Profiles: Advanced Provider Configuration¶
OpenFaaS Standard/for Enterprises
Profiles are part of OpenFaaS Standard/for Enterprises.
The OpenFaaS design allows it to provide a standard API across several different container orchestration tools: Kubernetes, containerd, and others. These faas-providers generally implement the same core features and allow your to functions to remain portable and be deployed on any certified OpenFaaS installation regardless of the orchestration layer. However, there are certain workloads or deployments that require more advanced features or fine tuning of configuration. To allow maximum flexibility without overloading the OpenFaaS function configuration, we have introduced the concept of Profiles. This is simply a reserved function annotation that the faas-provider can detect and use to apply the advanced configuration.
In some cases, there may be a 1:1 mapping between Profiles and Functions, this is to be expected for TopologySpreadConstraints, Affinity rules. We see no issue with performance or scalability.
In other cases, one Profile may serve more than one function, such as when using a toleration or a runtime class.
Multiple Profiles can be composed together for functions, if required.
Note: The general design is inspired by StorageClasses and IngressClasses in Kubernetes. If you are familiar with Kubernetes, these comparisons may be helpful, but they are not required to understand Profiles in OpenFaaS.
Using Profiles When You Deploy a Function¶
If you are a function author, using a Profile is a simple as adding an annotation to your function:
com.openfaas.profile: <profile_name>
You can do this with the faas-cli flags:
faas-cli deploy --annotation com.openfaas.profile=<profile_name>
Or in the stack YAML:
functions:
foo:
image: "..."
fprocess: "..."
annotations:
com.openfaas.profile: <profile_name>
If you need multiple profiles, you can use a comma separated value:
com.openfaas.profile: <profile_name1>,<profile_name2>
Profiles are created in the openfaas namespace, so typically will be created and maintained by Cluster Administrators.
Creating Profiles¶
Profiles must be pre-created, similar to Secrets, usually by the cluster admin. The OpenFaaS API does not provide a way to create Profiles because they are hyper specific to the orchestration tool.
Enable Profiles¶
When installing OpenFaaS on Kubernetes, Profiles use a CRD. This must be installed during or prior to start the OpenFaaS controller. When using the official Helm chart this will happen automatically. Alternatively, you can apply this YAML to install the CRD.
Available Options¶
Profiles in Kubernetes work by injecting the supplied configuration directly into the correct locations of the Function's Deployment. This allows us to directly expose the underlying API without any additional modifications. Currently, it exposes the following Pod and Container options from the Kubernetes API.
The configuration use the exact options that you find in the Kubernetes documentation.
Configuration priority
Resources set in the function spec take precedence over resources set through Profiles.
Examples¶
Implement the restricted Pod Security Standard¶
This example requires OpenFaaS for Enterprises with (faas-netes:0.5.65 or higher) and is aimed at securing enterprise and multi-tenant workloads.
Pod Security Standards were introduced in K8s v1.25 and are a set of best practices for securing your Pods. The restricted profile is the most secure option.
The below example deploys a function which will pass the restricted Pod Security Standard.
It defines:
- A new namespace for functions called
restricted-fn, which has been labeled withpod-security.kubernetes.io/enforce: restricted - A new Profile called
restrictedwhich sets the Pod Security Context to useRuntimeDefaultandrunAsNonRoot: true- any name can be used, or you could update an existing Profile that you're already using - A function called
envwhich uses therestrictedProfile
---
# Namespace "restricted-fn"
apiVersion: v1
kind: Namespace
metadata:
name: restricted-fn
labels:
kubernetes.io/metadata.name: dev
pod-security.kubernetes.io/enforce: restricted
annotations:
openfaas: "1"
---
# Profile "restricted"
apiVersion: openfaas.com/v1
kind: Profile
metadata:
name: restricted
namespace: openfaas
spec:
podSecurityContext:
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
---
# Function "restricted-fn"
apiVersion: openfaas.com/v1
kind: Function
metadata:
name: env
namespace: restricted-fn
spec:
name: env
image: ghcr.io/openfaas/alpine:latest
environment:
fprocess: "env"
annotations:
com.openfaas.profile: restricted
The securityContext for the container is not exposed as a separate configuration item since all required values (apart from capabilities are set at the Pod level instead.
By default, OpenFaaS for Enterprises will drop all Linux capabilities from the container. This is a requirement of the restricted Pod Security Standard.
The following will be added to the container's securityContext:
securityContext:
capabilities:
drop:
- ALL
allowPrivilegeEscalation: false
When the Helm chart has the following value set:
functions:
setNonRootUser: true
Then the following additional fields will be set in the container's securityContext:
securityContext:
runAsUser: 12000
runAsNonRoot: true
To upgrade existing functions, upgrade OpenFaaS via Helm, then run:
kubectl label namespace openfaas-fn pod-security.kubernetes.io/enforce=restricted
Then edit or redeploy each affected function to add the com.openfaas.profile: restricted annotation.
apiVersion: openfaas.com/v1
kind: Function
metadata:
name: env
namespace: restricted-fn
spec:
name: env
image: ghcr.io/openfaas/alpine:latest
environment:
fprocess: "env"
+ annotations:
+ com.openfaas.profile: restricted
If you are deploying functions via the REST API, update the deployment request as follows:
{
"name": "env",
"image": "ghcr.io/openfaas/alpine:latest",
"fprocess": "env",
"annotations": {
+ "com.openfaas.profile": "restricted"
}
}
The name restricted can be changed to any name you like, but you should include the podSecurityContext values as shown above.
If you're already using Profiles for your functions, then you can update the podSecurityContext values in the existing Profile.
Use an Alternative RuntimeClass¶
OpenFaaS for Enterprises
This feature is part of the OpenFaaS for Enterprises distribution.
A popular alternative container runtime class is gVisor that provides additional sandboxing between containers. If you have created a cluster that is using gVisor, you will need to set the runTimeClass on the Pods that are created. This is not exposed in the OpenFaaS API, but it can be set via a Profile.
-
Create a Profile to apply the runtime class
kubectl apply -f- << EOF kind: Profile apiVersion: openfaas.com/v1 metadata: name: gvisor namespace: openfaas spec: runtimeClassName: gvisor EOF -
Let your developers know that they need to use this annotation
com.openfaas.profile: gvisor
The following stack file will deploy a SHA512 generating file in a cluster with gVisor
provider:
name: openfaas
gateway: http://127.0.0.1:8080
functions:
stronghash:
skip_build: true
image: ghcr.io/openfaas/alpine:latest
fprocess: "sha512sum"
annotations:
com.openfaas.profile: gvisor
Set an elevated Pod priority with priorityClassName¶
In some cases, you may want to set a higher priority for certain functions to ensure they are scheduled first, or evicted last by the scheduler. This can be done by setting the priorityClassName in a Profile.
Example:
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: high-priority
value: 1000000
globalDefault: false
description: "This priority class should be used for XYZ service pods only."
---
apiVersion: openfaas.com/v1
kind: Profile
metadata:
name: high-priority-functions
namespace: openfaas
spec:
priorityClassName: "high-priority-functions"
Then add an annotation to the function:
com.openfaas.profile: high-priority-functions
Specify a nodeSelector to schedule functions to specific nodes¶
This example works for OpenFaaS Standard and OpenFaaS for Enterprises only, but you should consider using TopologySpreadConstraints or Affinity rules instead, which are more versatile.
"nodeSelector is the simplest recommended form of node selection constraint. You can add the nodeSelector field to your Pod specification and specify the node labels you want the target node to have. Kubernetes only schedules the Pod onto nodes that have each of the labels you specify."
Read the Kubernetes docs: Assign Pods to Nodes
Label a node as follows:
cat <<EOF > kind-nodeselectors.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraPortMappings:
- containerPort: 31112
hostPort: 31112
- role: worker
- role: worker
EOF
kind create cluster --config kind-nodeselectors.yaml
kubectl label node/kind-worker customer=1
kubectl label node/kind-worker2 customer=2
kind-worker will take on functions with a constraint of customer=1 and kind-worker2 will take on the workloads for customer 2.
Now deploy a function with a nodeselector:
cat <<EOF > stack.yml
version: 1.0
provider:
name: openfaas
gateway: http://127.0.0.1:8080
functions:
customer1-env:
skip_build: true
image: ghcr.io/openfaas/alpine:latest
fprocess: env
constraints:
- "customer=1"
customer2-env:
skip_build: true
image: ghcr.io/openfaas/alpine:latest
fprocess: env
constraints:
- "customer=1"
EOF
faas-cli deploy stack.yml
Confirm the scheduling:
$ kubectl get deploy -o wide -n openfaas-fn
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
customer1-env 1/1 1 1 18s customer1-env ghcr.io/openfaas/alpine:latest faas_function=customer1-env
customer2-env 1/1 1 1 18s customer2-env ghcr.io/openfaas/alpine:latest faas_function=customer2-env
This will also work if you have several nodes dedicated to a particular customer, just apply the label to each node and add the constraint at deployment time.
You may also want to consider using a taint and toleration to ensure OpenFaaS workload components do not get scheduled to these nodes.
Spreading your functions out across different zones for High Availability¶
The topologySpreadConstraints feature of Kubernetes provides a more flexible alternative to Pod Affinity / Anti-Affinity rules for scheduling functions.
"You can use topology spread constraints to control how Pods are spread across your cluster among failure-domains such as regions, zones, nodes, and other user-defined topology domains. This can help to achieve high availability as well as efficient resource utilization."
Imagine a cluster with two nodes, each in a different availability zone.
Let's simulate that with KinD:
cat <<EOF > kind-zones.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraPortMappings:
- containerPort: 31112
hostPort: 31112
- role: worker
- role: worker
EOF
kind create cluster --config kind-zones.yaml
kubectl label node/kind-worker topology.kubernetes.io/zone=a
kubectl label node/kind-worker2 topology.kubernetes.io/zone=b
Deploy a profile called env-tsc
kind: Profile
apiVersion: openfaas.com/v1
metadata:
name: env-tsc
namespace: openfaas
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
faas_function: env
Deploy a function with this profile:
faas-cli store deploy env \
--annotation com.openfaas.profile=tsc
Scale the function to 6 replicas:
kubectl scale -n openfaas-fn deploy/env --replicas=6
Notice how the pods are spread evenly between the nodes in the two zones:
kubectl get pod -n openfaas-fn -o wide -w
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
env-5f5f697594-5clh6 1/1 Running 0 11s 10.244.2.13 kind-worker <none> <none>
env-5f5f697594-6fxhq 1/1 Running 0 21s 10.244.2.12 kind-worker <none> <none>
env-5f5f697594-92ntn 1/1 Running 0 11s 10.244.1.7 kind-worker2 <none> <none>
env-5f5f697594-bz6sb 1/1 Running 0 11s 10.244.1.8 kind-worker2 <none> <none>
env-5f5f697594-hsx98 1/1 Running 0 21s 10.244.1.6 kind-worker2 <none> <none>
env-5f5f697594-nqxbh 1/1 Running 0 24s 10.244.2.11 kind-worker <none> <none>
A note on whenUnsatisfiable:
The constraint of whenUnsatisfiable: DoNotSchedule will mean pods are not scheduled if they cannot be balanced evenly. This may become an issue for you if your nodes are of difference sizes, therefore you may also want to consider changing this value to ScheduleAnyway
Use Tolerations and Affinity to Separate Workloads¶
This example is for OpenFaaS Pro because it uses Affinity.
While the OpenFaaS API exposes the Kubernetes NodeSelector via constraints, affinity/anti-affinity and taint/tolerations can be used to further expand the types of constraints you can express. OpenFaaS Profiles allow you to set these options. They allow you to more accurately isolate workloads, keep certain workloads together on the same nodes, or to keep certain workloads separate.
For example, a mixture of taints and affinity can put less critical functions on preemptable vms that are cheaper while keeping critical functions on standard nodes with higher availability guarantees.
In this example, we create a Profile using taints and affinity to place functions on the node with NVME storage. We will also ensure that only functions that require NVME are scheduled on these nodes. This ensures that the functions that need to faster storage are not blocked by other standard functions taking resources on these special nodes.
-
Install the latest
faas-netesrelease and the CRD. The is most easily done witharkadearkade install openfaas \ --set openfaasPro=trueThis default installation will enable Profiles.
-
Label and Taint the node with the NVME
kubectl label nodes <NODE_NAME> nvme=installed kubectl taint nodes <NODE_NAME> nvme:NoSchedule -
Create a Profile to that allows functions to run on this node
kubectl apply -f- << EOF kind: Profile apiVersion: openfaas.com/v1 metadata: name: withnvme namespace: openfaas spec: tolerations: - key: "nvme" operator: "Exists" effect: "NoSchedule" affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: nvme operator: In values: - installed EOF -
Then add this annotation to your stack.yml file:
com.openfaas.profile: withnvme
Configure DNS for function pods¶
There are cases when you might want to set a custom DNS configuration per function instead of using the cluster level DNS settings. For example if you are building a multi tenant functions platform and need different DNS configuration for functions from different tenants. Profiles support setting the dnsPolicy and dnsConfig for a function pod.
-
Create a profile with a custom DNS configuration. In this example we configure custom nameservers.
kubectl apply -f- << EOF kind: Profile apiVersion: openfaas.com/v1 metadata: name: function-dns namespace: openfaas spec: dnsPolicy: None dnsConfig: nameservers: - "8.8.8.8" - "1.1.1.1" EOF -
Deploy a function with a profile annotation to apply the dns profile.
faas-cli store deploy nslookup --annotation "com.openfaas.profile=function-dns"
Invoke to nslookup function to see it uses the custom nameservers.
echo openfaas.com | faas-cli invoke nslookup
Server: 8.8.8.8
Address: 8.8.8.8#53
Non-authoritative answer:
Name: openfaas.com
Address: 185.199.108.153
Name: openfaas.com
Address: 185.199.109.153
Name: openfaas.com
Address: 185.199.111.153
Name: openfaas.com
Address: 185.199.110.153
Schedule functions with GPU¶
You will have to make sure GPU nodes in your cluster are set up with GPU drivers and run the corresponding device plugin from the GPU vendor. See the kubernetes documentation for detailed information on scheduling GPUs
Once you have installed the plugin, your cluster exposes a custom schedulable resource such as amd.com/gpu or nvidia.com/gpu. These are not exposed through the resources in the OpenFaaS Function spec but can be applied using Profiles.
Here's an example of a Profile that requests one NVIDIA GPU for a function:
kind: Profile
apiVersion: openfaas.com/v1
metadata:
name: nvidia-gpu
namespace: openfaas
spec:
runtimeClass: nvidia
resources:
limits:
nvidia.com/gpu: 1 # requesting 1 GPU
Note: runtimeClass also needs to be set to use the relevant container runtime if your cluster has multiple runtimes.
Add this profile to the cluster and use the com.openfaas.profile annotation to apply the profile to functions that need access to a GPU:
com.openfaas.profile: nvidia-gpu
With the default RollingUpdate strategy, updating a function is not possible if all GPUs are in use.
Kubernetes will try to create a new function pod before shutting down the old one but the newly created pod can not start because no more free GPUs are available.
You may want to consider switching the update strategy for functions using GPU to Recreate if you plan on using all available GPUs in your cluster. Keep in mind that this may cause the function to be offline for a moment during updates.
The update strategy type for function deployments can be added to the profile:
kind: Profile
apiVersion: openfaas.com/v1
metadata:
name: nvidia-gpu
namespace: openfaas
spec:
runtimeClass: nvidia
+ strategy:
+ type: Recreate
resources:
limits:
nvidia.com/gpu: 1 # requesting 1 GPU
Set default RAM/CPU for all functions¶
You might want to set default memory and CPU resources for all your functions. This can be done by creating a Profile and applying it to all your functions by default.
Example of a profile that sets Memory/CPU limits and requests:
kind: Profile
apiVersion: openfaas.com/v1
metadata:
name: default-resources
namespace: openfaas
spec:
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
Add the profiles annotation to all functions to apply this profile.
com.openfaas.profile: default-resources
It is still possible to override the default settings on a per function basis by setting different values in the function stack.yaml: see Memory/CPU limits. Resources set in the function spec take precedence over resources set through Profiles.