Retries for functions¶
OpenFaaS Pro offers an upgraded queue-worker which allows for messages to be retried a number of times using an exponential back-off algorithm to mitigate the impact associated with retrying messages.
Note: This feature is included for OpenFaaS Pro customers.
There are two primary use-cases for retrying asynchronous messages:
- When a downstream API is likely to fail some of the time, and cannot be fixed easily, retrying can work around this problem
- When
max_inflight
has been set on a function such as a web scraper, or memory-intensive function and you do not want to overload the function. When a function cannot accept any more connections due to the inflight setting, it will return a 429 error, which indicates the message can be retried at a later time.
Installation¶
You can install the Pro version of the queue worker by editing the values.yaml file of the OpenFaaS chart.
You must also set:
openfaasPro: true
queueWorkerPro:
maxRetryAttempts: "10"
maxRetryWait: "120s"
initialRetryWait: "10s"
httpRetryCodes: "429,502,500,504,408"
Timings are specified in Golang durations such as 10s
or 1m
.
initialRetryWait
is the amount of time to wait for the first retry, this can be set 0s if desired, or a higher value.maxRetryWait
is the maximum amount of time to wait between retries.maxRetryAttempts
is the amount of times to try sending a message to a function before discarding it.httpRetryCodes
is a comma-separated list of HTTP status codes which the queue worker will retry when received from a function.
Usage¶
To test the retry functionality, you can use our chaos function, which allows a function to be configured to return a canned response, or to timeout with a given duration.
We will configure a testing function to output a 429 HTTP response, which indicates that the function is too busy to process the request. This is one of the core use-cases for retrying an invocation. Once we've observed the retrying functionality, we will update the function to return a 200 response, then we'll see the request complete.
You can also add a long timeout or a different HTTP code, see the testing function's notes on GitHub: alexellis/chaos-fn
git clone https://github.com/alexellis/chaos-fn --depth=1
cd chaos-fn
faas-cli template pull stack -f chaos-fn.yml
# Deploy the function
faas-cli deploy -f chaos-fn.yml
Open a separate terminal to observe the retrying mechanism with the logs of the queue-worker.
kubectl logs deploy/queue-worker -n openfaas -f
Configure the chaos-fn to start failing with a 429
error code:
export CODE=429
curl -i http://127.0.0.1:8080/function/chaos-fn/set --data-binary '
{ "status": '$CODE',
"delay": "1s",
"body": "1 second delay, then '$CODE'"}
' --header "Content-type: application/json"
You will always see a 202 Accepted
from the /set
path, if you see an error then check the format of your request. The status code must be an integer.
Invoke the function synchronously, to see that it's returning the expected error code:
curl -i http://127.0.0.1:8080/function/chaos-fn
HTTP/1.1 429 Conflict
Content-Length: 24
Content-Type: text/plain; charset=utf-8
Date: Tue, 08 Mar 2022 11:37:01 GMT
X-Call-Id: 24ae517c-ad0d-4654-8160-d2a7f7d68fab
X-Duration-Seconds: 1.001162
X-Start-Time: 1646739420463153513
1 second delay, then 429
Now invoke the function asynchronously.
curl -i http://127.0.0.1:8080/async-function/chaos-fn -d "input"
The invocation will fail, returning a 429 message. This will be received by the queue-worker which will then retry the request. Check the logs of the queue-worker in the other terminal to see this happening.
Allow it to fail several times, seeing the retry time back off exponentially.
For undelivered messages that reach maximum retries the response is posted to the callback url.
Then, whenever you like, fix the error by changing the canned HTTP response to "200 OK". This will allow the next retry to complete:
curl -i http://127.0.0.1:8080/function/chaos-fn/set --data-binary '
{"status": 200,
"delay": "1ms",
"body": "1ms second delay, then 200"}
' --header "Content-type: application/json"
Would you like a demo?¶
Feel free to reach out to us for a demo or to ask any questions you may have.