Lucas will show you how to test OpenFaaS functions written in Python with the Pytest framework.

In a recent Pull Request, we integrated unit-testing to the build process of the OpenFaaS Python Flask templates. Since there are multiple test frameworks including pytest, we opted to integrate with a test runner called tox, so that you can pick whichever unit test framework you prefer.

The upshot is that you can focus on implementing your functions and tests and then let OpenFaaS handle the rest. The faas-cli up command can then be run locally or in a CI/CD environment to build, test, and deploy your Python functions with a single.

In blog post, I’ll show you how to take advantage of unit tests written with Pytest in your OpenFaaS workflow.

Introduction

In this post we are going to build a very small calculator function and then write a few tests that show how we can ensure our calculator works before we deploy it. We will show you how to run the tests locally during development and then show how this is integrated into the OpenFaaS build flow so that you you can run the tests automatically in your CI/CD flows with a single command.

Setup the project

All of the code in this example can be found on Github, if you are already familiar with the python3-flask template and pytest, then you can jump ahead and see what the final implementation looks like.

Fetch the template from the store:

$ mkdir pytest-sample
$ cd pytest-sample
$ faas-cli template store pull python3-flask

Create a new function called calc and then rename its YAML file to the default stack.yml, to avoid needing the -f flag later on.

$ faas-cli new --lang python3-flask calc
$ mv calc.yml stack.yml

Since the templates are never committed to Git, each time someone clones the repository, they would need to run faas-cli template store pull. Fortunately, there is a work-around which adds the template name and source to the stack.yml file to automate this task.

configuration:
  templates:
    - name: python3-flask
      source: https://github.com/openfaas/python-flask-template

Setup the local python environment

It’s possible to install dependencies like flask, tox and pytest directly to your system using pip3 install, however Python practitioners will often use a virtual environment so that each project or function can have a different version of dependencies.

See also: RealPython’s primer on virtual environments

I use conda for my local virtual environments, but you can of course use virtualenv or venv.

In short this creates an isolated and repeatable development environment which you can delete and recreate if you need.

$ conda create -n pytest-sample tox
$ conda activate pytest-sample
$ cat <<EOF >> requirements.txt
pydantic==1.7.3
flask==1.1.2
EOF

$ cat <<EOF >> dev.txt
tox
pytest
black
pylint
EOF
$ conda install --yes --file requirements.txt
$ conda install --yes --file dev.txt

Now we are ready to develop the calculator.

The calculator implementation

The calculator will be a simple web endpoint that accepts payloads like the following:

{
"op":"+",
"var1":"1",
"var2":"2"
}

This is the output you can expect:

{
"value":"3"
}

There are several ways that this request could fail. To help validate the request (and to give us more a few more things to test) we can use pydantic to do the hard validation work and keep our function lean.

Update your requirements.txt to include

pydantic==1.7.3

Put this into your handler.py:

from pydantic import BaseModel, ValidationError
from enum import Enum, unique


@unique
class OperationType(Enum):
    ADD = "+"
    SUBTRACT = "-"
    MULTIPLY = "*"
    DIVIDE = "/"
    POWER = "^"


class Calculation(BaseModel):
    op: OperationType
    var1: float
    var2: float

    def execute(self) -> float:
        if self.op is OperationType.ADD:
            return self.var1 + self.var2

        if self.op is OperationType.MULTIPLY:
            return self.var1 * self.var2

        raise ValueError("unknown operation")


def handle(req) -> (dict, int):
    """handle a request to the function
    Args:
        req (str): request body
    """

    try:
        c = Calculation.parse_raw(req)
    except ValidationError as e:
        return {"message": e.errors()}, 422
    except Exception as e:
        return {"message": e}, 500

    return {"value": c.execute()}, 200

At this point we could deploy the function and use it

$ faas-cli deploy
Deploying: calc.

Deployed. 202 Accepted.
URL: http://127.0.0.1:8080/function/calc
$ echo '{"op":"+", "var1": 1, "var2": 1}' | faas-cli invoke calc
{"value":2.0}

We can see the nice work pydantic does for us by sending an empty {} payload

$  echo '{}' | faas-cli invoke calc
Server returned unexpected status code: 422 - {"message":[{"loc":["op"],"msg":"field required","type":"value_error.missing"},{"loc":["var1"],"msg":"field required","type":"value_error.missing"},{"loc":["var2"],"msg":"field required","type":"value_error.missing"}]}

Adding tests

pytest is a popular testing framework that provides automated test discovery and detailed info on failing assert statements, among other features. We will setup our project so that pytest works out of the box, this means we will name the test files *_test.py and prefix out test functions with test_.

Create a new file in your function

$ touch calc/handler_test.py

then add the following test cases for a couple of our happy paths

from . import handler as h

class TestParsing:
    def test_operation_addition(self):
        req = '{"op": "+", "var1": "1.0", "var2": 0}'
        resp, code = h.handle(req)
        assert code == 200
        assert resp["value"] == 1.0

    def test_operation_multiplication(self):
        req = '{"op": "*", "var1": "100.01", "var2": 1}'
        resp, code = h.handle(req)
        assert code == 200
        assert resp["value"] == 100.01

Note that we import then handler from ., we don’t use an absolute import like from calc import handler as h. This is required to be compatible with the the OpenFaaS build process.

Now, change into the function directory and run the tests using pytest

$ pytest
==================== test session starts =====================
platform linux -- Python 3.8.8, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: pytest-sample/calc
collected 2 items

handler_test.py ..                                     [100%]

===================== 2 passed in 0.03s ======================

If you want to see an error, just change the assertion in one of the tests to a “wrong” value and run pytest again:

pytest
==================== test session starts =====================
platform linux -- Python 3.8.8, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /home/lucas/code/openfaas/sandbox/pytest-sample/calc
collected 2 items

handler_test.py F.                                     [100%]

========================== FAILURES ==========================
____________ TestParsing.test_operation_addition _____________

self = <calc.handler_test.TestParsing object at 0x7f01f8a08400>

    def test_operation_addition(self):
        req = '{"op": "+", "var1": "1.0", "var2": 0}'
        resp, code = h.handle(req)
        assert code == 200
>       assert resp["value"] == 2.0
E       assert 1.0 == 2.0

handler_test.py:51: AssertionError
================== short test summary info ===================
FAILED handler_test.py::TestParsing::test_operation_addition
================ 1 failed, 1 passed in 0.04s =================

A test for the validation will look like

def test_operation_parsing_error_on_empty_obj(self):
    req = '{}'

    resp, code = h.handle(req)
    assert code == 422
    # should be a list of error
    errors = resp.get("message", [])
    assert len(errors) == 3
    assert errors[0].get("loc") == ('op', )
    assert errors[0].get("msg") == "field required"

    assert errors[1].get("loc") == ('var1', )
    assert errors[1].get("msg") == "field required"

    assert errors[2].get("loc") == ('var2', )
    assert errors[2].get("msg") == "field required"

Checkout the example repo for the other example tests.

Integrate testing into the OpenFaaS workflow

The python3-flask template can run pytest unit tests automatically. If one of your tests fails, the build will fail and you can see the pytest output

$ faas-cli build
# truncated ....
#24 7.477 test run-test: commands[0] | pytest
#24 7.662 ============================= test session starts ==============================
#24 7.662 platform linux -- Python 3.7.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
#24 7.662 cachedir: .tox/test/.pytest_cache
#24 7.662 rootdir: /home/app/function
#24 7.662 collected 8 items
#24 7.662
#24 7.662 handler_test.py ...F....                                                 [100%]
#24 7.689
#24 7.689 =================================== FAILURES ===================================
#24 7.689 _____________________ TestParsing.test_operation_addition ______________________
#24 7.689
#24 7.689 self = <function.handler_test.TestParsing object at 0x7fc06f52bf50>
#24 7.689
#24 7.689     def test_operation_addition(self):
#24 7.689         req = '{"op": "+", "var1": "1.0", "var2": 0}'
#24 7.689         resp, code = h.handle(req)
#24 7.689         assert code == 200
#24 7.689 >       assert resp["value"] == 2.0
#24 7.689 E       assert 1.0 == 2.0
#24 7.689
#24 7.689 handler_test.py:51: AssertionError
#24 7.689 =========================== short test summary info ============================
#24 7.689 FAILED handler_test.py::TestParsing::test_operation_addition - assert 1.0 == 2.0
#24 7.689 ========================= 1 failed, 7 passed in 0.07s ==========================
#24 7.709 ERROR: InvocationError for command /home/app/function/.tox/test/bin/pytest (exited with code 1)
#24 7.709 ___________________________________ summary ____________________________________
#24 7.709   lint: commands succeeded
#24 7.709 ERROR:   test: commands failed
#24 ERROR: executor failed running [/bin/sh -c if [ "$TEST_ENABLED" == "false" ]; then     echo "skipping tests";    else     eval "$TEST_COMMAND";     fi]: exit code: 1
------
 > [stage-1 18/19] RUN if [ "true" == "false" ]; then     echo "skipping tests";    else     eval "tox";     fi:
------
executor failed running [/bin/sh -c if [ "$TEST_ENABLED" == "false" ]; then     echo "skipping tests";    else     eval "$TEST_COMMAND";     fi]: exit code: 1
[0] < Building calc done in 16.67s.
# truncated ....

If you are working locally and need to disable the tests for a build, you can use the build arg --build-arg TEST_ENABLED=false.

Running the tests in CI

If you are a fan of Github Actions, you only need two steps:

- name: Setup tools
  env:
    ARKADE_VERSION: "0.6.21"
  run: |
    curl -SLs https://github.com/alexellis/arkade/releases/download/$ARKADE_VERSION/arkade > arkade
    chmod +x ./arkade
    ./arkade get faas-cli

- name: Build and Test Functions
  run: faaas-cli build

Here the arkade tool created by the OpenFaaS community is used as a downloader for faas-cli.

Wrapping up

In this post we’ve shown how to add unit tests to your Python functions and how to run those tests in your local development and CI environments.

Do you have any tips and tricks for testing in Python? Let us know on Twitter @openfaas or join us on on Slack to chat more.

Would you like to keep learning? The Python 3 template is a core part of the new Introduction to Serverless course by the LinuxFoundation

If Python is not your language of choice, then the Go, Node12, and the Node14 templates also have testing integrated into the build process.

Would like to have automated testing in your favorite language template? Checkout out the implementation in the python3-flask template and let us know how to adapt it to your favorite template.

Lucas Roesler

Core Team @openfaas. Senior Engineer and Team Lead @contiamo