Validate Your Network with pytest - Part 1 - Setup

Published: 2021-07-30
Author: Tom Ammon

What is pytest?

Understanding the operational state of a network is a challenging problem. Understanding the operational readiness of the software that runs the control plane of your network is even more challenging. As our networks grow more complex and more mission-critical, we are being forced to find new ways to verify that they are operating in their intended state. The pytest framework can give us that understanding. pytest “makes it easy to write small tests, yet scales to support complex functional testing for applications and libraries." With the help of a few plugins, we can use pytest to measure the operational state of a network against our own arbitrary criteria. This blog series will give you a running start at using pytest to test your network’s operational state.

Multipart Series

This article is part of a multipart series. Here are the other parts:

Lab Setup

This section will focus on the implementation details of the lab I am using to demonstrate the concepts in the rest of the series. If you’re not planning to build out your own version of this environment, you can skip this section and go straight to pytest Hello World - there is nothing particularly applicable to pytest in the lab setup, it’s just giving us some virtual devices to test against.

For all of the examples in this series I am launching pytest from a desktop PC running Ubuntu 20. The devices under test (DUTs) are Debian 10 VMs and Cisco CSR1000v VMs, running on a Debian 10 KVM hypervisor. Here’s how things are connected:

Topology

Lab Topology

A few notes about the topology:

Development environment

We’ll need to install a few things on our workstation to get our development environment ready:

We’ll use a python virtual environment to keep our work contained. I use pyenv with virtualenv. If you’re not familiar with pyenv, here’s a great tutorial.

Inside your virtual environment, install python packages:

$ pip install pytest pytest-testinfra paramiko
Collecting pytest
  Using cached pytest-6.2.4-py3-none-any.whl (280 kB)
Collecting pytest-testinfra
  Using cached pytest_testinfra-6.4.0-py3-none-any.whl (71 kB)
Collecting paramiko
  Using cached paramiko-2.7.2-py2.py3-none-any.whl (206 kB)
Collecting packaging
  Using cached packaging-21.0-py3-none-any.whl (40 kB)
Collecting iniconfig
  Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting py>=1.8.2
  Using cached py-1.10.0-py2.py3-none-any.whl (97 kB)
Collecting pluggy<1.0.0a1,>=0.12
  Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting toml
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting attrs>=19.2.0
  Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting bcrypt>=3.1.3
  Using cached bcrypt-3.2.0-cp36-abi3-manylinux2010_x86_64.whl (63 kB)
Collecting pynacl>=1.0.1
  Using cached PyNaCl-1.4.0-cp35-abi3-manylinux1_x86_64.whl (961 kB)
Collecting cryptography>=2.5
  Using cached cryptography-3.4.7-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
Collecting pyparsing>=2.0.2
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting cffi>=1.1
  Using cached cffi-1.14.6-cp39-cp39-manylinux1_x86_64.whl (405 kB)
Collecting six>=1.4.1
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting pycparser
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Installing collected packages: pyparsing, packaging, iniconfig, py, pluggy, toml, attrs, 
pytest, pytest-testinfra, pycparser, cffi, six, bcrypt, pynacl, cryptography, paramiko
Successfully installed attrs-21.2.0 bcrypt-3.2.0 cffi-1.14.6 cryptography-3.4.7 
iniconfig-1.1.1 packaging-21.0 paramiko-2.7.2 pluggy-0.13.1 py-1.10.0 pycparser-2.20 
pynacl-1.4.0 pyparsing-2.4.7 pytest-6.2.4 pytest-testinfra-6.4.0 six-1.16.0 toml-0.10.2

We won’t be using pytest-testinfra or paramiko right away, but we will need them later on.

Now our development environment is ready to go.

pytest Hello World

With our environment set up, we can finally get down to writing code. Let’s write a single test, execute it, and discuss what happened. I’m stealing part of this from the excellent pytest docs. You’ll also want to become familiar with how assert works in python, here’s a quick reference.

1
2
3
4
5
6

# contents of test_first.py

print("this is the start of my test")

def test_myfirsttest():
    assert 5 == 10

Now, it’s time to invoke pytest at the command line. If you’re following along on your own computer (which I highly recommend), you’ll also notice that pytest colorizes the output to make it easier to spot the failed tests.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


$ pytest test_first.py
================================ test session starts ================================
platform linux -- Python 3.9.4, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/tom/local/repos/blogrepos/network-validation-with-pytest
plugins: testinfra-6.4.0
collected 1 item                                                                    

test_first.py F                                                               [100%]

===================================== FAILURES ======================================
_________________________________ test_myfirsttest __________________________________

    def test_myfirsttest():
>       assert 5 == 10
E       assert 5 == 10

test_first.py:6: AssertionError
============================== short test summary info ==============================
FAILED test_first.py::test_myfirsttest - assert 5 == 10
================================= 1 failed in 0.02s =================================

What’s happening here?

pytest found a function prefixed with “test” on line 5 in test_first.py and executed it for us. By default, pytest runs all functions prefixed with “test”. In pytest parlance, this behavior is called “test discovery”, see the docs for more details.

Look at the output on line 16, and you’ll see that pytest is reporting the results for each individual function/test that failed, in this case, that was test_myfirsttest(). Can you see why this test failed?

Line 18 in the output reveals that the test failed with an “AssertionError” - this means that the condition in the test following the assert did not evaluate to True. In this case, 5 is not equal to 10, and this is what produced the AssertionError. Notice that pytest also prints out some of the python code around the part of the test that failed, and even points you to the line in the file with the failing assert: test_first.py:6: AssertionError. All of this will become very valuable as your library of tests grows.

One last thing: Notice that the print statement didn’t execute. This is because pytest is executing only functions prefixed with “test”. This is another example of how test discovery works.

Wrap Up

We’ve installed pytest and written a very simple Hello World test. You probably noticed that this test isn’t very useful - comparing the value of two integers on your desktop computer is fun (for some definition of “fun”), but it won’t help you validate the current running state of your network. In the next article, we’ll test some conditions on an actual remote linux router, including the status of a BGP peer - and we’ll see how pytest reacts when that BGP session goes down.

References

Tags: pytest, python