Validate Your Network with pytest - Part 2 - Kick the Tires

Published: 2021-08-03
Author: Tom Ammon

This article is part of a multipart series that introduces pytest as a tool to validate the operational state of a network. We’ve validated that our lab and development environment are ready to go. Now it’s time to see pytest in action on a real network. If you weren’t here for Part 1, go take a look at it. Otherwise, let’s get to writing some tests!

Articles in this Series

Test a Remote Device

Let’s take this one step at a time.

We’re going to do something simple that has nothing to do with network infrastructure, to help us understand basic usage of pytest. Then, once we’ve got our legs under us, we can layer on more logic and testing of the network.

We’ll start off by connecting to one our linux routers, specifically we’ll be looking at debian-1. We’ll do a simple test to see if a file on the filesystem of that router contain the contents we expect it to.

Here’s the file, /opt/TomStatus.txt:

root@debian-1:~# echo "Tom is a goofball" > /opt/TomStatus.txt
root@debian-1:~# cat /opt/TomStatus.txt 
Tom is a goofball

Build the Test

You can find the complete source for examples used in this article in its Github repo.

# we'll connect to devices and run commands a lot, so let's 
# put this component into its own function right off the bat.
def run_remote_command(host, cmd):
    output = host.run(cmd)
    return output

# here's the actual test function that pytest will execute. 
# we get the contents of /opt/TomStatus.txt from the remote
# device, and compare it to what we expect those contents to be
def test_file_contents(host):
    command = 'cat /opt/TomStatus.txt'
    filecontents = run_remote_command(host, command)
    # if the output of the cat command above, against the host
    # we specified in --hosts at the CLI, matches the string below,
    # this test passes
    assert filecontents.stdout.strip() == 'Tom is a goofball'

Run the Test

Now let’s fire it up. We’re running with more verbosity to make the example more meaningful.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
$ pytest test_debian1.py --hosts 100.66.12.202 -v
============================== test session starts ===============================
platform linux -- Python 3.9.4, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /home/
tom/.pyenv/versions/3.9.4/envs/network-pytest/bin/python3.9
cachedir: .pytest_cache
rootdir: /home/tom/local/repos/blogrepos/network-validation-with-pytest
plugins: testinfra-6.4.0
collected 1 item                                                                 

test_debian1.py::test_file_contents[paramiko://100.66.12.202] PASSED       [100%]

================================ warnings summary ================================
test_debian1.py::test_file_contents[paramiko://100.66.12.202]
  /home/tom/.pyenv/versions/3.9.4/envs/network-pytest/lib/python3.9/site-packages/
  paramiko/client.py:835: UserWarning: Unknown ssh-rsa host key for 100.66.12.202: 
  b'13e0914173a54983f693acbdc444228c'  warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================== 1 passed, 1 warning in 0.37s ==========================

You can ignore the warning about the ssh key for now

We see in line 10 the name of the specific test function that passed, as well as which remote device it passed on (100.66.12.202). Line 19 gives us a summary of all the results. This summary will become more useful later on when we have a lot of tests to run.

Notice here, that a passing test doesn’t tell you what caused the test to pass. Now let’s see what happens when a test fails.

Let’s break this thing

To make a test fail, we’ll update the contents of /opt/TomStatus.txt and run again.

root@debian-1:~# echo "Tom is a goofballl" > /opt/TomStatus.txt
root@debian-1:~# cat /opt/TomStatus.txt
Tom is a goofballl

Now kick off the test against your freshly-damaged target host. Notice the highlighted lines, then we’ll walk through them after the output.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
$ pytest test_debian1.py --hosts 100.66.12.202 -v
============================== test session starts ===============================
platform linux -- Python 3.9.4, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /home/
tom/.pyenv/versions/3.9.4/envs/network-pytest/bin/python3.9
cachedir: .pytest_cache
rootdir: /home/tom/local/repos/blogrepos/network-validation-with-pytest
plugins: testinfra-6.4.0
collected 1 item                                                                 

test_debian1.py::test_file_contents[paramiko://100.66.12.202] FAILED       [100%]

==================================== FAILURES ====================================
__________________ test_file_contents[paramiko://100.66.12.202] __________________

host = <testinfra.host.Host paramiko://100.66.12.202>

    def test_file_contents(host):
        command = 'cat /opt/TomStatus.txt'
        filecontents = run_remote_command(host, command)
>       assert filecontents.stdout.strip() == 'Tom is a goofball'
E       AssertionError: assert 'Tom is a goofballl' == 'Tom is a goofball'
E         - Tom is a goofball
E         + Tom is a goofballl
E         ?                  +

test_debian1.py:13: AssertionError
================================ warnings summary ================================
test_debian1.py::test_file_contents[paramiko://100.66.12.202]
  /home/tom/.pyenv/versions/3.9.4/envs/network-pytest/lib/python3.9/site-packages
  /paramiko/client.py:835: UserWarning: Unknown ssh-rsa host key for 100.66.12.202: 
  b'13e0914173a54983f693acbdc444228c'  warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/warnings.html
============================ short test summary info =============================
FAILED test_debian1.py::test_file_contents[paramiko://100.66.12.202] - Assertio...
========================== 1 failed, 1 warning in 0.35s ==========================

Here’s what failure looks like. pytest breaks it down for us in great detail:

Test the State of the Network

Enough of this Hello World stuff - let’s check out the network. We’re working with a very simple topology. The router debian-1 is directly connected to the router debian-2, both are running FRR, and have established one BGP session with each other over their directly-connected link. Here’s the BGP configuration from debian-1:

root@debian-1:~# vtysh

Hello, this is FRRouting (version 7.5.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

debian-1.ammonfamily.net# sh run
Building configuration...
(...output omitted...)
!
router bgp 100
 no bgp ebgp-requires-policy
 neighbor 192.168.1.2 remote-as 101
 !
 address-family ipv4 unicast
  network 1.1.1.1/32
 exit-address-family
!
line vty
!
end

Before we start writing our test, let’s see what the healthy BGP session looks like:

root@debian-1.ammonfamily.net:~# vtysh

Hello, this is FRRouting (version 7.5.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

debian-1.ammonfamily.net# sh ip bgp summ

IPv4 Unicast Summary:
BGP router identifier 1.1.1.1, local AS number 100 vrf-id 0
BGP table version 2
RIB entries 3, using 576 bytes of memory
Peers 1, using 21 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
192.168.1.2     4        101        40        40        0    0    0 00:35:20            1        2

Total number of neighbors 1

Also, as an aside, FRR will provide json output for a lot of its state monitoring commands. Here’s the same data as above, formatted as json. I think you can see where this is going:

debian-1.ammonfamily.net# sh ip bgp summ json
{
"ipv4Unicast":{
  "routerId":"1.1.1.1",
  "as":100,
  "vrfId":0,
  "vrfName":"default",
  "tableVersion":2,
  "ribCount":3,
  "ribMemory":576,
  "peerCount":1,
  "peerMemory":21816,
  "peers":{
    "192.168.1.2":{
      "hostname":"debian-2.ammonfamily.net",
      "remoteAs":101,
      "version":4,
      "msgRcvd":41,
      "msgSent":41,
      "tableVersion":0,
      "outq":0,
      "inq":0,
      "peerUptime":"00:36:18",
      "peerUptimeMsec":2178000,
      "peerUptimeEstablishedEpoch":1627955534,
      "pfxRcd":1,
      "pfxSnt":2,
      "state":"Established",
      "connectionsEstablished":1,
      "connectionsDropped":0,
      "idType":"ipv4"
    }
  },
  "failedPeers":0,
  "totalPeers":1,
  "dynamicPeers":0,
  "bestPath":{
    "multiPathRelax":"false"
  }
}
}

The neighbor is up, some prefixes have been sent and received, and life seems pretty good so far.

Build the Test

Our first test will be simple. We know that on debian-1, we should see a BGP session up and in Established state with debian-2 (192.168.1.2). So let’s write the test.

# This is the same function we used last time, to connect to the device
# and capture the output and exit code generated from the device
def run_remote_command(host, cmd):
    output = host.run(cmd)
    return output

def test_bgp_peer_up(host):
    # FRR syntax to request output in json
    command = 'vtysh -c "show ip bgp summary json"'
    output = run_remote_command(host, command)
    # deserialize the json object found in output.stdout into the bgp_data dict
    bgp_data = json.loads(output.stdout)
    # find the session state in the dict and ensure that it meets our criteria
    assert bgp_data['ipv4Unicast']['peers']['192.168.1.2']['state'] == "Established"

A few notes here:

Run the Test

$ pytest test_debian1.py --hosts 100.66.12.202 -v
============================== test session starts ===============================
platform linux -- Python 3.9.4, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /home/tom/.pyenv...
cachedir: .pytest_cache
rootdir: /home/tom/local/repos/blogrepos/network-validation-with-pytest
plugins: testinfra-6.4.0
collected 2 items                                                                

test_debian1.py::test_file_contents[paramiko://100.66.12.202] PASSED       [ 50%]
test_debian1.py::test_bgp_peer_up[paramiko://100.66.12.202] PASSED         [100%]

================================ warnings summary ================================
test_debian1.py::test_file_contents[paramiko://100.66.12.202]
  /home/tom/.pyenv/versions/3.9.4/envs/network-pytest/lib/python3.9/site-packages/...
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================== 2 passed, 1 warning in 0.49s ==========================

Cool! Our test passed. Based on the data we obtained manually before we wrote the test, this is not a surprise. Let’s see what pytest does when BGP is unhappy.

Let’s break this thing

We’ll drop the BGP session and then experience the chaos:

debian-1.ammonfamily.net# conf t
debian-1.ammonfamily.net(config)# router bgp
debian-1.ammonfamily.net(config-router)# neighbor 192.168.1.2 shutdown
debian-1.ammonfamily.net(config-router)# 
debian-1.ammonfamily.net(config-router)# do sh ip bgp summ

IPv4 Unicast Summary:
BGP router identifier 1.1.1.1, local AS number 100 vrf-id 0
BGP table version 3
RIB entries 1, using 192 bytes of memory
Peers 1, using 21 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
192.168.1.2     4        101        58        60        0    0    0 00:00:07 Idle (Admin)        0

Total number of neighbors 1

It’s pretty clear from the State/PfxRcd column that the BGP session is not where we want it to be. Now let’s see what pytest thinks of the situation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
pytest test_debian1.py --hosts 100.66.12.202 -v
============================== test session starts ===============================
platform linux -- Python 3.9.4, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /home/tom/...
cachedir: .pytest_cache
rootdir: /home/tom/local/repos/blogrepos/network-validation-with-pytest
plugins: testinfra-6.4.0
collected 2 items                                                                

test_debian1.py::test_file_contents[paramiko://100.66.12.202] PASSED       [ 50%]
test_debian1.py::test_bgp_peer_up[paramiko://100.66.12.202] FAILED         [100%]

==================================== FAILURES ====================================
___________________ test_bgp_peer_up[paramiko://100.66.12.202] ___________________

host = <testinfra.host.Host paramiko://100.66.12.202>

    def test_bgp_peer_up(host):
        command = 'vtysh -c "show ip bgp summary json"'
        output = run_remote_command(host, command)
        bgp_data = json.loads(output.stdout)
>       assert bgp_data['ipv4Unicast']['peers']['192.168.1.2']['state'] == "Established"
E       AssertionError: assert 'Idle (Admin)' == 'Established'
E         - Established
E         + Idle (Admin)

test_debian1.py:22: AssertionError
================================ warnings summary ================================
test_debian1.py::test_file_contents[paramiko://100.66.12.202]
  /home/tom/.pyenv/versions/3.9.4/envs/network-pytest/lib/python3.9/site-packages/...
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/warnings.html
============================ short test summary info =============================
FAILED test_debian1.py::test_bgp_peer_up[paramiko://100.66.12.202] - AssertionE...
===================== 1 failed, 1 passed, 1 warning in 0.52s =====================

You’ll need to read the FRR docs, and experiment with different failure modes to understand how FRR’s CLI output responds to various conditions in the control plane.

For example, after a little poking around, you’ll notice that the data in ['ipv4Unicast']['totalPeers'] doesn’t describe the total number of peers that are Established, rather, it describes the total number of peers that are configured. When you combine this with the number of peers who have state “Established” (which you can obtain by iterating over the bgp_data dictionary above), you can now pretty quickly write a test that looks for the total number of configured peers to be the same as peers that are in “Established” state.

Wrap Up, and Homework

We’re starting to gain some momentum with pytest. We understand how to write tests, and we can test conditions on our local computer as well as on remote devices. Our test suite is starting to grow, with 2 total tests in it so far. In the next article, we’ll leave the comfortable and safe territory of pure linux and venture into testing in the wild west of vendor-controlled network appliances.

Now that you have the basic framework for pytest, you can customize your code to your specific requirements. Here are a few challenges you can use to help dig further down into pytest.

Tags: pytest, python