Validate Your Network with pytest - Part 2 - Kick the Tires
Author: Tom Ammon
This article is part of a multipart series that introduces pytest as a tool to validate the operational state of a network. We’ve validated that our lab and development environment are ready to go. Now it’s time to see pytest in action on a real network. If you weren’t here for Part 1, go take a look at it. Otherwise, let’s get to writing some tests!
Articles in this Series
- Part 1 - Setup
- Part 2 - Kick the Tires (You are Here)
- Part 3 - Add Support for Vendor Devices
- Part 4 - Scale Your Approach
Test a Remote Device
Let’s take this one step at a time.
We’re going to do something simple that has nothing to do with network infrastructure, to help us understand basic usage of pytest. Then, once we’ve got our legs under us, we can layer on more logic and testing of the network.
We’ll start off by connecting to one our linux routers, specifically we’ll be looking at debian-1. We’ll do a simple test to see if a file on the filesystem of that router contain the contents we expect it to.
Here’s the file, /opt/TomStatus.txt:
root@debian-1:~# echo "Tom is a goofball" > /opt/TomStatus.txt
root@debian-1:~# cat /opt/TomStatus.txt
Tom is a goofball
Build the Test
You can find the complete source for examples used in this article in its Github repo.
# we'll connect to devices and run commands a lot, so let's
# put this component into its own function right off the bat.
def run_remote_command(host, cmd):
output = host.run(cmd)
return output
# here's the actual test function that pytest will execute.
# we get the contents of /opt/TomStatus.txt from the remote
# device, and compare it to what we expect those contents to be
def test_file_contents(host):
command = 'cat /opt/TomStatus.txt'
filecontents = run_remote_command(host, command)
# if the output of the cat command above, against the host
# we specified in --hosts at the CLI, matches the string below,
# this test passes
assert filecontents.stdout.strip() == 'Tom is a goofball'
Run the Test
Now let’s fire it up. We’re running with more verbosity to make the example more meaningful.
|
|
You can ignore the warning about the ssh key for now
We see in line 10 the name of the specific test function that passed, as well as which remote device it passed on (100.66.12.202). Line 19 gives us a summary of all the results. This summary will become more useful later on when we have a lot of tests to run.
Notice here, that a passing test doesn’t tell you what caused the test to pass. Now let’s see what happens when a test fails.
Let’s break this thing
To make a test fail, we’ll update the contents of /opt/TomStatus.txt and run again.
root@debian-1:~# echo "Tom is a goofballl" > /opt/TomStatus.txt
root@debian-1:~# cat /opt/TomStatus.txt
Tom is a goofballl
Now kick off the test against your freshly-damaged target host. Notice the highlighted lines, then we’ll walk through them after the output.
|
|
Here’s what failure looks like. pytest breaks it down for us in great detail:
- line 10 - tells us which test function failed, and on which host
- line 13 - starting on line 13, we’ll get a printout showing the actual code of the test function that failed. This makes bug-hunting a lot faster.
- line 20 - the
>points to the line that actually failed the test - line 21 - explains in plain english why it failed
- line 26 - gives the error output message, in this case this message hasn’t been customized by us.
Test the State of the Network
Enough of this Hello World stuff - let’s check out the network. We’re working with a very simple topology. The router debian-1 is directly connected to the router debian-2, both are running FRR, and have established one BGP session with each other over their directly-connected link. Here’s the BGP configuration from debian-1:
root@debian-1:~# vtysh
Hello, this is FRRouting (version 7.5.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
debian-1.ammonfamily.net# sh run
Building configuration...
(...output omitted...)
!
router bgp 100
no bgp ebgp-requires-policy
neighbor 192.168.1.2 remote-as 101
!
address-family ipv4 unicast
network 1.1.1.1/32
exit-address-family
!
line vty
!
end
Before we start writing our test, let’s see what the healthy BGP session looks like:
root@debian-1.ammonfamily.net:~# vtysh
Hello, this is FRRouting (version 7.5.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
debian-1.ammonfamily.net# sh ip bgp summ
IPv4 Unicast Summary:
BGP router identifier 1.1.1.1, local AS number 100 vrf-id 0
BGP table version 2
RIB entries 3, using 576 bytes of memory
Peers 1, using 21 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
192.168.1.2 4 101 40 40 0 0 0 00:35:20 1 2
Total number of neighbors 1
Also, as an aside, FRR will provide json output for a lot of its state monitoring commands. Here’s the same data as above, formatted as json. I think you can see where this is going:
debian-1.ammonfamily.net# sh ip bgp summ json
{
"ipv4Unicast":{
"routerId":"1.1.1.1",
"as":100,
"vrfId":0,
"vrfName":"default",
"tableVersion":2,
"ribCount":3,
"ribMemory":576,
"peerCount":1,
"peerMemory":21816,
"peers":{
"192.168.1.2":{
"hostname":"debian-2.ammonfamily.net",
"remoteAs":101,
"version":4,
"msgRcvd":41,
"msgSent":41,
"tableVersion":0,
"outq":0,
"inq":0,
"peerUptime":"00:36:18",
"peerUptimeMsec":2178000,
"peerUptimeEstablishedEpoch":1627955534,
"pfxRcd":1,
"pfxSnt":2,
"state":"Established",
"connectionsEstablished":1,
"connectionsDropped":0,
"idType":"ipv4"
}
},
"failedPeers":0,
"totalPeers":1,
"dynamicPeers":0,
"bestPath":{
"multiPathRelax":"false"
}
}
}
The neighbor is up, some prefixes have been sent and received, and life seems pretty good so far.
Build the Test
Our first test will be simple. We know that on debian-1, we should see a BGP session up and in Established state with debian-2 (192.168.1.2). So let’s write the test.
# This is the same function we used last time, to connect to the device
# and capture the output and exit code generated from the device
def run_remote_command(host, cmd):
output = host.run(cmd)
return output
def test_bgp_peer_up(host):
# FRR syntax to request output in json
command = 'vtysh -c "show ip bgp summary json"'
output = run_remote_command(host, command)
# deserialize the json object found in output.stdout into the bgp_data dict
bgp_data = json.loads(output.stdout)
# find the session state in the dict and ensure that it meets our criteria
assert bgp_data['ipv4Unicast']['peers']['192.168.1.2']['state'] == "Established"
A few notes here:
- I fixed the problem with /opt/TomStatus.txt behind your back
- we are making use of the
jsonoutput that FRR offers, which will make parsing the output text really simple. - as with our test_file_contents() function, we are making use of the pytest-testinfra plugin to allow us to send commands to the remove device and then retrieve their output. Since it’s just linux on the remote device we don’t need to do anything different to obtain network state information.
- I’m leaving out the test_file_contents() function in the code block above for brevity, but it is still there in test_debian1.py.
Run the Test
$ pytest test_debian1.py --hosts 100.66.12.202 -v
============================== test session starts ===============================
platform linux -- Python 3.9.4, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /home/tom/.pyenv...
cachedir: .pytest_cache
rootdir: /home/tom/local/repos/blogrepos/network-validation-with-pytest
plugins: testinfra-6.4.0
collected 2 items
test_debian1.py::test_file_contents[paramiko://100.66.12.202] PASSED [ 50%]
test_debian1.py::test_bgp_peer_up[paramiko://100.66.12.202] PASSED [100%]
================================ warnings summary ================================
test_debian1.py::test_file_contents[paramiko://100.66.12.202]
/home/tom/.pyenv/versions/3.9.4/envs/network-pytest/lib/python3.9/site-packages/...
warnings.warn(
-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================== 2 passed, 1 warning in 0.49s ==========================
Cool! Our test passed. Based on the data we obtained manually before we wrote the test, this is not a surprise. Let’s see what pytest does when BGP is unhappy.
Let’s break this thing
We’ll drop the BGP session and then experience the chaos:
debian-1.ammonfamily.net# conf t
debian-1.ammonfamily.net(config)# router bgp
debian-1.ammonfamily.net(config-router)# neighbor 192.168.1.2 shutdown
debian-1.ammonfamily.net(config-router)#
debian-1.ammonfamily.net(config-router)# do sh ip bgp summ
IPv4 Unicast Summary:
BGP router identifier 1.1.1.1, local AS number 100 vrf-id 0
BGP table version 3
RIB entries 1, using 192 bytes of memory
Peers 1, using 21 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
192.168.1.2 4 101 58 60 0 0 0 00:00:07 Idle (Admin) 0
Total number of neighbors 1
It’s pretty clear from the State/PfxRcd column that the BGP session is not where we want it to be. Now let’s see what pytest thinks of the situation:
|
|
- lines 9 and 10 give the summary of which tests passed and failed
- lines 21-24 point out the specifics of the failure
- line 26 tells you which line in the python source contains the failing test
You’ll need to read the FRR docs, and experiment with different failure modes to understand how FRR’s CLI output responds to various conditions in the control plane.
For example, after a little poking around, you’ll notice that the data in ['ipv4Unicast']['totalPeers'] doesn’t describe the total number of peers that are Established, rather, it describes the total number of peers that are configured. When you combine this with the number of peers who have state “Established” (which you can obtain by iterating over the bgp_data dictionary above), you can now pretty quickly write a test that looks for the total number of configured peers to be the same as peers that are in “Established” state.
Wrap Up, and Homework
We’re starting to gain some momentum with pytest. We understand how to write tests, and we can test conditions on our local computer as well as on remote devices. Our test suite is starting to grow, with 2 total tests in it so far. In the next article, we’ll leave the comfortable and safe territory of pure linux and venture into testing in the wild west of vendor-controlled network appliances.
Now that you have the basic framework for pytest, you can customize your code to your specific requirements. Here are a few challenges you can use to help dig further down into pytest.
- Challenge #1: Custom AssertionError message
test_debian1.py:22: AssertionErroris not very informative. What if you wanted a non-developer to take some kind of action based on the results of your test suite?- Challenge: Figure out how to customize this message. For example, you could make it print out “BGP peer not in expected state of Established”.
- Challenge #2: Unknown host-key problem
- This warning is really annoying.
- Challenge: Make this warning go away.
- Challenge #3: Additional error handling logic
- The pytest-testinfra plugin returns not only stdout, but also stderr as well as exit codes. Different failure modes on the device being tested will manifest themselves in different ways.
- Challenge: Add additional error handling logic to present Assertion errors in a more fine-grained manner, giving your users maximum data against which to make operational decisions. For example, you could print one Assertion Error message if the BGP session data shows the neighbor in state
Idle(Admin)and a different message if the session data shows a different value for that field.