- Kali On ARM
- Gem PDA
- Raspberry Pi - Full Disk Encryption
- Galaxy Note 10.1
- Raspberry Pi
- Acer Tegra Chromebook 13"
- ASUS Chromebook Flip
- BeagleBone Black
- Cubieboard 2
- HP Chromebook
- ODROID U2
- Raspberry Pi - Disk Encryption
- Raspberry Pi 2
- Samsung ChromeBook
- Samsung Chromebook 2
- USB Armory
- Utilite Pro
- Kali NetHunter Documentation
- NetHunter Rootless
- Installing NetHunter
- Installing NetHunter On the Gemini PDA
- NetHunter Components
- NetHunter Home Screen
- NetHunter Chroot Manager
- NetHunter KeX Manager
- NetHunter USB-Arsenal
- NetHunter BadUSB Attack
- NetHunter Application - Terminal
- NetHunter Custom Commands
- NetHunter DuckHunter Attacks
- NetHunter HID Keyboard Attacks
- NetHunter Kali Services
- NetHunter MAC Changer
- NetHunter MANA Evil Access Point
- NetHunter Man In The Middle Framework
- NetHunter Metasploit Payload Generator
- NetHunter Nmap Scan
- NetHunter Exploit Database SearchSploit
- Wireless Cards and NetHunter
- Building a New Device File
- Building NetHunter
- Porting NetHunter to New Devices
- Testing Checklist
- Patching the Kernel
- Configuring the Kernel - General
- Configuring the Kernel - Network
- Configuring the Kernel - Wifi
- Configuring the Kernel - SDR
- Configuring the Kernel - USB
- General Use
- Kali Network Repositories (/etc/apt/sources.list)
- HiDPI (High Dots Per Inch) Display
- Install NVIDIA GPU Drivers
- Kali Linux XFCE FAQ
- Kali Linux Forensics Mode
- Kali Linux Metapackages
- Configuring Yubikeys for SSH Authentication
- Kali In The Browser (Guacamole)
- Kali In The Browser (noVNC)
- All about sudo
- Updating Kali
- Kali's Domains
- Kali Development
- Public Packaging
- ARM Cross-Compilation
- Building Custom Kali ISOs
- Custom Beaglebone Black Image
- Custom Chromebook Image
- Custom CuBox Image
- Custom EfikaMX Image
- Custom MK/SS808 Image
- Custom ODROID X2 U2 Image
- Custom Raspberry Pi Image
- Generate an Updated Kali ISO
- Live Build a Custom Kali ISO
- Preparing a Kali Linux ARM chroot
- Rebuilding a Source Package
- Recompiling the Kali Linux Kernel
- Contributing run-time tests with autopkgtest
Contributing run-time tests with autopkgtest
Why Kali could benefit from your help
With Kali Linux being a rolling distribution it will occasionally have packages that are not stable. To best combat this, we have set up automated runtime tests that have to be passed in order to allow the packages to migrate into kali-rolling. However, most packages at this time don’t have a comprehensive autopkgtest associated with it causing this process to not reach its full potential. This is where the community can help us, by contributing test for packages that lack them.
A bit of autopkgtest background
Autopkgstests were created as a way to test a package as accurately as possible on a real Debian system. This method can leverage virtualization and containers to create an accurate and re-usable environment that will be the testing area. There is a lot more information available on this topic. For a more in depth look than what will be covered here please see the following articles (which served as sources of information during writing):
- The official Debian CI doc page
- A great talk covering autopkgtests
- A paper going in-depth on writing As-Installed tests
- The best practices Debian wiki page
We encourage users to spend some time looking through these after finishing reading the rest of this doc. We will only be really covering a small portion of what is a large topic to explore and learn.
For more information on the environment itself, the
autopkgtest-virt-qemu man pages are beneficial. As are the docs located at
Before getting started with writing tests, we need to get our environment set up. So we can validate our test once it is written.
kali@kali:~$ sudo apt install -y autopkgtest vmdb2 kali@kali:~$ kali@kali:~$ sudo mkdir /srv/autopkgtest-images/ kali@kali:~$ kali@kali:~$ sudo autopkgtest-build-qemu kali-rolling /srv/autopkgtest-images/kali-rolling.img http://http.kali.org/kali kali@kali:~$
Learning by examples
There are a few things to learn about with autopkgtests that we can see in action already. We will look at what tests can look like, what types are more beneficial, and a few tricks that we can see.
The cloud_enum control file looks like the following:
cloud_enum control file
Test-Command: cloud_enum --help Depends: @ Restrictions: superficial
Whereas the python-pip control file looks like the following:
python-pip control file
Tests: pip3-root.sh Restrictions: needs-root Tests: pip3-user.sh Restrictions: breaks-testbed # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=823358 Tests: pip3-editable.sh
With includes a test that looks like:
python-pip pip3-user.sh test file
#!/bin/sh export HOME=$AUTOPKGTEST_TMP export PATH=$PATH:$HOME/.local/bin export PIP_DISABLE_PIP_VERSION_CHECK=1 if [ $(id -u) = 0 ] then adduser --quiet --system --group --no-create-home testing user=testing mkdir $HOME/.cache chown -R $user $HOME runuser="runuser -p -u $user --" else runuser="" fi $runuser python3 -m pip install world $runuser python3 -m pip list --format=columns $runuser python3 -m pip show world ls -ld $HOME/.local/lib/python3.*/site-packages/world-*.dist-info $runuser python3 -m pip uninstall -y world $runuser python3 -m pip list --format=columns # Temporarily disabled. See #912379 #$runuser python3 -m pip list --outdated if [ $(id -u) = 0 ] then deluser --quiet testing fi
It is apparent that there is a big difference between the two. Both will technically create a pass or a fail, and one might think that the first option, the cloud_enum control file, seems easier and better to use. However, this is a superficial test which means “The test does not provide significant test coverage, so if it passes, that does not necessarily mean that the package under test is actually functional”. Because of this, we would rather have the python-pip test environment. It is more detailed and includes tests of different types of end-user environments.
We did just throw a bunch of text and code at you all to look at without much explanation, as it was important to note the differences before getting too involved. But lets take a step back now and look more closely at this.
Looking more closely at the
cloud_enum control file, we have three things going on.
Test-Command: cloud_enum --help Depends: @ Restrictions: superficial
The first thing we do is tell what command we are running. Because this is just a superficial test, we do not tell it to run a script and instead just do one line to test if it outputs the help. We then tell it to depend on the same dependencies that the package depends on, by using ‘@’. Should the test need other dependencies, we can list them here. In another test we will look at, this will be seen in action. We finally tell it that yes, this test is just a superficial one. We could tell it other things, like that
cloud_enum needs root, or that after running this test the testing environment will need to be scrapped.
The next control file we will learn from is pytest-factoryboy’s.
Tests: test3-pytest-factoryboy Depends: @, python3-pytest-pep8
As we can see, it depends on a python module. This python module will help us to test the tool, as we can see being done in the test:
#!/bin/sh set -e cp -r tests "$AUTOPKGTEST_TMP/" && cd "$AUTOPKGTEST_TMP" for py in $(py3versions -i); do $py -Wd -m pytest -v -x tests 2>&1; done
We can also see something else being done that is a good practice.
$AUTOPKGTEST_TMP can be used during these tests, and will help to clean a system and start on a next test, assuming the
breaks-testbed flag is not set.
Another package that can show a lot of info is hyperion. However, due to the size of the files and the breakdowns associated that will not be explained here.
What to do
So you have a system set up, have seen tests in practice, but now you need to actually write the tests. Once you find a package that has no test or a superficial test, you should learn the tool. Let us say that we have a tool that pings a server and then depending on the response will tell the user something. What does a fail look like? What does a success look like? Are there any edge cases that might mess up your test?
The next step after learning about a tool’s purpose is to figure out what you will need to tell autopkgtest. Using our tool from the previous example, will it need root to ping properly? What else might need to be done with the machine or environment to get the tool to work? What about after the test? Will there be an output file, and so therefore you should use $AUTOPKGTEST_TMP?
Once you learn and note what you can about the tool, you should start to write the test using the best practices and tips given earlier. There are many available resources that stem from those resources, and lots more packages that have solid tests that you can learn from example with if need be.
Running the tests after you have written them is fairly simple. All you have to do is the following command;
autopkgtest -u debci ~/packages/mytest/ -- qemu kali-rolling and point to the directory that contains the package you want to test. There are many other variations which will potentially be needed, and can be found at /usr/share/doc/autopkgtest/README.running-tests.html.