Testing

The testing of software is an essential part of developing a high-quality code base and ensuring that functionality remains intact as changes are introduced.

This document details the testing standards and guidelines in place around KubOS.

Standards

All software modules in the Kubos repo are expected to have some level of testing in place. The appropriate level of testing will vary from module to module.

Libraries and APIs should have unit tests for private functionality and integration tests for public interfaces when possible.

Binary services or applications should provide integration tests when possible.

Project-level unit and integration tests should be included in the CI process. Branches must have all tests passing in CI to be accepted into master.

Guidelines

Running Tests

Rust

The cargo tool has a built-in test runner under the cargo test subcommand.

Running cargo test from within the crate’s folder will run all tests within the crate.

Running cargo test from the root of the Kubos repo will run all Rust tests in the repo.

Python

Tests for Python modules should be run under Python3 from within the module’s folder. The test files may exist as independent python scripts.

Any additional dependencies needed for running the tests will need to be added to the module’s requirements.txt file, in addition to the SDK and CI environments.

If a new Python module has been created, the CI configuration will need to be changed for the new tests to be run in the CI process. The CI configuration file is .circleci/config.yml, and the job which will need to be edited is non_rust_tests. Add a new step to run the tests for the new module.

non_rust_tests:
  docker:
    - image: kubos/kubos-dev:latest
  steps:
  ...
  - run: cd apis/new-api; python3 test_api.py

C

Unit tests can be run locally by navigating to the test folder under the module folder, creating a build dir in the test folder and running cmake .. && make.

To run the tests the same way that CircleCI does, navigate to the top level of the Kubos repo and issue this command:

$ python3 $PWD/tools/ci_c.py

If a new C module has been created, it will need to be added to the projects list in tools/ci_c.py before the script will run the tests.

Unit Tests

Unit tests should cover at least the following cases:

• Good cases for all functions
• Null pointer cases for each function pointer argument (when writing C)
• Out-of-bounds cases for each function argument which is limited by more than its size (ex. uint8_t but max value of 3)

When testing functionality which is tightly coupled with an external dependency (hardware device, network interface, etc), we suggest leveraging mocking to control and test with various dependency inputs and outputs.

In general we suggest decoupling any business, decision-making, or parsing logic from external dependencies to ease the testing of these functional units.

Rust

Rust has native support for unit tests.

Mocking in Rust is done by creating a trait to abstract away functionality and inject mocked out dependencies.

For instance, if you are writing a Radio API, and the device will communicate over UART, you can mock out the UART interface to simulate interactions with the radio.

pub trait UartDevice {
    fn read() -> Result<Vec<u8>, String>
    fn write(data: &[u8]) -> Result<(), String>
}

pub struct RadioDevice {
    comms: Box<UartDevice>
}

impl RadioDevice {
    pub fn new(comms: Box<UartDevice>) -> Self { .. }
}

When writing tests, a mocked out implementation of the trait can be either generated by a mocking library, or manually implemented. This page gives a good overview of mocking libraries currently available for Rust.

Warning

Many popular mocking libraries require unstable Rust, however KubOS uses stable rust.

Rust modules should include example code in the documentation. It is ok to use no_run when writing examples for docs, as sometimes these examples require external dependencies to actually run. However all examples should be buildable in the SDK and CI environment.

The general convention for Rust tests is to include unit tests in the same file as the code under test, in a tests module, and to place integration tests in a tests folder at the top level of the crate. See the test organization section of the book for more details on these conventions.

The app-service is a great Rust project to look at for examples of Rust code under test.

Python

Python’s unittest and mock packages should be used to create unit tests for Python modules.

The pumpkin-mcu-api is a great Python project to look at for an example of testing Python code using mock. The file mcu_api.py contains the Python class under test, while test_mcu_api.py contains the actual test code.

C

Unit tests for modules written in C are run using CMocka, which gives testers the ability to use mocking in their testing.

The C module should contain a test folder with a subfolder containing the test set/s (most modules will only have one test set).

Within each test set should be three files:

• <test-set>.c - The file containing the actual tests
• sysfs.c - Stub functions for the underlying sysfs calls
• stubs.cmake - Makes the stub functions available to the test builder/runner

The kubos-hal is a great C project to look at for an example of C testing using CMocka. The file source/i2c.c contains the code under test, the folder test/i2c contains all of the test code.

Integration Tests

Integration tests are built to exercise the public interfaces or end-to-end functionality of software. When writing these tests it is important to keep the end consumer of the software under test in mind.

Typically integration tests are run alongside unit tests when running the KubOS repo’s test suite and can live in the project’s folder. However if an integration test is long running (30 seconds or more), it should be split out into an independent project in test/integration and run separately from the suite of unit tests.