Testing ======= The testing of software is an essential part of developing a high-quality code base and ensuring that functionality remains intact as changes are introduced. This document details the testing standards and guidelines in place around KubOS. Standards --------- All software modules in the Kubos repo are expected to have some level of testing in place. The appropriate level of testing will vary from module to module. Libraries and APIs should have unit tests for private functionality and integration tests for public interfaces when possible. Binary services or applications should provide integration tests when possible. Project-level unit and integration tests should be included in the CI process. Branches **must** have all tests passing in CI to be accepted into master. Guidelines ---------- Running Tests ~~~~~~~~~~~~~ Rust ^^^^ The ``cargo`` tool has a built-in test runner under the ``cargo test`` subcommand. Running ``cargo test`` from within the crate's folder will run all tests within the crate. Running ``cargo test`` from the root of the Kubos repo will run all Rust tests in the repo. Python ^^^^^^ Tests for Python modules should be run under Python3 from within the module's folder. The test files may exist as independent python scripts. Any additional dependencies needed for running the tests will need to be added to the module's ``requirements.txt`` file, in addition to the SDK and CI environments. If a new Python module has been created, the CI configuration will need to be changed for the new tests to be run in the CI process. The CI configuration file is ``.circleci/config.yml``, and the job which will need to be edited is ``non_rust_tests``. Add a new step to run the tests for the new module. .. code-block:: yaml non_rust_tests: docker: - image: kubos/kubos-dev:latest steps: ... - run: cd apis/new-api; python3 test_api.py C ^ Unit tests can be run locally by navigating to the test folder under the module folder, creating a ``build`` dir in the test folder and running ``cmake .. && make``. To run the tests the same way that CircleCI does, navigate to the top level of the Kubos repo and issue this command:: $ python3 $PWD/tools/ci_c.py If a new C module has been created, it will need to be added to the ``projects`` list in ``tools/ci_c.py`` before the script will run the tests. Unit Tests ~~~~~~~~~~ Unit tests should cover at least the following cases: - Good cases for all functions - Null pointer cases for each function pointer argument (when writing C) - Out-of-bounds cases for each function argument which is limited by more than its size (ex. ``uint8_t`` but max value of 3) When testing functionality which is tightly coupled with an external dependency (hardware device, network interface, etc), we suggest leveraging mocking to control and test with various dependency inputs and outputs. In general we suggest decoupling any business, decision-making, or parsing logic from external dependencies to ease the testing of these functional units. Rust ^^^^ Rust has `native support for unit tests `__. Mocking in Rust is done by creating a ``trait`` to abstract away functionality and inject mocked out dependencies. For instance, if you are writing a Radio API, and the device will communicate over UART, you can mock out the UART interface to simulate interactions with the radio. .. code-block:: rust pub trait UartDevice { fn read() -> Result, String> fn write(data: &[u8]) -> Result<(), String> } pub struct RadioDevice { comms: Box } impl RadioDevice { pub fn new(comms: Box) -> Self { .. } } When writing tests, a mocked out implementation of the trait can be either generated by a mocking library, or manually implemented. This `page `__ gives a good overview of mocking libraries currently available for Rust. .. warning:: Many popular mocking libraries require unstable Rust, however KubOS uses stable rust. Rust modules should include example code in the documentation. It is `ok` to use ``no_run`` when writing examples for docs, as sometimes these examples require external dependencies to actually run. However all examples should be buildable in the SDK and CI environment. The general convention for Rust tests is to include `unit tests` in the same file as the code under test, in a `tests` module, and to place `integration tests` in a `tests` folder at the top level of the crate. See the `test organization `__ section of the book for more details on these conventions. The `app-service `__ is a great Rust project to look at for examples of Rust code under test. Python ^^^^^^ Python's ``unittest`` and ``mock`` packages should be used to create unit tests for Python modules. The `pumpkin-mcu-api `__ is a great Python project to look at for an example of testing Python code using ``mock``. The file ``mcu_api.py`` contains the Python class under test, while ``test_mcu_api.py`` contains the actual test code. C ^ Unit tests for modules written in C are run using `CMocka `__, which gives testers the ability to use mocking in their testing. The C module should contain a ``test`` folder with a subfolder containing the test set/s (most modules will only have one test set). Within each test set should be three files: - ``.c`` - The file containing the actual tests - ``sysfs.c`` - Stub functions for the underlying `sysfs` calls - ``stubs.cmake`` - Makes the stub functions available to the test builder/runner The `kubos-hal `__ is a great C project to look at for an example of C testing using `CMocka`. The file ``source/i2c.c`` contains the code under test, the folder ``test/i2c`` contains all of the test code. Integration Tests ~~~~~~~~~~~~~~~~~ Integration tests are built to exercise the public interfaces or end-to-end functionality of software. When writing these tests it is important to keep the end consumer of the software under test in mind. Typically integration tests are run alongside unit tests when running the KubOS repo's test suite and can live in the project's folder. However if an integration test is long running (30 seconds or more), it should be split out into an independent project in ``test/integration`` and run separately from the suite of unit tests.