KubOS Linux on the ISIS iOBC

Overview

Note

Just looking to install KubOS Linux onto an iOBC? Skip ahead to the Default Installation Process.

The goal of this document is to create a KubOS Linux installation for the iOBC that can then run the satellite services (telemetry, payload communication, etc) needed for the ISIS customers’ missions.

The Working with the iOBC doc can then be used to create and load a user application on top of the new KubOS Linux install.

Ideally, the user should never have to mess with the kernel themselves. It should be pre-loaded onto the iOBC.

Software Components

ISIS Bootloader

The ISIS bootloader lives in the very beginning of the NOR flash. It should come pre-loaded on the board and should not need to be modified. It initializes the memory hardware and then copies U-Boot into the SDRAM and starts its execution.

If for some reason this bootloader needs to be reloaded, the relevant instructions can be found in section 8.1 of the ISIS-OBC Quickstart Guide.

U-Boot

Wiki

Site Page

Kubos U-Boot Repo

U-Boot, at the most basic level, is responsible for loading the kernel from the SD card into the SDRAM. However, it also provides a basic OS and CLI which can be used to configure and debug the kernel before it’s loaded.

Additionally, we’ve made some changes to allow us to use it as a kernel upgrade and recovery system. At boot time it will check for available upgrade packages or a corrupted Linux kernel and will then upgrade or rollback the kernel and rootfs as necessary.

Kernel

Linux

Version Overview

We’re using Linux 4.4. This is the current long-term support version (LTS) and will be supported until early 2018.

Glibc

Overview

We use the standard GNU C library to build our toolchains. We are currently building using v2.23.

BusyBox

Overview

BusyBox provides many of the common Linux console and shell commands, but in a smaller package.

BuildRoot

Overview

The current development tool for building all of the components required for running embedded Linux. Using this allows us to pass in a basic configuration file and then have all of the required packages and options brought in and compiled automatically. This reduces the amount of time to configure KubOS Linux for a new board.

SAM-BA

Product Page

The software tool used to flash the kernel and components onto the iOBC.

Note

The ISIS-OBC SDK includes the SAM-BA application. You should install this version, rather than the default Atmel version, since it is packaged with several iOBC configuration files which are required to successfully connect to the board.

KubOS Linux Build Process

If for some reason you want or need to modify and rebuild the KubOS Linux components, follow the steps in this section.

Build the OS Files

Warning

The OS files cannot be built using a synced folder in a Vagrant box (or regular VM). VirtualBox does not support hard links in shared folders, which are crucial in order to complete the build.

Create new folder

$ mkdir kubos-linux

Enter the new folder

$ cd kubos-linux

Download BuildRoot-2016.11 (more current versions of BuildRoot may work as well, but all testing has been done against 2016.11)

$ wget https://buildroot.uclibc.org/downloads/buildroot-2016.11.tar.gz && tar xvzf buildroot-2016.11.tar.gz && rm buildroot-2016.11.tar.gz

Pull the kubos-linux-build repo

$ git clone http://github.com/kubostech/kubos-linux-build

Move into the buildroot directory

$ cd buildroot-2016.11

Point BuildRoot to the external kubos-linux-build folder and tell it to build the iOBC.

Note

You will need to build with sudo if you are using the default iOBC configuration, since it points the output toolchain to “/usr/bin/iobc_toolchain”, which is a protected directory.

$ sudo make BR2_EXTERNAL=../kubos-linux-build at91sam9g20isis_defconfig

Build everything

$ sudo make

The full build process will take a while. Running on a Linux VM, it took about an hour. Running in native Linux, it took about ten minutes. Once this build process has completed once, you can run other BuildRoot commands to rebuild only certain sections and it will go much more quickly (<5 min).

BuildRoot documentation can be found **here**

The generated files will be located in buildroot-2016.11/output/images. They are:

  • uboot.bin - The U-Boot binary
  • zImage - The compressed Linux kernel file
  • at91sam9g20isis.dtb - The Device Tree Binary that Linux uses to configure itself for the iOBC
  • rootfs.tar - The root file system. Contains BusyBox and other libraries

Changing the Output Toolchain Directory (optional)

If you would like to build your toolchain in somewhere other than the “/usr/bin/iobc_toolchain” directory, update the BR2_HOST_DIR variable in the “configs/at91sam9g20isis_defconfig” file.

If you would like BuildRoot to just build the toolchain locally, you may remove the BR2_HOST_DIR variable entirely. The toolchain will then be built under the main “buildroot-2016.11” directory in a new “output/host” folder.

Create an SD Card Image

Note

The image script will use 4GB of system RAM during execution. By default, the Kubos Vagrant box only provides 2GB. As a result, you’ll either need to increase the amount of RAM provided to your box, or run the script natively.

A script is available to create an SD card image with the latest KubOS Linux kernel and root filesystem.

Navigate to the ‘kubos-linux-build/tools’ directory.

Run the format-image.sh script. You might need to run as root to get permissions for certain steps.

The script has optional parameters:

  • -d {device} - Sets the SD card device name to flash the newly created image to (does not flash by default)
  • -i {name} - Specifies the output file name of the image file to be created. (default: “kubos-linux.img”)
  • -p - Specify that existing kpack-base.itb and kernel files should be copied into the appropriate partitions
  • -pp - Specify that the kpack-base.itb and kernel files should be built and then copied to their partitions
  • -ppp - Only build and copy the kpack and kernel files. Skip all other steps.
  • -s {size} - Size, in MB, of the SD card. The default is 3800 (~4GB).
  • -b {branch} - Specify the branch name of U-Boot that has been built. The default is ‘master’. This option should not need to be used outside of development.

So if I wanted to create a custom-named image with brand new kernel files, I would run:

$ ./format-image.sh -i kubos-linux-v1.0.img -pp

Create an Upgrade Package

If you would like to distribute your changes as a Kubos upgrade package instead, please refer to the Upgrade Creation instructions.

Default Installation Process

The KubOS Linux installation process is composed of two high-level steps:

  • Flashing the SD card
  • Flashing the on-board NOR flash

To perform a default installation, three files are needed:

  • A KubOS Linux SD card image
  • u-boot.bin
  • at91sam9g20isis.dtb

All of these files can be obtained from our KubOS Linux Releases page on GitHub

Download the latest KubOS_Linux.zip file and then unzip the files for the iOBC. They’re located in the KubOS_Linux/{version}/iOBC folder.

Install the SD Card Files

All users should install the SD card files using a distributed KubOS Linux image, unless they have created a custom KubOS Linux build. In that case, the SD card files can be installed by either flashing a complete KubOS Linux image onto an SD card or by alternate means.

Pre-Requisites

  1. Obtain an SD card that is at least 4GB.

Note

The KubOS Linux SD images are created for a 4GB SD card. The image can be applied to a larger SD card, but the resulting system will still only have 4GB of space available to it.

  1. Install Etcher. Other software to flash SD cards does exist, but Etcher is the Kubos software of choice.
  2. Obtain a KubOS Linux image

Flash the SD Card

Using Etcher:

  • Select the KubOS Linux image to flash
  • Make sure the SD card device is correct (may be auto-detected if there is only one SD card present in your system.)
  • Click the “Flash!” button to start the flashing process
Etcher Setup

Etcher Setup

It should take roughly 10 minutes for a 4GB image to be loaded onto an SD card.

Once the program has finished successfully, the SD card is ready to be inserted into the iOBC’s SD Card 0 slot.

Install the NOR Flash Files

The NOR flash files will be loaded onto the iOBC using the Atmel SAM-BA software.

This can be done by using the provided command line script or using the SAM-BA GUI.

The SD card does not need to be inserted into the iOBC in order for this step to work.

Warning

The SAM-BA software currently only supports using the SAM-ICE JTAG with host machines running Windows. This means that you must use a Windows OS in order to initially flash the iOBC.

Once KubOS Linux has been installed, the device tree, which is located in the NOR flash, can be updated using the standard Upgrade Installation process with a kpack-nor-*.itb file.

Pre-Requisites

  1. Obtain an Atmel SAM-ICE programmer/debugger.

  2. Install programming drivers from https://www.segger.com/jlink-software.html.

  3. Install FTDI USB-to-serial drivers from http://www.ftdichip.com/Drivers/VCP.htm

  4. Install SAM-BA from the ISIS-OBC SDK installer. (Refer to Section 3.3 of the ISIS-OBC Quick Start Guide)

    Note: You must use the ISIS version of SAM-BA, rather than the default Atmel installation. It includes several configuration files that are required to connect to the iOBC.

  5. Setup the iOBC board for serial connection and programming. (Refer to Chapter 4 of the ISIS-OBC Quick Start Guide)

  6. Connect the programming and serial connection cables to your computer.

Warning

Make sure the red jumper on the programming board is in place; it bypasses the watchdog. If you don’t, the board will continually reboot and you won’t be able to flash anything.

  1. Turn on the board.

  2. Copy the kubos-nor-flash.tcl script from the tools/at91sam9g20isis folder in the kubos-linux-build repo into the SAM-BA application folder.

  3. Change line 44 in {path to SAM-BA}/tcl_lib/boards.tcl from this:

    "at91sam9g20-ISISOBC"    "at91sam9g20-ISISOBC/at91sam9g20-ISISOBC.tcl"

    to this:

    "at91sam9g20-isisobc"    "at91sam9g20-ISISOBC/at91sam9g20-ISISOBC.tcl"

    (the SAM-BA application converts everything to lower case, which will lead to a “board not found” error if you don’t change this file)

Boot into U-Boot

(Skip this section if you’ve never put Linux on your board before)

If you already have Linux running on your board, you’ll need to boot into the U-Boot console rather than the Linux console in order to be able to flash the board.

You’ll need to establish a serial connection with the board in order to connect to the console.

You can do this via a Kubos Vagrant image with the minicom kubos command after booting the board.

The default login information for an iOBC is kubos/Kubos123.

Issue the reboot command in order to restart the system.

Hold down any key while the board is restarting. This will exit out of the auto-boot and bring up the CLI.

U-Boot Console

U-Boot Console

The board is now ready to be flashed.

Flash the Files

The flashing script can be called from the standard command prompt using this command:

$ {path to SAM-BA}/sam-ba.exe \jlink\ARM0 at91sam9g20-ISISOBC
      {path to SAM-BA}/kubos-nor-flash.tcl {input arguments} [> {logfile}]

Where the input arguments are as follows:

  • uboot={uboot file} - Path to U-Boot binary
  • dtb={dtb file} - Path to Device Tree binary
  • altos={alt file} - Path to alternate OS binary

Multiple input arguments can be specified and should be space-separated.

The optional logfile parameter is highly recommended, as the SAM-BA application will not give any other response to this command. The log file will contain all of the output as the script connects to the board and transfers the files.

Example command:

$ C:/ISIS/applications/samba/sam-ba.exe /jlink/ARM0 at91sam9g20-ISISOBC
      C:/ISIS/applications/samba/kubos-nor-flash.tcl uboot=new-u-boot.bin dtb=new-dtb.dtb
      > logfile.log

If you’d like to confirm that the command ran successfully, open the log file. You should see this message for each file you attempted to flash:

Sent file & Memory area content (address: [...], size: [...] bytes) match exactly !

Reboot the System

If you have not already done so, insert the SD card into the iOBC’s first SD card slot while the board is not powered.

After new files have been loaded, the board will need to be powered off and back on again in order to go through the normal boot process.

Status LEDs

There are four yellow LEDs present on the iOBC which give some indication of what state the board is in, along with one red LED which is lit when the system is powered:

  • Three LEDS (solid) - The system is currently running U-Boot
  • One LED (blinking) - The system is currently running KubOS Linux

Connect to the System

You should now be able to set up a serial connection to your board and interact with the KubOS Linux environment.

You’ll need to establish a serial connection with the board in order to connect to the console. Set up a serial connection to the board at a baudrate of 115200.

If you have a Kubos Vagrant image currently running, the FTDI connection will be automatically passed through. You can use the included minicom configuration to quickly connect to the iOBC via the minicom kubos command in the VM’s command console.

Note

If a Kubos Vagrant image is running, you will be unable to establish a serial connection on your host machine. You must instead connect to the device through the VM.

$ minicom kubos

If the board is already powered, hit the Enter key to display the login dialog.

If you power the board after starting the minicom session, the end of the boot messages will look like this:

...
EXT4-fs (mmcblk0p1): mounted filesystem with ordered data mode. Opts: (null)
EXT4-fs (mmcblk0p7): recovery complete
EXT4-fs (mmcblk0p7): mounted filesystem with ordered data mode. Opts: (null)
Initializing random number generator... random: dd: uninitialized urandom read (512 bytes read, 25 bits of entropy available)
done.
Starting network: OK
Starting kubos-c2-daemon:
OK
Starting linux-telemetry-service:
OK

Welcome to KubOS Linux
Kubos login:

By default, there are two user accounts available: “root” (the superuser), and “kubos” (a normal user). Both have a default password of “Kubos123”. For more information, see the User Accounts section.

Using KubOS Linux

For information on how to create and run applications on your new KubOS Linux system, see the Working with the ISIS-OBC guide.

Upgrade Process

If you already have KubOS Linux installed on your system, but would like to upgrade to the latest version, check out the Upgrade Installation section. Alternatively, if you would like to rollback to a previously installed version, refer to the Upgrade Rollback section.

Recovery Process

Should your KubOS Linux kernel become corrupted (as indicated by failing to successfully boot into Linux several times), the system will automatically try to recover during the next boot.

It will go through the following steps, if each is present (system will reboot after attempting each step):

  1. Reload the current version of KubOS Linux from the kpack*.itb file in the upgrade partition
  2. Reload the previous version of KubOS Linux from the kpack*.itb file in the upgrade partition
  3. Reload the base version of KubOS Linux from the kpack-base.itb file in the upgrade partition
  4. Boot into the alternate OS

If none of these steps work, then the system will boot into the U-Boot CLI. From here, some basic troubleshooting and debugging abilities should be available.

More information about the recovery process and architecture can be found in the KubOS Linux Recovery doc

Resetting the Environment

If the system goes through the full recovery process, you will need to reset the environment in order to resume the normal boot process.

From the U-Boot CLI:

$ env default bootcmd
$ env default bootcount
$ env default recovery_available
$ saveenv
$ reset

These commands will:

  • Restore the relevant environment variables to their default values
  • Save the new values to persistent storage
  • Reboot the system

As long as a valid kernel and rootfs are available, your system should now successfully boot into KubOS Linux.

Non-Default Installation Process

There are alternate ways to install KubOS Linux onto the board, in case you want to create a custom installation, or are having issues with the default installation work flow.

Alternate SD Card Setup

If you do not have a KubOS Linux image, you can load the required files onto an SD card:

  • by using our flashing script

    or

  • manually

Pre-Requisites

Since you are not using a KubOS Linux image, you will need to go through the OS build process locally in order to create the kernel and rootfs files.

In order to write the files to the SD card your build system needs be able to a) see the SD card device and b) read/write to multiple partitions.

If you’re running Mac OS or Windows, you’ll need to pass the SD card through to your Vagrant box.

If you’re running Linux, you can either pass through the SD card to your Vagrant box via the VirtualBox Manager, or run the whole build process natively.

Get the Device Name

To start, find the name of your SD card in your system:

$ sudo fdisk -l

You should see a device that looks like this:

Disk /dev/sdb: 3.8 GiB, 4025483264 bytes, 7862272 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xf39e6ab1

In this example ‘/dev/sdb’ is the name of the SD card. You might also see ‘/dev/mmcblk0’. You’ll need to use this name in all future commands.

Method 1: Run the Formatting/Flashing Script

A script is available to format the SD card and then load the latest KubOS Linux kernel and root filesystem.

Navigate to the ‘kubos-linux-build/tools’ directory.

Run the format-sd.sh script. You might need to run as root to get permissions for certain steps.

The script has optional parameters:

  • -d {device} - Specify the name of the SD card device. The default is ‘/dev/sdb’
  • -s {size} - Size, in MB, of the SD card. The default is 4000 (4GB).
  • -w - Specify that the SD card should be wiped before formatting. Useful if there was any data previously on the card. Note Wiping a 4GB SD card takes about 10 minutes.
  • -p - Specify that existing kpack-base.itb and kernel files should be copied into the appropriate partitions
  • -pp - Specify that the kpack-base.itb and kernel files should be built and then copied to their partitions
  • -ppp - Specify that the SD card should not be formatted. Only build and copy the kpack and kernel files.
  • -b {branch} - Specify the branch name of U-Boot that has been built. The default is ‘master’. This option should not need to be used outside of development.

So if I wanted to wipe my SD card and then build and load the new kernel files, I would run:

$ ./format-sd.sh -wpp

Once the script has finished successfully, the SD card is ready to be inserted into the iOBC’s SD Card 0 slot.

Method 2: Manual Format/Flash Process

If for some reason you’d like to format the SD card and load the bare minimum files onto it manually, follow this process.

Partition the SD Card

First, you’ll need to set up the partitions on the SD card (for readability, we’ll be using device name ‘/dev/sdb’. Be sure to replace with the device name of your SD card):

Create a partition table

$ sudo parted /dev/sdb mklabel msdos y

Create the partitions

$ sudo parted /dev/sdb mkpart primary ext4 4M 3856M
$ sudo parted /dev/sdb mkpart extended 3856M 3996M
$ sudo parted /dev/sdb mkpart logical fat16 3856M 3876M
$ sudo parted /dev/sdb mkpart logical ext4 3876M 3896M
$ sudo parted /dev/sdb mkpart logical ext4 3896M 3996M

Configure the partitions (ex. /dev/sdb1)

$ sudo mkfs.ext4 /dev/sdb1
$ sudo mkfs.fat /dev/sdb5
$ sudo mkfs.ext4 /dev/sdb6
$ sudo mkfs.ext4 /dev/sdb7
Create the Kernel File

The BuildRoot build process creates the zImage file, which is a self-extracting kernel image. In order to help detect corruption, we package that into an *.itb file, which includes a checksum value that can be validated during boot time.

Navigate to your ‘kubos-linux-build’ folder and open the ‘tools’ directory.

Run the kubos-kernel.sh script.

The script has optional parameters (which are unlikely to be needed):

  • -i {input-file} - Specify the name of the *.its file to use. This file describes the files that will be packaged and their usage configuration options. The default is ‘kubos-kernel.its’, which should also be located in the ‘tools’ directory.
  • -b {branch} - Specify the branch name of U-Boot that has been built. The default is ‘master’. This option should not need to be used outside of development.

The script will create the ‘kubos-kernel.itb’ file.

Copy the files

Next, you’ll need to copy the kernel file into the boot partition and the rootfs into the rootfs partition

From your project folder:

Create mount folders

$ mkdir boot
$ mkdir rootfs

Mount the partitions (replace ‘/dev/sdb’ with the name of your SD card device)

$ sudo mount /dev/sdb5 boot
$ sudo mount /dev/sdb6 rootfs

Copy the kubos-kernel.itb file into partition 5. It will need to be renamed to ‘kernel’.

$ sudo cp buildroot-2016.11/output/images/kubos-kernel.itb boot/kernel

Untar the rootfs into partition 6

$ sudo tar -xvf buildroot-2016.11/output/images/rootfs.tar -C rootfs

Unmount the partitions

$ sudo umount /dev/sdb5
$ sudo umount /dev/sdb6

Remove the SD card and insert it into iOBC SD card slot 0.

Alternate NOR Flash Setup

You must still use the Atmel SAM-BA software on a Windows machine in order to flash the required files into the iOBC NOR flash. However, you can use the SAM-BA software directly to do so, rather than using the previously provided script.

Pre-Requisites

  1. Obtain an Atmel SAM-ICE programmer/debugger.

  2. Install programming drivers from https://www.segger.com/jlink-software.html.

  3. Install FTDI USB-to-serial drivers from http://www.ftdichip.com/Drivers/VCP.htm

  4. Install SAM-BA from the ISIS-OBC SDK installer. (Refer to Section 3.3 of the ISIS-OBC Quick Start Guide)

    Note: You must use the ISIS version of SAM-BA, rather than the default Atmel installation. It includes several configuration files that are required to connect to the iOBC.

  5. Setup the iOBC board for serial connection and programming. (Refer to Chapter 4 of the ISIS-OBC Quick Start Guide)

  6. Connect the programming and serial connection cables to your computer.

Warning

Make sure the red jumper on the programming board is in place; it bypasses the watchdog. If you don’t, the board will continually reboot and you won’t be able to flash anything.

  1. Turn on the board.

  2. Obtain the NOR flash files either from Kubos, or from your own local build:

    • u-boot.bin
    • at91sam9g20isis.dtb

Boot into U-Boot

(Skip this section if you’ve never put Linux on your board before)

If you already have Linux running on your board, you’ll need to boot into the U-Boot console rather than the Linux console in order to be able to flash the board.

You’ll need to establish a serial connection with the board in order to connect to the console.

You can do this via a Kubos Vagrant image with the minicom kubos command after booting the board.

The default login information for an iOBC is kubos/Kubos123.

Issue the reboot command in order to restart the system.

Hold down any key while the board is restarting. This will exit out of the auto-boot and bring up the CLI.

U-Boot Console

U-Boot Console

The board is now ready to be flashed.

Installation

Start SAM-BA

Start up SAM-BA. You’ll want to select the at91sam9g20-ISISOBC option from the ‘Select your board’ drop-down.

SAM-BA Connection Selection

SAM-BA Connection Selection

Enable Flashing

Execute the ‘Enable NorFlash’ script. This will prep the board to enable flashing.

SAM-BA Enable NorFlash

SAM-BA Enable NorFlash

Flash U-Boot

Select the uboot.bin file in the ‘Send File Name’ field.

Make sure that ‘Address’ is set to 0xA000.

Click ‘Send File’

SAM-BA Send U-Boot

SAM-BA Send U-Boot

Click ‘Compare sent file with memory’ after the file transfer has completed to confirm that all data was sent successfully.

Flash Device Tree

Select the at91sam9g20isis.dtb file in the ‘Send File Name’ field (you’ll need to view all file types in order to see the .dtb file)

Set ‘Address’ to 0x70000.

Click ‘Send File’

SAM-BA Send DTB

SAM-BA Send DTB

Click ‘Compare sent file with memory’ after the file transfer has completed to confirm that all data was sent successfully.

Reboot the System

After new files have been loaded, the board will need to be powered off and back on again in order to go through the normal boot process.