Going Real-Time with Xenomai on VAB-820

It has been quite a few years since I’ve last used Xenomai, the real-time application framework for Linux, during the work on my thesis back at the university physics lab. When I’ve found that the VIA VAB-820 board is listed on Xenomai’s compatible hardware page, it brought back a lot of interesting memories and I got excited trying it out again. Now that the new (testing) kernel for the VAB-820 is released recently, I thought it’s a good chance to revisit real-time operating systems.

Setting up

Buildroot is a very handy tool to build a kernel and simple file system for embedded devices, and just recently accepted our submitted patch adding support for VAB-820. Buildroot already has support for Xenomai, so everything’s set up to create a system complete with the required patches & tools.

Running xeno-test on the VAB-820

One outstanding issue was a bit of adaptation. Xenomai (at least the stable 2.6.4 version I targeted) needs a suitable Adeos I-pipe patch applied to the Linux kernel, which enables secondary real-time kernel on the same hardware (see the Xenomai FAQ). The closest kernel version that had an available patch was 3.10.18, and I’ve backported that to the 3.10.17 kernel used in our latest release (can download the patch from Github).

After this comes setting up Buildroot. Besides the base configuration for the VAB-820, I had to add the following changes:

  • Kernel -> Linux Kernel Extensions -> Adeos/Xenomai Real-time patch(and add path for the downloaded patch file linked above
    • Target packages -> Real-Time -> (add your required Xenomai userspace skins; add test suite optionally)
    • Enable netcat in the busybox configuration (“make busybox-menuconfig”) if want to use network load in testing (optional)
    • Enable Target packages -> Debugging, profiling and benchmark -> ltp-testsuite for load testing  (optional, just need hackbench provided by that package)

This is not an exhaustive list, there are some dependencies (including the toolchain, I used the latest eglibc), but the required choices are quite obvious during the Buildroot setup using “make menuconfig”.

Now I can run make, wait for the compilation to finish (~30 minutes on my laptop), prepare an SD card with the new system, and boot up your VAB-820 or AMOS-820.

Running benchmarks

Of course I want some proof that the system is really running in real-time mode. The Xenomai knowledgebase has a pretty straightforward article on benchmarking with xeno-test, used that as a base to set up my testing environment.

I’ve connected the VAB-820 to the network, and the testing was run xeno-test like this, explained below:

xeno-test -l “dohell -s SERVERIP -p PORT -b PATH-TO-HACKBENCH 7200” -g results

dohell is a script that generates load for the latency benchmarks. If there’s no load, it’s very easy for a computer to do things reliably on time, the whole point of real time systems is delivering regardless of the load.

The dohell script can use a couple of different sources of load. For example, if provided with the “-s SERVERIP -p PORT” parameters (fill in the blank for them), then it generates network load. It needs netcat (nc) to work, and the easiest way I found on the other end to set up a server receiving the load is to run a simple network sink on the server on the port given to the script above:

nc -v -v -n -l -p PORT >/dev/null

Other load include is loading the CPU: that’s what hackbench is used for, “-b PATH-TO-HACKBENCH” tells the script where to find that tool.

Also have to tell the script how long to run, in seconds: “7200” = 2h, it does generate very nice plots, though I think you can have pretty good results in “900”  = 15mins.

Other flags are added to the latency tool running the actual measurements. Here’s “-g result” is telling it to save binned latency measurements ready to be plotted in a file called “result”. Other flags that can be useful is “-t 0|1|2”, which sets the test mode: put 0 there for user mode task timing (this is the default), 1 for kernel mode tasks, or 2 for IRQ timing. For most use cases user mode tasks are the most relevant information.

I’ve run 3 benchmarks: one run, 2h for each test mode (that’s a nice little 6h window to do other things). Below is the result of the benchmarks:

VAB-820 Xenomai results (click to enlarge)

The results are 16.5us mean / 57us max latency for user tasks, 11.6us mean / 43us max latency for kernel tasks, 4.9us mean / 24us max latency for timer IRQ, while being under full load. These are pretty close to the results I could find for other ARM systems out there, so looking good!

Future work

The upcoming Xenomai 3 is a rewritten version of Xenomai that can provide different ways of creating a real-time system, possibly allowing even stricter response time limits. It’s still in it’s 3rd release candidate form and Buildroot has no support for it, so trying it out would mean once again rolling up the sleeves and do the hard work.

Before that, it should be more interesting to find some real use cases for our brand new real-time system that’s running on the board, let it be for example drone control, industrial automation, or scientific research (and gather some inspiration from out there, eg. these slides: Real-Time Linux in Industrial Appliances).

If you have real-time OS requirements, would love to hear your experience too! What’s your use case, what are your tools, and experience with them? Leave us a note in the comments!