Dragon/Falcon Software

[krytek]

Just occurred to me that software is probably one of the least discussed and obscure topics around here.

Here's an interesting tidbit -
"Our Flight Software Group currently has opportunities developing software for embedded flight hardware using Linux and VxWorks as well as ground simulation software using Linux"
source - http://www.startuphire.com/job/software-engineer-embedded-linux-159520
So I guess there's no danger of Dragon getting a BSOD 

Anyway, can someone explain what a real time OS like VxWorks actually does differently, and how it combines with Linux in the flight harware?

[goretexguy]

A RTOS is designed to provide consistently reliable response times- in other words, programmers don't need to worry if their flight control program will be interrupted or delayed by something else. A RTOS guarantees that information will flow back and forth within certain limits, as opposed to Windows or MacOS which may have great variability in response times. Humans may not care about 30 milliseconds, but rockets and nuclear reactors do. A RTOS like vxworks is also extremely reliable... Visit wikipedia for more info.
Linux will be used to host the apps which provide flight simulation data- pretending to be rockets, pumps, navigation, etc. and feeding all that data into the actual flight control systems.

[butters]

The Linux kernel is primarily designed so that lots of different processes running on behalf of many different users can safely and fairly share the system resources. Each process runs in its own virtual address space, isolated from all other processes. User threads may only enter the kernel via one hardware exception handler which enforces strict protection of kernel resources. For example, there is a dedicated stack reserved for each thread when it is running in kernel. User processes are not to be trusted.

VxWorks is a single-user embedded OS designed to run a known set of real-time threads with static priority levels in a single kernel mode address space. The scheduling is fully-preemptive, but while they are scheduled to run, threads are trusted to play nice and to not barf all over resources belonging to other threads or the core kernel. This simple design permits very fast context switches between threads or between schedulable threads and interrupt service routines.

Over the years, Linux has made great big leaps in minimizing scheduling latency, which is perhaps the single most important measure of real-time capability. In Linux 2.0, the kernel could only service on thread at a time, and it always completed each system call before scheduling a different thread. Since Linux 2.6, threads can run in the kernel on all processors simultaneously, and threads can be preempted in favor of a higher-priority thread even while running in the kernel. The scheduler can also run concurrently on all processors, and no matter how many threads are queued, it always takes the same amount of time to find the highest-priority thread to run.

But no matter how small they squeeze the critical sections where kernel preemption or interrupts are disabled, the Linux kernel will still have non-deterministic response times because it makes some compromises to prevent device drivers from monopolizing the system and preventing user processes from running.

For example, Linux runs any pending "bottom halves" of device drivers whenever the kernel returns from any interrupt handler. But if a device driver reschedules its bottom half to run again while the processor is still executing bottom halves, then it is queued to special low-priority kernel threads (ksoftirqd) which wait to run until the processor is otherwise idle. This helps prevent device drivers from hogging the system (e.g. denial of service attacks which flood the network card with requests), but it also results in non-deterministic response time under heavy I/O loads.

In short, Linux is designed to withstand hostile multi-user networked environments while delivering excellent performance in client-server applications. But although response times have gotten very fast for soft real-time, there are no hard real-time guarantees.

[starsilk]

realtime OS like vxworks guarantee (absolutely, positively) that hardware interrupts (requests for attention) will be serviced within a certain time. they also tend to impose hard limits on processor time utilization by processes.

in a nutshell: they are simple OS which provide cast iron guarantees about timing. you can see how that would be useful for flight software.

linux is a multitasking OS designed for interacting with humans, who tend to care little if things are delayed by half a second once in a while. as a result, it can be far more complex and offer many more features. (same applies to Windows, OS X etc). needless to say a half second delay in rocket engine control would result in disaster...

because realtime OS are simple, and often impose draconian programming requirements to meet their guarantees, you often see a combination approach - realtime OS where it is absolutely needed, and a non-realtime OS where it can be safely used.

an example: tesla is also using vxworks for engine management, but linux for the giant touchscreen display in the Model S.

in reality, the line is being blurred somewhat. there are various patches or third party products to make linux a 'realtime' OS - either by providing a small subset of features and offering guarantees to 'realtime' processes, or by running the entire linux kernel as a low priority process in a small realtime OS wrapper.

for example, the linux 'realtime' patches do the former, but are generally claimed to be 'soft realtime' - ie: we *think* we guarantee this, but once in a blue moon we may not.. they are approaching true realtime though, for example: http://www.h-online.com/open/news/item/OSADL-experimentally-analyses-Linux-s-real-time-capabilities-1500366.html shows many, many thousands of hours with no deviations... but to be truly considered realtime the algorithms generally have to be mathematically proven, not shown by experiment.

[baldusi]

Let's not forget the L4 based linux versions. Those are not only RT, but the L4 kernel is demonstrated as bug free. Or my loved QNX, which get's its deterministic performance by using a microkernel, so even driver problems won't create problems. It does have a performance hit on the context switch front, though.

[LegendCJS]

Sometimes there is nothing real time kernels can do if its on bad hardware.  I've been using the mathworks Real Time Target (formerly Real Time Windows Workshop) for real time control in simulink, and I don't always get hard real time.  Apparently the newer your hardware is the more likely it has some hardware level feature (BUG) energy management framework that automatically takes over everything for a few moment every now and then to make sure you aren't wasting energy.  This is just one example of problems with trying to go real time on some store bought computer.

I'm certain that the pros know how to take care of this with careful hardware selection.

[IRobot]

This is the new approach, pass it on to software. The old approach for realtime control would be to use Xilinx. 10 years ago I would never think that software engineers would have so much power...

[grr]

To add to this, vxworks (like any realtime OS), allows you to control the deterministically control scheduling of a thread/process via set numbers . By that, you can state that we will give 4 units of time to a particular thread  out of say every 100 units.  This scheduling is important for making calculations on, well, real-time issues.

OTOH, regular Linux uses non-deterministic scheduling. You set priorities, etc. and HOPE to have a fair response time for all thread/processes. There are real-time extensions for Linux, but to be honest, these are considered soft real time vs. hard. Many of the RTOS companies have to deal with the popularity of Linux.

The 3 big RTOS out there would be QNX, VxWorks, and greenhills. However, in the aviation and space arena, you need a DO-178B OS. Basically, a hard RTOS generally with tight security. That would be VwWorks, GreenHills and Linux with RT extension.  Besides, QNX is falling out of favor esp. since RIM bought them. You will see it in their blackberry tablets, but to be honest, RIM is done for. Most likely QNX will fall to the side.

One of the biggest differences between greenhills and VxWorks, is that VxWorks is working closely with the Linux community, while Greenhill is fighting them tooth and nail.  As such, VxWorks API and development env. being ported to Linux allows a coder to come up to speed relatively quickly in a well known env and develop some useful code. Later, if needed, you can move that code to VxWorks quickly.  As such, greenhill is also dying a slow death.

However, what is also interesting is that they picked WindRiver's vxworks.  Says a lot about them.

[Antares]

(Not a fan boy but just suspicious whenever anybody demands perfect software. Made doubly so by experience in cellular when poor quality systems trumped 5-9s when 5-9 infrastructure sold at 20x the cost of one nine....)

Flawed analogy. Cell phones don't kill people when the software fails. (Dragon may be unmanned, but it's flying an approach to a manned international space station that cost over $50B to build). No one expects perfect software, but mission-critical software is held to higher standards and undergoes more rigorous testing. The space shuttle software was considered the "gold standard" in this category:

http://www.fastcompany.com/magazine/06/writestuff.html

No one expects SpaceX (or any other commercial space company) to ever reach that level again, but there are still better analogies than cell phones:
Aircraft flight software
Air traffic control software
Nuclear powerplant control software

All of these must be more-or-less bug-free in normal operation, or people die.

Add in medical devices and high-volume passenger trains and subways.

[Jorge]

(Not a fan boy but just suspicious whenever anybody demands perfect software. Made doubly so by experience in cellular when poor quality systems trumped 5-9s when 5-9 infrastructure sold at 20x the cost of one nine....)

Flawed analogy. Cell phones don't kill people when the software fails. (Dragon may be unmanned, but it's flying an approach to a manned international space station that cost over $50B to build). No one expects perfect software, but mission-critical software is held to higher standards and undergoes more rigorous testing. The space shuttle software was considered the "gold standard" in this category:

http://www.fastcompany.com/magazine/06/writestuff.html

No one expects SpaceX (or any other commercial space company) to ever reach that level again, but there are still better analogies than cell phones:
Aircraft flight software
Air traffic control software
Nuclear powerplant control software

All of these must be more-or-less bug-free in normal operation, or people die.

Add in medical devices and high-volume passenger trains and subways.

Good additions. Good thread move, too.

[Jorge]

1) Don't forget about the dedicated SpaceX software thread http://forum.nasaspaceflight.com/index.php?topic=28611.msg893373#msg893373

2) The hard part in most software writing is bridging the gap between the hardware engineers who own the parts that the software is controlling and the coders.  The coder can build perfect code that doesn't do what the valve engineer really wanted.

Especially when the hardware is changing. Configuration management is critical. Shuttle software at least had the advantage of a (relatively) stable hardware platform, once initial development was complete.

[watermod]

(Not a fan boy but just suspicious whenever anybody demands perfect software. Made doubly so by experience in cellular when poor quality systems trumped 5-9s when 5-9 infrastructure sold at 20x the cost of one nine....)

Flawed analogy. Cell phones don't kill people when the software fails

I beg to differ. I believe our customer  had some base stations bombed several times when some Yakuza calls were dropped.  (Nuff said and slinking off on that - Recent memory brings up the bus contention in a  famous brand of autos (with hundreds of embedded procs) was claimed to have led to some deaths and almost took down that company)

On another note read the Errata on both compilers, OS's and processors.  They are usually long and huge. For practical purposes errata are bugs.    There is no such thing as BUG FREE CODE.

As to the Linux kernel and RealTime...   There are many optional kernel schedulers and scheduling techniques.  They are usually compile time options for the Kernel but some are modules.   It's possible to radically change the scheduling behavior of a Linux kernel in several "Real Time" directions.   Hardware permitting it's possible to radically change the minimal clocking too.   If you look in the kernel code you see that ticks are macro-defined at compile time dependent on target hardware.  (obviously the std pc hardware clock is not going to cut fine grained time).  I should point out that syncing clocking among multiple machines below 500 ns is non-trivial. One should bear that in mind if you ever require tightly synchronized real time computers and commission a thesis level study first.

[Jorge]

(Not a fan boy but just suspicious whenever anybody demands perfect software. Made doubly so by experience in cellular when poor quality systems trumped 5-9s when 5-9 infrastructure sold at 20x the cost of one nine....)

Flawed analogy. Cell phones don't kill people when the software fails

I beg to differ. I believe our customer  had some base stations bombed several times when some Yakuza calls were dropped.  (Nuff said and slinking off on that

s'alright. I needed the laugh!

- Recent memory brings up the bus contention in a  famous brand of autos (with hundreds of embedded procs) was claimed to have led to some deaths and almost took down that company)

I think it's quite wise of SpaceX to compare themselves with the auto company (especially concerning capital) and consider (internally) whether they would survive a similar issue. They've shown considerable integrity in standing down to fix issues. For a company with their burn rate, that's a real sacrifice.

On another note read the Errata on both compilers, OS's and processors.  They are usually long and huge. For practical purposes errata are bugs.    There is no such thing as BUG FREE CODE.

And that is why I was quite careful to say "more-or-less bug-free in normal operation" and "No one expects perfect software". The space shuttle demonstrates one can come very close to perfection, but at considerable cost ($83/SLOC/year in the mid-90s, if I did the math right). But other high-reliability applications such as the ones Antares and I listed show that one can still approach reasonably close to perfection at lower cost.

[mnagy]

Let's not forget the L4 based linux versions. Those are not only RT, but the L4 kernel is demonstrated as bug free.

What do you mean by that? Surely no one is making a claim that a non-trivial software is free from bugs..

[butters]

Let's not forget the L4 based linux versions. Those are not only RT, but the L4 kernel is demonstrated as bug free.

What do you mean by that? Surely no one is making a claim that a non-trivial software is free from bugs..

L4 is a true microkernel. It does interrupt routing, thread scheduling, (physical) memory allocation, and interprocess communication. That's it. It doesn't have device drivers or filesystems or network protocols. It's actually more like a hypervisor than an OS, and its feature set is small enough that it can be exhaustively tested.

But unless your application can run on bare metal, implementing its own I/O stacks and so forth, then you need to run an OS such a Linux in usermode on top of L4. In this setup, the Linux kernel is a process running alongside your application on L4, and when your application makes system calls to the Linux kernel, they are implemented via L4's very lean and clever message-passing primitives.

There are certain advantages to this setup. For example, multiple Linux instances can run on top of the single L4 instance, and because they run in usermode, if there is a kernel panic or other failure in a Linux instance, it can be restarted without affecting the other instances. So flight-critical processes can be isolated from the rest of the system or even replicated in a redundant set.