SpaceX CRS-1 Software/Computer Design Discussion Thread

[mlindner]

An uneducated guess: on a slow system the realtime requirements may not be met with Linux and C++.

I doubt it. You can do cycle-perfect simulations of Apollo hardware in Javascript in a browser nowadays, so that can't be it. Console video games run on limited hardware too, and C++ is the language of choice for that.

That is quite a few orders of magnitude slower. Some here on the forum were even surprised they use Linux at all because it is not hard realtime.

Linux is a re-implementation of Unix.  Soft real time Unix made its living controlling telephone exchanges.  For SpaceX it probably comes down to how fast a rocket engine can gimbal.

I'm pretty sure SpaceX isn't using Linux in that portion of their avionics... probably some other embedded, fully real-time operating system.

I doubt that. Having to support an entirely different operating system splits your workforce. From the little I talked to them they have mainly two groups: flight software, and avionics. The avionics groups seem to use mainly hardware controls that interact with software and the flight software is all in these redundant linux modules.

[mlindner]

So digging in the careers section of spacex.com should have been done a while ago:
Summarized: I deleted common traits and traits that were generic.

Software Engineer (Embedded uC):
Broad knowledge of microprocessor technology including (but not limited to); PowerPC, ARM, OMAP, 8051, etc.
Extensive experience programming in C

Software Engineer (Flight Software):
Extensive knowledge of Unix/Linux and Linux Internals
Several years of software development experience
Extensive experience programming in C++
Experience programming for high reliability systems

Software Engineer (Embedded Linux):
Extensive knowledge of Linux system programming
Extensive knowledge of Linux kernel internals including device drivers and board support
Experience developing embedded systems, particularly 'Board Bring Up'
Extensive experience programming in C


A few comments:
They use a lot of linux. They don't use x86. Looks like PowerPC and ARM mainly.

[john smith 19]

So digging in the careers section of spacex.com should have been done a while ago:
Summarized: I deleted common traits and traits that were generic.
A few comments:
They use a lot of linux. They don't use x86. Looks like PowerPC and ARM mainly.

Wonder what the 8051 is for? uC rolls up a bunch of Unix commands and a shell into 1 single block to speed up loading. It also looks like the design boards in house. I'd guessed they'd gone with a *single* architecture but it suggests they went (mostly) with ARM & PPC instead.
I wonder what decides how the jobs are split up?

Nice work.  Thank you for the effort.

[mlindner]

uC rolls up a bunch of Unix commands and a shell into 1 single block to speed up loading.

uC stands for microcontroller (Atmel AVR, PIC, TI MSP430 are a few common ones), so I'm not sure what you meant by this.

As a side note, my group here flies MSP430s in space. We and other groups have quite often flown them in space on cubesats. They have quite high reliability, I haven't really heard of any permanently failing, occasionally they crash and have to be rebooted, but thats why we fly everything with watchdog timers (to reboot them). More so the newer ones are FRAM (Ferroelectric RAM) based which has inherent radiation hardening based on the technology because the data is stored in magnetic fields rather than electrons that could be disrupted by radiation. I should also note that they cost around $6 USD per chip.

[john smith 19]

uC rolls up a bunch of Unix commands and a shell into 1 single block to speed up loading.

uC stands for microcontroller (Atmel AVR, PIC, TI MSP430 are a few common ones), so I'm not sure what you meant by this.

Oops. Still had my Linux glasses on when I read that. It's one of the approaches for doing 1sec boot Linux implementations.

As a side note, my group here flies MSP430s in space. We and other groups have quite often flown them in space on cubesats. They have quite high reliability, I haven't really heard of any permanently failing, occasionally they crash and have to be rebooted, but thats why we fly everything with watchdog timers (to reboot them). More so the newer ones are FRAM (Ferroelectric RAM) based which has inherent radiation hardening based on the technology because the data is stored in magnetic fields rather than electrons that could be disrupted by radiation. I should also note that they cost around $6 USD per chip.

I've not come across MSP430 before. Is the instruction set completely new or is a uC version of a better known architecture?
BTW I think you'll find ferroelectric is not magnetic (the ferro- bit is a misnomer) its more like a static piezoelectric effect. But I think you're right it is *substantially* more rad hard. It doesn't sound that expensive either.

OT but how did they fair over the South Atlantic Anomaly?

[baldusi]

This is a mis-representation of "real-time." If your system runs fast enough then even if it is not "real-time," it acts as if it is. As long as you can service events fast enough.

No it isn't. Real-Time means bounded maximum latency. You are thinking about the average latency. Non RT OS can't guarantee that the maximum latency between CPU slices is below certain (critical, might get 100ms, but not nano second) amount. Things like DMA, memory or file handle access might block in non deterministic ways, you can get races on resources and such. It's very low level but it doesn't matter how fast your CPU is.
In fact, even the architecture might have limitations. I still remember how some very old soekris boxes had better RT behavior than the latest i7 motherboard. It might have to do with the way the system accesses the clock, or how the usb controller is wired to the system bus.
Many times, maximum latency and average latency are opposites and you have to trade one for the other. In fact, maximizing average latency might get you to O(C) while minimizing maximum latency might get to be a O(N) problem.

[mlindner]

OT but how did they fair over the South Atlantic Anomaly?

Not sure on that. I'm not directly involved with the mission that has logged the most time in space. We don't (yet) actually fly radiation monitors in space, so we can only tell when it resets. You can take a look at http://rax.engin.umich.edu/
It flies an msp430 as its flight computer, older flash based model, apparently works fine, doing great science. Nanosats generally don't fly with any redundancy anywhere because of mass and space requirements, if it breaks it breaks.

[john smith 19]

OT but how did they fair over the South Atlantic Anomaly?

Not sure on that. I'm not directly involved with the mission that has logged the most time in space. We don't (yet) actually fly radiation monitors in space, so we can only tell when it resets. You can take a look at http://rax.engin.umich.edu/
It flies an msp430 as its flight computer, older flash based model, apparently works fine, doing great science. Nanosats generally don't fly with any redundancy anywhere because of mass and space requirements, if it breaks it breaks.

I've seen pictures taken with digital cameras and closed shutters over the SAA Vs other parts of their orbit.

It's an impressive demonstration of the *relative* radiation level.

[Mader Levap]

I've seen pictures taken with digital cameras and closed shutters over the SAA Vs other parts of their orbit.
It's an impressive demonstration of the *relative* radiation level.

Links?

[john smith 19]

I've seen pictures taken with digital cameras and closed shutters over the SAA Vs other parts of their orbit.
It's an impressive demonstration of the *relative* radiation level.

Links?

I'd google south Atlantic anomaly but it was at a presentation give by Henry Spencer at Space Access.

[Prober]

... I've seen a lot of equipment and more than one life lost because of redundancy induced complacency. (A phrase I just invented for this post)...

Same argument applies to ANY reliability increase, does it not?

For instance, SpaceX is going to improve their error-handling capability for the next mission, making resyncing automatic. This will make them more robust to future problems.

this might be a fix or it might be the wrong direction.   Remember the "weakest link";  auto resyncing with an error prone processor makes the whole system weak. 

[mlindner]

... I've seen a lot of equipment and more than one life lost because of redundancy induced complacency. (A phrase I just invented for this post)...

Same argument applies to ANY reliability increase, does it not?

For instance, SpaceX is going to improve their error-handling capability for the next mission, making resyncing automatic. This will make them more robust to future problems.

this might be a fix or it might be the wrong direction.   Remember the "weakest link";  auto resyncing with an error prone processor makes the whole system weak. 

Not sure if you have to worry about that. If it is error prone then it will be kicked back out of sync again. Also keep in mind that every processor is itself in 2-way voting to see if its sure. So if it keeps erroring it won't even finish rebooting before it errors again.

Conceivably it could be an error that only occurs in a certain location in the code and that location happens immediately after resync so you are constantly going in and out of sync. Well as long as they can guarantee a maximum bound on the length of resync they can wait till they are not busy with anything system critical. (No point in doing a resync during stage sep for example.) Presumably there is a lockout switch where they can disable auto-resync when anything critical could occur. (Disable during launch, stage sep, docking, undocking, etc.) That would enable a somewhat more reliable setup.