Author Topic: SpaceX CRS-1 Software/Computer Design Discussion Thread  (Read 36093 times)

Offline pippin

  • Regular
  • Senior Member
  • *****
  • Posts: 2575
  • Liked: 312
  • Likes Given: 45
Re: SpaceX CRS-1 Software/Computer Design Discussion Thread
« Reply #40 on: 11/21/2012 06:19 am »
I have no knowledge about SpaceX avionics architecture, but I'd be shocked beyond words if it were ARM based today.
Actually judging by the comments about the architecture I'd bet money on it being ARM.
Especially the point about having so many different computers points into that direction.
ARM is the de-facto standard for cores in embedded controllers these days and if you want to be scalable and keep your software modules it helps a lot to have a processor architecture that spans the range.

x86 is more complex and less predictable as an architecture, if I want something reliable I would not use it. Too many implementation bugs in there simply due to complexity, ARM is much better to control.
And then it's not available in small, low power controllers so if you want to use x86 in some computers, you'd have to support more than one architecture.

It's not true that using Linux, and C++ solves all your portability issues. It somewhat works with x86 and ARM, mainly because ARM supports both endianness modes. Even here you have to be careful with network and driver level code. The two other platforms that are somewhat supported are PPC and MIPS. PPC is getting out of flavor and you find fewer and fewer designs with it and MIPS is not available for high processing power requirements anymore.

It'll be at least mostly ARM. Maybe a few GPCs with x86 if they needs lots of floating point power and don't care about power but the majority of the smaller ones will be ARM.

Offline garidan

  • Full Member
  • *
  • Posts: 103
  • Italy
  • Liked: 19
  • Likes Given: 21
Re: SpaceX CRS-1 Software/Computer Design Discussion Thread
« Reply #41 on: 11/21/2012 08:08 am »
As a newbie I found intersting this paper about mars human exploration and radiation issues http://www.marsjournal.org/contents/2006/0004/files/rapp_mars_2006_0004.pdf

You can use rad hardened hardware but if you want to carry humans you have to work on shields.
In the long run it could be "smarter" for SpaceX to focus on shields and gain experience on it, investing bucks on it and saving on not rad hardened electronics.

I "believe" in redundancy , but a strong event (rad storm ?) could destroy all your redundant parts and leave you naked.

That paper says Hydrogen rich materials shield better: could it be feasible to put CPU boards and ram in a sphere inside the future methane tanks for free shielding ?
And what about redundant CPU boards put in orthogonal direction to minimize the damage in case of directional rays (sun bursts)?
Last thought, modern CPUs and RAM are way smaller, so they lessen the chance of a hit, but I presume the smaller transistors are damaged by smaller energy levels than bigger ones, is it true ?

Offline mlindner

  • Software Engineer
  • Senior Member
  • *****
  • Posts: 2908
  • Space Capitalist
  • Silicon Valley, CA
  • Liked: 2204
  • Likes Given: 818
Re: SpaceX CRS-1 Software/Computer Design Discussion Thread
« Reply #42 on: 11/21/2012 08:32 am »
You can use rad hardened hardware but if you want to carry humans you have to work on shields.
In the long run it could be "smarter" for SpaceX to focus on shields and gain experience on it, investing bucks on it and saving on not rad hardened electronics.
Radiation shielding is already a well understood topic, AFAIK the only interesting research going on in the field is electromagnetic shielding using some sort of plasma bubble (google M2P2), but this is still very theoretical and very far from implementation. The most likely solution will just be to point a large-ish tank of water directly at the sun. Elon Musk has already stated this as a good way of doing this as well. The other option is to just eat the radiation, "acceptable risk."
Quote
I "believe" in redundancy , but a strong event (rad storm ?) could destroy all your redundant parts and leave you naked.
I haven't heard of this being the case.
Quote
That paper says Hydrogen rich materials shield better: could it be feasible to put CPU boards and ram in a sphere inside the future methane tanks for free shielding?
This seems unlikely. The very cold temperatures inside the fuel are probably out of the operating temperature range of many semiconductors.
Quote
And what about redundant CPU boards put in orthogonal direction to minimize the damage in case of directional rays (sun bursts)?
A good idea, assuming you can maintain the orientation. Definitely possible, if not done already. This doesn't necessarily help the damage amounts though. If you get hit by a heavy ion it could go through many cells at once when flying through edge on.
Quote
Last thought, modern CPUs and RAM are way smaller, so they lessen the chance of a hit, but I presume the smaller transistors are damaged by smaller energy levels than bigger ones, is it true ?
There is a difference between a bit flip and permanent damage. Indeed if the transistor is smaller than it takes less energy to discharge/charge a RAM cell. The chip itself is not any larger or smaller though, there are just more transistors.
« Last Edit: 11/21/2012 08:34 am by mlindner »
LEO is the ocean, not an island (let alone a continent). We create cruise liners to ride the oceans, not artificial islands in the middle of them. We need a physical place, which has physical resources, to make our future out there.

Offline JBF

  • Full Member
  • ****
  • Posts: 1459
  • Liked: 472
  • Likes Given: 914
Re: SpaceX CRS-1 Software/Computer Design Discussion Thread
« Reply #43 on: 11/21/2012 11:31 am »
Replying to two items from this thread:

I watch flight flight boards get populated/placed and assembled in unit quantities down to qty=1 in the lab next to mine- on a regular basis.

Space rated PWA and PWB assembly is a different game than nearly anything commercial and absolutely everything high-volume.

Which is exactly why they cost so much.
"In principle, rocket engines are simple, but that’s the last place rocket engines are ever simple." Jeff Bezos

Offline MP99

A properly designed system should be plug-n-play being able to drop any number of computational "modules" into the loop as you want. As long as they didn't hardcode the 3 modules of 2 computing units architecture, it may already be capable of dropping additional modules into the loop.

This sounds like something a software engineer would say :)

Once you start adding hardware inputs/outputs into the equation, it becomes much harder to abstract your 'modules' in such a way. 

I found this to be a fantastic read on the subject of various architectures for tying multiple flight computers together. Most informative, and ISTM very relevant to this thread.

cheers, Martin

Offline Nomadd

  • Senior Member
  • *****
  • Posts: 8840
  • Lower 48
  • Liked: 60430
  • Likes Given: 1305
Re: SpaceX CRS-1 Software/Computer Design Discussion Thread
« Reply #45 on: 11/21/2012 12:29 pm »
 I've fabricated single boards/systems and fabricated batches of 50. If SpaceX bought dozens of these RAD hardened units I'd expect the unit price to be much lower. Can't say I'm sure of the process here, but making ten of anything at once often costs little more than making a single item.
Those who danced were thought to be quite insane by those who couldn't hear the music.

Offline MP99


With added redundancy, you have the additional benefit of more "near misses," which gives you more opportunities to improve the system. Without redundancy, you either succeed or you fail hard, much fewer near misses.


As far as I can tell not a single person has suggested there should be no redundancy.  Redundancy is obvious thing to have.

The discussion is wheter or not unlimted redundancy should be the anwer-all for unreliability. 

I suggest there is a middle ground where reliability is high but redundancy is there because things happen. 
Nobody disagrees with you on the idea there should be a middle ground. Or at least, nobody should.

I agree with this as well. The problem is there is a very large range of possible middle grounds. The inflection point could be much further out than conventionally thought. I trust SpaceX to do this calculation. My personal belief though is that there has to be a better solution rather than using 15 year old technology.

Following Moore's law there have been 10 doubling's in transistor density since then. Which implies a roughly 1024x fold increase in computation power since then. Meaning assuming you do distributed computing (even more radiation prone) and assuming that distributed computing scales linearly (it doesn't) you need roughly 1000 of these processors to get to the speed of one modern processor.

If your hardware elements (transistors, memory cells, etc) are 1000 times smaller, wouldn't that make them much more susceptible to individual rad hits, perhaps of lower energy?

Also, I wonder if a hit that would have affected one component before might now affect multiple?

cheers, Martin

Offline IRobot

  • Full Member
  • ****
  • Posts: 1312
  • Portugal & Germany
  • Liked: 310
  • Likes Given: 272
Re: SpaceX CRS-1 Software/Computer Design Discussion Thread
« Reply #47 on: 11/21/2012 01:38 pm »
If your hardware elements (transistors, memory cells, etc) are 1000 times smaller, wouldn't that make them much more susceptible to individual rad hits, perhaps of lower energy?

Also, I wonder if a hit that would have affected one component before might now affect multiple?
It also makes them more susceptible to burn on induced currents. Of course if tracks between transistors are smaller, induced currents are smaller.

Offline john smith 19

  • Senior Member
  • *****
  • Posts: 10351
  • Everyplaceelse
  • Liked: 2431
  • Likes Given: 13606
Re: SpaceX CRS-1 Software/Computer Design Discussion Thread
« Reply #48 on: 11/21/2012 02:27 pm »
Can't say I'm sure of the process here, but making ten of anything at once often costs little more than making a single item.
Quite probably
But that is the *cost* to BAe to make.

This is the *space* business.

You want some kind of buy-in-bulk *price* like a real business? :)
(I'm only *partly* joking about this).
MCT ITS BFR SS. The worlds first Methane fueled FFSC engined CFRP SS structure A380 sized aerospaceplane tail sitter capable of Earth & Mars atmospheric flight.First flight to Mars by end of 2022 TBC. T&C apply. Trust nothing. Run your own #s "Extraordinary claims require extraordinary proof" R. Simberg."Competitve" means cheaper ¬cheap SCramjet proposed 1956. First +ve thrust 2004. US R&D spend to date > $10Bn. #deployed designs. Zero.

Offline Robotbeat

  • Senior Member
  • *****
  • Posts: 39270
  • Minnesota
  • Liked: 25240
  • Likes Given: 12115
Re: SpaceX CRS-1 Software/Computer Design Discussion Thread
« Reply #49 on: 11/21/2012 02:30 pm »

With added redundancy, you have the additional benefit of more "near misses," which gives you more opportunities to improve the system. Without redundancy, you either succeed or you fail hard, much fewer near misses.


As far as I can tell not a single person has suggested there should be no redundancy.  Redundancy is obvious thing to have.

The discussion is wheter or not unlimted redundancy should be the anwer-all for unreliability. 

I suggest there is a middle ground where reliability is high but redundancy is there because things happen. 
Nobody disagrees with you on the idea there should be a middle ground. Or at least, nobody should.

I agree with this as well. The problem is there is a very large range of possible middle grounds. The inflection point could be much further out than conventionally thought. I trust SpaceX to do this calculation. My personal belief though is that there has to be a better solution rather than using 15 year old technology.

Following Moore's law there have been 10 doubling's in transistor density since then. Which implies a roughly 1024x fold increase in computation power since then. Meaning assuming you do distributed computing (even more radiation prone) and assuming that distributed computing scales linearly (it doesn't) you need roughly 1000 of these processors to get to the speed of one modern processor.

If your hardware elements (transistors, memory cells, etc) are 1000 times smaller, wouldn't that make them much more susceptible to individual rad hits, perhaps of lower energy?

Also, I wonder if a hit that would have affected one component before might now affect multiple?

cheers, Martin
I've seen arguments go the other way WRT radiation bit-flipping. Modern techniques in manufacturing them make them less susceptible. They are also smaller, thus less likely to be hit... And you can afford more features like ECC that provide more resiliency.
Chris  Whoever loves correction loves knowledge, but he who hates reproof is stupid.

To the maximum extent practicable, the Federal Government shall plan missions to accommodate the space transportation services capabilities of United States commercial providers. US law http://goo.gl/YZYNt0

Offline jimvela

  • Member
  • Full Member
  • ****
  • Posts: 1662
  • Liked: 900
  • Likes Given: 71
Re: SpaceX CRS-1 Software/Computer Design Discussion Thread
« Reply #50 on: 11/21/2012 02:34 pm »
Replying to two items from this thread:

I watch flight flight boards get populated/placed and assembled in unit quantities down to qty=1 in the lab next to mine- on a regular basis.

Space rated PWA and PWB assembly is a different game than nearly anything commercial and absolutely everything high-volume.

Which is exactly why they cost so much.

That cost is in the noise compared to the cost of a failure- which is why they are built that way.

Offline john smith 19

  • Senior Member
  • *****
  • Posts: 10351
  • Everyplaceelse
  • Liked: 2431
  • Likes Given: 13606
Re: SpaceX CRS-1 Software/Computer Design Discussion Thread
« Reply #51 on: 11/21/2012 02:39 pm »
That cost is in the noise compared to the cost of a failure- which is why they are built that way.
So it's the QA in design & build coupled with testing *after* mfg and population that soaks up the cash?

I'd guessed it might have something to do with needing some kind of forced flow (either gas or liquid) cooling due to zero g.

This also raises a point. Are layer counts and line widths for space rated PWA's and PWC's (Those sound like IBM terms, I thought most people call them PCB's) behind those of terrestrial boards in the same way as space rated parts tend to be a generation or 2 behind their ground based equivalents?
MCT ITS BFR SS. The worlds first Methane fueled FFSC engined CFRP SS structure A380 sized aerospaceplane tail sitter capable of Earth & Mars atmospheric flight.First flight to Mars by end of 2022 TBC. T&C apply. Trust nothing. Run your own #s "Extraordinary claims require extraordinary proof" R. Simberg."Competitve" means cheaper ¬cheap SCramjet proposed 1956. First +ve thrust 2004. US R&D spend to date > $10Bn. #deployed designs. Zero.

Offline jimvela

  • Member
  • Full Member
  • ****
  • Posts: 1662
  • Liked: 900
  • Likes Given: 71
Re: SpaceX CRS-1 Software/Computer Design Discussion Thread
« Reply #52 on: 11/21/2012 02:52 pm »
That cost is in the noise compared to the cost of a failure- which is why they are built that way.
So it's the QA in design & build coupled with testing *after* mfg and population that soaks up the cash?

Yes.  Plus, the cost of resolving a test anomaly can exceed the cost of building a brand new board- particularly with an unverified failure.

Quote
I'd guessed it might have something to do with needing some kind of forced flow (either gas or liquid) cooling due to zero g.

It is my experience that flight boards are cooled by conduction then radiating the heat. 

Quote
This also raises a point. Are layer counts and line widths for space rated PWA's and PWC's (Those sound like IBM terms, I thought most people call them PCB's) behind those of terrestrial boards in the same way as space rated parts tend to be a generation or 2 behind their ground based equivalents?

In my experience, yes.  Especially when you have whole planes dedicated to thermal management.


Online mmeijeri

  • Senior Member
  • *****
  • Posts: 7772
  • Martijn Meijering
  • NL
  • Liked: 397
  • Likes Given: 822
Re: SpaceX CRS-1 Software/Computer Design Discussion Thread
« Reply #53 on: 11/21/2012 05:42 pm »
An uneducated guess: on a slow system the realtime requirements may not be met with Linux and C++.

I doubt it. You can do cycle-perfect simulations of Apollo hardware in Javascript in a browser nowadays, so that can't be it. Console video games run on limited hardware too, and C++ is the language of choice for that.
Pro-tip: you don't have to be a jerk if someone doesn't agree with your theories

Online mmeijeri

  • Senior Member
  • *****
  • Posts: 7772
  • Martijn Meijering
  • NL
  • Liked: 397
  • Likes Given: 822
Re: SpaceX CRS-1 Software/Computer Design Discussion Thread
« Reply #54 on: 11/21/2012 05:43 pm »
I think SpaceX is just trying to follow Amdahl's Law in that you shouldn't optimize a small part of the problem.

That could well be, but that's not what they appeared to be saying.
Pro-tip: you don't have to be a jerk if someone doesn't agree with your theories

Offline guckyfan

  • Senior Member
  • *****
  • Posts: 7438
  • Germany
  • Liked: 2332
  • Likes Given: 2891
Re: SpaceX CRS-1 Software/Computer Design Discussion Thread
« Reply #55 on: 11/21/2012 06:28 pm »
An uneducated guess: on a slow system the realtime requirements may not be met with Linux and C++.

I doubt it. You can do cycle-perfect simulations of Apollo hardware in Javascript in a browser nowadays, so that can't be it. Console video games run on limited hardware too, and C++ is the language of choice for that.

That is quite a few orders of magnitude slower. Some here on the forum were even surprised they use Linux at all because it is not hard realtime.


Offline A_M_Swallow

  • Elite Veteran
  • Senior Member
  • *****
  • Posts: 8906
  • South coast of England
  • Liked: 500
  • Likes Given: 223
Re: SpaceX CRS-1 Software/Computer Design Discussion Thread
« Reply #56 on: 11/21/2012 06:45 pm »
An uneducated guess: on a slow system the realtime requirements may not be met with Linux and C++.

I doubt it. You can do cycle-perfect simulations of Apollo hardware in Javascript in a browser nowadays, so that can't be it. Console video games run on limited hardware too, and C++ is the language of choice for that.

That is quite a few orders of magnitude slower. Some here on the forum were even surprised they use Linux at all because it is not hard realtime.

Linux is a re-implementation of Unix.  Soft real time Unix made its living controlling telephone exchanges.  For SpaceX it probably comes down to how fast a rocket engine can gimbal.
« Last Edit: 11/21/2012 06:46 pm by A_M_Swallow »

Offline Robotbeat

  • Senior Member
  • *****
  • Posts: 39270
  • Minnesota
  • Liked: 25240
  • Likes Given: 12115
Re: SpaceX CRS-1 Software/Computer Design Discussion Thread
« Reply #57 on: 11/21/2012 06:47 pm »
An uneducated guess: on a slow system the realtime requirements may not be met with Linux and C++.

I doubt it. You can do cycle-perfect simulations of Apollo hardware in Javascript in a browser nowadays, so that can't be it. Console video games run on limited hardware too, and C++ is the language of choice for that.

That is quite a few orders of magnitude slower. Some here on the forum were even surprised they use Linux at all because it is not hard realtime.

Linux is a re-implementation of Unix.  Soft real time Unix made its living controlling telephone exchanges.  For SpaceX it probably comes down to how fast a rocket engine can gimbal.
I'm pretty sure SpaceX isn't using Linux in that portion of their avionics... probably some other embedded, fully real-time operating system.
« Last Edit: 11/21/2012 06:47 pm by Robotbeat »
Chris  Whoever loves correction loves knowledge, but he who hates reproof is stupid.

To the maximum extent practicable, the Federal Government shall plan missions to accommodate the space transportation services capabilities of United States commercial providers. US law http://goo.gl/YZYNt0

Offline john smith 19

  • Senior Member
  • *****
  • Posts: 10351
  • Everyplaceelse
  • Liked: 2431
  • Likes Given: 13606
Re: SpaceX CRS-1 Software/Computer Design Discussion Thread
« Reply #58 on: 11/21/2012 11:30 pm »
I'm pretty sure SpaceX isn't using Linux in that portion of their avionics... probably some other embedded, fully real-time operating system.

You may be right but there's a fairly active embedded Linux community and they've been tightening up Linux's RT performance for about the last *decade*. At least one groups aim is to consistently get their improvements into the *core* approved version rather than as build options. There are also strategies you can employ. IIRC Armadillo's stuff all runs on Linux and they use a *single* monolithic app on the RT control boards. 1 job, 1 board. Poor *commercial* development practice but works just fine in *this* environment, whre you have *total* control of what code runs on what hardware. They don't really do "task switching" as such but leverage the Linux toolset.

One small details about the AvWeek interview was he said "Different processors," not different *architectures*. My instinct is they have *one* architecture they support. If they need more MIPS they swallow the power budget and go to a higher clock frequency or a variant with on chip peripherals better tuned to the needs of that function.

As to *what* that architecture is I have no idea. Someone said no way is it ARM but did not explain *why* and I think the front runners are x86 and ARM architectures. They are widespread, and *most* support tools target them as core options and hardware MMU's makes memory protection fairly easy.
« Last Edit: 11/21/2012 11:54 pm by john smith 19 »
MCT ITS BFR SS. The worlds first Methane fueled FFSC engined CFRP SS structure A380 sized aerospaceplane tail sitter capable of Earth & Mars atmospheric flight.First flight to Mars by end of 2022 TBC. T&C apply. Trust nothing. Run your own #s "Extraordinary claims require extraordinary proof" R. Simberg."Competitve" means cheaper ¬cheap SCramjet proposed 1956. First +ve thrust 2004. US R&D spend to date > $10Bn. #deployed designs. Zero.

Offline mlindner

  • Software Engineer
  • Senior Member
  • *****
  • Posts: 2908
  • Space Capitalist
  • Silicon Valley, CA
  • Liked: 2204
  • Likes Given: 818
Re: SpaceX CRS-1 Software/Computer Design Discussion Thread
« Reply #59 on: 11/22/2012 01:18 pm »
An uneducated guess: on a slow system the realtime requirements may not be met with Linux and C++.

I doubt it. You can do cycle-perfect simulations of Apollo hardware in Javascript in a browser nowadays, so that can't be it. Console video games run on limited hardware too, and C++ is the language of choice for that.

That is quite a few orders of magnitude slower. Some here on the forum were even surprised they use Linux at all because it is not hard realtime.

This is a mis-representation of "real-time." If your system runs fast enough then even if it is not "real-time," it acts as if it is. As long as you can service events fast enough.
LEO is the ocean, not an island (let alone a continent). We create cruise liners to ride the oceans, not artificial islands in the middle of them. We need a physical place, which has physical resources, to make our future out there.

Tags:
 

Advertisement NovaTech
Advertisement Northrop Grumman
Advertisement
Advertisement Margaritaville Beach Resort South Padre Island
Advertisement Brady Kenniston
Advertisement NextSpaceflight
Advertisement Nathan Barker Photography
0