Author Topic: SpaceX COTS Demo 2 Updates (Read 448896 times)

NotGncDude · « **Reply #420 on:** 05/21/2011 06:42 pm »

Quote from: marsavian on 05/21/2011 06:09 pm

Quote from: mr. mark on 05/21/2011 05:48 pm
It's probably wiser for me refrain from comments until I know the extent of any problems in the chain. It may simply be growing pains as happens with any business including my own. A good management team will overcome these initial problems before moving on to full production. Spacex is still in the testing phase of COTS and Falcon 9 and can expect disruptions.

This looks more like a design fault than an initial teething problem, something whose bad effects will only be found out from a random radiation event. They need to seriously look at this again if they are just using standard PC chips. If they want to use standard commercial than a HP NonStop (Tandem) Itanium blade or small IBM System Z Mainframe would be much better than a $1K PC, they are designed for the best fault tolerance commercial computer systems on Earth. Even the latest E3/E7 Xeons would be better and they still use x86 chips like PCs and could run the same software.

http://h20223.www2.hp.com/NonStopComputing/cache/307953-0-0-0-121.html
http://www-03.ibm.com/systems/z/
http://www.theinquirer.net/inquirer/news/2041343/hp-releases-80-core-intel-xeon-e7-server

It's most certainly not a design flaw, but designed like this on purpose. SpaceX uses triple computers with voting *everywhere*. Nice and easy way of getting around the radiation hardening issue.

They don't use x86 either, I don't think. Somebody from SpaceX explained the setup to me once and it blew my mind (about how complex it is, not necessarily as a good thing). Something about PowerPC cores running on FPGAs etc, very bizarre, but he wasn't a computer architecture guy so he didn't go into many details.

jimvela · « **Reply #421 on:** 05/21/2011 06:45 pm »

Quote from: marsavian on 05/21/2011 06:09 pm

If they want to use standard commercial than a HP NonStop (Tandem) Itanium blade or small IBM System Z Mainframe would be much better than a $1K PC, they are designed for the best fault tolerance commercial computer systems on Earth.

You're still not taking the subtle hint. Avionics boxes aren't designed and built that way.

NotGncDude · « **Reply #422 on:** 05/21/2011 06:45 pm »

Quote from: marsavian on 05/21/2011 02:20 pm

Well they have chosen not to use hardened chips for some reason but you can buy commercial server chips that are designed to be very recoverable from radiation effects. Case in point ...

http://science.nasa.gov/science-news/science-at-nasa/2005/18nov_eaftc/

Oh, you pointed it out too.

joek · « **Reply #423 on:** 05/21/2011 06:58 pm »

Quote from: Nate_Trost on 05/21/2011 06:30 pm

Even if they aren't using one of the rad-hard CPUs, it's far more likely they are using established embedded platforms and one of the conventional RTOSes.

VxWorks is used on F9 and Dragon; not sure about ~~Dragon or~~ F1. (IIRC Musk mentioned it in one of his factory tour videos and Wind River had a post about it. See also DragonLab data sheet.)

edit: Dragon also.

Nascent Ascent · « **Reply #424 on:** 05/21/2011 07:25 pm »

.....It's most certainly not a design flaw, but designed like this on purpose. SpaceX uses triple computers with voting *everywhere*. Nice and easy way of getting around the radiation hardening issue.....

GNC,

Come on now. You're ruining the fun for the SpaceX crapehangers! It's fun to be able to pull out a single line from a PPT presentation and inflate it to the point of SpaceX failure.

I understand that one should be careful not to be a blind cheerleader but it seems that on the flip side, there's just a bit of schadenfreude here.

RocketEconomist327 · « **Reply #425 on:** 05/21/2011 07:44 pm »

I'm calling Elon now and telling him to shut it down. Clearly the posters here on NSF can armchair engineer this much better than anything that SpaceX can do.

Stick a fork in it.

Falcon is dead.

Dragon is dead.

Falcon Heavy is dead.

Oh snap, wasn't the world supposed to end today!?

*waves arms* oh noez

butters · « **Reply #426 on:** 05/21/2011 07:47 pm »

Quote from: marsavian on 05/21/2011 06:09 pm

This looks more like a design fault than an initial teething problem, something whose bad effects will only be found out from a random radiation event. They need to seriously look at this again if they are just using standard PC chips. If they want to use standard commercial than a HP NonStop (Tandem) Itanium blade or small IBM System Z Mainframe would be much better than a $1K PC...

What do Itanium and System Z have in common? They were largely overtaken by fault-tolerant clustering of relatively inexpensive machines, except where it wouldn't be cost-effective to port existing mainframe applications. One of my first jobs was developing a clustering platform for IBM's midrange UNIX boxes. Even IBM understands that when Mark Twain (or Andrew Carnegie) said "put all your eggs in one basket, and watch that basket", he was just kidding.

You have at least three computers, and if two vote against one, you switch the bus master if necessary and reset the failed computer.You assume that memory corruption is inevitable and design the system so that it isn't a big deal.

renclod · « **Reply #427 on:** 05/21/2011 08:01 pm »

Reading from a couple presentations from CNES (that came up first in a web search), it looks like there are multiple fault/failure-tolerant architectures with Commercial Off-The-Shelf (COTS) electronics onboard spacecraft.

COTS "provide system-level performance that their Hi-Rel/Rad-Tol counterparts cannot achieve", and of course much lower procurement cost.

The problems are with cost of lot acceptance test + cost of validation.

The proposed validation methodology seems especially cumbersome (to me!); laser beam, heavy-ion beam ...

So when Mr. Gerstenmaier said "Whether this could cause loss of mission is unknown" it could simply mean that no funds have yet been allocated to perform a full validation of Dragon avionics tolerance to single-event upsets.

edit. In other words, it's a money issue.

edit again . I will just attach one of the informative CNES papers.

Robotbeat · « **Reply #428 on:** 05/21/2011 08:15 pm »

How could the Station-to-Dragon comm frequency (and power) issue crop up only now? Was it believed before that it'd be fine, but now some bureaucrat is making a fuss of it, or is this a real oversight by NASA and/or SpaceX?

ugordan · « **Reply #429 on:** 05/21/2011 08:17 pm »

I'm reminded of one of the old Soviet Mars probes that also employed 2-out-of-3 voting computers. Except in that case the computers chips were defective - known before launch but politics/schedule pressure made them launch anyway. Well, in the end they got 2 faulty computers outvoting the healthy one. Mission lost.

Then again, those weren't due to bit flips but actual electronics degradation (i.e. unrecoverable), but it's an amusing story nonetheless. Probably not so amusing to those scientists and engineers back then...

gospacex · « **Reply #430 on:** 05/21/2011 08:22 pm »

Quote from: Chris-A on 05/21/2011 06:09 pm

Processors are considered to be very reliable for consumer and server grade electronics. Memory like RAM, is a much different story. I can't imagine SDRAM with error correcting features working in orbit.

ISS crew apparently uses *ordinary, consumer-grade laptops*, and from what I hear they don't have to use 10 cm thick lead boxes to keep them from failing.

Don't create mountain out of mole hill.

Jorge · « **Reply #431 on:** 05/21/2011 08:24 pm »

Quote from: gospacex on 05/21/2011 08:22 pm

Quote from: Chris-A on 05/21/2011 06:09 pm
Processors are considered to be very reliable for consumer and server grade electronics. Memory like RAM, is a much different story. I can't imagine SDRAM with error correcting features working in orbit.

ISS crew apparently uses *ordinary, consumer-grade laptops*, and from what I hear they don't have to use 10 cm thick lead boxes to keep them from failing.

They do fail quite frequently, though. They would never be trusted for critical work.

Quote

Don't create mountain out of mole hill.

Don't minimize it, either.

marsavian · « **Reply #432 on:** 05/21/2011 08:26 pm »

Quote from: ugordan on 05/21/2011 08:17 pm

I'm reminded of one of the old Soviet Mars probes that also employed 2-out-of-3 voting computers. Except in that case the computers chips were defective - known before launch but politics/schedule pressure made them launch anyway. Well, in the end they got 2 faulty computers outvoting the healthy one. Mission lost.

Then again, those weren't due to bit flips but actual electronics degradation (i.e. unrecoverable), but it's an amusing story nonetheless. Probably not so amusing to those scientists and engineers back then...

Yes there is the possibility that two or even three cores could be taken out which is why the core chosen should be server grade fault-tolerant to minimize the risk. Power 7/Sparc64 too are in the same class.

Robotbeat · « **Reply #433 on:** 05/21/2011 08:30 pm »

You can get RAM in a modern server which can handle multiple simultaneous bit errors, even the failure of an entire RAM chip (with the ability to use another chip as a spare and memory mirroring). That's actually COTS and very cheap (motherboards with these features are $x00, though aren't necessarily what you'd use), though I don't think it's used for laptops (like the ones on Station).

Combined with a realtime watchdog and a capability for very quick rebooting, you have a relatively robust system at low cost. Without the expense of RAD hardening (and RAD hardening doesn't make you immune, either... it's possible all these features add up to better reliability for the RAM than RAD-hard RAM without these features, if not the main CPU). But of course, that's not the "way it's always been done..."

marsavian · « **Reply #434 on:** 05/21/2011 08:32 pm »

Chips have internal fast memory (cache) that needs protecting too and not all chips do it to the same degree with some not at all.

Jorge · « **Reply #435 on:** 05/21/2011 08:33 pm »

Quote from: RocketScientist327 on 05/21/2011 07:44 pm

I'm calling Elon now and telling him to shut it down. Clearly the posters here on NSF can armchair engineer this much better than anything that SpaceX can do.

Considering that the posters here on NSF can armchair engineer better than NASA (just ask them), why not?

Jim · « **Reply #436 on:** 05/21/2011 08:37 pm »

All this talk about resets and reboots is fine and dandy for on orbit but not for ascent

Rocket Science · « **Reply #437 on:** 05/21/2011 08:38 pm »

Quote from: Robotbeat on 05/21/2011 08:15 pm

How could the Station-to-Dragon comm frequency (and power) issue crop up only now? Was it believed before that it'd be fine, but now some bureaucrat is making a fuss of it, or is this a real oversight by NASA and/or SpaceX?

Oversight, remember this...?
http://articles.cnn.com/1999-09-30/tech/9909_30_mars.metric_1_mars-orbiter-climate-orbiter-spacecraft-team?_s=PM:TECH

Robotbeat · « **Reply #438 on:** 05/21/2011 08:40 pm »

Quote from: Rocket Science on 05/21/2011 08:38 pm

Quote from: Robotbeat on 05/21/2011 08:15 pm
How could the Station-to-Dragon comm frequency (and power) issue crop up only now? Was it believed before that it'd be fine, but now some bureaucrat is making a fuss of it, or is this a real oversight by NASA and/or SpaceX?
Oversight, remember this...?
http://articles.cnn.com/1999-09-30/tech/9909_30_mars.metric_1_mars-orbiter-climate-orbiter-spacecraft-team?_s=PM:TECH

This is a different issue. I believe this is about Station-Dragon comms interfering with other nations' radio frequencies. But yeah, point taken.

Rocket Science · « **Reply #439 on:** 05/21/2011 08:42 pm »

Quote from: Robotbeat on 05/21/2011 08:40 pm

Quote from: Rocket Science on 05/21/2011 08:38 pm
Quote from: Robotbeat on 05/21/2011 08:15 pm
How could the Station-to-Dragon comm frequency (and power) issue crop up only now? Was it believed before that it'd be fine, but now some bureaucrat is making a fuss of it, or is this a real oversight by NASA and/or SpaceX?
Oversight, remember this...?
http://articles.cnn.com/1999-09-30/tech/9909_30_mars.metric_1_mars-orbiter-climate-orbiter-spacecraft-team?_s=PM:TECH
This is a different issue. I believe this is about Station-Dragon comms interfering with other nations' radio frequencies. But yeah, point taken.

I know....