SpaceX CRS-1 Software/Computer Design Discussion Thread

[GalacticIntruder]

SpX says rad problems expected. much ado about nothing. Their non hardened, but tolerant systems approach are superior. I find it hard to believe they could do that in a Mars transit, but they have a plan. I just have to take their word for it.


http://www.aviationweek.com/Blogs.aspx?plckBlogId=Blog:04ce340e-4b63-4d23-9695-d49ab661f385&plckPostId=Blog%3a04ce340e-4b63-4d23-9695-d49ab661f385Post%3aa8b87703-93f9-4cdf-885f-9429605e14df

[Go4TLI]

Yes, you have an opinion on everthing and seem to act like an expert on just about everything as well.  As I said, carry on.   

To quote Jim: Pot, meet kettle.

 

But I have actually been part of a team that has developed, tested, engineered, built, and flown hardware in space and brought it back on multiple programs for about 15 years now. 

I don't claim to be an expert on everything and I don't comment where I don't have insight, hence why I am void from certain parts of this forum or on certain subject matter, unlike some. 

Don't mistake experience for anything else but what it is.  If you and others don't like it, don't read it or don't comment on it.  Hell, it's not like I don't have other things to do anyway if true experience is not welcome here.

[Nomadd]

 

Don't mistake experience for anything else but what it is.  If you and others don't like it, don't read it or don't comment on it.  Hell, it's not like I don't have other things to do anyway if true experience is not welcome here.

 Your posts are always welcome as far as I'm concerned.
 I agree with you. Using redundancy can be a good way to increase reliability, but it's often used as a way to slack off on standards. I've seen a lot of equipment and more than one life lost because of redundancy induced complacency. (A phrase I just invented for this post)
 Saying that one computer failing isn't a big deal because there are two more is a great way to insure a LOM. It's no more acceptable than ignoring the loss on an engine because you still had 8 working ones.
 Knowing SpaceX they'll get into the radiation hardened electronics business now. Probably mine their own silicon.

[JBF]

 

Don't mistake experience for anything else but what it is.  If you and others don't like it, don't read it or don't comment on it.  Hell, it's not like I don't have other things to do anyway if true experience is not welcome here.

 Your posts are always welcome as far as I'm concerned.
 I agree with you. Using redundancy can be a good way to increase reliability, but it's often used as a way to slack off on standards. I've seen a lot of equipment and more than one life lost because of redundancy induced complacency. (A phrase I just invented for this post)
 Saying that one computer failing isn't a big deal because there are two more is a great way to insure a LOM. It's no more acceptable than ignoring the loss on an engine because you still had 8 working ones.
 Knowing SpaceX they'll get into the radiation hardened electronics business now. Probably mine their own silicon.

They didn't loose a computer it rebooted just fine. NASA just didn't want them to resysnc it while it was at the ISS due to the complication of explaining it to all the partners. According to what's been reported elsewhere SpaceX just plans to make the resyncing an automatic operation.

[Go4TLI]

They didn't loose a computer it rebooted just fine. NASA just didn't want them to resysnc it while it was at the ISS due to the complication of explaining it to all the partners. According to what's been reported elsewhere SpaceX just plans to make the resyncing an automatic operation.

This is from some of the reports on what went wrong:

•One of three flight computers failed while Dragon was docked at ISS due to a suspected radiation hit. The computer was restarted but could not re-synchronize with the other two units. The computer was restarted but was not resynchronized with the other two units. SpaceX says that NASA felt it was not necessary to continue the mission.
•One of three GPS units, the Propulsion and Trunk computers and Ethernet switch also experienced suspected radiation hits, but they were recovered during a power cycle.

This is for a about a 2 week long flight where the majority of the time it was not doing anything and just attached to ISS. 

While these anomolies were managed, on a more active flight it becomes an unnecessary distraction and issues that must be worked.  It takes away from the reason for the flight in the first place and, law of averages suggest, that someday they will have to deal with a more serious flight issue. 

These issues that crop up because their electronics are unreliable or under the constant threat of being disrupted by rad hits will make it much more difficult and just inputs additional risk into the mission for little reason as far as I can tell. 

[mlindner]

Knowing SpaceX they'll get into the radiation hardened electronics business now. Probably mine their own silicon.

I've noticed a problem on this board that while everyone here is _very_ knowledgable on rocket technology, there seems to be quite an absence of knowledge of how the integrated circuit industry works and the design processes involved. (Not picking on anyone specifically.)

There is a reason a giant like Apple had to _buy_ a company to make the _designs_ of the hardware. Even more so, the company they bought, itself, _buys_ parts of the design of the hardware from ARM. And then even then, they still don't actually burn anything to silicon, they contract that out to Samsung (oh the irony).

There is no way in hell SpaceX will actually start manufacturing its own integrated circuits. They said they don't even manufacture their own printed circuit boards in house when I asked them about it at the career fair here a few months ago.

The whole idea of making rad-hardened extreme-expense parts is part of the era of too-big-to-fail philosophy. Using enough computation and redundancy you can automatically correct and adjust for failure in hardware.

[mlindner]

They didn't loose a computer it rebooted just fine. NASA just didn't want them to resysnc it while it was at the ISS due to the complication of explaining it to all the partners. According to what's been reported elsewhere SpaceX just plans to make the resyncing an automatic operation.

These issues that crop up because their electronics are unreliable or under the constant threat of being disrupted by rad hits will make it much more difficult and just inputs additional risk into the mission for little reason as far as I can tell. 

I suggest you read this url.

[Go4TLI]

The whole idea of making rad-hardened extreme-expense parts is part of the era of too-big-to-fail philosophy. Using enough computation and redundancy you can automatically correct and adjust for failure in hardware.

For the rest of your post, I have to say I'm pretty sure the person you were responding to was saying tongue-in-cheek. 

Now for this, please see my previous post.  I would not call rad-hardened components "extreme expense".  It takes a certain amount of money and effort to qualify components to be rad-hardened but once they are, they are only moderately more expensive than non rad-hardened. 

This is why rad-hardened electronics lag the industry here on the ground.  Trade that against buying all the parts necessary to add whatever layer of redundancy to compensate for non-reliability and it is a wash or your solution is more expensive. 

Finally your "too-big-to-fail" line is inaccurate at best.  You want to imply that this is a product of by-gone era but on the flip side being cavalier about it is equally disturbing.  Especially for a company that claims it will send people, etc to Mars. 

Perhaps there are other ways of doing things, never suggested otherwise, but discounting hard-won experience operating in the most dangerous environment possible is naive. 

[alk3997]

I think I'll make this my last append on the subject since I only got interested because someone was misquoting GPC specs.  However, I'd like to make a few points, if you don't mind...

1) This is particle radiation that is impacting memory, registers, etc.  Particle radiation comes in different flavors, whether that be protons or heavy ions or positrons.  Each generates a different type of effect.  For instance, a heavy ion can impact multiple memory cells.  These particles are moving fast and are high energy for the extremely small size of the particle.  Some equipment can be pretty immune to protons and very soft to heavy ions.

2) There are a few rules-of-thumb.  First is that if you see a pattern, it isn't a radiation hit (I think I may have written that previously).  Second, an SEU can do anything that bad programming can do.  So when you are looking for effects of SEUs, just imagine if a compiler changes a random 1 to a 0 when creating the object code.  What will that do?  Maybe someone can work out the different probabilities.  Also writing to a location will remove the SEU.

3) There are no absolutes.  An SEU is a random event.  Where it occurs in memory is also a random location.  So, to say that an SEU will occur at this particular time is incorrect.  You can average them or say that over a flight you'll get this many *on average*, but being more specific than that is not possible.

4) A slight clarification to #1 and #3 is that any good discussion of SEUs in LEO must include the South Atlantic Anomaly.  More proton events occur over the SAA than anywhere else.  We had 6 ThinkPads go down nearly simultaneously while over the SAA on one flight.  Guess which type of Shuttle flight that was?  (hint: high altitude)

5) The LEO environment is much different than beyond the Van Allen belts.

6) We haven't even discussed latch-ups rather than SEUs.  Latch-ups are always a possibility (although small in LEO for memory).

Hope all of that helps your informed discussions.

Andy

[mmeijeri]

Latch-ups are always a possibility (although small in LEO for memory).

Isn't Silicon on Insulator supposed to be immune to latchups?

[Robotbeat]

... I've seen a lot of equipment and more than one life lost because of redundancy induced complacency. (A phrase I just invented for this post)...

Same argument applies to ANY reliability increase, does it not?

With added redundancy, you have the additional benefit of more "near misses," which gives you more opportunities to improve the system. Without redundancy, you either succeed or you fail hard, much fewer near misses.

For instance, SpaceX is going to improve their error-handling capability for the next mission, making resyncing automatic. This will make them more robust to future problems.

[Go4TLI]

With added redundancy, you have the additional benefit of more "near misses," which gives you more opportunities to improve the system. Without redundancy, you either succeed or you fail hard, much fewer near misses.

As far as I can tell not a single person has suggested there should be no redundancy.  Redundancy is obvious thing to have.

The discussion is wheter or not unlimted redundancy should be the anwer-all for unreliability. 

I suggest there is a middle ground where reliability is high but redundancy is there because things happen. 

[Robotbeat]

With added redundancy, you have the additional benefit of more "near misses," which gives you more opportunities to improve the system. Without redundancy, you either succeed or you fail hard, much fewer near misses.

As far as I can tell not a single person has suggested there should be no redundancy.  Redundancy is obvious thing to have.

The discussion is wheter or not unlimted redundancy should be the anwer-all for unreliability. 

I suggest there is a middle ground where reliability is high but redundancy is there because things happen. 

Nobody disagrees with you on the idea there should be a middle ground. Or at least, nobody should.

[Go4TLI]

Nobody disagrees with you on the idea there should be a middle ground. Or at least, nobody should.

"You can go a LONG ways just adding redundancy and still end up with a much /more/ reliable system even with FEWER reliable components. This is more true with computer systems than it is for other engineered systems."

That's a quote from you.  I read that as suggesting reliability is unimportant as long as there is sufficient redundancy to compensate. 

The problem is how deep that redundancy has to be is likely a variable that is a function of the issue at hand the operations being performed at the time. 

[JBF]

This is very off topic, but as someone who designs PCB, no one except the very largest manufacturers etch their own PCBs; and unless you are manufacturing at least several hundred assemblies a month it's generally not worth it to do your own component placement either.

The primary reason rad-hardened components are so expensive is scale of manufacture. The smaller the run of silicon wafers the more expensive the set-up time is per wafer.

There is no way in hell SpaceX will actually start manufacturing its own integrated circuits. They said they don't even manufacture their own printed circuit boards in house when I asked them about it at the career fair here a few months ago.

[mlindner]

With added redundancy, you have the additional benefit of more "near misses," which gives you more opportunities to improve the system. Without redundancy, you either succeed or you fail hard, much fewer near misses.

As far as I can tell not a single person has suggested there should be no redundancy.  Redundancy is obvious thing to have.

The discussion is wheter or not unlimted redundancy should be the anwer-all for unreliability. 

I suggest there is a middle ground where reliability is high but redundancy is there because things happen. 

Nobody disagrees with you on the idea there should be a middle ground. Or at least, nobody should.

I agree with this as well. The problem is there is a very large range of possible middle grounds. The inflection point could be much further out than conventionally thought. I trust SpaceX to do this calculation. My personal belief though is that there has to be a better solution rather than using 15 year old technology.

Following Moore's law there have been 10 doubling's in transistor density since then. Which implies a roughly 1024x fold increase in computation power since then. Meaning assuming you do distributed computing (even more radiation prone) and assuming that distributed computing scales linearly (it doesn't) you need roughly 1000 of these processors to get to the speed of one modern processor.

[Robotbeat]

Nobody disagrees with you on the idea there should be a middle ground. Or at least, nobody should.

"You can go a LONG ways just adding redundancy and still end up with a much /more/ reliable system even with FEWER reliable components. This is more true with computer systems than it is for other engineered systems."

That's a quote from you.  I read that as suggesting reliability is unimportant as long as there is sufficient redundancy to compensate. 

That is something that certainly isn't true for many engineered systems but for computer systems, it really can be true. It takes more software development prowess, which can often turn out to be more expensive than just throwing hardware (i.e. better rad-hard capabilities) at it, but it essentially is true.

The problem is how deep that redundancy has to be is likely a variable that is a function of the issue at hand the operations being performed at the time. 

Your statement is more true for mechanical systems than it is for modern computer systems, owing to the fact that there have been several, several orders of magnitude improvements in capabilities beyond what's strictly necessary. Perhaps a factor of a million greater than strictly necessary.

Another point is that while throwing redundancy at the problem can indeed solve almost any reliability issue, the cost of such a decision may be an order of magnitude increase in complexity and software development costs.

[mlindner]

Another point is that while throwing redundancy at the problem can indeed solve almost any reliability issue, the cost of such a decision may be an order of magnitude increase in complexity and software development costs.

A properly designed system should be plug-n-play being able to drop any number of computational "modules" into the loop as you want. As long as they didn't hardcode the 3 modules of 2 computing units architecture, it may already be capable of dropping additional modules into the loop.

[dunderwood]

A properly designed system should be plug-n-play being able to drop any number of computational "modules" into the loop as you want. As long as they didn't hardcode the 3 modules of 2 computing units architecture, it may already be capable of dropping additional modules into the loop.

This sounds like something a software engineer would say

Once you start adding hardware inputs/outputs into the equation, it becomes much harder to abstract your 'modules' in such a way. 

[Go4TLI]

Another point is that while throwing redundancy at the problem can indeed solve almost any reliability issue, the cost of such a decision may be an order of magnitude increase in complexity and software development costs.

And there you go.  So redundancy alone is not the fix-all you were suggesting.

Also redundancy does NOT solve reliability issues.  It is a mitigation to the fact that one has poor reliability, that is a major difference. 

And operationally speaking, if one takes a step back and looks at the big picture of why these systems and vehicles exist, there are phases of potential mission scenarios where it is not optimal to have to assume one has poor reliability and then rely solely on redundancies that may require crew/ground input at less then ideal times and/or circumstances

[Robotbeat]

Another point is that while throwing redundancy at the problem can indeed solve almost any reliability issue, the cost of such a decision may be an order of magnitude increase in complexity and software development costs.

And there you go.  So redundancy alone not is the fix-all you were suggesting.

Where did I say it will solve all issues ever? It does solve reliability issues, not necessarily in the most cost-effective way.

Also redundancy does NOT solve reliability issues.  It is a mitigation to the fact that one has poor reliability, that is a major difference. 
...

Yes, it does, and it isn't actually the major difference you make it out to be.

Since you're speaking in absolutes, let me use an example from my experience: It's actually better for reliability to have two consumer drives in a RAID 1 (especially if you are doing filesystem-level check-summing, another form of redundancy) than a single Enterprise drive by itself. There's more complication, sure, but reliability is absolutely dealt with using redundancy.

SSDs and hard-dives at the hardware/firmware level also use tons of redundancy (ESPECIALLY SSDs) to improve the reliability of each block. On top of that, you use RAID. On top of that, you use redundant mirrored systems and backups.

Defense-in-depth with redundancy (even if you're using crappy consumer drives) beats using a single drive with extreme tolerances every single time when it comes to reliability and data integrity. The only exception to this (besides software glitches or common design errors, which can affect you in either case but have less of an impact if you have backups) would be when rebuild time approaches the same order of magnitude as MTBF (for that unit). But there are ways around that, too.

You don't even realize the extent to which redundancy in data increases reliability in even our communication right here because it is essentially completely transparent. Checksumming occurs all over the place. You think all the cables that make up the Internet have such high S/N ratios that they are designed to never produce a flipped bit? Absolutely not. There is check-summing all over the place. Even in your computer's PCI-E bus, there are cyclic redundancy checks occurring with each transaction.

[Chris Bergin]

Hopefully I've not messed this up, but it seems we can have a standalone for the software rad issues. So a split thread.

[mmeijeri]

I'd love to hear more details about why SpaceX went with non-hardened components. So far I've heard C++ and Linux as an explanation, which I don't find very convincing since both work just fine on several rad-hard processors. There must be more to it.

[A_M_Swallow]

I'd love to hear more details about why SpaceX went with non-hardened components. So far I've heard C++ and Linux as an explanation, which I don't find very convincing since both work just fine on several rad-hard processors. There must be more to it.

I can understand why SpaceX did it in their control room, the computers have key boards and displays.  However most Dragon computers are embedded.  A rocket engine looks nothing like a display.

[guckyfan]

I'd love to hear more details about why SpaceX went with non-hardened components. So far I've heard C++ and Linux as an explanation, which I don't find very convincing since both work just fine on several rad-hard processors. There must be more to it.

An uneducated guess: on a slow system the realtime requirements may not be met with Linux and C++.

[MikeAtkinson]

I'd love to hear more details about why SpaceX went with non-hardened components. So far I've heard C++ and Linux as an explanation, which I don't find very convincing since both work just fine on several rad-hard processors. There must be more to it.

A major reason must be cost. A quick search came up with $23,000 for a processor, so say $50,000 for a processor board. Six of these in a processing unit = $300,000. 18 processing units per dragon = $5.4M

They also probably take more power, so larger solar panels and radiators.

Then there is limited selection of chips available, which may lead to compromises in design.

If they are slower than non-rad hard parts, the software may require more optimisation, which can be very costly both in development and maintenance.

[mlindner]

I'd love to hear more details about why SpaceX went with non-hardened components. So far I've heard C++ and Linux as an explanation, which I don't find very convincing since both work just fine on several rad-hard processors. There must be more to it.

So I've been trying to make this point several times now, but apparently people are thinking about this differently than I. I don't know if its a generational issue (I'm 23) or being a student in computer engineering or what. I think SpaceX is just trying to follow Amdahl's Law in that you shouldn't optimize a small part of the problem.

Why would you want to spend large amounts of money (relatively) in buying rad-hardened parts when you can just use multiple processors in parallel checking each other. Current top of the line rad-hardened parts get you (using stated prices for fastest rad hardened parts) 3 orders of magnitude less in speed for an increase of cost of also 3 orders of magnitude for a net 6 orders of magnitude increase.

[Robotbeat]

Plus, don't forget that radiation is just one cause of failure. A redundant design can guard against several.

They calculated (apparently correctly) that the error rate would be low enough that a triply redundant computer system with ability to reboot and resync the computers should allow for high enough reliability. In light of the successful mission and lower-than-expected error rate (if we believe SpaceX), that view was justified, with the caveat that they didn't automate resyncing and NASA told them not to resync manually, which can be classed as an oversight that should be (and appears to be) corrected.

[mlindner]

And operationally speaking, if one takes a step back and looks at the big picture of why these systems and vehicles exist, there are phases of potential mission scenarios where it is not optimal to have to assume one has poor reliability and then rely solely on redundancies that may require crew/ground input at less then ideal times and/or circumstances

Which is exactly why SpaceX is making re-syncing automatic in future software. A possible reason they didn't make it auto-resync initially is to not make the system overly complex right away and get some flight heritage on the existing system before they added that feature.

[john smith 19]

I'd love to hear more details about why SpaceX went with non-hardened components. So far I've heard C++ and Linux as an explanation, which I don't find very convincing since both work just fine on several rad-hard processors. There must be more to it.

AFAIK the going price for the BA 750 board (POWER PC architecture) is in the $400-800k range. I'd expect that's "price competitve" in this market with similar products running lesser know instruction sets like the USAF 1750A and the USN ANsomething-or-other. IIRC this is about the capability of a mid 90s Apple Mac. Aside from the *eyewatering* price I think you'll find these boards are *mostly* instruction set compatible with other POWER PCs, but not *exactly*, much as the European equivalent (Mongoose?) is based on the SPARC 7 architecture.

So on the upside the hardware is mfg in a rad hard process (SOS/SOI substrates are only the *start*). from the transistor up, *all* registers are likely to have 3 way voting, as is all I/O and the watchdog timer so you get defense in depth (*provided* your software make appropriate use of the features).

*but* you've got not-quite compatibility with less popular instruction sets (possibly with *substantial* limitations, like a 1MB address space on 1750A, still used by ULA IIRC or the Shuttle's 4Pi architecture) probably favoring military standard 1553b bus protocols (with mil spec pricing) and a clock frequency at most in the low 100s of MHz with *no* control over the form factor and any additional peripherals will be available at the same "competitive" pricing.

I note in all this talk I've not seen any comment on what Spacex actually *uses*. My instinct is x86 compatibles or ARM's (which have enjoyed *much* better power consumption.

[schaban]

Could ITAr or other limitations be one of the reason not to choose rad-hardened hardware?

Especially if Musk mentioned that ultimatly, he could try to sell Dragons to 3rd party, possibly outside of US...

[guckyfan]

Could ITAr or other limitations be one of the reason not to choose rad-hardened hardware?

Especially if Musk mentioned that ultimatly, he could try to sell Dragons to 3rd party, possibly outside of US...

You could also consider the possibility that SpaceX was not lying when they stated their reasons for not using rad-hardened in that article.

[Robotbeat]

I'd love to hear more details about why SpaceX went with non-hardened components. So far I've heard C++ and Linux as an explanation, which I don't find very convincing since both work just fine on several rad-hard processors. There must be more to it.

AFAIK the going price for the BA 750 board (POWER PC architecture) is in the $400-800k range. I'd expect that's "price competitve" in this market with similar products running lesser know instruction sets like the USAF 1750A and the USN ANsomething-or-other. IIRC this is about the capability of a mid 90s Apple Mac. Aside from the *eyewatering* price I think you'll find these boards are *mostly* instruction set compatible with other POWER PCs, but not *exactly*, much as the European equivalent (Mongoose?) is based on the SPARC 7 architecture.

So on the upside the hardware is mfg in a rad hard process (SOS/SOI substrates are only the *start*). from the transistor up, *all* registers are likely to have 3 way voting, as is all I/O and the watchdog timer so you get defense in depth (*provided* your software make appropriate use of the features).

*but* you've got not-quite compatibility with less popular instruction sets (possibly with *substantial* limitations, like a 1MB address space on 1750A, still used by ULA IIRC or the Shuttle's 4Pi architecture) probably favoring military standard 1553b bus protocols (with mil spec pricing) and a clock frequency at most in the low 100s of MHz with *no* control over the form factor and any additional peripherals will be available at the same "competitive" pricing.

I note in all this talk I've not seen any comment on what Spacex actually *uses*. My instinct is x86 compatibles or ARM's (which have enjoyed *much* better power consumption.

Quite informative...

So, four or five of those puppies would get to be in the millions of dollars, not counting peripherals. That becomes a significant portion of the spacecraft's cost... SpaceX is a company that likes to spend as little as possible on outside components. And presumably, they would want similiarity to their rocket's avionics as well. That would mean millions for each Falcon 9 or even Falcon 1 (back when they were still pursuing it) or the extra overhead of having two very different platforms.

[cleonard]

From that Avation Week link it's obvious that SpaceX spent a good deal of time engineering a computing solution.  They did a lot of analysis and even a good amount of testing.  The result is the current set of computing resources used in the SpaceX vehicles.  So far it's worked out. 

Please remember that Radiation Hardened means a lot of different things.  There are the transient effects of particles hitting computer components and there is the total dose over time.  Even hardened components suffer from SEU and you have to deal with that no matter what type of parts you use.

The total dose that a Dragon computer might see in a LEO mission is low.  just guessing I'd say 1 rad or so.  The Curiosity rover has a RAD750 computer that is specified for 100k rads.  To get that 100k rad you get to pay a reported $400k or so for it.

Now SpaceX had said that the Dragon could land on any solid surface in the solar system.  Good luck landing on Io with the current computer setup.  At the surface of Io you get about 2 rads per minute.   The current computer system would not survive the radiation environment for long.  What about Mars?  I'd say that is a maybe or maybe not. 

[Robotbeat]

From that Avation Week link it's obvious that SpaceX spent a good deal of time engineering a computing solution.  They did a lot of analysis and even a good amount of testing.  The result is the current set of computing resources used in the SpaceX vehicles.  So far it's worked out. 

Please remember that Radiation Hardened means a lot of different things.  There are the transient effects of particles hitting computer components and there is the total dose over time.  Even hardened components suffer from SEU and you have to deal with that no matter what type of parts you use.

The total dose that a Dragon computer might see in a LEO mission is low.  just guessing I'd say 1 rad or so.  The Curiosity rover has a RAD750 computer that is specified for 100k rads.  To get that 100k rad you get to pay a reported $400k or so for it.

Now SpaceX had said that the Dragon could land on any solid surface in the solar system.  Good luck landing on Io with the current computer setup.  At the surface of Io you get about 2 rads per minute.   The current computer system would not survive the radiation environment for long.  What about Mars?  I'd say that is a maybe or maybe not. 

Good one about Io... Not much could survive there! It'd be a challenge for even a very good rad-hard computer. You'd need additional shielding.

[john smith 19]

So, four or five of those puppies would get to be in the millions of dollars, not counting peripherals.

I guess that's more or less the going rate for this kind of hardware.

That becomes a significant portion of the spacecraft's cost... SpaceX is a company that likes to spend as little as possible on outside components.

It does mount up. The AvWeek article said they have about 54 processors on the whole LV/capsule doing various things. Commonality seems to be  a *very* big Spacex trait. Why support 2 (or 3?) architectures when you can standardize on 1?

And presumably, they would want similiarity to their rocket's avionics as well. That would mean millions for each Falcon 9 or even Falcon 1 (back when they were still pursuing it) or the extra overhead of having two very different platforms.

Exactly.
Note a classic issue with redundancy management is what happens if the SEU happens inside the *voting* logic. This *could* be done with off the shelf rad hard logic, acting as a "gatekeeper" on the processors I/O.

[IRobot]

This is from some of the reports on what went wrong:

•One of three flight computers failed while Dragon was docked at ISS due to a suspected radiation hit. The computer was restarted but could not re-synchronize with the other two units. The computer was restarted but was not resynchronized with the other two units. SpaceX says that NASA felt it was not necessary to continue the mission.
•One of three GPS units, the Propulsion and Trunk computers and Ethernet switch also experienced suspected radiation hits, but they were recovered during a power cycle.

This is for a about a 2 week long flight where the majority of the time it was not doing anything and just attached to ISS. 

Unsure if someone already mentioned it, but these could all be caused by EMI or static electricity. That would make sense as they were attached to ISS.

AFAIK, they had some EMI issues some months ago...

Radiation is not the only thing that causes this symptoms.

[jimvela]

Replying to two items from this thread:

This is very off topic, but as someone who designs PCB, no one except the very largest manufacturers etch their own PCBs; and unless you are manufacturing at least several hundred assemblies a month it's generally not worth it to do your own component placement either.

I watch flight flight boards get populated/placed and assembled in unit quantities down to qty=1 in the lab next to mine- on a regular basis.

Space rated PWA and PWB assembly is a different game than nearly anything commercial and absolutely everything high-volume.

Aside from the *eyewatering* price I think you'll find these boards are *mostly* instruction set compatible with other POWER PCs, but not *exactly*

Mostly is... mostly correct. 

I have testbench hardware that sometimes substitutes COTS PPC hardware in place of a rad hard board. 

There's another option in that you can buy non-flight boards from the same vendors as the rad hard boards that save quite a bit of money for applications like a testbed because most of the flight assembly processing and some flight packaging/finishing is omitted.

, much as the European equivalent (Mongoose?) is based on the SPARC 7 architecture.

The ESA Sparc is a LEON- it's commercially available and very widely used. 

probably favoring military standard 1553b bus protocols (with mil spec pricing)

1553 is indeed common, but newer busses like spacewire are becoming common place alongside it.  1553 isn't terribly expensive- at least not rad hard flight processor board expensive.

I note in all this talk I've not seen any comment on what Spacex actually *uses*. My instinct is x86 compatibles or ARM's (which have enjoyed *much* better power consumption.

I have no knowledge about SpaceX avionics architecture, but I'd be shocked beyond words if it were ARM based today.

[mlindner]

That becomes a significant portion of the spacecraft's cost... SpaceX is a company that likes to spend as little as possible on outside components.

It does mount up. The AvWeek article said they have about 54 processors on the whole LV/capsule doing various things. Commonality seems to be  a *very* big Spacex trait. Why support 2 (or 3?) architectures when you can standardize on 1?

From the AvWeek article:

We've got 54 in a Dragon – and they're all different kinds of computers, different kinds of processors.

They don't standardize on one.

This here is one of the reasons they use linux and C++. If the processor has minimal elements like MMU (memory management unit) and timer interrupts and it supports POSIX then you can run linux on it.

[mlindner]

I note in all this talk I've not seen any comment on what Spacex actually *uses*. My instinct is x86 compatibles or ARM's (which have enjoyed *much* better power consumption.

I have no knowledge about SpaceX avionics architecture, but I'd be shocked beyond words if it were ARM based today.

Why do you say that? ARM is the leader in full-function (32-bit) low power applications. (You can go lower power but it generally means going below 32-bit processors.) The main reason new applications use x86 is for binary/assembly compatibility with previous x86 code. In today's era of cross-compiliation though this is a non-issue.

[pippin]

I have no knowledge about SpaceX avionics architecture, but I'd be shocked beyond words if it were ARM based today.

Actually judging by the comments about the architecture I'd bet money on it being ARM.
Especially the point about having so many different computers points into that direction.
ARM is the de-facto standard for cores in embedded controllers these days and if you want to be scalable and keep your software modules it helps a lot to have a processor architecture that spans the range.

x86 is more complex and less predictable as an architecture, if I want something reliable I would not use it. Too many implementation bugs in there simply due to complexity, ARM is much better to control.
And then it's not available in small, low power controllers so if you want to use x86 in some computers, you'd have to support more than one architecture.

It's not true that using Linux, and C++ solves all your portability issues. It somewhat works with x86 and ARM, mainly because ARM supports both endianness modes. Even here you have to be careful with network and driver level code. The two other platforms that are somewhat supported are PPC and MIPS. PPC is getting out of flavor and you find fewer and fewer designs with it and MIPS is not available for high processing power requirements anymore.

It'll be at least mostly ARM. Maybe a few GPCs with x86 if they needs lots of floating point power and don't care about power but the majority of the smaller ones will be ARM.

[garidan]

As a newbie I found intersting this paper about mars human exploration and radiation issues http://www.marsjournal.org/contents/2006/0004/files/rapp_mars_2006_0004.pdf

You can use rad hardened hardware but if you want to carry humans you have to work on shields.
In the long run it could be "smarter" for SpaceX to focus on shields and gain experience on it, investing bucks on it and saving on not rad hardened electronics.

I "believe" in redundancy , but a strong event (rad storm ?) could destroy all your redundant parts and leave you naked.

That paper says Hydrogen rich materials shield better: could it be feasible to put CPU boards and ram in a sphere inside the future methane tanks for free shielding ?
And what about redundant CPU boards put in orthogonal direction to minimize the damage in case of directional rays (sun bursts)?
Last thought, modern CPUs and RAM are way smaller, so they lessen the chance of a hit, but I presume the smaller transistors are damaged by smaller energy levels than bigger ones, is it true ?

[mlindner]

You can use rad hardened hardware but if you want to carry humans you have to work on shields.
In the long run it could be "smarter" for SpaceX to focus on shields and gain experience on it, investing bucks on it and saving on not rad hardened electronics.

Radiation shielding is already a well understood topic, AFAIK the only interesting research going on in the field is electromagnetic shielding using some sort of plasma bubble (google M2P2), but this is still very theoretical and very far from implementation. The most likely solution will just be to point a large-ish tank of water directly at the sun. Elon Musk has already stated this as a good way of doing this as well. The other option is to just eat the radiation, "acceptable risk."

I "believe" in redundancy , but a strong event (rad storm ?) could destroy all your redundant parts and leave you naked.

I haven't heard of this being the case.

That paper says Hydrogen rich materials shield better: could it be feasible to put CPU boards and ram in a sphere inside the future methane tanks for free shielding?

This seems unlikely. The very cold temperatures inside the fuel are probably out of the operating temperature range of many semiconductors.

And what about redundant CPU boards put in orthogonal direction to minimize the damage in case of directional rays (sun bursts)?

A good idea, assuming you can maintain the orientation. Definitely possible, if not done already. This doesn't necessarily help the damage amounts though. If you get hit by a heavy ion it could go through many cells at once when flying through edge on.

Last thought, modern CPUs and RAM are way smaller, so they lessen the chance of a hit, but I presume the smaller transistors are damaged by smaller energy levels than bigger ones, is it true ?

There is a difference between a bit flip and permanent damage. Indeed if the transistor is smaller than it takes less energy to discharge/charge a RAM cell. The chip itself is not any larger or smaller though, there are just more transistors.

[JBF]

Replying to two items from this thread:

I watch flight flight boards get populated/placed and assembled in unit quantities down to qty=1 in the lab next to mine- on a regular basis.

Space rated PWA and PWB assembly is a different game than nearly anything commercial and absolutely everything high-volume.

Which is exactly why they cost so much.

[MP99]

A properly designed system should be plug-n-play being able to drop any number of computational "modules" into the loop as you want. As long as they didn't hardcode the 3 modules of 2 computing units architecture, it may already be capable of dropping additional modules into the loop.

This sounds like something a software engineer would say

Once you start adding hardware inputs/outputs into the equation, it becomes much harder to abstract your 'modules' in such a way. 

I found this to be a fantastic read on the subject of various architectures for tying multiple flight computers together. Most informative, and ISTM very relevant to this thread.

cheers, Martin

[Nomadd]

 I've fabricated single boards/systems and fabricated batches of 50. If SpaceX bought dozens of these RAD hardened units I'd expect the unit price to be much lower. Can't say I'm sure of the process here, but making ten of anything at once often costs little more than making a single item.

[MP99]

With added redundancy, you have the additional benefit of more "near misses," which gives you more opportunities to improve the system. Without redundancy, you either succeed or you fail hard, much fewer near misses.

As far as I can tell not a single person has suggested there should be no redundancy.  Redundancy is obvious thing to have.

The discussion is wheter or not unlimted redundancy should be the anwer-all for unreliability. 

I suggest there is a middle ground where reliability is high but redundancy is there because things happen. 

Nobody disagrees with you on the idea there should be a middle ground. Or at least, nobody should.

I agree with this as well. The problem is there is a very large range of possible middle grounds. The inflection point could be much further out than conventionally thought. I trust SpaceX to do this calculation. My personal belief though is that there has to be a better solution rather than using 15 year old technology.

Following Moore's law there have been 10 doubling's in transistor density since then. Which implies a roughly 1024x fold increase in computation power since then. Meaning assuming you do distributed computing (even more radiation prone) and assuming that distributed computing scales linearly (it doesn't) you need roughly 1000 of these processors to get to the speed of one modern processor.

If your hardware elements (transistors, memory cells, etc) are 1000 times smaller, wouldn't that make them much more susceptible to individual rad hits, perhaps of lower energy?

Also, I wonder if a hit that would have affected one component before might now affect multiple?

cheers, Martin

[IRobot]

If your hardware elements (transistors, memory cells, etc) are 1000 times smaller, wouldn't that make them much more susceptible to individual rad hits, perhaps of lower energy?

Also, I wonder if a hit that would have affected one component before might now affect multiple?

It also makes them more susceptible to burn on induced currents. Of course if tracks between transistors are smaller, induced currents are smaller.

[john smith 19]

Can't say I'm sure of the process here, but making ten of anything at once often costs little more than making a single item.

Quite probably
But that is the *cost* to BAe to make.

This is the *space* business.

You want some kind of buy-in-bulk *price* like a real business?
(I'm only *partly* joking about this).

[Robotbeat]

With added redundancy, you have the additional benefit of more "near misses," which gives you more opportunities to improve the system. Without redundancy, you either succeed or you fail hard, much fewer near misses.

As far as I can tell not a single person has suggested there should be no redundancy.  Redundancy is obvious thing to have.

The discussion is wheter or not unlimted redundancy should be the anwer-all for unreliability. 

I suggest there is a middle ground where reliability is high but redundancy is there because things happen. 

Nobody disagrees with you on the idea there should be a middle ground. Or at least, nobody should.

I agree with this as well. The problem is there is a very large range of possible middle grounds. The inflection point could be much further out than conventionally thought. I trust SpaceX to do this calculation. My personal belief though is that there has to be a better solution rather than using 15 year old technology.

Following Moore's law there have been 10 doubling's in transistor density since then. Which implies a roughly 1024x fold increase in computation power since then. Meaning assuming you do distributed computing (even more radiation prone) and assuming that distributed computing scales linearly (it doesn't) you need roughly 1000 of these processors to get to the speed of one modern processor.

If your hardware elements (transistors, memory cells, etc) are 1000 times smaller, wouldn't that make them much more susceptible to individual rad hits, perhaps of lower energy?

Also, I wonder if a hit that would have affected one component before might now affect multiple?

cheers, Martin

I've seen arguments go the other way WRT radiation bit-flipping. Modern techniques in manufacturing them make them less susceptible. They are also smaller, thus less likely to be hit... And you can afford more features like ECC that provide more resiliency.

[jimvela]

Replying to two items from this thread:

I watch flight flight boards get populated/placed and assembled in unit quantities down to qty=1 in the lab next to mine- on a regular basis.

Space rated PWA and PWB assembly is a different game than nearly anything commercial and absolutely everything high-volume.

Which is exactly why they cost so much.

That cost is in the noise compared to the cost of a failure- which is why they are built that way.

[john smith 19]

That cost is in the noise compared to the cost of a failure- which is why they are built that way.

So it's the QA in design & build coupled with testing *after* mfg and population that soaks up the cash?

I'd guessed it might have something to do with needing some kind of forced flow (either gas or liquid) cooling due to zero g.

This also raises a point. Are layer counts and line widths for space rated PWA's and PWC's (Those sound like IBM terms, I thought most people call them PCB's) behind those of terrestrial boards in the same way as space rated parts tend to be a generation or 2 behind their ground based equivalents?

[jimvela]

That cost is in the noise compared to the cost of a failure- which is why they are built that way.

So it's the QA in design & build coupled with testing *after* mfg and population that soaks up the cash?

Yes.  Plus, the cost of resolving a test anomaly can exceed the cost of building a brand new board- particularly with an unverified failure.

I'd guessed it might have something to do with needing some kind of forced flow (either gas or liquid) cooling due to zero g.

It is my experience that flight boards are cooled by conduction then radiating the heat. 

This also raises a point. Are layer counts and line widths for space rated PWA's and PWC's (Those sound like IBM terms, I thought most people call them PCB's) behind those of terrestrial boards in the same way as space rated parts tend to be a generation or 2 behind their ground based equivalents?

In my experience, yes.  Especially when you have whole planes dedicated to thermal management.

[mmeijeri]

An uneducated guess: on a slow system the realtime requirements may not be met with Linux and C++.

I doubt it. You can do cycle-perfect simulations of Apollo hardware in Javascript in a browser nowadays, so that can't be it. Console video games run on limited hardware too, and C++ is the language of choice for that.

[mmeijeri]

I think SpaceX is just trying to follow Amdahl's Law in that you shouldn't optimize a small part of the problem.

That could well be, but that's not what they appeared to be saying.

[guckyfan]

An uneducated guess: on a slow system the realtime requirements may not be met with Linux and C++.

I doubt it. You can do cycle-perfect simulations of Apollo hardware in Javascript in a browser nowadays, so that can't be it. Console video games run on limited hardware too, and C++ is the language of choice for that.

That is quite a few orders of magnitude slower. Some here on the forum were even surprised they use Linux at all because it is not hard realtime.

[A_M_Swallow]

An uneducated guess: on a slow system the realtime requirements may not be met with Linux and C++.

I doubt it. You can do cycle-perfect simulations of Apollo hardware in Javascript in a browser nowadays, so that can't be it. Console video games run on limited hardware too, and C++ is the language of choice for that.

That is quite a few orders of magnitude slower. Some here on the forum were even surprised they use Linux at all because it is not hard realtime.

Linux is a re-implementation of Unix.  Soft real time Unix made its living controlling telephone exchanges.  For SpaceX it probably comes down to how fast a rocket engine can gimbal.

[Robotbeat]

An uneducated guess: on a slow system the realtime requirements may not be met with Linux and C++.

I doubt it. You can do cycle-perfect simulations of Apollo hardware in Javascript in a browser nowadays, so that can't be it. Console video games run on limited hardware too, and C++ is the language of choice for that.

That is quite a few orders of magnitude slower. Some here on the forum were even surprised they use Linux at all because it is not hard realtime.

Linux is a re-implementation of Unix.  Soft real time Unix made its living controlling telephone exchanges.  For SpaceX it probably comes down to how fast a rocket engine can gimbal.

I'm pretty sure SpaceX isn't using Linux in that portion of their avionics... probably some other embedded, fully real-time operating system.

[john smith 19]

I'm pretty sure SpaceX isn't using Linux in that portion of their avionics... probably some other embedded, fully real-time operating system.

You may be right but there's a fairly active embedded Linux community and they've been tightening up Linux's RT performance for about the last *decade*. At least one groups aim is to consistently get their improvements into the *core* approved version rather than as build options. There are also strategies you can employ. IIRC Armadillo's stuff all runs on Linux and they use a *single* monolithic app on the RT control boards. 1 job, 1 board. Poor *commercial* development practice but works just fine in *this* environment, whre you have *total* control of what code runs on what hardware. They don't really do "task switching" as such but leverage the Linux toolset.

One small details about the AvWeek interview was he said "Different processors," not different *architectures*. My instinct is they have *one* architecture they support. If they need more MIPS they swallow the power budget and go to a higher clock frequency or a variant with on chip peripherals better tuned to the needs of that function.

As to *what* that architecture is I have no idea. Someone said no way is it ARM but did not explain *why* and I think the front runners are x86 and ARM architectures. They are widespread, and *most* support tools target them as core options and hardware MMU's makes memory protection fairly easy.

[mlindner]

An uneducated guess: on a slow system the realtime requirements may not be met with Linux and C++.

I doubt it. You can do cycle-perfect simulations of Apollo hardware in Javascript in a browser nowadays, so that can't be it. Console video games run on limited hardware too, and C++ is the language of choice for that.

That is quite a few orders of magnitude slower. Some here on the forum were even surprised they use Linux at all because it is not hard realtime.

This is a mis-representation of "real-time." If your system runs fast enough then even if it is not "real-time," it acts as if it is. As long as you can service events fast enough.

[mlindner]

An uneducated guess: on a slow system the realtime requirements may not be met with Linux and C++.

I doubt it. You can do cycle-perfect simulations of Apollo hardware in Javascript in a browser nowadays, so that can't be it. Console video games run on limited hardware too, and C++ is the language of choice for that.

That is quite a few orders of magnitude slower. Some here on the forum were even surprised they use Linux at all because it is not hard realtime.

Linux is a re-implementation of Unix.  Soft real time Unix made its living controlling telephone exchanges.  For SpaceX it probably comes down to how fast a rocket engine can gimbal.

I'm pretty sure SpaceX isn't using Linux in that portion of their avionics... probably some other embedded, fully real-time operating system.

I doubt that. Having to support an entirely different operating system splits your workforce. From the little I talked to them they have mainly two groups: flight software, and avionics. The avionics groups seem to use mainly hardware controls that interact with software and the flight software is all in these redundant linux modules.

[mlindner]

So digging in the careers section of spacex.com should have been done a while ago:
Summarized: I deleted common traits and traits that were generic.

Software Engineer (Embedded uC):
Broad knowledge of microprocessor technology including (but not limited to); PowerPC, ARM, OMAP, 8051, etc.
Extensive experience programming in C

Software Engineer (Flight Software):
Extensive knowledge of Unix/Linux and Linux Internals
Several years of software development experience
Extensive experience programming in C++
Experience programming for high reliability systems

Software Engineer (Embedded Linux):
Extensive knowledge of Linux system programming
Extensive knowledge of Linux kernel internals including device drivers and board support
Experience developing embedded systems, particularly 'Board Bring Up'
Extensive experience programming in C


A few comments:
They use a lot of linux. They don't use x86. Looks like PowerPC and ARM mainly.

[john smith 19]

So digging in the careers section of spacex.com should have been done a while ago:
Summarized: I deleted common traits and traits that were generic.
A few comments:
They use a lot of linux. They don't use x86. Looks like PowerPC and ARM mainly.

Wonder what the 8051 is for? uC rolls up a bunch of Unix commands and a shell into 1 single block to speed up loading. It also looks like the design boards in house. I'd guessed they'd gone with a *single* architecture but it suggests they went (mostly) with ARM & PPC instead.
I wonder what decides how the jobs are split up?

Nice work.  Thank you for the effort.

[mlindner]

uC rolls up a bunch of Unix commands and a shell into 1 single block to speed up loading.

uC stands for microcontroller (Atmel AVR, PIC, TI MSP430 are a few common ones), so I'm not sure what you meant by this.

As a side note, my group here flies MSP430s in space. We and other groups have quite often flown them in space on cubesats. They have quite high reliability, I haven't really heard of any permanently failing, occasionally they crash and have to be rebooted, but thats why we fly everything with watchdog timers (to reboot them). More so the newer ones are FRAM (Ferroelectric RAM) based which has inherent radiation hardening based on the technology because the data is stored in magnetic fields rather than electrons that could be disrupted by radiation. I should also note that they cost around $6 USD per chip.

[john smith 19]

uC rolls up a bunch of Unix commands and a shell into 1 single block to speed up loading.

uC stands for microcontroller (Atmel AVR, PIC, TI MSP430 are a few common ones), so I'm not sure what you meant by this.

Oops. Still had my Linux glasses on when I read that. It's one of the approaches for doing 1sec boot Linux implementations.

As a side note, my group here flies MSP430s in space. We and other groups have quite often flown them in space on cubesats. They have quite high reliability, I haven't really heard of any permanently failing, occasionally they crash and have to be rebooted, but thats why we fly everything with watchdog timers (to reboot them). More so the newer ones are FRAM (Ferroelectric RAM) based which has inherent radiation hardening based on the technology because the data is stored in magnetic fields rather than electrons that could be disrupted by radiation. I should also note that they cost around $6 USD per chip.

I've not come across MSP430 before. Is the instruction set completely new or is a uC version of a better known architecture?
BTW I think you'll find ferroelectric is not magnetic (the ferro- bit is a misnomer) its more like a static piezoelectric effect. But I think you're right it is *substantially* more rad hard. It doesn't sound that expensive either.

OT but how did they fair over the South Atlantic Anomaly?

[baldusi]

This is a mis-representation of "real-time." If your system runs fast enough then even if it is not "real-time," it acts as if it is. As long as you can service events fast enough.

No it isn't. Real-Time means bounded maximum latency. You are thinking about the average latency. Non RT OS can't guarantee that the maximum latency between CPU slices is below certain (critical, might get 100ms, but not nano second) amount. Things like DMA, memory or file handle access might block in non deterministic ways, you can get races on resources and such. It's very low level but it doesn't matter how fast your CPU is.
In fact, even the architecture might have limitations. I still remember how some very old soekris boxes had better RT behavior than the latest i7 motherboard. It might have to do with the way the system accesses the clock, or how the usb controller is wired to the system bus.
Many times, maximum latency and average latency are opposites and you have to trade one for the other. In fact, maximizing average latency might get you to O(C) while minimizing maximum latency might get to be a O(N) problem.

[mlindner]

OT but how did they fair over the South Atlantic Anomaly?

Not sure on that. I'm not directly involved with the mission that has logged the most time in space. We don't (yet) actually fly radiation monitors in space, so we can only tell when it resets. You can take a look at http://rax.engin.umich.edu/
It flies an msp430 as its flight computer, older flash based model, apparently works fine, doing great science. Nanosats generally don't fly with any redundancy anywhere because of mass and space requirements, if it breaks it breaks.

[john smith 19]

OT but how did they fair over the South Atlantic Anomaly?

Not sure on that. I'm not directly involved with the mission that has logged the most time in space. We don't (yet) actually fly radiation monitors in space, so we can only tell when it resets. You can take a look at http://rax.engin.umich.edu/
It flies an msp430 as its flight computer, older flash based model, apparently works fine, doing great science. Nanosats generally don't fly with any redundancy anywhere because of mass and space requirements, if it breaks it breaks.

I've seen pictures taken with digital cameras and closed shutters over the SAA Vs other parts of their orbit.

It's an impressive demonstration of the *relative* radiation level.

[Mader Levap]

I've seen pictures taken with digital cameras and closed shutters over the SAA Vs other parts of their orbit.
It's an impressive demonstration of the *relative* radiation level.

Links?

[john smith 19]

I've seen pictures taken with digital cameras and closed shutters over the SAA Vs other parts of their orbit.
It's an impressive demonstration of the *relative* radiation level.

Links?

I'd google south Atlantic anomaly but it was at a presentation give by Henry Spencer at Space Access.

[Prober]

... I've seen a lot of equipment and more than one life lost because of redundancy induced complacency. (A phrase I just invented for this post)...

Same argument applies to ANY reliability increase, does it not?

For instance, SpaceX is going to improve their error-handling capability for the next mission, making resyncing automatic. This will make them more robust to future problems.

this might be a fix or it might be the wrong direction.   Remember the "weakest link";  auto resyncing with an error prone processor makes the whole system weak. 

[mlindner]

... I've seen a lot of equipment and more than one life lost because of redundancy induced complacency. (A phrase I just invented for this post)...

Same argument applies to ANY reliability increase, does it not?

For instance, SpaceX is going to improve their error-handling capability for the next mission, making resyncing automatic. This will make them more robust to future problems.

this might be a fix or it might be the wrong direction.   Remember the "weakest link";  auto resyncing with an error prone processor makes the whole system weak. 

Not sure if you have to worry about that. If it is error prone then it will be kicked back out of sync again. Also keep in mind that every processor is itself in 2-way voting to see if its sure. So if it keeps erroring it won't even finish rebooting before it errors again.

Conceivably it could be an error that only occurs in a certain location in the code and that location happens immediately after resync so you are constantly going in and out of sync. Well as long as they can guarantee a maximum bound on the length of resync they can wait till they are not busy with anything system critical. (No point in doing a resync during stage sep for example.) Presumably there is a lockout switch where they can disable auto-resync when anything critical could occur. (Disable during launch, stage sep, docking, undocking, etc.) That would enable a somewhat more reliable setup.