SpaceX Falcon 9 - AMOS-6 - (Pad Failure) - DISCUSSION THREAD (2)

[spacekid]

So what about the RP and LOX?  Where do they come from?  Is the LOX manufactured on site?  How are vendors involved?  Have there been problems with these before?

LOX comes from a Praxair plant in Mims, FL that is used by all launch vehicles at the Cape.

impurities in the LOX that aren't an issue until supercooled? would not affect other vehicles... or same with RP but this issue seems to be LOX related.

hard to see how that would be a 'business process issue' unless they elected not to pay for the 'extra super pure' LOX.

edit: or of course the Helium. much easier to see how impurities from buying 'cheaper' could cause a problem there.

LOX is made by cryogenic distillation of air. Cool down air enough and various gasses start condensing out of it based on the vaporization temperature. I thought this process generates high-purity gas on its own, kind of how distillation works.

I think impurities would come from contaminants in the storage tanks and hoses used. Probably similar to rockets, cleaning material left behind would also be a source.

[Fred Bonyea]

Distillation as a single step does not create pure products - it takes multiple steps to separate ethanol from water, and it is almost impossible to distil methanol and isopropyl alcohol from ethyl.
It will be interesting to see if impure LOX is in some way responsible - which I frankly doubt - however, variations in purity will change boiling points and such.
 
We know the Helium bottle broke - an 'business' cause usually means a choice was made to reduce cost. So purity and any changes without substantial testing or verification fall in 'the business' fault tree.  Whatever the cost savings, it was substantially less than this billion dollar loss.

[Norm38]

Rule #5 for accident investigators:  ask ‘why’ seven times. It is much too easy to come to a first level conclusion and leave the investigation.  That is guaranteed to result in future accidents.[/i]I have no idea why the train crash occurred, but let’s take an imaginary trip through the kind of questions that an accident investigator should ask later in the investigation when a proximate cause is identified.  Here is that strictly hypothetical example:

Q1: Why did the train not stop? A1:  The brakes failed to apply when commanded by the operator. 

Q2: Why did the brakes fail?  A2:  part X in the braking system failed. 

Q3: Why did part X in the braking system fail?  A3 It was installed improperly at the last maintenance period.

Q4:  Why was part X installed improperly? A4:  The maintenance installation procedure was incorrect. 

Q5:  Why was the maintenance procedure incorrect? A5:  The procedure was not updated when a new part manufacturer was selected to build part X. 

Q6: Why was the procedure not updated?  A6:  The process for updating maintenance procedures did not allow for a change in part manufacturer. 

Q7:  Why did the process not allow for a new manufacturer:  A7:  It was not foreseen that a new part manufacturer would make a part that needed new installation procedures. 

Following this hypothetical case – and just note that I know nothing about the train crash, I am just making this up as a teaching tool – an accident investigator would find that the proximate cause of the accident was a braking failure, but the root cause was an inadequate process to account for new part manufacturers and the corrective action is to update the maintenance procedure change process to ensure that when a new part is introduced, the maintenance procedures are updated properly.

Is 7 Whys enough?  This example is just starting to get to the *true* root cause, the business practices.  To continue the hypothetical:

Q8: Why was it not foreseen?
A8: The company laid off the senior component engineer.

Q9: Why was he laid off?
A9: To save money.

Q10: Why did money need to be saved?
A10: The CEO wanted to boost the share price.

Ask a few more whys and you see why the question of corporate culture does come up.  Is it enough to just say "this failure mode was unforeseen, and now it's fixed"?  What else is unforeseen?

[Jim]

So what about the RP and LOX?  Where do they come from?  Is the LOX manufactured on site?  How are vendors involved?  Have there been problems with these before?

LOX comes from a Praxair plant in Mims, FL that is used by all launch vehicles at the Cape.

impurities in the LOX that aren't an issue until supercooled? would not affect other vehicles... or same with RP but this issue seems to be LOX related.

hard to see how that would be a 'business process issue' unless they elected not to pay for the 'extra super pure' LOX.

edit: or of course the Helium. much easier to see how impurities from buying 'cheaper' could cause a problem there.

LOX is made by cryogenic distillation of air. Cool down air enough and various gasses start condensing out of it based on the vaporization temperature. I thought this process generates high-purity gas on its own, kind of how distillation works.

I think impurities would come from contaminants in the storage tanks and hoses used. Probably similar to rockets, cleaning material left behind would also be a source.

Impurities in the LOX have nothing to do with this

[john smith 19]

On a different note, Spaceflight101 published a decent article summarizing the current state of the investigation (besides the rather click bait title that is...).

Good summary article.

I had no idea the fluid loading schedule was that aggressive. I'd thought at the least the COPV's would be submerged before they started high pressure loading leading to a temperature rise in the COPV.

Rapid improvement requires rapid iteration which requires failing fast. Failing fast is great, it's served SpaceX well up to this point, allowing fast growth and extensive market disruption.

But failing isn't an option once a customer payload is mounted. There has to be a hard stop to any unproven changes at some point prior to risking a payload. Test AS you fly is great, test WHILE you fly is not.

This is why I wonder if commercial crew wasn't a bad move this early in SpaceX's plan.

If they only launch satellites, then some amount of lesser reliability is worth it for lesser cost. If the total cost to the customer of losing the satellite is say $1 billion, and you're 6% likely to lose it on Falcon and 1% likely on Atlas... as long as the riskier option is at least $50 million, it's the rational choice.

Accepting say 1 failure in 15 in exchange for lower operating costs might be perfectly sensible, for a pure satellite launcher.

But when people's lives are involved, not so much.

EDIT: fixed quote

Which is why Commercial Crew has a Launch Escape System in its design.

We're still close to 2 years from a crewed launch on F9.

I'd suggest your concerns are premature. 

Following this hypothetical case – and just note that I know nothing about the train crash, I am just making this up as a teaching tool – an accident investigator would find that the proximate cause of the accident was a braking failure, but the root cause was an inadequate process to account for new part manufacturers and the corrective action is to update the maintenance procedure change process to ensure that when a new part is introduced, the maintenance procedures are updated properly.

Bold is my emphasis.

I think his point is that fixing the superficial cause will stop that exact same accident happening again but fixing the root cause will stop a whole class of similar accidents (those caused by a new supplier delivering a slightly different part not triggering a review, and update of the procedures for that part).

Incidentally this is the process IBM Federal Systems (as they were) went through with Shuttle software not just to fix a bug, but to fix the class of bugs it came from and to fix the system to stop that class of bugs coming through.

This does make the point that  ultimately most problems come down to a failure of process, including having no process to cope with an event.

I find it very hard to believe that SX don't have some kind of change management process in place for both pad operations and mfg simply because given the complexity of what's happening I don't think they could have launched this many rockets without blowing more of them up without one.

How rigorous it is, and how it needs to be tightened, is another matter. 

Playing Devils advocate for a moment I will note that every time you mate the payload for a test and then de- mate it for storage you multiply the risk of a mishap to the  payload or it's shroud. You save multiple blocks of time if you just leave it in place. As others have pointed out that's quite reasonable provided  the vehicle and it's GSE and procedures are basically frozen. This is simply not how SX operates with its process of continuous (but controlled) improvements to everything.

I suspect SX will be more cautious about involving payloads in any process changes in future although I doubt they will radically reduce the rate at which they make changes to either their designs or their processes. 

[Thorny]

We're still close to 2 years from a crewed launch on F9.

2? Wasn't it supposed to be around this time next year?

[rsdavis9]

I think impurities would come from contaminants in the storage tanks and hoses used. Probably similar to rockets, cleaning material left behind would also be a source.

For any liquids fractionating column is needed.
Otherwise you get a mixture defined by the ratio of the vapor pressures of the 2 liquids at the temperature you are boiling/condensing at.

[sdsds]

changes should be tested out in Texas

as to the testing in FL vs TX... the initial business case could have been that the GSE at both sites was identical.

Even if they tested a procedure in Texas or on a prior flight there would be question about whether they tested it enough. Oscillators can by finicky; a small difference in initial conditions can lead to a large difference in outcome.

Chaos happens. Operating on the bleeding edge is a business decision.

[akm]

changes should be tested out in Texas

as to the testing in FL vs TX... the initial business case could have been that the GSE at both sites was identical.

Even if they tested a procedure in Texas or on a prior flight there would be question about whether they tested it enough. Oscillators can by finicky; a small difference in initial conditions can lead to a large difference in outcome.

Chaos happens. Operating on the bleeding edge is a business decision.

As to testing in FL vs TX it wasn't suppose to be taken literally. It's more that risky testing, such as changes to procedures, should be carried out in such a way that failures are not so expensive.  If it can't be done in TX then good, if not at least take the expensive payload off the rocket.  Failures happen the part of pushing test to limit, but I thing more effort needs to be put into making failures less expensive.

[CraigLieb]

There can be differences in testing environment vs use environment like temperature ranges, humidity differences, changes in the vehicle or components en-route due to road vibrations, a cycle difference where the failure  isn't going to show up until the 5th, 6th or 10th cycle of the process, on that part, etc.

Then there may also be an  oversight including some employee bumping into a COPV and not reporting it. (business process failure in this case is: damage that wasn't detected on the COPV.
  - Possible causes: because it wasn't properly shielded during the manufacturing process. 
   Why not reported?  Fear of consequences by the employee or possibly they didn't even know they bumped it. )

One company a long time ago had a chemical process specification to do something at "room temperature". In the original location, that worked, but the process was moved from the South to the North. Room temperature up North (middle of winter in the unheated garage) was like 20 degrees F. So the process (specification) didn't work properly.  Parts had to be scrapped and at least it was caught in subsequent inspection and testing.

[Coastal Ron]

The problem there is that almost anything can be reduced to a business process failure. The CRS-7 can be reduced to that since their process did not include internal testing of the struts, for example.

There are design failures that happen, which though you could say you have a process in place to catch such problems (which would be a business process), the end result is that the design was flawed.

For the Challenger accident the Shuttle was flown in colder conditions than it was designed for, so not a design problem, but bad business process decision making.

For the CRS-7 there were business processes in place that, if followed, would have resulted in struts of the proper strength being delivered to SpaceX.  They know those processes were not followed consistently, and they theorized from that fact that an understrength strut caused the loss of vehicle.  So not a design flaw, but not a business process flaw per se at SpaceX, since if all the documented processes would have been followed at the supplier the flawed parts would not have been used.  But overall a business process flaw.

[Wolfram66]

What is inside the COPV's and He system Prior to loading of HighPressure Helium and the loading of the subcooled LOX? GN2??

[scubadown]

I have heard rumors that SpaceX saves a lot of money by not cleaning their LOx GSE as thoroughly as we do across the river.   

[virnin]

I have heard rumors that SpaceX saves a lot of money by not cleaning their LOx GSE as thoroughly as we do across the river.   

While the choice of GSE cleaning procedures falls under the "business process" heading, I believe that any post that starts with "I have heard rumors" should go in the other (wild) thread.

[Rocket Science]

We're still close to 2 years from a crewed launch on F9.

2? Wasn't it supposed to be around this time next year?

As far as I can recall crewed test flights are supposed to be in 2017.... anyone else?

[spacekid]

We're still close to 2 years from a crewed launch on F9.

2? Wasn't it supposed to be around this time next year?

As far as I can recall crewed test flights are supposed to be in 2017.... anyone else?

SpaceX was supposed to launch in 2017 but Boeing had slipped into 2018. Now Boeing has slipped their launch to ISS to Dec 2018 but SpaceX hasn't announced any delays. I would expect SpaceX to also slip to 2018 at least.

[Rocket Science]

We're still close to 2 years from a crewed launch on F9.

2? Wasn't it supposed to be around this time next year?

As far as I can recall crewed test flights are supposed to be in 2017.... anyone else?

SpaceX was supposed to launch in 2017 but Boeing had slipped into 2018. Now Boeing has slipped their launch to ISS to Dec 2018 but SpaceX hasn't announced any delays. I would expect SpaceX to also slip to 2018 at least.

I'll eat my peas if Boeing flys in December 2018...

[deruch]

As far as I can recall crewed test flights are supposed to be in 2017.... anyone else?

Yeah, prior to the AMOS-6 mishap they were essentially targeting December 31, 2017 so they would still make it under the wire (though, IIRC, the last date I remember actually being bandied about was the ~20th).  Which I read as: We're still officially targeting a 2017 launch but in reality everyone who's paying attention should expect us to launch in 2018 because there will inevitably be some slip as we get closer to crunch time. 

[Jet Black]

The problem there is that almost anything can be reduced to a business process failure. The CRS-7 can be reduced to that since their process did not include internal testing of the struts, for example.

There are design failures that happen, which though you could say you have a process in place to catch such problems (which would be a business process), the end result is that the design was flawed.

For the Challenger accident the Shuttle was flown in colder conditions than it was designed for, so not a design problem, but bad business process decision making.

For the CRS-7 there were business processes in place that, if followed, would have resulted in struts of the proper strength being delivered to SpaceX.  They know those processes were not followed consistently, and they theorized from that fact that an understrength strut caused the loss of vehicle.  So not a design flaw, but not a business process flaw per se at SpaceX, since if all the documented processes would have been followed at the supplier the flawed parts would not have been used.  But overall a business process flaw.

I see your point. I guess that they are pretty linked, since one could always argue that there is a business process that should catch a design/manufacturing/operating flaw and ultimately it is always one of the latter that blows up a rocket. I feel I can play ping-pong in my head between business process and everything else.

[alang]

There can be differences in testing environment vs use environment like temperature ranges, humidity differences, changes in the vehicle or components en-route due to road vibrations, a cycle difference where the failure  isn't going to show up until the 5th, 6th or 10th cycle of the process, on that part, etc.

Then there may also be an  oversight including some employee bumping into a COPV and not reporting it. (business process failure in this case is: damage that wasn't detected on the COPV.
  - Possible causes: because it wasn't properly shielded during the manufacturing process. 
   Why not reported?  Fear of consequences by the employee or possibly they didn't even know they bumped it. )

One company a long time ago had a chemical process specification to do something at "room temperature". In the original location, that worked, but the process was moved from the South to the North. Room temperature up North (middle of winter in the unheated garage) was like 20 degrees F. So the process (specification) didn't work properly.  Parts had to be scrapped and at least it was caught in subsequent inspection and testing.

Reminds me of a short story by Primo Levi who had been an industrial chemist before circumstances drove him to write about his life.: The story was based around a problem with a batch of paint. In the end the problem was found to be due to the interaction of a particular kind of pollen in the country it had been sold to. I don't know if the story is based on truth but there's a lot of interesting chemistry in paint and insulators and I wonder if mechanical and electronic and engineers are always fully prepared for that. (I'm an ignorant coder).