SpaceX Falcon 9 - AMOS-6 - (Pad Failure) - DISCUSSION THREAD (2)

[dglow]

Sorry if i'm being cranky today, please forgive me.

This forum has a very high bar when it comes to crankiness. You'll need to keep at it. 

[old_sellsword]

Ah, I missed it – thank you, Raul!

Neither Eutelsat nor the missions which followed had any scrubs, according to our launch log. So: six loadings with extra-extra-cold helium.

I don't think we can assume all six FT propellant loadings were identical. AMOS-6 may have been a first. SpaceX has said that tweaks to the loading process were ongoing.

Matthew

In fact, it has been mentioned that Amos-6 was only the second time they'd used this particular Helium loading procedure with a flight-ready Falcon 9.

It is also understood that SpaceX was testing modifications to the countdown sequence on the Static Fire Test for the previous Falcon 9 mission with JCSat-16 to introduce window management capabilities for the FT version of Falcon 9 that initially had to launch very shortly after propellant loading finished in order to avoid the chilled propellants warming up inside the tanks. These modified countdown steps include adjustments to engine chilldown as well as the propellant and pressurant loading sequence.

[PreferToLurk]

As has been discussed upthread, ... The problem is what happens if that LOX freezes.

As has been mentioned upthread, the SpaceX update says the LOX freezing (aka SOX) exacerbates the problem. in simple english that means it is not the root problem.

I know that SpaceX/Musk mentioned several times earlier in the investigation that the cause was SOX, but based on the final update it seems the conclusion was not so clear, yet it seems many on this thread continue to repeat it as if it were fact.

I mentioned this earlier, nobody came back to say otherwise, if anyone is still convinced SOX was the root cause please speak up and convince me, I'll be happy to hear. (SOX seems like a much easier problem to solve so everyone prefers to focus on that).

Sorry if i'm being cranky today, please forgive me.

<delurk>

This is how I understand the issue in fairly plain english:

Two (mostly) new things happened with the destroyed rocket, and one thing that's been happening since the beginning.  The old thing is lox intrusion into the COPV composite overwrap.  The new things are buckles in the COPV liner, and conditions conducive to the creation of SOX in the LOX soaked overwrap. 

The accident investigation team cannot rule out that LOX, pooled in the COPV buckles, became trapped, and ignited the overwrap without necessarily transitioning to SOX first.  This scenario is probably viewed by Musk as being so unlikely that it can be discounted as the actual cause of the accident.  This is due to how incredibly unlikely it is that the overwrap (after letting the LOX penetrate in the first place) then sealed itself up and trapped said LOX with enough pressure to ignite the overwrap.  This is also likely discounted by Musk due to the fact that these COPV's have been getting soaked with LOX and then pressurized for a very long time without any incident until now.

The new factors are the buckles and the possibility of SOX formation.  While it is technically possible for the buckles themselves to lead to a scenario where LOX becomes entrapped, SOX in the buckle magnifies the possibility of overwrap ignition many times over. 

SpaceX plan for return to flight (approved by the same board that maintains the possibility of a LOX entrapment ignition) is to first return the loading procedures to a place where SOX cannot form, and buckles of the liner are significantly more unlikely.  The long term solution is to re-engineer the COPV's to remove the risk of buckles entirely.

So there you go.  The root cause (physically speaking) is oxygen pooling in buckles and igniting the overwrap under pressure. The LIKELY root cause is SOX entrapment in buckles igniting the overwrap under pressure.  But I believe SpaceX is undertaking corrective actions to address both the likely and unlikely root causes. 

</delurk>

[mn]

As has been discussed upthread, ... The problem is what happens if that LOX freezes.

As has been mentioned upthread, the SpaceX update says the LOX freezing (aka SOX) exacerbates the problem. in simple english that means it is not the root problem.

I know that SpaceX/Musk mentioned several times earlier in the investigation that the cause was SOX, but based on the final update it seems the conclusion was not so clear, yet it seems many on this thread continue to repeat it as if it were fact.

I mentioned this earlier, nobody came back to say otherwise, if anyone is still convinced SOX was the root cause please speak up and convince me, I'll be happy to hear. (SOX seems like a much easier problem to solve so everyone prefers to focus on that).

Sorry if i'm being cranky today, please forgive me.

<delurk>

This is how I understand the issue in fairly plain english:

Two (mostly) new things happened with the destroyed rocket, and one thing that's been happening since the beginning.  The old thing is lox intrusion into the COPV composite overwrap.  The new things are buckles in the COPV liner, and conditions conducive to the creation of SOX in the LOX soaked overwrap. 

The accident investigation team cannot rule out that LOX, pooled in the COPV buckles, became trapped, and ignited the overwrap without necessarily transitioning to SOX first.  This scenario is probably viewed by Musk as being so unlikely that it can be discounted as the actual cause of the accident.  This is due to how incredibly unlikely it is that the overwrap (after letting the LOX penetrate in the first place) then sealed itself up and trapped said LOX with enough pressure to ignite the overwrap.  This is also likely discounted by Musk due to the fact that these COPV's have been getting soaked with LOX and then pressurized for a very long time without any incident until now.

The new factors are the buckles and the possibility of SOX formation.  While it is technically possible for the buckles themselves to lead to a scenario where LOX becomes entrapped, SOX in the buckle magnifies the possibility of overwrap ignition many times over. 

SpaceX plan for return to flight (approved by the same board that maintains the possibility of a LOX entrapment ignition) is to first return the loading procedures to a place where SOX cannot form, and buckles of the liner are significantly more unlikely.  The long term solution is to re-engineer the COPV's to remove the risk of buckles entirely.

So there you go.  The root cause (physically speaking) is oxygen pooling in buckles and igniting the overwrap under pressure. The LIKELY root cause is SOX entrapment in buckles igniting the overwrap under pressure.  But I believe SpaceX is undertaking corrective actions to address both the likely and unlikely root causes. 

</delurk>

Thank you, clear enough.

Now for my take on this: I suspect there were lots of arguments among the investigators as to the likelihood of failure without SOX.

It seems clear to me who won that argument.

The update says "the investigation team concluded the failure was likely due to the accumulation of oxygen between the COPV liner and overwrap in a void or a buckle in the liner, leading to ignition and the subsequent failure of the COPV". This says nothing about SOX.

Only in the NEXT paragraph does it mention that 'in addition..' SOX exacerbates the problem, my interpretation is that this sentence was a concession to those who were arguing that SOX is the cause, but it was not the consensus opinion.

It's obvious why we all wish SOX was the cause, but it seems to me that investigators found enough evidence of possible failure without SOX.

OK I hope that's enough kremlinology for one day.

[Reflectiv]

Hard to make a definitive conclusion, without knowing, how precisely the short update reflects the actual report.

[geza]

The situation is quite worrisome. Design for reliability was one of the main selling points for SpaceX. Now they have an issue that is difficult to understand and control. It reminds me the Shuttle after Columbia: it was impossible to get rid of foam shredding entirelly. NASA people did their bests to minimize the risk and decided to terminate the program asap. However, Falcon 9 is to stay and carry people soon. Fuelling while crew on board was considered a probem by many even before Sept 1.

[Fred Bonyea]

...

It seems clear to me who won that argument.

The update says "the investigation team concluded the failure was likely due to the accumulation of oxygen between the COPV liner and overwrap in a void or a buckle in the liner, leading to ignition and the subsequent failure of the COPV". This says nothing about SOX.

Only in the NEXT paragraph does it mention that 'in addition..' SOX exacerbates the problem, my interpretation is that this sentence was a concession to those who were arguing that SOX is the cause, but it was not the consensus opinion.

It's obvious why we all wish SOX was the cause, but it seems to me that investigators found enough evidence of possible failure without SOX.

OK I hope that's enough kremlinology for one day.

Very good. One more speculative decypher: My best guess is that they got an unexpected kablam during testing at a time when SOX formation was not feasible; giving them a failure rate (when all testing and launches are figured in) that is greater than they would like, but low enough to risk resuming launching.  The other possibilitie(s) are that they were not able to recreate conditions that confirm SOX formation; or induce a failure when they did so.

[dglow]

Very good. One more speculative decypher: My best guess is that they got an unexpected kablam during testing at a time when SOX formation was not feasible; giving them a failure rate (when all testing and launches are figured in) that is greater than they would like, but low enough to risk resuming launching.

That sounds about right. Perhaps the key question pertaining to RTF risk factor is this: do they see failure only at three-COPV pressure levels? What happens with four?

[bstrong]

Very good. One more speculative decypher: My best guess is that they got an unexpected kablam during testing at a time when SOX formation was not feasible; giving them a failure rate (when all testing and launches are figured in) that is greater than they would like, but low enough to risk resuming launching.  The other possibilitie(s) are that they were not able to recreate conditions that confirm SOX formation; or induce a failure when they did so.

If I squint at it just right, it reads like someone in another organization who had a say in the matter was worried that it might happen without SOX, even though all failures in testing involved SOX. We just don't know (and probably never will). I'm all for belt and suspenders at this point, though.

[dglow]

Very good. One more speculative decypher: My best guess is that they got an unexpected kablam during testing at a time when SOX formation was not feasible; giving them a failure rate (when all testing and launches are figured in) that is greater than they would like, but low enough to risk resuming launching.  The other possibilitie(s) are that they were not able to recreate conditions that confirm SOX formation; or induce a failure when they did so.

If I squint at it just right, it reads like someone in another organization who had a say in the matter was worried that it might happen without SOX, even though all failures in testing involved SOX. We just don't know (and probably never will). I'm all for belt and suspenders at this point, though.

No, SpaceX made it clear the failure does not require SOX. "The investigation team identified several credible causes for the COPV failure, all of which involve accumulation of super chilled LOX or SOX in buckles under the overwrap."

[bstrong]

Very good. One more speculative decypher: My best guess is that they got an unexpected kablam during testing at a time when SOX formation was not feasible; giving them a failure rate (when all testing and launches are figured in) that is greater than they would like, but low enough to risk resuming launching.  The other possibilitie(s) are that they were not able to recreate conditions that confirm SOX formation; or induce a failure when they did so.

If I squint at it just right, it reads like someone in another organization who had a say in the matter was worried that it might happen without SOX, even though all failures in testing involved SOX. We just don't know (and probably never will). I'm all for belt and suspenders at this point, though.

No, SpaceX made it clear the failure does not require SOX. "The investigation team identified several credible causes for the COPV failure, all of which involve accumulation of super chilled LOX or SOX in buckles under the overwrap."

That was the conclusion of the failure investigation. Whether that was proven by testing or is the result of taking a conservative approach is unclear. I think a conservative approach is warranted, so I have no bone to pick here. I'm just pointing out that we don't have enough information to make concrete statements about what may or may not have been directly reproduced in testing.

[dglow]

Okay. Squint as much as you need to.

[foltster]

Just a thought...have you ever thrown a bottle of almost frozen water and seen it crystallize after impact?  Is something akin to that possible? 

The idea being that LOX near the freezing point and in contact with the aluminum liner already is pushed past the freezing point due to a rapid movement pressure change due to movement between the liner and overwrap.  This could be cause by temperature change and/or pressure increase from incoming helium.  This could create extremely viscous or even possibly short lived SOX that immediately lead to failures.  Probably not, as I have poor understanding of how LOX and SOX behave in this environment. 

[Fred Bonyea]

Just a thought...have you ever thrown a bottle of almost frozen water and seen it crystallize after impact?  Is something akin to that possible? 

The idea being that LOX near the freezing point and in contact with the aluminum liner already is pushed past the freezing point due to a rapid movement pressure change due to movement between the liner and overwrap.  This could be cause by temperature change and/or pressure increase from incoming helium.  This could create extremely viscous or even possibly short lived SOX that immediately lead to failures.  Probably not, as I have poor understanding of how LOX and SOX behave in this environment.

Good question.
Freezing point suppression is a very big deal with water. Water expands when it freezes, so sudden freezing at suppressed temperatures creates an enormous expansion force. LOX freezing to solid LOX does not; which, in my opinion, makes this a curious mechanism for creating a breach in bottle integrity ; and it is, again in my opinion, much more probable that internal stresses within a fiber-wound bottle during rapid temperature/pressure shifts that lead to somewhat rare and unpredictable failures.

[jpo234]

Just a thought...have you ever thrown a bottle of almost frozen water and seen it crystallize after impact?  Is something akin to that possible? 

The idea being that LOX near the freezing point and in contact with the aluminum liner already is pushed past the freezing point due to a rapid movement pressure change due to movement between the liner and overwrap.  This could be cause by temperature change and/or pressure increase from incoming helium.  This could create extremely viscous or even possibly short lived SOX that immediately lead to failures.  Probably not, as I have poor understanding of how LOX and SOX behave in this environment.

Good question.
Freezing point suppression is a very big deal with water. Water expands when it freezes, so sudden freezing at suppressed temperatures creates an enormous expansion force. LOX freezing to solid LOX does not; which, in my opinion, makes this a curious mechanism for creating a breach in bottle integrity ; and it is, again in my opinion, much more probable that internal stresses within a fiber-wound bottle during rapid temperature/pressure shifts that lead to somewhat rare and unpredictable failures.

If the LOX/SOX gets trapped before the COPV is fully pressured, the "enormous expansion force" comes from the expanding liner, when the pressure rises.
That's why I think SpaceX's solution is to fully pressurize the COPVs before submerging them into the LOX. Any LOX that still finds a nook inside the overwrap (buckles in the liner will be gone thanks to the pressure) will not be subjected to the pressure exerted by the expanding liner (because it's already expanded).

[MarekCyzio]

Just a thought...have you ever thrown a bottle of almost frozen water and seen it crystallize after impact?  Is something akin to that possible? 

The idea being that LOX near the freezing point and in contact with the aluminum liner already is pushed past the freezing point due to a rapid movement pressure change due to movement between the liner and overwrap.  This could be cause by temperature change and/or pressure increase from incoming helium.  This could create extremely viscous or even possibly short lived SOX that immediately lead to failures.  Probably not, as I have poor understanding of how LOX and SOX behave in this environment.

Good question.
Freezing point suppression is a very big deal with water. Water expands when it freezes, so sudden freezing at suppressed temperatures creates an enormous expansion force. LOX freezing to solid LOX does not; which, in my opinion, makes this a curious mechanism for creating a breach in bottle integrity ; and it is, again in my opinion, much more probable that internal stresses within a fiber-wound bottle during rapid temperature/pressure shifts that lead to somewhat rare and unpredictable failures.

If the LOX/SOX gets trapped before the COPV is fully pressured, the "enormous expansion force" comes from the expanding liner, when the pressure rises.
That's why I think SpaceX's solution is to fully pressurize the COPVs before submerging them into the LOX. Any LOX that still finds a nook inside the overwrap (buckles in the liner will be gone thanks to the pressure) will not be subjected to the pressure exerted by the expanding liner (because it's already expanded).

It may be way more complex - you can pressurize COPV but as soon as you cool them down with LOX, pressure will drop and buckling may occur.

[vanoord]

If the LOX/SOX gets trapped before the COPV is fully pressured, the "enormous expansion force" comes from the expanding liner, when the pressure rises.
That's why I think SpaceX's solution is to fully pressurize the COPVs before submerging them into the LOX. Any LOX that still finds a nook inside the overwrap (buckles in the liner will be gone thanks to the pressure) will not be subjected to the pressure exerted by the expanding liner (because it's already expanded).

Or / and buckling is less likely as the COPVs are pressurised before the loading of the LOX, so the liner isn't exposed to high pressure outside it, which *may* have been one cause of the bucking.

[feynmanrules]

this has been a quite detailed and very informative thread.  thanks to all who participated on it.

if any of the mods or chris are watching, it would be interesting to read an official article summarizing this thread. 

not sure it's worth the effort but I thought I'd express my .02 w/my thanks.

[deruch]

In thinking about the LOX vs SOX discussion and the implications of the wording in the latest investigation update, my biggest question is whether an immediate and catastrophic failure is the only possible result?  Both for the COPV and for the vehicle as a whole?  Clearly, sometimes the bullets are dodged altogether as they survived testing and McGreggor operations prior to the AMOS-6 failure.  And replicating this failure mode took some effort, so it can't be as straightforward as A+B always results in C.  But, can this process also result in a small/medium/large leak instead of a total failure of the COPV?  We know they've had leaky COPVs in the past.  Were those a result of non-catastrophic damage via this process?  Some here have argued that the new information raises doubts in their minds about the results of the CRS-7 failure investigation (strut failure), though I don't think the timing of that failure (during flight) or its publicized description (Helium system pressure drop followed by return) fits very well.  I'm actually more interested in the Orbcomm OG2-1 Static Fire helium leak (NSF discussion starting here) which also took place during loading operations [Source].  That event sounds almost exactly the same as this one, except it occurred well before the switch to subcooled LOX and fast loading sequence.  So, I'm assuming it's much less likely for SOX to have been involved.  At the time, I just assumed that it was a failure due to bad hardware as opposed to a procedure issue but maybe they just got really lucky and had this same issue only without destroying a vehicle and payload.  Anyways, uncertainty about that could explain why they are unwilling to pin SOX formation as a necessary prerequisite for this type of failure.

[jpo234]

http://thespacereview.com/article/3170/1