SpaceX Falcon 9 - AMOS-6 - (Pad Failure) - DISCUSSION THREAD (2)

[CyndyC]

Since day one my instinct has been that the 1st stage was somehow implicated in this event, no matter how unapparent the possibility. In my search for a post with information on a different subject, I came across a post that might explain the connection I've been groping for.

Wouldn't there have to be a designed-in conductive path from the second stage through the interstage to the first stage to keep all structure at the same potential during ascent?

http://forum.nasaspaceflight.com/index.php?topic=30981.msg1582782#msg1582782

What could an electrical conductivity question possibly have to do with an overpressure/failure of a He-filled COPV?

The ignition source.

[CyndyC]

Before anyone says it, I understand a conventional ignition source would not be necessary if/when a COPV explodes, but something else has been on my mind.

Among the COPV expert's statements in the SpaceNews article, he said, "The kerosene needs to be pulverized, such as in a spray, to ignite."

The thing was, I read that not long after a kerosene lamp helped get me through Hurricane Matthew in Jacksonville. I was not hunkering down and pulverizing kerosene inside my house, yet I had plenty of light from the lamp.

So now I'm wondering if the LOX feed line that runs both ways through the center of the RP-1 tank could have behaved like a wick. No problem w/o an ignition source, big problem if there was one.

[Okie_Steve]

So, how much infra structure would be needed in Texas for a permanent stack for testing procedure changes off pad? Is there enough room if a test went south?

[docmordrid]

So, how much infra structure would be needed in Texas for a permanent stack for testing procedure changes off pad? Is there enough room if a test went south?

About 2014 SpaceX expanded their footprint from 922 to 4,280 acres.

Link....

[john smith 19]

They apparently proof to only 1.25x MEOP, which is sporty.  Burst is only 1.5x MEOP, which is definitely sporty; I'd use not less than 2x.  Their approach is acceptable for metal vessels, not (IMHO) for composite.

Wow, that is sporty.

My memory was COPV's were so well behaved because they had large margins, more like 4x MEOP. I was thinking of the GHe tanks on the SSME's.

[john smith 19]

Just a note that "fluid hammer" brought on by sudden valve closures can result in 3x the standard line pressure and that has caused trouble (IE serious burns or fatalities) in lines with steam so even thought it's just a gas (or a vapor) does not mean to say it is trouble free.

A good question is did the Texas testing replicate the process on the day or did they over stress to get the failure. 

That raises the further question of why (if it exactly the conditions on the day) they did not pick it up in testing before they applied it to a vehicle with a payload on it.

My guess is SX did do testing but it may have required a "perfect storm" of marginal conditions to line up just right to cause this and fairly minor changes in any of them would break the chain that lead to this incident.

If so then changing any of those events will stop it happening again and they can move ahead with RTF while continuing to work the root cause.

[Fred Bonyea]

Just a note that "fluid hammer" brought on by sudden valve closures can result in 3x the standard line pressure and that has caused trouble (IE serious burns or fatalities) in lines with steam so even thought it's just a gas (or a vapor) does not mean to say it is trouble free.

A good question is did the Texas testing replicate the process on the day or did they over stress to get the failure. 

That raises the further question of why (if it exactly the conditions on the day) they did not pick it up in testing before they applied it to a vehicle with a payload on it.

My guess is SX did do testing but it may have required a "perfect storm" of marginal conditions to line up just right to cause this and fairly minor changes in any of them would break the chain that lead to this incident.

If so then changing any of those events will stop it happening again and they can move ahead with RTF while continuing to work the root cause.

"Gas hammer" can occur in a line that does not contain fluid: It happens when a high pressure gas is vented into a much lower pressure system.  In this case, they seem to be focused upon a micro-channel event; a very small intrusion of lox into the composite matrix. I read this as they have theorized that LOX froze when the helium (which is extremely thermally conductive at low temperatures) loading reduced the temperature of the bottle rapidly and froze the LOX, trapping LOX within a microchannel. When the helium pressure surged (hammered), the pinched-off pressurized liquid may have flash-boiled and increased the temperature within the channel to the ignition point (or the pressure within the channel to a bubble burst point), rupturing a bullet-sized hole in the bottle which rapidly unraveled and/or burned.

A remaining question, is whether or not frozen LOX is necessary to cause a microchannel event: Would it would be possible, in a sloshing LOX environment, for differential pressure, within the fiber matrix, to pinch-off a microchannel void and cause a failure without freezing the LOX? Is it possible to reduce the pressure differentials within the bottle enough to eliminate microchannel failure modes?

These are difficult questions for composite engineers and yes, the design margins are sporty.

[jpo234]

Gunman on a hill?

Yikes!  I had not heard of this theory yet!

SpaceX took it seriously enough to hire a marksman. The reference is now gone.

Edit: it's not completely gone, SpaceNews still has fragments

[obi-wan]

They apparently proof to only 1.25x MEOP, which is sporty.  Burst is only 1.5x MEOP, which is definitely sporty; I'd use not less than 2x.  Their approach is acceptable for metal vessels, not (IMHO) for composite.

Wow, that is sporty.

My memory was COPV's were so well behaved because they had large margins, more like 4x MEOP. I was thinking of the GHe tanks on the SSME's.

NASA STD-5001B specifies the required structural margins for flight vehicles. For COPVs, 5001B defers to ANSI/AIAA S-081, which specifies burst testing at 1.5xMEOP, and proof testing at 1.25 MEOP (or, exactly what SpaceX is evidently doing). It should be noted that MEOP is not the _nominal_ expected operating condition, but the worst-case condition, which NASA specifies as worst case pressure following two independent component or system failures.

[rsdavis9]

Gunman on a hill?

Yikes!  I had not heard of this theory yet!

SpaceX took it seriously enough to hire a marksman. The reference is now gone.

Edit: it's not completely gone, SpaceNews still has fragments

I think they will take it seriously enough to make sure they have appropriate means to identify and get direction. I think microphones should be enough. Bullets leave good signatures in the air.

[edkyle99]

This is also an operations problem - but it is not an operations problem which is consistent between scenarios.

The specific root cause differs in each case, but there is an overarching commonality.  It is the design of the system, the use of  high pressure COPV helium tanks immersed in super-cooled LOX.  Will changing a propellant loading procedure really fix the problem? 

 - Ed Kyle

[envy887]

This is also an operations problem - but it is not an operations problem which is consistent between scenarios.

The specific root cause differs in each case, but there is an overarching commonality.  It is the design of the system, the use of  high pressure COPV helium tanks immersed in super-cooled LOX.  Will changing a propellant loading procedure really fix the problem? 

 - Ed Kyle

This has been noted before, but... F9 FT does NOT use supercooled LOX. It uses subcooled LOX. There is a significant difference. And CRS 7 was on v1.1, which didn't even use subcooled LOX.

If CRS-7 was indeed a strut failure, then it's not obvious that moving the COPV outside the LOX tank would have saved it from LOM; the strut could still break due to manufacturing flaws, and the resultant helium system breach likely results in the second stage failing to reach orbit.

Manufacturing and operational processes can't be entirely foolproofed by design processes on systems with this type of margins.

[tdperk]

The specific root cause differs in each case, but there is an overarching commonality.  It is the design of the system, the use of  high pressure COPV helium tanks immersed in super-cooled LOX.  Will changing a propellant loading procedure really fix the problem? 

 - Ed Kyle

Taking strut manufacture in house fixed the strut problem, more careful and automatic profiling of helium temperature will fix that problem.

[edkyle99]

This is also an operations problem - but it is not an operations problem which is consistent between scenarios.

The specific root cause differs in each case, but there is an overarching commonality.  It is the design of the system, the use of  high pressure COPV helium tanks immersed in super-cooled LOX.  Will changing a propellant loading procedure really fix the problem? 

 - Ed Kyle

This has been noted before, but... F9 FT does NOT use supercooled LOX. It uses subcooled LOX. There is a significant difference. And CRS 7 was on v1.1, which didn't even use subcooled LOX.

If CRS-7 was indeed a strut failure, then it's not obvious that moving the COPV outside the LOX tank would have saved it from LOM; the strut could still break due to manufacturing flaws, and the resultant helium system breach likely results in the second stage failing to reach orbit.

Manufacturing and operational processes can't be entirely foolproofed by design processes on systems with this type of margins.

Good points, and thanks for the correction.  That leaves as the commonality the design using COPV tanks immersed in "cryogenic" LOX.

 - Ed Kyle

[Jet Black]

This is also an operations problem - but it is not an operations problem which is consistent between scenarios.

The specific root cause differs in each case, but there is an overarching commonality.  It is the design of the system, the use of  high pressure COPV helium tanks immersed in super-cooled LOX.  Will changing a propellant loading procedure really fix the problem? 

 - Ed Kyle

Super-cooled LOX aside, yes loading could fix it. Saying there is an overarching commonality is a truism, given that for the most part, it's the same rocket.

[Fred Bonyea]

The specific root cause differs in each case, but there is an overarching commonality.  It is the design of the system, the use of  high pressure COPV helium tanks immersed in super-cooled LOX.  Will changing a propellant loading procedure really fix the problem? 

 - Ed Kyle

Taking strut manufacture in house fixed the strut problem, more careful and automatic profiling of helium temperature will fix that problem.

Unless the failure mode is possible only if LOX is frozen, I don't see how. Remember, the pressure differentials and liquid levels are highly dynamic during the actual flight. And if the failure is tied to LOX freezing, they will still need to demonstrate that this cannot occur during flight, while the helium is going through rapid adiabatic transition.

[russianhalo117]

Relevant, but yet to be confirmed - by Chris Gebhardt:
https://www.nasaspaceflight.com/2016/10/spacex-prepares-upcoming-falcon-9-amos-6/

[Fred Bonyea]

Relevant, but yet to be confirmed - by Chris Gebhardt:
https://www.nasaspaceflight.com/2016/10/spacex-prepares-upcoming-falcon-9-amos-6/

From the article:

“The root cause of the breach has not yet been confirmed, but attention has continued to narrow to one of the three COPVs inside the LOX tank.”...

..."Importantly for SpaceX, 'Through extensive testing in Texas, SpaceX has shown that it can re-create a COPV failure entirely through helium loading conditions.  These conditions are mainly affected by the temperature and pressure of the helium being loaded.'”

I am reminded of the definition of a scientist: A person who can tell you with utter confidence and sincerity exactly why something, anything happened that they assured you was impossible, just before it happened.

[pogo661]

My experience is limited to devops, so take this with a grain of salt, but it seems to me that while it may be possible to instrument and automate He loading in such a way as to prevent a recurrence of this failure,  it may also be difficult to introduce such complexity without introducing additional failure modes.  It's easy (for me) to imagine that a manual procedure change may well be the best way to address this for now.  (edit) Especially if this was not operator error but a planned experiment.

[Kabloona]

Especially if this was not operator error but a planned experiment.

Loading a flight vehicle with a payload on top is probably not the time they'd want to be experimenting.

An alternative possibility to "operator error" is that they had to red-line (ie make an on-the-spot change to) the loading procedure because of, say, a stuck valve or other GSE problem that they had to work around, possibly resulting in performing some steps of the procedure out of the normal sequence.

That's something that would require engineering approval, but the cognizant engineer may have decided the change in procedure was minor enough to sign off on. And then they found out the hard way that what they thought was an insignificant change in procedure affected the COPV/LOX interaction in an unexpected way.

Launch vehicle manufacturing and integration procedures routinely get red-lined because things rarely go exactly as the engineers who wrote the procedure planned it, although by now the loading procedure and GSE should be pretty well debugged. OTOH, the change to subcooled LOX has introduced a relatively new variable, and we know they had some early problems with the subcooled LOX GSE and/or loading procedures.

So maybe they got bit by a loading procedure red-line that seemed innocuous at the time.