SpaceX Falcon 9 - AMOS-6 - (Pad Failure) - DISCUSSION THREAD (2)

[JasonAW3]

That is not a  "business process error", having LOX penetrate the overwrap is a design flaw.   There can be work done to the COPV's to prevent this.

Having just finished a DFMEA review, I agree.  Yes, risk can be managed by careful control of operating procedures to keep the LOX from locally freezing.  But risk is better mitigated by not allowing LOX to penetrate the COPV in the first place.

The only practical way to prevent LOX from penetrating the composite overwrap of a COPV is by not immersing the COPV in LOX. I guess none of you here have heard of thermal micro-cracking of composite materials? NASA, Ball Aerospace, ESA and Air Liquide have boatloads of knowledge on this thanks to work done for spacecraft such as IRAS, COBE, ISO and Herschell.

Going into speculation from this point forward but I have a feeling that on Amos-6 SpaceX has discovered an unexpected result of using sub-cooled LOX.

Interestingly, the matrix material for these vessels is urethane, which should minimize or eliminate micro-cracking, as its CTE is well matched to the fiber (unlike epoxy).  I've tested a lot of carbon-epoxy + urethane-coated composites, but never got ignition at normal tank pressures (we are talking >1300 tests).  But to take your point, I've never used sub-cooled LOX.

So in other words, at 66K, the urethane may have solidified and cracked, allowing subcooled oxygen to seep into the space between the composite and the actual helium tank, where it solidified, due to a further drop in temperature from the helium fill, and was put under extreme enough pressure, as the helium tank filled, that it ignited the carbon fiber?

     Can't really see that this was truly foreseeable, as they'd already tested the fill technique several times, both in Texas and at the Cape.  The repeated cold and pressures would be sufficient to cause that kind of crack, and wouldn't really be anything anyone would, or could catch.

[guckyfan]

     Can't really see that this was truly foreseeable, as they'd already tested the fill technique several times, both in Texas and at the Cape.  The repeated cold and pressures would be sufficient to cause that kind of crack, and wouldn't really be anything anyone would, or could catch.

It was not the LOX getting into micro cracks that caused the problem. It was the LOX turning to ice that caused it. I understand if they can avoid forming ice there they are ok. Should be possible with some change of procedure.

[Jet Black]

     Can't really see that this was truly foreseeable, as they'd already tested the fill technique several times, both in Texas and at the Cape.  The repeated cold and pressures would be sufficient to cause that kind of crack, and wouldn't really be anything anyone would, or could catch.

It was not the LOX getting into micro cracks that caused the problem. It was the LOX turning to ice that caused it. I understand if they can avoid forming ice there they are ok. Should be possible with some change of procedure.

Well it depends on why the ice formed. It could have formed around impurities which act as nucleation points (which could include imperfections in surfaces such as micro cracks) or possibly due to cold spots if gas was expanding (in the COPV perhaps) or a few other things. I have actually been bothered for a while about the sudden jump to subcooled LOX and whether they really understand the physics of it as well as they think they do.

[mvpel]

How sure are we that this actually has been said by Elon? How credible is the source?
Sorry if this already has been established. If so feel free to remove this post.

The marksman comment in the thread came from a known SpaceX employee.

Also, there's photos of the shredded remains of the mock-up out there.

[Space Ghost 1962]

In a cryo oxy environment, there are few substances that can act as a "wetting agent" to flow into micro-fractures/cracks, at solid oxygen temps.

Oxygen itself has interesting wetting properties, at supercritical point it can be used as a lubricant. Like on NK-33/AJ26 ... which is why it is a problem to begin with here ...

add:

As to the debate over flaw being process (operations) or vehicle, here's the gist of that:

vehicle - irrespective of operations, the vehicle should not be vulnerable to a props loading issue such that it explodes.

process - irrespective of the vehicle, the operations should not make the vehicle vulnerable to explosion.

To - mah - to, to - mate - oh.

Can we just fix the problem and start flying again ...

[JasonAW3]

     Can't really see that this was truly foreseeable, as they'd already tested the fill technique several times, both in Texas and at the Cape.  The repeated cold and pressures would be sufficient to cause that kind of crack, and wouldn't really be anything anyone would, or could catch.

It was not the LOX getting into micro cracks that caused the problem. It was the LOX turning to ice that caused it. I understand if they can avoid forming ice there they are ok. Should be possible with some change of procedure.

Well it depends on why the ice formed. It could have formed around impurities which act as nucleation points (which could include imperfections in surfaces such as micro cracks) or possibly due to cold spots if gas was expanding (in the COPV perhaps) or a few other things. I have actually been bothered for a while about the sudden jump to subcooled LOX and whether they really understand the physics of it as well as they think they do.

     Geez, I should have remembed.  Urethane is spontaneous combustible too, with the right combination of O2 and pressure.

[mfck]

     Can't really see that this was truly foreseeable, as they'd already tested the fill technique several times, both in Texas and at the Cape.  The repeated cold and pressures would be sufficient to cause that kind of crack, and wouldn't really be anything anyone would, or could catch.

It was not the LOX getting into micro cracks that caused the problem. It was the LOX turning to ice that caused it. I understand if they can avoid forming ice there they are ok. Should be possible with some change of procedure.

So, the business decision to compress time from rollout to launch causes a more aggressive prop loading sequence which breaks the temperature regime acceptable for that construction of the COPV. Specifically, causing a temperature drop inside the COPV that cooled the outer shell of the vessel to below the freezing temperature of Oxygen, then stressing it (pressurising the COPV toward its 6000psi working pressure). The LOX, that is normally allowed to permeate the carbon overwrap until it is displaced out of the fiber by the expanding of the COPV, froze between the fibers and the crystals cut the fibers, since the fibers are strong in tension, but weak in shear which the crystals applied.

Am I parsing this correctly?


Edit. ... yes the combustion part. Ignited instead of shear failed?

[Jakusb]

     Can't really see that this was truly foreseeable, as they'd already tested the fill technique several times, both in Texas and at the Cape.  The repeated cold and pressures would be sufficient to cause that kind of crack, and wouldn't really be anything anyone would, or could catch.

It was not the LOX getting into micro cracks that caused the problem. It was the LOX turning to ice that caused it. I understand if they can avoid forming ice there they are ok. Should be possible with some change of procedure.

So, the business decision to compress time from rollout to launch causes a more aggressive prop loading sequence which breaks the temperature regime acceptable for that construction of the COPV. Specifically, causing a temperature drop inside the COPV that cooled the outer shell of the vessel to below the freezing temperature of Oxygen, then stressing it (pressurising the COPV toward its 6000psi working pressure). The LOX, that is normally allowed to permeate the carbon overwrap until it is displaced out of the fiber by the expanding of the COPV, froze between the fibers and the crystals cut the fibers, since the fibers are strong in tension, but weak in shear which the crystals applied.

Am I parsing this correctly?


Edit. ... yes the combustion part. Ignited instead of shear failed?

Sounds like a very unfortunate, unexpected, but scientifically explainable sequence of events.
Assuming it is close to what went wrong, how would one go about replicating this or significant parts of this theory?
Just curious.
I am no expert, but sounds like something you could prove by seeking the extremes and then explicitly prevent by changing procedures.

Regarding discussion business process vs design: you need to press the stage to make it strong enough. That is business process (press before clamp release), or is it a design flaw that the stage would collapse if pressure is allied too late?!

[HVM]

Reddit thread is removed and comments deleted... so we will see

[rsdavis9]

I was wondering where the YouTube of musks speech was. So now it didn't happen? Was musk speaking to the nro today?  What is the nro?

[mfck]

National Reconnaissance Office?

[garidan]

So you change a timing/procedure on such complex matter, while preparing for a launch, keeping the payload aboard ?
And NOW they are learning a lot ? And people find lots of previous knowledge about this being at risk?
This is not unfortunate, or a business process failure.
SpaceX often talks about Information Technology and its way of doing.
When your software drives something complex and critical you never ever do something out of what you tested. You never do something first-time in production environment, expecially if you have people on an airplane under control, or a patient under a medical device, or a payload of somebody else.
Something very wrong happened, it sounds.
And it's worse than a bad COPV.
I feel a little disillusioned.
Please SpaceX keep going, but you did a big mess.

presumptuous
(of a person or their behavior) failing to observe the limits of what is permitted or appropriate

[M_Puckett]

How much payload would they lose if they switched to Titanium?

[garidan]

Falcon 9 had to be the most reliable rocket, and cheap. It has 9 engines and can lose one. It has only two stages to minimize risks. Uses RP1 and no solids, has 3 seconds on clamps on takeoff to do more safety checks ....
And then you go sub temp, with COPV inside LOX, and try to shorten timings too.
I see as all this fits the need to master composites and low temps for the Mars project.
They cannot fear to manage and control this matters, but there has to be time to learn, and prudence is a virtue.
You can get out of business very quickly if customers feel you put at risk their payloads with continuous experiments.
Please keep a line, harden Falcon 9 design, consolidate. I hope so.

[HMXHMX]

How much payload would they lose if they switched to Titanium?

The wit in me says: maybe all of it.

Titanium is generally viewed as not being LOX-compatible.

[tleski]

How much payload would they lose if they switched to Titanium?

The wit in me says: maybe all of it.

Titanium is generally viewed as not being LOX-compatible.

Helpful read for people interested in LOX/titanium compatibility:

www.dtic.mil/dtic/tr/fulltext/u2/273489.pdf

[CameronD]

So in other words, at 66K, the urethane may have solidified and cracked, allowing subcooled oxygen to seep into the space between the composite and the actual helium tank, where it solidified, due to a further drop in temperature from the helium fill, and was put under extreme enough pressure, as the helium tank filled, that it ignited the carbon fiber?

Can't really see that this was truly foreseeable, as they'd already tested the fill technique several times, both in Texas and at the Cape.  The repeated cold and pressures would be sufficient to cause that kind of crack, and wouldn't really be anything anyone would, or could catch.

For reference, here are a few excerpts from the timeline posted ages back relevant to He and LOX filling:
T-0:22:00   Stage 2 Fuel Loading Complete
T-0:19:30   Stage 2 Liquid Oxygen Loading
T-0:13:15   Stage 2 Helium Loading
T-0:13:00   Stage 2 LOX Flow Adjustment for Helium Cryo Load
T-0:10:00   Stage 2 Venting for LOX Fast Fill
T-0:08:00 approx... Bang!

The above seems to indicate that filing of the He tanks doesn't start until 6 minutes into the LOX loading, at which point they tweak the LOX flow rate (downwards?) and, if one makes the WAG that they only "fast fill" once the COPVs are covered with liquid, they don't ramp it up again until 3 minutes later.  Approx 2 minutes after that it all came crashing down.

According to the timeline, LOX filling doesn't finish until T-0:02:05 so even though the LOX tank should not have been much more than half full there was still plenty of time for LOX penetration.

I'm guessing they got the flow rates wrong this time around.

[docmordrid]

At the risk of overdoing it with copying other people's reddit posts, here's the top voted comment on that same reddit thread.  The post provides a plausible seeming explanation which I think might provide a useful contribution to this discussion

https://www.reddit.com/r/spacex/comments/57balr/details_from_elons_speech_at_the_nro/d8qies7

ergzay 136 points an hour ago*

For people who aren't understanding the "solid oxygen" comment, here's a likely scenario where this occurs:

The helium COPVs are sitting inside the oxygen tank. The oxygen tank is then filled with subcooled oxygen that's very close to oxygen's freezing temperature. The helium tanks are immersed in and covered by this liquid oxygen. The helium tanks are then begun to be filled. When you fill the tanks, initially before too much pressure builds up, the helium will be expanding from the pressure lines into the pressure vessels. This will cause an associated temperature drop from adiabatic cooling. This could cause the surface of the vessel to drop below the freezing temperature of Oxygen. There will now be tiny frozen/solid Oxygen crystals that develop in between the aluminum tank and the carbon fiber wrapping of the COPV. As the vessel is then pressurized these crystals are strongly compressed as the COPV expands by a few centimeters as it pressurizes. When you strongly compress an oxidizer and a fuel source (carbon fiber) you can cause spontaneous combustion if the pressures are high enough. This causes tank and COPV rupture and then the runaway explosion of the vehicle.

As noted above, Helium doesn't adiabatically cool from expansion (at these temps, at least). Need another cooling mechanism.

Acoustic chilling from the harmonics/thermoacoustic heat engine effect.

[jaufgang]

So you change a timing/procedure on such complex matter, while preparing for a launch, keeping the payload aboard ?
And NOW they are learning a lot ? And people find lots of previous knowledge about this being at risk?
This is not unfortunate, or a business process failure.
SpaceX often talks about Information Technology and its way of doing.
When your software drives something complex and critical you never ever do something out of what you tested. You never do something first-time in production environment, expecially if you have people on an airplane under control, or a patient under a medical device, or a payload of somebody else.
Something very wrong happened, it sounds.
And it's worse than a bad COPV.
I feel a little disillusioned.
Please SpaceX keep going, but you did a big mess.

presumptuous
(of a person or their behavior) failing to observe the limits of what is permitted or appropriate

Perhaps they did test the new timing/procedure for filling the propellant tanks, and the test was successful, but later during the static fire test the same timing caused the failure. Perhaps this new fill procedure only leaves the vehicle susceptible to this mode of failure only some of the time (e.g. after multiple fueling cycles?)

At any rate, if they were indeed experimenting with untested variations of the fuelling procedure on the launch pad with the payload on top, that will undoubtedly be pointed out as a serious lapse in the failure investigation report  It will be interesting to see if that is the case.

[JBF]

At any rate, if they were indeed experimenting with untested variations of the fuelling procedure on the launch pad with the payload on top, that will undoubtedly be pointed out as a serious lapse in the failure investigation report  It will be interesting to see if that is the case.

It is equally possible that the approved and tested method has an unexpected and extremely low probability failure mode.