SpaceX Falcon 9 - AMOS-6 - (Pad Failure) - DISCUSSION THREAD (2)

[Kabloona]

Point is that it's not "all-telling" because it's not a precise technical term... 

No, because it's a euphemism for "exploded," "ruptured," or "burst," terms which senior aerospace executives typically try to avoid using when describing their hardware.

[kaiser]

Point is that it's not "all-telling" because it's not a precise technical term... 

No, because it's a euphemism for "exploded," "ruptured," or "burst," terms which senior aerospace executives typically try to avoid using when describing their hardware.

Fair enough.  Point still is that it doesn't fully implicate the COPV -- a failure elsewhere in the system could cause a COPV to let go.

[Herb Schaltegger]

Point is that it's not "all-telling" because it's not a precise technical term... 

No, because it's a euphemism for "exploded," "ruptured," or "burst," terms which senior aerospace executives typically try to avoid using when describing their hardware.

Fair enough.  Point still is that it doesn't fully implicate the COPV -- a failure elsewhere in the system could cause a COPV to let go.

There is L2 info today at least tangentially on this point.

[FinalFrontier]

From today interview to Mrs Shotwell:

On Sept. 1 it wasn’t clear whether the cause lay in the ground support equipment, or inside the rocket. You have made the determination that it was inside the rocket and not some procedure during preparation for the static test?

We believe that the composite over wrapped pressure vessel [the helium bottle], known as a COPv, let go in the tank. What caused it, the exact reason it let go, we’re still investigating. I don’t believe it was a ground system cause, but we’re still looking at the data.

Was this public knowledge before?

"Massive breach in the cryogenic helium system" had already been released via the SpaceX anomaly updates. Some folks here (me included) figured that THAT piece of information, along with the hi-speed event (93 milliseconds) could mean only one thing: catastrophic COPV failure.

This is now confirmed by Gwynne. And it also pretty much shuts-up the folks pointing to other parts of the cryogenic helium system.

Well there you go. Still surprises me that it was actually a COPV and not something else in the plumbing, but we already had a plausible failure mode in which a burst COPV ignites material in the tank.

This seems to match up with what they are saying.

So, now the question becomes why did a COPV burst on the pad, but somehow make it through acceptance testing and fill and drain cycles at McGregor without failing? Does this mean that in fact COPV itself may have contributed to CRS7 and this was simply missed? What does it say about issues SpaceX has had with COPV up to this point wrt quality control and risk management to have another failure?

But most importantly, did it fail because of material defect or mishandling, or did it fail because it was over-filled or over pressurized as the result of a control mistake?

Much as it would be the logical theory I am still not convinced of material failure being the root cause. SpaceX has already had problems with material failure on COPV's long before now, they should have learned enough from those and applied enough improvements in the FT variant of the vehicle to avoid future failures of this type. On top of that, it didn't fail on the test stand in Texas, it had to have gone through at least a few cyro cycles without failing, but did fail on the pad. That is also interesting.


We will have to see how it plays out, but I am not going to say "I told you so" even if material failure (my initial guess at the outset of the mishap) turns out to be the root cause.


What interests me the most is, if it was material failure, how will this effect timetable for introduction of the new US?.

The way Elon was talking at IAC it made it seem like he wants to get away from helium altogether even in the existing LV family, or at least for the upper stage. I am wondering if they were pressed, how fast they could phase out MVAC US in favor of a raptor US. Does 39A have infrastructure in place for methane yet? Ect.

Lots of questions now.

[FinalFrontier]

From today interview to Mrs Shotwell:

On Sept. 1 it wasn’t clear whether the cause lay in the ground support equipment, or inside the rocket. You have made the determination that it was inside the rocket and not some procedure during preparation for the static test?

We believe that the composite over wrapped pressure vessel [the helium bottle], known as a COPv, let go in the tank. What caused it, the exact reason it let go, we’re still investigating. I don’t believe it was a ground system cause, but we’re still looking at the data.

Was this public knowledge before?

"Massive breach in the cryogenic helium system" had already been released via the SpaceX anomaly updates. Some folks here (me included) figured that THAT piece of information, along with the hi-speed event (93 milliseconds) could mean only one thing: catastrophic COPV failure.

This is now confirmed by Gwynne. And it also pretty much shuts-up the folks pointing to other parts of the cryogenic helium system.

Eh, if a fitting or hose let go downstream from the COPV, the quick release of pressure could have caused the COPV to break free/torque/bust pretty immediately.

Edit:  Still likely COPV, just saying that the above statement doesn't prove it was a COPV.

I disagree. "Letting go" is an industry term for catastrophic failure of a pressure vessel or burst disk. The term goes back to at least the 1970's. I've seen "letting go" being used in this context in multiple failure investigation reports with the oldest one dating back to 1971. Gwynne using the same term "letting go" in direct relation to a COPV is IMO all-telling.

And I've used "letting go" in other ways throughout my career, and I've seen senior aerospace engineers use it in other ways also.  It's not a technical term.

The first thing that I thought of when they said "let go" was that it broke free from mounts (but maybe not via a strut failing) and shot off like a projectile.  On a pressurized vessel, that's what I think of when people say "let go".  If it catastrophically failed and just exploded, I've also seen that described as "let go" also though.

It could mean different things to different people, and again -- it very likely is a COPV issue, but the statement by Shotwell does not specifically say that it has to be the COPV.  Point is that it's not "all-telling" because it's not a precise technical term for a root cause.    The COPV could have catastrophically failed, and the root cause of that failure be somewhere other than the COPV.

It blew up. That is almost universally used as a nice term for saying your pressure vessel exploded.

But being that she said "operations" this suggests again, that it was overfilled. Or at least that is what they are going with.

I really hope they are sure before they refly, november is a month away.

[rsdavis9]

from spacenews

billsimpson • an hour ago

I'm sticking with my original guesswork theory from a couple of weeks ago. The heat from the hot compressed helium expanded the inside aluminum liner, as the cold LOX shrunk the outer carbon fiber wrap bonded to it. That put so much stress on the fiber that it broke the bond between the two and somehow caused the inner tank to lose enough support to fail. It's probably wrong, but sounds good to an amateur. Slower helium or LOX filling might help. Maybe they could cool the helium as it is being pumped in, since the super cooled LOX can't sit in the rocket warming up for very long.

They could test the theory by building a test rig using a colder liquid gas to submerge the helium tank in. Do that a dozen times, and if it doesn't explode, then that probably isn't the problem.

Or just cool the copv before the helium is introduced. Lox has a greater specific heat so it would cool things quicker. For example: Put pyrex glass in cold water before filling with hot air versus putting hot air in pyrex glass then dunking in cold water.

[LouScheffer]

From today interview to Mrs Shotwell:

We believe that the composite over wrapped pressure vessel [the helium bottle], known as a COPv, let go in the tank. What caused it, the exact reason it let go, we’re still investigating. I don’t believe it was a ground system cause, but we’re still looking at the data.

Was this public knowledge before?

"Massive breach in the cryogenic helium system" had already been released via the SpaceX anomaly updates. Some folks here (me included) figured that THAT piece of information, along with the hi-speed event (93 milliseconds) could mean only one thing: catastrophic COPV failure.

This is now confirmed by Gwynne. And it also pretty much shuts-up the folks pointing to other parts of the cryogenic helium system.

This is not all clear from the statements, where Gwynne explicitly states they don't know if the COPV failure is a cause or an effect.  If some part of the 5000 PSI plumbing broke, it could fly around with great force and smash a tank.  Or the resonance theories, where fluid flow could set up a resonance (like a whistle) that overstressed parts, including the tanks.  Or the tank was somehow overpressurized.  In any of these cases the tank design and manufacturing could be perfectly fine, and the fault lies elsewhere.  In my opinion, the "fault elsewhere caused a tank to explode" is more likely than "tank spontaneously failed while being used within its specification".

[fthomassy]

Can someone connect the dots from a source of pressure high enough to burst the suspect COPV and the COPV?

[iamlucky13]

That was a pretty solid interview by Gwynne - good questions, and good answers. Obviously not as much detail as any of us want, but there never is enough info for this site. The question about struts was fair, but difficult. Her answer seems appropriate.

Is there a different report being referenced than this one? I haven't found a public copy of the December 2015 Launch Services Program report mentioned:
https://oig.nasa.gov/audits/reports/FY16/IG-16-025.pdf

SpaceX recovered parts of the Falcon 9 rocket and, through telemetry analysis and other testing, determined the most probable cause for the mishap was a strut assembly failure in the rocket’s second stage.
...
NASA’s Launch Services Program (LSP) conducted a separate, independent review of the failure, briefing its results to senior NASA leadership on December 18, 2015.24 LSP did not identify a single probable cause for the launch failure, instead listing several “credible causes.” In addition to the material defects in the strut assembly SpaceX found during its testing, LSP pointed to manufacturing damage or improper installation of the assembly into the rocket as possible initiators of the failure. LSP also highlighted improper material selection and such practices as individuals standing on flight hardware during the assembly process, as possible contributing factors.

[Norm38]

What’s the possibility that there’s a design issue with that helium bottle?
I don’t think it’s a design issue with the bottle. I think it probably is more focused on the operations, which is one of the reasons we believe we can get back to flight so quickly.
But we have to finish the investigation. We’re not going to fly until we’re ready to fly.

On Sept. 1 it wasn’t clear whether the cause lay in the ground support equipment, or inside the rocket. You have made the determination that it was inside the rocket and not some procedure during preparation for the static test?
We believe that the composite over wrapped pressure vessel [the helium bottle], known as a COPv, let go in the tank. What caused it, the exact reason it let go, we’re still investigating. I don’t believe it was a ground system cause, but we’re still looking at the data.

These two statements are hard to reconcile.  Shotwell is leaning towards operations, not design.  That would rule out manufacturing defects, installation mistakes, all of that.  Any problem there would preclude getting back to flight quickly, I would think.

So operations, what would cause a COPV to "let go" besides over-pressure.  Where would that pressure come from except the GSE?  If it's not the GSE, and it wasn't defective hardware, then what?  Jim had said the individual  helium bottles don't have valves, so it can't be that all the GSE pressure was diverted to one bottle or point in the system?
I thought I was confused before...

[rsdavis9]

What’s the possibility that there’s a design issue with that helium bottle?
I don’t think it’s a design issue with the bottle. I think it probably is more focused on the operations, which is one of the reasons we believe we can get back to flight so quickly.
But we have to finish the investigation. We’re not going to fly until we’re ready to fly.

On Sept. 1 it wasn’t clear whether the cause lay in the ground support equipment, or inside the rocket. You have made the determination that it was inside the rocket and not some procedure during preparation for the static test?
We believe that the composite over wrapped pressure vessel [the helium bottle], known as a COPv, let go in the tank. What caused it, the exact reason it let go, we’re still investigating. I don’t believe it was a ground system cause, but we’re still looking at the data.

Actually the 2nd statement from Gwynne says that the blast occurred inside the rocket. The cause of the blast was not yet determined.
The 1st statement is less conclusive and says she doesn't think its design but its operation. Does operation include QA? I take what she said, to mean the filling operation.

[Mike_1179]

Shotwell is leaning towards operations, not design.  That would rule out manufacturing defects, installation mistakes, all of that.  Any problem there would preclude getting back to flight quickly, I would think.

So operations, what would cause a COPV to "let go" besides over-pressure.  Where would that pressure come from except the GSE?  If it's not the GSE, and it wasn't defective hardware, then what?  Jim had said the individual  helium bottles don't have valves, so it can't be that all the GSE pressure was diverted to one bottle or point in the system?

Let's go through a very simple fault tree to see how she could have come to this place:

This COPV was filled at least once before under when submerged in LOX - during testing at McGregor. If you consider that the McGregor testing itself didn't cause damage (you'd have to prove this but they have some flight history on these COPVs with several of them in both stages of all Falcon 9) and it wasn't damaged in transit (you'd have to prove with close-out photos, test data, etc) then you can consider interactions between the stage and GSE at McGregor or LC-40 to see if there is something different in their design or operation that could have caused this. If the flow rate is different at these sites, you could set up something like Lou mentioned a few posts above.

Or the resonance theories, where fluid flow could set up a resonance (like a whistle) that overstressed parts, including the tanks.  Or the tank was somehow overpressurized.

Other examples could include the manner in which the bottles are filled. To control the temperature and pressure of the COPVs, there are probably some procedures around the flow rate of He into the COPVs; those filling parameters are further impacted by the filling of the tank with LOX (which would also impact the temperature and pressure inside the COPV). Even if the He filling equipment is the same, maybe the LOX filling equipment is different or the procedures used this time were different (or not followed as expected) so that components of the He system we exposed to pressures or temperatures beyond their design margin.

[envy887]

So operations, what would cause a COPV to "let go" besides over-pressure.  Where would that pressure come from except the GSE?  If it's not the GSE, and it wasn't defective hardware, then what?
...

GSE hardware is different from operations. Helium pressure is a function of temperature and flow rate, so the timing and rate of both the helium and LOX fills affect the COPV pressure as much as the source pressure does.

[Jim]

What interests me the most is, if it was material failure,how will this effect timetable for introduction of the new US?

What new US?

[meekGee]

Point is that it's not "all-telling" because it's not a precise technical term... 

No, because it's a euphemism for "exploded," "ruptured," or "burst," terms which senior aerospace executives typically try to avoid using when describing their hardware.

Yup.  If something is under stress (especially tensile stress) and it "lets go", then yeah, it broke.

Not to be confused with "let's go".

However - "let go" indicates empathy with the part.

For example, if a tank exploded because it was over-pressurized, you're more likely to say "let go" then "failed".

[Norm38]

So operations, what would cause a COPV to "let go" besides over-pressure.  Where would that pressure come from except the GSE?  If it's not the GSE, and it wasn't defective hardware, then what?
...

GSE hardware is different from operations. Helium pressure is a function of temperature and flow rate, so the timing and rate of both the helium and LOX fills affect the COPV pressure as much as the source pressure does.

They were still filling the LOX.  So you're saying that for a given flow rate, if the COPVs were covered by LOX they don't burst, but if they're not yet covered by LOX, they could? And so a possible root cause is that they simply loaded the He too early?
Or another possible root cause, the flow rate was too high, meaning the He didn't have time to shed heat to the LOX?

[Mike_1179]

They were still filling the LOX.  So you're saying that for a given flow rate, if the COPVs were covered by LOX they don't burst, but if they're not yet covered by LOX, they could? And so a possible root cause is that they simply loaded the He too early?
Or another possible root cause, the flow rate was too high, meaning the He didn't have time to shed heat to the LOX?

There are several interactions here. Helium has a negative Joule-Thomson coefficient which means its temperature increases if it decreases pressure isenthalpically. When you open the valve from the high pressure He tanks on the pad to the COPV, the helium expands and heats up. It enters the COPV and cools down and contracts. It's not as simple as "filled with LOX too early" but there are different interactions between LOX tank temperature, LOX fill rate, LOX fill level, He pressure drop, He fill rate, He fill level and probably a dozen other things that could impact the pressure and temperature inside the helium system.

Not to mention there might be some system that monitoring overall pressure or temperature of the He lines which would have controlled the filling parameters to keep the system safe (or ruptured a burst disk in a safe area). So then you have to investigate why this didn't work (if there is such a system in place).

And this is all for one possible failure mode.

[spacekid]

Here's my COPV theory: The tanks are submerged under LOX and more HE is stored in the COPV tanks because of the cooling. As a result of heat from the COPVs the LOX became gaseous and caused the HE pressure to rise. Perhaps LOX fast fill sloshing around the tank caused the LOX to not fully cover the COPVs? Maybe some effect similar to propellers cavitating?

[Rocket Science]

What interests me the most is, if it was material failure,how will this effect timetable for introduction of the new US?

What new US?

The one that doesn't blowup...

[CameronD]

From today interview to Mrs Shotwell:

On Sept. 1 it wasn’t clear whether the cause lay in the ground support equipment, or inside the rocket. You have made the determination that it was inside the rocket and not some procedure during preparation for the static test?

We believe that the composite over wrapped pressure vessel [the helium bottle], known as a COPv, let go in the tank. What caused it, the exact reason it let go, we’re still investigating. I don’t believe it was a ground system cause, but we’re still looking at the data.

So.. for the ASDS Bingo prize (which isn't mine to give, BTW) who was the first on NSF to outline COPV failure as the root cause??  IIRC it was jongoff, but it could have been Jim... or me. 

Any other takers??