SpaceX Falcon 9 - AMOS-6 - (Pad Failure) - DISCUSSION THREAD (2)

[Herb Schaltegger]

From today interview to Mrs Shotwell:

On Sept. 1 it wasn’t clear whether the cause lay in the ground support equipment, or inside the rocket. You have made the determination that it was inside the rocket and not some procedure during preparation for the static test?

We believe that the composite over wrapped pressure vessel [the helium bottle], known as a COPv, let go in the tank. What caused it, the exact reason it let go, we’re still investigating. I don’t believe it was a ground system cause, but we’re still looking at the data.

So.. for the ASDS Bingo prize (which isn't mine to give, BTW) who was the first on NSF to outline COPV failure as the root cause??  IIRC it was jongoff, but it could have been Jim... or me. 

Any other takers??
 

The COPV is not the root cause; it's the proximate case (e.g., the thing that made the stage kaboom). We - and SpaceX - don't yet know the root cause (the thing that caused the COPV to fail - Manufacturing flaw? Test procedure anomaly? Material deficiency? Etc).

[FinalFrontier]

From today interview to Mrs Shotwell:

On Sept. 1 it wasn’t clear whether the cause lay in the ground support equipment, or inside the rocket. You have made the determination that it was inside the rocket and not some procedure during preparation for the static test?

We believe that the composite over wrapped pressure vessel [the helium bottle], known as a COPv, let go in the tank. What caused it, the exact reason it let go, we’re still investigating. I don’t believe it was a ground system cause, but we’re still looking at the data.

So.. for the ASDS Bingo prize (which isn't mine to give, BTW) who was the first on NSF to outline COPV failure as the root cause??  IIRC it was jongoff, but it could have been Jim... or me. 

Any other takers??

I highlighted copv very early on and was vilified for it. However,  I was not the only one to say this quite a few of the seasoned folks here indicated it was unlikely another part of the vehicle could cause the failure mode as fast and as violent as we had here.  This was known early on. That said,  copv is not the root cause what caused the bottle to fail is the root cause and we do not have enough yet to know whether it was material failure or operation error.

[Kabloona]

From today interview to Mrs Shotwell:

On Sept. 1 it wasn’t clear whether the cause lay in the ground support equipment, or inside the rocket. You have made the determination that it was inside the rocket and not some procedure during preparation for the static test?

We believe that the composite over wrapped pressure vessel [the helium bottle], known as a COPv, let go in the tank. What caused it, the exact reason it let go, we’re still investigating. I don’t believe it was a ground system cause, but we’re still looking at the data.

So.. for the ASDS Bingo prize (which isn't mine to give, BTW) who was the first on NSF to outline COPV failure as the root cause??  IIRC it was jongoff, but it could have been Jim... or me. 

It was Jim who first raised the question of COPV failure.

http://forum.nasaspaceflight.com/index.php?topic=30981.msg1576251#msg1576251

[mclumber1]

The funny thing is, we learn from the same source, that the JCSat-14 S1 has gone through 11 hot fires:
1x pre-launch testing in McGregor
1x static fire on the pad
1x launch
8x post launch tests in McGregor
and it is still in one piece. My understanding is, that the pressure system of S1 and S2 is very similar.

Do we know where in the S1 LOX tank the COPVs are mounted?  Is it possible that they are mounted high enough in the LOX tank that they never actually come into contact with the super cold liquid oxygen, and are only ever exposed to gaseous oxygen and helium ullage?

[mme]

The funny thing is, we learn from the same source, that the JCSat-14 S1 has gone through 11 hot fires:
1x pre-launch testing in McGregor
1x static fire on the pad
1x launch
8x post launch tests in McGregor
and it is still in one piece. My understanding is, that the pressure system of S1 and S2 is very similar.

Do we know where in the S1 LOX tank the COPVs are mounted?  Is it possible that they are mounted high enough in the LOX tank that they never actually come into contact with the super cold liquid oxygen, and are only ever exposed to gaseous oxygen and helium ullage?

The main point of putting the COPVs in the LOX tank is to cool the He which allows each COPV to contain significantly more He and reduces the number of COPVs required.

[HMXHMX]

The funny thing is, we learn from the same source, that the JCSat-14 S1 has gone through 11 hot fires:
1x pre-launch testing in McGregor
1x static fire on the pad
1x launch
8x post launch tests in McGregor
and it is still in one piece. My understanding is, that the pressure system of S1 and S2 is very similar.

Do we know where in the S1 LOX tank the COPVs are mounted?  Is it possible that they are mounted high enough in the LOX tank that they never actually come into contact with the super cold liquid oxygen, and are only ever exposed to gaseous oxygen and helium ullage?

Yes.  No.

[mclumber1]

Yes.  No.

Isn't that the Stage 2 LOX tank?

[Kabloona]

Yes.  No.

Isn't that the Stage 2 LOX tank?

Yes, and the same logic for immersing the COPV's in LOX applies to both stages.

[mclumber1]

Yes, and the same logic for immersing the COPV's in LOX applies to both stages.

The logic makes sense to me that submerging the COPVs would allow for more helium to fit in them.  However, if what happened last month was due to some kind of thermal stress issue with one of the vessels, shouldn't we have seen this issue on the first stage as well?  SpaceX now has a booster that has seen 10+ tanking/detanking cycles and hasn't yet experienced any sort of COPV failure.

[Kabloona]

Yes, and the same logic for immersing the COPV's in LOX applies to both stages.

The logic makes sense to me that submerging the COPVs would allow for more helium to fit in them.  However, if what happened last month was due to some kind of thermal stress issue with one of the vessels, shouldn't we have seen this issue on the first stage as well?  SpaceX now has a booster that has seen 10+ tanking/detanking cycles and hasn't yet experienced any sort of COPV failure.

Gwynne Shotwell was quoted as saying they believe the failure may have been due to "operations" , which implies they don't think it was a manufacturing or design flaw. So they may have done something out of sequence or too fast during tanking.

Also, her comment about doing tests in Texas and "learning a lot" supports the hypothesis that some procedure was done differently during tanking operations and now they're learning how that difference (ie in flow rates, loading sequences, etc)  may affect the hardware in ways they didn't expect.

Early on, a reddit commenter quoted a SpaceX source as saying they had observed some "weird harmonics" in a COPV during tanking, and this could also be consistent with some operational irregularity causing unexpected phenomena.

[Fred Bonyea]

Shotwell is leaning towards operations, not design.  That would rule out manufacturing defects, installation mistakes, all of that.  Any problem there would preclude getting back to flight quickly, I would think.

So operations, what would cause a COPV to "let go" besides over-pressure.  Where would that pressure come from except the GSE?  If it's not the GSE, and it wasn't defective hardware, then what?  Jim had said the individual  helium bottles don't have valves, so it can't be that all the GSE pressure was diverted to one bottle or point in the system?

It is hard for me to come up with an 'operational' error that would lead to a bottle burst: It is almost inconceivable that it would be possible to increase the pressure of the system as described above engineered limits.

So Jim's pointing to an operational failure that stressed the bottle seems much more likely than a gross helium overload.

One think I do know, is that if there was shrapnel of any kind, it could take as little as a 50 mm breach in a pressurized bottle to cause it to 'give way' - rapidly unravel. Just as a chain is only as strong as its weakest link, a filament wound bottle is only as strong as a 50 - 150 mm wide stretch of woven fibers.

[FinalFrontier]

Yes, and the same logic for immersing the COPV's in LOX applies to both stages.

The logic makes sense to me that submerging the COPVs would allow for more helium to fit in them.  However, if what happened last month was due to some kind of thermal stress issue with one of the vessels, shouldn't we have seen this issue on the first stage as well?  SpaceX now has a booster that has seen 10+ tanking/detanking cycles and hasn't yet experienced any sort of COPV failure.

No not necessarily.

[jongoff]

From today interview to Mrs Shotwell:

On Sept. 1 it wasn’t clear whether the cause lay in the ground support equipment, or inside the rocket. You have made the determination that it was inside the rocket and not some procedure during preparation for the static test?

We believe that the composite over wrapped pressure vessel [the helium bottle], known as a COPv, let go in the tank. What caused it, the exact reason it let go, we’re still investigating. I don’t believe it was a ground system cause, but we’re still looking at the data.

So.. for the ASDS Bingo prize (which isn't mine to give, BTW) who was the first on NSF to outline COPV failure as the root cause??  IIRC it was jongoff, but it could have been Jim... or me. 

Any other takers??

Honestly, I can't remember who claimed it first. It just seemed like the one high energy source in the tank at the time (at least to a rocket guy without any insider information).

~Jon

[Nigeluna]

I believe the LOX in F9 is only a degree or so above its triple point so in order to have it cold for flight they whack it all in over a period less than 30 minutes. Therefore the bulk He load can really only occur during this time. In compressing the gas in the COPV's there will be heat generated and this has to be lost to the surrounding LOX to cram in the amount of He required. The He and LOX charge curves with time must actually be quite complex to achieve the final pre-launch state in a timely manner.

Sounds a lot like they fell off this tightrope and got a transient overpressure of a bottle or attached plumbing that caused it to go pop. Probably not a bottle failure but something about the charging profile on the day took it outside the Safe Operating Area of its spec. All FT vehicles may have been waving close to this point and for some reason they didn't see the transient, or a component may have been a little off tune on the day. Standard chemical engineering problem and more comfortable than malevalency. At least it wasn't a Seveso or Bhopal event!

[jgoldader]

If this ends up being due to a mix of procedure and equipment spec, I wonder if the report (which we'll almost surely never see, alas) will have a long account of missed signals.  That has been a very painful spot for spaceflight.  I've started thinking of such things as "unknown knowns," when you think you understand something, but don't.

[vanoord]

Gwynne Shotwell was quoted as saying they believe the failure may have been due to "operations" , which implies they don't think it was a manufacturing or design flaw. So they may have done something out of sequence or too fast during tanking.

Also, her comment about doing tests in Texas and "learning a lot" supports the hypothesis that some procedure was done differently during tanking operations and now they're learning how that difference (ie in flow rates, loading sequences, etc)  may affect the hardware in ways they didn't expect.

Early on, a reddit commenter quoted a SpaceX source as saying they had observed some "weird harmonics" in a COPV during tanking, and this could also be consistent with some operational irregularity causing unexpected phenomena.

The Reddit comment was interesting:

just had dinner with a credible source i trust that spacex is about 99% sure a COPV issue was the cause. 'explosion' originated in the LOX tank COPV container that had some weird harmonics while loading LOX.
i dont have any more detailed info beyond that, just wanted to share.

That was, I think, less than 12 hours before SpaceX confirmed that the Helium system was at fault; and since then Gwynne Shotwell has implicated the COPV.

I guess on that basis, he 'weird harmonics' comment has to be viewed as potentially credible.

IIRC there was an early suggestion of an LOX pump problem during loading? Something along the lines of one of two pumps failing?

Were that the case, I could see potential for strange flow in the LOX system - although I'm a bit flummoxed how that could lead to a COPV failing so critically.

[sandrot]

Same Reddit thread referenced by vanoord: "I remember a portion of Angle of Attack by Harrison Storms that recounted how the S2 titanium pressure vessels were failing and they didn't know why. Turns out the titanium supplier knew it was for the Apollo program and made sure the titanium was ultra-pure. Chemically pure titanium is brittle at cryogenic temperature; slightly alloyed titanium is much more tolerant of cold. Of course this is blind conjecture, but I wonder if something similar happened here."

[spacenut]

I thought the helium system in the first stage was in the kerosene tanks and the lox tanks in the second stage.  A redesign of the helium system to the kerosene tanks on the second stage might be in order.  However the common bulkhead might have to be adjusted to make up the difference in volume. 

The first stage seems to be working quite well, and the second might have to use the same first stage design to avoid this helium system problem in the future. 

[rsdavis9]

I thought the helium system in the first stage was in the kerosene tanks and the lox tanks in the second stage.  A redesign of the helium system to the kerosene tanks on the second stage might be in order.  However the common bulkhead might have to be adjusted to make up the difference in volume. 

The first stage seems to be working quite well, and the second might have to use the same first stage design to avoid this helium system problem in the future.

copv's in lox tank in both stages.

[JamesH65]

Yes, and the same logic for immersing the COPV's in LOX applies to both stages.

The logic makes sense to me that submerging the COPVs would allow for more helium to fit in them.  However, if what happened last month was due to some kind of thermal stress issue with one of the vessels, shouldn't we have seen this issue on the first stage as well?  SpaceX now has a booster that has seen 10+ tanking/detanking cycles and hasn't yet experienced any sort of COPV failure.

Gwynne Shotwell was quoted as saying they believe the failure may have been due to "operations" , which implies they don't think it was a manufacturing or design flaw. So they may have done something out of sequence or too fast during tanking.

Also, her comment about doing tests in Texas and "learning a lot" supports the hypothesis that some procedure was done differently during tanking operations and now they're learning how that difference (ie in flow rates, loading sequences, etc)  may affect the hardware in ways they didn't expect.

Early on, a reddit commenter quoted a SpaceX source as saying they had observed some "weird harmonics" in a COPV during tanking, and this could also be consistent with some operational irregularity causing unexpected phenomena.

I try not to comment on things like this when as a complete outsider I/We have no real data to work with. So I not claiming this as any sort of fact! It's a complete guess, but the harmonics point does sound interesting. Most people will have encountered weird harmonics in a water system at some point in their lives - I have one at the moment caused by a ball valve which make the whole house buzz, very loudly. Caused by a particular flow rate (main water supply) going through a constriction (the valve). They can produce a lot of energy. When combined with something akin to 'water hammer', I wonder if there could be enough amplified energy to rupture something unfortunate, ie the COPV, or something very close to it, a connector perhaps.