SpaceX Falcon 9 - AMOS-6 - (Pad Failure) - DISCUSSION THREAD (2)

[gospacex]

Some more fun facts about helium, taken from comments on another website, when SpaceX had a problem with the helium system that delayed the Turkmen/Monaco Sats launch for a week in March of last year, not after the CRS-7 anomaly in June that year as I originally thought.

Inert gas my ass. It does what it wants to, when it wants to, hahaha.

What that even means? Helium _is_ chemically inert.

Helium has a very small atomic cross section. It leaks where other gasses will not. Pain in the butt...

(mentioned earlier)

Helium is tricky though; it can leak through solid steel albeit slowly.

Which is hardly a problem in this case. Slowly leaking He would not make COPV go boom.

The main reason for use of helium in such systems is not that helium is inert (in such case one may use argon, which is chemically as inert as helium, but also much cheaper and easier to handle), or that it weighs less (its mass is negligible compared with mass of whole rocket

Wrong. Rocket equation is a harsh mistress. Every kg counts, especially for the upper stage.

That "expert" does not sound too "experty" to me.

[Steven Pietrobon]

The simplest possible approach to helium loading would be to just let ambient temperature helium flow into the tank from the high-pressure railroad cars, then let the LOX cool it.  However, it's not clear that the helium in the COPV tanks could cool quickly enough.

Well, I was sure they loaded the helium at cryogenic temperatures. Listened to the last technical broadcast and now I'm not so sure. They mention cryohelium (which is what I remembered) as well as just helium.

T-12:53 Stage 2 LOX throttling back. Preparation for helium load.
T-8:12 Stage 1 cryohelium is in topping.
T-7:01 Stage 1 helium cushion secured.
T-6:21 Stage 2 cryohelium is in topping.
T-4:41 Stage 1 helium level secured.
T-1:16 Stage 1 and 2 cryohelium and ACS secured for flight.
T-0:25 Ground helium closeout complete.

[Steven Pietrobon]

From the Wall Street Journal. Enter the link in Google to read the whole article.
 
http://www.wsj.com/articles/spacex-probe-into-blast-focuses-on-fueling-issues-1477042203

"Investigators believe operational issues linked to fueling procedures, rather than a manufacturing flaw, likely caused a Space Exploration Technologies Corp. rocket to explode during ground tests last month, according to people familiar with the details.

Seven weeks after the high-profile accident, which occurred during a routine fueling exercise, company experts are conducting tests to try to determine the precise sequence of events. They hope to re-create the complex interplay of variables—including pressure, temperature and fill rate—suspected of rupturing a helium tank inside the Falcon 9 booster’s upper stage, the people said."

[edkyle99]

This is beginning to remind me of the still-running, 100 year debate about the Royal Navy's battlecruisers at Jutland.   Three suffered massive magazine explosions, costing more than 3,000 lives.  Arguments continue today about whether it was due to design flaws (they had battleship guns but relatively little armor), operational issues (they fought the more heavily armored, but lesser-gunned German battlecruisers at closer range than they should have, gun crews shortcut blast protection systems, etc), or something else (Royal Navy cordite turned out to be more volatile than other navy's propellant), etc..

Will we similarly always wonder about Falcon 9?  Two helium system related explosions, so far.  Is it faulty design, or was the second failure due only to faulty procedures?  Struts or frozen LOX?  Bad welds?  Gunman on a hill?

 - Ed Kyle

[LouScheffer]

The simplest possible approach to helium loading would be to just let ambient temperature helium flow into the tank from the high-pressure railroad cars, then let the LOX cool it.  However, it's not clear that the helium in the COPV tanks could cool quickly enough.

Well, I was sure they loaded the helium at cryogenic temperatures. Listened to the last technical broadcast and now I'm not so sure. They mention cryohelium (which is what I remembered) as well as just helium.

T-12:53 Stage 2 LOX throttling back. Preparation for helium load.
T-8:12 Stage 1 cryohelium is in topping.
T-7:01 Stage 1 helium cushion secured.
T-6:21 Stage 2 cryohelium is in topping.
T-4:41 Stage 1 helium level secured.
T-1:16 Stage 1 and 2 cryohelium and ACS secured for flight.
T-0:25 Ground helium closeout complete.

Perhaps where you remember this from is: http://spaceflight101.com/falcon-9-ft-countdown-timeline/

T-0:25:00   All three Cryo Helium Pumps active
T-0:22:00   Stage 2 Fuel Loading Complete
T-0:19:30   Stage 2 Liquid Oxygen Loading
T-0:17:20   Stage 1 LOX Flowrate Adjustment for Stage 2 Fast Fill
T-0:13:15   Stage 2 Helium Loading
T-0:13:00   Stage 2 LOX Flow Adjustment for Helium Cryo Load

Although I have no idea where this timeline came from, and how reliable it is, the fact that they say "Cryo Helium pumps active" to me implies they are doing something more than just passive loading from even-higher pressure tanks.

[woods170]

This is beginning to remind me of the still-running, 100 year debate about the Royal Navy's battlecruisers at Jutland.   Three suffered massive magazine explosions, costing more than 3,000 lives.  Arguments continue today about whether it was due to design flaws (they had battleship guns but relatively little armor), operational issues (they fought the more heavily armored, but lesser-gunned German battlecruisers at closer range than they should have, gun crews shortcut blast protection systems, etc), or something else (Royal Navy cordite turned out to be more volatile than other navy's propellant), etc..

Will we similarly always wonder about Falcon 9?  Two helium system related explosions, so far.  Is it faulty design, or was the second failure due only to faulty procedures?  Struts or frozen LOX?  Bad welds?  Gunman on a hill?

 - Ed Kyle

Minor nit. CRS-7 did not explode. The Helium system ruptured, over-pressurizing and subsequently rupturing the LOX tank. Amos-6 was quite different in that it was way more energetic to start with and also a whole lot more rapid in nature.

[Mike_1179]

The simplest possible approach to helium loading would be to just let ambient temperature helium flow into the tank from the high-pressure railroad cars, then let the LOX cool it.  However, it's not clear that the helium in the COPV tanks could cool quickly enough.

Well, I was sure they loaded the helium at cryogenic temperatures. Listened to the last technical broadcast and now I'm not so sure. They mention cryohelium (which is what I remembered) as well as just helium.

T-12:53 Stage 2 LOX throttling back. Preparation for helium load.
T-8:12 Stage 1 cryohelium is in topping.
T-7:01 Stage 1 helium cushion secured.
T-6:21 Stage 2 cryohelium is in topping.
T-4:41 Stage 1 helium level secured.
T-1:16 Stage 1 and 2 cryohelium and ACS secured for flight.
T-0:25 Ground helium closeout complete.

I guess one question is what do they mean by "cryo-helium".

Do they run the He through a heat exchanger to cool it before it fills the COPV? If they didn't, it would heat up the sub-cooled LOX.

Assuming the stage is full of LOX before COPV loading starts:

If you're running regular old LOX, then heating it up when the COPVs are filled doesn't matter as much - you'd cause more boil-off until an equilibrium is reached but would be able to top-off with LOX as this faster boil-off occurs. At some point, COPV is filled and boil-off returns to pre-He loading levels.

If you're cooling the LOX, then heating it more during COPV fill goes the wrong way. Chilling the He to get it to take up less space eats up performance from lower LOX load. So you can account for the heat at the bottom of the LOX tank by cycling the less-chilled LOX out and replacing it with new cold stuff or you could pre-condition the He so it's not as cold when it enters the COPV.

Does the F9FT pre-chill the He?

[intrepidpursuit]

I don't think it should be any surprise that both major failures and other issues were caused by the helium system. It is likely the most complex mechanical part of the whole system outside of the engines and gets much less full-up testing because it requires the whole system to truly test. If they had repeated failures of much simpler systems it would be cause for alarm.

[scubadown]

At KSC we use what we call rechargers. They take cryogenic liquid, pump it up to a high pressure (usually 6K PSIG) and then vaporizer the liquid, leaving a high pressure gas. Liquid air, and LN2 are the most common commodities we do this with. A number of years ago a vendor developed (with NASA funding and support) a recharger for LHe.  We didn't put into use at the CCF due to business reasons but maybe that is what SpaceX uses? You end up with high pressure gas and if you size the vaporizer right it would be very cold.

[DavidH]

This is beginning to remind me of the still-running, 100 year debate about the Royal Navy's battlecruisers at Jutland.   Three suffered massive magazine explosions, costing more than 3,000 lives.  Arguments continue today about whether it was due to design flaws (they had battleship guns but relatively little armor), operational issues (they fought the more heavily armored, but lesser-gunned German battlecruisers at closer range than they should have, gun crews shortcut blast protection systems, etc), or something else (Royal Navy cordite turned out to be more volatile than other navy's propellant), etc..

Will we similarly always wonder about Falcon 9?  Two helium system related explosions, so far.  Is it faulty design, or was the second failure due only to faulty procedures?  Struts or frozen LOX?  Bad welds?  Gunman on a hill?

 - Ed Kyle

Hmph. Or maybe a bit like the Orbital fairing separation problems.

“However, no root cause for the fairing’s failure to separate was able to be determined.”

[CyndyC]

Inert gas my ass. It does what it wants to, when it wants to, hahaha.

What that even means? Helium _is_ chemically inert.

The person knows helium is inert, but was pointing out the irony and attempting to add some humor.

Helium has a very small atomic cross section. It leaks where other gasses will not. Pain in the butt...

Helium is tricky though; it can leak through solid steel albeit slowly.

Which is hardly a problem in this case. Slowly leaking He would not make COPV go boom.

Not by itself, but could further distort the thermal gradients, and small leaks often become larger leaks under additional stress.

The main reason for use of helium in such systems is not that helium is inert (in such case one may use argon, which is chemically as inert as helium, but also much cheaper and easier to handle), or that it weighs less (its mass is negligible compared with mass of whole rocket

Wrong. Rocket equation is a harsh mistress. Every kg counts, especially for the upper stage.

That "expert" does not sound too "experty" to me.

The same person later said he is a professional chemist who has experience working with cryogenic substances, and he reasserted his previous statement, but note that he only claimed the low boiling point is the "main" reason helium is used.

[The Amazing Catstronaut]

This is beginning to remind me of the still-running, 100 year debate about the Royal Navy's battlecruisers at Jutland.   Three suffered massive magazine explosions, costing more than 3,000 lives.  Arguments continue today about whether it was due to design flaws (they had battleship guns but relatively little armor), operational issues (they fought the more heavily armored, but lesser-gunned German battlecruisers at closer range than they should have, gun crews shortcut blast protection systems, etc), or something else (Royal Navy cordite turned out to be more volatile than other navy's propellant), etc..

Will we similarly always wonder about Falcon 9?  Two helium system related explosions, so far.  Is it faulty design, or was the second failure due only to faulty procedures?  Struts or frozen LOX?  Bad welds?  Gunman on a hill?

 - Ed Kyle

What are you suggesting, Ed? Insinuation isn't the bonemeal of this site.


There's not a single fault. Overlapping similarities and events that lead to the same outcome due to the propagation of the failure tree, but not a big, illusive spectre.

It's a roughly seven week investigation.  Have you seen how long some air accident investigations go on for? They are confident of a near-term RTF. This is not schedule panic, this is evidence of having most of the data points and just needing the last few questions answered to give a comprehensive narrative to the world.

As a Brit and a naval enthusiast, I can say that all of the listed battlecruiser's "faults" applied - but they were an example of an operations problem. This is also an operations problem - but it is not an operations problem which is consistent between scenarios. Localisation is comparable, as in both the historical pseudo-example you provided. Armour and structure on the ships, helium system on the other. But it is dangerous reasoning to draw equivalence because it is convenient to do so. Everything should be case-by-case in history.

[Kabloona]

It's a roughly seven week investigation.  Have you seen how long some air accident investigations go on for? They are confident of a near-term RTF. This is not schedule panic, this is evidence of having most of the data points and just needing the last few questions answered to give a comprehensive narrative to the world.

The point Ed was making has nothing to do with the length of the investigation. It's that both F9 failure investigations have apparently been hampered by lack of conclusive data. SpaceX had to do a good deal of interpretation of the flight data they had in order to come up with the strut failure hypothesis. And we've been told that there are still people within NASA who believe the first failure may have involved a COPV failure, not (or maybe in addition to) a strut failure. And if this latest failure was in fact caused by solid LOX being compressed within the COPV overwrap, it's quite unlikely there was any instrumentation in the vicinity to prove it.

So in both cases, SpaceX is having to draw conclusions based on data sets that aren't as definitive as they would like, and it leaves room for the concern that they (and their customers) can't be 100% certain of the root causes and their fixes.

Thus the analogy with other accidents in which there wasn't enough information to completely dispel all doubt as to the cause.

[Lars-J]

[copied from the update thread]

SpaceX just posted an update - seems like they are making progress and can at least partially reproduce what happened:

http://www.spacex.com/news/2016/09/01/anomaly-updates

October 28, 4:00pm EDT

The Accident Investigation Team continues to make progress in examining the anomaly on September 1 that led to the loss of a Falcon 9 and its payload at Launch Complex 40 (LC-40), Cape Canaveral Air Force Station, Florida.

Since the incident, investigators from SpaceX, the FAA, NASA, the US Air Force and industry experts have been working methodically through an extensive fault tree to investigate all plausible causes. As part of this, we have conducted tests at our facility in McGregor, Texas, attempting to replicate as closely as possible the conditions that may have led to the mishap.

The investigation team has made significant progress on the fault tree. Previously, we announced the investigation was focusing on a breach in the cryogenic helium system of the second stage liquid oxygen tank. The root cause of the breach has not yet been confirmed, but attention has continued to narrow to one of the three composite overwrapped pressure vessels (COPVs) inside the LOX tank. Through extensive testing in Texas, SpaceX has shown that it can re-create a COPV failure entirely through helium loading conditions. These conditions are mainly affected by the temperature and pressure of the helium being loaded.

SpaceX’s efforts are now focused on two areas – finding the exact root cause, and developing improved helium loading conditions that allow SpaceX to reliably load Falcon 9. With the advanced state of the investigation, we also plan to resume stage testing in Texas in the coming days, while continuing to focus on completion of the investigation. This is an important milestone on the path to returning to flight.

Pending the results of the investigation, we continue to work towards returning to flight before the end of the year. Our launch sites at Kennedy Space Center, Florida, and Vandenberg Air Force Base, California, remain on track to be operational in this timeframe.

[Fred Bonyea]

"Through extensive testing in Texas, SpaceX has shown that it can re-create a COPV failure entirely through helium loading conditions. "

This is good news. One heartburn question is: Does it take a specific cycling with the potential of creating solid O2, or are there a variety of cycling conditions that will compromise COPV performance? In one case, operational safeguards can prevent failures; if cycling along produces failures, it may require a complete system redesign.

[rsdavis9]

"Through extensive testing in Texas, SpaceX has shown that it can re-create a COPV failure entirely through helium loading conditions. "

This is good news. One heartburn question is: Does it take a specific cycling with the potential of creating solid O2, or are there a variety of cycling conditions that will compromise COPV performance? In one case, operational safeguards can prevent failures; if cycling along produces failures, it may require a complete system redesign.

Considering they have cycled the recovered booster 8 times now it would seem it isn't cycling but specific temperature and pressure. (as stated in the update).

I am so happy that they have reproduced the issue. My biggest worry was they were never going to get a reproduction of the explosion and would have to end the investigation with a series of probabilities. Something which would leave doubt as to what the issue really was. (not that this will stop the doubters).

[Rocket Science]

I need a video... seeing is believing...

[Robotbeat]

 Just because they "reproduced "the same conditions and the same result does not mean that they found the root cause. Remember, they reproduce the results using a sniper and a bullet.

[Jimmy_C]

It is good that they are closer to figuring out the physical root cause of the explosion. But why did SpaceX allow a new loading procedure during the static fire? Are they underestimating the risk of their incremental tweaks to the launch vehicle?

The last failure taught them to perform more validation testing of certified components. Will this failure teach SpaceX to perform more testing before making changes to their in-production equipment and procedures?

On one hand, this could slow down their rate of innovation. On the other hand, more testing and slower updates could help improve their reliability. I have faith that they will learn from the incident, but their next challenge will be to prove it.

[Coastal Ron]

Considering they have cycled the recovered booster 8 times now it would seem it isn't cycling but specific temperature and pressure. (as stated in the update).

However the recovered booster is a 1st stage, not a 2nd stage like the one that failed.  So while there are similarities in the designs between the 1st & 2nd stages, it's not an apples-to-apples comparison.

I am so happy that they have reproduced the issue.

I too think this is very important, just like they were able to find under-strength struts on the CRS-8 failure.