SpaceX’s Crew Dragon spacecraft static fire anomaly - THREAD 3

[ChrisWilson68]

This: it's in a DTIC memo from the 60s (page 9).  I don't know why it's "unexpected" for titanium to ignite when NTO is rammed into it at high pressure.

Not related to this thread, but this little memo made me pause for an instant and awe to the myriad of knowledge generated in the 60s for the Mercury, Gemini and Apollo missions. How many more memos like that are out there forgotten by the current professionals dealing with the same questions and that will provoke "unexpected" issues  in the future?

Back to thread

Seems like the perfect kind of things to have all scanned and fed into an AI?

There's a certain type of person who, when faced with a problem, will say, "I know, l'll use AI." Now they have two problems.

In all seriousness, this is why you need senior M&P engineers with lots of experience as well as well defined processes that, in addition to explaining what to do and what not to do, give clear rationale.

This is the type of problem where SpaceX's habit of maintaining a young workforce by chewing through and burning out its workers puts it at a disadvantage.

I'd say this is a case in point for why SpaceX would be reluctant to put out information quickly after an incident.

People seize on any little thing, add speculation to speculation and bring that to an unjustified conclusion about SpaceX.

There is definitely not enough information in the press release to justify the conclusion that this happened because SpaceX has too young a workforce and they missed things known decades ago.

In fact, while SpaceX's workforce included many younger people, they've also hired many older people with decades of experience in "old Space".

[ChrisWilson68]

Short the finger pointing, I suppose this is a good thing, even if that first press conference is going to be a bunch of people with serious looks on their faces going,

"No, we don't yet know the root cause of the incident."

"No, we really don't know the proximate cause either."

"No, we don't know how much this is going to affect the schedule, if at all."

"No, I don't know why I'm standing here, except that we don't want the media to be frustrated, I guess."

One thing that they could say that would decrease media frustration is "Here is a public domain video of the anomaly".   The detailed answers will come later, and the media knows that, but this gives them something to put in their article that will be fascinating to their audience.  It also avoids any appearance of a coverup.

SpaceX has a lot of powerful enemies that like to latch onto anything they can to sow unjustified fear, uncertainty, and doubt.  Releasing a video of a failure just gives ammunition to that kind of unfair treatment.  I can understand why SpaceX would not want to do that, particularly at a time when they didn't have the information themselves to authoritatively refute such attacks.

Yes when rational discussion ends reach for the tinfoil hat. That’s how this post comes across.

No tinfoil, and everything I said was perfectly rational.

There's lots of SpaceX bashing through misleading statements by powerful people with a huge vested interests against SpaceX.  Just look at the public comments by certain members of Congress at the time of Zuma.  Look at all the public statements by people involved with Ariane in the early days of SpaceX.

[mmeijeri]

Seems odd to believe that SpaceX did not carefully word their written statement.

And my experience with failures (and software and bugs of all kinds) is that it is rarely the expected thing (keep MMH and NTO from mixing via the He lines!) which fails.  It seems more likely (to me) than an unexpected failure would result from precisely something which was *not* obviously a problem.  Why does it matter if NTO gets into the He lines which are going to vent into the NTO tank anyway?  Well, there's some Ti there.  But NTO tanks are made of Ti all the time! So no big deal, even if it were to leak.  And so the unexpected happens...

For all we know NTO from GSE might indeed have ended up in the He pressure vessel, but wouldn't it then have likely also ended up in the hydrazine tank, which as far as we know it didn't? After all, both propellant tanks are pressurised from the same He pressure vessel. It's a possible interpretation, but is it probable?

[SWGlassPit]

It's not clear that the statement was sufficiently precisely worded to withstand such close reading. But isn't it more logical that the check valve was intended to prevent NTO from getting into the helium line (and from there eventually to the hydrazine tank, with catastrophic consequences) than to prevent helium from getting into the NTO tank prematurely?

Seems odd to believe that SpaceX did not carefully word their written statement.

And my experience with failures (and software and bugs of all kinds) is that it is rarely the expected thing (keep MMH and NTO from mixing via the He lines!) which fails.  It seems more likely (to me) than an unexpected failure would result from precisely something which was *not* obviously a problem.  Why does it matter if NTO gets into the He lines which are going to vent into the NTO tank anyway?  Well, there's some Ti there.  But NTO tanks are made of Ti all the time! So no big deal, even if it were to leak.  And so the unexpected happens...

I supppose that is a trivially true statement: if the thing you expected to fail fails, then what good was your test?

Tanks aren't subject to impact or water hammer the way valves are.  If you damage a tank to the point that you can knock the oxide layer off the tank and react with the NTO, it's probably going to rupture anyway.

[yg1968]

I couldn't find the teleconference but this video has small excerpts:

[su27k]

It should, Titanium is used everywhere for NTO. The fact it can burn it I don't think is well known. Titanium is generally pretty non-reactive.

Interesting tweets on this topic by Ben Brockert:

https://twitter.com/wikkit/status/1150855184924336128?s=20

SpaceX update on the disappearing Dragon. (link: https://www.spacex.com/news/2019/07/15/update-flight-abort-static-fire-anomaly-investigation) spacex.com/news/2019/07/1… “It is worth noting that the reaction between titanium and NTO at high pressure was not expected” is a frightening sentence; titanium is well known to have poor oxidizer compatibility and be easy to set on fire. “resistant to N2O4 except under impact... found that Ti impacts sporadically under reasonably well controlled test conditions; the ignition freq. is increased markedly by Ti filings or glass particles on the impact surface” Compatibility of Materials w/ Rocket Prop and Oxidizers

This: it's in a DTIC memo from the 60s (page 9).  I don't know why it's "unexpected" for titanium to ignite when NTO is rammed into it at high pressure.

This NTRS document seems to show Titanium is compatible with N2O4: Material compatibility with space storable propellants

See Page 59, Table 7 N2O4 Summary Compatibility Chart

Edit: Added the entire table covering all metals, looking at the entire table, for a Rating 1 metal, Titanium is a very good choice.

[dondar]

After all of this I do have a few outstanding questions:

1) Due to the lack of clarity in the statement, what actually leaked?

2) More importantly, WHY did it leak?
  - Design flaw?
  - Manufacturing defect?
  - Environmental exposure?  (After-all, this was a flown component)

Basically, the 80% is that they now have confidence in what happened and how to prevent it... That remaining 20% is the bear, where it is needed to determine if there is anything else that could be impacted in a similar way.

It leaks because of physics.
The design solution, that SpaceX implements, implies that the check valves are a culprit.
It is well known that they are not 100% leak proof, but nobody bothered to check their robustness against 2k psi hydrodynamic shock.
Probably the pressure discharge after draco test had introduced some TNO into helium plumbing which was sufficient to hammer, brittle and ignite the check valve.
It is possible  that such drops of NTO or MMH were leaked before, but it is also probable that during previous tests the time off between engines runs was sufficiently long for drops of NTO to dissipate into vapor and to become irrelevant contamination.
They do not say directly what had caused the leakage because they do not know factually. Such events don't leave forensic prints. Just like they were saying nothing in the beginning because they did not know anything. They had nothing to say.

From the point of view of it's current duties Dragon 2 propulsion plumbing is over-engineered. Introduction of the burst disks makes it even more equivalent to the solid-fuel abort systems used on all classical crafts. Pity but it is very understandable decision, they want to close Dragon chapter and move on.

[SWGlassPit]

It should, Titanium is used everywhere for NTO. The fact it can burn it I don't think is well known. Titanium is generally pretty non-reactive.

Interesting tweets on this topic by Ben Brockert:

https://twitter.com/wikkit/status/1150855184924336128?s=20

SpaceX update on the disappearing Dragon. (link: https://www.spacex.com/news/2019/07/15/update-flight-abort-static-fire-anomaly-investigation) spacex.com/news/2019/07/1… “It is worth noting that the reaction between titanium and NTO at high pressure was not expected” is a frightening sentence; titanium is well known to have poor oxidizer compatibility and be easy to set on fire. “resistant to N2O4 except under impact... found that Ti impacts sporadically under reasonably well controlled test conditions; the ignition freq. is increased markedly by Ti filings or glass particles on the impact surface” Compatibility of Materials w/ Rocket Prop and Oxidizers

This: it's in a DTIC memo from the 60s (page 9).  I don't know why it's "unexpected" for titanium to ignite when NTO is rammed into it at high pressure.

This NTRS document seems to show Titanium is compatible with N2O4: Material compatibility with space storable propellants

See Page 59, Table 7 N2O4 Summary Compatibility Chart

Edit: Added the entire table covering all metals, looking at the entire table, for a Rating 1 metal, Titanium is a very good choice.

Right there on the same page, it reports, "Some shock sensitivity reported."

That's a big red flag to explore what it means and be very careful using it on items that may experience shock loading, like a valve that may slam shut or experience water hammer.

[su27k]

It should, Titanium is used everywhere for NTO. The fact it can burn it I don't think is well known. Titanium is generally pretty non-reactive.

Interesting tweets on this topic by Ben Brockert:

https://twitter.com/wikkit/status/1150855184924336128?s=20

SpaceX update on the disappearing Dragon. (link: https://www.spacex.com/news/2019/07/15/update-flight-abort-static-fire-anomaly-investigation) spacex.com/news/2019/07/1… “It is worth noting that the reaction between titanium and NTO at high pressure was not expected” is a frightening sentence; titanium is well known to have poor oxidizer compatibility and be easy to set on fire. “resistant to N2O4 except under impact... found that Ti impacts sporadically under reasonably well controlled test conditions; the ignition freq. is increased markedly by Ti filings or glass particles on the impact surface” Compatibility of Materials w/ Rocket Prop and Oxidizers

This: it's in a DTIC memo from the 60s (page 9).  I don't know why it's "unexpected" for titanium to ignite when NTO is rammed into it at high pressure.

This NTRS document seems to show Titanium is compatible with N2O4: Material compatibility with space storable propellants

See Page 59, Table 7 N2O4 Summary Compatibility Chart

Edit: Added the entire table covering all metals, looking at the entire table, for a Rating 1 metal, Titanium is a very good choice.

Right there on the same page, it reports, "Some shock sensitivity reported."

That's a big red flag to explore what it means and be very careful using it on items that may experience shock loading, like a valve that may slam shut or experience water hammer.

If you read the explanation of the "shock sensitivity reported" on page 74, it looks very mild, plus additional test performed by McDonnell Douglas doesn't seem to be able to reproduce it.

Also Ben Brockert's claim is that "titanium is well known to have poor oxidizer compatibility and be easy to set on fire.", I believe this NASA document pretty much refutes his claim entirely.

[tdperk]

It should, Titanium is used everywhere for NTO. The fact it can burn it I don't think is well known. Titanium is generally pretty non-reactive.

Interesting tweets on this topic by Ben Brockert:

https://twitter.com/wikkit/status/1150855184924336128?s=20

SpaceX update on the disappearing Dragon. (link: https://www.spacex.com/news/2019/07/15/update-flight-abort-static-fire-anomaly-investigation) spacex.com/news/2019/07/1… “It is worth noting that the reaction between titanium and NTO at high pressure was not expected” is a frightening sentence; titanium is well known to have poor oxidizer compatibility and be easy to set on fire. “resistant to N2O4 except under impact... found that Ti impacts sporadically under reasonably well controlled test conditions; the ignition freq. is increased markedly by Ti filings or glass particles on the impact surface” Compatibility of Materials w/ Rocket Prop and Oxidizers

This: it's in a DTIC memo from the 60s (page 9).  I don't know why it's "unexpected" for titanium to ignite when NTO is rammed into it at high pressure.

This NTRS document seems to show Titanium is compatible with N2O4: Material compatibility with space storable propellants

See Page 59, Table 7 N2O4 Summary Compatibility Chart

Edit: Added the entire table covering all metals, looking at the entire table, for a Rating 1 metal, Titanium is a very good choice.

Right there on the same page, it reports, "Some shock sensitivity reported."

That's a big red flag to explore what it means and be very careful using it on items that may experience shock loading, like a valve that may slam shut or experience water hammer.

And when you "read the memo" that says the ignition does not propagate, when you observe such systems are commonly used in spacecraft, why doesn't the flag go back down?

" In all seriousness, this is why you need senior M&P engineers with lots of experience as well as well defined processes that, in addition to explaining what to do and what not to do, give clear rationale. "

That seems like a statement made with justification only in hindsight.

[SWGlassPit]

Also Ben Brockert's claim is that "titanium is well known to have poor oxidizer compatibility and be easy to set on fire.", I believe this NASA document pretty much refutes his claim entirely.

Yeah, and lets not forget that Brockert deleted his tweet pretty quickly.

What are you talking about?  It's still there.  I just pulled it up.

https://twitter.com/wikkit/status/1150855184924336128

https://twitter.com/wikkit/status/1150862156658667520

[SWGlassPit]

And when you "read the memo" that says the ignition does not propagate, when you observe such systems are commonly used in spacecraft, why doesn't the flag go back down?

" In all seriousness, this is why you need senior M&P engineers with lots of experience as well as well defined processes that, in addition to explaining what to do and what not to do, give clear rationale. "

That seems like a statement made with justification only in hindsight.

Because I don't want ignition anywhere near my propellant storage or delivery infrastructure, regardless of whether it propagated in lab tests.

I make the M&P comment because all three major anomalies SpaceX has experienced have had material compatibility issues in the failure chain.

 -- Titanium + NTO + shock loading
 -- Carbon fiber immersed in LOX
 -- A martensitic steel casting used as a tie rod end immersed in cryos in a high vibe environment.

Hindsight can uncover patterns.

[tdperk]

And when you "read the memo" that says the ignition does not propagate, when you observe such systems are commonly used in spacecraft, why doesn't the flag go back down?

" In all seriousness, this is why you need senior M&P engineers with lots of experience as well as well defined processes that, in addition to explaining what to do and what not to do, give clear rationale. "

That seems like a statement made with justification only in hindsight.

Because I don't want ignition anywhere near my propellant storage or delivery infrastructure, regardless of whether it propagated in lab tests.

I make the M&P comment because all three major anomalies SpaceX has experienced have had material compatibility issues in the failure chain.

 -- Titanium + NTO + shock loading
 -- Carbon fiber immersed in LOX
 -- A martensitic steel casting used as a tie rod end immersed in cryos in a high vibe environment.

Hindsight can uncover patterns.

And the distinctions you aren't making is in the first two, there were no design incompatibility issues because the materials were not supposed to meet, and in the latter, for the design loading, the part spec'ed was drastically overstrength for it's use in that cryogenic vibration application--and failure occurred because the part did not meet spec, and not even nearly so.

[Semmel]

From my unserstanding, the discussion about titanium or not is misleading. It doesn't matter what material the check valve was. The route cause was the leak into the helium line. Even without combustion, the check valve would have ruptured by the force of n2o4 slamming into it. You would have had a bad day, no matter what material was there. It just would have been a different bad day.

[tdperk]

From my unserstanding, the discussion about titanium or not is misleading. It doesn't matter what material the check valve was. The route cause was the leak into the helium line. Even without combustion, the check valve would have ruptured by the force of n2o4 slamming into it. You would have had a bad day, no matter what material was there. It just would have been a different bad day.

Agreed if there was a design error, it was in the GSE and associated procedures.

[HVM]

Tanks have bladders for freefall functionality(?), so how NTO leaked to the helium side of the tank?

[su27k]

And here's some Titanium valves that ArianeGroup is selling that is compatible with nitrogen tetroxide: http://www.space-propulsion.com/brochures/valves/space-propulsion-valves.pdf

I believe MON is nitrogen tetroxide with some nitric oxide added.

"titanium is well known to have poor oxidizer compatibility and be easy to set on fire."? Better tell ArianeGroup that...

[envy887]

And when you "read the memo" that says the ignition does not propagate, when you observe such systems are commonly used in spacecraft, why doesn't the flag go back down?

" In all seriousness, this is why you need senior M&P engineers with lots of experience as well as well defined processes that, in addition to explaining what to do and what not to do, give clear rationale. "

That seems like a statement made with justification only in hindsight.

Because I don't want ignition anywhere near my propellant storage or delivery infrastructure, regardless of whether it propagated in lab tests.

I make the M&P comment because all three major anomalies SpaceX has experienced have had material compatibility issues in the failure chain.

 -- Titanium + NTO + shock loading
 -- Carbon fiber immersed in LOX
 -- A martensitic steel casting used as a tie rod end immersed in cryos in a high vibe environment.

Hindsight can uncover patterns.

Hot high pressure NTO spraying out of a fractured stainless or inconel valve might have had the same result, since under those conditions NTO will happily oxidize many materials and I doubt all the surrounding components are any more resistant to NTO than the plumbing itself is.

[envy887]

Tanks have bladders for freefall functionality(?), so how NTO leaked to the helium side of the tank?

SD's don't fire in microgee, but the regular Dracos do and they are supposed to use the same tanks, so this it a good question.

[LouScheffer]

A question for propulsion experts:

Lots of spacecraft have used check valves to prevent upstream migration of NTO.  (Here's an an example from Galileo, that in turn refers to an earlier example from Viking.  "Check valves were provided to minimize MMH and NTO vapor migration upstream of the propellant tanks."

And for as long as they have been used, these valves have been known to leak (note the above sentence says minimize, not eliminate).  Presumably, once pressurized, all these other systems also pushed NTO through the check valve, too.   But this alone never seems to have caused a serious problem - it was only when MMH and NTO met upstream of valve.

So why did this cause a problems when similar systems did not?   I can think of four hypotheses:
   - Old check valves did not use titanium.
   - They did use titanium.  No problem occurs at 20 bar, but the reaction is much worse at 160 bar.
   - More NTO was leaked (by whatever mechanism) than in previous examples.
   - Previous accident reports missed this problem.  The accident reports I have seen tie themselves in knots figuring out how MMH and NTO could meet in the pressurization lines.  Maybe there was in fact no MMH leak, and the this particular problem has been happening all along, but was wrongly thought to be due to MMH and NTO mixing.