SpaceX’s Crew Dragon spacecraft static fire anomaly - THREAD 3

[umaurer]

It should, Titanium is used everywhere for NTO. The fact it can burn it I don't think is well known. Titanium is generally pretty non-reactive.

Interesting tweets on this topic by Ben Brockert:

https://twitter.com/wikkit/status/1150855184924336128?s=20

SpaceX update on the disappearing Dragon. (link: https://www.spacex.com/news/2019/07/15/update-flight-abort-static-fire-anomaly-investigation) spacex.com/news/2019/07/1… “It is worth noting that the reaction between titanium and NTO at high pressure was not expected” is a frightening sentence; titanium is well known to have poor oxidizer compatibility and be easy to set on fire. “resistant to N2O4 except under impact... found that Ti impacts sporadically under reasonably well controlled test conditions; the ignition freq. is increased markedly by Ti filings or glass particles on the impact surface” Compatibility of Materials w/ Rocket Prop and Oxidizers

Selective quoting perhaps.   The very next sentence (just after the quote) says "The ignitions do not spread beyond the impact area."

This is reinforced by a more detailed report about titanium reacting with N2O4, up to and including rifle bullet strikes: "Propagation of the reaction does not occur even though sufficient N2O4 is present to allow complete oxidation of the metal", and  "In no cases, however, has ignition been observed to cover more than a small fraction of the impact area as a surface fusion only". 

Most of these reports were not done under high pressure, but you'd think the 30-06 bullet impact (about 3000 joules) would create some extremely high pressure, at least temporarily.  Even in this case, "no propagation of the ignition was observed, and neither container was damaged to a greater extent than the empty impacted container."

So at least to me, nothing in this literature jumps out at me as a "Duh!  They should have seen this" moment.

I think that SpaceX still took risk using Titanium. (Maybe it was calculated, we won't know)
In an other source Titanium is explicitly forbidden in use with NTO:

Titanium must be avoided because of the impact sensitivity in the presence of a strong oxidizing agent.

(Source: 22.3.10, Sheet 91, "Materials Compatibility With Liquid Rocket Propellants", Boeing D2-113073-1, March 1970)
I can't find the pdf to link to at the moment, sorry.

I'm interested in the reason for their choice of Titanium. Why not use SS, Brass or Inconel? Is the check valve really that mass-critical?

I don't know what kinds of mass-flows are present in the system, however having worked with cryo-rated check-valves rated up to 200 bar, I don't think they are gaining much using Titanium...

[ValmirGP]

This: it's in a DTIC memo from the 60s (page 9).  I don't know why it's "unexpected" for titanium to ignite when NTO is rammed into it at high pressure.

Not related to this thread, but this little memo made me pause for an instant and awe to the myriad of knowledge generated in the 60s for the Mercury, Gemini and Apollo missions. How many more memos like that are out there forgotten by the current professionals dealing with the same questions and that will provoke "unexpected" issues  in the future?

Back to thread

[TrueBlueWitt]

This: it's in a DTIC memo from the 60s (page 9).  I don't know why it's "unexpected" for titanium to ignite when NTO is rammed into it at high pressure.

Not related to this thread, but this little memo made me pause for an instant and awe to the myriad of knowledge generated in the 60s for the Mercury, Gemini and Apollo missions. How many more memos like that are out there forgotten by the current professionals dealing with the same questions and that will provoke "unexpected" issues  in the future?

Back to thread

Seems like the perfect kind of things to have all scanned and fed into an AI?

[cscott]

It was raised before but I'll repeat it, because I think it's a good point: the "leaking" component and the titanium check valve might not be the same. (In fact, given the wording, likely they are not.). Cf the Mars Observer situation again, where the NTO that leaked past the check valve was then blown in a different direction past a different check valve into the MMH.

This might be the Occam's razor explanation for "why did SpaceX use Titanium for NTO?" Ie: they didn't; the titanium check valve (or the titanium side of the valve) was supposed to see only Helium.  But when the leaking ground side equipment let NTO in where it shouldn't be, then suddenly we had NTO+Ti+impact which, although perhaps not entirely unthought of, turned out to be a bigger problem than expected.

As usual, a belt and suspenders solution will fix both the leaking ground side process/component *and* the titanium, even though in "normal" operation the titanium part still ought never be exposed to NTO.  The SpaceX release seems to focus on the changes needed on the vehicle, emphasizing they are minor, but presumably there are many other changes to procedures, ground side equipment, etc, that are not being explicitly enumerated.

It's usual in failure investigations for there to be an "unlucky" sequence of several events, all of which had to go wrong to create the Bad Day.  My reading of the release is that leaking NTO into the Helium system was unexpected, but not necessarily sufficient; and it wasn't where the NTO went next that turned out to be a problem, but that there happened to be some Ti in its way.

[SWGlassPit]

This: it's in a DTIC memo from the 60s (page 9).  I don't know why it's "unexpected" for titanium to ignite when NTO is rammed into it at high pressure.

Not related to this thread, but this little memo made me pause for an instant and awe to the myriad of knowledge generated in the 60s for the Mercury, Gemini and Apollo missions. How many more memos like that are out there forgotten by the current professionals dealing with the same questions and that will provoke "unexpected" issues  in the future?

Back to thread

Seems like the perfect kind of things to have all scanned and fed into an AI?

There's a certain type of person who, when faced with a problem, will say, "I know, l'll use AI." Now they have two problems.

In all seriousness, this is why you need senior M&P engineers with lots of experience as well as well defined processes that, in addition to explaining what to do and what not to do, give clear rationale.

This is the type of problem where SpaceX's habit of maintaining a young workforce by chewing through and burning out its workers puts it at a disadvantage.

[envy887]

This: it's in a DTIC memo from the 60s (page 9).  I don't know why it's "unexpected" for titanium to ignite when NTO is rammed into it at high pressure.

Not related to this thread, but this little memo made me pause for an instant and awe to the myriad of knowledge generated in the 60s for the Mercury, Gemini and Apollo missions. How many more memos like that are out there forgotten by the current professionals dealing with the same questions and that will provoke "unexpected" issues  in the future?

Back to thread

Seems like the perfect kind of things to have all scanned and fed into an AI?

There's a certain type of person who, when faced with a problem, will say, "I know, l'll use AI." Now they have two problems.

In all seriousness, this is why you need senior M&P engineers with lots of experience as well as well defined processes that, in addition to explaining what to do and what not to do, give clear rationale.

This is the type of problem where SpaceX's habit of maintaining a young workforce by chewing through and burning out its workers puts it at a disadvantage.

NASA used titanium tanks to store NTO in the STS OMS pods.

https://science.ksc.nasa.gov/shuttle/technology/sts-newsref/sts-oms.html

It's not at all clear to me that an experienced engineer would have decided based solely on material incompatibilities to not use titanium components in an NTO system.

[envy887]

With a burst disk, after it has burst propellant can flow both ways. Presumably the sequence is

1) Open valve from high pressure helium tank to propellant tank.
2) Disk burst.
3) High pressure helium flows into propellant tank.

After the abort, there may be some propellant left in the tanks. The only way I can think of to stop the propellants mixing via the pressurisation lines is to close a valve while helium is still blowing through. So the system might be, showing a redundant valve configuration:

/----------------\
|High Pressure He|
\-------+--------/
        |
 +---+--+--+---+
 |   |     |   |
<-> <->   <-> <-> Valves normally closed
 |   |     |   |
 o   o     o   o  Burst disks
 |   |     |   |
 +-+-+     +-+-+
   |         |
  <|>       <|> Valves normally open
   |         |
  <|>       <|>
   |         |
/--+--\   /--+--\
| NTO |   | MMH |
\--+--/   \--+--/
   |         |
    To Engines

They could keep the check valves upstream of the burst valves, that way no fluid can get to the check valves on a nominal flight, but nothing will backflow after an abort is concluded.

[gongora]

It was raised before but I'll repeat it, because I think it's a good point: the "leaking" component and the titanium check valve might not be the same. (In fact, given the wording, likely they are not.).

Sometimes you just have to get over a poorly worded sentence.  Unless you can show some other point of entry for the propellant that leaked into the pressurant line between the helium tank and the check valve, I don't see how it's possible to conclude the check valve wasn't leaking.

[cscott]

I think we'd need access to the plumbing schematics of Crew Dragon and its ground systems to answer gongora's question with the certainty he seems to demand.

Here's a strawman. On the GSE, a high pressure helium line is used both to fill the He tanks on Dragon and to push the NTO into the Dragon tanks.  NTO from the GSE storage tanks leaks past a GSE valve back into the He supply line on the GSE side, then is forced into the Dragon He lines when the Dragon's NTO-pressurant He tank is filled.  But this isn't a problem, because the line is just a stub for filling, and the NTO just sits there in a pool at the bottom of the stub He fill line.  But in this particular case we got a water hammer effect on a Ti surface when the stub was pressurized, and that caused problems even though the stub led nowhere except out of the vehicle and was never supposed to see anything but He.

You can poke holes in this schematic design, it's just a strawman. I'm just pointing out that it's *feasible* that the GSE and vehicle side plumbing interacted in a bad way, that the vehicle-side Ti was never meant to see NTO, and that the leaking GSE valve was not the vehicle-side check valve with Ti components.

(It also seems likely that there were failure modes considered during design where after some off-nominal event (like NTO leaking past a GSE valve) the NTO might contact Ti, and the assessment based on the literature was that this might cause localized pitting/wear on the Ti in this off-nominal case and thus require inspection/replacement or even cause a leak but not initiate an explosion.  The energy of the event was the surprising thing.)

And I know I'm beating this to death, but I'm not saying the design was blameless, just pointing out the limits of our knowledge.  We shouldn't assume that the Ti was or was not designed to contact NTO, or that the leaking valve was or was not the same as the detonating one.  We don't (yet) know.

[Star One]

Additionally, the SuperDraco thrusters recovered from the test site remained intact, underscoring their reliability.

...so nobody's gonna point out how ridiculous this line is?

Not at all. It rules out the SDs as a potential root cause.

Agree, this is an attempt to emphasize the fact that the SDs were not involved in the mishap.

But it comes off ham-fisted. Pointing out the engines are intact when the rest of the capsule was destroyed? It sounds like something Elon might say at a press conference. (theory: he learned the SDs were found intact and demanded this detail be included)

Good intention, sub-optimal communication.

Much less of the capsule was destroyed than the leaked video would suggest. Several of the pressurized propellant tanks remained intact, pressurized and carrying propellant, and had to be dealt with in the weeks following the incident. Other sturdy components, not fully destroyed by the explosions and later recovered, included electronics boxes, substantial parts of the pressure vessel, both hatches and many other items. The capsule was destroyed to the point that it was no longer a coherent spacecraft. But there were many major pieces remaining.
The DM-1 spacecraft was not destroyed into tiny little pieces. To do so would have required much more energy than what was released during the incident.

Didn’t know that. Most of the reporting I read would have you believe that it was utterly destroyed.

[Kabloona]

NASA used titanium tanks to store NTO in the STS OMS pods.

https://science.ksc.nasa.gov/shuttle/technology/sts-newsref/sts-oms.html

It's not at all clear to me that an experienced engineer would have decided based solely on material incompatibilities to not use titanium components in an NTO system.

Yes. It's common practice to use titanium tanks for NTO. Look at pretty much any biprop satellite. No one at NASA or SpaceX would have thought twice about using titanium anywhere in an NTO system. That's what makes this failure mode such a unicorn.

[Cbar]

If upstream contamination of the helium pressurization system cannot be guaranteed, are there NTO check valves designed to operate at 2400 psi, so it does  not matter if it's NTO or helium (mix) on both sides of the valve?
That way you retain testabilty and reuse without replacing components.

[Cherokee43v6]

Additionally, the SuperDraco thrusters recovered from the test site remained intact, underscoring their reliability.

...so nobody's gonna point out how ridiculous this line is?

Not at all. It rules out the SDs as a potential root cause.

Agree, this is an attempt to emphasize the fact that the SDs were not involved in the mishap.

But it comes off ham-fisted. Pointing out the engines are intact when the rest of the capsule was destroyed? It sounds like something Elon might say at a press conference. (theory: he learned the SDs were found intact and demanded this detail be included)

Good intention, sub-optimal communication.

Much less of the capsule was destroyed than the leaked video would suggest. Several of the pressurized propellant tanks remained intact, pressurized and carrying propellant, and had to be dealt with in the weeks following the incident. Other sturdy components, not fully destroyed by the explosions and later recovered, included electronics boxes, substantial parts of the pressure vessel, both hatches and many other items. The capsule was destroyed to the point that it was no longer a coherent spacecraft. But there were many major pieces remaining.
The DM-1 spacecraft was not destroyed into tiny little pieces. To do so would have required much more energy than what was released during the incident.

A simple site photo of the aftermath would definitely clarify this and put paid to the 'blown to smithereens' argument.  Hopefully something like that will be part of any final public report.

[cscott]

Let's re-read that statement carefully:

Evidence shows that a leaking component allowed liquid oxidizer – nitrogen tetroxide (NTO) – to enter high-pressure helium tubes during ground processing. A slug of this NTO was driven through a helium check valve at high speed during rapid initialization of the launch escape system, resulting in structural failure within the check valve. The failure of the titanium component in a high-pressure NTO environment was sufficient to cause ignition of the check valve and led to an explosion.

There's another factor here which I think we've missed: NTO and He have very different compressibilities.  The SpaceX wording says that the "helium check valve" got pummelled with a "slug" of NTO causing "structural failure within the check valve".  So it wasn't a direct ignition, it was the fact that first a valve designed for compressible helium got incompressible NTO slammed into it and broke violently, releasing energy.  The combination of that initiating energy plus the oxider plus the exposed unoxidized titanium plus the now-failed valve (so ambient atmosphere and whatever was on the other side of the valve) caused the explosion.

But the root cause was that NTO shouldn't have been introduced into the He system "during ground processing".  And the secondary cause was throwing a liquid at a valve meant for a gas.

[mmeijeri]

It's not clear that the statement was sufficiently precisely worded to withstand such close reading. But isn't it more logical that the check valve was intended to prevent NTO from getting into the helium line (and from there eventually to the hydrazine tank, with catastrophic consequences) than to prevent helium from getting into the NTO tank prematurely?

[rakaydos]

This: it's in a DTIC memo from the 60s (page 9).  I don't know why it's "unexpected" for titanium to ignite when NTO is rammed into it at high pressure.

Not related to this thread, but this little memo made me pause for an instant and awe to the myriad of knowledge generated in the 60s for the Mercury, Gemini and Apollo missions. How many more memos like that are out there forgotten by the current professionals dealing with the same questions and that will provoke "unexpected" issues  in the future?

Back to thread

Seems like the perfect kind of things to have all scanned and fed into an AI?

There's a certain type of person who, when faced with a problem, will say, "I know, l'll use AI." Now they have two problems.

In all seriousness, this is why you need senior M&P engineers with lots of experience as well as well defined processes that, in addition to explaining what to do and what not to do, give clear rationale.

This is the type of problem where SpaceX's habit of maintaining a young workforce by chewing through and burning out its workers puts it at a disadvantage.

[Cherokee43v6]

After all of this I do have a few outstanding questions:

1) Due to the lack of clarity in the statement, what actually leaked?

2) More importantly, WHY did it leak?
  - Design flaw?
  - Manufacturing defect?
  - Environmental exposure?  (After-all, this was a flown component)

Basically, the 80% is that they now have confidence in what happened and how to prevent it... That remaining 20% is the bear, where it is needed to determine if there is anything else that could be impacted in a similar way.

[gongora]

A simple site photo of the aftermath would definitely clarify this and put paid to the 'blown to smithereens' argument.  Hopefully something like that will be part of any final public report.

When I read woods170's description I did not think it contradicted the "blown to smithereens" description at all.

[cscott]

It's not clear that the statement was sufficiently precisely worded to withstand such close reading. But isn't it more logical that the check valve was intended to prevent NTO from getting into the helium line (and from there eventually to the hydrazine tank, with catastrophic consequences) than to prevent helium from getting into the NTO tank prematurely?

Seems odd to believe that SpaceX did not carefully word their written statement.

And my experience with failures (and software and bugs of all kinds) is that it is rarely the expected thing (keep MMH and NTO from mixing via the He lines!) which fails.  It seems more likely (to me) than an unexpected failure would result from precisely something which was *not* obviously a problem.  Why does it matter if NTO gets into the He lines which are going to vent into the NTO tank anyway?  Well, there's some Ti there.  But NTO tanks are made of Ti all the time! So no big deal, even if it were to leak.  And so the unexpected happens...

[SWGlassPit]

A simple site photo of the aftermath would definitely clarify this and put paid to the 'blown to smithereens' argument.  Hopefully something like that will be part of any final public report.

When I read woods170's description I did not think it contradicted the "blown to smithereens" description at all.

Once you get to a failure that is, by pretty much any account, non-survivable, I'm not sure it makes sense to quibble over degrees beyond non-survivability.