SpaceX’s Crew Dragon spacecraft static fire anomaly - THREAD 3

[gongora]

https://spaceflightnow.com/2019/07/15/spacex-points-to-leaky-valve-as-culprit-in-crew-dragon-test-accident/

“The burst disk we have now is definitely the safer approach overall going forward,” Koenigsmann said. “We didn’t really expect this to be a problem prior to that (accident), but that’s what you learn when you test. You find out some components might be better off exchanged with other components.”

The check valves are designed with a spring to open and close as needed.

“The problem is that sometimes the spring is a little bit sticky,” Koenigsmann said. “The valve has moving parts, and so that’s why things sometimes, especially at low pressure, are not quite sealing as well as they’re supposed to in check valves.”

[Kabloona]

I realize Hans is speaking theoretically here, but he's talking about the possibility of a one-cup volume of NTO leakage. That's quite a (theoretical) leak.

Contained inside a 1-inch inside diameter tube, for example, 1 cup of liquid would form a slug about 18 inches long. Inside a 1/2-inch ID tube, it becomes 73 inches long. Either way, that is a considerable water hammer at high pressure.

“If you have a propellant tank, and you fill that tank, and you do have a check valve, it’s conceivable that the check valve leaks backwards … and you push propellant into the pressurization system,” Koenigsmann said. “The amount might be a cup or something like that, or more than a cup, it depends on how the system is being built up. And then it’s there for a while after loading, and when you pressurize you basically open the valves really, really fast.”

 (quote from article linked above by gongora.)

So it's confirmed that it was, in fact, the check valve itself that leaked, not some other component.

This is exactly why check valves are often installed in redundant pairs. But either Dragon didn't have redundant check valves, or the redundant valves were installed far enough apart (in series) to allow a slug of (maybe, theoretically,) one cup or so in volume to form in between, with enough empty volume in the line downstream of the slug to allow it to accelerate sufficiently to do damage upon pressurization.

[cscott]

*A* check valve leaked.  The one which leaked is still not necessarily the one which exploded.  The leaking valve occurred during "ground processing".

There are lots of valves and check valves in and around rockets.

[mlindner]

Good breakdown of what occurred with diagrams:

[ChrisWilson68]

*A* check valve leaked.  The one which leaked is still not necessarily the one which exploded.  The leaking valve occurred during "ground processing".

There are lots of valves and check valves in and around rockets.

Part of the fix is to replace a check valve with a burst disk.  That implies that the check valve being replaced by the burst disk was responsible for the leak.

[su27k]

Good breakdown of what occurred with diagrams:

He quoted the Mars Observer Failure Investigation Report. A causal browsing shows the Mars Observer used a lot of Titanium in its propulsion systems, including pyro valves, service valves, and yes, check valves, all have flight heritage.

Looks like the idea that Titanium cannot be used in MMH/NTO propulsion system is hearsay.

[ChrisWilson68]

I'd say that the multiple, detailed documents su27k has posted make it clear that the idea that Titanium is well-known to be a bad idea to use with oxidizers and competent, knowledgeable engineers wouldn't have used it is complete nonsense.

Even without these documents, it's pretty obviously a bogus claim.  If SpaceX were so incompetent as to miss something obvious, they wouldn't have been able to have all the successes they have had.  And NASA did an extensive technical review of all Dragon 2 systems.  It's just very, very difficult to believe that NASA would miss something as obvious as this was claimed to be when they're reviewing a system they're planning to put their own astronauts on.

[jig]

just a quick question:

why is it the assumption that the NTO in the helium tubing was in a liquid state?  at the temps that day, NTO would be gaseous, unless the tubing was only a little bit less pressurized than the NTO tank was.  the tubing would have had to have been less pressurized in order for the NTO to leak into it.

also, am i correct that the same H2 system is used to pressurize the dracos and the superdracos?  if so, there's a bunch of other actuated valves in the system that might have failed (i think the NTO for the superdracos is stored at a higher pressure and so is separate from that used in the dracos, but i wouldn't put it past spacex to provide for a system to divert NTO from the superdraco system into the draco, in case of emergency).

anyway, great discussion.  i wonder if the "leaking component" was made to leak by the vibration testing going on.  i could see that vibration causing "a" check valve to leak back into the helium tubing.

my worry is that the burst disk changes the timing of the superdraco triggers, possibly contaminates the fuel downstream, and probably requires additional redundancy (over the relief valve system) since it can't be non-destructively checked.  it's not hard to engineer, but it could be costly in terms of weight and reliability testing time.  plus, those disks, to be reliable against all their failure modes, are expensive AND you have to rely on the manufacturer to get all that correct each time.  that's a problem that spacx doesn't like to incorporate into their designs.  a simple check valve can be tested in-situ and is much, much easier to manufacture reliably, even at these pressures and with these gasses.

if it were me, i'd be attempting to make the check valve design work with different materials and/or some reliable way to ensure no leaking NTO in the space where it ended up this time.

[Kabloona]

just a quick question:

why is it the assumption that the NTO in the helium tubing was in a liquid state?  at the temps that day, NTO would be gaseous, unless the tubing was only a little bit less pressurized than the NTO tank was.  the tubing would have had to have been less pressurized in order for the NTO to leak into it.

Because SpaceX called it a "slug." Slugs are liquid, not gaseous. Liquid slugs produce the type of water-hammer effect SpaceX described. Vapor does not.

[whitelancer64]

just a quick question:

*snip*

also, am i correct that the same H2 system is used to pressurize the dracos and the superdracos?  if so, there's a bunch of other actuated valves in the system that might have failed (i think the NTO for the superdracos is stored at a higher pressure and so is separate from that used in the dracos, but i wouldn't put it past spacex to provide for a system to divert NTO from the superdraco system into the draco, in case of emergency).

anyway, great discussion.  i wonder if the "leaking component" was made to leak by the vibration testing going on.  i could see that vibration causing "a" check valve to leak back into the helium tubing.

my worry is that the burst disk changes the timing of the superdraco triggers, possibly contaminates the fuel downstream, and probably requires additional redundancy (over the relief valve system) since it can't be non-destructively checked.  it's not hard to engineer, but it could be costly in terms of weight and reliability testing time.  plus, those disks, to be reliable against all their failure modes, are expensive AND you have to rely on the manufacturer to get all that correct each time.  that's a problem that spacx doesn't like to incorporate into their designs.  a simple check valve can be tested in-situ and is much, much easier to manufacture reliably, even at these pressures and with these gasses.

if it were me, i'd be attempting to make the check valve design work with different materials and/or some reliable way to ensure no leaking NTO in the space where it ended up this time.

The Dracos and SuperDracos share the same fuel / oxidizer tanks. The high pressure for the SuperDracos comes from pressuring the tanks with lots of helium. I'm not sure if there is a separate high pressure helium system that is only activated when firing the SuperDracos. My guess is not, that they just turn the helium valves wide open. I'm sure someone will correct or confirm this.

[Kabloona]

*A* check valve leaked.  The one which leaked is still not necessarily the one which exploded.  The leaking valve occurred during "ground processing".

There are lots of valves and check valves in and around rockets.

Hans said the slug was driven "back" into the check valve. Which implies the slug initially leaked "forward" through the check valve, then was driven "back" in the opposite direction...into the same check valve.

SpaceX accomplished a successful test of the lower-pressure Draco thrusters before pressing on to a SuperDraco hot fire test April 20. As the abort system pressurized, roughly 100 milliseconds before the SuperDraco engines were set to ignite, “we think that this slug (of nitrogen tetroxide) was driven back into the check valve,” Koenigsmann said Monday.

“Imagine a lot of pressure driving back a slug of liquid (that) has significant force, and that basically destroyed the check valve and caused an explosion,” Koenigsmann said.

He used the word "back" in a directional sense twice (my bolding). Not a coincidence, IMO. You don't get "back" into your car unless you just got out of your car.

[whitelancer64]

*snip*
   - Previous accident reports missed this problem.  The accident reports I have seen tie themselves in knots figuring out how MMH and NTO could meet in the pressurization lines.  Maybe there was in fact no MMH leak, and the this particular problem has been happening all along, but was wrongly thought to be due to MMH and NTO mixing.

Should be fairly easy to prove if that's the case, by demonstrating via experiments that even a mild interaction of NTO (liquid or vapor) with titanium will produce combustion.

[Okie_Steve]

I'm curious about the phrase "ground handling" and how that contributed to off-nominal.

[cscott]

For the record, I believe in closely parsing written statements. I don't believe in closely parsing every word of Hans' *oral* answers, especially when he is trying to "dumb it down" for the press.

I expect/hope there will be a NASA report at some point, I'm content to wait for that to settle the matter.

[Spindog]

The burst disk plan is simple and should be reliable. Maybe in the future they'll look at something like keeping low helium pressure in the pressurant lines or perhaps adding an additional valve to prevent backflow in order to retain the multiple firing capability they had.

[Lar]

This: it's in a DTIC memo from the 60s (page 9).  I don't know why it's "unexpected" for titanium to ignite when NTO is rammed into it at high pressure.

Not related to this thread, but this little memo made me pause for an instant and awe to the myriad of knowledge generated in the 60s for the Mercury, Gemini and Apollo missions. How many more memos like that are out there forgotten by the current professionals dealing with the same questions and that will provoke "unexpected" issues  in the future?

Back to thread

Seems like the perfect kind of things to have all scanned and fed into an AI?

There's a certain type of person who, when faced with a problem, will say, "I know, l'll use AI." Now they have two problems.

In all seriousness, this is why you need senior M&P engineers with lots of experience as well as well defined processes that, in addition to explaining what to do and what not to do, give clear rationale.

This is the type of problem where SpaceX's habit of maintaining a young workforce by chewing through and burning out its workers puts it at a disadvantage.

Feeding a large corpus of knowledge into Watson or a similar system has had good results in other domains. There is no reason to beleive that it might not in this domain. Would it be a substitute for experience? No. But current AI is capable of coping with questions like "are there any likely adverse effects of using titanium valves with NTO" quite easily.  IF you've fed it the right corpus. Is it sufficient? No, also do your research and talk to your grayhairs. But it might well catch something worth more investigating.

Disclaimer: This is next to my wheelhouse. I've been involved in projects to feed Watson.  But I felt it necessary to correct this misconception. This sort of thing is one of the things AI actually is good at. Keyword search will not find these sorts of things easily, or will return so much that you get overwhelmed.

[Lar]

SpaceX has a lot of powerful enemies that like to latch onto anything they can to sow unjustified fear, uncertainty, and doubt.  Releasing a video of a failure just gives ammunition to that kind of unfair treatment.  I can understand why SpaceX would not want to do that, particularly at a time when they didn't have the information themselves to authoritatively refute such attacks.

Yes when rational discussion ends reach for the tinfoil hat. That’s how this post comes across.

No tinfoil, and everything I said was perfectly rational.

There's lots of SpaceX bashing through misleading statements by powerful people with a huge vested interests against SpaceX.  Just look at the public comments by certain members of Congress at the time of Zuma.  Look at all the public statements by people involved with Ariane in the early days of SpaceX.

Two points, one from each hat.
(fan) SpaceX DOES have powerful enemies. Far more than most other companies in this business. It's tinfoil-hattishness to deny this. These powerful enemies (and their allies, the concern trolls) love to twist everything around to make SpaceX look bad. Often this is to protect the turf of their competitors. It's tinfoil-hattishness to deny this as well.

Chris is correct in their statements and going back and forth to try to claim counterfactuals?  not good.

(mod) the entire subtopic is off topic for this very fast moving thread. Last post was just made,  by me, just now, on this subtopic.

[Kabloona]

I'm curious about the phrase "ground handling" and how that contributed to off-nominal.

There are a number of ways it might have happened. For discussion purposes, here's a very basic sketch of a typical propulsion tank schematic.

Normally the check valve would be placed above the 1-g vertical liquid level, so that in ground handling and on the pad you would minimize the risk of liquid leaking through the check valve.

But there are several scenarios in which NTO liquid could contact the check valve during ground ops including:

1) mistakenly overfilling the NTO tank (though I doubt this happened).
2) rotating the Dragon capsule for handling/transport after a test while some residual liquid still remains in the tank.
3) applying positive pressure through the fill/drain vent after a test in order to aid draining the NTO tank, while the capsule is tilted enough to have liquid NTO in contact with the check valve.
4) after a test, venting the He pressure in the line upstream of the tank, while the capsule is again tilted enough for liquid NTO to be in contact with the check valve, and while the NTO tank still has pressure, resulting in NTO flow back up the helium supply line.

and probably other ways I haven't thought of.

So a slug of NTO leaks through the stuck open check valve, then is driven "back into the check valve" (per Hans) when He pressure is applied.

[tdperk]

especially at low pressure, are not quite sealing as well as they’re supposed to in check valves.”

Let's please keep mind he surely meant low-pressure differential across the valve, although that is not how he phrased it.

[Kabloona]

especially at low pressure, are not quite sealing as well as they’re supposed to in check valves.”

Let's please keep mind he surely meant low-pressure differential across the valve, although that is not how he phrased it.

Not necessarily so. He said that the leak happened during ground handling, and there are several ground handling ops that can occur with low internal pressure and some NTO in the tanks/lines. See my post above for a list of ground handling operations that occur at low pressure, during which NTO could have leaked through the valve.