SpaceX’s Crew Dragon spacecraft static fire anomaly - THREAD 3

[tdperk]

just a quick question:

why is it the assumption that the NTO in the helium tubing was in a liquid state?  at the temps that day, NTO would be gaseous, unless the tubing was only a little bit less pressurized than the NTO tank was.  the tubing would have had to have been less pressurized in order for the NTO to leak into it.

also, am i correct that the same H2 system is used to pressurize the dracos and the superdracos?  if so, there's a bunch of other actuated valves in the system that might have failed (i think the NTO for the superdracos is stored at a higher pressure and so is separate from that used in the dracos, but i wouldn't put it past spacex to provide for a system to divert NTO from the superdraco system into the draco, in case of emergency).

anyway, great discussion.  i wonder if the "leaking component" was made to leak by the vibration testing going on.  i could see that vibration causing "a" check valve to leak back into the helium tubing.

my worry is that the burst disk changes the timing of the superdraco triggers, possibly contaminates the fuel downstream, and probably requires additional redundancy (over the relief valve system) since it can't be non-destructively checked.  it's not hard to engineer, but it could be costly in terms of weight and reliability testing time.  plus, those disks, to be reliable against all their failure modes, are expensive AND you have to rely on the manufacturer to get all that correct each time.  that's a problem that spacx doesn't like to incorporate into their designs.  a simple check valve can be tested in-situ and is much, much easier to manufacture reliably, even at these pressures and with these gasses.

if it were me, i'd be attempting to make the check valve design work with different materials and/or some reliable way to ensure no leaking NTO in the space where it ended up this time.

The steady state for NTO might be vapor, that is pressure dependent.  The helium dumping into the tubing would certainly have forced a slug of NTO into liquidity ahead of it, I am sure the helium is admitted quickly enough it has a Joule-Thompson throttling effect going into the piping.  The He2 is going into the piping faster than the speed of sound in any NTO vapor present.

I'll bet 100 bucks over it.

[tdperk]

especially at low pressure, are not quite sealing as well as they’re supposed to in check valves.”

Let's please keep mind he surely meant low-pressure differential across the valve, although that is not how he phrased it.

Not necessarily so. He said that the leak happened during ground handling, and there are several ground handling ops that can occur with low internal pressure and some NTO in the tanks/lines. See my post above for a list of ground handling operations that occur at low pressure, during which NTO could have leaked through the valve.

Err, low pressure during ground handling would be low pressure across the valve.

[Kabloona]

especially at low pressure, are not quite sealing as well as they’re supposed to in check valves.”

Let's please keep mind he surely meant low-pressure differential across the valve, although that is not how he phrased it.

Not necessarily so. He said that the leak happened during ground handling, and there are several ground handling ops that can occur with low internal pressure and some NTO in the tanks/lines. See my post above for a list of ground handling operations that occur at low pressure, during which NTO could have leaked through the valve.

Err, low pressure during ground handling would low pressure across the valve.

Yes.

I thought you were implying the leak may have happened with high pressure on both sides of the check valve, but low differential pressure across the valve, ie both sides at essentially the same high pressure.

OTOH, there are a number of ground operations that might have been done with the system at low pressure, on both sides of the check valve, during which the leak may have occurred, and the way I read it, Hans was referring to ground operations during which the system was basically unpressurized. That's how stuck check valves stay open, because there's no pressure on the NTO side to force it back closed.

[rsdavis9]

I know wikipedia is not that reliable but it does reference impact and oxidizers.
https://en.wikipedia.org/wiki/Titanium#Precautions

I can't access the references but:

Basically
1. powdered or shavings in heated air can explode.
2. In LOX impact can ignite.

Sort of similar to magnesium which burns hotter with CO2.
Mg Al and Ti owe their corrosion resistance to a tough oxide layer. Remove the oxide and they are reactive.

[jig]

just a quick question:

why is it the assumption that the NTO in the helium tubing was in a liquid state?  at the temps that day, NTO would be gaseous, unless the tubing was only a little bit less pressurized than the NTO tank was.  the tubing would have had to have been less pressurized in order for the NTO to leak into it.

also, am i correct that the same H2 system is used to pressurize the dracos and the superdracos?  if so, there's a bunch of other actuated valves in the system that might have failed (i think the NTO for the superdracos is stored at a higher pressure and so is separate from that used in the dracos, but i wouldn't put it past spacex to provide for a system to divert NTO from the superdraco system into the draco, in case of emergency).

anyway, great discussion.  i wonder if the "leaking component" was made to leak by the vibration testing going on.  i could see that vibration causing "a" check valve to leak back into the helium tubing.

my worry is that the burst disk changes the timing of the superdraco triggers, possibly contaminates the fuel downstream, and probably requires additional redundancy (over the relief valve system) since it can't be non-destructively checked.  it's not hard to engineer, but it could be costly in terms of weight and reliability testing time.  plus, those disks, to be reliable against all their failure modes, are expensive AND you have to rely on the manufacturer to get all that correct each time.  that's a problem that spacx doesn't like to incorporate into their designs.  a simple check valve can be tested in-situ and is much, much easier to manufacture reliably, even at these pressures and with these gasses.

if it were me, i'd be attempting to make the check valve design work with different materials and/or some reliable way to ensure no leaking NTO in the space where it ended up this time.

The steady state for NTO might be vapor, that is pressure dependent.  The helium dumping into the tubing would certainly have forced a slug of NTO into liquidity ahead of it, I am sure the helium is admitted quickly enough it has a Joule-Thompson throttling effect going into the piping.  The He2 is going into the piping faster than the speed of sound in any NTO vapor present.

I'll bet 100 bucks over it.

gentleman's bet.  but, there's a lag when attempting to flash condense a gas into a liquid simply by applying an abrupt pressure increase (some of the energy has to go into the phase change itself, and it can't happen super quickly) .  the He might cool initially when going through the valve, but the valve would be instant full-on, and my fairly extensive experience with helium being released from a 2000psi tank into both an empty 500 tank manifold and atmosphere, at a high rate of flow (full open, 1/4in tubing) tells me that the cooling caused by the He pulse wouldn't be enough to affect the phase change of the n2o4 on the time scales we're talking about.

i'm fairly certain that if the n2o4 was gas when the helium was released, it was still gas when it hit the relief valve.  "slug" can refer to either liquid or gas - it's a generic term for a spatially localized volume of gas or liquid, so i don't think we can reliably infer he meant liquid when using that term.

if the n2o4 was liquid in the helium tubing, then the question is how could it leak into the tubing but be kept at high enough pressure that it stayed liquid.  one explanation would be that they purged the helium line enough to get a leak into the tubing from a relatively high pressure n2o4 tank, but then re-pressurized the helium line to just below the relief valve pressure to shorten the response time associated with starting the superdracos.  given enough time at that higher lead-in pressure, i could see the leaked n2o4 collecting as a liquid in some unfortunate spot in the helium line, ready to liquid-hammer the check valve when the pressure pulsed.  i'd still like a more detailed description.

anyway, it's fun to speculate.

also, when i wrote H2 above i meant He.  Not sure why I wrote H2 other than boom effect.

[Robotbeat]

Additionally, the SuperDraco thrusters recovered from the test site remained intact, underscoring their reliability.

...so nobody's gonna point out how ridiculous this line is?

Not at all. It rules out the SDs as a potential root cause.

Agree, this is an attempt to emphasize the fact that the SDs were not involved in the mishap.

But it comes off ham-fisted. Pointing out the engines are intact when the rest of the capsule was destroyed? It sounds like something Elon might say at a press conference. (theory: he learned the SDs were found intact and demanded this detail be included)

Good intention, sub-optimal communication.

Much less of the capsule was destroyed than the leaked video would suggest. Several of the pressurized propellant tanks remained intact, pressurized and carrying propellant, and had to be dealt with in the weeks following the incident. Other sturdy components, not fully destroyed by the explosions and later recovered, included electronics boxes, substantial parts of the pressure vessel, both hatches and many other items. The capsule was destroyed to the point that it was no longer a coherent spacecraft. But there were many major pieces remaining.
The DM-1 spacecraft was not destroyed into tiny little pieces. To do so would have required much more energy than what was released during the incident.

A simple site photo of the aftermath would definitely clarify this and put paid to the 'blown to smithereens' argument.  Hopefully something like that will be part of any final public report.

Sounds like a terrible idea, even if "intact." It'd be all over the press, evoking an emotional response. In other words, pretty dumb from a PR perspective.

[Okie_Steve]

Given that there was not a tremendously long between Draco tests and (ATTEMPTED) Super Draco tests I wonder if the ground handling issue happened earlier and was lurking in the high pressure He plumbing for a while.

[tdperk]

Looking for a psychrometric chart for NTO.

Anyone have such?

EDIT:  Please nevermind.  I looked for "N2O4" & "properties" and found it.

[koraldon]

Seems whoever wrote the PR has zero to little knowledge of hypergolics and mainly tried to get good press and praise for spacex

A few notes:
- Getting NTO into the pressurant line is very bad, regardless of the final failure mode. Corrosion / impact and so on.

- It was not a surprise nor advancing the state of the art to get the explosion, just PR speak, not engineering.

- Putting only a check valve between the propellant and pressurant is a surprising engineering decision. Common practice is to put both an isolation valve (pyro/latch/burst) for ground operation + priming and a check valve to mitigate propellant transfer in space.
I assume the spacex engineer either didn’t know or just assumed everyone else was conservative...

- SpaceX guys know that NTO leaks happens through check valves. One of their hypergolic processing engineers told me stories of his days in the shuttle program and all the NTO leaks they got through check valves.
So PR just have no clue and it seems that the system designer on SpaceX didn’t talk to their own hypergolic engineers in the Cape.

- For the leak to occur during processing means they probably had reverse pressure on the check valves. This should have been monitored and detected during ground ops.

- Replacing the check valve with a burst disc sounds strange - They have different roles.
It means that the NTO and MMH will mix after priming, unless they have separate  pressure sources, which will surprise me.

- Using a burst disc by itself is also strange - a pyro valve is much more robust but also a one shot device, so a latch valve seems like a better solution. Unless they don’t have the avionics interfaces for active isolation valves, so they have to use a burst disc.

- Replacing a burst disc is not simple, as hypergolic require welded connections - it basically means cutting out the section. Of course, as an abort system I assume reuse is not a major concern, assuming the plumbing for the super Draco is separate from the Draco.

[envy887]

...
- Putting only a check valve between the propellant and pressurant is a surprising engineering decision. Common practice is to put both an isolation valve (pyro/latch/burst) for ground operation + priming and a check valve to mitigate propellant transfer in space.
I assume the spacex engineer either didn’t know or just assumed everyone else was conservative...
...

Probably a bad assumption. An oxidizer isolation valve upstream of the check valves is standard per the OMS design (and SpaceX surely has one) but doesn't help prevent this failure mode. Any NTO that leaks past the check valve can still sit downsteam of the isolation valve and get hammered back into the check valve when pressurant flow starts.

[Cherokee43v6]

I realize Hans is speaking theoretically here, but he's talking about the possibility of a one-cup volume of NTO leakage. That's quite a (theoretical) leak.

Contained inside a 1-inch inside diameter tube, for example, 1 cup of liquid would form a slug about 18 inches long. Inside a 1/2-inch ID tube, it becomes 73 inches long. Either way, that is a considerable water hammer at high pressure.

These two numbers seem awful high.  A cup is 8 ounces.  The standard coffee mug holds about 12 ounces and is roughly 2 1/2 - 3 inches in diameter and 3 1/2 - 4 inches tall. Somehow I cannot see reducing that diameter to 1/2 inch making it over 6 feet tall.

[LouScheffer]

I realize Hans is speaking theoretically here, but he's talking about the possibility of a one-cup volume of NTO leakage. That's quite a (theoretical) leak.

Contained inside a 1-inch inside diameter tube, for example, 1 cup of liquid would form a slug about 18 inches long. Inside a 1/2-inch ID tube, it becomes 73 inches long. Either way, that is a considerable water hammer at high pressure.

These two numbers seem awful high.  A cup is 8 ounces.  The standard coffee mug holds about 12 ounces and is roughly 2 1/2 - 3 inches in diameter and 3 1/2 - 4 inches tall. Somehow I cannot see reducing that diameter to 1/2 inch making it over 6 feet tall.

No, seems about right.   Reducing the diameter from 2.5 inch to 0.5 inch decreases the area by 25x (5^2).  To get the same volume, you now need 25 times the height.  3 inches x 25 = 75 inches, or about 6 feet.

[Kabloona]

- Putting only a check valve between the propellant and pressurant is a surprising engineering decision. Common practice is to put both an isolation valve (pyro/latch/burst) for ground operation + priming and a check valve to mitigate propellant transfer in space.
I assume the spacex engineer either didn’t know or just assumed everyone else was conservative...

No doubt there's an iso valve somewhere between the NTO tank and the He tank. The question is where. If it was downstream of the check valve, it was obviously open both at the time of the leak and the time of the accident. If it was upstream of the check valve, it was irrelevant.

- For the leak to occur during processing means they probably had reverse pressure on the check valves. This should have been monitored and detected during ground ops.

Not necessarily. Once the check valve is stuck open, ground handling in an off-vertical orientation could allow residual NTO to leak through the valve just by gravity, if the capsule is tilted enough in the right (wrong) direction.

- Using a burst disc by itself is also strange - a pyro valve is much more robust but also a one shot device, so a latch valve seems like a better solution. Unless they don’t have the avionics interfaces for active isolation valves, so they have to use a burst disc.

Actually, a burst disc is a beautifully simple KISS solution. Any kind of electrically-actuated/initiated valve is a PITA, requiring extensive avionics with safeties, and now you've introduced a whole 'nother layer of complexity and failure modes, in a system (abort) where you want the absolute minimum number of failure modes.

 A burst disc is dirt-simple and virtually foolproof. Open the He iso valve and the disc is 100% guaranteed to burst, assuming you design with sufficient margins.

The two downsides to burst disks are potential leakage around the flanges, which you now have to check with each installation, and the fact that you have to replace the disk every time.

Other than that, I love the burst disk plan.

- Replacing the check valve with a burst disc sounds strange - They have different roles.
It means that the NTO and MMH will mix after priming, unless they have separate  pressure sources, which will surprise me.

Not necessarily. They may in fact have separate pressure sources, or they may have separate pressure lines coming off one He tank, each with its own regulator going to the respective NTO and MMH tanks. Either way makes it virtually impossible for NTO/MMH mixing upstream.

Many of the people who designed and/or reviewed this biprop system will know about the Mars Observer (possible) failure mode of NTO getting through a check valve and mixing upstream with MMH, so I would be greatly surprised if the upstream pressurant plumbing wasn't designed specifically to eliminate such a possibility, even if NTO or MMH did leak past its first check valve.

[koraldon]

...
- Putting only a check valve between the propellant and pressurant is a surprising engineering decision. Common practice is to put both an isolation valve (pyro/latch/burst) for ground operation + priming and a check valve to mitigate propellant transfer in space.
I assume the spacex engineer either didn’t know or just assumed everyone else was conservative...
...

Probably a bad assumption. An oxidizer isolation valve upstream of the check valves is standard per the OMS design (and SpaceX surely has one) but doesn't help prevent this failure mode. Any NTO that leaks past the check valve can still sit downsteam of the isolation valve and get hammered back into the check valve when pressurant flow starts.

Yep,
It seems that this involves both a bad call by spacex system designers and an issue in the ground processing.
I doubt we will ever get the root cause for this, only the outcome, so it is hard to pinpoint.

The scenario you describe sound highly unlikely, as a short tube between isolation valve and check valve will not cause a significant priming effect, especially as the diameter will be large since this a high flow system.
Unless if for some strange reason spacex used a long tube between those elements, which I find hard to believe.

Anyway, not sure on what did you base that they have an isolation valve upstream - did you work on the system or have it’s schematics?

p.s. generally speaking the initial
Pressure in the contaminated section will be relatively high, due to the high NTO vapor pressure. Hard to tell if it contributed or mitigated the phenomena without knowing the actual values.

[koraldon]

- For the leak to occur during processing means they probably had reverse pressure on the check valves. This should have been monitored and detected during ground ops.

Not necessarily. Once the check valve is stuck open, ground handling in an off-vertical orientation could allow residual NTO to leak through the valve just by gravity, if the capsule is tilted enough in the right (wrong) direction.

- Using a burst disc by itself is also strange - a pyro valve is much more robust but also a one shot device, so a latch valve seems like a better solution. Unless they don’t have the avionics interfaces for active isolation valves, so they have to use a burst disc.

Actually, a burst disc is a beautifully simple KISS solution. Any kind of electrically-actuated/initiated valve is a PITA, requiring extensive avionics with safeties, and now you've introduced a whole 'nother layer of complexity and failure modes, in a system (abort) where you want the absolute minimum number of failure modes.

 A burst disc is dirt-simple and virtually foolproof. Open the He iso valve and the disc is 100% guaranteed to burst, assuming you design with sufficient margins.

The two downsides to burst disks are potential leakage around the flanges, which you now have to check with each installation, and the fact that you have to replace the disk every time.

Other than that, I love the burst disk plan.

A) SpaceX pointing at the ground processing as the reasons for the NTO leak.
Getting a check-valve stuck is rare failure - leak is commonplace and exactly why I wrote it seems a strange design call not to accommodate that failure mode.

B) what leak around the flanges? Have you ever designed / tested a burst disc?
A burst disc is a pain in the ass since it is very sensitive to ground operations.
Avionics for latch valves / pyro were complex in the 60’s, this is why those times burst discs were used.
In addition burst disc create contamination downstream. Another PITA.

[Kabloona]

B) what leak around the flanges? Have you ever designed / tested a burst disc?

I spent over a year running high-pressure solid propellant combustion experiments in a shock tube, using burst disks to trigger the shock tube. So, yes.

As you know, replacing the burst disk requires that you have a bolted interface with O-rings, instead of welded butt joints like you have with check valves.

So now, by replacing a sealed weld joint with a bolted interface, you've introduced the possibility of leakage past the O-rings in the bolted flanges. (I'm talking about leakage of the fluid to the outside of the flange, not leakage past the burst disk.)

And yes, there are other potential issues. I was using basic metal burst disks back in  the late '80's, and fragmentation was a problem. But the Google says there are now non-fragmenting disks, though I don't have experience with those. I'm assuming SpaceX is smart enough to use non-fragmenting disks and/or make accommodations for fragment capture.

Look, we both know there's no perfect solution. Check valves leak. Pyro valves are a PITA. Latch valves are also another layer of complexity.

Burst disks aren't perfect either, but they have the virtue of absolute simplicity and are virtually guaranteed to work when you need to abort quickly.

I cursed burst disks plenty during late nights in the lab when I screwed up and blew the burst disk backwards towards the wrong end of the shock tube. But if I were sitting in the Dragon when the sh*t hits the fan, I'd rather have a burst disk saving me than an electrically-actuated/initiated valve of any sort.

[mn]

Regarding burst disk fragments: Would that be a problem considering the SuperDraco design?

[envy887]

...
- Putting only a check valve between the propellant and pressurant is a surprising engineering decision. Common practice is to put both an isolation valve (pyro/latch/burst) for ground operation + priming and a check valve to mitigate propellant transfer in space.
I assume the spacex engineer either didn’t know or just assumed everyone else was conservative...
...

Probably a bad assumption. An oxidizer isolation valve upstream of the check valves is standard per the OMS design (and SpaceX surely has one) but doesn't help prevent this failure mode. Any NTO that leaks past the check valve can still sit downsteam of the isolation valve and get hammered back into the check valve when pressurant flow starts.

Yep,
It seems that this involves both a bad call by spacex system designers and an issue in the ground processing.
I doubt we will ever get the root cause for this, only the outcome, so it is hard to pinpoint.

The scenario you describe sound highly unlikely, as a short tube between isolation valve and check valve will not cause a significant priming effect, especially as the diameter will be large since this a high flow system.
Unless if for some strange reason spacex used a long tube between those elements, which I find hard to believe.

Anyway, not sure on what did you base that they have an isolation valve upstream - did you work on the system or have it’s schematics?

p.s. generally speaking the initial
Pressure in the contaminated section will be relatively high, due to the high NTO vapor pressure. Hard to tell if it contributed or mitigated the phenomena without knowing the actual values.

I'm not personally familiar with the Dragon design, but the Shuttle OMS system is very similar in size and function and SpaceX undoubtedly referenced it when designing Dragon. The OMS plumbing schematic is attached.

Reading SpaceX's statement, it sounds to me like there was a ground processing issue that contributed to or caused the leak. We will need to wait for the report to be published to get a more definitive root cause, but I see to reason to expect that the addition of the burst disk will not resolve all the proximate causes and completely eliminate the failure mode.

[ZChris13]

B) what leak around the flanges? Have you ever designed / tested a burst disc?

I spent over a year running high-pressure solid propellant combustion experiments in a shock tube, using burst disks to trigger the shock tube. So, yes.

As you know, replacing the burst disk requires that you have a bolted interface with O-rings, instead of welded butt joints like you have with check valves.

Bolted interface with O-rings is simply the easiest solution to replacing sections of pipe, not the only. If you're a masochist or the design calls for it, there's no reason you can't weld it in.

[Kabloona]

B) what leak around the flanges? Have you ever designed / tested a burst disc?

I spent over a year running high-pressure solid propellant combustion experiments in a shock tube, using burst disks to trigger the shock tube. So, yes.

As you know, replacing the burst disk requires that you have a bolted interface with O-rings, instead of welded butt joints like you have with check valves.

Bolted interface with O-rings is simply the easiest solution to replacing sections of pipe, not the only. If you're a masochist or the design calls for it, there's no reason you can't weld it in.

I'm talking about the seal between the flange and the burst disk, not the seal between the flange and the pressurant line. Yes, you can/would weld the flange to the line. But you can't weld the flanges to each other, because you have to be able to take out the old burst disk and install a new one.

So you still have a bolted interface that clamps the flanges of the burst disk holder to either side of the burst disk. That interface can leak, because someone installed the burst disk incorrectly, or scratched the mating surface, or whatever.

Now every time you replace the burst disk, you have to do a leak check. I'm not saying it's a major problem, but it is now a potential leak area that you have to check every time. As opposed to a welded-in-place valve body that you check once for integrity after welding and never have to leak-check again.

Also remember that we're talking about high-pressure helium, which is extraordinarily good at finding tiny leak paths.