SpaceX’s Crew Dragon spacecraft static fire anomaly - THREAD 3

[Kabloona]

Will SpaceX ever find/announce the root cause or will SpaceX settle for the proximal cause of the anomaly and move on using the burst disks?

Since the check valve was damaged in the explosion, it may be impossible for SpaceX to know for sure why the check valve leaked, but from Hans' comments it seems they believe the valve spring got stuck in a slightly compressed position in a prior operation. That's a not-uncommon failure mode for check valves, and that may be the closest SpaceX can get to knowing what happened.

The check valves are designed with a spring to open and close as needed.

“The problem is that sometimes the spring is a little bit sticky,” Koenigsmann said. “The valve has moving parts, and so that’s why things sometimes, especially at low pressure, are not quite sealing as well as they’re supposed to in check valves.”

https://spaceflightnow.com/2019/07/15/spacex-points-to-leaky-valve-as-culprit-in-crew-dragon-test-accident/

Note that Hans points out this is exactly where check valves work worst...in low differential pressure conditions where the spring gets little help from the fluid/gas. So this is where you'd expect a stuck-spring leak to manifest itself (ie during or after filling the NTO tank) which I expect gives them added confidence that this is indeed what happened.

[vulture4]

Note that Hans points out this is exactly where check valves work worst...in low differential pressure conditions where the spring gets little help from the fluid/gas. So this is where you'd expect a stuck-spring leak to manifest itself (ie during or after filling the NTO tank) which I expect gives them added confidence that this is indeed what happened.

If the valve stuck in an open position one would assume it was previously cycled. Was the NTO tank pressurized at some point? The low spring pressure could contribute to sticking in the open position but also indicates the valve had an elastomeric seal that could have been subject to degradation and leakage. 

[DAZ]

What you’re discussing is what the root cause is?  The check valve was an aerospace designed part built with aerospace design standards with aerospace tracked materials and tested to aerospace design standards for its intended use.  All this aerospace stuff means that the part is supposed to do what it’s intended to do as long as it’s used within its intended and designed usage parameters.  So, in this case, a check valve is supposed to only work in one direction and not leak.  If it doesn’t function as it’s intended all of this aerospace stuff is supposed to allow you to figure out the likely root cause of the failure.

The root cause could be an error in the design or choice of materials, which in this case would not seem likely.

The root cause could be flawed workmanship and the quality control inspection of the part.  Which would show up as the part is used in real-world applications as this one was so that would seem unlikely.

It could be a flaw in the material and the quality control of the material.  This supposedly was what caused the 2nd stage failure when the structure broke holding down the helium tanks.  This would also seem unlikely for the same reason as above.

The park could’ve been used outside of its design specs in previous usages.  For example, if the part was subjected to a higher amplitude vibration limit and then had been used before.  Check valves can be very sensitive to external vibrations, especially if their locations happen to correspond with possible harmonic points that would amplify the vibrations at that location.  These can be very difficult to detect without tests to specifically isolate those effects.  They can also be very time-consuming and expensive.  You would essentially need to build an entire actual working system exactly as you are to use it then subject it to all possible vibration scenarios on a shake table.

Almost all possible root causes would be eliminated by removing the check valve and replacing it with a burst disk.  SpaceX might have an idea or probably does know what the actual root cause is.  But as they don’t intend to reuse the system, the simplest way to eliminate them is to eliminate all possible root causes.

[vulture4]

As far as I can tell from press reports the Super Draco NTO pressurization line check valve was a custom in-house design, i.e. it was fabricated from titanium. I agree that SpaceX made a logical design chance by replacing the valve with a burst disk as tank pressurization occurs at most once per flight. However reliability of hypergolic propellant valves has been a problem for decades and understanding the exact failure mode would aid future designs. We should learn as much as possible from every failure.

[gongora]

As far as I can tell from press reports the Super Draco NTO pressurization line check valve was a custom in-house design, i.e. it was fabricated from titanium.

Are you saying you think it's a custom in-house design because it was made from titanium?

[Star One]

As far as I can tell from press reports the Super Draco NTO pressurization line check valve was a custom in-house design, i.e. it was fabricated from titanium.

Are you saying you think it's a custom in-house design because it was made from titanium?

That’s how I read it as well?

[ChrisWilson68]

I don't think that's sufficient evidence to conclude that it was designed and/or built in-house.

[Star One]

I don't think that's sufficient evidence to conclude that it was designed and/or built in-house.

Well quite. It seemed an odd idea to draw that conclusion from the material of its construction.

[Lemurion]

As far as I can tell from press reports the Super Draco NTO pressurization line check valve was a custom in-house design, i.e. it was fabricated from titanium. I agree that SpaceX made a logical design chance by replacing the valve with a burst disk as tank pressurization occurs at most once per flight. However reliability of hypergolic propellant valves has been a problem for decades and understanding the exact failure mode would aid future designs. We should learn as much as possible from every failure.

Considering that a search for titanium check valves brings up over 10 million results, most of which are offers for sale it's hard to believe that titanium check valves in general are anything but a commodity part.

[Lar]

Considering that a search for titanium check valves brings up over 10 million results, most of which are offers for sale it's hard to believe that titanium check valves in general are anything but a commodity part.

... nod. Even "aerospace grade" ones.

[dondar]

as a amazing people but otherwise nothing special, I have heard these public statements from SpaceX:
1. Crew Dragon is designed to be reusable (specifically, that Dragon 2 has a bunch of changes that make it easier and faster to reuse than Dragon 1)
2. Crew Dragon will not be reused (I don't know if there are any qualifiers on this statement)
3. Cargo Dragon 2 has differences from Crew Dragon

I think this implies that Cargo Dragon 2 might be reused like Dragon 1 is, but I have not heard much talk about Cargo Dragon 2 at all before the CRS-18 press conference.

Dragon crew 2 mk 2018 (i.e. the real design) is not the same Dragon Crew they were pitching in 2017, not speaking about 2016 etc.
So you can safely erase anything what was said before they started the final test phase approved by NASA. Dragon crew ultimately is designed by following NASA specifications and according to their continuous engineering input and suggestions.

Just like now you can safely abandon everything which was  and is ever said about BFR until SpaceX will finalize orbital testing (hardware prototyping). Hardware physics will dictate the actual choices to be made in BFR design and the numbers "BFR"/"StarShip"/... will show.

Whatever Elon Musk is saying now about BFR is literally "the shaking of the simulation", whatever  Gwynne Shotwell is saying is commercial pitching and they repeat exactly the same procedure they did in 2012-2017 with the Dragon Crew program: transmuting something very difficult and expensive in the real deal.

About leaking valve.
The  only serious difference which separates this test from all previous SuperDracos tests is Draco test performed on the same system short time before.
The question is: did they routed superdraco and dracos He plumbing before valve or did they use separated valves for different pressures. Draco pressures are very "low" comparably to the 2500 psi of the SuperDraco pressurizing system.
The mere choice of check valves says they use separated COPVs for every subsystem...

And yes it is obvious that the valve is in-house production. We would know if it would be otherwise.
SpaceX has titanium 3D printing in house since 2011(?) and is using 3D printed titanium in-house designed valves in F9 since 2014 (see SRC-5).

[OTV Booster]

Snip ...

About leaking valve.
The  only serious difference which separates this test from all previous SuperDracos tests is Draco test performed on the same system short time before.
The question is: did they routed superdraco and dracos He plumbing before valve or did they use separated valves for different pressures. Draco pressures are very "low" comparably to the 2500 psi of the SuperDraco pressurizing system.
The mere choice of check valves says they use separated COPVs for every subsystem...

And yes it is obvious that the valve is in-house production. We would know if it would be otherwise.
SpaceX has titanium 3D printing in house since 2011(?) and is using 3D printed titanium in-house designed valves in F9 since 2014 (see SRC-5).

I don’t know either way on in house titanium valves for the F9 but I’ll take your word on it. And ISTM that in house titanium 3D makes more sense than not. BUT even if true these two facts do not make it obvious that the Dragon valves were made in house.

Do you have something else that points to this?

Phil

[dondar]

Snip ...

About leaking valve.
The  only serious difference which separates this test from all previous SuperDracos tests is Draco test performed on the same system short time before.
The question is: did they routed superdraco and dracos He plumbing before valve or did they use separated valves for different pressures. Draco pressures are very "low" comparably to the 2500 psi of the SuperDraco pressurizing system.
The mere choice of check valves says they use separated COPVs for every subsystem...

And yes it is obvious that the valve is in-house production. We would know if it would be otherwise.
SpaceX has titanium 3D printing in house since 2011(?) and is using 3D printed titanium in-house designed valves in F9 since 2014 (see SRC-5).

I don’t know either way on in house titanium valves for the F9 but I’ll take your word on it. And ISTM that in house titanium 3D makes more sense than not. BUT even if true these two facts do not make it obvious that the Dragon valves were made in house.

Do you have something else that points to this?

Phil

If I had precise information I would not be able to write here. I had my share of neck tightening dutch NDAs, and I would respect buddies who have similar (granted a bit softer) burden in USA.

There were multiple reports and a least a couple of interviews with Musk about in-house 3D titanium production starting from 2011. I put sign of question because I believe they started in 2008, though I can not find open reports connecting to that date.
since apparently it is not that trivial to find report about valves in Falcon 9 there is SpaceX news report:
https://www.spacex.com/news/2014/07/31/spacex-launches-3d-printed-part-space-creates-printed-engine-chamber-crewed
see first part.

There are tens of interviews with Musk etc. where they talk about preference of 3d printing and bringing as much possible in-house.
The main real difference is certification. It's infinitely easier and it is much cheaper to certificate components in-house.

Dragon 2 being a crew carrier and under NASA watchdogs is inevitably uptight and less "flexible" vehicle than the motley crew of Falcon 9 boosters. It would be wrong for SpaceX to rely on suppliers if they can build controllable pieces themselves for such high profile piece. So far SpaceX shows to be quite normal and very predictable engineering company. I can bet that Draco/SuperDraco plumbing is made in-house.

[FutureSpaceTourist]

https://twitter.com/stephenclark1/status/1163472902861275136

1/2 SpaceX’s Hans Keonigsmann, at AIAA Prop & Energy Forum: “We’re almost ready to tie a bow around” report on the Crew Dragon abort static fire anomaly. "It's a slow process, but overall I'm confident we'll get this over with before the next month, basically."

https://twitter.com/stephenclark1/status/1163473179488202753

2/2 Koenigsmann says the the Crew Dragon in-flight abort test is scheduled in the October/November timeframe. “Then right after that, hopefully this year, we have the Demo-2 flight coming up."

[FutureSpaceTourist]

https://twitter.com/emrekelly/status/1163474745985261574

#SpaceX's Koenigsmann at AIAA: "NTO issue (Crew Dragon incident) was a surprise." Says he feels more optimistic now than he was before.

[Lar]

Dragon 2 being a crew carrier and under NASA watchdogs is inevitably uptight and less "flexible" vehicle than the motley crew of Falcon 9 boosters. It would be wrong for SpaceX to rely on suppliers if they can build controllable pieces themselves for such high profile piece. So far SpaceX shows to be quite normal and very predictable engineering company. I can bet that Draco/SuperDraco plumbing is made in-house.

Maybe. However I don't think that having additive manufacturing that works for titanium means the valves are made (solely) that way. My understanding is that valve seats tend to need the really precise tolerances that machining provides.

[OTV Booster]

Snip ...

About leaking valve.
The  only serious difference which separates this test from all previous SuperDracos tests is Draco test performed on the same system short time before.
The question is: did they routed superdraco and dracos He plumbing before valve or did they use separated valves for different pressures. Draco pressures are very "low" comparably to the 2500 psi of the SuperDraco pressurizing system.
The mere choice of check valves says they use separated COPVs for every subsystem...

And yes it is obvious that the valve is in-house production. We would know if it would be otherwise.
SpaceX has titanium 3D printing in house since 2011(?) and is using 3D printed titanium in-house designed valves in F9 since 2014 (see SRC-5).

I don’t know either way on in house titanium valves for the F9 but I’ll take your word on it. And ISTM that in house titanium 3D makes more sense than not. BUT even if true these two facts do not make it obvious that the Dragon valves were made in house.

Do you have something else that points to this?

Phil

If I had precise information I would not be able to write here. I had my share of neck tightening dutch NDAs, and I would respect buddies who have similar (granted a bit softer) burden in USA.

There were multiple reports and a least a couple of interviews with Musk about in-house 3D titanium production starting from 2011. I put sign of question because I believe they started in 2008, though I can not find open reports connecting to that date.
since apparently it is not that trivial to find report about valves in Falcon 9 there is SpaceX news report:
https://www.spacex.com/news/2014/07/31/spacex-launches-3d-printed-part-space-creates-printed-engine-chamber-crewed
see first part.

There are tens of interviews with Musk etc. where they talk about preference of 3d printing and bringing as much possible in-house.
The main real difference is certification. It's infinitely easier and it is much cheaper to certificate components in-house.

Dragon 2 being a crew carrier and under NASA watchdogs is inevitably uptight and less "flexible" vehicle than the motley crew of Falcon 9 boosters. It would be wrong for SpaceX to rely on suppliers if they can build controllable pieces themselves for such high profile piece. So far SpaceX shows to be quite normal and very predictable engineering company. I can bet that Draco/SuperDraco plumbing is made in-house.

Warning! Pedantic Alert!

Well, that article nails down valves being Done 3D and it nails down titanium in 3D, but not specifically titanium 3D valves.

The main oxidizer valve sounds like something that would be a custom build and definitely not COTS. It was pointed out that titanium valves, presumably including check valves, are widely available. After the Amos debacle I can understand shying away from outside vendors so its kogical that they would do the valve in house and probably 3D.

BUT there logic in many things that are not true. I guess my objection is using the word ‘obvious’ because while it makes sense, it isn’t obvious.

Peace bro

Phil

[OTV Booster]

Dragon 2 being a crew carrier and under NASA watchdogs is inevitably uptight and less "flexible" vehicle than the motley crew of Falcon 9 boosters. It would be wrong for SpaceX to rely on suppliers if they can build controllable pieces themselves for such high profile piece. So far SpaceX shows to be quite normal and very predictable engineering company. I can bet that Draco/SuperDraco plumbing is made in-house.

Maybe. However I don't think that having additive manufacturing that works for titanium means the valves are made (solely) that way. My understanding is that valve seats tend to need the really precise tolerances that machining provides.

The tightest tolerances on a valve seat are done by grinding the seat and valving object together like a glass stopper in lab glassware. Grinding will give much higher precision than cutting. This is how the valves in an engine are done.

There might be objection to the possibility of residue embedded but I can think of some admittedly theoretical ways to get past that.

Additive mgfr might create too open a matrix for a good sealing surface. A seat insert will get around this at the cost of one more thing to go wrong but at very low probability if done correctly.

Is SX decides they want to 3D a valve they’ll figure out some way to make it work. I just don’t know if they did that or not.

Phil

[Nomadd]

Dragon 2 being a crew carrier and under NASA watchdogs is inevitably uptight and less "flexible" vehicle than the motley crew of Falcon 9 boosters. It would be wrong for SpaceX to rely on suppliers if they can build controllable pieces themselves for such high profile piece. So far SpaceX shows to be quite normal and very predictable engineering company. I can bet that Draco/SuperDraco plumbing is made in-house.

Maybe. However I don't think that having additive manufacturing that works for titanium means the valves are made (solely) that way. My understanding is that valve seats tend to need the really precise tolerances that machining provides.

The tightest tolerances on a valve seat are done by grinding the seat and valving object together like a glass stopper in lab glassware. Grinding will give much higher precision than cutting. This is how the valves in an engine are done.

There might be objection to the possibility of residue embedded but I can think of some admittedly theoretical ways to get past that.

Additive mgfr might create too open a matrix for a good sealing surface. A seat insert will get around this at the cost of one more thing to go wrong but at very low probability if done correctly.

Is SX decides they want to 3D a valve they’ll figure out some way to make it work. I just don’t know if they did that or not.

Phil

Not everybody is a fan of lapping valves. If the valves and seats expand slightly differently, you can have problems. Perfection isn't always good if it costs you margin.

[goretexguy]

At the risk of getting off-topic, why use metal valve seats as opposed to ceramic disk or ball valves?