SpaceX Falcon 9 - AMOS-6 - (Pad Failure) - DISCUSSION THREAD (2)

[Robotbeat]

What are the odds of them pointing a tiger team at developing an autogenous pressurization system for the F9/FH LOX tanks? Just to retire the cLOX v He issue once and for all. Implement as "v1.2.1"

Would require an engine redesign and reduction of performance

The RP-1 COPV's are in the RP-1 tank

Are you certain? Since the helium has to exchange heat with the engines anyway (going all the way down and back up) before going into the propellant tank, I would think they'd stick both in the LOx since you could fit like 4x as much Helium in the same tank due to the much colder temperature of LOx.

But I guess you should know.

And you would get SOME of the benefit from cooling due to the supercooling of the kerosene, I suppose.

[FinalFrontier]

It would probably depend upon whether Space X can conclusively determine the exact cause of the failure. If they can, then probably the COPVs can stay where they are. If, on the other hand, it’s a bit nebulous as to what happened or multiple issues are identified, then maybe a larger redesign is warranted — or needed to keep key customers happy — that includes relocating the COPVs. Time will tell.

I dunno. The engineer in me just doesn't like the idea of putting tanks pressurized to hundreds or thousands of pounds inside a tank that's built for 30.

Yeah, it was a cool idea but so were a lot of my FUBARs, too. Sometimes you just have to backtrack.

Honestly, what difference would it make in flight? If the pressurization system fails in flight, you're screwed anyway. Maybe in a less dramatic fashion, but so what? (The difference I would suggest: DON'T do static fires with the payload attached! Just don't!)

They're not going to put the tanks outside. There's nowhere to put them.

The suggestions here are to make essentially an entirely new stage. No way in heck that's going to happen before return to flight.

No, they'll find the root cause, hopefully find a belt-and-suspenders way of preventing it from happening again (such as double inspections, 1.5x current FoS on the tank, plus an inert sealant on the COPV, if that's the root cause), then move on.

Hopefully this will spur development of a reusable methane upper stage since it uses a different technique entirely. But SpaceX isn't going to abandon the entire design like people are suggesting.

SpaceX isn't going to abandon the entire design like people are suggesting.

What they should do actually is just focus on finishing the methane US sooner as a phased replacement. That and better quality control and testing processes once they do finish it. But in this sense they should "abandon" the design, just merely the kerolox mvac stage they have as opposed to internal tankage.

I don't see any reason the internal deep cooled design for the helium system cannot be made to work even under the rather exotic conditions they are running it at (super cold LOX and cyro helium), but it needs to be beefed up or handled more carefully in testing.

Like I said in the other thread earlier the real root cause here IS the same as CRS 7 and that is quality control. Doesn't matter if this ultimately turns out to have been the result of a design flaw (which given previous flight rationale I think is only 20% likely), it was still ultimately not caught/caused by bad quality control/inadequate testing (or perhaps excessively stressful testing or transportation) and thus ended up in the stage and causing LOV.

But again, I somehow doubt it's a design flaw. It's probably something as simple as too many cycles or too much stress, or not enough stress testing on the second stage, and/or not enough proper checkout during the pre launch flow that led to this, same as it did with the strut on CRS 7 because they simply assumed the material would work at rated stress levels instead of testing it to find out.


Either way you can eliminate both a potential design issue and inadequate procedures by re-doing everything to move to a new upper stage and something tells me this will be the direction they go in. They have done this in the past after failures, CRS 7 included (IE upgrading to FT ect, using the time to just redesign things all the way around before flying again). Now in this case I don't expect to see them move that fast and suddenly have the new US by November, but I would expect to not see many more flights on the old US/2017 Q 2-3 sometime for new. New US is tied at the hip to the engine as well however so that is an added wild card.


The short version: revise quality control, testing, and checkout procedures relating to stage 2 and revise static fire procedures or eliminate static fire altogether (or at least ban the idea of putting the payload on during testing).

[Robotbeat]

The methane stage is years away and would be totally un-flight-tested, so may not be ANY safer initially.

I would probably increase priority of a methane stage, but I wouldn't wait for it. I'd fix this problem and fly out F9 for a few years until the methane stage is ready.

Summarising what some knowledgeable NSF members posted in the last thread:
- Falcon is unusual (possibly exceptional) in having LOX-immersed COPVs
- some voices at NASA were not wholly convinced that SpaceX had truly got to the root of CRS-7, therefore it could be premature to completely rule out any connection
- SpaceX's COPVs are unusual in having Al liners, rather than Ti, which has the potential for differential thermal expansion issues.

The thing that strikes me as very odd is that there has not been a He system failure in a first stage yet. These stages have been fired many more times than the second stages have, so you would predict that failures would crop up there instead- unless there are fundamental differences in the He systems?

I don't think aluminum liners are unusual in COPVs.  I know of at least one example that has been flying for years and is still in use.  Of course, its not submerged in LOX, though!

Aluminum liners are the rule, as far as I can tell, given my work on COPVs.

[matthewkantar]

(The difference I would suggest: DON'T do static fires with the payload attached! Just don't!)

This is the biggest mystery of all to me. It seems to me the whole static fire/rehearsal thing is for the safety of the payload. Either drop the static fire, or just launch the thing. There will be some number of issues that the payload can survive, but the risk of loss is unacceptable.

When I first got news of the anomaly, my worry was for the safety of the people on the ground in Florida. I was relieved to see no one was hurt. The next thought was that SpaceX had dodged a bullet by uncovering some problem without the loss of a payload. Not so much as it turns out. I watch this stuff pretty diligently, but I would not have guessed they were doing static fires with payloads mated.

Matthew

[Rocket Science]

There's a thought that's been going around in my head for the past three weeks about the two LOV events when it comes to the structural differences between S1 and S2... I call it simply, "what's the same, what's different"... S1 solid bird from launch to landing, S2 cantankerous... Why is this "if" there is so much commonality in materials, production technique, tooling, employees assembling them, handling and transportation, testing...etc?

[jbbo78]

*snip*

Like I said in the other thread earlier the real root cause here IS the same as CRS 7 and that is quality control. Doesn't matter if this ultimately turns out to have been the result of a design flaw (which given previous flight rationale I think is only 20% likely), it was still ultimately not caught/caused by bad quality control/inadequate testing (or perhaps excessively stressful testing or transportation) and thus ended up in the stage and causing LOV.

But again, I somehow doubt it's a design flaw. It's probably something as simple as too many cycles or too much stress, or not enough stress testing on the second stage, and/or not enough proper checkout during the pre launch flow that led to this, same as it did with the strut on CRS 7 because they simply assumed the material would work at rated stress levels instead of testing it to find out.

*snip*

The short version: revise quality control, testing, and checkout procedures relating to stage 2 and revise static fire procedures or eliminate static fire altogether (or at least ban the idea of putting the payload on during testing).

So they either didn't test enough or they tested too much? You can basically assign this same blame to any failure ever.  You didn't catch the one in a million failure case? Then you should have tested a million times.  Your testing a million times caused the material to weaken and fail? Bad you for testing too much.  Not particularly useful logic.

Eliminating static fires would simply have moved this failure to launch day. That's not a solution.  I do agree that having the payload on there for a static fire doesn't seem like the best idea though.

[Johnnyhinbos]

Whew, man - no offense, but I'm glad you guys don't work for SpaceX, cause if you did I think the F9 would be looking like this by now! Perhaps we can ease off the COPV talk for a bit. That's what I would call low hanging fruit. I think the issue is a little more nuanced than that. Perhaps now is a good time to just say, "hey - there are a lot of really talented folks armed with a lot more info then we have, working incredibly hard on this complicated issue and we're just going to wait to hear what the resolution is"

[spacekid]

(The difference I would suggest: DON'T do static fires with the payload attached! Just don't!)

This is the biggest mystery of all to me. It seems to me the whole static fire/rehearsal thing is for the safety of the payload. Either drop the static fire, or just launch the thing. There will be some number of issues that the payload can survive, but the risk of loss is unacceptable.

When I first got news of the anomaly, my worry was for the safety of the people on the ground in Florida. I was relieved to see no one was hurt. The next thought was that SpaceX had dodged a bullet by uncovering some problem without the loss of a payload. Not so much as it turns out. I watch this stuff pretty diligently, but I would not have guessed they were doing static fires with payloads mated.

Matthew

This failure was prior to the static fire. I believe NASA requires a wet countdown demonstration for planetary missions (short launch windows) which was done recently for the Atlas comet mission (which did not have the payload mounted).

[Jim]

(The difference I would suggest: DON'T do static fires with the payload attached! Just don't!)

This is the biggest mystery of all to me. It seems to me the whole static fire/rehearsal thing is for the safety of the payload. Either drop the static fire, or just launch the thing. There will be some number of issues that the payload can survive, but the risk of loss is unacceptable.

When I first got news of the anomaly, my worry was for the safety of the people on the ground in Florida. I was relieved to see no one was hurt. The next thought was that SpaceX had dodged a bullet by uncovering some problem without the loss of a payload. Not so much as it turns out. I watch this stuff pretty diligently, but I would not have guessed they were doing static fires with payloads mated.

Matthew

This failure was prior to the static fire. I believe NASA requires a wet countdown demonstration for planetary missions (short launch windows) which was done recently for the Atlas comet mission (which did not have the payload mounted).

That is for schedule assurance and not mission assurance

[Kabloona]

Whew, man - no offense, but I'm glad you guys don't work for SpaceX, cause if you did I think the F9 would be looking like this by now! Perhaps we can ease off the COPV talk for a bit. That's what I would call low hanging fruit. I think the issue is a little more nuanced than that. Perhaps now is a good time to just say, "hey - there are a lot of really talented folks armed with a lot more info then we have, working incredibly hard on this complicated issue and we're just going to wait to hear what the resolution is.

As much as the COPV's are being beaten to death here, engineering logic says they're a primary suspect.

It's hard to imagine what other He component failure could overpressure the LOX tank in less than 100 msec, unless a smaller component failed and a metal fragment got propelled through the common bulkhead. I've never seen metal tubing fail at 6000 psi, but I imagine it could produce high-velocity shrapnel.

I have, however, seen a 1000 psi Helium line blow out and almost take someone's head off. I was working on my Masters thesis in a high-pressure solid propellant combustion lab. A friend and I were pressurizing a shock tube with Helium when one of the stainless steel Helium line fittings let loose, and a section of stainless tubing went ballistic and almost hit my fellow student in the head. Instead it dented the corrugated steel wall right next to him. We both went home pretty shaken. That accident gave me instant respect for the power of high pressure gases.

[mn]

... . They have done this in the past after failures, CRS 7 included (IE upgrading to FT ect, using the time to just redesign things all the way around before flying again).

For the record CRS7 was the last of the previous design and the next flight was always going to be FT, that was a done deal long before the failure.

[FinalFrontier]

Summarising what some knowledgeable NSF members posted in the last thread:
- Falcon is unusual (possibly exceptional) in having LOX-immersed COPVs
- some voices at NASA were not wholly convinced that SpaceX had truly got to the root of CRS-7, therefore it could be premature to completely rule out any connection
- SpaceX's COPVs are unusual in having Al liners, rather than Ti, which has the potential for differential thermal expansion issues.

The thing that strikes me as very odd is that there has not been a He system failure in a first stage yet. These stages have been fired many more times than the second stages have, so you would predict that failures would crop up there instead- unless there are fundamental differences in the He systems?

I don't think aluminum liners are unusual in COPVs.  I know of at least one example that has been flying for years and is still in use.  Of course, its not submerged in LOX, though!

This particular one is unusual primarily in that its an Al liner being submerged. On top of that it's being submerged in super dense, super cold LOX which is a very exotic form of LOX whose features and behavior is not well understood even by SpaceX who has admitted they are still learning how to handle it (see prior LOX off temperature/off density scrubs on F9 FT flights).


This is the main issue here and why the quality control thing was/is so important and why it's where they keep messing up. You are dealing with:

A composite material that has unknown unknown's when it comes to cold gas exposure
A material that is not only being exposed to cold, but submerged entirely in a cryogenic liquid.
A material that is being submerged in a super-cooled cryogenic liquid with temperatures far more extreme than normal LOX.
A material which must also itself contain another extremely cold cryogenic liquid (the helium) that has a very different density and specific gravity, as well as different behaviors
A tanking procedure that is very complex and dynamic as propellant and helium loading occurs simultaneously very close to flight pressurization and launch.
A series of cyrogenic environments which are not yet well understood even by the manufacturer.
A system which has previously exhibited problems even prior to the introduction of exotic LOX.
And finally, a system in which it appears (based on prior analysis), to have failed prior to even reaching full capacity or flight pressure (He system was thought to be in the process of filling but not yet full or pressurized to flight conditions).



This is a very dynamic environment by any measure and since nobody, I think including SpaceX, fully understands the stresses, torquing, and cycling going on in that second stage tank during a loading procedure, this just further drives home the reason why you need adequate testing and quality control. If you are dealing with cutting edge technology and bleeding edge propellants and temperatures you need to make damn sure you do proper checkout and don't over-stress the dang thing in testing or fail to test adequately and miss a failure mode.

Which is exactly what happened. And its exactly why there has been so much criticism of SpaceX over this. They should have known better by now than to push things this far. I understand (and love them) for the way they pursue and push the bleeding edge of the design curve in these vehicles but there is a very fine line here between ambitious and foolish, and clearly it is not being respected. Blowing up a vehicle with cold engines during a test when all you are doing is loading propellants looks as stupid as it is: which is to say its pretty dang silly, no offense to SpaceX.


With all that said, I think they will learn alot from this and come back in a much better position, hopefully in the long run this moves up introduction of the new upper stage which would potentially solve alot of the problems related to the cryo environment anyway. And/or alternative He tank materials wouldn't hurt. But of course, the big one, it will for sure (we think) change how they test, close out, ship, ect, and will improve quality control.


Either that or they will blow another one up 6 months from now and Jeff Bezos will end up taking their contracts away in the next decade. Entirely up to SpaceX how this plays out now.

[Coastal Ron]

- some voices at NASA were not wholly convinced that SpaceX had truly got to the root of CRS-7, therefore it could be premature to completely rule out any connection

However, regardless if it was the root cause of the CRS-7 accident or not, out of spec struts were discovered to be coming from their supplier, even though the supplier was certifying each strut.  So if it wasn't CRS-7 it could have been a future flight that suffered an inflight accident.

Taking that into account, addressing the strut problem did make the Falcon 9 potentially safer.

And I have no doubt that all of the non-SpaceX members of the accident board are actively wondering if the two accidents are related (and no doubt many in SpaceX are wondering if they made the right conclusions), so I would be surprised if they have completely ruled it out yet, but just haven't been able to rule it in yet.

[FinalFrontier]

... . They have done this in the past after failures, CRS 7 included (IE upgrading to FT ect, using the time to just redesign things all the way around before flying again).

For the record CRS7 was the last of the previous design and the next flight was always going to be FT, that was a done deal long before the failure.

FT would have used the same struts and same material for the struts more likely than not had that failure not happened. Additionally, I (and I am not alone) remain unconvinced a strut failing was the actual failure mode as opposed to the liner or part of the COPV where the strut attached failing because it de-laminated.

We will never know because it would produce almost exactly the same readings on flight instrumentation, but we do know some struts in the same batch were found to have material defects in post accident investigation.

Here is why it doesn't matter though: the bottom line was the helium pressurization system failed due to quality control lapse. The same thing happened again here though its probably for a very different technical reason.

The issue being the event chain and result were the same and have a very common problem even though actual failure/material failure is different.

[Kabloona]

If you are dealing with cutting edge technology and bleeding edge propellants and temperatures you need to make damn sure you do proper checkout and don't over-stress the dang thing in testing or fail to test adequately and miss a failure mode.

Which is exactly what happened.

With respect, you list at least three different possible design/analysis/process problems, and we don't know which of them was at fault, so saying "which is exactly what happened" seems a bit premature. To summarize your areas of concern:

1. Design/Analysis: Part was improperly designed or a failure mode was overlooked, which is an analytic (FMEA) failure, not a QC/testing failure. You can't inspect/test for a failure mode or critical material property that your analysis failed to identify.
2. Testing: test procedures/methods were inadequate to verify a part meets spec, or testing damaged a part.
3. Quality Control: a substandard part got past inspection/test.
4. Unknown unknowns: COPV's in supercooled LOX are so cutting-edge that lots of unknown bad stuff may be happening. (Maybe some overlap with #1 above, but this is more about materials science unknowns than straighforward engineering FMEA lapses.)

There can be some overlap in QC failure and failure to test adequately/properly. But missing a failure mode in analysis has nothing to do with QC or testing.

So you have a list of three or four areas of possible concern. All of which may be valid. But I'd stop there before saying "that's exactly what happened." Which one(s) happened? I doubt we'll know until SpaceX tells us.

[mn]

... . They have done this in the past after failures, CRS 7 included (IE upgrading to FT ect, using the time to just redesign things all the way around before flying again).

For the record CRS7 was the last of the previous design and the next flight was always going to be FT, that was a done deal long before the failure.

FT would have used the same struts and same material for the struts more likely than not had that failure not happened.

I understand your post to assert that they did a redesign (switched to the FT design)  as a result of the failure and that is just not true.
 
(The rest is opinions which I'm not going to argue about.)

[toren]

S2 LOX was apparently loading at the time of the kaboom.  Is it known whether there is a back flow check valve on the LOX feed to S2? And if so, could it handle the sudden back pressure of a breached He system?

[PlanetStorm]

... . They have done this in the past after failures, CRS 7 included (IE upgrading to FT ect, using the time to just redesign things all the way around before flying again).

For the record CRS7 was the last of the previous design and the next flight was always going to be FT, that was a done deal long before the failure.

FT would have used the same struts and same material for the struts more likely than not had that failure not happened. Additionally, I (and I am not alone) remain unconvinced a strut failing was the actual failure mode as opposed to the liner or part of the COPV where the strut attached failing because it de-laminated.

We will never know because it would produce almost exactly the same readings on flight instrumentation, but we do know some struts in the same batch were found to have material defects in post accident investigation.

Here is why it doesn't matter though: the bottom line was the helium pressurization system failed due to quality control lapse. The same thing happened again here though its probably for a very different technical reason.

The issue being the event chain and result were the same and have a very common problem even though actual failure/material failure is different.

But respectfully, as much as you and others may be unconvinced that a failed strut caused CRS7 , I doubt that you have access to the data that is required for coming to a rational conclusion. As I understand it, there was a fluctuating pressure signal that was incompatible with a catastrophic COPV failure and which could only be explained in terms of a broken strut allowing the strongly buoyant COPV to bend a pressure line. If you have evidence or knowledge to the contrary, please share it - please don't just declare it unsatisfactory in your judgement if you have no data to base your judgement on.

I'm also unconvinced by your declaration that there was a quality control lapse in this latest anomaly. Where is your evidence that there was no design flaw, no mishandling, no environmental factor...?

[Rocket Science]

Question, perhaps rhetorical, perhaps not..? Were not the LOX/RP-1 loaded at the same temps and flow into S2 as was into S1? If it was, then what's different beside the TPS outside of the stages (ignoring engines,RCS, Grid Fins, etc) COPV's, struts, material gauge, plumbing, all subject to the same cold shock/stress during loading..?

[hrissan]

COPV burst test...