SpaceX Falcon 9 - AMOS-6 - (Pad Failure) - DISCUSSION THREAD (2)

[mfck]

How did we get from "modifying the fill procedures" to "why for f***'s sake do they have the payload on top during the static fire"?

We've been Jimmed.

[meekGee]

So to summarize - GS said something that is vague and actually rather confusing, everyone interpret it to mean their favorite theory, and now we've run out of steam and are waiting for another statement - yes?

[CameronD]

So to summarize - GS said something that is vague and actually rather confusing, everyone interpret it to mean their favorite theory, and now we've run out of steam and are waiting for another statement - yes?

Something like that.

In a commercial space company, every process is a business process.  Reading the least possible into her sentences, I think she's just saying the process that failed was not in engineering nor manufacturing. 

A couple of points:
1. Shotwell was probably only trying to emphasise their faith in the design and manufacture of the F9, but may have gone too far the other way.  As I see it, a failed "business process" (whatever that may be) could be a worse outcome than a manufacturing defect or even faulty design because your insurers and clients are going to start asking difficult questions about whether or not they should be dealing with you.  Questions like: "Why didn't you follow your own Management Of Change procedures? ..and, come to think of it, do you actually have any anyway?!?"

2. No matter what she did or didn't say, it seems mighty obvious from the discussions here and elsewhere over the many days (weeks?) since the incident, that if there's nothing wrong with either the design and manufacture of the stage, then perhaps their propellant loading procedures need serious internal review.
 

[envy887]

How did we get from "modifying the fill procedures" to "why for f***'s sake do they have the payload on top during the static fire"?

More like "why are they testing modifications to operational procedure with a payload on top?"

Static fire with new operational procedures is fine. Static fire with payload on top is fine. Doing both at once is not fine.

[CameronD]

How did we get from "modifying the fill procedures" to "why for f***'s sake do they have the payload on top during the static fire"?

More like "why are they testing modifications to operational procedure with a payload on top?"

Static fire with new operational procedures is fine. Static fire with payload on top is fine. Doing both at once is not fine.

As has been said before on this thread at least once, payload on top or not during static fire is the customer's decision - not SpaceX's

..and, other than adding a whole extra dimension to the clean-up, it has nothing to do with this incident anyways.

[envy887]

How did we get from "modifying the fill procedures" to "why for f***'s sake do they have the payload on top during the static fire"?

More like "why are they testing modifications to operational procedure with a payload on top?"

Static fire with new operational procedures is fine. Static fire with payload on top is fine. Doing both at once is not fine.

As has been said before on this thread at least once, payload on top or not during static fire is the customer's decision - not SpaceX's

..and, other than adding a whole extra dimension to the clean-up, it has nothing to do with this incident anyways.

Changing propellant loading procedures with a payload on top might not have been up to the customer though. I wonder if they signed off on that.

[Navier–Stokes]

Changing propellant loading procedures with a payload on top might not have been up to the customer though. I wonder if they signed off on that.

Unless I'm mistaken, we don't know that any propellant loading procedures were changed during the AMOS-6 static fire. All we know is that they adjusted the engine chilldown and propellant/pressurant loading sequence during the JCSAT-16 static fire (which didn't have the payload integrated).

On a different note, Spaceflight101 published a decent article summarizing the current state of the investigation (besides the rather click bait title that is...).

[Rocket Science]

SpaceX's version of "faster, cheaper, better"..?

[Jim]

As has been said before on this thread at least once, payload on top or not during static fire is the customer's decision - not SpaceX's

no, it is a Spacex decision with customer's concurrence.  The customer didn't ask to be put on top.

[Vultur]

Rapid improvement requires rapid iteration which requires failing fast. Failing fast is great, it's served SpaceX well up to this point, allowing fast growth and extensive market disruption.

But failing isn't an option once a customer payload is mounted. There has to be a hard stop to any unproven changes at some point prior to risking a payload. Test AS you fly is great, test WHILE you fly is not.

This is why I wonder if commercial crew wasn't a bad move this early in SpaceX's plan.

If they only launch satellites, then some amount of lesser reliability is worth it for lesser cost. If the total cost to the customer of losing the satellite is say $1 billion, and you're 6% likely to lose it on Falcon and 1% likely on Atlas... as long as the riskier option is at least $50 million, it's the rational choice.

Accepting say 1 failure in 15 in exchange for lower operating costs might be perfectly sensible, for a pure satellite launcher.

But when people's lives are involved, not so much.

EDIT: fixed quote

[Herb Schaltegger]

You know, a couple days ago when Ms. Shotwell used the phrase "let go" with regard to the COPV, everyone was quick to point out how specific a term "let go" is, and how it means only one thing in terms of pressure vessels.

Now, when she uses the phrase "business processes", everyone is quick to point out how imprecise the term is, and how susceptible it is to multiple interpretations based on the biases implicit in the reader.

So the interesting question to me is, "Why so precise in one interview, and why so vague in a second?" My supposition is that "COPV let go" is a pretty easy proximate cause to point a finger at, and doesn't castigate anyone personally; it's just hardware, after all. Saying "it let go" doesn't actually say why, how or what caused it to let go.

Interestingly, "business processes" is also a way to point a finger without personalizing it. To ME, what that phrase represents is an over-arching category of things: the test engineering oversight process that allows procedural errors in a hot fire to exist, sufficient to destroy the vehicle and pad infrastructure; the change management that allows steps to be changed, rearranged (and thus maybe even missed by accident in the shuffle!) with a customer payload at risk; the programmatic operational management that continues to put "agility" ahead of the end result. After all, every engineer understands the old saw that "Perfect is the enemy of Good Enough." At what point do you say, "I'm done changing basic operational parameters and design features" and simply build and fly your rocket?

These are all the questions - very hard questions - that SpaceX has to come to grips with as they RTF.

[Lee Jay]

People always say things like "space is hard".  It seems that it's hard for a very simple reason - "good enough" is not good enough.

All sorts of things, large and small can go wrong on an airliner, and the ending is happy.  "Good enough" is almost always good enough.

Rockets have such a difficult obstacle to overcome (Earth's gravity well and the velocity needed to be in orbit), that margins have to be very small and energy levels have to be very high.  This means the only thing that's good enough for rockets is "perfect", or nearly so.  And this seems to apply to every aspect of a flight, from vehicle design to construction through to payload deploy.  If not, bad days seem to result.

[envy887]

Unless I'm mistaken, we don't know that any propellant loading procedures were changed during the AMOS-6 static fire. All we know is that they adjusted the engine chilldown and propellant/pressurant loading sequence during the JCSAT-16 static fire (which didn't have the payload integrated).

IIRC there was a statement from SpaceX that they were trying some changes (specifically later LOX load) to improve hold time with subcooled propellant. Looking back I see some speculation in this thread and the first one, but I can't find a real source. But, it fits all too well with what Shotwell said, so...

Anyone have a definitive answer to this?

[Rocket Science]

People always say things like "space is hard".  It seems that it's hard for a very simple reason - "good enough" is not good enough.

All sorts of things, large and small can go wrong on an airliner, and the ending is happy.  "Good enough" is almost always good enough.

Rockets have such a difficult obstacle to overcome (Earth's gravity well and the velocity needed to be in orbit), that margins have to be very small and energy levels have to be very high.  This means the only thing that's good enough for rockets is "perfect", or nearly so.  And this seems to apply to every aspect of a flight, from vehicle design to construction through to payload deploy.  If not, bad days seem to result.

True LeeJay, but a passenger airliner stays pretty much away from the edges of it's envelope whereas a rocket it's pretty much at the edges of the envelope which is the nature of the beast I guess...

[Brovane]

As has been said before on this thread at least once, payload on top or not during static fire is the customer's decision - not SpaceX's

no, it is a Spacex decision with customer's concurrence.  The customer didn't ask to be put on top.

Jim-What you are saying is that doing the Static Fire with the payload attached was the default option for SpaceX?  However the customer is always free, to say No this option.  However they have to tell SpaceX no, to doing the static fire with the payload attached.  It is kind of like when you sign-up for a new account at a website and the option to receive advertising e-mails is already selected and you have to manually un-select it. 

[Herb Schaltegger]

People always say things like "space is hard".  It seems that it's hard for a very simple reason - "good enough" is not good enough.

All sorts of things, large and small can go wrong on an airliner, and the ending is happy.  "Good enough" is almost always good enough.

Rockets have such a difficult obstacle to overcome (Earth's gravity well and the velocity needed to be in orbit), that margins have to be very small and energy levels have to be very high.  This means the only thing that's good enough for rockets is "perfect", or nearly so.  And this seems to apply to every aspect of a flight, from vehicle design to construction through to payload deploy.  If not, bad days seem to result.

All the more reason not to tinker and tweak a working design indefinitely in the hopes of squeezing out another half-percent in performance, or adding 10 seconds to a hold window or whatever other rabbit-hole parameter which might tempt an "agile" business to keep leaping into. "Business processes" - I believe - is a deliberate phrase chosen for a deliberate reason. F9-1.2 is a reasonably well-designed and working vehicle. I think Shotwell's statement means what it says in my post above: time to re-think whatever they did or failed to do, if the root cause turns out to be an operational error in the count; but more broadly, re-think what led to that happening in the first place.

[spacekid]

It seems that SpaceX has started to fall into the ways of those they once criticized.

I thought that SpaceX originally stayed away from the risky high performance launch vehicles in favor of the tried and true. Now, to squeeze extra performance out of their launch vehicles, they are perhaps going to extra risky lengths.

[Kabloona]

My first suspect is the fact that this static fire was being used to test new loading procedures that would extend the time that the chilled LOX would last.   Avoiding the costly (time, materials, schedule, reputation) scrub is a business driven decision.  Did that decision drive procedural changes that were not properly change managed?  Did all the right people sign off on them?  Change management rigor is often tailored  to a product based on business need and risk.  (for example, using manager signoffs in place of a formal CCB)   Did the business assign proper rigor to the management of the operational changes?

An aerospace exec pinning a launch vehicle failure on a "business process" falls somewhere between "rarely happens" and "never happens," so it's understandable we're left scratching our heads. But your hypothesis best fits the piecemeal info we have, IMO:

-probable COPV failure per GS
-credible source cited "weird harmonics" in COPV during tanking
-GS initially suggested "operational" cause, then cited "business process"

All of which seems consistent with a change to the tanking procedure driven by a business decision.

[Robotbeat]

IMHO, the only thing standing between SpaceX and their Mars ambitions, long-term, is getting the Falcon rocket family to work reliably.

They need to get their workhorse tweaked. They need to address whatever business issues that lead to failures.

And I think abandoning the reusable upper stage may help this because it means they're separating their cutting-edge development side from their operations side, which needs to be highly reliable.

Anyway, that's what I hope for, anyway. If they can manage to learn from this in that way, to perhaps change part of themselves to something they've never had before (consistent, professional, reliable operations), then I think they'll succeed on putting people on Mars.

[dorkmo]

for us visual folks, here's what a business process looks like mapped out

https://en.wikipedia.org/wiki/Business_process_mapping#Example

as you can see someone chose the wrong process for this scenario

Input: Falcon 9  ->  Make Breakfast -> Output: Scrambled Eggs