SpaceX Falcon 9 - AMOS-6 - (Pad Failure) - DISCUSSION THREAD (2)

[.Scott]

There's more to vendor relations than picking the price.  Perhaps the "business" issue relates to certification of vendors - or the QA responsibilities of vendors.  Quality standards and procedures need to be passed down through the supply chain.  Perhaps this was the business issue.

[envy887]

That doesn't necessarily mean they need to fundamentally change they way they do R&D, development, testing, manufacturing, etc, which is where most of the cost is IMHO.

I think that's the point where we disagree.
Manufacturing and testing: yes, to a certain degree. R&D (which includes development) is overrated except for extreme low volume applications like SLS.
And that's not just what I think, that's also what SpaceX have publicly stated they think.

Maybe. If they have to fundamentally change manufacturing and/or testing processes then all the hardware those processes have built has to be requalified. No way will they be flying in a month. Changing operations processes (like integration and launch ops) that don't output hardware can be done faster, especially if they can nail down the failure fault tree soon.

[ugordan]

And I might point out that V1.0 had no total mission failures, but both V1.1 and V1.2 have. 

Statistically insignificant bollocks and you know it. Both v1.1 and v1.2 have had more flights before the first major anomaly than v1.0 had in total. There is nothing to suggest v1.0 was inherently more reliable.

[CyndyC]

I suspect there's a better description of what she meant but it sure seems like a poor choice of words.

Or you could say the poor choice of words was earlier, when she did say somewhere that the cause might be rooted in operations, and then realized the implications in her role as COO, chief of operations.

And I don't see what she'd gain by calling something like that "a business process" issue.

Keeping her position at SpaceX.

[meekGee]

Losing 1 rocket/yr is unacceptable, but I don't have enough data to know whether this is indicative of something larger or was bad luck.

And neither do you, not unless you know what went wrong.

... So it wasn't indicative of a cultural issue.

I never said it was. Jim did.

My point was: since SpaceX claims that their processes are tweaked to the max and that's what gives them their competitive advantage, any change to these processes they will have to do will likely make that advantage smaller.
There's very little speculation in that, it's all stuff Elon said...

I don't think SpaceX claims that their processes are "tweaked to the max", since as with any statistical issue, there is no "max".

I think SpaceX will tell you that they are cheaper because they are an integrated operation, and that they have less fat.

.. we're drifting a bit.  Back to the root cause.

If this was a result of some intentional variation in the process, then that's not indicative of a "culture" or anything.

If this was a result of lack of process - that's something to frown upon.

[te_atl]

There's more to vendor relations than picking the price.  Perhaps the "business" issue relates to certification of vendors - or the QA responsibilities of vendors.  Quality standards and procedures need to be passed down through the supply chain.  Perhaps this was the business issue.

By her saying it was a business issue and not a vehicle issue, that implies we aren't talking a quality of product or supply chain problem.

Also by looking at a quick return, then were probably aren't talking about replacing vendors, reworking product quality, etc.

My first suspect is the fact that this static fire was being used to test new loading procedures that would extend the time that the chilled LOX would last.   Avoiding the costly (time, materials, schedule, reputation) scrub is a business driven decision.  Did that decision drive procedural changes that were not properly change managed?  Did all the right people sign off on them?  Change management rigor is often tailored  to a product based on business need and risk.  (for example, using manager signoffs in place of a formal CCB)   Did the business assign proper rigor to the management of the operational changes?

[Jim]

If this was a result of some intentional variation in the process, then that's not indicative of a "culture" or anything.

If this was a result of lack of process - that's something to frown upon.

Ah, yes it is.  The culture allowed a variation in the process that put a payload at risk.

[meekGee]

If this was a result of some intentional variation in the process, then that's not indicative of a "culture" or anything.

If this was a result of lack of process - that's something to frown upon.

Ah, yes it is.  The culture allowed a variation in the process that put a payload at risk.

If the change is intentional, and controlled, but had a negative result, that's not indicative of any culture issue.  All vendors make changes.

If there was no process to control what's happening and the change was unintentional, then that's an issue - maybe - depending on the circumstances.

[envy887]

If this was a result of some intentional variation in the process, then that's not indicative of a "culture" or anything.

If this was a result of lack of process - that's something to frown upon.

Ah, yes it is.  The culture allowed a variation in the process that put a payload at risk.

Bingo.

[te_atl]

Ah, yes it is.  The culture allowed a variation in the process that put a payload at risk.

That's too broad a generalization.   It implies variations only occur because of culture.   They can occur because of engineering and technical reasons as opposed to cultural ones.  Variations and waivers should always be planned for and a process should exist and be documented on how to manage them if they do arise.  Management of a variation may include rejection of the variation.

[Jim]

If this was a result of some intentional variation in the process, then that's not indicative of a "culture" or anything.

If this was a result of lack of process - that's something to frown upon.

Ah, yes it is.  The culture allowed a variation in the process that put a payload at risk.

If the change is intentional, and controlled, but had a negative result, that's not indicative of any culture issue.  All vendors make changes.

If there was no process to control what's happening and the change was unintentional, then that's an issue - maybe - depending on the circumstances.

The change isn't the issue, it is risking  the payload is the issue.

[akm]

I go with Jim on this one, changes should be tested out in Texas where you have room for failure, not with a payload attached to the rocket.

[jgoldader]

Ah, yes it is.  The culture allowed a variation in the process that put a payload at risk.

Sometimes, unexpected things happen.

I can't get my mind around the possibility that SpaceX was cavalier about their vehicle and its payload.  If Jim is suggesting hubris is at fault here, then okay, that would be a cultural issue, but we aren't even sure it's hubris.  And we should remember that even hubris isn't willful negligence.   

We simply don't know the sequence of events.  Perhaps there was a flaw in an analysis that suggested whatever variance in procedure may have happened should have been okay.

On L2, there's a really interesting thread about Titan launches, and the great majority of the early ones were dismal failures with associated fireballs.  Every time I look at the thread, I'm reminded just how difficult launching rockets is.  At some level, you can't test every change on a rocket with a payload-free flight.  But SpaceX's demonstrated ability to recover boosters means the days of frequent test flights to test out small design changes might be coming.  Until then, we have to hope the engineers do their jobs right, with all the process checking and such that idea entails.

[Toast]

If there was no process to control what's happening and the change was unintentional, then that's an issue

The change isn't the issue, it is risking  the payload is the issue.

I'd argue it's both, but leaning towards Jim's interpretation. Working with change control, it's pretty common for everything is analyzed in terms of likelihood of an adverse event, and potential impact of an adverse event. Both are assigned a score (which can be essentially arbitrary), and the total risk is assigned as the product of the two. Insufficient testing of new procedures may have potentially increased the likelihood of an adverse event, but it's inarguable that integrating the payload increased the impact of the adverse event.
Aside from addressing the specific root cause of this event, if SpaceX wants to reduce risk they ought to (and probably do) perform an analysis of their procedure and identify potential sources of risk and score them in terms of likelihood and impact. Since payload integration has a relatively small benefit (a day or so shorter launch campaign) and a relatively high potential impact (loss of payload), I'd guess it's worth avoiding in the future. Of course, they could always do what Jim suggests and skip the static fire altogether--It's not readily apparent that there's any risks that the static fire telemetry could identify that couldn't also be identified with telemetry received before releasing the hold downs.

[virnin]

I go with Jim on this one, changes should be tested out in Texas where you have room for failure, not with a payload attached to the rocket.

NOTE: Just cherry-picking a recent failure candidate here for illustrative purposes.  Not casting a vote.

I don't think McGregor has the capability to test the integrated launch sequence since they don't stack the stages.  Yes, testing each stage independently might have revealed problems but you can't be certain that interactions between the stages and the GSE will be revealed.

I.E. fast fill changes for S2 might have gone perfectly well in Texas.  The resonance issue didn't manifest simply because the GSE at SLC-40 connects to both stage's He systems rather than only one at a time.

[te_atl]

I go with Jim on this one, changes should be tested out in Texas where you have room for failure, not with a payload attached to the rocket.

I don't disagree with the payload bit.  I can't see how its presence added value to the testing, so I can't come up with a scenario why it should have been there during a test of a procedural change.

However, as to the testing in FL vs TX... the initial business case could have been that the GSE at both sites was identical.  Then technical variations could cause them to drift.  Its not inconceivable to imagine that business risk analysis decisions (with proper engineering input) could be made that said the cost to keep them identical exceeded the likelihood of the risks that any differences incurred.  If this case happened, then a operational procedural change would only be able to be tested in FL.

[ddeflyer]

To me, the "business process" comment brings up two major fault paths:

1. sufficient definition of acceptable edge conditions (ie. some series of unrelated GSE variations might have seemed OK according to checklists but actually compounded into a problem)
2. documentation of assumed conditions for hardware operating conditions (so a new procedure might cause an assumed condition (or a condition not stated in primary documents) to become invalid without triggering any red flags in its evaluation).

Of those two, the second one really speaks to me; in a large organization it can be easy to get some of those little items buried over time when dealing across teams. In software, you sometimes create bugs that are missed in QA by twiddling little details of code that shouldn't (according to API documentation) cause any problems. This is part of why code maintenance can be so difficult.

[meekGee]

How did we get from "modifying the fill procedures" to "why for f***'s sake do they have the payload on top during the static fire"?

[feynmanrules]

Well since nobody knows what "business process" she's talking about, what's the point second guessing her, and then attacking the very speculation you just made as if it were fact?

Ok, then spin it another way to make it a positive thing that she said

In a commercial space company, every process is a business process.  Reading the least possible into her sentences, I think she's just saying the process that failed was not in engineering nor manufacturing.   So like much of how they've communicated all along... it was kind of a blanket statement that was meant to be comforting to business people w/out too much specifics (or false precision as word elon likes).   My impression was it seems they have an idea of what issue may be, but they're trying to increase the certainty via testing.    Then they'll know which business processes need modification.

[Jet Black]

If the change is intentional, and controlled, but had a negative result, that's not indicative of any culture issue.  All vendors make changes.

it means the change was not tested well enough. let's say for simplicity this was an issue due to using supercooled LOX that did something unexpected that resulted in this explosion. That means the processes in the business which led to them using supercooled LOX were inadequate to catch this problem. The culture was one of careless rushing. Now I like SpaceX a lot, btu I do think that they rush rather a lot. There is no real need for them to try and get almost everything done within a day or two of the launch, but they do. It reminds me of the knife game - it's really cool watching someone stab away with a crowd around them egging them on, until someone cuts off a finger.