SpaceX Falcon 9 FT - ORBCOMM-2 - Dec. 21, 2015 (Return To Flight) DISCUSSION

[Herb Schaltegger]

Look, all this QC stuff has been debated ad nauseum for 4-1/2 months. No need to rehash it even further now.

What WOULD be useful and instructive would be to discuss is what is apparently holding up testing of the new FT configuration at Gregor and how that impacts SpaceX's mission flow and manifest plans.

[Craftyatom]

What WOULD be useful and instructive would be to discuss is what is apparently holding up testing of the new FT configuration at Gregor and how that impacts SpaceX's mission flow and manifest plans.

AFAIK the second stage is still considered the rate-limiting factor, even after all the various delays in first stage testing due to results and/or weather, which raises the question: where's the second stage now?  Keeping tabs on the first stage is easy, it's sitting out at McGregor, but I'm not entirely sure where the second stage should be.  Is it still at Hawthorne, and if so, how finished is it?  Is it at the Cape but undergoing changes?  Did it get shipped and then have to be shipped back?  Some argue that second stage failure implies slower second stage production, but the struts were used on both stages (IIRC), and the FT upgrades shook up everything, so I can't imagine what would cause it to be so behind the first stage.

... with that said, I fully realize how swift and smooth an RTF this will have been.  Partly because the issue was just in a structural element and the fix was "make sure it meets strength requirements" and partly because it's SpaceX and they've proven to move through development phases quite quickly, it should be less than half a year between F9 launches - which, for some, could actually just be considered a slow launch cadence.  I can't say I wouldn't have preferred a quicker path through this pre-launch flow, but I also have no right to complain.

[ChrisWilson68]

There's no need to invoke supernatural beings! Yes, all parts can have defects. Most defects are of kinds that can be detected. In general there is no need to test-to-destruction to eliminate defective parts.

Putting this back into context: what SpaceX is doing now, they could have done before.

They're not buying struts from that supplier any more, that's what they're doing differently.  Now they know that supplier is unreliable.  It's not clear they had any reason to believe that before.

Some problems can reasonably be foreseen, some can not.  You can always do more testing for less and less likely problems, but that costs money and time.  At some point, you have to make a trade-off and decide which tests to do.

Did SpaceX make the wrong choice here, or did they just get unlucky?  If you stand on a 20 in blackjack, that can be the right bet to make given the information available even if it turns out in hindsight you find out you would have been better off hitting.

I don't think we have enough information on the details of the defect and what kind of testing would have been needed to detect it to know whether SpaceX made the right choice.

[Kabloona]

There's no need to invoke supernatural beings! Yes, all parts can have defects. Most defects are of kinds that can be detected. In general there is no need to test-to-destruction to eliminate defective parts.

Putting this back into context: what SpaceX is doing now, they could have done before.

They're not buying struts from that supplier any more, that's what they're doing differently.

And perhaps more importantly, they're load testing each strut.

[guckyfan]

And perhaps more importantly, they're load testing each strut.

Going from trust to trust but verify. Which is just another way of saying, DON'T trust.

[Karloss12]

There's no need to invoke supernatural beings! Yes, all parts can have defects. Most defects are of kinds that can be detected. In general there is no need to test-to-destruction to eliminate defective parts.

Putting this back into context: what SpaceX is doing now, they could have done before.

They're not buying struts from that supplier any more, that's what they're doing differently.

And perhaps more importantly, they're load testing each strut.

Where has it been said that they aren't buying struts from that supplier?

As I remember, the problem was with the bolts in the strap rather than the strap structure itself.  Elon said that SpaceX tested thousands of bolts to find the primary failure suspect and will now as a result test all bolts in future.

He specifically did not blame any supplier at all.  With the bolt being the primary suspect, there may be no need to redesign the strut or change strut supplier.  As for the Bolt sub-supllier, if you fasten something critical together without testing the spindly little fastener or don't provide a redundant fastener then you are asking for trouble even with a massive bolt load safety factor.

With the information I have the only change needed is testing of every bolt.  A change in suppler would only have the effect of make SpaceX's customer feel happy and joyous inside and nothing else.

[guckyfan]

With the information I have the only change needed is testing of every bolt.  A change in suppler would only have the effect of make SpaceX's customer feel happy and joyous inside and nothing else.

It also sends a signal to suppliers that selling junk to SpaceX carries a risk.

[Kabloona]

There's no need to invoke supernatural beings! Yes, all parts can have defects. Most defects are of kinds that can be detected. In general there is no need to test-to-destruction to eliminate defective parts.

Putting this back into context: what SpaceX is doing now, they could have done before.

They're not buying struts from that supplier any more, that's what they're doing differently.

And perhaps more importantly, they're load testing each strut.

Where has it been said that they aren't buying struts from that supplier?

As I remember, the problem was with the bolts in the strap rather than the strap structure itself.

All your questions are answered here:

http://shitelonsays.com/transcript/elon-musk-talks-failed-crs-7-dragon-mission-2015-07-20

 Elon may have used the word "bolt" as shorthand for "special attachment fitting on the end of the strut" because he didn't want to get into too much technical detail in public. Otherwise they'd simply be changing bolts or bolt test protocol instead of changing the strut design and the supplier.

[jfallen]

So what is the latest update on meeting the projected launch date for RTF?  I am eager for rockets to fly again so the forums aren't bogged down with irrelevant speculation.

[Mader Levap]

One can imagine a design that doesn't incorporate parts that can have undetectable defects. But a company doesn't end up with that kind of design as a result of "luck."

This request is impossible, nonsensical and unreasonable.

[LouScheffer]

One could imagine a scenario in which SpaceX [...] just got unlucky because [...] there was no reasonable way to detect the defect.

One can imagine a design that doesn't incorporate parts that can have undetectable defects. But a company doesn't end up with that kind of design as a result of "luck."

Since all parts made by human beings and their machines can possibly have undetected defects (unless every single part is tested to destruction, in which case, you still cannot have completely, provably defect-free parts riding upon  your rocket), then I suppose you are advocating that we use parts for rockets made by supernatural beings?  Since, after all, nothing in the natural world can possibly create parts that have zero potential for undetected defects.

Just sayin'...

No need for super-natural beings.   What you do is find the size of defect you can reliably detect, then design your part such that any defects smaller than that will not cause failure before the next inspection, or end-of-life of the part.  So while you cannot avoid parts with undetectable defects, you can avoid  failure-causing undetectable defects.  This is the system used in designing airplanes, with their A,B,C and D inspections, and it works.

Of course the system can fail - defects that should be caught can be missed, the design can improperly account for small defects that might be present, and so on.   But I can't recall any accident report, ever, that said "That's an undetectable defect that caused this crash.  We'll just hope it does not happen again."  Instead any such problem is regarded as a failure of inspection, design, usage, or some combination, and in general you never get another crash from the exact same problem.   So in practice, though you can't eliminate undetectable defects, you can design out fatal undetectable defects, which is good enough.  This is the task at which airframe builders by and large succeed at, but SpaceX failed to do.

[the_other_Doug]

Of course the system can fail - defects that should be caught can be missed, the design can improperly account for small defects that might be present, and so on.   But I can't recall any accident report, ever, that said "That's an undetectable defect that caused this crash.  We'll just hope it does not happen again."  Instead any such problem is regarded as a failure of inspection, design, usage, or some combination, and in general you never get another crash from the exact same problem.   So in practice, though you can't eliminate undetectable defects, you can design out fatal undetectable defects, which is good enough.  This is the task at which airframe builders by and large succeed at, but SpaceX failed to do.

By and large, I totally agree, Lou.  However, please recall that there have been a rather significant number of cases in which airframe builders have missed fatal defects, easily detectable or not.  Some have been design defects, others materials defects, and likely the majority have been maintenance or procedural defects.

I recall one major airframe -- I believe it was the DC-10, but I can't recall right now -- in which the rear cargo door was designed with a latching mechanism that was not sufficient, under all flight conditions, to keep the door closed.  This resulted in more than one fatal crash when the rear cargo door blew open in flight.

That was certainly an undetected problem.  Was it undetectable?  That I can't answer -- but at least two planes crashed before the manufacturer realized that there was a problem with either the design or the materials.

Of course, there are always errors that will creep in due to the well-known human "will to be stupid" -- especially in maintenance and procedural errors.  The good old "Hey, I removed all the bolts holding the vertical stabilizer down, in preparation for a maintenance check -- it's not my fault if the guy who took over the shift when I left didn't notice it."  That kind of thing.  That will happen during initial construction, as well.

So, again, I don't think the line "This is the task at which airframe builders by and large succeed at, but SpaceX failed to do" is really fair.  SpaceX has had one failure along these lines, and airframe builders over the decades have had hundreds -- which, unlike the CRS-7 event, cost lives.  Yes, there have been thousands of airframe designs built, and not very many Falcon 9 airframes.  But still, I think that all human-built mechanical contraptions, of all kinds, are subject to defects of all kinds, reasonably detectable or not.

I firmly believe you only get into "culpable" ground when a defect is glaringly obvious and you simply failed to look for glaringly obvious defects.  From the small amount of information we have from the SpaceX accident investigation, I would have to say that this doesn't appear to have been the case for CRS-7.  Had there been zero defect testing of the struts in question (by either/both SpaceX and the manufacturer), then you might have a case for saying that Space "failed to do" acceptable defect detection.  But I really don't think we can say, especially with what information we have available (i.e., not just based on speculation that "well, I bet they didn't do a good job" but rather on "I know for a fact, based on the evidence I will now present, that they didn't do a good job") that SpaceX failed to do acceptable defect testing -- testing that simply did not reveal a defect that standard industry-wide testing criteria would generally fail to detect.  Like parts that had been spot-tested later showing major defects in a small percentage of the whole lot, but which could well be missed by the industry-standard technique of testing a percentage of items and then applying that percentage test to the entire lot.

YMMV.

[Kabloona]

One could imagine a scenario in which SpaceX [...] just got unlucky because [...] there was no reasonable way to detect the defect.

One can imagine a design that doesn't incorporate parts that can have undetectable defects. But a company doesn't end up with that kind of design as a result of "luck."

Since all parts made by human beings and their machines can possibly have undetected defects (unless every single part is tested to destruction, in which case, you still cannot have completely, provably defect-free parts riding upon  your rocket), then I suppose you are advocating that we use parts for rockets made by supernatural beings?  Since, after all, nothing in the natural world can possibly create parts that have zero potential for undetected defects.

Just sayin'...

No need for super-natural beings.   What you do is find the size of defect you can reliably detect, then design your part such that any defects smaller than that will not cause failure before the next inspection, or end-of-life of the part.  So while you cannot avoid parts with undetectable defects, you can avoid  failure-causing undetectable defects.  This is the system used in designing airplanes, with their A,B,C and D inspections, and it works.

Of course the system can fail - defects that should be caught can be missed, the design can improperly account for small defects that might be present, and so on.   But I can't recall any accident report, ever, that said "That's an undetectable defect that caused this crash.  We'll just hope it does not happen again."  Instead any such problem is regarded as a failure of inspection, design, usage, or some combination, and in general you never get another crash from the exact same problem.   So in practice, though you can't eliminate undetectable defects, you can design out fatal undetectable defects, which is good enough.  This is the task at which airframe builders by and large succeed at, but SpaceX failed to do.

Strut defect(s) was/ were found after the fact by load testing, so it seems to me this discussion of "undetectable" flaws is missing the point a bit.

What Elon meant was that the flaws they found were undetectable either by visual inspection (and maybe also by standard NDE methods, but it sounded to me like he was only referring to the fact that the flaw was invisible to the naked eye).

The defect probably was detectable, if the simple practice of 100% load testing for acceptance had been employed. Which SpaceX will now do.

[whitelancer64]

One could imagine a scenario in which SpaceX [...] just got unlucky because [...] there was no reasonable way to detect the defect.

One can imagine a design that doesn't incorporate parts that can have undetectable defects. But a company doesn't end up with that kind of design as a result of "luck."

Since all parts made by human beings and their machines can possibly have undetected defects (unless every single part is tested to destruction, in which case, you still cannot have completely, provably defect-free parts riding upon  your rocket), then I suppose you are advocating that we use parts for rockets made by supernatural beings?  Since, after all, nothing in the natural world can possibly create parts that have zero potential for undetected defects.

Just sayin'...

No need for super-natural beings.   What you do is find the size of defect you can reliably detect, then design your part such that any defects smaller than that will not cause failure before the next inspection, or end-of-life of the part.  So while you cannot avoid parts with undetectable defects, you can avoid  failure-causing undetectable defects.  This is the system used in designing airplanes, with their A,B,C and D inspections, and it works.

Of course the system can fail - defects that should be caught can be missed, the design can improperly account for small defects that might be present, and so on.   But I can't recall any accident report, ever, that said "That's an undetectable defect that caused this crash.  We'll just hope it does not happen again."  Instead any such problem is regarded as a failure of inspection, design, usage, or some combination, and in general you never get another crash from the exact same problem.   So in practice, though you can't eliminate undetectable defects, you can design out fatal undetectable defects, which is good enough.  This is the task at which airframe builders by and large succeed at, but SpaceX failed to do.

How about detecting a failure that, based on the design of the part, should never have happened?

The only possible way to have caught the defective strut assembly would have been to test all of them to full flight loads, something that should have been unnecessary since the part was over-designed by a factor of 3. As you suggested, the design should have ruled out a failure at less than full flight loads.

[Lars-J]

No, the point if this thread is now the mission, which just so happens to be a "RTF" mission.

This is where we want to discuss the mission, not generalities about returning to flight. IMO. What you are doing is about as on-topic as a general discussion of orbital mechanics would be for any satellite launch.

[Comga]

I went back to the first post to find support for Lars-J's position, with which I agreed, that this should be a thread about the Orbcomm-2 mission, but saw this:

"A thread to discuss SpaceX's efforts to Return To Flight (RTF) following the loss of the Falcon 9 and Dragon during the CRS-7 Launch, now assigned as the ORBCOMM-2 Mission."

If one were pedantic, the subject of the sentence is "efforts" to Return To Flight, and ORBCOMM-2 is the last detail, and it doesn't mention the "Full Thrust" aspect at all. That's more like what Space Ghost is saying.

That said, it would be good to have some separate thread to discuss the ORBCOMM mission separately from the discussion of quality assurance on flight hardware.  Where we would discuss the Full Thrust aspects is academic, because there haven't been many (any?) posts specifically about that.

[vulture4]

The only possible way to have caught the defective strut assembly would have been to test all of them to full flight loads, something that should have been unnecessary since the part was over-designed by a factor of 3. As you suggested, the design should have ruled out a failure at less than full flight loads.

This is an important point. The fault should have been identified before flight, not from the design (which was apparently adequate for the flight loads) and not from 100% testing, which is not possible in situations where the test would damage the component.

The failure could and should have been prevented by the implementation of scientific quality control procedures, as first developed by W. Edwards Deming ( https://en.wikipedia.org/wiki/W._Edwards_Deming ). As Deming demonstrated in great detail, these procedures do not include 100% testing. They are based on an understanding of the source of variation in the quality of manufactured products. Quality is not random; it depends on the continuous measurement of variations in the process, indentification of nonrandom variations indicating inadequate control, identification of the sources of the variation, and correction of the process to restore quality.

Periodic sampling and quantitative testing to failure of the struts would have indicated (as it apparently did when the customer, SpaceX tested a sample of the struts) that control of the process at some point was lost due to an unintended change in the process, possibly because of improper metallurgy in the raw material or improper welding procedures. Such a change is manifested by a change in the statistical distribution of quantitative properties in a sample of parts. This is not the same as testing all the parts, or testing whether or not they meet purchase specifications.

Within NASA the meaning of "quality control" has changed over the years, and today it refers essentially to having more people verify that written procedures are followed. This has very little relevance to the actual science of quality control as Demings, an industrial engineer, developed it under the pressure of World War II.

Ironically in the postwar era US industry largely ignored Deming, and he eventually found a more receptive audience in Japan, which was hobbled by its reputation for making goods that were cheap and shoddy. As US manufacturers became complacent in the 60's and 70's and Japanese products overcame their inferior reputation and gained a permanent place in the US market, the practical value of Deming's ideas was, I feel, demonstrated in the most convincing terms.

[watermod]

The only possible way to have caught the defective strut assembly would have been to test all of them to full flight loads, something that should have been unnecessary since the part was over-designed by a factor of 3. As you suggested, the design should have ruled out a failure at less than full flight loads.

This is an important point. The fault should have been identified before flight, not from the design (which was apparently adequate for the flight loads) and not from 100% testing, which is not possible in situations where the test would damage the component.

The failure could and should have been prevented by the implementation of scientific quality control procedures, as first developed by W. Edwards Deming ( https://en.wikipedia.org/wiki/W._Edwards_Deming ). As Deming demonstrated in great detail, these procedures do not include 100% testing. They are based on an understanding of the source of variation in the quality of manufactured products. Quality is not random; it depends on the continuous measurement of variations in the process, indentification of nonrandom variations indicating inadequate control, identification of the sources of the variation, and correction of the process to restore quality.

Periodic sampling and quantitative testing to failure of the struts would have indicated (as it apparently did when the customer, SpaceX tested a sample of the struts) that control of the process at some point was lost due to an unintended change in the process, possibly because of improper metallurgy in the raw material or improper welding procedures. Such a change is manifested by a change in the statistical distribution of quantitative properties in a sample of parts. This is not the same as testing all the parts, or testing whether or not they meet purchase specifications.

Within NASA the meaning of "quality control" has changed over the years, and today it refers essentially to having more people verify that written procedures are followed. This has very little relevance to the actual science of quality control as Demings, an industrial engineer, developed it under the pressure of World War II.

Ironically in the postwar era US industry largely ignored Demings, and he eventually found a more receptive audience in Japan, which was hobbled by its reputation for making goods that were cheap and shoddy. As US manufacturers became complacent in the 60's and 70's and Japanese products overcame their inferior reputation and gained a permanent place in the US market, the practical value of Demings' ideas was, I feel, demonstrated in the most convincing terms.

Deming's ideas are not as important in fields undergoing a high rate of change.   A friend was on the Motorola team supporting KDDI's (Japan) expansion in the early 2000s.  He noticed none of the Japanese were using Motorola phones.   (developed with Deming always in mind) He dropped his to the floor and it had no effect.  He dropped one of the customer's Hitachi phones and it exploded to pieces.   He said "Look Deming's principles make a better phone!"   The guy with the Hitachi phone responded: "your's cost $1500 mine cost $60 from the corner vendor.  The rate of change in the phone field makes both obsolete in one or two years.   Mine costs $60 to replace so I can do it any time....why should I spend $1500 for a Rolls Royce?"
 

[cambrianera]

Attention to the process to get a finished product on spec without much testing is always worthwile.
No field is excluded.
It can be done also for intellectual products.
But two things are paramount to implement Deming's ideas:
belief and committment.

[vulture4]

   Mine costs $60 to replace so I can do it any time....why should I spend $1500 for a Rolls Royce?"

Point taken, but the economics might be different for a launch vehicle.