Author Topic: SpaceX Falcon 9 v1.1 - SpX-7/CRS-7 DRAGON FAILURE - Discussion Thread 2  (Read 564331 times)

Offline AJW

  • Full Member
  • ****
  • Posts: 596
  • Liked: 490
  • Likes Given: 63
Was discussing how in the aircraft industry, prototypes often include significant numbers of additional sensors and data recorders but this approach doesn't work as well for SpaceX when your stages are lost.   What I learned was that GH2 had far more sensors than other Falcons and that the philosophy was that if you wanted to add a sensor for testing, there were few restrictions so long as there was bandwidth.

The loss of GH2 may have had more impact than many realized.

Offline Alastor

  • Full Member
  • **
  • Posts: 272
  • Liked: 148
  • Likes Given: 330
A lot of people on this thread seem to forget that no one ever should expect a complex system such as a rocket to be a 100% reliable.
We don't for a plane, we don't for a car, and we shouldn't for a rocket.

We are here speaking of a rocket that was not manned, so that the consequences of a LOV (or LOM for that matter) wouldn't be much worse that a loss of money and time for the parties involved. Every one in the industry aknowledges and accepts this as a risk.
Even for manned rockets there is a degree of risk involved that the astronauts accept.

So the question is not and shouldn't be how the vehicle or the process should be designed so that it never ever fails.
It seems here that the design was a 2x security factor. That means that they expect some struts to fail below that. Do they expect some to fail at 60% load ? Well, it is possible that they would expect it as an extreme event, after all, that leaves a 10% margin.
But ultimately, wether you design the strut to hold a 1000%, or put 10 struts instead of one, there is always a low probalility failiure. And at some point this probability will be low enough that some other part of the vehicle will have a higher probability of failiure. In the end, though, there is always something that will cause the vehicle to fail a some point in the future.

What is really in question here is wether or not they did take reasonable measures (compared to the cost in both money and time (so money) incured by a loss) to prevent such a failiure, and if they can take reasonable measures (same criteria) to improve the reliability of the spacecraft.

Sure, this question becomes a bit more messy once you put maned spacecraft into the equation, because putting a cost next to a human life is always something tricky (and here, they plan to use the vehicle for manned spacecraft, so the point might be somewhat valid). But in the end, imaginning that anything has a 0.00000...% probability of failiure is just pure delusion.

I thik we should really start speaking about how much it costs to ensure reliability of these parts and how much it would cost to improve it, rather than waving hands about wether or not it would be feasible to scan each and every strut to the molecular level to look if atom A is in the right spot or not (okay, I may be caricaturing a little).
My point is, we can do a lot of things when it comes to materials testing ... But that does not mean that evertything is a realistic technique to be used !

Offline Rocket Science

  • Senior Member
  • *****
  • Posts: 8927
  • NASA Educator Astronaut Candidate Applicant 2002
  • Liked: 2823
  • Likes Given: 7437
Elon has been on record from the beginning that a rocket need not cost so much and doesn’t need exotic components and should make use of “off the shelf” items... Google it and read for yourselves... "That's all I have to say about that..." to quote Forest Gump...
"The laws of physics are unforgiving"
~Rob: Physics instructor, Aviator, Vintage auto racer

Offline LouScheffer

  • Full Member
  • ****
  • Posts: 1908
  • Liked: 2496
  • Likes Given: 279
As for critical bolts - like on bridges (looked this up) state governments require sampling and testing of every lot (purchase order) by an independent QA house.
Like any other statistical test, this is making critical assumptions on the distribution of errors.  If your bolts are in general too weak, this is the right test.  But if 99 of every 100 bolts are perfect, and one is completely and utterly defective, then some lots will pass with a perfect score and yet you'll be building bridges with at least some defective bolts.

I suspect this is exactly what happened here.  The supplier's process, for whatever reason, produced struts where the vast majority are fine, but the ones that were defective were very defective.  (One could imagine lots of reasons for this - a few counterfeit bolts mixed with real bolts, a heat-treating machine that intermittently fails, etc.).  Any QA process that uses sampling can miss this. 

Unfortunately, the only way to *know* the distribution is to test every bolt.  And then you don't need to know the distribution.

Offline baldusi

  • Senior Member
  • *****
  • Posts: 7437
  • Buenos Aires, Argentina
  • Liked: 1446
  • Likes Given: 4499
Actually, there's another statistical explanation. The bolt's strength is distributed like a normal (or probably like a Chi). So, the statistical sample is not big enough to catch the outliers.

Offline cscott

  • Senior Member
  • *****
  • Posts: 2949
  • Liked: 2055
  • Likes Given: 664
Actually, there's another statistical explanation. The bolt's strength is distributed like a normal (or probably like a Chi). So, the statistical sample is not big enough to catch the outliers.
SpaceX certainly has tested enough bolts by this point to know whether the strength distribution is normal/Chi.  From the tiny bit of data we have ("some" failed at 6000lb, 2 failed at 2000lb) I think it's unlikely it's a normal distribution: you'd expect to see more failures between 2000-6000lb before you saw another 2000lb failure.  You could compute an exact probability given some reasonable assumptions.

Offline johnx98374

  • Member
  • Posts: 94
  • USA
  • Liked: 67
  • Likes Given: 40
As for critical bolts - like on bridges (looked this up) state governments require sampling and testing of every lot (purchase order) by an independent QA house.
Like any other statistical test, this is making critical assumptions on the distribution of errors.  If your bolts are in general too weak, this is the right test.  But if 99 of every 100 bolts are perfect, and one is completely and utterly defective, then some lots will pass with a perfect score and yet you'll be building bridges with at least some defective bolts.

I suspect this is exactly what happened here.  The supplier's process, for whatever reason, produced struts where the vast majority are fine, but the ones that were defective were very defective.  (One could imagine lots of reasons for this - a few counterfeit bolts mixed with real bolts, a heat-treating machine that intermittently fails, etc.).  Any QA process that uses sampling can miss this. 

Unfortunately, the only way to *know* the distribution is to test every bolt.  And then you don't need to know the distribution.

I agree random defects are possible, but I hope buying space hardware is not like playing Russian roulette.

Unless Musk has thrown the QA book out the window, there are many other controls required.  There is a lot more record keeping and QA required.  Parts are built in lots with date codes.  Factory records are kept for the lots.  Parts are serialized.  Do you buy food that doesn't have the plant number, date code, lot code and other info on it? 

Musk did not say "after testing an enormous number of parts -- all parts met the x pound spec except for two samples that failed at 6000 pounds and 2000 pounds.   These are random unexplainable defects that our best quality assurance and statistical sampling were unable to detect." 

Here is what Musk actually said:
Quote
40:46 Elon Musk: … we tested a whole bunch of struts, none of them failed at a level, like the lowest we saw anyone fail was at six thousand pounds of force…… So we thought well, that couldn't be the issue…

... Then we got just a huge number of these struts with this particular bolt and after testing some enormous number we were able to find one that failed below the two thousand pound level. So it was sort of a statistical thing

IMHO, our industrial civilization would not be possible if all manufacturing processes randomly produce bad parts.  Especially critical parts.
« Last Edit: 08/07/2015 09:36 PM by johnx98374 »

Offline baldusi

  • Senior Member
  • *****
  • Posts: 7437
  • Buenos Aires, Argentina
  • Liked: 1446
  • Likes Given: 4499
Well, the fact I that these are big numbers. Testing 1000s of bolts to characterize the outliers es very difficult. Let's say that they use 100 bolts and plan to launch 100 time with this design. They would have to test 200.000 bolts to get any significant confidence that they won't get an outlier in the life of the program. And then any change in the manufacturing process will invalidate the data. It's quite easier work with an individual acceptance test.

Offline Mike_1179

  • Member
  • Full Member
  • ****
  • Posts: 562
  • New Jersey
  • Liked: 231
  • Likes Given: 51
 Depending on what they're doing, there sampling might not have been designed to detect defects. Sometimes all your sampling is doing, is trying to ensure that you followed a validated process. If you have a big lot of parts, you're really pretty low on th O-C curve if you only test a few samples out of that lot

LQs are generally really low on lot acceptance tests - you're not going to find that one random defect in sampling. That's what making sure you'd manufacturing process is in control is all about.

Offline Prober

  • Senior Member
  • *****
  • Posts: 10313
  • Save the spin....I'm keeping you honest!
  • Nevada
  • Liked: 702
  • Likes Given: 728
Why are we have this same line of questions about the struts and bolts?  Sorry but its getting old :o :(

https://en.wikipedia.org/wiki/Statistical_process_control

"Statistical process control (SPC) is a method of quality control which uses statistical methods. SPC is applied in order to monitor and control a process. Monitoring and controlling the process ensures that it operates at its full potential. At its full potential, the process can make as much conforming product as possible with a minimum (if not an elimination) of waste (rework or scrap). SPC can be applied to any process where the "conforming product" (product meeting specifications) output can be measured. Key tools used in SPC include control charts; a focus on continuous improvement; and the design of experiments. An example of a process where SPC is applied is manufacturing lines."

Emphasis on early detection

An advantage of SPC over other methods of quality control, such as "inspection", is that it emphasizes early detection and prevention of problems, rather than the correction of problems after they have occurred.




2017 - Everything Old is New Again.
I have not failed. I've just found 10,000 ways that won't work. ~ by Thomas Alva Edison

Offline baldusi

  • Senior Member
  • *****
  • Posts: 7437
  • Buenos Aires, Argentina
  • Liked: 1446
  • Likes Given: 4499
I'm an econometrist. Threr's nothing more dangerous than trusting statistical methods. And if those are spelled in an international specification, it is like playing with fire. You use them because it is the only way forward most of the time. But you have to be awfully aware of its limitations and underlying assumptions. It is the frequency of the outliers that appear to be the problem here. It is a very specific statistical problem where those not used to develop their own PDF, tend to assume normality, weibull or something. Even the non parametric models have assumptions. They already did acceptance test of pneumatic valves with a working pressure of 600psi at 100psi (CRS-1 anomaly).

Offline pargoo

  • Lifelong space fan
  • Full Member
  • ****
  • Posts: 693
  • Australia
    • Buran - wait, the Russians had a Space Shuttle?
  • Liked: 76
  • Likes Given: 1
Maybe OT, but still no close-up launch pics from SpaceX.  I've been checking daily...

Offline johnx98374

  • Member
  • Posts: 94
  • USA
  • Liked: 67
  • Likes Given: 40
Actually, there's another statistical explanation. The bolt's strength is distributed like a normal (or probably like a Chi). So, the statistical sample is not big enough to catch the outliers.
SpaceX certainly has tested enough bolts by this point to know whether the strength distribution is normal/Chi.  From the tiny bit of data we have ("some" failed at 6000lb, 2 failed at 2000lb) I think it's unlikely it's a normal distribution: you'd expect to see more failures between 2000-6000lb before you saw another 2000lb failure.  You could compute an exact probability given some reasonable assumptions.

Musk didn't say those were the only two parts that didn't meet the spec.  I interpret his words as meaning there was a distribution -- "it was sort of a statistical thing."  If statistical 'outliers' fall outside the worst case specifications then there is something very wrong - you can't should not design with parts like that.

Maybe another explanation that fits your outlier conjecture:  I have seen many designs in the electronic world where the designer claims he can not build some widget based on worst case part specs.  He will rationalize based on his experience, other past experience, common sense - worst case numbers not realistic, cost, or how you'd need this and this unlikely event etc.  I have also seen cases where the designer misinterpreted part specs and the same rationale is used to justify use as is.

Offline Lee Jay

  • Elite Veteran
  • Global Moderator
  • Senior Member
  • *****
  • Posts: 6683
  • Liked: 982
  • Likes Given: 140
I think this is interesting:

http://naca.central.cranfield.ac.uk/reports/arc/rm/3654.pdf

Table 2, steel bolts in tension on a nut.

Coefficient of Variation (standard deviation / mean) is listed at 2%-7%.  That's pretty good.

If that represents the real variability, the likelihood of a part being 60-80% below the mean is vanishingly small.  Either the CoV is really much too large, or they have a process that can produce a bad batch or a bad single parts.

Offline johnx98374

  • Member
  • Posts: 94
  • USA
  • Liked: 67
  • Likes Given: 40
Why are we have this same line of questions about the struts and bolts?  Sorry but its getting old :o :(

It just restarts every 15 pages.  Interpreting Musk's words is like the Bible interpretation that's been going on for 3000+ years.  We already have different religions now.  We need some new words or signs from higher up. :)

Offline Prober

  • Senior Member
  • *****
  • Posts: 10313
  • Save the spin....I'm keeping you honest!
  • Nevada
  • Liked: 702
  • Likes Given: 728
I think this is interesting:

http://naca.central.cranfield.ac.uk/reports/arc/rm/3654.pdf

Table 2, steel bolts in tension on a nut.

Coefficient of Variation (standard deviation / mean) is listed at 2%-7%.  That's pretty good.

If that represents the real variability, the likelihood of a part being 60-80% below the mean is vanishingly small.  Either the CoV is really much too large, or they have a process that can produce a bad batch or a bad single parts.

you might find this interesting from Honda, and the F9 is way more complex than an auto.
This video demonstrates how Honda Motor Corporation uses Statistical Process Control to monitor Quality.



2017 - Everything Old is New Again.
I have not failed. I've just found 10,000 ways that won't work. ~ by Thomas Alva Edison

Offline Brovane

  • Full Member
  • ****
  • Posts: 1110
  • United States
  • Liked: 605
  • Likes Given: 1143
'Congress asks question that makes them appear silly.' Should be the headline. 

First  there's an engine explosion on the stand, then a propulsive failure on the vehicle that involved an engine failure indicating a failure to fix the root cause applied on top of the inherent doubt about using a 30 year old engine. 

Then you have a first time LOM with a vehicle with a much more solid track record ... Gee I wonder why they are being treated somewhat differently.

Further, the finger pointing war between gencorp and orbital(atk) certainly doesn't suggest the root cause can be settled without another opinion. 

Someone more familiar with the details feel free to correct any misperceptions I have.

It isn't a silly question.  I am very interested to see what NASA's official response to this is. 

"Look at that! If anybody ever said, "you'll be sitting in a spacecraft naked with a 134-pound backpack on your knees charging it", I'd have said "Aw, get serious". - John Young - Apollo-16

Offline Chris Bergin

What a dire end to the thread. Boring, so locking it (PS Not just a random Chris drive by - was sent here by report to mods. I don't do drive bys, too busy ;D ). RTF is where this is now heading, which includes the investigation results as that feeds into RTF.

Take politics to Space Policy or another site. Most of you should know better. ;)

Technically the next discussion thread in the path:
http://forum.nasaspaceflight.com/index.php?topic=38149.0

(And the plan was always to keep thread going for the interim before moving to that new thread, so let's do that now).
« Last Edit: 08/08/2015 09:24 AM by Chris Bergin »

Tags: