Author Topic: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle  (Read 12850 times)

Offline Michael Baylor

  • NSF Reporter
  • Full Member
  • ****
  • Posts: 901
  • Liked: 4868
  • Likes Given: 865
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #20 on: 11/10/2018 05:37 pm »
Did they actually use the 14 flight option, or the 6 flight option?  There have been 6 flights of Block 5, and they have a good amount of insight into that vehicle from the Commercial Crew program.
Yeah, I removed that part earlier just to be safe. I was under the impression that the other two options were for new variants of launch vehicles that already had Cat 3 certification. However, D4H got certification with less than 14, and I don't think D4M ever had it?

Offline niwax

  • Full Member
  • ****
  • Posts: 1422
  • Germany
    • SpaceX Booster List
  • Liked: 2040
  • Likes Given: 166
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #21 on: 11/10/2018 07:00 pm »
Did they actually use the 14 flight option, or the 6 flight option?  There have been 6 flights of Block 5, and they have a good amount of insight into that vehicle from the Commercial Crew program.
Yeah, I removed that part earlier just to be safe. I was under the impression that the other two options were for new variants of launch vehicles that already had Cat 3 certification. However, D4H got certification with less than 14, and I don't think D4M ever had it?

I wouldn't be surprised if these rules are more like suggestions when NASA really needs to launch something. Most rockets never had the launch rate for 14 flights in under a few years. And while F9B5 has flown a few times, the vehicle is yet to have the new COPVs. So possibly NASA decided to certify it based on engineering reviews as a side effect of the commercial crew certification.
Which booster has the most soot? SpaceX booster launch history! (discussion)

Offline deruch

  • Senior Member
  • *****
  • Posts: 2422
  • California
  • Liked: 2006
  • Likes Given: 5634
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #22 on: 11/11/2018 04:54 pm »
Just to add a bit of additional clarity, the number of flights requirements are for flights of a "Common Launch Vehicle Configuration" which is a defined term.  And then they go on to define changes that amount to a new configuration and thus restart the counter and changes which are allowable without restarting the counting but still require delta-qualification to remain certified, etc. 

From the outside, without someone from SpaceX or LSP explicitly telling us about the process, we're not going to have a good idea of exactly how this got done.  IMO, debating the hypothetical ways it might have been done seems a less than valuable use of time.  YMMV.
Shouldn't reality posts be in "Advanced concepts"?  --Nomadd

Offline LouScheffer

  • Senior Member
  • *****
  • Posts: 3383
  • Liked: 6111
  • Likes Given: 837
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #23 on: 11/12/2018 12:11 am »
Just wondering. Where does the "14 consecutive successful missions" number come from? For 95% reliability wouldn't you need 19? I count 36 consecutive successful missions so far for SX.
After 14 consecutive successes you have more than 50% confidence in 95% reliability. This is calculated as 0.95^14 = .487% This 48.7% is the average probability of getting at least one failure if the actual reliability is 95%. The confidence level is 100% less this number, or 51.3%.

19 successful missions does not "prove" 95% reliability, it only establishes a 62.3% (100% - 0.95^19) confidence that the true reliability is at least 95%.
That's not how these probabilities work.  [Examples deleted]

I hope this helps illustrate that everything depends on the prior probabilities.  That is, it depends on the population of launch vehicles you are sampling from.  [...]  It's not a very satisfying answer, because we don't really know much about the prior probabilities.  So we have to assume something.  But we should be aware that our calculations are based on these (very uncertain) prior probabilities and be appropriately cautious in our use of the probability figures we get from them.
The '14' is pretty much a made-up number.  I'm sure they have some formula, and when you plug 14 into that formula it gives an acceptable reliability.  But the model itself has lots of assumptions, variables, and parameters, all of which are basically guesses.

As an example where 14 consecutive successes proves little, suppose you have a really cheap launcher that fails 50% of the time.   You keep launching it until you get 14 consecutive successes.  Is it then a reliable launcher?

Even worse, if you are trying for accuracy, is that the probability of failure is not stationary.  (That's statistician-speak for the probability changing over the course of the measurement.)  This can go both directions.  Normally an initial design becomes more reliable as bugs and near-misses are fixed.  But the other way can happen too, as Proton and Soyuz show, where an initially reliable vehicle becomes worse.

Plus there are lots of priors that are hard to capture in a model.   Have the folks who designed it done a similar design before?  Are the technologies used old and well understood, or relatively new?  What's the organizational culture for reporting suspicious events up the management chain?

So I'm sure they have some model, which everyone knows is just guesswork.  But the model says '14', and folks say "that seems about right".  So they require 14 straight successes.
Quote
this is something that nearly everyone who isn't a professional statistician gets wrong.  Probabilities are often non-intuitive.  That's why people hire professional statisticians.
A professional statistician won't be much help here.  The problem is the assumptions behind the model (the priors) and not the mathematics of probability.

Offline ChrisWilson68

  • Senior Member
  • *****
  • Posts: 5266
  • Sunnyvale, CA
  • Liked: 4992
  • Likes Given: 6459
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #24 on: 11/15/2018 07:42 pm »
This is all about comparing the odds that you've gotten this result because it's a reliable vehicle versus that you've gotten this result by chance even though the vehicle is not reliable.

Which is why the confidence is 50%. There is still an equal chance that the actual reliability is less than 95%, including due to weird population distributions.

Incorrect.  See my previous posts where I worked out concrete examples where this is not true.

I am not a reliability engineer, but I do know from working with reliability engineers that this is how they calculate confidence levels for reliability in populations with unknown distributions.

You're misunderstanding what they're doing.  They're not calculating it this way because these calculations give numbers that are true no matter what the prior distributions are.  They're calculating it this way because, if they want to get numbers, they need to assume *something* about the priors.  These calculations build in a particular set of assumptions about the priors.

That's the whole danger I was trying to point out: people see formulas like this and they misunderstand them.  A lot of people think what you are claiming here, that these formulas give results that are true no matter what the priors are.  But that's not true.  The formulas assume something about the priors.  They have to.  They assume something about the priors that is a useful heuristic.

There's nothing wrong with doing a calculation like that as long as you understand what it actually means and don't get overconfident about the numbers it gives.
« Last Edit: 11/15/2018 07:43 pm by ChrisWilson68 »

Offline ChrisWilson68

  • Senior Member
  • *****
  • Posts: 5266
  • Sunnyvale, CA
  • Liked: 4992
  • Likes Given: 6459
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #25 on: 11/15/2018 07:50 pm »
Quote
this is something that nearly everyone who isn't a professional statistician gets wrong.  Probabilities are often non-intuitive.  That's why people hire professional statisticians.
A professional statistician won't be much help here.  The problem is the assumptions behind the model (the priors) and not the mathematics of probability.

Thanks for the very good post.  You brought up a lot of other issues with reliability predictions that are also valid issues beyond the one narrow point I was making.

I agree that professional statisticians won't have the domain expertise to give the correct assumptions to use in the models.  But they can help people understand where there are assumptions behind the model and what the model really captures.  You seem to have a good handle on it, but many people, even some smart people who are knowledgeable in their fields do not understand all the subtleties of statistical models.

Offline envy887

  • Senior Member
  • *****
  • Posts: 8144
  • Liked: 6801
  • Likes Given: 2965
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #26 on: 11/16/2018 03:00 am »
This is all about comparing the odds that you've gotten this result because it's a reliable vehicle versus that you've gotten this result by chance even though the vehicle is not reliable.

Which is why the confidence is 50%. There is still an equal chance that the actual reliability is less than 95%, including due to weird population distributions.

Incorrect.  See my previous posts where I worked out concrete examples where this is not true.

Your examples assume a particular distribution of reliabilities, the distribution you explicitly state. This changes the probability of a given reliability by eliminating many other distributions that are also possible. Since we don't actually know the distribution, we have to also account for the probability of LV reliabilities actually having that particular distribution. In the case of your examples, the likelihood of that distribution is right around 0, based on what we observe from other launch vehicles.

Quote
I am not a reliability engineer, but I do know from working with reliability engineers that this is how they calculate confidence levels for reliability in populations with unknown distributions.

You're misunderstanding what they're doing.

No, I think you're misunderstanding what I wrote. Evidently I wasn't clear enough.

Quote
They're not calculating it this way because these calculations give numbers that are true no matter what the prior distributions are.  They're calculating it this way because, if they want to get numbers, they need to assume *something* about the priors.  These calculations build in a particular set of assumptions about the priors.

I understand that just fine, and never indicated otherwise.

Quote
A lot of people think what you are claiming here, that these formulas give results that are true no matter what the priors are.

Say what? ??? I did not claim that. I do not think that. I'm well aware of the many assumptions that go into reliability calculations and how they are often wildly inaccurate.

Quote
But that's not true.  The formulas assume something about the priors.  They have to.  They assume something about the priors that is a useful heuristic.

There's nothing wrong with doing a calculation like that as long as you understand what it actually means and don't get overconfident about the numbers it gives.

I didn't say that they aren't making assumptions about the prior distributions. I said they don't KNOW the prior distributions.

They assume that all reliabilities are possible and equally likely. This is not exactly true, but it's a decent approximation, and is entirely reasonable based on what we see with existing vehicles. It's far more reasonable than the rest of the assumptions going into the formula, and particularly compared to the one about launches being independent events.
« Last Edit: 11/16/2018 01:28 pm by envy887 »

Tags:
 

Advertisement NovaTech
Advertisement Northrop Grumman
Advertisement
Advertisement Margaritaville Beach Resort South Padre Island
Advertisement Brady Kenniston
Advertisement NextSpaceflight
Advertisement Nathan Barker Photography
0