Author Topic: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle  (Read 7000 times)

Offline Chris Bergin

SpaceX presser:

__
NASA’s Launch Services Program (LSP) has certified Falcon 9 as a Category 3 launch vehicle. Category 3 launch vehicles are certified to support NASA’s highest cost and most complex scientific missions. The following statement is from SpaceX President and Chief Operating Officer Gwynne Shotwell:

“LSP Category 3 certification is a major achievement for the Falcon 9 team and represents another key milestone in our close partnership with NASA. We are honored to have the opportunity to provide cost-effective and reliable launch services to the country’s most critical scientific payloads."

The process of designating launch vehicles as Category 3 is designed to assure the highest practicable probability of success. Falcon 9 has completed over 60 missions, including the NASA LSP missions Jason-3 and TESS.
__

« Last Edit: 11/08/2018 08:25 pm by Chris Bergin »

Online gongora

  • Global Moderator
  • Senior Member
  • *****
  • Posts: 4579
  • US
  • Liked: 4141
  • Likes Given: 2358
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #1 on: 11/08/2018 08:28 pm »

Offline TGMetsFan98

Cat 3 is required to fly crew, so F9 was going to need this certification soon. Would like to know if this covers nuclear payloads, though.
"Mankind was born on Earth. It was never meant to die here." -Coop

Offline whitelancer64

Cat 3 is required to fly crew, so F9 was going to need this certification soon. Would like to know if this covers nuclear payloads, though.

It is a prerequisite.

In addition to the Category 3 rating, a NASA nuclear certification requires an additional review of the rocket's self-destruct system, and an analysis of what would happen to the nuclear payload in that event.
"One bit of advice: it is important to view knowledge as sort of a semantic tree -- make sure you understand the fundamental principles, ie the trunk and big branches, before you get into the leaves/details or there is nothing for them to hang on to." - Elon Musk
"There are lies, damned lies, and launch schedules." - Larry J

Offline Chris Bergin

Word is nuclear payloads need separate certification.

Edit: Whitelancer beat me to it.  ;D
« Last Edit: 11/08/2018 09:07 pm by Chris Bergin »

Online DigitalMan

  • Full Member
  • ****
  • Posts: 667
  • Liked: 185
  • Likes Given: 11
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #5 on: 11/08/2018 10:01 pm »
Word is nuclear payloads need separate certification.

Edit: Whitelancer beat me to it.  ;D

If nuclear certification requires more insight into the destruct system, I wonder if the best path for SpaceX is to pursue the removal of the destruct system while still meeting requirements for range safety.  Originally they didn't want to have one, no?

Offline speedevil

  • Senior Member
  • *****
  • Posts: 3316
  • Fife
  • Liked: 1770
  • Likes Given: 2151
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #6 on: 11/08/2018 10:02 pm »
The last line, giving 14 consecutive successes as a route to Cat 3 is an interesting one in the context of possibly highly reusable vehicles.
 
A great milestone for F9, let's hope it only makes another hundred or so flights :)

Offline J-V

  • Full Member
  • *
  • Posts: 100
  • Liked: 29
  • Likes Given: 35
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #7 on: 11/09/2018 06:18 am »
I wonder how much recovered cores helped this process. Examining the booster post-flight can certainly tell you things telemetry can't, but is NASA willing to use this data is another question.

Do we have any (un)official information on this?

Offline woods170

  • IRAS fan
  • Senior Member
  • *****
  • Posts: 9048
  • IRAS fan
  • The Netherlands
  • Liked: 6495
  • Likes Given: 2212
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #8 on: 11/09/2018 08:45 am »
Cat 3 is required to fly crew, so F9 was going to need this certification soon. Would like to know if this covers nuclear payloads, though.

Wrong.

Launch Services Program (LSP) is for UNmanned launch services only.

Certifying F9 to carry a crewed vehicle is NOT dependent on, nor tied to, LSP Category 3 certification.

CCP does its own certification of F9, independent from LSP certification efforts. The requirements for both types of certification differ substantially.

The only connection between the two (LSP and CCP) is that both programs share the available information on launch vehicles so that each program gains exceptional insight with regards to those launch vehicles.
« Last Edit: 11/09/2018 08:53 am by woods170 »


Offline psherriffs

  • Member
  • Posts: 3
  • Cebu, Philippines
  • Liked: 0
  • Likes Given: 1
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #10 on: 11/09/2018 10:05 pm »
Just wondering. Where does the "14 consecutive successful missions" number come from? For 95% reliability wouldn't you need 19? I count 36 consecutive successful missions so far for SX.

Offline envy887

  • Senior Member
  • *****
  • Posts: 5600
  • Liked: 3410
  • Likes Given: 1660
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #11 on: 11/09/2018 10:35 pm »
Just wondering. Where does the "14 consecutive successful missions" number come from? For 95% reliability wouldn't you need 19? I count 36 consecutive successful missions so far for SX.

After 14 consecutive successes you have more than 50% confidence in 95% reliability. This is calculated as 0.95^14 = .487% This 48.7% is the average probability of getting at least one failure if the actual reliability is 95%. The confidence level is 100% less this number, or 51.3%.

19 successful missions does not "prove" 95% reliability, it only establishes a 62.3% (100% - 0.95^19) confidence that the true reliability is at least 95%.

Offline ChrisWilson68

  • Senior Member
  • *****
  • Posts: 3881
  • Sunnyvale, CA
  • Liked: 2602
  • Likes Given: 3373
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #12 on: 11/09/2018 11:32 pm »
Just wondering. Where does the "14 consecutive successful missions" number come from? For 95% reliability wouldn't you need 19? I count 36 consecutive successful missions so far for SX.

After 14 consecutive successes you have more than 50% confidence in 95% reliability. This is calculated as 0.95^14 = .487% This 48.7% is the average probability of getting at least one failure if the actual reliability is 95%. The confidence level is 100% less this number, or 51.3%.

19 successful missions does not "prove" 95% reliability, it only establishes a 62.3% (100% - 0.95^19) confidence that the true reliability is at least 95%.

That's not how these probabilities work.

To illustrate the point, lets take an extreme, unrealistic example.  Lets assume that half of all launch vehicles are 95% reliable and all the other half are 90% reliable.

You don't know if this particular vehicle is 90% reliable or 95% reliable.  So lets look at how likely it is that it's 95% reliable versus 90% reliable.

If it happens that the vehicle is 95% reliable and you launch 14 times, the odds all 14 are successful are .488, as calculated above.  If it happens the vehicle is 90% reliable and you lauch 14 times, the odds all 14 are successful are .229 (.9^14).

Since the 90% vehicles and 95% vehicles are equally represented in the population from which you are sampling, the odds you have a 95% reliable vehicle are .488 / (.488 + .229) = 0.680.  So the odds you have a 95% reliable vehicle are 0.68.

Now, suppose instead that 95%-reliable are one in a billion and that the others are all 90%-reliable.  Now, if you see 14 launches in a row that are successful, the odds you have a 95%-reliable vehicle are .488 / (.488 + (.229 * (10^9 - 1)), which is approximately one in a billion.

I hope this helps illustrate that everything depends on the prior probabilities.  That is, it depends on the population of launch vehicles you are sampling from.  If launch vehicles in general are likely to be reliable, you need far fewer launches to be confident in a new launch vehicle than if launch vehicles in general are unlikely to be reliable.

It's not a very satisfying answer, because we don't really know much about the prior probabilities.  So we have to assume something.  But we should be aware that our calculations are based on these (very uncertain) prior probabilities and be appropriately cautious in our use of the probability figures we get from them.

By the way, this is something that nearly everyone who isn't a professional statistician gets wrong.  Probabilities are often non-intuitive.  That's why people hire professional statisticians.

Offline envy887

  • Senior Member
  • *****
  • Posts: 5600
  • Liked: 3410
  • Likes Given: 1660
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #13 on: 11/10/2018 01:05 am »
Just wondering. Where does the "14 consecutive successful missions" number come from? For 95% reliability wouldn't you need 19? I count 36 consecutive successful missions so far for SX.

After 14 consecutive successes you have more than 50% confidence in 95% reliability. This is calculated as 0.95^14 = .487% This 48.7% is the average probability of getting at least one failure if the actual reliability is 95%. The confidence level is 100% less this number, or 51.3%.

19 successful missions does not "prove" 95% reliability, it only establishes a 62.3% (100% - 0.95^19) confidence that the true reliability is at least 95%.

That's not how these probabilities work.

To illustrate the point, lets take an extreme, unrealistic example.  Lets assume that half of all launch vehicles are 95% reliable and all the other half are 90% reliable.

You don't know if this particular vehicle is 90% reliable or 95% reliable.  So lets look at how likely it is that it's 95% reliable versus 90% reliable.

If it happens that the vehicle is 95% reliable and you launch 14 times, the odds all 14 are successful are .488, as calculated above.  If it happens the vehicle is 90% reliable and you lauch 14 times, the odds all 14 are successful are .229 (.9^14).

Since the 90% vehicles and 95% vehicles are equally represented in the population from which you are sampling, the odds you have a 95% reliable vehicle are .488 / (.488 + .229) = 0.680.  So the odds you have a 95% reliable vehicle are 0.68.

Now, suppose instead that 95%-reliable are one in a billion and that the others are all 90%-reliable.  Now, if you see 14 launches in a row that are successful, the odds you have a 95%-reliable vehicle are .488 / (.488 + (.229 * (10^9 - 1)), which is approximately one in a billion.

I hope this helps illustrate that everything depends on the prior probabilities.  That is, it depends on the population of launch vehicles you are sampling from.  If launch vehicles in general are likely to be reliable, you need far fewer launches to be confident in a new launch vehicle than if launch vehicles in general are unlikely to be reliable.

It's not a very satisfying answer, because we don't really know much about the prior probabilities.  So we have to assume something.  But we should be aware that our calculations are based on these (very uncertain) prior probabilities and be appropriately cautious in our use of the probability figures we get from them.

By the way, this is something that nearly everyone who isn't a professional statistician gets wrong.  Probabilities are often non-intuitive.  That's why people hire professional statisticians.

Ok, I meant "more than 50% confidence in 95% reliability OR BETTER". It's not a >50% confidence that the reliability is exactly 95%, just a >50% confidence that reliability is higher than 95%. You don't need to know the reliability population distribution of all launch vehicles to establish this after 14 consecutive successes of the same design.


Offline ChrisWilson68

  • Senior Member
  • *****
  • Posts: 3881
  • Sunnyvale, CA
  • Liked: 2602
  • Likes Given: 3373
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #14 on: 11/10/2018 01:24 am »
Just wondering. Where does the "14 consecutive successful missions" number come from? For 95% reliability wouldn't you need 19? I count 36 consecutive successful missions so far for SX.

After 14 consecutive successes you have more than 50% confidence in 95% reliability. This is calculated as 0.95^14 = .487% This 48.7% is the average probability of getting at least one failure if the actual reliability is 95%. The confidence level is 100% less this number, or 51.3%.

19 successful missions does not "prove" 95% reliability, it only establishes a 62.3% (100% - 0.95^19) confidence that the true reliability is at least 95%.

That's not how these probabilities work.

To illustrate the point, lets take an extreme, unrealistic example.  Lets assume that half of all launch vehicles are 95% reliable and all the other half are 90% reliable.

You don't know if this particular vehicle is 90% reliable or 95% reliable.  So lets look at how likely it is that it's 95% reliable versus 90% reliable.

If it happens that the vehicle is 95% reliable and you launch 14 times, the odds all 14 are successful are .488, as calculated above.  If it happens the vehicle is 90% reliable and you lauch 14 times, the odds all 14 are successful are .229 (.9^14).

Since the 90% vehicles and 95% vehicles are equally represented in the population from which you are sampling, the odds you have a 95% reliable vehicle are .488 / (.488 + .229) = 0.680.  So the odds you have a 95% reliable vehicle are 0.68.

Now, suppose instead that 95%-reliable are one in a billion and that the others are all 90%-reliable.  Now, if you see 14 launches in a row that are successful, the odds you have a 95%-reliable vehicle are .488 / (.488 + (.229 * (10^9 - 1)), which is approximately one in a billion.

I hope this helps illustrate that everything depends on the prior probabilities.  That is, it depends on the population of launch vehicles you are sampling from.  If launch vehicles in general are likely to be reliable, you need far fewer launches to be confident in a new launch vehicle than if launch vehicles in general are unlikely to be reliable.

It's not a very satisfying answer, because we don't really know much about the prior probabilities.  So we have to assume something.  But we should be aware that our calculations are based on these (very uncertain) prior probabilities and be appropriately cautious in our use of the probability figures we get from them.

By the way, this is something that nearly everyone who isn't a professional statistician gets wrong.  Probabilities are often non-intuitive.  That's why people hire professional statisticians.

Ok, I meant "more than 50% confidence in 95% reliability OR BETTER". It's not a >50% confidence that the reliability is exactly 95%, just a >50% confidence that reliability is higher than 95%. You don't need to know the reliability population distribution of all launch vehicles to establish this after 14 consecutive successes of the same design.

But that's still not true.  Take my example where only one launch vehicle design in a billion is 95% reliable and the rest are 90% reliable.  In this case, even if you have 14 consecutive successes with the same design, the odds you've gotten that one 95% reliable design in a billion is only 2 point something in a billion.

This is all about comparing the odds that you've gotten this result because it's a reliable vehicle versus that you've gotten this result by chance even though the vehicle is not reliable.  And that is very sensitive to how likely the vehicle was to be reliable in the first place.

Offline envy887

  • Senior Member
  • *****
  • Posts: 5600
  • Liked: 3410
  • Likes Given: 1660
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #15 on: 11/10/2018 12:57 pm »
This is all about comparing the odds that you've gotten this result because it's a reliable vehicle versus that you've gotten this result by chance even though the vehicle is not reliable.

Which is why the confidence is 50%. There is still an equal chance that the actual reliability is less than 95%, including due to weird population distributions.

I am not a reliability engineer, but I do know from working with reliability engineers that this is how they calculate confidence levels for reliability in populations with unknown distributions.

Online gongora

  • Global Moderator
  • Senior Member
  • *****
  • Posts: 4579
  • US
  • Liked: 4141
  • Likes Given: 2358
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #16 on: 11/10/2018 01:49 pm »
Did they actually use the 14 flight option, or the 6 flight option?  There have been 6 flights of Block 5, and they have a good amount of insight into that vehicle from the Commercial Crew program.

Online oldAtlas_Eguy

  • Senior Member
  • *****
  • Posts: 3608
  • Florida
  • Liked: 2073
  • Likes Given: 247
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #17 on: 11/10/2018 02:18 pm »
Does the cat 3 cert include FH? If so it clears they way for FH as a candidate LV for Europa Clippper. Else FH would still need a cat 3 cert.

Offline hopalong

  • Full Member
  • *
  • Posts: 136
  • Milton Keynes
  • Liked: 68
  • Likes Given: 53
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #18 on: 11/10/2018 02:32 pm »
Does the cat 3 cert include FH? If so it clears they way for FH as a candidate LV for Europa Clippper. Else FH would still need a cat 3 cert.

My understanding it does not. But FH could gain Cat 3 late next year (min 3 flights, plus Falcon 9 heritage).

Offline joek

  • Senior Member
  • *****
  • Posts: 3128
  • Liked: 874
  • Likes Given: 444
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #19 on: 11/10/2018 03:31 pm »
Does the cat 3 cert include FH? If so it clears they way for FH as a candidate LV for Europa Clippper. Else FH would still need a cat 3 cert.
My understanding it does not. But FH could gain Cat 3 late next year (min 3 flights, plus Falcon 9 heritage).

No, this definitely does not include FH.  As to leveraging F9 heritage, maybe.  Previously NPD 8610 included (emphasis added):
Quote from: NPD 86.10 (old)
3 consecutive successful flights of a common launch vehicle configuration from an evolved vehicle family developed by an LSC with a previously certified launch vehicle for Risk Category 3, instrumented to provide design verification & flight performance data
Some of that has been removed... 
Quote from: NPD 86.10 (current)
3 (minimum 2 consecutive) successful flights of a common launch vehicle configuration, instrumented to provide design verification & flight performance data

However it also includes the following, which allows for a more tailored approach to certification...
Quote from: NPD 86.10 (current)
A.7 NASA IV&V is a comprehensive assessment of the common launch vehicle analyses conducted by the launch service contractor. It will be based on the complexities and heritage of the vehicle. The LSP will propose the appropriate level of technical assessment (LSP independent modeling, launch service contractor model evaluation, and/or launch service contractor analytical review) at a FPB used to assess the certification strategy for a specified common launch vehicle configuration

Online Michael Baylor

  • Full Member
  • ****
  • Posts: 626
  • Liked: 1767
  • Likes Given: 723
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #20 on: 11/10/2018 05:37 pm »
Did they actually use the 14 flight option, or the 6 flight option?  There have been 6 flights of Block 5, and they have a good amount of insight into that vehicle from the Commercial Crew program.
Yeah, I removed that part earlier just to be safe. I was under the impression that the other two options were for new variants of launch vehicles that already had Cat 3 certification. However, D4H got certification with less than 14, and I don't think D4M ever had it?

Offline niwax

  • Full Member
  • ****
  • Posts: 411
  • Germany
    • SpaceX Booster List
  • Liked: 280
  • Likes Given: 59
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #21 on: 11/10/2018 07:00 pm »
Did they actually use the 14 flight option, or the 6 flight option?  There have been 6 flights of Block 5, and they have a good amount of insight into that vehicle from the Commercial Crew program.
Yeah, I removed that part earlier just to be safe. I was under the impression that the other two options were for new variants of launch vehicles that already had Cat 3 certification. However, D4H got certification with less than 14, and I don't think D4M ever had it?

I wouldn't be surprised if these rules are more like suggestions when NASA really needs to launch something. Most rockets never had the launch rate for 14 flights in under a few years. And while F9B5 has flown a few times, the vehicle is yet to have the new COPVs. So possibly NASA decided to certify it based on engineering reviews as a side effect of the commercial crew certification.
Which booster has the most soot? SpaceX booster launch history! (discussion)

Offline deruch

  • Senior Member
  • *****
  • Posts: 2426
  • California
  • Liked: 1990
  • Likes Given: 5506
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #22 on: 11/11/2018 04:54 pm »
Just to add a bit of additional clarity, the number of flights requirements are for flights of a "Common Launch Vehicle Configuration" which is a defined term.  And then they go on to define changes that amount to a new configuration and thus restart the counter and changes which are allowable without restarting the counting but still require delta-qualification to remain certified, etc. 

From the outside, without someone from SpaceX or LSP explicitly telling us about the process, we're not going to have a good idea of exactly how this got done.  IMO, debating the hypothetical ways it might have been done seems a less than valuable use of time.  YMMV.
Shouldn't reality posts be in "Advanced concepts"?  --Nomadd

Offline LouScheffer

  • Full Member
  • ****
  • Posts: 1998
  • Liked: 2780
  • Likes Given: 299
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #23 on: 11/12/2018 12:11 am »
Just wondering. Where does the "14 consecutive successful missions" number come from? For 95% reliability wouldn't you need 19? I count 36 consecutive successful missions so far for SX.
After 14 consecutive successes you have more than 50% confidence in 95% reliability. This is calculated as 0.95^14 = .487% This 48.7% is the average probability of getting at least one failure if the actual reliability is 95%. The confidence level is 100% less this number, or 51.3%.

19 successful missions does not "prove" 95% reliability, it only establishes a 62.3% (100% - 0.95^19) confidence that the true reliability is at least 95%.
That's not how these probabilities work.  [Examples deleted]

I hope this helps illustrate that everything depends on the prior probabilities.  That is, it depends on the population of launch vehicles you are sampling from.  [...]  It's not a very satisfying answer, because we don't really know much about the prior probabilities.  So we have to assume something.  But we should be aware that our calculations are based on these (very uncertain) prior probabilities and be appropriately cautious in our use of the probability figures we get from them.
The '14' is pretty much a made-up number.  I'm sure they have some formula, and when you plug 14 into that formula it gives an acceptable reliability.  But the model itself has lots of assumptions, variables, and parameters, all of which are basically guesses.

As an example where 14 consecutive successes proves little, suppose you have a really cheap launcher that fails 50% of the time.   You keep launching it until you get 14 consecutive successes.  Is it then a reliable launcher?

Even worse, if you are trying for accuracy, is that the probability of failure is not stationary.  (That's statistician-speak for the probability changing over the course of the measurement.)  This can go both directions.  Normally an initial design becomes more reliable as bugs and near-misses are fixed.  But the other way can happen too, as Proton and Soyuz show, where an initially reliable vehicle becomes worse.

Plus there are lots of priors that are hard to capture in a model.   Have the folks who designed it done a similar design before?  Are the technologies used old and well understood, or relatively new?  What's the organizational culture for reporting suspicious events up the management chain?

So I'm sure they have some model, which everyone knows is just guesswork.  But the model says '14', and folks say "that seems about right".  So they require 14 straight successes.
Quote
this is something that nearly everyone who isn't a professional statistician gets wrong.  Probabilities are often non-intuitive.  That's why people hire professional statisticians.
A professional statistician won't be much help here.  The problem is the assumptions behind the model (the priors) and not the mathematics of probability.

Offline ChrisWilson68

  • Senior Member
  • *****
  • Posts: 3881
  • Sunnyvale, CA
  • Liked: 2602
  • Likes Given: 3373
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #24 on: 11/15/2018 07:42 pm »
This is all about comparing the odds that you've gotten this result because it's a reliable vehicle versus that you've gotten this result by chance even though the vehicle is not reliable.

Which is why the confidence is 50%. There is still an equal chance that the actual reliability is less than 95%, including due to weird population distributions.

Incorrect.  See my previous posts where I worked out concrete examples where this is not true.

I am not a reliability engineer, but I do know from working with reliability engineers that this is how they calculate confidence levels for reliability in populations with unknown distributions.

You're misunderstanding what they're doing.  They're not calculating it this way because these calculations give numbers that are true no matter what the prior distributions are.  They're calculating it this way because, if they want to get numbers, they need to assume *something* about the priors.  These calculations build in a particular set of assumptions about the priors.

That's the whole danger I was trying to point out: people see formulas like this and they misunderstand them.  A lot of people think what you are claiming here, that these formulas give results that are true no matter what the priors are.  But that's not true.  The formulas assume something about the priors.  They have to.  They assume something about the priors that is a useful heuristic.

There's nothing wrong with doing a calculation like that as long as you understand what it actually means and don't get overconfident about the numbers it gives.
« Last Edit: 11/15/2018 07:43 pm by ChrisWilson68 »

Offline ChrisWilson68

  • Senior Member
  • *****
  • Posts: 3881
  • Sunnyvale, CA
  • Liked: 2602
  • Likes Given: 3373
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #25 on: 11/15/2018 07:50 pm »
Quote
this is something that nearly everyone who isn't a professional statistician gets wrong.  Probabilities are often non-intuitive.  That's why people hire professional statisticians.
A professional statistician won't be much help here.  The problem is the assumptions behind the model (the priors) and not the mathematics of probability.

Thanks for the very good post.  You brought up a lot of other issues with reliability predictions that are also valid issues beyond the one narrow point I was making.

I agree that professional statisticians won't have the domain expertise to give the correct assumptions to use in the models.  But they can help people understand where there are assumptions behind the model and what the model really captures.  You seem to have a good handle on it, but many people, even some smart people who are knowledgeable in their fields do not understand all the subtleties of statistical models.

Offline envy887

  • Senior Member
  • *****
  • Posts: 5600
  • Liked: 3410
  • Likes Given: 1660
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #26 on: 11/16/2018 03:00 am »
This is all about comparing the odds that you've gotten this result because it's a reliable vehicle versus that you've gotten this result by chance even though the vehicle is not reliable.

Which is why the confidence is 50%. There is still an equal chance that the actual reliability is less than 95%, including due to weird population distributions.

Incorrect.  See my previous posts where I worked out concrete examples where this is not true.

Your examples assume a particular distribution of reliabilities, the distribution you explicitly state. This changes the probability of a given reliability by eliminating many other distributions that are also possible. Since we don't actually know the distribution, we have to also account for the probability of LV reliabilities actually having that particular distribution. In the case of your examples, the likelihood of that distribution is right around 0, based on what we observe from other launch vehicles.

Quote
I am not a reliability engineer, but I do know from working with reliability engineers that this is how they calculate confidence levels for reliability in populations with unknown distributions.

You're misunderstanding what they're doing.

No, I think you're misunderstanding what I wrote. Evidently I wasn't clear enough.

Quote
They're not calculating it this way because these calculations give numbers that are true no matter what the prior distributions are.  They're calculating it this way because, if they want to get numbers, they need to assume *something* about the priors.  These calculations build in a particular set of assumptions about the priors.

I understand that just fine, and never indicated otherwise.

Quote
A lot of people think what you are claiming here, that these formulas give results that are true no matter what the priors are.

Say what? ??? I did not claim that. I do not think that. I'm well aware of the many assumptions that go into reliability calculations and how they are often wildly inaccurate.

Quote
But that's not true.  The formulas assume something about the priors.  They have to.  They assume something about the priors that is a useful heuristic.

There's nothing wrong with doing a calculation like that as long as you understand what it actually means and don't get overconfident about the numbers it gives.

I didn't say that they aren't making assumptions about the prior distributions. I said they don't KNOW the prior distributions.

They assume that all reliabilities are possible and equally likely. This is not exactly true, but it's a decent approximation, and is entirely reasonable based on what we see with existing vehicles. It's far more reasonable than the rest of the assumptions going into the formula, and particularly compared to the one about launches being independent events.
« Last Edit: 11/16/2018 01:28 pm by envy887 »

Tags: