Author Topic: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle  (Read 6386 times)

Offline Chris Bergin

SpaceX presser:

__
NASA’s Launch Services Program (LSP) has certified Falcon 9 as a Category 3 launch vehicle. Category 3 launch vehicles are certified to support NASA’s highest cost and most complex scientific missions. The following statement is from SpaceX President and Chief Operating Officer Gwynne Shotwell:

“LSP Category 3 certification is a major achievement for the Falcon 9 team and represents another key milestone in our close partnership with NASA. We are honored to have the opportunity to provide cost-effective and reliable launch services to the country’s most critical scientific payloads."

The process of designating launch vehicles as Category 3 is designed to assure the highest practicable probability of success. Falcon 9 has completed over 60 missions, including the NASA LSP missions Jason-3 and TESS.
__

« Last Edit: 11/08/2018 08:25 PM by Chris Bergin »

Offline gongora

  • Global Moderator
  • Senior Member
  • *****
  • Posts: 4013
  • US
  • Liked: 3448
  • Likes Given: 1986
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #1 on: 11/08/2018 08:28 PM »

Offline TGMetsFan98

Cat 3 is required to fly crew, so F9 was going to need this certification soon. Would like to know if this covers nuclear payloads, though.
"Mankind was born on Earth. It was never meant to die here." -Coop

Offline whitelancer64

Cat 3 is required to fly crew, so F9 was going to need this certification soon. Would like to know if this covers nuclear payloads, though.

It is a prerequisite.

In addition to the Category 3 rating, a NASA nuclear certification requires an additional review of the rocket's self-destruct system, and an analysis of what would happen to the nuclear payload in that event.
"One bit of advice: it is important to view knowledge as sort of a semantic tree -- make sure you understand the fundamental principles, ie the trunk and big branches, before you get into the leaves/details or there is nothing for them to hang on to." - Elon Musk
"There are lies, damned lies, and launch schedules." - Larry J

Offline Chris Bergin

Word is nuclear payloads need separate certification.

Edit: Whitelancer beat me to it.  ;D
« Last Edit: 11/08/2018 09:07 PM by Chris Bergin »

Online DigitalMan

  • Full Member
  • ****
  • Posts: 597
  • Liked: 129
  • Likes Given: 10
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #5 on: 11/08/2018 10:01 PM »
Word is nuclear payloads need separate certification.

Edit: Whitelancer beat me to it.  ;D

If nuclear certification requires more insight into the destruct system, I wonder if the best path for SpaceX is to pursue the removal of the destruct system while still meeting requirements for range safety.  Originally they didn't want to have one, no?

Offline speedevil

  • Senior Member
  • *****
  • Posts: 3189
  • Fife
  • Liked: 1631
  • Likes Given: 1956
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #6 on: 11/08/2018 10:02 PM »
The last line, giving 14 consecutive successes as a route to Cat 3 is an interesting one in the context of possibly highly reusable vehicles.
 
A great milestone for F9, let's hope it only makes another hundred or so flights :)

Offline J-V

  • Member
  • Posts: 99
  • Liked: 25
  • Likes Given: 35
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #7 on: 11/09/2018 06:18 AM »
I wonder how much recovered cores helped this process. Examining the booster post-flight can certainly tell you things telemetry can't, but is NASA willing to use this data is another question.

Do we have any (un)official information on this?

Offline woods170

  • IRAS fan
  • Senior Member
  • *****
  • Posts: 8686
  • IRAS fan
  • The Netherlands
  • Liked: 5458
  • Likes Given: 1788
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #8 on: 11/09/2018 08:45 AM »
Cat 3 is required to fly crew, so F9 was going to need this certification soon. Would like to know if this covers nuclear payloads, though.

Wrong.

Launch Services Program (LSP) is for UNmanned launch services only.

Certifying F9 to carry a crewed vehicle is NOT dependent on, nor tied to, LSP Category 3 certification.

CCP does its own certification of F9, independent from LSP certification efforts. The requirements for both types of certification differ substantially.

The only connection between the two (LSP and CCP) is that both programs share the available information on launch vehicles so that each program gains exceptional insight with regards to those launch vehicles.
« Last Edit: 11/09/2018 08:53 AM by woods170 »


Offline psherriffs

  • Member
  • Posts: 2
  • Cebu, Philippines
  • Liked: 0
  • Likes Given: 0
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #10 on: 11/09/2018 10:05 PM »
Just wondering. Where does the "14 consecutive successful missions" number come from? For 95% reliability wouldn't you need 19? I count 36 consecutive successful missions so far for SX.

Offline envy887

  • Senior Member
  • *****
  • Posts: 4826
  • Liked: 2712
  • Likes Given: 1446
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #11 on: 11/09/2018 10:35 PM »
Just wondering. Where does the "14 consecutive successful missions" number come from? For 95% reliability wouldn't you need 19? I count 36 consecutive successful missions so far for SX.

After 14 consecutive successes you have more than 50% confidence in 95% reliability. This is calculated as 0.95^14 = .487% This 48.7% is the average probability of getting at least one failure if the actual reliability is 95%. The confidence level is 100% less this number, or 51.3%.

19 successful missions does not "prove" 95% reliability, it only establishes a 62.3% (100% - 0.95^19) confidence that the true reliability is at least 95%.

Offline ChrisWilson68

  • Senior Member
  • *****
  • Posts: 3813
  • Sunnyvale, CA
  • Liked: 2460
  • Likes Given: 3192
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #12 on: 11/09/2018 11:32 PM »
Just wondering. Where does the "14 consecutive successful missions" number come from? For 95% reliability wouldn't you need 19? I count 36 consecutive successful missions so far for SX.

After 14 consecutive successes you have more than 50% confidence in 95% reliability. This is calculated as 0.95^14 = .487% This 48.7% is the average probability of getting at least one failure if the actual reliability is 95%. The confidence level is 100% less this number, or 51.3%.

19 successful missions does not "prove" 95% reliability, it only establishes a 62.3% (100% - 0.95^19) confidence that the true reliability is at least 95%.

That's not how these probabilities work.

To illustrate the point, lets take an extreme, unrealistic example.  Lets assume that half of all launch vehicles are 95% reliable and all the other half are 90% reliable.

You don't know if this particular vehicle is 90% reliable or 95% reliable.  So lets look at how likely it is that it's 95% reliable versus 90% reliable.

If it happens that the vehicle is 95% reliable and you launch 14 times, the odds all 14 are successful are .488, as calculated above.  If it happens the vehicle is 90% reliable and you lauch 14 times, the odds all 14 are successful are .229 (.9^14).

Since the 90% vehicles and 95% vehicles are equally represented in the population from which you are sampling, the odds you have a 95% reliable vehicle are .488 / (.488 + .229) = 0.680.  So the odds you have a 95% reliable vehicle are 0.68.

Now, suppose instead that 95%-reliable are one in a billion and that the others are all 90%-reliable.  Now, if you see 14 launches in a row that are successful, the odds you have a 95%-reliable vehicle are .488 / (.488 + (.229 * (10^9 - 1)), which is approximately one in a billion.

I hope this helps illustrate that everything depends on the prior probabilities.  That is, it depends on the population of launch vehicles you are sampling from.  If launch vehicles in general are likely to be reliable, you need far fewer launches to be confident in a new launch vehicle than if launch vehicles in general are unlikely to be reliable.

It's not a very satisfying answer, because we don't really know much about the prior probabilities.  So we have to assume something.  But we should be aware that our calculations are based on these (very uncertain) prior probabilities and be appropriately cautious in our use of the probability figures we get from them.

By the way, this is something that nearly everyone who isn't a professional statistician gets wrong.  Probabilities are often non-intuitive.  That's why people hire professional statisticians.

Offline envy887

  • Senior Member
  • *****
  • Posts: 4826
  • Liked: 2712
  • Likes Given: 1446
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #13 on: 11/10/2018 01:05 AM »
Just wondering. Where does the "14 consecutive successful missions" number come from? For 95% reliability wouldn't you need 19? I count 36 consecutive successful missions so far for SX.

After 14 consecutive successes you have more than 50% confidence in 95% reliability. This is calculated as 0.95^14 = .487% This 48.7% is the average probability of getting at least one failure if the actual reliability is 95%. The confidence level is 100% less this number, or 51.3%.

19 successful missions does not "prove" 95% reliability, it only establishes a 62.3% (100% - 0.95^19) confidence that the true reliability is at least 95%.

That's not how these probabilities work.

To illustrate the point, lets take an extreme, unrealistic example.  Lets assume that half of all launch vehicles are 95% reliable and all the other half are 90% reliable.

You don't know if this particular vehicle is 90% reliable or 95% reliable.  So lets look at how likely it is that it's 95% reliable versus 90% reliable.

If it happens that the vehicle is 95% reliable and you launch 14 times, the odds all 14 are successful are .488, as calculated above.  If it happens the vehicle is 90% reliable and you lauch 14 times, the odds all 14 are successful are .229 (.9^14).

Since the 90% vehicles and 95% vehicles are equally represented in the population from which you are sampling, the odds you have a 95% reliable vehicle are .488 / (.488 + .229) = 0.680.  So the odds you have a 95% reliable vehicle are 0.68.

Now, suppose instead that 95%-reliable are one in a billion and that the others are all 90%-reliable.  Now, if you see 14 launches in a row that are successful, the odds you have a 95%-reliable vehicle are .488 / (.488 + (.229 * (10^9 - 1)), which is approximately one in a billion.

I hope this helps illustrate that everything depends on the prior probabilities.  That is, it depends on the population of launch vehicles you are sampling from.  If launch vehicles in general are likely to be reliable, you need far fewer launches to be confident in a new launch vehicle than if launch vehicles in general are unlikely to be reliable.

It's not a very satisfying answer, because we don't really know much about the prior probabilities.  So we have to assume something.  But we should be aware that our calculations are based on these (very uncertain) prior probabilities and be appropriately cautious in our use of the probability figures we get from them.

By the way, this is something that nearly everyone who isn't a professional statistician gets wrong.  Probabilities are often non-intuitive.  That's why people hire professional statisticians.

Ok, I meant "more than 50% confidence in 95% reliability OR BETTER". It's not a >50% confidence that the reliability is exactly 95%, just a >50% confidence that reliability is higher than 95%. You don't need to know the reliability population distribution of all launch vehicles to establish this after 14 consecutive successes of the same design.


Offline ChrisWilson68

  • Senior Member
  • *****
  • Posts: 3813
  • Sunnyvale, CA
  • Liked: 2460
  • Likes Given: 3192
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #14 on: 11/10/2018 01:24 AM »
Just wondering. Where does the "14 consecutive successful missions" number come from? For 95% reliability wouldn't you need 19? I count 36 consecutive successful missions so far for SX.

After 14 consecutive successes you have more than 50% confidence in 95% reliability. This is calculated as 0.95^14 = .487% This 48.7% is the average probability of getting at least one failure if the actual reliability is 95%. The confidence level is 100% less this number, or 51.3%.

19 successful missions does not "prove" 95% reliability, it only establishes a 62.3% (100% - 0.95^19) confidence that the true reliability is at least 95%.

That's not how these probabilities work.

To illustrate the point, lets take an extreme, unrealistic example.  Lets assume that half of all launch vehicles are 95% reliable and all the other half are 90% reliable.

You don't know if this particular vehicle is 90% reliable or 95% reliable.  So lets look at how likely it is that it's 95% reliable versus 90% reliable.

If it happens that the vehicle is 95% reliable and you launch 14 times, the odds all 14 are successful are .488, as calculated above.  If it happens the vehicle is 90% reliable and you lauch 14 times, the odds all 14 are successful are .229 (.9^14).

Since the 90% vehicles and 95% vehicles are equally represented in the population from which you are sampling, the odds you have a 95% reliable vehicle are .488 / (.488 + .229) = 0.680.  So the odds you have a 95% reliable vehicle are 0.68.

Now, suppose instead that 95%-reliable are one in a billion and that the others are all 90%-reliable.  Now, if you see 14 launches in a row that are successful, the odds you have a 95%-reliable vehicle are .488 / (.488 + (.229 * (10^9 - 1)), which is approximately one in a billion.

I hope this helps illustrate that everything depends on the prior probabilities.  That is, it depends on the population of launch vehicles you are sampling from.  If launch vehicles in general are likely to be reliable, you need far fewer launches to be confident in a new launch vehicle than if launch vehicles in general are unlikely to be reliable.

It's not a very satisfying answer, because we don't really know much about the prior probabilities.  So we have to assume something.  But we should be aware that our calculations are based on these (very uncertain) prior probabilities and be appropriately cautious in our use of the probability figures we get from them.

By the way, this is something that nearly everyone who isn't a professional statistician gets wrong.  Probabilities are often non-intuitive.  That's why people hire professional statisticians.

Ok, I meant "more than 50% confidence in 95% reliability OR BETTER". It's not a >50% confidence that the reliability is exactly 95%, just a >50% confidence that reliability is higher than 95%. You don't need to know the reliability population distribution of all launch vehicles to establish this after 14 consecutive successes of the same design.

But that's still not true.  Take my example where only one launch vehicle design in a billion is 95% reliable and the rest are 90% reliable.  In this case, even if you have 14 consecutive successes with the same design, the odds you've gotten that one 95% reliable design in a billion is only 2 point something in a billion.

This is all about comparing the odds that you've gotten this result because it's a reliable vehicle versus that you've gotten this result by chance even though the vehicle is not reliable.  And that is very sensitive to how likely the vehicle was to be reliable in the first place.

Offline envy887

  • Senior Member
  • *****
  • Posts: 4826
  • Liked: 2712
  • Likes Given: 1446
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #15 on: 11/10/2018 12:57 PM »
This is all about comparing the odds that you've gotten this result because it's a reliable vehicle versus that you've gotten this result by chance even though the vehicle is not reliable.

Which is why the confidence is 50%. There is still an equal chance that the actual reliability is less than 95%, including due to weird population distributions.

I am not a reliability engineer, but I do know from working with reliability engineers that this is how they calculate confidence levels for reliability in populations with unknown distributions.

Offline gongora

  • Global Moderator
  • Senior Member
  • *****
  • Posts: 4013
  • US
  • Liked: 3448
  • Likes Given: 1986
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #16 on: 11/10/2018 01:49 PM »
Did they actually use the 14 flight option, or the 6 flight option?  There have been 6 flights of Block 5, and they have a good amount of insight into that vehicle from the Commercial Crew program.

Offline oldAtlas_Eguy

  • Senior Member
  • *****
  • Posts: 3535
  • Florida
  • Liked: 1943
  • Likes Given: 237
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #17 on: 11/10/2018 02:18 PM »
Does the cat 3 cert include FH? If so it clears they way for FH as a candidate LV for Europa Clippper. Else FH would still need a cat 3 cert.

Offline hopalong

  • Full Member
  • *
  • Posts: 122
  • Milton Keynes
  • Liked: 61
  • Likes Given: 44
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #18 on: 11/10/2018 02:32 PM »
Does the cat 3 cert include FH? If so it clears they way for FH as a candidate LV for Europa Clippper. Else FH would still need a cat 3 cert.

My understanding it does not. But FH could gain Cat 3 late next year (min 3 flights, plus Falcon 9 heritage).

Offline joek

  • Senior Member
  • *****
  • Posts: 2966
  • Liked: 728
  • Likes Given: 375
Re: NASA LSP certify Falcon 9 as a Cat 3 launch vehicle
« Reply #19 on: 11/10/2018 03:31 PM »
Does the cat 3 cert include FH? If so it clears they way for FH as a candidate LV for Europa Clippper. Else FH would still need a cat 3 cert.
My understanding it does not. But FH could gain Cat 3 late next year (min 3 flights, plus Falcon 9 heritage).

No, this definitely does not include FH.  As to leveraging F9 heritage, maybe.  Previously NPD 8610 included (emphasis added):
Quote from: NPD 86.10 (old)
3 consecutive successful flights of a common launch vehicle configuration from an evolved vehicle family developed by an LSC with a previously certified launch vehicle for Risk Category 3, instrumented to provide design verification & flight performance data
Some of that has been removed... 
Quote from: NPD 86.10 (current)
3 (minimum 2 consecutive) successful flights of a common launch vehicle configuration, instrumented to provide design verification & flight performance data

However it also includes the following, which allows for a more tailored approach to certification...
Quote from: NPD 86.10 (current)
A.7 NASA IV&V is a comprehensive assessment of the common launch vehicle analyses conducted by the launch service contractor. It will be based on the complexities and heritage of the vehicle. The LSP will propose the appropriate level of technical assessment (LSP independent modeling, launch service contractor model evaluation, and/or launch service contractor analytical review) at a FPB used to assess the certification strategy for a specified common launch vehicle configuration

Tags: