A Different Take On Reliability

[Toast]

Many of you may have seen the Launch Vehicle Reliability Stats on Space Launch Report that are generated by forum member Ed Kyle. Lately, my interest is in how to improve the overall quality of the predicted rate of success for rocket launches.

My Methodology and Rationale:

Before beginning into my approach to addressing these issues, I should inform you that I am not a statistician. I enjoy studying statistics as a hobby, but I am not by any stretch of the imagination an expert and I would gladly welcome feedback from those of you who are. That said, we can move on.

I would like to make some key improvements to the previous predicted rates generated on Space Launch Report, as they are calculated using a method that has a few problems, mostly stemming from unjustified or undisclosed assumptions. Specifically:

First: The method uses an uniformative prior, which is inappropriate here. As has been pointed out in discussions elsewhere, this is essentially stating that our prior assumption for an unkown rocket's success rate is 50%. This is absurd, especially for rockets with low flight rates, as the prior overwhelms the observed data. For example, with the (questionable) distinction of the Falcon 9 FT as a separate rocket from previous Falcon 9 iterations, the Full Thrust variant has a projected reliability of 75%. This doesn’t pass the smell test, so to speak: nobody expects that 5 of the next 20 Falcon 9 launches will fail.
 
Second: This projects an inappropriate level of precision, as there is only a single projected rate rather than an interval of likely rates. We are working with a fairly sparse amount of data, and it is very hard to estimate the true reliability of a rocket from a small handful of launches. Only with a few rockets (e.g. Soyuz) do we have a sufficiently large dataset to be relatively confident of the total reliability.

Third: The methodology in distinguishing rockets is sometimes questionable. This certainly applies to the Falcon 9, where it has been broken up in to F9 1.0, F9 1.1 and F9 FT, but also applies to others, including the Soyuz 2 and Ariane 5. This problem is compounded with the use of an uniformative prior, as it is quite pessimistic in the context of rocket launches. It may seem reasonable to distinguish these variants, as it is not unreasonable to expect that (for example) different upper stages in the Ariane 5 variants would not have equal reliability. However, by not grouping similar rockets together we are saying that the prior is more informative that the reliability of related rockets. To use an example, this is stating that for the F9 FT, we expect the reliability will be closer to 50% than to the previous observed reliability for Falcon 9 launches (95%). That’s quite clearly wrong, and we can be fairly sure that while the F9 FT’s reliability may differ from previous iterations, it’s undoubtedly closer to 95% than to 50%.

With that out of the way, here is my methodology in approaching this problem:

I would like to make clear, at each stage, what my assumptions are and why I am making them.
First, I cleaned up and modified Ed Kyle’s dataset, giving me this:



I merged many related platforms because (as discussed above) while the reliability of variants on a rocket may not be equal to each other, they are probably more closely correlated to each other than to the prior. In simple terms, I expect that (for example) the Soyuz 2a’s long-term reliability to be closer to the Soyuz 2b than to the average of all rockets.

That brings us to the next item: the prior that I am using to draw these conclusions is based on the overall success rate of rocket platforms in active use today. Specifically, the prior I am using is a beta distribution with α = 0.95618 and β = 0.04382. Graphically, we are stating that the probability distribution is as follows, where the x-axis is the reliability (where 1 = 100%) and the y-axis is the probability density:



The assumption here is that, in the absence of platform-specific historical data, a good starting point for rocket reliability is the reliability of previous launches. This is obviously better than starting with 50:50 odds of success or failure, but it does have a few weaknesses. Most notably, this does not prevent predictions of 100% maximum reliability. My decision to use this method stems, in part, from the fact that I am hampered by a lack of good pre-existing datasets. Alternate approaches that I considered include:

Use a prior distribution based on cumulative failures per platform over time (the sequence of launches/failures). This would have the advantage of more accurately reflecting the increased risk in new rockets as well as the continuous improvement that takes place as a platform matures. This is a case where available data is the biggest problem: Space Launch Report has by far the best dataset in the genre, but unfortunately does not denote failure modes in a way that is easily computer-readable when looking at historical data for rocket platforms. Specifically, while the datasets for platform launch reliability denote success/failure/partial failure quite clearly, the platform launch history uses footnotes both for failures and for unique or notable launches. This makes it difficult to automate the construction of a dataset that includes the sequence of launches/failures. Eventually, I may revisit this methodology as I think it could represent a significant improvement.

Use a prior distribution for each rocket manufacturer independently. This would have the advantage of not penalizing highly reliable manufacturers by applying a prior based in part on less reliable manufacturer’s success rates (and vice versa). The problem with this approach comes down to the amount of available data. Very few manufacturers have a large enough dataset to allow this methodology, leaving relatively new platforms (such as Antares) with insufficient data to generate a good prior. Further, the benefit of this methodology is questionable as most manufacturers that do have a large number of launches to their name tend to have fairly similar success rates—even the relatively unreliable Proton has a 90% success rate. Combined with the diluted impact of the prior, this shouldn’t significantly penalize successful manufacturers (at the very least, it is a huge improvement from an uninformative prior).


Finally, I took the data on each launch platform and calculated a distribution using binom.bayes in r with the inputs from the prior distribution and the number of successes and failures from each platform. The results are as follows:



That’s a bit hard to parse, though, so let’s go rocket-by-rocket through a few of these.
 
Antares



Reliability: Mean 82.59%, Range 53.83% to ~100%

We’ll start with an instance where the prior overwhelms the data. We’ve got a very wide range this time due to Antares’ single failure, but due to the sparse dataset it’s hard to make any firm projections yet this early in the rocket’s lifespan.

Ariane 5


 
Reliability: Mean 97.62%, Range 94.43% to 99.94%
Moving on, here’s a rocket with a reasonably large launch history, the Ariane 5. Here, we can make predictions with significantly more confidence. The Ariane 5 family has only had two failures in 85 launches, both fairly early in its history.

Atlas 5



Reliability: Mean 98.34%, Range 95.15% to ~100.00%
Another rocket with an excellent and long track record. Atlas 5 has only suffered one complete failure out of 62 launches, so again we can be fairly confident in predicting a high success rate in future launches.

Delta II



Reliability: Mean 98.67%, Range 96.87% to 99.97%
One of the most reliable platforms to date, the Delta II has a good history and a large number of launches. Again, we can strongly predict a high likelihood of success in Delta II’s remaining launches.
 
Delta IV



Reliability: Mean 96.74%, Range 90.53% to ~100.00%
The Delta IV doesn’t have the long track record of its little brother, but still has a strong overall success rate. Due to the smaller amount of data available as well as the single launch failure, the high predicted success rate is accompanied by a relatively long tail.

Falcon 9



Reliability: Mean 95.46%, Range 86.92% to ~100.00%
A rocket that I’m sure many of you are interested in. The Falcon 9 had a strong start, but was hampered by a failure in its 19th launch. We’re still early in the Falcon 9’s career, so we’ve got a fairly large range of probabilities.

Proton M


 
Reliability: Mean 90.64%, Range 85.33% to 95.56%
Proton has a long launch history, but it’s significantly more checkered than the ones we’ve looked at so far. This is the first rocket we’ve looked at where we can confidently say that it will not exceed an upper bound of about 96% reliability in future launches.

Soyuz-2


 
Reliability: Mean 90.83%, Range 83.19% to 97.55%
The newcomer with big shoes to fill, the Soyuz 2 has had a rocky start but still has an overall positive record. Still, without significant improvement it will not reach the reliability of its older sibling, the Soyuz-U.

Soyuz-U


 
Reliability: Mean 97.18%, Range 95.97% to 98.31%
Soyuz-U has the longest track record of any rocket to date, and it has been quite successful. As you can see from the Beta Density on the chart, we can have extreme confidence in this prediction, and this is the shortest interval we’ll see.

Super Strypi


 
Reliability: Mean 47.78%, Range 0.00% to 94.07%
For the sake of showing the extremes of the model, we’ll look at the analysis of the Super Strypi, which failed its first launch attempt a few months back. We’ve got the largest interval in our dataset here, leaning towards the lower end of the scale. Despite our prior informing the model that the average rocket enjoys a 95% success rate, the S. Strypi is off to a rough start, and we can’t be very confident about where it will go from here.


That's about all I've got in me for now, it's taken quite a while to put this together so I'll leave it here for now. If you want any of the probability distribution graphs for a platform I didn't include here, shoot me a message and I'll try to get one to you. If you've got feedback, I'd love to hear it. Moving forward, the next step I'd like to take is to overhaul this methodology to reflect the sequence of success/failure. Rockets tend to fail more frequently early in their history, after which they become more and more reliable through continuous improvement, and these distributions do not take that into account. Beyond that, I'd like to delve a bit more deeply into failure modes. For these intervals, I only used complete failures (e.g. RUD, failure to orbit, etc.). A more accurate picture would also incorporate partial failures, such as missing a target orbit or losing a secondary payload. If there's something else you'd like me to address, let me know!

[Chris Bergin]

I don't have anything useful to add, but only a "like" doesn't do this justice. What a super thread starter. It's not in a viable format for an article, but that has more information than an article. Fine work!

I'm sure some people will have potential refinements, but that's the whole point of a forum, to allow discussion.

[DAZ]

I very much like what you are trying to do here as it is an attempt to go beyond just success and failures.  I have a suggestion for going beyond just the simple.  How about you assign aging to each data point.  Older data points would have less weight than newer data points.  This would take into account how the systems seem to fail earlier in their lives in as the problems are discovered become more reliable with time.  But if they are not becoming more reliable with time this might be a way to highlight that.  For example data points that are 20 years old could age to a value of 0 whereas the most recent launch would have a weight of 100.  You could also use a similar system for waiting partial failures as opposed to total successes or failures.  You would have to be careful on how you waited your data as you could skew the results to the point of leading you to false conclusions.  In the above example of a data point that is 20 years or older having a value of 0 might be an example of very few or none of the people who were working on the rocket at that time are still working on that rocket.  This would mean that the reliability would be based on the inherent robustness of the design and the strength of the systems that were set up to build the rocket.

[Stan-1967]

The addition of the Beta Density axis is very a very helpful metric.   The conversations around here could be much better informed by overlaying of different rockets on the same scaled chart.   This method makes a very informative graphic of the differences that just cant be summarized by a %reliability guestimate.

[deaville]

I'm not qualified to comment when assessing the reliability of launch vehicles. However, I did use a very simple analogy when teaching to show just how much testing was done to achieve the reliability of the Saturn rockets.

If your car starts on first try 9 times out of ten it is 90% reliable. If you have two cars and they both start on first try 9 times out of ten then the combined cars have an 81% reliability [9x9 over 10x10]. With three cars with the same characteristic it is 72% reliability [9x9x9 over 10x10x10].

The Saturn/Apollo complex was made up of several million components and was reckoned to be 99.9% reliable. Now that is mind boggling ..... !

On edit - The vehicle had 5,600,000 parts in 1,500,000 systems and assemblies. If all functioned with a 99.9% reliability factor, there would still be over 5,600 defects or malfunctions on the mission. Some space experts claimed that this was not worth the gamble even though the trip would face fewer unknowns than faced by Columbus.

[savuporo]

Would be awesome to see manned spacecraft side by side here too. Soyuz, Shenzhou, STS, Gemini, Apollo

[deaville]

Would be awesome to see manned spacecraft side by side here too. Soyuz, Shenzhou, STS, Gemini, Apollo

What about Mercury, Vostok and Voskhod? 

[Tev]

This is great addition to Ed's great project! I hope he will see it that way too and figure out how to include it in his Space Launch Report.

[rocx]

I think the merging of launcher types, while giving a better result than keeping them completely separate, is throwing away useful data. Right now there are two levels of reliability: all launchers, and launcher type. I think reliability could be tracked on four levels: all launchers, manufacturer, type and version. And aging of data points should also give an improvement in the results, once the right weighting of parameters is found.

[Proponent]

How Is failure defined?  For instance yes there was one catastrophic failure.  But there was a satellite or two that didn't achieve their orbits.  *cough* interesting *cough*

Ideally, one would keep statistics according to multiple definitions of success, such as "complete success," "primary payload deli0vered as promised" and "payload delivered to trajectory significantly impairing operations."

[Proponent]

I think the merging of launcher types, while giving a better result than keeping them completely separate, is throwing away useful data. Right now there are two levels of reliability: all launchers, and launcher type. I think reliability could be tracked on four levels: all launchers, manufacturer, type and version. And aging of data points should also give an improvement in the results, once the right weighting of parameters is found.

Aging is interesting, and not just for new launch vehicles.  One factor cited by Wayne Eleazor in several launch failures is the the impending phase-out of a vehicle, resulting in a lack of manpower and attention.

[Toast]

Hey everybody, haven't had time to work on this much lately but I'm anticipating some free time in a week or two. I'll be updating all the numbers with the launches we've seen over the last little while, and I'll start reworking my model to include some of these suggestions.

Specifically, I'll try to work in:
   -Consecutive successes/weighting of recent results
   -Breakout for crewed vehicles
   -Breakdown of complete success/partial success/failure
   -Split version of reliability using global/manufacturer/rocket/variant-specific data

I'll also probably just upload this all as a pdf or other downloadable document, that will be easier than separately uploading each graph.

Additionally (with a big maybe) I'll try to start working on some infographics. I love the "Rockets of the World" infographic, but it is perpetually out of date (with launches almost every week, it's hard to keep your tallies updated). I'm trying to work on a few ways of scripting the generation of such a chart or infographic so that I can have it update without any intervention on my part. Don't hold your breath on this part, it's something I'm determined to do but it'll require some time to get set up and running, and I'm still looking for the right tools to automate the job.

[vaporcobra]

I just found this while googling launch vehicle reliability, had no idea it existed! Bravo, Toast

I'd be very curious to see this updated for Soyuz, Proton, and Falcon 9.

[Toast]

Yeah, it's definitely getting a bit dated where the Falcon 9 is concerned, we're at twice the launch history now. Unfortunately I lost the R script that I used, so I can't just plug in some new numbers, and I've been too busy/lazy (and really more the latter) to get around to rebuilding it. I've been wondering about shaking up the methodology too, but I'm just not enough of a statistician to do much to improve it.

But soon^TM I'll get around to it...

[vaporcobra]

Yeah, it's definitely getting a bit dated where the Falcon 9 is concerned, we're at twice the launch history now. Unfortunately I lost the R script that I used, so I can't just plug in some new numbers, and I've been too busy/lazy (and really more the latter) to get around to rebuilding it. I've been wondering about shaking up the methodology too, but I'm just not enough of a statistician to do much to improve it.

But soon^TM I'll get around to it...

Nooooo! Sorry to hear that, I can sympathize with the annoyance of losing work. No worries at all, just wanted to let you know that the interest is there if you can ever spare the time Life comes first!

[S.Paulissen]

How is SpaceX reliability faring these days?

[envy887]

How is SpaceX reliability faring these days?

It's now 54/56=.964 vs the 21/22=.954 listed here (counting AMOS-6 as a failure and not counting the partial failure on CRS-1).

I can't rerun the analysis here, but I's guess the mean hiked up to about .96, and the spread tightened quite a bit. Both failures are now early in the F9's history (man, that sounds really weird to say only 18 months after AMOS), so that would help a lot here.

[rocketguy101]

Wow!  Nice job!  I missed this first time around...I have nothing to add, except I liked this story from “APOLLO The Race to the Moon” by Charles Murray and Catherine Bly Cox...

The joke that made the rounds of NASA was that the Saturn V had a reliability rating of .9999.  In the story, a group from headquarters goes down to Marshall and asks Wernher von Braun how reliable the Saturn is going to be. Von Braun turns to four of his lieutenants and asks, "Is there any reason why it won't work?" to which they answer: "Nein." "Nein." "Nein." "Nein." Von Braun then says to the men from headquarters, "Gentlemen. I have a reliability of four nines."

[edkyle99]

How is SpaceX reliability faring these days?

SpaceX altogether, or just the currently-operated launch vehicle variants? 

Falcon 9 v1.2 has flown 35 times with no launch failures, but this ignores the AMOS 6 failure which involved a fully stacked vehicle with payload during a pre-launch test.  If AMOS 6 is included, Falcon 9 v1.2 sports a 0.85-0.99-ish 95% confidence interval range with a 0.95 point estimate, placing it about 8th or so among the world's major launchers (behind H-2A but ahead of the CZ-4 series, all incrementally).  If AMOS 6 is ignored, the numbers are 0.91-1.0 for the range and 0.97 for the point estimate, slightly behind Atlas 5 and slightly ahead of Delta 4 Medium and Ariane 5 ECA. 

For SpaceX entirely, the company has performed 61 launches with 5 failures, not including AMOS 6.  If AMOS 6 is included, the total is 56 successes in 62 "campaigns", a 0.80-0.96 95% confidence interval range with a 0.89 point estimate (not as good as, say, Rokot, but better than, say, Pegasus, or, more apropos, slightly better than OSC-Orbital-ATK's 68/77 (0.79-0.94, 0.87 point estimate). 

SpaceX reliability has obviously improved from the early days, but we don't really need statistics to tell us that result.

 - Ed Kyle

[EnigmaSCADA]

Would you implement the aging of data based on time, number of launches elapsed, or some percentage of the total number of launches?

To me, if I'm buying launch services, and option A flies once annually and has a near perfect record over the last 15 years vs option B which flies ~30 times a year and has 3 failures that happened 8, 12, and 15 years ago. I'm leaning toward trusting option B with my expensive payload.

The same could be said of choosing a surgeon. Do you want a guy who has performed a procedure twice a year over 20 years or a guy who does it nearly every day for the past 10 years?

So aging by number of launches in the past makes sense to me vs by time in the past. Maybe a way to compound the two?

[kraisee]

Another factor that is often considered by some, is the time between failures, and whether that is increasing, decreasing or remaining stable.

For example, Shuttle first failed on its 25th launch, but it took 87 more flights before it failed a second time.   While those are only two data points, it still suggests that the program's reliability was improving over time.

More data points would help to indicate more useful trends in this regard.

Ross.

[su27k]

I'm just amazed F9 v1.2 already flew 35 times, more flights than Delta 4 Medium, and it only started flying 2.5 years ago.

[Lars-J]

Another factor that is often considered by some, is the time between failures, and whether that is increasing, decreasing or remaining stable.

For example, Shuttle first failed on its 25th launch, but it took 87 more flights before it failed a second time.   While those are only two data points, it still suggests that the program's reliability was improving over time.

More data points would help to indicate more useful trends in this regard.

Ross.

Two data points suggest no such thing. You could just as easily argue that the reliability was about to go over a cliff. Or using the bathtub curve, that Shuttle was nearing the time where failures were about to be more common.

[edkyle99]

Two data points suggest no such thing. You could just as easily argue that the reliability was about to go over a cliff. Or using the bathtub curve, that Shuttle was nearing the time where failures were about to be more common.

I don't believe that the bathtub curve applies to this problem.  The bathtub curve applies to product or part reliability as the product or part ages in use, as I understand things.  Neither of the STS failures were due to part aging.

Historically, most launch vehicles have become more reliable over time, as the bugs are worked out of their designs and processes.  With Shuttle, we aren't just looking at two data points, we are looking at 135 mission "samples", spread over time, that include two outright destructive failures. 

Here's a graph that shows a view of a LaPlace point reliability estimate over the life of the STS program.

 - Ed Kyle

[Lars-J]

Two data points suggest no such thing. You could just as easily argue that the reliability was about to go over a cliff. Or using the bathtub curve, that Shuttle was nearing the time where failures were about to be more common.

I don't believe that the bathtub curve applies to this problem.  The bathtub curve applies to product or part reliability as the product or part ages in use, as I understand things.  Neither of the STS failures were due to part aging.

But future issues, if the program had kept going, would eventually rear their heads. But don't take it too literal. In the case of the NASA and the STS program something similar might be caused by aging ground infrastructure, retiring workforce, lack of proper training of new employees, and so on.

Historically, most launch vehicles have become more reliable over time, as the bugs are worked out of their designs and processes.  With Shuttle, we aren't just looking at two data points, we are looking at 135 mission "samples", spread over time, that include two outright destructive failures.

Here's a graph that shows a view of a LaPlace point reliability estimate over the life of the STS program.

How would such a graph look for Proton?

Anyway, reliability estimates are by their nature not very exact. They can be reasonable predictors of existing known failure points, but mean little if they do take into account the unknown issues that have yet to lead to a failure.

[S.Paulissen]

Two data points suggest no such thing. You could just as easily argue that the reliability was about to go over a cliff. Or using the bathtub curve, that Shuttle was nearing the time where failures were about to be more common.

I don't believe that the bathtub curve applies to this problem.  The bathtub curve applies to product or part reliability as the product or part ages in use, as I understand things.  Neither of the STS failures were due to part aging.

Historically, most launch vehicles have become more reliable over time, as the bugs are worked out of their designs and processes.  With Shuttle, we aren't just looking at two data points, we are looking at 135 mission "samples", spread over time, that include two outright destructive failures. 

Here's a graph that shows a view of a LaPlace point reliability estimate over the life of the STS program.

 - Ed Kyle

Can you point me toward a comprehensible (comprehensive is not necessary) explanation of the LaPlace point statistical analysis?

[Coastal Ron]

I don't believe that the bathtub curve applies to this problem.  The bathtub curve applies to product or part reliability as the product or part ages in use, as I understand things.  Neither of the STS failures were due to part aging.

Historically, most launch vehicles have become more reliable over time, as the bugs are worked out of their designs and processes.  With Shuttle, we aren't just looking at two data points, we are looking at 135 mission "samples", spread over time, that include two outright destructive failures. 

Here's a graph that shows a view of a LaPlace point reliability estimate over the life of the STS program.

Of course you're only plotting complete failures, and not partial failures too. There were many times the Shuttle had ice coming off of the ET that didn't strike the Shuttle, but if it had it could have lead to a complete failure. Same with the SRM's, where they found there was some burn-through at joints.

So I think there should be a graph that takes into account the POTENTIAL for failure, in addition to the ones that lucked out and the ones that didn't. Because a condition that COULD have lead to a loss of vehicle but didn't because of luck could be assumed to be the same as a flight that did fail by the same means.

[Rocket Science]

Absolute confidence in the statistical reliability model is proportional to the amount of crystals you added to the Kool-Aid you are prepared to drink...

[edkyle99]

Can you point me toward a comprehensible (comprehensive is not necessary) explanation of the LaPlace point statistical analysis?

This page includes some concise descriptions of several methods.  The Laplace point estimate represents the peak of the probability distribution within the confidence interval.   The two should be used together, because a "confidence interval ...  is much more informative than a point estimate ..."
https://measuringu.com/wald/

 - Ed Kyle

[Lars-J]

Can you point me toward a comprehensible (comprehensive is not necessary) explanation of the LaPlace point statistical analysis?

This page includes some concise descriptions of several methods.  The Laplace point estimate represents the peak of the probability distribution within the confidence interval.   The two should be used together, because a "confidence interval ...  is much more informative than a point estimate ..."
https://measuringu.com/wald/

 - Ed Kyle

But I don't see both figures in your "launch vehicle reliability stats" on your site. Or am I missing it?

[edkyle99]

But I don't see both figures in your "launch vehicle reliability stats" on your site. Or am I missing it?

I've found the point estimate to be a useful and concise means of ranking launch vehicle reliabilities over time.  The  probability distrubutions would not change that ranking much, if at all.  I'm trying to keep things uncluttered, essentially. 

 - Ed Kyle   

[edkyle99]

Chasing this a bit, I've made three tables comparing several methods to rank reliabilities.
The first table is the Laplace point estimate.  The second table uses the lower bound of the
95% Confidence Interval using the Adjusted Wald method.  The third table uses Wilson's
point estimate, which is the midpoint of the Adjusted Wald 95% Confidence Interval.

The latter two methods put more weight on total number of launches, moving rockets
like Proton M/Briz M up the list versus the Laplace ranking.

Launch Vehicles with 20 or More Orbital Attempts

Ranked by LaPlace point estimate
================================================================ 
Vehicle     Successes/Tries Realzd Pred  Consc. Last     Dates    
                             Rate  Rate* Succes Fail    
================================================================ 
Soyuz-FG          53    53   1.00  .98     53    None     2001-
Delta 2          152   154    .99  .98     99    1/17/97  1989-
Atlas 5           77    78    .99  .98     68    6/15/07  2002-
Falcon 9 v1.2     35    35(D)1.00  .97     35    None     2015-  
Delta 4M(+)       27    27   1.00  .97     27    None     2002-
Ariane 5-ECA      64    66    .97  .96      1    01/25/18 2002-   
CZ-2D             38    39    .97  .95      7    12/28/16 1992-
H-2A              37    38    .97  .95     32    11/29/03 2001-
CZ-4(A/B/C)       54    56    .96  .95      6    08/31/16 1988-
CZ-2(C)(/SD/SM)   47    49    .96  .94     13    08/18/11 1974-
CZ-3B/3C          58    61    .95  .94      6    06/18/17 1996-
PSLV              40    43    .93  .91      2    08/31/17 1993-
CZ-3/3A           35    38    .92  .90     25    8/18/96  1984-
Rokot/Briz/K(M)   27    29    .93  .90     13    02/01/11 1994-
Proton-M/Briz-M   85    94    .90  .90     12    10/21/14 2001-
Soyuz 2-1b/Fregat 27    30    .90  .88      1    11/28/17 2006-
Pegasus (H/XL)    38    43    .88  .87     29    11/4/96  1991-
================================================================ 


Ranked by Adjusted Wald 95% Confidence Interval Lower Limit
================================================================ 
Vehicle     Successes/Tries Realzd Adj Wald 
                                   95%CI 
                                   Lower  Consc. Last     Dates    
                             Rate  Limit* Succes Fail    
================================================================
Delta 2          152   154    .99  .95     99    1/17/97  1989-
Atlas 5           77    78    .99  .92     68    6/15/07  2002- 
Soyuz-FG          53    53   1.00  .92     53    None     2001-
Ariane 5-ECA      64    66    .97  .89      1    01/25/18 2002- 
Falcon 9 v1.2     35    35(D)1.00  .88     35    None     2015-
CZ-4(A/B/C)       54    56    .96  .87      6    08/31/16 1988-
CZ-3B/3C          58    61    .95  .86      6    06/18/17 1996-
CZ-2D             38    39    .97  .86      7    12/28/16 1992-
CZ-2(C)(/SD/SM)   47    49    .96  .86     13    08/18/11 1974-
H-2A              37    38    .97  .85     32    11/29/03 2001-  
Delta 4M(+)       27    27   1.00  .85     27    None     2002-  
Proton-M/Briz-M   85    94    .90  .83     12    10/21/14 2001-
PSLV              40    43    .93  .81      2    08/31/17 1993-
CZ-3/3A           35    38    .92  .78     25    8/18/96  1984-
Rokot/Briz/K(M)   27    29    .93  .77     13    02/01/11 1994- 
Pegasus (H/XL)    38    43    .88  .75     29    11/4/96  1991-
Soyuz 2-1b/Fregat 27    30    .90  .74      1    11/28/17 2006-
================================================================ 


Ranked by Wilson's Point Estimate
================================================================ 
Vehicle     Successes/Tries Realzd Wilsons 
                                   Point  Consc. Last     Dates    
                             Rate  Est*   Succes Fail    
================================================================
Delta 2          152   154    .99  .98     99    1/17/97  1989- 
Soyuz-FG          53    53   1.00  .97     53    None     2001-
Atlas 5           77    78    .99  .96     68    6/15/07  2002- 
Falcon 9 v1.2     35    35(D)1.00  .95     35    None     2015-
Ariane 5-ECA      64    66    .97  .94      1    01/25/18 2002-  
Delta 4M(+)       27    27   1.00  .94     27    None     2002-
CZ-4(A/B/C)       54    56    .96  .93      6    08/31/16 1988-
CZ-2D             38    39    .97  .93      7    12/28/16 1992-
CZ-2(C)(/SD/SM)   47    49    .96  .93     13    08/18/11 1974-
H-2A              37    38    .97  .93     32    11/29/03 2001-
CZ-3B/3C          58    61    .95  .92      6    06/18/17 1996-  
Proton-M/Briz-M   85    94    .90  .89     12    10/21/14 2001-
PSLV              40    43    .93  .89      2    08/31/17 1993-
CZ-3/3A           35    38    .92  .88     25    8/18/96  1984-
Rokot/Briz/K(M)   27    29    .93  .88     13    02/01/11 1994- 
Pegasus (H/XL)    38    43    .88  .85     29    11/4/96  1991-
Soyuz 2-1b/Fregat 27    30    .90  .85      1    11/28/17 2006-
================================================================ 

 - Ed Kyle

[hplan]

Chasing this a bit, I've made three tables comparing several methods to rank reliabilities.
The first table is the Laplace point estimate.  The second table uses the lower bound of the
95% Confidence Interval using the Adjusted Wald method.  The third table uses Wilson's
point estimate, which is the midpoint of the Adjusted Wald 95% Confidence Interval.

The latter two methods put more weight on total number of launches, moving rockets
like Proton M/Briz M up the list versus the Laplace ranking.

Launch Vehicles with 20 or More Orbital Attempts

Ranked by LaPlace point estimate
================================================================ 
Vehicle     Successes/Tries Realzd Pred  Consc. Last     Dates    
                             Rate  Rate* Succes Fail    
================================================================ 
Soyuz-FG          53    53   1.00  .98     53    None     2001-
================================================================ 

I'm not a statistician, but something strikes me as strange about this method of estimating reliability. If all the launches were successful, why would any statistical method ever predict a nonzero failure rate? There must be some kind of assumption built in that rockets sometimes fail, even in the absence of any data to that effect.

Perhaps that is necessary in order to develop a reasonable estimate of likely failure rate, but it is surely dependent on assumptions not resulting from looking at the data for this particular rocket.

Are these assumptions made explicit somewhere, e.g. that in the absence of data 50% reliability is assumed or some such?

Do these models assume that the reliability of a rocket is constant over time?

[Proponent]

I'm not a statistician, but something strikes me as strange about this method of estimating reliability. If all the launches were successful, why would any statistical method ever predict a nonzero failure rate? There must be some kind of assumption built in that rockets sometimes fail, even in the absence of any data to that effect.

Just because a rocket has not failed yet doesn't mean it will never fail in the future, right?  Consider the Shuttle or Falcon 9, for example, which enjoyed multiple successes before their first failures.

Are these assumptions made explicit somewhere, e.g. that in the absence of data 50% reliability is assumed or some such?

The Laplace point estimate makes precisely that assumption.  The estimated success rate after n launches of which s are successful is (s + 1)/(n + 2).

Do these models assume that the reliability of a rocket is constant over time?

Yes, in that only the total numbers of trials and successes, regardless of how long ago the occurred, matter.

[hplan]

I'm not a statistician, but something strikes me as strange about this method of estimating reliability. If all the launches were successful, why would any statistical method ever predict a nonzero failure rate? There must be some kind of assumption built in that rockets sometimes fail, even in the absence of any data to that effect.

Just because a rocket has not failed yet doesn't mean it will never fail in the future, right?  Consider the Shuttle or Falcon 9, for example, which enjoyed multiple successes before their first failures.

Are these assumptions made explicit somewhere, e.g. that in the absence of data 50% reliability is assumed or some such?

The Laplace point estimate makes precisely that assumption.  The estimated success rate after n launches of which s are successful is (s + 1)/(n + 2).

Do these models assume that the reliability of a rocket is constant over time?

Yes, in that only the total numbers of trials and successes, regardless of how long ago the occurred, matter.

And yet from the data, both of these assumptions would appear to be false.

This situation reminds me of the clash of cultures between statisticians and practitioners of machine learning (ML). Statisticians sometimes criticize the ML crowd as "not doing science," because they are trying to get the best possible results for a particular dataset, instead of doing what theoretical statisticians do--proving that a certain method has optimal results when certain assumptions about the distribution of input data are met.

Sadly, real data never meets the statisticians' assumptions.

 

[edkyle99]

Allow me to introduce a proposal for updating my ranking tables.   

First, I propose to replace the Laplace point estimate with what I'm calling a "Lewis Point
Estimate",  using the method presented by James Lewis and Jeff Sauro in 2006.
http://uxpajournal.org/wp-content/uploads/sites/8/pdf/JUS_Lewis_May2006.pdf

Lewis uses the Laplace method if the MLE (Maximum Liklihood Estimate, x/n) >=0.9,
so nothing changes there.  For 0.5<MLE<0.9, the MLE itself is the point estimate. 
For MLE<=0.5, the Wilson method is used.  See Lewis and Sauro for reasoning, and
my footnote below for specifics.

Second, I propose to add an Adjusted-Wald 95% Confidence Interval Range column, to
give a better sense of the certainty of the predictions. 

To keep the table reasonably tidy, I've removed the MLE column, which is easy to figure
regardless.

            Launch Vehicle Reliability
            Ranked by Lewis Point Estimate
================================================================ 
                              Lewis 
                              Point AdjWald Consc. Last    Dates 
Vehicle      Successes/Tries  Est*  95%CI*  Succes Fail    
================================================================
Soyuz-FG          53    53    .98  .92-1.00  53   None     2001-
Delta 2          152   154    .98  .95-1.00  99   01/17/97 1989-
Atlas 5           77    78    .98  .92-1.00  68   06/15/07 2002-
Falcon 9 v1.2     35    35(D) .97  .88-1.00  35   None     2015-
Delta 4M(+)       27    27    .97  .85-1.00  27   None     2002-
Ariane 5-ECA      64    66    .96  .89-1.00   1   01/25/18 2002-
CZ-2D             38    39    .95  .86-1.00   7   12/28/16 1992-
H-2A              37    38    .95  .85-1.00  32   11/29/03 2001-
CZ-4(A/B/C)       54    56    .95  .87-1.00   6   08/31/16 1988-
CZ-2(C)(/SD/SM)   47    49    .94  .86-1.00  13   08/18/11 1974-
CZ-3B/3C          58    61    .94  .86-0.99   6   06/18/17 1996-
CZ-2F(T/Y)        13    13    .93  .73-1.00  13   None     1999-
Minotaur 1        11    11    .92  .70-1.00  11   None     2000-
Vega              10    10xx  .92  .68-1.00  10   None     2012-
PSLV              40    43    .91  .81-0.98   2   08/31/17 1993-
CZ-3/3A           35    38    .90  .78-0.98  25   08/18/96 1984-
Soyuz 2-1a/Fregat 18    19#   .90  .74-1.00  16   05/21/09 2006-
Rokot/Briz/K(M)   27    29    .90  .77-0.99  13   02/01/11 1994-
Soyuz 2-1b         8     8    .90  .63-1.00   8   None     2008-
Proton-M/Briz-M   85    94    .90  .83-0.95  12   10/21/14 2001-
Ariane 5ES         7     7    .89  .60-1.00   7   None     2008-
Delta 4 Heavy      8     9    .89  .54-1.00   8   12/21/04 2004-
H-2B               6     6    .88  .56-1.00   6   None     2009-
Soyuz 2-1b/Fregat 27    30    .88  .74-0.97   1   11/28/17 2006-
Pegasus (H/XL)    38    43    .88  .75-0.95  29   11/04/96 1991-
Soyuz 2-1a        12    13#   .87  .65-1.00   6   04/28/15 2004-
Minotaur 4/5       5     5++  .86  .51-1.00   5   None     2010-
Zenit 3F/FregSB    4     4    .83  .45-1.00   4   None     2011-
CZ-11              4     4    .83  .45-1.00   4   None     2015-
GSLV Mk2           5     6    .83  .42-0.99   5   04/15/10 2001-
Strela             3     3    .80  .38-1.00   3   None     2003-
Kuaizhou 1(A)      3     3    .80  .38-1.00   3   None     2013-
Epsilon            3     3    .80  .38-1.00   3   None     2013-
Antares 2xx        3     3    .80  .38-1.00   3   None     2016-
CZ-6               2     2    .75  .29-1.00   2   None     2015-
Shtil'             2     2    .75  .29-1.00   2   None     1998-
CZ-7               2     2    .75  .29-1.00   2   None     2016-
Shavit(-1,-2)      8    11    .73  .43-0.91   4   9/6/04   1988-
Taurus (XL)        7    10    .70  .39-0.90   1   3/4/11   1994-
Soyuz 2-1v/Volga   2     3    .67  .20-0.94   1   12/05/15 2013-
Falcon Heavy       1     1    .67  .17-1.00   1   None     2018-
Soyuz 2-1a/Volga   1     1    .67  .17-1.00   1   None     2016-
Angara A5          1     1    .67  .17-1.00   1   None     2014-
GSLV Mk3           1     1z   .67  .17-1.00   1   None     2017-
KT-2               1     1    .67  .17-1.00   1   None     2017-
Soyuz 2-1v         1     1    .67  .17-1.00   1   None     2018-
Safir              5     8(C) .63  .30-0.87   1   09/02/12 2008-
Electron           1     2    .50  .09-0.91   1   05/25/17 2017-
SS-520             1     2    .50  .09-0.91   1   01/14/17 2017-
CZ-5               1     2    .50  .09-0.91   0   07/02/17 2016-
Unha (TD-2)        2     5%   .44  .12-0.77   2   04/12/12 2006-
Proton-M/DM-03     1     3    .43  .06-0.80   1   07/02/13 2010-
================================================================
* Lewis Point Estimate Determined as Follows.
  
  Maximum Liklihood Estimate (MLE)= x/n 
      where x=success, n=tries
  If MLE<=0.5, use Wilson Method = (x+2)/(n+4)
  If 0.5<MLE<0.9, use MLE = x/n
  If MLE>=0.9, use Laplace Method = (x+1)/(n+2)
 
  Lewis, J. & Lauro, J., "Improving the Accuracy of Small-Sample 
   Estimates of Completion Rates", Journal of Usability Studies, 
   Issue 3, Vol. 1, May 2006, pp. 136-150.

  Adjusted-Wald 95% Confidence Interval Range Also Provided.

 - Ed Kyle

[Rocket Science]

This is instructional how reliability numbers can be subjective, pre/post Challenger, station and before Columbia... Pick your flavor or Kool-Aid...
https://spectrum.ieee.org/tech-history/heroic-failures/the-space-shuttle-a-case-of-subjective-engineering

“Statistics don’t count for anything,” declared Will Willoughby, the National Aeronautics and Space Administration’s former head of reliability and safety during the Apollo moon landing program. “They have no place in engineering anywhere.” Now director of reliability management and quality assurance for the U.S. Navy, Washington, D.C., he still holds that risk is minimized not by statistical test programs, but by “attention taken in design, where it belongs.” His design-­oriented view prevailed in NASA in the 1970s, when the space shuttle was designed and built by many of the engineers who had worked on the Apollo program.

“The real value of probabilistic risk analysis is in understanding the system and its vulnerabilities,” said Benjamin Buchbinder, manager of NASA’s two-year-old risk management program. He maintains that probabilistic risk analysis can go beyond design-oriented qualitative techniques in looking at the interactions of subsystems, ascertaining the effects of human activity and environmental conditions, and detecting common-cause failures.

NASA started experimenting with this program in response to the Jan. 28, 1986, Challenger accident that killed seven astronauts. The program’s goals are to establish a policy on risk management and to conduct risk assessments independent of normal engineering analyses. But success is slow because of past official policy that favored “engineering judgment” over “probability numbers,” resulting in NASA’s failure to collect the type of statistical test and flight data useful for quantitative risk assessment.

This Catch 22–the agency lacks appropriate statistical data because it did not believe in the technique requiring the data, so it did not gather the relevant data–is one example of how an organization’s underlying culture and explicit policy can affect the overall reliability of the projects it undertakes."

[AncientU]

Allow me to introduce a proposal for updating my ranking tables.   

...
 - Ed Kyle

This is an excellent and balanced treatment of the data!

[Proponent]

And yet from the data, both of these assumptions would appear to be false.

This situation reminds me of the clash of cultures between statisticians and practitioners of machine learning (ML). Statisticians sometimes criticize the ML crowd as "not doing science," because they are trying to get the best possible results for a particular dataset, instead of doing what theoretical statisticians do--proving that a certain method has optimal results when certain assumptions about the distribution of input data are met.

Sadly, real data never meets the statisticians' assumptions.

There is a saying attributed to statistician George Box that "all models are wrong, but some are useful."  In my experience, it's usually best to tackle a problem with the simplest approach that covers the major factors.  In this case, I'd say that a Laplace point estimate fits the bill.  Then elaborate allow for other effects, such as the reliability changing over time.  But there comes a point when additional complexity reduces the power.

[Proponent]

Allow me to introduce a proposal for updating my ranking tables....

Thanks very much for the link to the paper.

I think what you're proposing is a sensible improvement.

Because launch-vehicle reliability can change over time, would it perhaps make sense to also introduce some sort of windowing, so that recent results weigh more heavily than launches from that either occurred a long time ago or, if a vehicle is new, early in the program.  Maybe the way to approach this would be to offer estimates from different subsets of each vehicle's history.

[john smith 19]

Hey everybody, haven't had time to work on this much lately but I'm anticipating some free time in a week or two. I'll be updating all the numbers with the launches we've seen over the last little while, and I'll start reworking my model to include some of these suggestions.

It's clear this has taken some time to prepare such a large piece of work. 

Thank you for providing the basis for a potentially more nuanced discussion on what reliability means and why some vehicles can charge a premium and get it.

[Fequalsma]

Interesting discussion.  Ed, where does the Shuttle fit into your latest analysis?
F=ma

[edkyle99]

Interesting discussion.  Ed, where does the Shuttle fit into your latest analysis?
F=ma

Its rank among retired launch vehicles remains the same, I believe.

===============================================================
                    SPACE LAUNCH REPORT

        RETIRED LAUNCH VEHICLE RELIABILITY STATISTICS
================================================================
           by Ed Kyle      
================================================================
Space launch vehicles retired since ~1980 ranked by their 
predicted orbital success rate*.  Failures include incorrect 
orbits.

            Ranked by Lewis Point Estimate
================================================================ 
                        Lewis 
              Successes Point  AdjWald  Consc. Last    Dates 
Vehicle       /Attempts Est*   95%CI*   Succes Fail    
================================================================
Atlas 2/2AS     63/63   0.98  0.93-1.00  63   None     1991-2004
Tsyklon 2      104/105  0.98  0.94-1.00  92   4/25/73  1967-2006
Soyuz-U        755/776x 0.97  0.96-0.98   1   12/01/16 1973-2017
STS            132/135  0.97  0.93-1.00  22(A)2/1/03   1981-2011
Ariane 4       113/116  0.97  0.92-0.99  74   12/11/94 1988-2003
Delta ELT       89/93   0.95  0.89-0.99   8   05/03/86 1972-1990
Titan 2         17/17#  0.95  0.78-1.00  17   None     1964-2003
Kosmos 3M      423/446  0.95  0.92-0.97  22   11/20/00 1964-2010
Molniya M      277/296  0.93  0.90-0.96   4   6/21/05  1963-2010
Proton-K/DM-2M  40/42   0.93  0.83-1.00   7   11/25/02 1994-2006
Tsyklon 3      114/122  0.93  0.87-0.97   1   12/24/04 1977-2009
Proton-K/DM-2  101/109  0.92  0.86-0.96  15   10/27/99 1982-2012
Scout D-G       33/35   0.92  0.80-0.99  23   12/06/75 1972-1994
Dnepr           21/22   0.92  0.76-1.00  15   7/26/06  1999-2015
Soyuz FG/Fregat 10/10   0.92  0.68-1.00  10   None     2003-2012
H-1              9/9    0.91  0.66-1.00   9   None     1986-1992
Proton-K        26/29+  0.90  0.73-0.97   9   11/29/86 1968-2000
M-3             17/19   0.89  0.67-0.98   0   01/15/95 1974-1995
Zenit 3SL/DMSL  32/36   0.89  0.74-0.96   1   2/1/13   1999-2014
Falcon 9 v1.1   14/15   0.88  0.68-1.00   1   06/28/15 2013-2016
Titan 4B        15/17   0.88  0.64-0.98  12   4/30/99  1997-2005
Atlas E SUS     21/23   0.88  0.72-0.99  21   12/19/81 1980-1995
Ariane 5G(+,S)  22/25   0.88  0.68-0.97  15   7/12/01  1996-2009
Titan 4A        20/22   0.88  0.71-0.99   0   08/12/98 1989-1998
Atlas 3(A/B)     6/6    0.88  0.56-1.00   6   None     2000-2005
Soyuz-U/Ikar     6/6    0.88  0.56-1.00   6   None     1999-1999
Proton-M/DM-2    6/6    0.88  0.56-1.00   6   None     2007-2010
Proton-K/17S40   6/6    0.88  0.56-1.00   6   None     1997-2002
Atlas H SUS      5/5    0.86  0.51-1.00   5   None     1983-1987
Titan 2(Star)    6/7%   0.86  0.47-0.99   6   10/5/93  1964-2003
M-5              6/7    0.86  0.47-0.99   4   2/10/00  1997-2006
START(-1)        6/7    0.86  0.47-0.99   5   3/28/95  1993-2006
Soyuz-U/Fregat   4/4    0.83  0.45-1.00   4   None     2000-2000
Zenit 3SLB/DMSLB 5/6    0.83  0.42-0.99   5(B)4/28/08  2008-2015
Titan 34D       12/15   0.80  0.54-0.94   3   09/02/88 1982-1989
Falcon 9 v1.0    4/5    0.80  0.36-0.98   1   10/8/12  2010-2013
Antares 1xx      4/5    0.80  0.36-0.98   0   10/28/14 2013-2014
Zenit 2(M/SB)   30/38   0.79  0.63-0.89   7   09/09/98 1985-2015
Shitl'           2/2    0.75  0.29-1.00   2   None     1998-2006
Proton-K/Briz-M  3/4    0.75  0.29-0.97   3   07/05/99 1999-2003
Titan 3 Comm.    3/4    0.75  0.29-0.97   2   03/14/90 1990-1992
Atlas G/Centaur 13/18   0.72  0.49-0.88   4   03/25/93 1984-1997
H-2              5/7    0.71  0.35-0.92   0   11/15/99 1994-1999
Falcon 1         2/5    0.44  0.12-0.77   2   08/03/08 2006-2009
KSLV-1 (Angara)  1/3    0.43  0.06-0.80   1   06/10/10 2009-2013
Delta 3          1/3    0.43  0.06-0.80   1   05/05/99 1998-2000
GSLV Mk1         2/6    0.40  0.09-0.70   0   12/25/10 2001-2010
Super Strypi     0/1    0.40  0.00-0.83   0   11/04/15 2015-2015
Volna            0/1    0.40  0.00-0.83   0   06/21/05 2005-2005
Conestoga        0/1    0.40  0.00-0.83   0   10/23/95 1995-1995
VLS-1            0/2    0.33  0.00-0.71   0(E)12/11/99 1997-1999
================================================================

 - Ed Kyle

[john smith 19]

Interesting discussion.  Ed, where does the Shuttle fit into your latest analysis?
F=ma

Its rank among retired launch vehicles remains the same, I believe.

I found this report quite interesting.
I think it's interesting to see what happens when you put "BFR" in place of the vehicle described.
 

[edkyle99]

I found this report quite interesting.
I think it's interesting to see what happens when you put "BFR" in place of the vehicle described.
 

Richard Feynman's Appendix F  [1] shows what the great scientist discovered about launch vehicle reliability.
https://science.ksc.nasa.gov/shuttle/missions/51-l/docs/rogers-commission/Appendix-F.txt

I agree.  All of this talk about flying each BFR or New Glenn stage 100 times or so makes the obvious, and currently difficult - and maybe impossible - to support, assertion that such a rocket would succeed in both launch and landing that many times.

[1]  Which includes the famous ending "For a successful technology, reality must take precedence over
public relations, for nature cannot be fooled."

 - Ed Kyle

[envy887]

I found this report quite interesting.
I think it's interesting to see what happens when you put "BFR" in place of the vehicle described.
 

Richard Feynman's Appendix F  [1] shows what the great scientist discovered about launch vehicle reliability.
https://science.ksc.nasa.gov/shuttle/missions/51-l/docs/rogers-commission/Appendix-F.txt

I agree.  All of this talk about flying each BFR or New Glenn stage 100 times or so makes the obvious, and currently difficult - and maybe impossible - to support, assertion that such a rocket would succeed in both launch and landing that many times.

[1]  Which includes the famous ending "For a successful technology, reality must take precedence over
public relations, for nature cannot be fooled."

 - Ed Kyle

Nobody has ever worn out a launch vehicle, so based on the currently available data it's just as difficult to support the assertion that 100 reflights cannot be done. Every LV failure to date has been an infant mortality issue, not an old age issue. If a booster gets past the first flight unscathed, it might be overwhelmingly likely to complete 100 flights successfully. Or maybe not, we don't know yet.

[edkyle99]

Nobody has ever worn out a launch vehicle, so based on the currently available data it's just as difficult to support the assertion that 100 reflights cannot be done. Every LV failure to date has been an infant mortality issue, not an old age issue. If a booster gets past the first flight unscathed, it might be overwhelmingly likely to complete 100 flights successfully. Or maybe not, we don't know yet.

It seems to me that 100 consecutive successful launches, if not landings, is within the realm of possibility.  It has been done before by expendables, on two occasions,  (1983-86 and 1990-96) when R-7 based launchers recorded 133 consecutive mission successes.  That, of course, assumes that expendable results can be a predictor for reusable success. 

As for consecutive successful landings during orbital launches, the record so far is two with no attempts to even try for three to date.

 - Ed Kyle

[john smith 19]

Nobody has ever worn out a launch vehicle, so based on the currently available data it's just as difficult to support the assertion that 100 reflights cannot be done. Every LV failure to date has been an infant mortality issue, not an old age issue.

That's a belief, not a fact. Without recovery of a full LV to study we believe we understand why those stages failed, but it might be an illusion. Their design could just as easily have failed but the failure "signature" on telemetry was just the same.

If a booster gets past the first flight unscathed, it might be overwhelmingly likely to complete 100 flights successfully. Or maybe not, we don't know yet.

The fact it's taken to SX Block 5 to get to a level of 10 flights suggests the answer is "no." 
I'm curious about the shock loads on the both the structures and the Merlin turbines during the landing. I'm guessing they are quit high. People might think it's like a  carrier landing, but there the engine axis doesn't align to the vehicle descent axis.

[AncientU]

Nobody has ever worn out a launch vehicle, so based on the currently available data it's just as difficult to support the assertion that 100 reflights cannot be done. Every LV failure to date has been an infant mortality issue, not an old age issue. If a booster gets past the first flight unscathed, it might be overwhelmingly likely to complete 100 flights successfully. Or maybe not, we don't know yet.

It seems to me that 100 consecutive successful launches, if not landings, is within the realm of possibility.  It has been done before by expendables, on two occasions,  (1983-86 and 1990-96) when R-7 based launchers recorded 133 consecutive mission successes.  That, of course, assumes that expendable results can be a predictor for reusable success. 

As for consecutive successful landings during orbital launches, the record so far is two with no attempts to even try for three to date.

 - Ed Kyle

What will be the probabilities for SLS Block 1 after one successful flight, when multi-billion dollar Clipper flies?
After two, when crew flies?

[A_M_Swallow]

Nobody has ever worn out a launch vehicle, so based on the currently available data it's just as difficult to support the assertion that 100 reflights cannot be done. Every LV failure to date has been an infant mortality issue, not an old age issue. If a booster gets past the first flight unscathed, it might be overwhelmingly likely to complete 100 flights successfully. Or maybe not, we don't know yet.

There are vertical takeoff and landing (VTOL) rockets that have completed 100 flights. The small 'lunar landers' produced for various prizes. NASA's own Morpheus lander made several trips. Masten Space was trying to get into the Guinness Book of Records for the most landings.

[envy887]

Nobody has ever worn out a launch vehicle, so based on the currently available data it's just as difficult to support the assertion that 100 reflights cannot be done. Every LV failure to date has been an infant mortality issue, not an old age issue. If a booster gets past the first flight unscathed, it might be overwhelmingly likely to complete 100 flights successfully. Or maybe not, we don't know yet.

It seems to me that 100 consecutive successful launches, if not landings, is within the realm of possibility.  It has been done before by expendables, on two occasions,  (1983-86 and 1990-96) when R-7 based launchers recorded 133 consecutive mission successes.  That, of course, assumes that expendable results can be a predictor for reusable success. 

As for consecutive successful landings during orbital launches, the record so far is two with no attempts to even try for three to date.

 - Ed Kyle

Those were 133 consecutive successful first flights, which is even more impressive. Most reliability models suggest that the probability of failure decreases during the first half of the life of a vehicle. The available data supports this: in 143 orbital launches involving the reflight of a reusable vehicle as part of a partially reusable stack, only two mission failures have occurred and both were due to expendable components and assemblies flying the first and only first flight (new ET foam and new SRB assembly).

The record for successful landings after orbital flight is not 2 but 39, by Discovery. Not VTVL, but still highly stressful on the airframe. Perhaps more so than an F9 booster, due to the much higher off-axis loading and much hotter entry on the Orbiter.

Nobody has ever worn out a launch vehicle, so based on the currently available data it's just as difficult to support the assertion that 100 reflights cannot be done. Every LV failure to date has been an infant mortality issue, not an old age issue.

That's a belief, not a fact. Without recovery of a full LV to study we believe we understand why those stages failed, but it might be an illusion. Their design could just as easily have failed but the failure "signature" on telemetry was just the same.

If a booster gets past the first flight unscathed, it might be overwhelmingly likely to complete 100 flights successfully. Or maybe not, we don't know yet.

The fact it's taken to SX Block 5 to get to a level of 10 flights suggests the answer is "no." 
I'm curious about the shock loads on the both the structures and the Merlin turbines during the landing. I'm guessing they are quit high. People might think it's like a  carrier landing, but there the engine axis doesn't align to the vehicle descent axis.

Design failures that cause failure on first use are infant mortality failures, by definition. F9 and New Glenn still have the problem of infant mortality on new components, mainly upper stages. Once full reuse is implemented, that risk can be reduced or eliminated with a shakedown flight of every new vehicle before entering service.

A carrier landing isn't even nominally zero velocity/zero altitude, since a plane can't fly that way. An off-nominal hoverslam might have some high shock loading, but a nominal landing should be relatively soft.

We'll see how Block 5 progresses. Note the the 10 flights is intended to be with no maintenance, and lifespan could be much longer with occasional maintenance and repair, perhaps essentially indefinite. But BFR will take the lessons learned from F9 Block 5 and build them into the "next generation" booster.

[fthomassy]

Design failures that cause failure on first use are infant mortality failures, by definition. F9 and New Glenn still have the problem of infant mortality on new components, mainly upper stages. Once full reuse is implemented, that risk can be reduced or eliminated with a shakedown flight of every new vehicle before entering service.

The definition is better summarized by saying that infant mortality is an issue when first use is less reliable than second use. It is a fallacy to presume that first use failure means there is an early failure (bath tub) reliability issue. The simplest example is uniform failure rates where you are just as likely to fail the first time as the last. Figuring out what the situation is requires lots of expensive data.

Shakedown may or may not be prudent.

Edit: Look at the level of shakedown already in process. Engines tested before assembly, assembly tested before shipment, static test before flight, static test before re-flight. Will a full flight cycle really add value?

[incoming]

IMO, this is a case where a push for simplicity, while understandable, limits the utility of the metric. I think the OP was on the right track in trying to develop a methodology that better accounts for and communicates uncertainty.

Any projection of statistical reliability based on historical failure rates for a system where "n" is very low should be dominated by a very wide uncertainty band. For systems with low numbers of flights, the uncertainty around the reliability is far more important than the mean. Insurance companies would tell you that. Companies that buy launches would tell you that. That's reality.



 

[S.Paulissen]

I know this is serious thread necromancy, but has this type of analysis been done recently. Searches of the forum haven't turned anything up.

[Nomadd]

 I hadn't thought of the R-7 family 133 successful consecutive launch record for a few years till I read through this thread. The F9 is at 129 and should break the record sometime next month. They already have if you count the Heavy boosters.

[Toast]

Sorry, I really should get off my butt and see if I can find where I put these original files at some point. Falcon 9 in particular is insanely out of date already given SpaceX's crazy launch cadence, plus there's lots of new players in the field (like Rocketlab) that I'd love to run an analysis on.

[S.Paulissen]

Sorry, I really should get off my but and see if I can find where I put these original files at some point. Falcon 9 in particular is insanely out of date already given SpaceX's crazy launch cadence, plus there's lots of new players in the field (like Rocketlab) that I'd love to run an analysis on.

Well, consider the fact I was thinking about this work 3 years later. :p

[TrevorMonty]

Most customers are now comfortable with low flightrate boosters. I'm going say upto 5. This level should increase as Starlink pushes few boosters into high teens in 2022. 

Electron is unknown but given F9 success shouldn't have problem selling discounted launches to constellation customers. These customers need discount and can quickly replace any lost satellites from LV failure.

[spacenut]

Yeah, which someone would update those rocket figures.  The chart is now way out of date.  F9 version 1.2 or full thrust has landed over 100 times, several boosters have been reused at least 10 times, and it is only continuing to go up. 

Also, if a F9 booster using kerolox (which can have coking problems) can launch and land 10 or more times, I would think the new metholox rockets should be able to do more than 20-30 launch and landings, being conservative. 

As mentioned, we also need to look at Electron's numbers. 

[spacenut]

Maybe "edkyle99" will chime in with new figures I hope. 

[edkyle99]

================================================================
                   
        ACTIVE LAUNCH VEHICLE RELIABILITY STATISTICS
================================================================
           by Ed Kyle      as of April 21, 2022     
================================================================
Top active space launch vehicles ranked by Lewis point estimate 
of their orbital success rate*.  Failures include incorrect 
orbits.
================================================================ 
                           Lewis 
                Successes  Point AdjWald   Consc. Last    Dates 
Vehicle         /Attempts  Est*  95%CI*    Succes Fail    
================================================================
Falcon 9 v1.2    128/128(D)0.99  0.96-1.00 128   None     2015-
Atlas 5           91/92    0.98  0.94-1.00  82   06/15/07 2002-
CZ-2D             57/58    0.97  0.90-1.00  26   12/28/16 1992-
Ariane 5-ECA(+)   77/79    0.96  0.91-1.00  14   01/25/18 2002-
H-2A              44/45    0.96  0.87-1.00  39   11/29/03 2001-
CZ-4(A/B/C)       87/90    0.96  0.90-0.99  29   05/22/19 1988-
CZ-2(C)(/SD/SM/YZ)62/64    0.95  0.89-1.00  28   08/18/11 1974-
CZ-3B/3C          98/102   0.95  0.90-0.99  18   04/09/20 1996-
CZ-2F(T/Y/G)      16/16    0.94  0.77-1.00  16   None     1999-
Soyuz 2-1a        29/30#   0.94  0.82-1.00  23   04/28/15 2004-
Soyuz 2-1b/Fregat 57/60    0.94  0.86-0.99  30   11/28/17 2006-
Soyuz 2-1a/Fregat 28/29#   0.94  0.81-1.00  26   05/21/09 2006-
Soyuz 2-1b        13/13    0.93  0.73-1.00  13   None     2008-
Minotaur 1        12/12    0.93  0.72-1.00  12   None     2000-
CZ-11             12/12    0.93  0.72-1.00  12   None     2015-
PSLV              51/54    0.93  0.84-0.99  13   08/31/17 1993-
Antares 2xx       11/11    0.92  0.70-1.00  11   None     2016-
CZ-3/3A           37/40    0.90  0.79-0.98  27   08/18/96 1984-
CZ-6               8/8     0.90  0.63-1.00   8   None     2015-
Vega              17/19xx  0.89  0.67-0.98   3   11/17/20 2012-
Proton-M/Briz-M   90/100   0.89  0.82-0.94   0   12/13/21 2001-
Pegasus (H/XL)    40/45    0.89  0.76-0.96  31   11/04/96 1991-
Electron          22/25    0.88  0.69-0.97   5   05/15/21 2017-
Kuaizhou 1(A)     14/16    0.88  0.63-0.98   0   12/15/21 2013-
Minotaur 4/5       6/6++   0.88  0.56-1.00   6   None     2010-
Delta 4 Heavy     12/13    0.87  0.65-1.00  12   12/21/04 2004-
Epsilon            5/5     0.86  0.51-1.00   5   None     2013-
CZ-7               4/4     0.83  0.45-1.00   4   None     2016-
GSLV Mk3           3/3z    0.80  0.38-1.00   3   None     2017-
Falcon Heavy       3/3     0.80  0.38-1.00   3   None     2018-
Soyuz 2-1v/Volga   4/5     0.80  0.36-0.98   3   12/05/15 2013-
CZ-5               4/5     0.80  0.36-0.98   3   07/02/17 2016-
Shavit(-1,-2)      9/12    0.75  0.46-0.92   5   9/6/04   1988-
GSLV Mk2           6/8     0.75  0.40-0.94   0   08/12/21 2010-
Soyuz 2-1v         2/2     0.75  0.29-1.00   2   None     2018-
Angara A5/Briz-M   2/2     0.75  0.29-1.00   2   None     2014-
CZ-5B              2/2     0.75  0.29-1.00   2   None     2020-
Ceres-1            2/2     0.75  0.29-1.00   2   None     2020-
CZ-8               2/2     0.75  0.29-1.00   2   None     2020-
Qased              2/2     0.75  0.29-1.00   2   None     2020-
LauncherOne        3/4     0.75  0.29-0.97   3   05/25/20 2020-
Taurus (XL)        7/10    0.70  0.39-0.90   1   3/4/11   1994-
CZ-7A              2/3     0.67  0.20-0.94   2   03/16/20 2020-
CZ-6A              1/1     0.67  0.17-1.00   1   None     2022-
Proton M           1/1     0.67  0.17-1.00   1   07/21/21 2021-
Soyuz 2-1a/Volga   1/1     0.67  0.17-1.00   1   None     2016-
KT-2               1/1     0.67  0.17-1.00   1   None     2017-
Jielong-1          1/1     0.67  0.17-1.00   1   None     2019-
Proton-M/DM-03     3/5     0.60  0.23-0.88   3   07/02/13 2010-
Safir              5/9(C)  0.56  0.27-0.81   0   02/05/19 2008-
SS-520             1/2     0.50  0.09-0.91   1   01/14/17 2017-
Unha (TD-2)        2/5%    0.44  0.12-0.77   2   04/12/12 2006-
SQX-1              1/3     0.43  0.06-0.80   0   08/03/21 2019-
Astra Rocket 3     2/6     0.40  0.09-0.70   1   02/10/22 2020-
Angara A5/Persei   0/1     0.40  0.00-0.83   0   12/27/21 2021-
KSLV-2 (Nuri)      0/1     0.40  0.00-0.83   0   10/21/21 2021-
KZ-11              0/1     0.40  0.00-0.83   0   07/10/20 2020-  
ZQ-1               0/1     0.40  0.00-0.83   0   10/27/18 2018-
OS-M               0/1     0.40  0.00-0.83   0   03/27/19 2019-
Alpha              0/1     0.40  0.00-0.83   0   09/03/21 2021-
Simorgh            0/4(E)  0.28  0.00-0.62   0   12/31/21 2017- 
================================================================
* Lewis Point Estimate Determined as Follows.
  
  Maximum Liklihood Estimate (MLE)= x/n 
      where x=success, n=tries
  For MLE<=0.5, use Wilson Method = (x+2)/(n+4)
  For MLE Between 0.5 and 0.9, use MLE = x/n
  For MLE>=0.9, use Laplace Method = (x+1)/(n+2)
 
  Lewis, J. & Lauro, J., "Improving the Accuracy of Small-Sample 
   Estimates of Completion Rates", Journal of Usability Studies, 
   Issue 3, Vol. 1, May 2006, pp. 136-150.

* Adjusted-Wald 95% Confidence Interval Range Determined 
     as Follows.

     Pw = (x+1.9208)/(n+3.8416)

     CI = Pw +/- 1.96*sqrt[(Pw*(1-Pw))/(n+3.8416)]  
          (maximum range 1.00)
    
# Does not include one successful suborbital Soyuz 2-1a test 
   flight performed in 2004.

++Does not include two successful suborbital Minotaur 4 Lite 
   flights in 2010-11.

x Does not include Soyuz-U/Soyuz T-10-1 pre-launch fire that 
   resulted in escape tower firing saving crew, but destroying 
   launch vehicle on 9-26-1983.  Note that 10 additional 
   Soyuz-U launches with Ikar or Fregat upper stages (all 
   successful) are cataloged separately.

xx Does not include successful 2-11-15 suborbital flight with 
    IXV reentry demonstrator.

% Includes 2006 failure thought to be a two-stage suborbital 
   test launch attempt.

z Does not include successful inaugural suborbital test flight 
    on 12-18-14.

(C) Assumes that two unsuccessful, unreported Safir launch 
     attempts occurred during 2012 and one during 2019.
     Does not count 2019 prelaunch propellant loading explosion. 
 
(D) Does not include 09/01/16 explosion that destroyed F9-29 
     and AMOS 6 payload during propellant loading for prelaunch 
     hot fire test at CC 40 two days before planned launch.      
(D) Does not include successful 01/19/20 suborbital IFA Crew 
     Dragon launch by F9-80.  

(E) Assumes 7/27/17 launch was a failed orbital attempt and 
     that 4/19/16 launch was a suborbital test flight.