A Different Take On Reliability

[john smith 19]

Hey everybody, haven't had time to work on this much lately but I'm anticipating some free time in a week or two. I'll be updating all the numbers with the launches we've seen over the last little while, and I'll start reworking my model to include some of these suggestions.

It's clear this has taken some time to prepare such a large piece of work. 

Thank you for providing the basis for a potentially more nuanced discussion on what reliability means and why some vehicles can charge a premium and get it.

[Fequalsma]

Interesting discussion.  Ed, where does the Shuttle fit into your latest analysis?
F=ma

[edkyle99]

Interesting discussion.  Ed, where does the Shuttle fit into your latest analysis?
F=ma

Its rank among retired launch vehicles remains the same, I believe.

===============================================================
                    SPACE LAUNCH REPORT

        RETIRED LAUNCH VEHICLE RELIABILITY STATISTICS
================================================================
           by Ed Kyle      
================================================================
Space launch vehicles retired since ~1980 ranked by their 
predicted orbital success rate*.  Failures include incorrect 
orbits.

            Ranked by Lewis Point Estimate
================================================================ 
                        Lewis 
              Successes Point  AdjWald  Consc. Last    Dates 
Vehicle       /Attempts Est*   95%CI*   Succes Fail    
================================================================
Atlas 2/2AS     63/63   0.98  0.93-1.00  63   None     1991-2004
Tsyklon 2      104/105  0.98  0.94-1.00  92   4/25/73  1967-2006
Soyuz-U        755/776x 0.97  0.96-0.98   1   12/01/16 1973-2017
STS            132/135  0.97  0.93-1.00  22(A)2/1/03   1981-2011
Ariane 4       113/116  0.97  0.92-0.99  74   12/11/94 1988-2003
Delta ELT       89/93   0.95  0.89-0.99   8   05/03/86 1972-1990
Titan 2         17/17#  0.95  0.78-1.00  17   None     1964-2003
Kosmos 3M      423/446  0.95  0.92-0.97  22   11/20/00 1964-2010
Molniya M      277/296  0.93  0.90-0.96   4   6/21/05  1963-2010
Proton-K/DM-2M  40/42   0.93  0.83-1.00   7   11/25/02 1994-2006
Tsyklon 3      114/122  0.93  0.87-0.97   1   12/24/04 1977-2009
Proton-K/DM-2  101/109  0.92  0.86-0.96  15   10/27/99 1982-2012
Scout D-G       33/35   0.92  0.80-0.99  23   12/06/75 1972-1994
Dnepr           21/22   0.92  0.76-1.00  15   7/26/06  1999-2015
Soyuz FG/Fregat 10/10   0.92  0.68-1.00  10   None     2003-2012
H-1              9/9    0.91  0.66-1.00   9   None     1986-1992
Proton-K        26/29+  0.90  0.73-0.97   9   11/29/86 1968-2000
M-3             17/19   0.89  0.67-0.98   0   01/15/95 1974-1995
Zenit 3SL/DMSL  32/36   0.89  0.74-0.96   1   2/1/13   1999-2014
Falcon 9 v1.1   14/15   0.88  0.68-1.00   1   06/28/15 2013-2016
Titan 4B        15/17   0.88  0.64-0.98  12   4/30/99  1997-2005
Atlas E SUS     21/23   0.88  0.72-0.99  21   12/19/81 1980-1995
Ariane 5G(+,S)  22/25   0.88  0.68-0.97  15   7/12/01  1996-2009
Titan 4A        20/22   0.88  0.71-0.99   0   08/12/98 1989-1998
Atlas 3(A/B)     6/6    0.88  0.56-1.00   6   None     2000-2005
Soyuz-U/Ikar     6/6    0.88  0.56-1.00   6   None     1999-1999
Proton-M/DM-2    6/6    0.88  0.56-1.00   6   None     2007-2010
Proton-K/17S40   6/6    0.88  0.56-1.00   6   None     1997-2002
Atlas H SUS      5/5    0.86  0.51-1.00   5   None     1983-1987
Titan 2(Star)    6/7%   0.86  0.47-0.99   6   10/5/93  1964-2003
M-5              6/7    0.86  0.47-0.99   4   2/10/00  1997-2006
START(-1)        6/7    0.86  0.47-0.99   5   3/28/95  1993-2006
Soyuz-U/Fregat   4/4    0.83  0.45-1.00   4   None     2000-2000
Zenit 3SLB/DMSLB 5/6    0.83  0.42-0.99   5(B)4/28/08  2008-2015
Titan 34D       12/15   0.80  0.54-0.94   3   09/02/88 1982-1989
Falcon 9 v1.0    4/5    0.80  0.36-0.98   1   10/8/12  2010-2013
Antares 1xx      4/5    0.80  0.36-0.98   0   10/28/14 2013-2014
Zenit 2(M/SB)   30/38   0.79  0.63-0.89   7   09/09/98 1985-2015
Shitl'           2/2    0.75  0.29-1.00   2   None     1998-2006
Proton-K/Briz-M  3/4    0.75  0.29-0.97   3   07/05/99 1999-2003
Titan 3 Comm.    3/4    0.75  0.29-0.97   2   03/14/90 1990-1992
Atlas G/Centaur 13/18   0.72  0.49-0.88   4   03/25/93 1984-1997
H-2              5/7    0.71  0.35-0.92   0   11/15/99 1994-1999
Falcon 1         2/5    0.44  0.12-0.77   2   08/03/08 2006-2009
KSLV-1 (Angara)  1/3    0.43  0.06-0.80   1   06/10/10 2009-2013
Delta 3          1/3    0.43  0.06-0.80   1   05/05/99 1998-2000
GSLV Mk1         2/6    0.40  0.09-0.70   0   12/25/10 2001-2010
Super Strypi     0/1    0.40  0.00-0.83   0   11/04/15 2015-2015
Volna            0/1    0.40  0.00-0.83   0   06/21/05 2005-2005
Conestoga        0/1    0.40  0.00-0.83   0   10/23/95 1995-1995
VLS-1            0/2    0.33  0.00-0.71   0(E)12/11/99 1997-1999
================================================================

 - Ed Kyle

[john smith 19]

Interesting discussion.  Ed, where does the Shuttle fit into your latest analysis?
F=ma

Its rank among retired launch vehicles remains the same, I believe.

I found this report quite interesting.
I think it's interesting to see what happens when you put "BFR" in place of the vehicle described.
 

[edkyle99]

I found this report quite interesting.
I think it's interesting to see what happens when you put "BFR" in place of the vehicle described.
 

Richard Feynman's Appendix F  [1] shows what the great scientist discovered about launch vehicle reliability.
https://science.ksc.nasa.gov/shuttle/missions/51-l/docs/rogers-commission/Appendix-F.txt

I agree.  All of this talk about flying each BFR or New Glenn stage 100 times or so makes the obvious, and currently difficult - and maybe impossible - to support, assertion that such a rocket would succeed in both launch and landing that many times.

[1]  Which includes the famous ending "For a successful technology, reality must take precedence over
public relations, for nature cannot be fooled."

 - Ed Kyle

[envy887]

I found this report quite interesting.
I think it's interesting to see what happens when you put "BFR" in place of the vehicle described.
 

Richard Feynman's Appendix F  [1] shows what the great scientist discovered about launch vehicle reliability.
https://science.ksc.nasa.gov/shuttle/missions/51-l/docs/rogers-commission/Appendix-F.txt

I agree.  All of this talk about flying each BFR or New Glenn stage 100 times or so makes the obvious, and currently difficult - and maybe impossible - to support, assertion that such a rocket would succeed in both launch and landing that many times.

[1]  Which includes the famous ending "For a successful technology, reality must take precedence over
public relations, for nature cannot be fooled."

 - Ed Kyle

Nobody has ever worn out a launch vehicle, so based on the currently available data it's just as difficult to support the assertion that 100 reflights cannot be done. Every LV failure to date has been an infant mortality issue, not an old age issue. If a booster gets past the first flight unscathed, it might be overwhelmingly likely to complete 100 flights successfully. Or maybe not, we don't know yet.

[edkyle99]

Nobody has ever worn out a launch vehicle, so based on the currently available data it's just as difficult to support the assertion that 100 reflights cannot be done. Every LV failure to date has been an infant mortality issue, not an old age issue. If a booster gets past the first flight unscathed, it might be overwhelmingly likely to complete 100 flights successfully. Or maybe not, we don't know yet.

It seems to me that 100 consecutive successful launches, if not landings, is within the realm of possibility.  It has been done before by expendables, on two occasions,  (1983-86 and 1990-96) when R-7 based launchers recorded 133 consecutive mission successes.  That, of course, assumes that expendable results can be a predictor for reusable success. 

As for consecutive successful landings during orbital launches, the record so far is two with no attempts to even try for three to date.

 - Ed Kyle

[john smith 19]

Nobody has ever worn out a launch vehicle, so based on the currently available data it's just as difficult to support the assertion that 100 reflights cannot be done. Every LV failure to date has been an infant mortality issue, not an old age issue.

That's a belief, not a fact. Without recovery of a full LV to study we believe we understand why those stages failed, but it might be an illusion. Their design could just as easily have failed but the failure "signature" on telemetry was just the same.

If a booster gets past the first flight unscathed, it might be overwhelmingly likely to complete 100 flights successfully. Or maybe not, we don't know yet.

The fact it's taken to SX Block 5 to get to a level of 10 flights suggests the answer is "no." 
I'm curious about the shock loads on the both the structures and the Merlin turbines during the landing. I'm guessing they are quit high. People might think it's like a  carrier landing, but there the engine axis doesn't align to the vehicle descent axis.

[AncientU]

Nobody has ever worn out a launch vehicle, so based on the currently available data it's just as difficult to support the assertion that 100 reflights cannot be done. Every LV failure to date has been an infant mortality issue, not an old age issue. If a booster gets past the first flight unscathed, it might be overwhelmingly likely to complete 100 flights successfully. Or maybe not, we don't know yet.

It seems to me that 100 consecutive successful launches, if not landings, is within the realm of possibility.  It has been done before by expendables, on two occasions,  (1983-86 and 1990-96) when R-7 based launchers recorded 133 consecutive mission successes.  That, of course, assumes that expendable results can be a predictor for reusable success. 

As for consecutive successful landings during orbital launches, the record so far is two with no attempts to even try for three to date.

 - Ed Kyle

What will be the probabilities for SLS Block 1 after one successful flight, when multi-billion dollar Clipper flies?
After two, when crew flies?

[A_M_Swallow]

Nobody has ever worn out a launch vehicle, so based on the currently available data it's just as difficult to support the assertion that 100 reflights cannot be done. Every LV failure to date has been an infant mortality issue, not an old age issue. If a booster gets past the first flight unscathed, it might be overwhelmingly likely to complete 100 flights successfully. Or maybe not, we don't know yet.

There are vertical takeoff and landing (VTOL) rockets that have completed 100 flights. The small 'lunar landers' produced for various prizes. NASA's own Morpheus lander made several trips. Masten Space was trying to get into the Guinness Book of Records for the most landings.

[envy887]

Nobody has ever worn out a launch vehicle, so based on the currently available data it's just as difficult to support the assertion that 100 reflights cannot be done. Every LV failure to date has been an infant mortality issue, not an old age issue. If a booster gets past the first flight unscathed, it might be overwhelmingly likely to complete 100 flights successfully. Or maybe not, we don't know yet.

It seems to me that 100 consecutive successful launches, if not landings, is within the realm of possibility.  It has been done before by expendables, on two occasions,  (1983-86 and 1990-96) when R-7 based launchers recorded 133 consecutive mission successes.  That, of course, assumes that expendable results can be a predictor for reusable success. 

As for consecutive successful landings during orbital launches, the record so far is two with no attempts to even try for three to date.

 - Ed Kyle

Those were 133 consecutive successful first flights, which is even more impressive. Most reliability models suggest that the probability of failure decreases during the first half of the life of a vehicle. The available data supports this: in 143 orbital launches involving the reflight of a reusable vehicle as part of a partially reusable stack, only two mission failures have occurred and both were due to expendable components and assemblies flying the first and only first flight (new ET foam and new SRB assembly).

The record for successful landings after orbital flight is not 2 but 39, by Discovery. Not VTVL, but still highly stressful on the airframe. Perhaps more so than an F9 booster, due to the much higher off-axis loading and much hotter entry on the Orbiter.

Nobody has ever worn out a launch vehicle, so based on the currently available data it's just as difficult to support the assertion that 100 reflights cannot be done. Every LV failure to date has been an infant mortality issue, not an old age issue.

That's a belief, not a fact. Without recovery of a full LV to study we believe we understand why those stages failed, but it might be an illusion. Their design could just as easily have failed but the failure "signature" on telemetry was just the same.

If a booster gets past the first flight unscathed, it might be overwhelmingly likely to complete 100 flights successfully. Or maybe not, we don't know yet.

The fact it's taken to SX Block 5 to get to a level of 10 flights suggests the answer is "no." 
I'm curious about the shock loads on the both the structures and the Merlin turbines during the landing. I'm guessing they are quit high. People might think it's like a  carrier landing, but there the engine axis doesn't align to the vehicle descent axis.

Design failures that cause failure on first use are infant mortality failures, by definition. F9 and New Glenn still have the problem of infant mortality on new components, mainly upper stages. Once full reuse is implemented, that risk can be reduced or eliminated with a shakedown flight of every new vehicle before entering service.

A carrier landing isn't even nominally zero velocity/zero altitude, since a plane can't fly that way. An off-nominal hoverslam might have some high shock loading, but a nominal landing should be relatively soft.

We'll see how Block 5 progresses. Note the the 10 flights is intended to be with no maintenance, and lifespan could be much longer with occasional maintenance and repair, perhaps essentially indefinite. But BFR will take the lessons learned from F9 Block 5 and build them into the "next generation" booster.

[fthomassy]

Design failures that cause failure on first use are infant mortality failures, by definition. F9 and New Glenn still have the problem of infant mortality on new components, mainly upper stages. Once full reuse is implemented, that risk can be reduced or eliminated with a shakedown flight of every new vehicle before entering service.

The definition is better summarized by saying that infant mortality is an issue when first use is less reliable than second use. It is a fallacy to presume that first use failure means there is an early failure (bath tub) reliability issue. The simplest example is uniform failure rates where you are just as likely to fail the first time as the last. Figuring out what the situation is requires lots of expensive data.

Shakedown may or may not be prudent.

Edit: Look at the level of shakedown already in process. Engines tested before assembly, assembly tested before shipment, static test before flight, static test before re-flight. Will a full flight cycle really add value?

[incoming]

IMO, this is a case where a push for simplicity, while understandable, limits the utility of the metric. I think the OP was on the right track in trying to develop a methodology that better accounts for and communicates uncertainty.

Any projection of statistical reliability based on historical failure rates for a system where "n" is very low should be dominated by a very wide uncertainty band. For systems with low numbers of flights, the uncertainty around the reliability is far more important than the mean. Insurance companies would tell you that. Companies that buy launches would tell you that. That's reality.



 

[S.Paulissen]

I know this is serious thread necromancy, but has this type of analysis been done recently. Searches of the forum haven't turned anything up.

[Nomadd]

 I hadn't thought of the R-7 family 133 successful consecutive launch record for a few years till I read through this thread. The F9 is at 129 and should break the record sometime next month. They already have if you count the Heavy boosters.

[Toast]

Sorry, I really should get off my butt and see if I can find where I put these original files at some point. Falcon 9 in particular is insanely out of date already given SpaceX's crazy launch cadence, plus there's lots of new players in the field (like Rocketlab) that I'd love to run an analysis on.

[S.Paulissen]

Sorry, I really should get off my but and see if I can find where I put these original files at some point. Falcon 9 in particular is insanely out of date already given SpaceX's crazy launch cadence, plus there's lots of new players in the field (like Rocketlab) that I'd love to run an analysis on.

Well, consider the fact I was thinking about this work 3 years later. :p

[TrevorMonty]

Most customers are now comfortable with low flightrate boosters. I'm going say upto 5. This level should increase as Starlink pushes few boosters into high teens in 2022. 

Electron is unknown but given F9 success shouldn't have problem selling discounted launches to constellation customers. These customers need discount and can quickly replace any lost satellites from LV failure.

[spacenut]

Yeah, which someone would update those rocket figures.  The chart is now way out of date.  F9 version 1.2 or full thrust has landed over 100 times, several boosters have been reused at least 10 times, and it is only continuing to go up. 

Also, if a F9 booster using kerolox (which can have coking problems) can launch and land 10 or more times, I would think the new metholox rockets should be able to do more than 20-30 launch and landings, being conservative. 

As mentioned, we also need to look at Electron's numbers. 

[spacenut]

Maybe "edkyle99" will chime in with new figures I hope.