Author Topic: Paper: NASA's Joint Confidence Level Paradox - A History of Denial  (Read 13761 times)

Offline Robotbeat

  • Senior Member
  • *****
  • Posts: 39271
  • Minnesota
  • Liked: 25241
  • Likes Given: 12115
Here's a paper written for the 2009 Cost Estimating Symposium: http://science.ksc.nasa.gov/shuttle/nexgen/Nexgen_Downloads/Butts_NASA's_Joint_Cost-Schedule_Paradox_-_A_History_of_Denial.pdf

Quote
We introduce here  a landscape changing JCL  equation that mathematically compensates for the optimism bias inherently present in NASA cost estimating activity.  It is called the  Joint Confidence Level - Probabilistic Calculator (JCL-PC) and will correct the overly optimistic cost and schedule estimates that have long plagued the NASA cost estimating community
Chris  Whoever loves correction loves knowledge, but he who hates reproof is stupid.

To the maximum extent practicable, the Federal Government shall plan missions to accommodate the space transportation services capabilities of United States commercial providers. US law http://goo.gl/YZYNt0

Offline Lars_J

  • Senior Member
  • *****
  • Posts: 6160
  • California
  • Liked: 677
  • Likes Given: 195
Thanks for the link, interesting stuff indeed.

The latest SLS estimates (21 years & $38B) do seem so much larger that it might be due to a more honest estimation than previously.

Offline cro-magnon gramps

  • Full Member
  • ****
  • Posts: 1548
  • Very Ancient Martian National
  • Ontario, Canada
  • Liked: 843
  • Likes Given: 10996
Thanks for the link, interesting stuff indeed.

The latest SLS estimates (21 years & $38B) do seem so much larger that it might be due to a more honest estimation than previously.

the problem that I find with this estimation, is that it is predicated on the assumption that a 130 MT vehicle MUST be build before the US can use the Base Rocket, at 70 MT, in any meaningful way, other than Apollo 8 redux, and a flight to a NEO; something that are both pretty useless in terms of anything but technological show pieces;

that is why I prefer making the short term decision to build and UTILIZES a 70 MT LV and build the infrastructure, THEN if a larger LV is truly required expand on the system already in place;

using EELV to prepare the way, and then expand on that with a 70 MT LV in conjunction with EELVs makes more sense than immediately going for the UBER rocket that NASA can't afford; to my way of thinking NASA is heading off on the same path that it has already taken many times before; and there will be cost over runs;

btw, I began reading that pdf Robo, and saw 103 pages and said bugger that, will download and read at leisure  ;D but looks interesting; they certainly put a lot of work into it, and is going to be well worth the read Thx
Gramps "Earthling by Birth, Martian by the grace of The Elon." ~ "Hate, it has caused a lot of problems in the world, but it has not solved one yet." Maya Angelou ~ Tony Benn: "Hope is the fuel of progress and fear is the prison in which you put yourself."

Offline Lars_J

  • Senior Member
  • *****
  • Posts: 6160
  • California
  • Liked: 677
  • Likes Given: 195
Of course there are better ways of doing SLS. (Sticking to 70 mT being the primary one) But that is beside the point in the SLS estimates. NASA is estimating SLS based on what Congress wants.

Offline kraisee

  • Expert
  • Senior Member
  • *****
  • Posts: 10560
  • Liked: 807
  • Likes Given: 40
And they're all going to end up with nothing at all if someone doesn't start using some common sense pretty soon.

The uber-SLS as it looks today, is just plain UNAFFORDABLE on the current budget, let alone a reduced one.

A program that costs too much, at a time the budgets are being reduced.   Hmmm.   What happens next?    ::)

These are supposed to be rocket scientists.   If they're really having this much difficulty figuring this simple K-12 math problem out, perhaps it *IS* time for NASA to get out of the game and make way for the people who are making real progress (as opposed to PowerPoint Progress) in this industry.

Ross.
« Last Edit: 08/11/2011 01:59 am by kraisee »
"The meek shall inherit the Earth -- the rest of us will go to the stars"
-Robert A. Heinlein

Offline grr

  • Full Member
  • *
  • Posts: 162
  • Highlands Ranch, Colorado
  • Liked: 0
  • Likes Given: 0
And they're all going to end up with nothing at all if someone doesn't start using some common sense pretty soon.

The uber-SLS as it looks today, is just plain UNAFFORDABLE on the current budget, let alone a reduced one.

A program that costs too much, at a time the budgets are being reduced.   Hmmm.   What happens next?    ::)

These are supposed to be rocket scientists.   If they're really having this much difficulty figuring this simple K-12 math problem out, perhaps it *IS* time for NASA to get out of the game and make way for the people who are making real progress (as opposed to PowerPoint Progress) in this industry.

Ross.

Personally, I would love to see NASA do a COTS-SHLV for 2 LV. Do up to $10 billion/ <7 years each for development, 150 tonnes to LEO, and no more than $.5B to launch.
keep in mind that this is not NASA's vehicle. This is the senates.
But it makes sense for us to have 2 vehicles of similar size.

Offline Lars_J

  • Senior Member
  • *****
  • Posts: 6160
  • California
  • Liked: 677
  • Likes Given: 195
Personally, I would love to see NASA do a COTS-SHLV for 2 LV. Do up to $10 billion/ <7 years each for development, 150 tonnes to LEO, and no more than $.5B to launch.
keep in mind that this is not NASA's vehicle. This is the senates.
But it makes sense for us to have 2 vehicles of similar size.

But there isn't even a market for *one* 150 mT LV, never mind two.

Personally I think the 50-70 mT would be optimal for the affordability and capability perspectives. The solar system would be open to us even if we never got a bigger launcher.

Offline grr

  • Full Member
  • *
  • Posts: 162
  • Highlands Ranch, Colorado
  • Liked: 0
  • Likes Given: 0
Personally, I would love to see NASA do a COTS-SHLV for 2 LV. Do up to $10 billion/ <7 years each for development, 150 tonnes to LEO, and no more than $.5B to launch.
keep in mind that this is not NASA's vehicle. This is the senates.
But it makes sense for us to have 2 vehicles of similar size.

But there isn't even a market for *one* 150 mT LV, never mind two.

Personally I think the 50-70 mT would be optimal for the affordability and capability perspectives. The solar system would be open to us even if we never got a bigger launcher.

Assuming congress does not get in the way, there will be.
The fact is, that Bigelow wants the moon and Musk wants Mars.

Besides, it seems to me that as I look at Delta, Atlas, and falcon that any future SHLV will be a stack and not a single. Delta paved the way with their 3 core, but falcon has shown that the economics can be awesome. No doubt falcon X/XX would be winners, but I would guess that ULA would come up with a new system to take on SpaceX. And if they can launch a number of 50 tonnes, but have the capability to doing 150 tonnes, then it will be a winner. In fact, I would guess that ULA would try to one-up SpaceX and really make the core be re-usable.

Offline Robotbeat

  • Senior Member
  • *****
  • Posts: 39271
  • Minnesota
  • Liked: 25241
  • Likes Given: 12115
This problem of consistently over-optimistic schedule and budget estimates is not limited to NASA. Boeing has also been guilty of it. And so has SpaceX (and we've developed the "SpaceX multiplier" for estimating schedule, semi-jokingly). It's all over aerospace.
Chris  Whoever loves correction loves knowledge, but he who hates reproof is stupid.

To the maximum extent practicable, the Federal Government shall plan missions to accommodate the space transportation services capabilities of United States commercial providers. US law http://goo.gl/YZYNt0

Offline FinalFrontier

  • Senior Member
  • *****
  • Posts: 4490
  • Space Watcher
  • Liked: 1332
  • Likes Given: 173
This problem of consistently over-optimistic schedule and budget estimates is not limited to NASA. Boeing has also been guilty of it. And so has SpaceX (and we've developed the "SpaceX multiplier" for estimating schedule, semi-jokingly). It's all over aerospace.

Its a problem that the real world is not going to allow for much longer. In case folks in the Aerospace sector have not noticed, the world economy is not only shrinking, it is on the verge of collapse. There is a rather disturbing possibility that the European Union may dissolve entirely, the Euro and the Dollar are reaching their breaking point, and people are still getting laid off and no one is getting hired.


Forget about government programs even the private sector globally is shrinking


Times have changed, private space industry is going to have to get their respective crap in order if they want to survive let along profit.


The writing is on the wall, just as it was in 2007. The question is who decides to read it.
« Last Edit: 08/11/2011 05:36 am by FinalFrontier »
3-30-2017: The start of a great future
"Live Long and Prosper"

Offline Lars_J

  • Senior Member
  • *****
  • Posts: 6160
  • California
  • Liked: 677
  • Likes Given: 195
This problem of consistently over-optimistic schedule and budget estimates is not limited to NASA. Boeing has also been guilty of it. And so has SpaceX (and we've developed the "SpaceX multiplier" for estimating schedule, semi-jokingly). It's all over aerospace.

Its a problem that the real world is not going to allow for much longer. In case folks in the Aerospace sector have not noticed, the world economy is not only shrinking, it is on the verge of collapse. There is a rather disturbing possibility that the European Union may dissolve entirely, the Euro and the Dollar are reaching their breaking point, and people are still getting laid off and no one is getting hired.

... And the sky is falling. No, seriously, that has to be the most ridiculously alarmist post I have read for a long time. The world is not ending. The European Union is not dissolving.

Offline madscientist197

  • Regular
  • Full Member
  • ****
  • Posts: 1014
  • Liked: 6
  • Likes Given: 0
the European Union may dissolve entirely

There is no way that the EU as a whole is going to suddenly disappear. At most the currency union might end.

My guess would be that the majority of members stay in the Euro. Perhaps with some sort of joint fiscal policy/oversight, coupled with the ECB raising their inflation target significantly (at least to three or four percent) and lots more bond-buying/aid in the immediate future.

Ultimately, whether the currency union survives is going to depends an awful lot on German politics and whether they are willing to make the sacrifices necessary (like higher inflation) to allow the weaker members to stay in the union.
« Last Edit: 08/11/2011 08:52 am by madscientist197 »
John

Offline Justin Space

  • Extreme Veteran
  • Full Member
  • ****
  • Posts: 1368
  • England
  • Liked: 91
  • Likes Given: 293
Don't be so sure about that. Thank goodness the UK didn't become part of those United States of Europe.

Where in this presentation does it even mention SLS?

Offline madscientist197

  • Regular
  • Full Member
  • ****
  • Posts: 1014
  • Liked: 6
  • Likes Given: 0
Some interesting commentary:
Quote
It is found with overwhelming statistical significance that cost underestimation and overrun cannot be explained by error and seems to be best explained by strategic misrepresentation, namely lying, with a view to getting projects started.

I don't trust all their figures, though: for example, I'm a bit dubious about their claim that it would have cost over a trillion dollars to have made all 580 planned shuttle flights through 1990. It seems like they've just multiplied the total shuttle costs to that point by the ratio of intended to flown flights; this completely ignores the impact of fixed costs versus marginal costs. Surely a paper on cost-estimation can do better than that? For example, they could at least acknowledge the naivety of the estimate in the paper.

I finally got to the end of the paper and found that the JCL-PC methodology is a rather trivial heuristic, presented in a rather unprofessional way (they give pseudocode for Palisade @Risk, rather than writing it out in proper mathematical form).

They've obvious put a lot of research into it, but overall there seems to be too many oversimplifications to put a lot of trust in the paper.
« Last Edit: 08/11/2011 10:30 am by madscientist197 »
John

Offline john smith 19

  • Senior Member
  • *****
  • Posts: 10351
  • Everyplaceelse
  • Liked: 2431
  • Likes Given: 13606
Here's a paper written for the 2009 Cost Estimating Symposium: http://science.ksc.nasa.gov/shuttle/nexgen/Nexgen_Downloads/Butts_NASA's_Joint_Cost-Schedule_Paradox_-_A_History_of_Denial.pdf


Thank you for hosting this thread.

It's interesting the reactions you can get to a single posting.
MCT ITS BFR SS. The worlds first Methane fueled FFSC engined CFRP SS structure A380 sized aerospaceplane tail sitter capable of Earth & Mars atmospheric flight.First flight to Mars by end of 2022 TBC. T&C apply. Trust nothing. Run your own #s "Extraordinary claims require extraordinary proof" R. Simberg."Competitve" means cheaper ¬cheap SCramjet proposed 1956. First +ve thrust 2004. US R&D spend to date > $10Bn. #deployed designs. Zero.

Online Chris Bergin


Where in this presentation does it even mention SLS?

It doesn't. Which is why it was moved from the SLS thread.
Support NSF via L2 -- Help improve NSF -- Site Rules/Feedback/Updates
**Not a L2 member? Whitelist this forum in your adblocker to support the site and ensure full functionality.**

Offline john smith 19

  • Senior Member
  • *****
  • Posts: 10351
  • Everyplaceelse
  • Liked: 2431
  • Likes Given: 13606
Don't be so sure about that. Thank goodness the UK didn't become part of those United States of Europe.

Where in this presentation does it even mention SLS?
This paper was written in 2009, when CxP was still in progress.

It's about large organisations *consistent* inability to realistically cost and schedule large projects with examples from NASA (like the Shuttle) and Boeing and (possibly) how to compensate for this bias.

NASA is a large organisation.

SLS is a large project.

What are the chances NASA's cost/schedule prediction would be any more accurate this time?
MCT ITS BFR SS. The worlds first Methane fueled FFSC engined CFRP SS structure A380 sized aerospaceplane tail sitter capable of Earth & Mars atmospheric flight.First flight to Mars by end of 2022 TBC. T&C apply. Trust nothing. Run your own #s "Extraordinary claims require extraordinary proof" R. Simberg."Competitve" means cheaper ¬cheap SCramjet proposed 1956. First +ve thrust 2004. US R&D spend to date > $10Bn. #deployed designs. Zero.

Offline JohnFornaro

  • Not an expert
  • Senior Member
  • *****
  • Posts: 10974
  • Delta-t is an important metric.
  • Planet Eaarth
    • Design / Program Associates
  • Liked: 1257
  • Likes Given: 724
Quote from: That there paper that Robo linked
We introduce here  a landscape changing JCL  equation that mathematically compensates for the optimism bias inherently present in NASA cost estimating activity.  It is called the  Joint Confidence Level - Probabilistic Calculator (JCL-PC) and will correct the overly optimistic cost and schedule estimates that have long plagued the NASA cost estimating community.

It will not correct the estimates.

I propose the Subjunctive Grammatical Test: Any speculative proposal by NASA or any other entity, which uses imperative verb forms is already wrong, and should be required to be rewritten using the subjunctive verb form, and should be required to explicityly mention at least three of the underlying assumptions, which if demonstrated to be wrong, would negate the speculative proposal.

Quote
The Probabilistic Calculator will attempt to correct the overly optimistic cost and schedule estimates that have long plagued the NASA cost estimating community.  It will do this by applying a series of weighted factors to the several aspects of the estimate in question.  The proponents of the estimate will have the opportunity to respond to these factors, and to have the JCL-PC run again, in order to partially validate or partially refute the underlying assumptions of the estimate.

Something along those lines.

The very language that is used in these proposals and estimates that I've been reading about is the problem.  These various authors are stating, without cause, justification, or vetting, that the subject of their proposals "will solve" a given problem, or that their estimates "will prove" that their proposal is economically sound.
Sometimes I just flat out don't get it.

Offline KEdward5

  • Full Member
  • ****
  • Posts: 840
  • Dallas, TX
  • Liked: 57
  • Likes Given: 116
Don't be so sure about that. Thank goodness the UK didn't become part of those United States of Europe.

Where in this presentation does it even mention SLS?
This paper was written in 2009, when CxP was still in progress.

It's about large organisations *consistent* inability to realistically cost and schedule large projects with examples from NASA (like the Shuttle) and Boeing and (possibly) how to compensate for this bias.

NASA is a large organisation.

SLS is a large project.

What are the chances NASA's cost/schedule prediction would be any more accurate this time?

Very little, as NASA themselves said it was a worst case scenario. The desperation to associate negativity with SLS is very telling.

Offline Lars_J

  • Senior Member
  • *****
  • Posts: 6160
  • California
  • Liked: 677
  • Likes Given: 195
The desperation to associate negativity with SLS is very telling.

...As is the desperation by some not wanting to see the dire set of circumstances that surround SLS, caused by forces internal & external to NASA.

Those who do not learn from history are doomed to repeat it. A cliche statement at this point, but true. The lessons of past projects must be heeded by the SLS team(s).

Offline jongoff

  • Recovering Rocket Plumber/Space Entrepreneur
  • Senior Member
  • *****
  • Posts: 6807
  • Lafayette/Broomfield, CO
  • Liked: 3987
  • Likes Given: 1684
Personally, I would love to see NASA do a COTS-SHLV for 2 LV. Do up to $10 billion/ <7 years each for development, 150 tonnes to LEO, and no more than $.5B to launch.
keep in mind that this is not NASA's vehicle. This is the senates.
But it makes sense for us to have 2 vehicles of similar size.

But there isn't even a market for *one* 150 mT LV, never mind two.

Personally I think the 50-70 mT would be optimal for the affordability and capability perspectives. The solar system would be open to us even if we never got a bigger launcher.

And vehicles in the 50-70mT range, especially if they're 3-core designs might actually be commercially relevant to non-NASA customers, meaning you really could do a COTS-type program...  Not saying I'm a fan of even this, but it'd be better than Nelson and Shelby's Monster Rocket.  That said, not sure if this is on topic or not.

~Jon

Offline Robotbeat

  • Senior Member
  • *****
  • Posts: 39271
  • Minnesota
  • Liked: 25241
  • Likes Given: 12115
I should point out that between ESAS and the "Preliminary Report," NASA's cost estimation standards have changed in response to criticism from the GAO. This has the effect of making estimates more conservative.

"The GAO pointed out an average cost growth of about 35% and recommended the agency fully embrace and quantify risks. In response to these concerns NASA surveyed the best practices in the industry and determined that all projects should be estimated using probabilistic cost estimating methods and budgets should reflect a 70% probability the project could be completed for that budget request or lower. Initially, the confidence level calculation approaches were schedule insensitive and did not take full advantage of the “portfolio effect”. To account for the previously mentioned issues, NASA created Joint Cost and Schedule Confidence Level (JCL) policy language in NASA Procedure Directive (NPD) 1000.5 in early 2009."
http://www.nasa.gov/offices/pae/organization/CAD-faq.html


The FAQ continues to address anticipated questions about disagreements between lower level engineering teams (etc) and higher level estimations. I have a feeling we are seeing the friction in the discussion of the Preliminary Plan, where many of the existing factions in NASA don't agree with the newer, more conservative, cost estimates. This is inevitable.

It's related to SLS because the Preliminary Report probably represents the largest project yet to use these newer, more conservative cost estimates from the beginning.

And to be honest, anyone who thinks the Preliminary Report represents a LITERAL "worst-case" scenario apparently hasn't read any history. Worst-case is the project is canceled or a major disaster occurs. The Preliminary Report, according to the NASA policy (NASA Procedure Directive (NPD) 1000.5), ought to represent the case which should occur 70% of the time within the available budget.
« Last Edit: 08/11/2011 05:17 pm by Robotbeat »
Chris  Whoever loves correction loves knowledge, but he who hates reproof is stupid.

To the maximum extent practicable, the Federal Government shall plan missions to accommodate the space transportation services capabilities of United States commercial providers. US law http://goo.gl/YZYNt0

Tags:
 

Advertisement NovaTech
Advertisement Northrop Grumman
Advertisement
Advertisement Margaritaville Beach Resort South Padre Island
Advertisement Brady Kenniston
Advertisement NextSpaceflight
Advertisement Nathan Barker Photography
1