Author Topic: Weighing the available Augustine options with factors 1-10  (Read 5734 times)

Offline simon-th

  • Full Member
  • ****
  • Posts: 952
  • Liked: 0
  • Likes Given: 0
I created a spreadsheet to weigh the options + added 4 important political considerations that I don't think have been adequately included in the Augustine assessment matrix.

My personal weighing of the options resulted in Flexible Path with EELV or directly SDLV respectively the Moon option with STS to 2015 and direcly SDLV being in the lead without much difference between those options.

I would be interested in how people on here would themselves weigh the different assessment criteria and what they get out then.

Offline mmeijeri

  • Senior Member
  • *****
  • Posts: 7772
  • Martijn Meijering
  • NL
  • Liked: 397
  • Likes Given: 822
It may be interesting to note that Augustine himself was strongly opposed to this sort of exercise.
Pro-tip: you don't have to be a jerk if someone doesn't agree with your theories

Offline Halidon

  • Full Member
  • ****
  • Posts: 848
  • whereabouts unknown
  • Liked: 180
  • Likes Given: 533
I think tossing scoring charts back and forth is a waste unless reasons for the  scores are included. Otherwise we're just arguing over nonsense numbers. Additionally, putting in a weighted category called "Additional Factors" without describing them is pretty much an open invitation to skew results.

It may be interesting to note that Augustine himself was strongly opposed to this sort of exercise.
Very good point.

Offline sbt

  • Veteran
  • Full Member
  • ***
  • Posts: 328
  • Liked: 1
  • Likes Given: 0
I think tossing scoring charts back and forth is a waste unless reasons for the  scores are included. Otherwise we're just arguing over nonsense numbers. Additionally, putting in a weighted category called "Additional Factors" without describing them is pretty much an open invitation to skew results.

Plus there are more rigourous and logical ways to use judgement to rank options. I've never used them but, as an Operational Researcher, I am aware of their existance and their widespread use in 'Soft OR'.

IIRC An interesting property of some of the methods is that they show inconsistant (and therefore potentially dishonest) thinking by the Subject Matter Experts providing the judgement and provide measures of the level of agreement between SME's.

Recently the methods have been extended to 'Fuzzy' measures, allowing SME's to declare 'the factor is somewhere between 1 and 3 and definately not larger than 4' rather than 'the factor is 2'.

Rick
I am not interested in your political point scoring, Ad Hominem attacks, personal obsessions and vendettas. - No matter how cute and clever you may think your comments are.

Offline Proponent

  • Senior Member
  • *****
  • Posts: 7277
  • Liked: 2782
  • Likes Given: 1462
[T]here are more rigourous and logical ways to use judgement to rank options. I've never used them but, as an Operational Researcher, I am aware of their existance and their widespread use in 'Soft OR'.

Could you perhaps outline these techniques or provide some references?

Offline madscientist197

  • Regular
  • Full Member
  • ****
  • Posts: 1014
  • Liked: 6
  • Likes Given: 0
There are lots of different approaches you could use to do this, but one simple way is to agree on a scoring function and then calculate the expected score (statistical expectation) based on the choice of prior distributions for the different factors (essentially averaging the scores over all possible values of the uncertain variables assuming certain known statistical distributions). For example, if you want to assume that each score component has a Gaussian distribution, then you could rank them by the expected score for each option with larger variances for factors that are less certain etc. It all reduces to fairly simple statistics in the end and you can put confidence intervals on the rankings -- it gets much more interesting when you start ranking by the score at the 65th percentile (optimistic) or 35th percentile (pessimistic) etc.
« Last Edit: 10/12/2009 08:33 am by madscientist197 »
John

Offline PilatusB4

  • Member
  • Posts: 4
  • Liked: 0
  • Likes Given: 0
Could you perhaps outline these techniques or provide some references?

The "QFD" technique is tricky to learn but is really good at choosing among alternatives like this.  http://en.wikipedia.org/wiki/Quality_function_deployment

Offline mmeijeri

  • Senior Member
  • *****
  • Posts: 7772
  • Martijn Meijering
  • NL
  • Liked: 397
  • Likes Given: 822
The "QFD" technique is tricky to learn but is really good at choosing among alternatives like this.  http://en.wikipedia.org/wiki/Quality_function_deployment

Some interesting techniques to help you reach a reasonable consensus when estimates are subjective:

Delphi method
Wideband Delphi
Planning poker
Pro-tip: you don't have to be a jerk if someone doesn't agree with your theories

Offline sbt

  • Veteran
  • Full Member
  • ***
  • Posts: 328
  • Liked: 1
  • Likes Given: 0
[T]here are more rigourous and logical ways to use judgement to rank options. I've never used them but, as an Operational Researcher, I am aware of their existance and their widespread use in 'Soft OR'.

Could you perhaps outline these techniques or provide some references?
Sorry – I'm in the middle of a move of office and other issues so I didn't check back.

I was specifically thinking of Saaty's Analytic Hierarchy Process (AHP) which relies on paired rankings on options and matrix algebra to produce numerical, rather than purely qualitative, answers.

http://en.wikipedia.org/wiki/Analytic_Hierarchy_Process

Analytical Network Analysis generalises things in that the various decision factors are not assumed to be independent.

Whilst Saaty sells software to implement AHP/ANP the method has been published by Saaty and expanded upon by other researchers.

AHP is one example of a set of numerical approaches in an area called Multi-Criteria Decision Analysis However Delphi and other structured decision making systems are also options.

http://en.wikipedia.org/wiki/Multi-Criteria_Decision_Analysis

I'm very much the neophyte in this area – but we have experts in the area within our organisation I would turn to if the problem I was working on seemed to merit the use of this type of approach.

Rick
« Last Edit: 10/13/2009 08:31 pm by sbt »
I am not interested in your political point scoring, Ad Hominem attacks, personal obsessions and vendettas. - No matter how cute and clever you may think your comments are.

Offline kkattula

  • Member
  • Senior Member
  • *****
  • Posts: 3008
  • Melbourne, Australia
  • Liked: 656
  • Likes Given: 116
I expect the people making the decision will apply binary weights to each rating:

Politically acceptable:     1
Politically unacceptable:  0

Offline madscientist197

  • Regular
  • Full Member
  • ****
  • Posts: 1014
  • Liked: 6
  • Likes Given: 0
Re: Weighing the available Augustine options with factors 1-10
« Reply #10 on: 10/14/2009 12:50 am »
Some interesting techniques to help you reach a reasonable consensus when estimates are subjective:
I guess that's the big problem isn't it. If you can actually agree on an absolute scoring system, then the decision is already made for you -- but in reality, everyone disagrees on the scoring.

As to whether people necessarily disagree with the scoring until after they have seen the results is another matter altogether...
« Last Edit: 10/14/2009 12:52 am by madscientist197 »
John

Offline simon-th

  • Full Member
  • ****
  • Posts: 952
  • Liked: 0
  • Likes Given: 0
Re: Weighing the available Augustine options with factors 1-10
« Reply #11 on: 10/14/2009 10:16 am »
I expect the people making the decision will apply binary weights to each rating:

Politically acceptable:     1
Politically unacceptable:  0

I think that is probably closest to the scoring technique that will actually be used.

Offline khallow

  • Extreme Veteran
  • Full Member
  • ****
  • Posts: 1954
  • Liked: 8
  • Likes Given: 4
Re: Weighing the available Augustine options with factors 1-10
« Reply #12 on: 10/14/2009 04:23 pm »
I expect the people making the decision will apply binary weights to each rating:

Politically acceptable:     1
Politically unacceptable:  0

It's not that clear cut. NASA has a lot of power in deciding what is politically acceptable or not, merely by determining whether it can be done. "Sure Senator, we could try that, but it's not feasible so we'll probably be unable to spend all that money even though we're required to. But I'm sure we can find a more feasible way to fund your political contributors." NASA also comes up with the plans in the first place. It's straining credulity to believe that the only politically acceptable action in 2005 was to sponsor the Ares I/V rockets.

Karl Hallowell

Offline mmeijeri

  • Senior Member
  • *****
  • Posts: 7772
  • Martijn Meijering
  • NL
  • Liked: 397
  • Likes Given: 822
Re: Weighing the available Augustine options with factors 1-10
« Reply #13 on: 10/14/2009 04:30 pm »
It's straining credulity to believe that the only politically acceptable action in 2005 was to sponsor the Ares I/V rockets.

Agreed, in the sense that there would have been other solutions that could have satisfied various key politicians. But coming up with such a plan and convincing key players could have taken a lot of time, energy and political capital. Maybe space wasn't important enough for the government of the day to spend a lot of time, energy and political capital on. In that sense Ares I/V may really have been the only politically acceptable plan.
Pro-tip: you don't have to be a jerk if someone doesn't agree with your theories

Tags:
 

Advertisement NovaTech
Advertisement Northrop Grumman
Advertisement
Advertisement Margaritaville Beach Resort South Padre Island
Advertisement Brady Kenniston
Advertisement NextSpaceflight
Advertisement Nathan Barker Photography
1