Commercial Crew - Discussion Thread 3

[Vultur]

Try, "Companies that contract with NASA often lose money in the process." Is that a good thing?

It really falls on NASA to mostly enter into contracts where the contractor has a good chance to make a profit. Jonathan Goff is going to try to make the case that there's a contract like that out there somewhere with a non-Boeing, non-SpaceX commercial crew service provider. I sincerely hope he succeeds!

The alternative is "companies are guaranteed to make money even after 3x repeated failure at a high bid price".

FFP should be a good way to make money, but it can't be guaranteed or else it becomes a freebee money program. There's got to be a line, and Boeing is waayyy past that line.

Yeah. Old space companies designed for cost plus just shouldn't be involved in commercial space. So their losing out/refusing to bid is not necessarily a bad thing.

I know a SpaceX monopoly isn't ideal, but it's not meaningfully worse and possibly better than SpaceX + non performing company eating money. The answer to SpaceX monopoly is to give companies like Rocket Lab or Sierra contracts that will allow them to grow, not to throw good money after bad with Boeing.

(and there's maybe the possibility that with reuse, economies of scale will be powerful enough that launch will essentially become a natural monopoly - so the US government can sustain a second provider like ULA at greater cost & much lower cadence for assured access purposes, but a second SpaceX equivalent/truly commercially competitive provider just isn't plausible. I'm not saying that is the case, but I don't think it's been ruled out either.)

Gaganyaan has non commercial (national pride) justification to exist.

[DanClemmensen]

I know a SpaceX monopoly isn't ideal, but it's not meaningfully worse and possibly better than SpaceX + non performing company eating money. The answer to SpaceX monopoly is to give companies like Rocket Lab or Sierra contracts that will allow them to grow, not to throw good money after bad with Boeing.

I think the problem was that in 2014, Boeing's reputation as the premier Aerospace engineering company had not yet been seriously tarnished. NASA (and probably lots of other folks) believed Boeing's written assertions that were part of the CCP contract, so NASA did not seriously question Boeing's progress reports and did not see that more oversight was needed. Unnecessary oversight costs money, and leads to all sorts of complaints about unneeded government interference. Necessary but unperformed oversight costs even more, but we only see this in retrospect.

The way to avoid throwing good money after bad is to require that a company meet its contractual obligations. FFP is an ideal way to do this, but NASA needs to actually enforce their contracts. In the case of CCP, NASA should have cancelled the Starliner contract for nonperformance no later than about 2022.

There is no reason to penalize "Old Space" and nurture "New Space". Just award  contracts and enforce them.

[markbike528cbx]

......
I think the problem was that in 2014, Boeing's reputation as the premier Aerospace engineering company had not yet been seriously tarnished.

About the same time Ariane's Richard Bowles at the CASBAA Satellite Industry Forum 2013 stated that SpaceX was "selling a dream", while implying that Ariane 6 was a done deal.   Hindsight and internet record can be quite the drag.

I think some of the discussion will be seen as needless hand-wringing in retrospect.
Note that SpaceX BLEW UP a Dragon 2 (C204) in April 2019
The next flight - Inflight abort Test did test the system that caused the explosion.
Bob and Doug were the next flight, just over a year later than the explosion.

IF Starliner comes down correctly, I would think that in a few years, only a few space nerds will remember this episode.

Edit: date

[Vultur]

I know a SpaceX monopoly isn't ideal, but it's not meaningfully worse and possibly better than SpaceX + non performing company eating money. The answer to SpaceX monopoly is to give companies like Rocket Lab or Sierra contracts that will allow them to grow, not to throw good money after bad with Boeing.

I think the problem was that in 2014, Boeing's reputation as the premier Aerospace engineering company had not yet been seriously tarnished. NASA (and probably lots of other folks) believed Boeing's written assertions that were part of the CCP contract, so NASA did not seriously question Boeing's progress reports and did not see that more oversight was needed. Unnecessary oversight costs money, and leads to all sorts of complaints about unneeded government interference. Necessary but unperformed oversight costs even more, but we only see this in retrospect.

The way to avoid throwing good money after bad is to require that a company meet its contractual obligations. FFP is an ideal way to do this, but NASA needs to actually enforce their contracts. In the case of CCP, NASA should have cancelled the Starliner contract for nonperformance no later than about 2022.

There is no reason to penalize "Old Space" and nurture "New Space". Just award  contracts and enforce them.

Oh yeah I am not saying that NASA should intentionally penalize Old Space companies. But if they stop doing things like giving extra money to Boeing beyond the original contract for Starliner, the effect will be that Old Space companies stop trying for fixed price/commercial, and that isn't a bad thing IMO.

[sdsds]

[...]  the Firm Fixed Price (FFP) contract milestones that NASA created were insufficient to find out the true progress of what Boeing was doing [which] is what NASA needs to figure out better.

100% agreement here. What insight must NASA require, and when? And can NASA develop the ability to respond effectively when that insight implies an effort is off-track?

When did (vs. should have) NASA become aware that the Boeing thermal model for the thruster doghouse was inadequate? In a set of perfect milestones, when would Boeing have missed one?

[Thorny]

Note that SpaceX BLEW UP a Dragon 2 (C204) in April 2020.   

2019

[JayWee]

[...]  the Firm Fixed Price (FFP) contract milestones that NASA created were insufficient to find out the true progress of what Boeing was doing [which] is what NASA needs to figure out better.

100% agreement here. What insight must NASA require, and when? And can NASA develop the ability to respond effectively when that insight implies an effort is off-track?

When did (vs. should have) NASA become aware that the Boeing thermal model for the thruster doghouse was inadequate? In a set of perfect milestones, when would Boeing have missed one?

I still don't understand why it wasn't caught during OFT-2.

[AndrewM]

I do not know how much NASA has actually paid Boeing, which is an almost entirely different subject.

If I read this correctly, NASA has paid Boeing ~$2.24B of the up to ~$4.57B awarded.

https://www.usaspending.gov/award/CONT_IDV_NNK14MA75C_8000

[Eric Hedman]

I do not know how much NASA has actually paid Boeing, which is an almost entirely different subject.

If I read this correctly, NASA has paid Boeing ~$2.24B of the up to ~$4.57B awarded.

https://www.usaspending.gov/award/CONT_IDV_NNK14MA75C_8000

I may be wrong, but I read this as they have paid $380,299,609.00 listed as outlayed.  The obligated of $2,241,788,712.00 I would assume is owed when a milestone is reached.  Does anyone know what milestone this is? (completion of the test flights?)  The $4,556,076,044.00 is the potential total payments if all of the terms of the contract is met.  That is my guess as to what this means.

[Coastal Ron]

[...]  the Firm Fixed Price (FFP) contract milestones that NASA created were insufficient to find out the true progress of what Boeing was doing [which] is what NASA needs to figure out better.

100% agreement here. What insight must NASA require, and when? And can NASA develop the ability to respond effectively when that insight implies an effort is off-track?

When did (vs. should have) NASA become aware that the Boeing thermal model for the thruster doghouse was inadequate? In a set of perfect milestones, when would Boeing have missed one?

I still don't understand why it wasn't caught during OFT-2.

From the article on Ars Technica two days ago on this topic:

Several of the reaction control system thrusters stopped working as Starliner approached the space station on the OFT-2 mission, and another one failed on the return leg of the mission. Engineers thought they fixed the problem by introducing what was essentially a software fix to adjust timing and tolerance settings on sensors in the propulsion system, supplied by Aerojet Rocketdyne.

That didn't work. The problem lay elsewhere, as engineers discovered during testing this summer, when Starliner was already in orbit.

Maybe NASA didn't push hard enough for the root cause of the thruster issue? I'm not qualified to judge, but hopefully we hear more from NASA about how this problem was not caught earlier, and was misdiagnosed.

[DanClemmensen]

I do not know how much NASA has actually paid Boeing, which is an almost entirely different subject.

If I read this correctly, NASA has paid Boeing ~$2.24B of the up to ~$4.57B awarded.

https://www.usaspending.gov/award/CONT_IDV_NNK14MA75C_8000

I may be wrong, but I read this as they have paid $380,299,609.00 listed as outlayed.  The obligated of $2,241,788,712.00 I would assume is owed when a milestone is reached.  Does anyone know what milestone this is? (completion of the test flights?)  The $4,556,076,044.00 is the potential total payments if all of the terms of the contract is met.  That is my guess as to what this means.

Can someone translate this from federal contractese to English? I think I see $380 million actually paid, $ 2.2 Billion available to NASA to pay Boeing if milestones are met, and the rest of the $4.5 billion as a total of all payments to be made if Boeing completes the contract, including all six operational flights.

Do we know which milestone is the latest milestone that has been paid within the $380 million? Maybe the successful OFT or some CFT readiness milestone?

[tbellman]

If I read this correctly, NASA has paid Boeing ~$2.24B of the up to ~$4.57B awarded.

https://www.usaspending.gov/award/CONT_IDV_NNK14MA75C_8000

I may be wrong, but I read this as they have paid $380,299,609.00 listed as outlayed.  The obligated of $2,241,788,712.00 I would assume is owed when a milestone is reached.  Does anyone know what milestone this is? (completion of the test flights?)  The $4,556,076,044.00 is the potential total payments if all of the terms of the contract is met.  That is my guess as to what this means.

There's a glossary at the usaspending site, under "Find Resources" at the top right of the page.

But yes, you are basically right.  Outlayed amount is how much has actually been paid, while obligated amount is how much the agency has promised to pay (when the contractor delivers), and potential award amount is how much the contract is worth including all potential options (that the agency might not yet, or even ever, actually order).

Can someone translate this from federal contractese to English? I think I see $380 million actually paid, $ 2.2 Billion available to NASA to pay Boeing if milestones are met, and the rest of the $4.5 billion as a total of all payments to be made if Boeing completes the contract, including all six operational flights.

Almost.  Total obligated amount would include everything already paid, but "available" sounds like you are thinking of what's left in the budget to pay.

An example:
NASA wants to buy five pencils, gets an offer for $1/pencil, and accepts that.  However, they only have $3 left in their budget.  They would then enter a contract that actually orders three pencils, with the option of ordering one or two more pencils, pending congress appropriating another $2.  NASA then reserves those $3 in their budget, so they can pay when the contractor delivers.

• Potential award: $5.  Outlayed: $0.  Total obligated: $3.

The contractor now delivers two pencils.  NASA have those pencils go through acceptance testing, accepts them, and pays $2 to the contractor.  The total obligated amount stays at $3, but at least my layman's reading of your word "available" would be $1.

• Potential award: $5.  Outlayed: $2.  Total obligated: $3.

Congress now appropriates a full $3 more for NASA to buy pencils, a full $1 more than they requested!  NASA negotiates a contract modification to allow them the option of ordering up to five extra pencils (hey, Congress might feel generous and appropriate even more in the future, could be nice to not have to do a full procurement for a few extra pencils).

• Potential award: $8.  Outlayed: $2.  Total obligated: $3.

NASA now actually excercises the contract option, and orders three more pencils.

• Potential award: $8.  Outlayed: $2.  Total obligated: $6.

Contractor delivers the remaining four pencils ordered.  NASA pays for them.

• Potential award: $8.  Outlayed: $6.  Total obligated: $6.

NASA goes digital, buys iPads for $4711 each for all employees.  The six pencils they bought are thrown away ("expended").  The contractor goes bankrupt because they were staking their future on those final two pencils that were never ordered.  A year later, NASA is unable to save Voyager 1, because debugging it required using pencils, something NASA no longer owns.

[woods170]

[...]  the Firm Fixed Price (FFP) contract milestones that NASA created were insufficient to find out the true progress of what Boeing was doing [which] is what NASA needs to figure out better.

100% agreement here. What insight must NASA require, and when? And can NASA develop the ability to respond effectively when that insight implies an effort is off-track?

When did (vs. should have) NASA become aware that the Boeing thermal model for the thruster doghouse was inadequate? In a set of perfect milestones, when would Boeing have missed one?

I still don't understand why it wasn't caught during OFT-2.

Because OFT-2 didn't carry people onboard who, prior to docking, did some heavy-duty manual attitude control of Starliner.
Butch's and Suni's effort stressed Starliner's RCS thrusters in quite a different way than the autonomous control system does. Certain RCS thrusters were fired for longer periods of time and more often under manual control, than the autonomous control system would have fired those thrusters. And that wasn't accounted for in the thermal model of Starliner's doghouses and thrusters.

As a result, the manual steering led to overheated Teflon seals in certain thrusters, which led to the extruding problem now believed to be the prime cause of the thrusters going offline during rendez-vous with the ISS.


SpaceX on the other hand, did verification testing on its Crew Dragon RCS thrusters based on the actual simulator results of manual Crew Dragon attitude control. They realised that manual steering stressed the thrusters in quite a different way, so they went and ground-fired their Draco's in the same manner. Guess what: it revealed an issue with valve seals and throat overheating. SpaceX then went to fix those issues, long before DM-2 ever left the ground. What are formally called Draco engines on Crew Dragon today, are in fact much evolved and improved RCS thrusters when compared to the Dracos originally mounted on Dragon 1.


Continuous iteration and improvement offers many advantages over just buying a standard set of RCS thrusters from Aerojet. Something that Boeing has now found out the hard way.

[Mariusuiram]

I believe NASA provided Boeing with an NTP (Notice to Proceed) on 3 of their 6 missions in the CCtCap contract. That was probably necessary for Boeing to start procurement and such (1-2 year lead time if this mission had gone well in the spring).

So I would guess that the $2.24 billion represents all the development funding and the first 3 operational flights. By providing Boeing with an NTP, NASA obligated themselves to pay for those flights. If the ISS ends early and NASA has no need, they would still have to pay Boeing (or use the service somehow), whereas the final 3 are contracted but without the NTP are not obligated.

That is at least how I read it. But having half the contract not obligated yet seems high. That would mean there is $2.25 billion of the contract tied to the final 3 flights which seems too high ($750 million per flight). $2.25 billion divided by 6 flights seems like a more realistic number. ~$350 million per flight.

Or there is more than just flights in that $2.25 billion remaining and there are additional services

[mn]

[...]  the Firm Fixed Price (FFP) contract milestones that NASA created were insufficient to find out the true progress of what Boeing was doing [which] is what NASA needs to figure out better.

100% agreement here. What insight must NASA require, and when? And can NASA develop the ability to respond effectively when that insight implies an effort is off-track?

When did (vs. should have) NASA become aware that the Boeing thermal model for the thruster doghouse was inadequate? In a set of perfect milestones, when would Boeing have missed one?

I still don't understand why it wasn't caught during OFT-2.

Because OFT-2 didn't carry people onboard who, prior to docking, did some heavy-duty manual attitude control of Starliner.
Butch's and Suni's effort stressed Starliner's RCS thrusters in quite a different way than the autonomous control system does. Certain RCS thrusters were fired for longer periods of time and more often under manual control, than the autonomous control system would have fired those thrusters. And that wasn't accounted for in the thermal model of Starliner's doghouses and thrusters.

As a result, the manual steering led to overheated Teflon seals in certain thrusters, which led to the extruding problem now believed to be the prime cause of the thrusters going offline during rendez-vous with the ISS.


SpaceX on the other hand, did verification testing on its Crew Dragon RCS thrusters based on the actual simulator results of manual Crew Dragon attitude control. They realised that manual steering stressed the thrusters in quite a different way, so they went and ground-fired their Draco's in the same manner. Guess what: it revealed an issue with valve seals and throat overheating. SpaceX then went to fix those issues, long before DM-2 ever left the ground. What are formally called Draco engines on Crew Dragon today, are in fact much evolved and improved RCS thrusters when compared to the Dracos originally mounted on Dragon 1.


Continuous iteration and improvement offers many advantages over just buying a standard set of RCS thrusters from Aerojet. Something that Boeing has now found out the hard way.

Curious if there's any information available as to why manual control stressed the thrusters beyond expectations on both crew vehicles? Any analysis, is it something that could be addressed with better crew training perhaps?

[abaddon]

Manual piloting on both test flights wasn’t executing a straightforward approach and docking, but qualifying the system to be piloted.  Presumably that kind of flight profile is deliberately more complicated and stressful as a result/by design.

[yg1968]

My conspiracy bias leads me to assume that while there was a meeting — just as described in the media briefing — ending with unanimous consensus within NASA leadership, there were prior agreements made covertly among the power brokers. Just a theory: Boeing presented "We're prepared to conduct either crewed or uncrewed return" in exchange for a covert agreement that this offer would mean a successful uncrewed return puts them on a path to fly the next Starliner as a 4-crew rotation mission. That would be after the thruster over-heating issue is addressed to everyone's satisfaction of course.

This also explains the emphasis at the media briefing on this being a NASA decision. No good faith interpretation of the contract would include the idea that NASA, at its whim, could refuse to put its astronauts on the return leg and thereby put Boeing in violation of its contract obligations. I recognize that's twisted thinking, but contract lawyers are famous for that.

That seems incredibly unlikely. The most likely explanation is the one given by NASA and is also the simplest (Occam's razor). Until NASA gets Starliner back, no decision has been made or can be made since important data on the return portion of the mission is missing. Bowersox said that all options remain open at this point. 

[mn]

Manual piloting on both test flights wasn’t executing a straightforward approach and docking, but qualifying the system to be piloted.  Presumably that kind of flight profile is deliberately more complicated and stressful as a result/by design.

If that is deliberate and by design in both vehicles, then it should not have come as a surprise and require(d) changes after simulating (for SpaceX) or live testing (for Boeing)?

[abaddon]

Manual piloting on both test flights wasn’t executing a straightforward approach and docking, but qualifying the system to be piloted.  Presumably that kind of flight profile is deliberately more complicated and stressful as a result/by design.

If that is deliberate and by design in both vehicles, then it should not have come as a surprise and require(d) changes after simulating (for SpaceX) or live testing (for Boeing)?

Let me quote @woods170 from above:

SpaceX [...] did verification testing on its Crew Dragon RCS thrusters based on the actual simulator results of manual Crew Dragon attitude control.

So, SpaceX had astronauts fly the spacecraft in simulation.  They then used the thruster firing sequences in actual on-the-ground testing and discovered unexpected issues that weren't accounted for in their modeling and simulation of the spacecraft thrusters.  They redesigned the thruster systems to address these issues before the first crewed flight.  At what part in this sequence are you asserting that "it should not have come as a surprise"?  Do you think models and simulation should be perfect so no testing should be required?

As for Boeing... mistakes were made, clearly.

[DanClemmensen]

SpaceX on the other hand, did verification testing on its Crew Dragon RCS thrusters based on the actual simulator results of manual Crew Dragon attitude control. They realised that manual steering stressed the thrusters in quite a different way, so they went and ground-fired their Draco's in the same manner. Guess what: it revealed an issue with valve seals and throat overheating. SpaceX then went to fix those issues, long before DM-2 ever left the ground. What are formally called Draco engines on Crew Dragon today, are in fact much evolved and improved RCS thrusters when compared to the Dracos originally mounted on Dragon 1.

Wow. That's pretty damning. I absolutely believe you, but was this story known to the public before now? I assume it was known to NASA, and surely NASA would have at least mentioned it to Boeing.