Commercial Crew - Discussion Thread 3

[abaddon]

SpaceX on the other hand, did verification testing on its Crew Dragon RCS thrusters based on the actual simulator results of manual Crew Dragon attitude control. They realised that manual steering stressed the thrusters in quite a different way, so they went and ground-fired their Draco's in the same manner. Guess what: it revealed an issue with valve seals and throat overheating. SpaceX then went to fix those issues, long before DM-2 ever left the ground. What are formally called Draco engines on Crew Dragon today, are in fact much evolved and improved RCS thrusters when compared to the Dracos originally mounted on Dragon 1.

Wow. That's pretty damning. I absolutely believe you, but was this story known to the public before now? I assume it was known to NASA, and surely NASA would have at least mentioned it to Boeing.

It has been abundantly clear that Boeing barely listened to NASA at all, let alone paying attention to what anyone at SpaceX was doing.

[DanClemmensen]

SpaceX on the other hand, did verification testing on its Crew Dragon RCS thrusters based on the actual simulator results of manual Crew Dragon attitude control. They realised that manual steering stressed the thrusters in quite a different way, so they went and ground-fired their Draco's in the same manner. Guess what: it revealed an issue with valve seals and throat overheating. SpaceX then went to fix those issues, long before DM-2 ever left the ground. What are formally called Draco engines on Crew Dragon today, are in fact much evolved and improved RCS thrusters when compared to the Dracos originally mounted on Dragon 1.

Wow. That's pretty damning. I absolutely believe you, but was this story known to the public before now? I assume it was known to NASA, and surely NASA would have at least mentioned it to Boeing.

It has been abundantly clear that Boeing barely listened to NASA at all, let alone paying attention to what anyone at SpaceX was doing.

If true, then you have identified two different failures. NASA and Boeing need to address these failures. I have no idea how to classify the failures, or what the proper corrective action should be, but NASA has made such a correction before in response to the Shuttle disasters. It required a change in the culture in that case.

[mn]

Manual piloting on both test flights wasn’t executing a straightforward approach and docking, but qualifying the system to be piloted.  Presumably that kind of flight profile is deliberately more complicated and stressful as a result/by design.

If that is deliberate and by design in both vehicles, then it should not have come as a surprise and require(d) changes after simulating (for SpaceX) or live testing (for Boeing)?

Let me quote @woods170 from above:

SpaceX [...] did verification testing on its Crew Dragon RCS thrusters based on the actual simulator results of manual Crew Dragon attitude control.

So, SpaceX had astronauts fly the spacecraft in simulation.  They then used the thruster firing sequences in actual on-the-ground testing and discovered unexpected issues that weren't accounted for in their modeling and simulation of the spacecraft thrusters.  They redesigned the thruster systems to address these issues before the first crewed flight.  At what part in this sequence are you asserting that "it should not have come as a surprise"?  Do you think models and simulation should be perfect so no testing should be required?

As for Boeing... mistakes were made, clearly.

1. If we have two cases SpaceX and Boeing, both having the same situation where the actual manual control was different than modeled that suggests that there was a common misunderstanding of how manual control would play out, that seems interesting and I was wondering if there was any additional info about that. (and it's not like manual control was something new never done before)

2. Abbadon stated that "is deliberately more complicated and stressful", if it is deliberate, you can't really say that is the reason they had to make changes after testing. If on the other hand the test showed that it was even more stressful than the already 'expected and deliberate  more stressful' than the reason is because the model was not accurate, which I understand. that is just different than what I understood Abbadon to be saying.

[abaddon]

Manual piloting on both test flights wasn’t executing a straightforward approach and docking, but qualifying the system to be piloted.  Presumably that kind of flight profile is deliberately more complicated and stressful as a result/by design.

If that is deliberate and by design in both vehicles, then it should not have come as a surprise and require(d) changes after simulating (for SpaceX) or live testing (for Boeing)?

Let me quote @woods170 from above:

SpaceX [...] did verification testing on its Crew Dragon RCS thrusters based on the actual simulator results of manual Crew Dragon attitude control.

So, SpaceX had astronauts fly the spacecraft in simulation.  They then used the thruster firing sequences in actual on-the-ground testing and discovered unexpected issues that weren't accounted for in their modeling and simulation of the spacecraft thrusters.  They redesigned the thruster systems to address these issues before the first crewed flight.  At what part in this sequence are you asserting that "it should not have come as a surprise"?  Do you think models and simulation should be perfect so no testing should be required?

As for Boeing... mistakes were made, clearly.

1. If we have two cases SpaceX and Boeing, both having the same situation where the actual manual control was different than modeled that suggests that there was a common misunderstanding of how manual control would play out, that seems interesting and I was wondering if there was any additional info about that. (and it's not like manual control was something new never done before)

2. Abbadon stated that "is deliberately more complicated and stressful", if it is deliberate, you can't really say that is the reason they had to make changes after testing. If on the other hand the test showed that it was even more stressful than the already 'expected and deliberate  more stressful' than the reason is because the model was not accurate, which I understand. that is just different than what I understood Abbadon to be saying.

No, that's exactly what I am saying.

The manual approaches during the crewed the test flights are deliberately more stressful and complex and cover more scenarios than a nominal automated approach.  The goal is to qualify the spacecraft for such possible maneuvers for contigencies.  Based on what @woods170 posted, both SpaceX and Boeing simulation and modeling of these maneuvers did not uncover issues that happened with real-life testing (on the ground for SpaceX, in space for Boeing).  Despite the SpaceX simulation/modeling/testing showing this possibility, Boeing didn't conduct on-the-ground testing of these types of scenarios until the spacecraft experienced the issues on the crewed flight.

Hopefully that clarifies what I am saying.

[Nomadd]

 It would be interesting to see Aerojet specs compared to actual manned usage.
 One big mistake (to put it politely) I've seen a lot is using margin to pass something. When they do that, it's like the often stupid (Can't think of a more polite term) tendency to use similar redundancy to weaken standards, which can actually increase the probability of something going south.

[sdsds]

My conspiracy bias leads me to assume [...]

That seems incredibly unlikely. The most likely explanation is the one given by NASA and is also the simplest (Occam's razor). Until, NASA gets Starliner back, no decision has been made or can be made since important data on the return portion of the mission is missing. Bowersox said that all options remain open at this point. 

That's absolutely valid. The simplest explanation is usually correct. As simple as possible, but no simpler. It takes a somewhat Machiavellian world view to propose what's essentially the reverse of Hanlon's Razor. It would be something like, "Never blindly attribute to the stupidity of your adversary what could be explained by their cleverness."

Tangentially related but directly on the subject of Commercial Crew, it's interesting to consider if the best outcome for all parties at this point might be if Starliner CFT suffered a major anomaly during its return. The loss of Calypso would put Boeing in a situation where they couldn't fulfill the contract so they would have a perfectly sensible reason to exit a money-losing program, and that would leave NASA free to recompete (per @jongoff's suggestion) Commercial Crew services.

[abaddon]

Tangentially related but directly on the subject of Commercial Crew, it's interesting to consider if the best outcome for all parties at this point might be if Starliner CFT suffered a major anomaly during its return. The loss of Calypso would put Boeing in a situation where they couldn't fulfill the contract so they would have a perfectly sensible reason to exit a money-losing program, and that would leave NASA free to recompete (per @jongoff's suggestion) Commercial Crew services.

I really don't see how that benefits Boeing in any way.  The kind of news splash a lost capsule would generate, when considered with all of the other failures Boeing has had as a company that cost lives, after Boeing publicly lobbied for Starliner to return with crew?  The hit on the stock price and reputation of the company would be devastating.

The best outcome is Starliner returns flawlessly and Boeing works with NASA to fix the thruster issues to NASA's satisfaction and can fly out their six flights.

[jongoff]

Jonathan Goff argues in a blog post that NASA should cancel the Starliner contract and fund someone else to develop a replacement: https://selenianboondocks.com/2024/08/starliner-reponendum-est-boeings-troubled-capsule-needs-to-be-replaced/. According to his blog bio he is "Product Strategy Lead for the space station startup Gravitics", so he likely has a conflict of interest, but people with conflicts of interest can still make interesting arguments. He's apparently planning to post two more blog posts on this subject soon. (He didn't discuss who might develop the replacement but I guess the winner of the new procurement would most likely be either Sierra Space's DreamChaser or a Blue Origin vehicle.)

It seems to me that cancellation would be OK so NASA should hold Boeing to the contract even if they threaten to cancel. The only flexibility I would give Boeing is if another test flight is required and it delivers useful cargo NASA could pay Boeing the market price for delivering that cargo. I'm not yet convinced that NASA should proactively cancel, partly because I don't know how the contract handles termination by Boeing, termination by NASA for cause, and termination by NASA for convenience of the government.

Goff's blog post asserts that NASA has already decided that the CFT cannot be certified,so another CFT will be needed. He cites a NASA press release about Saturday's press conference:
    https://www.nasa.gov/news-release/nasa-decides-to-bring-starliner-spacecraft-back-to-earth-without-crew/
However, neither the press release nor my recollection of the press conference actually flatly said that the CFT would not be certified.

I was basing it off the statement "Following Starliner’s return, the agency will review all mission-related data to inform what additional actions are required to meet NASA’s certification requirements."

That to me implies that the "because the Starliner hasn’t met all of its test objectives, that it would not be considered certified for crew launches yet".

Though it is absolutely true that they could in theory just have some sort of review on whatever fixes they need, and NASA could count *that* as qualifying it for certification. Sorry if I was a little imprecise.

And yeah, before anyone else mentions it, I know I need to actually write the next two blog posts... I'm sorry I've gotten so rusty at getting those written in a timely fashion.

~Jon

[woods170]

Manual piloting on both test flights wasn’t executing a straightforward approach and docking, but qualifying the system to be piloted.  Presumably that kind of flight profile is deliberately more complicated and stressful as a result/by design.

If that is deliberate and by design in both vehicles, then it should not have come as a surprise and require(d) changes after simulating (for SpaceX) or live testing (for Boeing)?

Let me quote @woods170 from above:

SpaceX [...] did verification testing on its Crew Dragon RCS thrusters based on the actual simulator results of manual Crew Dragon attitude control.

So, SpaceX had astronauts fly the spacecraft in simulation.  They then used the thruster firing sequences in actual on-the-ground testing and discovered unexpected issues that weren't accounted for in their modeling and simulation of the spacecraft thrusters.  They redesigned the thruster systems to address these issues before the first crewed flight.  At what part in this sequence are you asserting that "it should not have come as a surprise"?  Do you think models and simulation should be perfect so no testing should be required?

As for Boeing... mistakes were made, clearly.

Emphasis mine.

Exactly. Initial models are only as good as the assumptions and calculations on which they are based. Any sensible person will eventually compare the models to measured and observed reality and adjust the models to better represent reality. That is exactly what SpaceX did, and Boeing somehow did not.


Of note: the Draco thruster models were already very good, closely modeled based on real-life performance of Draco during operational CRS missions. And yet, there still was an unexpected surprise when pilot-induced firing patterns were run on original unmodified Draco thrusters.


Boeing has now been presented with a similar surprise, but this time on a crewed test flight.

[woods170]

SpaceX on the other hand, did verification testing on its Crew Dragon RCS thrusters based on the actual simulator results of manual Crew Dragon attitude control. They realised that manual steering stressed the thrusters in quite a different way, so they went and ground-fired their Draco's in the same manner. Guess what: it revealed an issue with valve seals and throat overheating. SpaceX then went to fix those issues, long before DM-2 ever left the ground. What are formally called Draco engines on Crew Dragon today, are in fact much evolved and improved RCS thrusters when compared to the Dracos originally mounted on Dragon 1.

Wow. That's pretty damning. I absolutely believe you, but was this story known to the public before now? I assume it was known to NASA, and surely NASA would have at least mentioned it to Boeing.

NASA has been fully aware of the Draco issues discovered in the run up to orbital flight tests. They had full insight and had people imbedded in the teams that modified the Draco design. And no, given the proprietary- and in-house nature of the Draco design, the specifics of the issues were not communicated to Boeing, nor Aerojet.

Also, it is just one of many stories regarding the development of the Commercial Crew vehicles that never made it into the public spotlight. The issue was discovered during the development and ground testing phase, which usually doesn't attract much public attention.

However, test flights are high visibility events with the vehicles fully in the public spotlight. If a thruster issue rears it head then, it is widely noticed.

[DanClemmensen]

SpaceX on the other hand, did verification testing on its Crew Dragon RCS thrusters based on the actual simulator results of manual Crew Dragon attitude control. They realised that manual steering stressed the thrusters in quite a different way, so they went and ground-fired their Draco's in the same manner. Guess what: it revealed an issue with valve seals and throat overheating. SpaceX then went to fix those issues, long before DM-2 ever left the ground. What are formally called Draco engines on Crew Dragon today, are in fact much evolved and improved RCS thrusters when compared to the Dracos originally mounted on Dragon 1.

Wow. That's pretty damning. I absolutely believe you, but was this story known to the public before now? I assume it was known to NASA, and surely NASA would have at least mentioned it to Boeing.

NASA has been fully aware of the Draco issues discovered in the run up to orbital flight tests. They had full insight and had people imbedded in the teams that modified the Draco design. And no, given the proprietary- and in-house nature of the Draco design, the specifics of the issues were not communicated to Boeing, nor Aerojet.

Sure, NASA cannot leak Draco specifics to Boeing. However, I think there is a problem if this prevented NASA from telling Boeing that manual control added unexpected stress to the thrusters.

Yes, Boeing was still arrogant and dismissive back then (2015?) so it might not have been useful, and the tension between Boeing and Aerojet might have further degraded the information flow. I suppose the public will never know exactly why Boeing+Aerojet failed to run a simulation with actual astronaut-supplied firing sequences, but these artificial restrictions on information flow may have killed Starliner. There is a lesson to be learned in there somewhere. Is there any way to apply it?

[JayWee]

Yes, Boeing was still arrogant and dismissive back then (2015?) so it might not have been useful, and the tension between Boeing and Aerojet might have further degraded the information flow.

Absolutely agreed. Simple "Hey, simulate astro-fired manual sequence" would be all that's needed.

Keep in mind these tweets:
https://forum.nasaspaceflight.com/index.php?topic=49156.msg2358493#msg2358493

[DanClemmensen]

Yes, Boeing was still arrogant and dismissive back then (2015?) so it might not have been useful, and the tension between Boeing and Aerojet might have further degraded the information flow.

Absolutely agreed. Simple "Hey, simulate astro-fired manual sequence" would be all that's needed.

Keep in mind these tweets:
https://forum.nasaspaceflight.com/index.php?topic=49156.msg2358493#msg2358493

Except that the hypothetical conversation is:
NASA engineer: "I think you should simulate astro-fired manual sequence based on actual astronaut inputs in a flight simulator."
Boeing engineer Bob: "Well, ..."
Boeing manager: "Shut up, Bob. No that's a waste of time and money."

[FutureSpaceTourist]

https://arstechnica.com/space/2024/09/the-future-of-boeings-crewed-spaceflight-program-is-muddy-after-starliners-return/

The future of Boeing’s crewed spaceflight program is muddy after Starliner’s return
"The final chapter on Starliner has not been written yet."

by Eric Berger - Sep 11, 2024 3:01pm GMT

Nearly a decade ago to the day, I stood in the international terminal of Houston's main airport checking my phone. As I wanted to board a flight for Moscow, an announcement from NASA was imminent, with the agency due to make its selections for private companies that would transport astronauts to the International Space Station.

[sstli2]

https://arstechnica.com/space/2024/09/the-future-of-boeings-crewed-spaceflight-program-is-muddy-after-starliners-return/

One possible explanation is that Boeing has decided it will exit the Commercial Crew Program.

How do you reconcile that with Bill Nelson's claim that Boeing's CEO told him he remained committed to the program?

[GalacticIntruder]

https://arstechnica.com/space/2024/09/the-future-of-boeings-crewed-spaceflight-program-is-muddy-after-starliners-return/

One possible explanation is that Boeing has decided it will exit the Commercial Crew Program.

How do you reconcile that with Bill Nelson's claim that Boeing's CEO told him he remained committed to the program?

Nelson did say that but then immediately and ever since NASA and Boeing have been re-introducing ambiguity. Obviously both want to continue but reality and financial numbers and timelines are not on Boeing's side. New CEO has an opportunity to remove this albatross. Unless Boeing can get a bailout from Congress there is no way he can justify the program.

"But Boeing signed contract with NASA." So.

[DanClemmensen]

https://arstechnica.com/space/2024/09/the-future-of-boeings-crewed-spaceflight-program-is-muddy-after-starliners-return/

One possible explanation is that Boeing has decided it will exit the Commercial Crew Program.

How do you reconcile that with Bill Nelson's claim that Boeing's CEO told him he remained committed to the program?

My totally speculative and unsupported guess: Boeing's CEO and possibly Nelson also were implicitly assuming that NASA would find a way to to certify the CFT and pay Boeing for it, and then fly the first operational Starliner as the next Starliner flight. Boeing may feel that they will bail out if this cannot happen.

[abaddon]

My totally speculative and unsupported guess: Boeing's CEO and possibly Nelson were implicitly assuming that NASA would find a way to to certify the CFT and pay Boeing for it, also how they ended up with an expectation that we'll have it done by Dec 2024?, and then fly the first operational Starliner as the next Starliner flight. Boeing may feel that they will bail out if this cannot happen.

You refer to this in the past tense, but I don't think the possibility of this is at all closed.  I think that's the only way Boeing stays in; if NASA says they need ANOTHER self-funded CFT?  Buh-bye.

[joek]

My totally speculative and unsupported guess: Boeing's CEO and possibly Nelson were implicitly assuming that NASA would find a way to to certify the CFT and pay Boeing for it, also how they ended up with an expectation that we'll have it done by Dec 2024?, and then fly the first operational Starliner as the next Starliner flight. Boeing may feel that they will bail out if this cannot happen.

You refer to this in the past tense, but I don't think the possibility of this is at all closed.  I think that's the only way Boeing stays in; if NASA says they need ANOTHER self-funded CFT?  Buh-bye.

Yup. Think they will find a way to "conditionally" certify. That is, X% of requirements were met; Y% of requirements were not met. The remaining Y% have sufficient remediation or mitigation plans to reduce risk to an acceptable level and move to operational flights. (We've seen this movie before, although will be interesting to see if-how ASAP weighs in.)

[DanClemmensen]

My totally speculative and unsupported guess: Boeing's CEO and possibly Nelson were implicitly assuming that NASA would find a way to to certify the CFT and pay Boeing for it, also how they ended up with an expectation that we'll have it done by Dec 2024?, and then fly the first operational Starliner as the next Starliner flight. Boeing may feel that they will bail out if this cannot happen.

You refer to this in the past tense, but I don't think the possibility of this is at all closed.  I think that's the only way Boeing stays in; if NASA says they need ANOTHER self-funded CFT?  Buh-bye.

Yep. I was specifically referring to the situation at the time of the phone call between Nelson and the Boeing CEO. Yep, Boeing and possible Nelson may still be thinking that way. Whether or not the rest of NASA can sign up for this is the open question.

I personally hope that NASA remains firm and that Boeing bails out. As painful as that would be for everyone, I think it is less painful than the alternatives.