Commercial Crew - Discussion Thread 3

[mme]

Much of the Boeing software design staff and software QA team should be fired.  At least their management should be.

That is absolutely the wrong thing to do and could lead to a worse outcome since the new team could make even more mistakes. The people who wrote and managed the software will have learned a lot more than if the mission had gone perfectly, and will be stronger for it.

It doesn't work like that in the real world. There is a saying about the old dog and new tricks and it also applies to us, humans. There is a point where you need to reshuffle the team so questions like " Why " are flying again and critical thinking is employed. The Boeing team SW design and implementation is so poor, that just getting them a new boss will not repair the architecture& implementation errors. Their sufficiency and boredom/lack of involvement lead to this failure. Not management.

That's a lot of assuming - as is most of this thread. This was a really embarrassing outcome but we don't know why it happened. Before we assume everyone on the Boeing software team is an idiot maybe we should see if they figure out what the underlying cause was.

[Comga]

Much of the Boeing software design staff and software QA team should be fired.  At least their management should be.

That is absolutely the wrong thing to do and could lead to a worse outcome since the new team could make even more mistakes. The people who wrote and managed the software will have learned a lot more than if the mission had gone perfectly, and will be stronger for it.

It doesn't work like that in the real world. There is a saying about the old dog and new tricks and it also applies to us, humans. There is a point where you need to reshuffle the team so questions like " Why " are flying again and critical thinking is employed. The Boeing team SW design and implementation is so poor, that just getting them a new boss will not repair the architecture& implementation errors. Their sufficiency and boredom/lack of involvement lead to this failure. Not management.

That's a lot of assuming - as is most of this thread. This was a really embarrassing outcome but we don't know why it happened. Before we assume everyone on the Boeing software team is an idiot maybe we should see if they figure out what the underlying cause was.

No one is saying that they are all idiots.
Almost everyone may have done the task they were assigned with care and integrity.

But somewhere up in the hierarchy, several people seem to have messed up seriously, we can assume.

The architecture seems fragile, that “one line of code” done wrong could disrupt the mission and nearly cause an uncontrolled reentry. Then the testing didn’t find the immediate problem or the system fragility.

It’s way more than “embarrassing”. It’s extremely concerning IMO. If they didn’t find these seemingly simple issues what subtle ones remain unexplored?

These conclusions from the known facts are fairly generalized and not wild extrapolation or guesses.

[raketa]

I am IT architect and I see two basic issue:
-Design
-Testing
1/Design of time handshake between booster was accepted without additional checkup, for example comparing with timer on spacecraft and in case of big difference, stay on spacecraft  time and hold on and wait for launch team input.
I see lot of new developers make mistake to develop interface just for perfect day, not thinking about rainy days.

2/Test
You could not blame developer for failed code, this is test team responsibility. Developer is not able to find these error, since he is to familiar with code.
Most of unexpected errors happen in integration between systems.

[mme]

Much of the Boeing software design staff and software QA team should be fired.  At least their management should be.

That is absolutely the wrong thing to do and could lead to a worse outcome since the new team could make even more mistakes. The people who wrote and managed the software will have learned a lot more than if the mission had gone perfectly, and will be stronger for it.

It doesn't work like that in the real world. There is a saying about the old dog and new tricks and it also applies to us, humans. There is a point where you need to reshuffle the team so questions like " Why " are flying again and critical thinking is employed. The Boeing team SW design and implementation is so poor, that just getting them a new boss will not repair the architecture& implementation errors. Their sufficiency and boredom/lack of involvement lead to this failure. Not management.

That's a lot of assuming - as is most of this thread. This was a really embarrassing outcome but we don't know why it happened. Before we assume everyone on the Boeing software team is an idiot maybe we should see if they figure out what the underlying cause was.

No one is saying that they are all idiots.
Almost everyone may have done the task they were assigned with care and integrity.

But somewhere up in the hierarchy, several people seem to have messed up seriously, we can assume.

The architecture seems fragile, that “one line of code” done wrong could disrupt the mission and nearly cause an uncontrolled reentry. Then the testing didn’t find the immediate problem or the system fragility.

It’s way more than “embarrassing”. It’s extremely concerning IMO. If they didn’t find these seemingly simple issues what subtle ones remain unexplored?

These conclusions from the known facts are fairly generalized and not wild extrapolation or guesses.

Re-read the posts I am responding to. Not only claiming they are incompetent but they don't care and are basically saying everyone should be fired.

We have been told a timer was off and that it was set from a "wrong" memory location. I've debugged enough weird !@#$ in my life to know that things are not always what they seem. This could be a much more complicated edge case/unfortunate series of events than is obvious. Until I know what really went wrong and what the state the software was really in, I am not going to assume that a couple of statements for the general public tell the whole story.

[gongora]

Annual ASAP report is out, there are a few pages on Commercial Crew.

https://oiir.hq.nasa.gov/asap/documents/2019_ASAP_Report-TAGGED.pdf

Following the AMOS-6 mission launch pad anomaly in September 2016, there has been a tremendous amount of work done, both by NASA and by SpaceX, to better understand the design and operational parameters associated with composite overwrapped pressure vessels (COPVs). A multiyear study of COPVs has resulted in advancing the state of the art and increasing understanding of grain size, ignition risks, and non-destructive testing techniques. There has also been considerable progress made in evaluating both the benefits and the risks associated with crew insertion prior to propellant loading, sometimes referred to as “load-and-go.” During the past year, both issues and subsequent corrective actions have been thoroughly reviewed by NASA engineering and safety officials, and the residual risks have been accepted by the Program Manager. After many months of discussing these topics with both NASA and SpaceX, the ASAP is satisfied that NASA has executed an appropriate risk management process to address these issues

[woods170]

Annual ASAP report is out, there are a few pages on Commercial Crew.

https://oiir.hq.nasa.gov/asap/documents/2019_ASAP_Report-TAGGED.pdf

Following the AMOS-6 mission launch pad anomaly in September 2016, there has been a tremendous amount of work done, both by NASA and by SpaceX, to better understand the design and operational parameters associated with composite overwrapped pressure vessels (COPVs). A multiyear study of COPVs has resulted in advancing the state of the art and increasing understanding of grain size, ignition risks, and non-destructive testing techniques. There has also been considerable progress made in evaluating both the benefits and the risks associated with crew insertion prior to propellant loading, sometimes referred to as “load-and-go.” During the past year, both issues and subsequent corrective actions have been thoroughly reviewed by NASA engineering and safety officials, and the residual risks have been accepted by the Program Manager. After many months of discussing these topics with both NASA and SpaceX, the ASAP is satisfied that NASA has executed an appropriate risk management process to address these issues

Here's the short-short summary: Load-and-Go is no longer an issue.

[happyflower]

Is the big issue with Starliner a technical one or a legal one? In other words, is the discussion over the fact that the contract states "the capsule has to dock with the ISS" is causing issues?

As the administrator explained it, other crafts have also done their first human docking without an automated docking first. Can't NASA just say "we are changing the contract"? Contract is between these two entities after all and NASA will be the responsible party.

I don't think anybody wants to keep astronauts alive more than NASA.

[llanitedave]

No one is saying that they are all idiots.
Almost everyone may have done the task they were assigned with care and integrity.

But somewhere up in the hierarchy, several people seem to have messed up seriously, we can assume.

The architecture seems fragile, that “one line of code” done wrong could disrupt the mission and nearly cause an uncontrolled reentry. Then the testing didn’t find the immediate problem or the system fragility.

It’s way more than “embarrassing”. It’s extremely concerning IMO. If they didn’t find these seemingly simple issues what subtle ones remain unexplored?

These conclusions from the known facts are fairly generalized and not wild extrapolation or guesses.

Re-read the posts I am responding to. Not only claiming they are incompetent but they don't care and are basically saying everyone should be fired.

We have been told a timer was off and that it was set from a "wrong" memory location. I've debugged enough weird !@#$ in my life to know that things are not always what they seem. This could be a much more complicated edge case/unfortunate series of events than is obvious. Until I know what really went wrong and what the state the software was really in, I am not going to assume that a couple of statements for the general public tell the whole story.

This should not be an "edge case."  The launch was on time and the spacecraft was placed on the correct trajectory.  This should have been a nominal case.


You put the system in a strange and unusual circumstance and you find a bug... that's a problem that needs fixing but it's understandable.  But when everything is going right down the middle of the design parameter, and it still fails, then that's a BIG problem, and a lot less understandable.

[savantu]

That's a lot of assuming - as is most of this thread. This was a really embarrassing outcome but we don't know why it happened. Before we assume everyone on the Boeing software team is an idiot maybe we should see if they figure out what the underlying cause was.

Don't put words in my mouth; I didn't said they were idiots or incompetent, I said they were complacent and bored; typical of mature, bureacratic dinosaurs organisations. I know what that feels like as I work in a company 140 years old. We have tens of thousands of SW enginereers, system architects and so on. You couldn't imagine the absolute crap they produce, bugged, late and always missing requested features. Overpromising, over-cost and under-delivering. Just like Boeing.

[saliva_sweet]

Can't NASA just say "we are changing the contract"? Contract is between these two entities after all and NASA will be the responsible party.

Probably difficult. It's a high profile contract involving 4 billion taxpayer dollars. I think it will be hard for NASA to say "oops, we accidentally ordered an unnecessary ISS docking. Now lets all forget about it".

[abaddon]

Regarding the recently mused-about extension of DM-2 to be a longer duration mission...

If NASA chooses to extend DM-2, it would likely delay DM-2 long enough that Starliner would launch first.  Right now, it seems more probable to me SpaceX will be ready to launch first, although who really knows, and these things can and do change all the time.

There are good reasons for NASA to consider extending DM-2, of course, most specifically insurance against Boeing's first crewed flight being delayed.  Add those two together and it sounds like a win-win for NASA; Boeing gets a little more room to get first crack at it, and if they can't get it done in time, NASA gets astro coverage from SpaceX rather than just an eight day mission and a further large gap until the first SpaceX long duration mission.

I will admit there's a lot of guessing on my part, but I wouldn't be surprised to see DM-2 extended.

[mgeagon]

Does anyone know an average time astronauts need to train for a six month mission to the ISS? This might inform the length of rightward movement for DM-2. It sounds like the Boeing crew has been training for an extended mission since the middle of last year, so I suppose they are just waiting for the hardware. Also, isn’t the decision to extend DM-2 dependent upon the outcome of the OFT investigation?

Edit: grammar

[Semmel]

In the SpaceX launch abort post flight press conference, Jim was asked why the decision to extend the mission was not already taken. Independently of weather the DM-2 is extended or not, the delay in making this decision is dubious to me. I dont understand what they are waiting for. The question was not answered btw.

[JonathanD]

In the SpaceX launch abort post flight press conference, Jim was asked why the decision to extend the mission was not already taken. Independently of weather the DM-2 is extended or not, the delay in making this decision is dubious to me. I dont understand what they are waiting for. The question was not answered btw.

My impression from the Administrator's comments is that they were sticking with Plan A (not extended) until they had a better idea of when each vehicle would actually fly with crew.  They still need to do the last round of approvals on F9/Dragon prior to a DM-2 mission, and they have to make their decision on whether to waive the requirement for a second uncrewed mission from Starliner to demonstrate ISS docking.  Once these come into a little more focus I think they'll feel ready to make the decision on whether to extend.

One could certainly argue that they should have assumed they will wind up extending the first crewed Dragon mission and begun providing the additional training for Bob Behnken and Doug Hurley already.  But it may be more complicated than that in terms of timing, duration, and tasks.  I don't know, just speculating.

[rcoppola]

NASA may have gotten a little ahead of itself on this one:

After Dragon's abort fuel system test anomaly, SpaceX brought their short duration DM-2 capsule forward and used it for the IFA. Their originally intended first mission certified crew capsule, now slotted for DM-2, was now going to be long duration mission capable. But NASA probably thought it would take longer for SpaceX to recover from both the anomaly and the parachute redesign & testing program. Regardless, they mainly counted on Boeing having a nominal OFT since Boeing was already slated to have a long duration capable CST for their CFT and previously contracted for a long duration mission.

But...SpaceX recovered quickly, had a seemingly flawless IFA and MkIII  parachute qualification is almost completed with their DM-2 vehicle ready to go by March.

Meanwhile...Boeing had an off-nominal OFT. They never reached the ISS and no clear indication of what's next besides some vague notion of an arbitrary two month investigation. It hasn't been long but we went from, frankly a bizarre set of premature statements that, "if crew were on board everything would have been fine (paraphrase)..." to silence.

So why has NASA not immediately transitioned Doug and Bob to long duration mission training just in case? For an agency that prides itself on having a back-up plan for the back-up plan, it's a mystery. Would you really want the very first long duration mission performed with a vehicle that has never actually been to the ISS? Who's only mission to space was off-nominal? Again, why wouldn't NASA immediately get Bob and Doug training. They can always cut the training short and send them up per the current plan if CFT gets cleared. Would the additional training keep them in Houston too much and away from Hawthorne to complete current DM-2 requirements in a timely manner? Is it just about training logistics? Why not transfer Nicole and Mike to join Bob and Doug. They're already training for ISS and Dragon is 4 seat capable. Yeh, I know, it's not that simple.

Or is this more about how to work within this new public / private paradigm.  Boeing was going to get more money for the extended mission. If SpaceX does it instead, does Boeing lose out? How does that work in this new era? So is this financial and/or dare I say political as well? Wish I knew but there's a lot more to this then simple scheduling...imo.

[abaddon]

Or is this more about how to work within this new public / private paradigm.  Boeing was going to get more money for the extended mission. If SpaceX does it instead, does Boeing lose out? How does that work in this new era? So is this financial and/or dare I say political as well? Wish I knew but there's a lot more to this then simple scheduling...imo.

The Boeing CFT has been extended, regardless of whether NASA decides to extend SpaceX DM-2.  So this shouldn't be a factor.

[rcoppola]

Or is this more about how to work within this new public / private paradigm.  Boeing was going to get more money for the extended mission. If SpaceX does it instead, does Boeing lose out? How does that work in this new era? So is this financial and/or dare I say political as well? Wish I knew but there's a lot more to this then simple scheduling...imo.

The Boeing CFT has been extended, regardless of whether NASA decides to extend SpaceX DM-2.  So this shouldn't be a factor.

Perhaps but wasn't the primary reason NASA extended Boeing's CFT was because the original DM-2 wasn't going to be capable of it? But now that it is, and if they went with an extended DM-2, would they still need to have an extended CFT?

[saliva_sweet]

In the SpaceX launch abort post flight press conference, Jim was asked why the decision to extend the mission was not already taken. Independently of weather the DM-2 is extended or not, the delay in making this decision is dubious to me. I dont understand what they are waiting for. The question was not answered btw.

I think the answer was given in a recent GAO report. Boeing threatened to bail out. A plan was devised. Boeing can skip the CFT and move straight to operational missions AND get extra money for this. This was and is not necessary for SpaceX and therefore was not done.

[freddo411]

In the SpaceX launch abort post flight press conference, Jim was asked why the decision to extend the mission was not already taken. Independently of weather the DM-2 is extended or not, the delay in making this decision is dubious to me. I dont understand what they are waiting for. The question was not answered btw.

I think the answer was given in a recent GAO report. Boeing threatened to bail out. A plan was devised. Boeing can skip the CFT and move straight to operational missions AND get extra money for this. This was and is not necessary for SpaceX and therefore was not done.

Most parents know that one does not reward a toddler throwing a tantrum.   It's not in anyone's best interest.   It's time to take the ice cream cone away and send them to the corner.

[Coastal Ron]

I think the answer was given in a recent GAO report. Boeing threatened to bail out. A plan was devised. Boeing can skip the CFT and move straight to operational missions AND get extra money for this. This was and is not necessary for SpaceX and therefore was not done.

Most parents know that one does not reward a toddler throwing a tantrum.   It's not in anyone's best interest.   It's time to take the ice cream cone away and send them to the corner.

Except that Commercial Crew is not a parent-child relationship, it's a business one. And NASA needs Boeing more than Boeing needs NASA for this particular contract - or at least that is what Boeing likely is trying to make it seem.