-
#1280
by
Coastal Ron
on 01 Mar, 2020 00:38
-
(snip)
As soon as DM-2 is successful, NASA should end the Boeing contract.
Hold on.
The whole point of Commercial Crew is redundant and reliable crew transport to the ISS. SpaceX could do it alone. The question is, "should they?"
My answer: No way.
(snip)
Soyuz is our only backup, and THAT didn't look too good after October 2018 with Soyuz MS-10.
Way
That’s a well researched history but it doesn’t convince me.
Soyuz after MS-10 didn’t look good, but was adequate.
Soyuz and Dragon will be enough redundancy.
Redundancy for keeping a minimum crew on the ISS, but not redundancy for keeping the ISS fully staffed. You need two Commercial Crew vehicles for that.
Boeing was asked how much money they wanted for capsule development and they didn’t include adequate test systems, integrated system testing, or modern software methodology at 1.5X SpaceX’s contract.
I've worked for a number of large government contractors, and while perfection on executing a contract is the goal, sometimes things go wrong. Boeing is far enough along that they can recover from this and perform as required by the contract.
-
#1281
by
Vettedrmr
on 01 Mar, 2020 01:57
-
Hold on.
The whole point of Commercial Crew is redundant and reliable crew transport to the ISS.
Absolutely correct. A real shame that Boeing is so far away from the "reliable" part of crew transport. SpaceX hasn't demonstrated that either, but there are no known roadblocks to achieving that.
I really hope that Boeing gets their act together. If they don't, they're getting close to a precipice that may be extraordinarily painful as a corporation to recover from.
Have a good one,
Mike
-
#1282
by
Remes
on 01 Mar, 2020 04:53
-
2. No Agile, no continuous development, no continuous testing, or no continuous builds, Boeing using the 1970s Software Engineering method. Boeing finds mistakes too late, causing huge delays because the waterfall program management emphasizes testing after code-complete. In this far worse case, Boeing discovers their problems after launch because they did not discover inadequate testing which is one of the many benefits of continuous testing.
Agile is not suitable for every kind of project, especially not for safety critical highly complex systems spanning multiple entities. Also the requirements doesn't change all the time and are clear from the beginning.
3. Lots of talking around issues. For example, there is not an explanation why having the hardware propulsion controller would identify a bad valve mapping vs the emulator. How does a propulsion controller tell them they have the wrong mapping? It makes little sense.
I think we are talking hardware in the loop. The GNC sends out commands, you simulate how the spacecraft would behave and feed it back to the gnc. I assume the emulator had the right mapping and the thrust controller did not.
They did exactly this (told in an earlier interview) while on orbit and discovered the flaw.
I'm fine with that explanation and it makes sense to me. What doesn't make sense to me is, that there was only one single point in time, where they would make this tests and then they didn't have the right equipment. One test? I would expect these tests executed 24/7 in the weeks before the mission.
Looks to me like they were really in a hurry, getting this thing launched as fast as possible. Skipping the right amount of tests.
-
#1283
by
woods170
on 01 Mar, 2020 14:34
-
twitter.com/chrisg_nsf/status/1233437055415472128
Why didn't Boeing verify all the software before flight? Why only now?
John: Team developed what they thought would be adequate. testing. Hindsight proved different. "This team didn't take shortcuts. Abundance of testing. But we have gaps to fill." 1/2 #Beoing #Starliner
Emphasis mine.
Darnit. Why didn't anyone bother to ask the next logical questions (did the team have its chosen test strategy reviewed? And if yes, by whom?) after John revealed this little gem...
-
#1284
by
xyv
on 01 Mar, 2020 17:01
-
2. No Agile, no continuous development, no continuous testing, or no continuous builds, Boeing using the 1970s Software Engineering method. Boeing finds mistakes too late, causing huge delays because the waterfall program management emphasizes testing after code-complete. In this far worse case, Boeing discovers their problems after launch because they did not discover inadequate testing which is one of the many benefits of continuous testing.
Agile is not suitable for every kind of project, especially not for safety critical highly complex systems spanning multiple entities. Also the requirements doesn't change all the time and are clear from the beginning.
Some aspects of agile are suitable for any complex software project. In particular, continuous integration and test - the first time I heard the term "nightly build" I was terrified because a code build was something you rarely did in aerospace programs. It means that any code that is touched is checked back in, built into the "top of tree" code and tested using a test bench that grows with the code base. When you get into safety products (e.g. automated braking systems) the test specification is at least as important as the development specification and they are written together. Continuous test is a natural approach to this type of development.
3. Lots of talking around issues. For example, there is not an explanation why having the hardware propulsion controller would identify a bad valve mapping vs the emulator. How does a propulsion controller tell them they have the wrong mapping? It makes little sense.
I think we are talking hardware in the loop. The GNC sends out commands, you simulate how the spacecraft would behave and feed it back to the gnc. I assume the emulator had the right mapping and the thrust controller did not.
They did exactly this (told in an earlier interview) while on orbit and discovered the flaw.
I'm fine with that explanation and it makes sense to me. What doesn't make sense to me is, that there was only one single point in time, where they would make this tests and then they didn't have the right equipment. One test? I would expect these tests executed 24/7 in the weeks before the mission.
Looks to me like they were really in a hurry, getting this thing launched as fast as possible. Skipping the right amount of tests.
What I'm not fine with is the controller wasn't available for the test - this is the other thing that comes from a more agile like approach - a hardware rich environment. We have a bank of cameras that get the nightly build and are tested continuously. Boeing is likely treating these pieces of hardware as precious tracked items that are built in numbers defined by the contract instead of the numbers that the development team needs. Again, very much old space.
Remember all of the naysayers calling Space X "a software company"? All companies are software companies and not by choice - it is becoming the most critical discipline.
-
#1285
by
Khadgars
on 01 Mar, 2020 17:53
-
Hold on.
The whole point of Commercial Crew is redundant and reliable crew transport to the ISS.
Absolutely correct. A real shame that Boeing is so far away from the "reliable" part of crew transport. SpaceX hasn't demonstrated that either, but there are no known roadblocks to achieving that.
I really hope that Boeing gets their act together. If they don't, they're getting close to a precipice that may be extraordinarily painful as a corporation to recover from.
Have a good one,
Mike
If SpaceX can recover from two F9 failures, Boeing can recover from this. I agree Boeing has work to do, but they are not on the verge.
-
#1286
by
john smith 19
on 01 Mar, 2020 18:50
-
Note that for Commercial Cargo NASA have ended the contract for a supplier that failed to achieve its milestones.
Failure to achieve milestone --> No money paid for that milestone.
BTW Boeing required 63% more payment than SX to deliver so far substantially less.
SNC's Dream Chaser is still an option.
Of course it would be a massive blow to Boeing's corporate image, but isn't that Boeing's problem?
-
#1287
by
Khadgars
on 01 Mar, 2020 19:17
-
I find it amazing people are ready to write Boeing off. Yes there was a failure, but the capsule wasn't lost and completed over 80% of its objectives.
If this was SpaceX the opinions on here would be quite different.
-
#1288
by
Coastal Ron
on 01 Mar, 2020 19:43
-
I find it amazing people are ready to write Boeing off. Yes there was a failure, but the capsule wasn't lost and completed over 80% of its objectives.
If this was SpaceX the opinions on here would be quite different.
Oh, over the years SpaceX has been written off so many times that I know I've lost count. But it's a lot.
This is a software issue for Boeing, which in some ways is far less serious than the Super Draco explosion that SpaceX had on their crew vehicle, so as I mentioned in a post above I think Boeing can recover and proceed to execute the contract successfully.
-
#1289
by
meberbs
on 01 Mar, 2020 19:45
-
I find it amazing people are ready to write Boeing off. Yes there was a failure, but the capsule wasn't lost and completed over 80% of its objectives.
And I find it amazing anyone can make such distorted statements, failing 20% of objectives is terrible in this type of test, especially when failed objectives include very fundamental things.
If this was SpaceX the opinions on here would be quite different.
No they wouldn't. I am sick and tired of seeing these kinds of claims, they have no basis in reality, and serve no purpose other than to ignore the facts of whatever issue is at play. In this case even more so, since this is so serious, comparable examples of broken testing methodologies and general failures are hard to come by.
-
#1290
by
john smith 19
on 01 Mar, 2020 21:41
-
I find it amazing people are ready to write Boeing off.
No one is writing off Boeing. It's a company with turnover in the 10s of $Bn.
However the fee they requested was
substantially above what SX asked for and so far they have not delivered well.
Yes there was a failure, but the capsule wasn't lost and completed over 80% of its objectives.
The issue is
not the failures. It is the internal Boeing processes that
allowed those failures to work through their development process without being detected
before they got to a test flight.
If this was SpaceX the opinions on here would be quite different.
Almost certainly.
But wasn't that
why Boeing were selected to begin with?
Basically "We're Boeing. We built most of the US's human spaceflight vehicles. This is in our DNA. Your development programme is safe in our hands. We got this"
Except IRL it was their predecessor companies (or in some cases their predecessors predecessors predecessors companies) that did that building 40 years ago (in the case of NAA and the Shuttle) and the present crop of engineers are either unaware of how they did it or are not allowed to do it that way due to budget or time pressure.
If Boeing had charged 63%
less than SX rather than 63% more people would be saying they are amazed at how well they'd done given the minimal budget.
But they didn't. And the price premium does not seem to be showing up as increased quality of product
However, let us remember that Congress has
consistently underfunded the programme, for reasons that make little sense to me.
-
#1291
by
QuantumG
on 01 Mar, 2020 23:19
-
So much for NASA oversight.
-
#1292
by
Vettedrmr
on 01 Mar, 2020 23:26
-
So much for NASA oversight.
Yep. The stench of this isn't just stuck on Boeing...
-
#1293
by
EeeVee3
on 02 Mar, 2020 01:07
-
So much for NASA oversight.
NASA, NASA, NASA..... Commercial crew is a new procurement model for them too.
NASA has a budget.
NASA has an oversight crew.
I think they directed more resources to "keep an eye on the new kid on the block" and more modest resources towards Boing as "its Boing... they know what they are doing".
-
#1294
by
dglow
on 02 Mar, 2020 01:43
-
So much for NASA oversight.
Yep. The stench of this isn't just stuck on Boeing...
Sure, and perhaps most likely. At the same time we might ask whether NASA is sufficiently embedded and capable of catching either the MET or jet mapping problems OFT encountered.
Listen to
John Mulholland's Program Update from Friday. Both of these bugs sound pernicious. Should they have been caught, with better testing, before flight? Certainly.
But is it expected that oversight by NASA would have caught these oversights by Boeing? The "lack of end-to-end testing" headline turned out to be something much more subtle; the distribution of testing runs are what left the MET error undetected, and only by a matter of minutes. Should QA have performed a more thorough review and recommended greater temporal overlap of test runs? Yes. Would we expect that insight to have come from NASA? I'm not so sure.
The same could be said of the jet mapping error. Why would NASA know the emulator was faulty, or even recognize it was being used in lieu of flight hardware? I think this kind of responsibility falls squarely on Boeing.
Commercial Crew is supposed to benefit from a different set of responsibilities and incentives. NASA did not design Starliner. NASA is not buying Starliners.
-
#1295
by
Vettedrmr
on 02 Mar, 2020 01:50
-
I don't expect NASA to have the technical expertise to identify the software failures. I do expect them to be able to review the test coverage reports on the code, audit the traceability tables between the requirements and test cases, ensure that there are test cases that adequately exercise the defined interfaces, and finally to verify the integration tests between the various subsystems.
That's what our fighter aircraft customers did for every program I ever worked on.
-
#1296
by
Rocket Science
on 02 Mar, 2020 05:26
-
Something didn't seem right to me for some time and I never shared it here. I found it strange when they had to add an aero-skirt to the spacecraft and to me it seemed like last minute patch to the stack's aerodynamics. I'm glad they caught it, but it just appeared to be odd and rather late in the aero program... Just my 2 cents...
-
#1297
by
Vettedrmr
on 02 Mar, 2020 12:32
-
I had the same thought. What research I did said that it was added after some modelling (by ULA, IIRC) showed problems with supersonic flow onto the Atlas rocket. From what I've read it did the job just fine, although it looks awkward to me.
Have a good one,
Mike
-
#1298
by
haywoodfloyd
on 02 Mar, 2020 13:32
-
/rant
The more I read about this, the more I am gobsmacked.
"I really don't want you or anyone to have the impression that this team, tried to take shortcuts," said John Mulholland, Boeing Starliner program manager, told reporters Friday. The software team "did an abundance of testing. Obviously we have gaps to go fill."
The question is not "how much " testing was done but what kinds of testing.
I re-read this over and over again and I still can't make any sense of it.
When you build a car, you manufacture all the parts and test them to make sure they work. Then you put them all together and sell the car? Without testing the car with all the pieces in place??
Then he addressed the "end-to-end" testing by saying...
You can understand with the length of that run would be incredibly long," he said. "The team at the time decided that they would rather run multiple tests of different chunks of the mission."
So they couldn't set aside 24 hours to run a full-up test. Why. Did they have somewhere to go? That kind of statement just beggars belief, period, full stop.
What I have no trouble believing is that they obviously saw no problem with their methodology and that is the most worrisome aspect of all. And, on top of that NASA saw no problem with it either. Or maybe they didn't even look at it.
With all these "revelations" taken into account, why would anyone even consider anything other than another complete OFT?
/end rant
-
#1299
by
Khadgars
on 02 Mar, 2020 13:42
-
/rant
The more I read about this, the more I am gobsmacked.
"I really don't want you or anyone to have the impression that this team, tried to take shortcuts," said John Mulholland, Boeing Starliner program manager, told reporters Friday. The software team "did an abundance of testing. Obviously we have gaps to go fill."
The question is not "how much " testing was done but what kinds of testing.
I re-read this over and over again and I still can't make any sense of it.
When you build a car, you manufacture all the parts and test them to make sure they work. Then you put them all together and sell the car? Without testing the car with all the pieces in place??
Then he addressed the "end-to-end" testing by saying...
You can understand with the length of that run would be incredibly long," he said. "The team at the time decided that they would rather run multiple tests of different chunks of the mission."
So they couldn't set aside 24 hours to run a full-up test. Why. Did they have somewhere to go? That kind of statement just beggars belief, period, full stop.
What I have no trouble believing is that they obviously saw no problem with their methodology and that is the most worrisome aspect of all. And, on top of that NASA saw no problem with it either. Or maybe they didn't even look at it.
With all these "revelations" taken into account, why would anyone even consider anything other than another complete OFT?
/end rant
Hind sight is always 20/20. The testing methodology that Boeing used is not unusual, but they clearly missed important integration testing. But that is also the point of the live test.