-
#1320 by leovinus on 06 Mar, 2020 15:38
-
Steve Cowen (NASAWatch): 60+ problems in program given time and expense surprising. How can NASA and Boeing restore confidence? (Heavy paraphrase).
IRT 61 "recommendations" not "design issues or changes", all very actionable. Distributing learning as broadly as possible.
Listening to the audio conference atm. All high-level talk so far.
First reaction, as a taxpayer, I expect a better and more concrete line-by-line summary of those 61 issues, preferably on a few pages of paper. Like a GAO report. Would be appreciated if NASA can make that.
Secondly, the way they talk about software issues unfortunately show an alarming amount of lack-of-experience with complex software systems.
-
#1321 by leovinus on 06 Mar, 2020 15:49
-
(snip)
As soon as DM-2 is successful, NASA should end the Boeing contract.
Hold on.
The whole point of Commercial Crew is redundant and reliable crew transport to the ISS. SpaceX could do it alone. The question is, "should they?"
My answer: No way.
(snip)
Soyuz is our only backup, and THAT didn't look too good after October 2018 with Soyuz MS-10.
Way
That’s a well researched history but it doesn’t convince me.
Soyuz after MS-10 didn’t look good, but was adequate.
Soyuz and Dragon will be enough redundancy.
Boeing was asked how much money they wanted for capsule development and they didn’t include adequate test systems, integrated system testing, or modern software methodology at 1.5X SpaceX’s contract.
Allowing Boeing to continue will only reinforce their habit of wringing money out of NASA.
There should be real consequences for inexcusable failure.
Granted SpaceX has not yet flown their crewed mission.
They may still have a “tungsten” problem.
But after DM-2, Starliner will have served enough of its risk reduction purpose.
except the "real consequences" would be to the entire crewed program. The point is to create a commercial launch industry, NOT to substitute a SpaceX monopoly for the current NASA monopoly. SpaceX has proven much -- including the ability to blow up lots of vehicles -- among them being their most "successful" one that was "ready" to fly crew. Redundancy makes sense, and Boeing, for all its failures here, is the one who is by far the closest to bringing the redundancy intended.
[snip]
It's the kid failing Engineering 101 versus the doctorate student having a setback in their dissertation because they stumbled across previously unknown information.
Listening to the current press conference, I keep thinking about this one line summary which hits the nail on the head. -
#1322 by dglow on 06 Mar, 2020 15:56
-
What is going on with this call?
Doug and Kathy from NASA spend multiple minutes describing why a decision on an OFT re-fly cannot be made yet, and will not be made without a thorough review of Boeing's forthcoming proposal. They fall over themselves explaining to the press why a decision is not yet warranted.
Following this, Jim Chilton of Boeing jumps in with a curt, "We're ready to re-fly, just need a decision from NASA."
Something about this contrast didn't feel right. Like Boeing either isn't playing ball, or is trying to pass the buck.
-
#1323 by leovinus on 06 Mar, 2020 15:57
-
Steve Cowen (NASAWatch): 60+ problems in program given time and expense surprising. How can NASA and Boeing restore confidence? (Heavy paraphrase).
IRT 61 "recommendations" not "design issues or changes", all very actionable. Distributing learning as broadly as possible.
[snip]
First reaction, as a taxpayer, I expect a better and more concrete line-by-line summary of those 61 issues, preferably on a few pages of paper. Like a GAO report. Would be appreciated if NASA can make that.
Cool, this question was just asked by "Joey from Reuters". See update thread. Thanks Joey and abaddon!Joey from Reuters: Can NASA make public a list of 61 corrective actions either in full or lightly reactive. Since IRT is sticking around on speed dial what will their involvement look like.
Lovarro: Haven't had that conversation yet.
Leuders: IRT needs to stick around while a solution for comm outage is investigated (still being worked). Make sure from a NASA Boeing joint team perspective to make sure corrective actions are fully implemented and vetted.
John: Asked IRT as developing specific work plan, running back to IRT to make sure action fully captured. Make sure everyone agrees. -
#1324 by kessdawg on 06 Mar, 2020 15:58
-
What is going on with this call?
Doug and Kathy from NASA spend multiple minutes describing why a decision on an OFT re-fly cannot be made yet, and will not be made without a thorough review of Boeing's forthcoming proposal. They fall over themselves explaining to the press why a decision is not yet warranted.
Following this, Jim Chilton of Boeing jumps in with a curt, "We're ready to re-fly, just need a decision from NASA."
Something about this contrast didn't feel right. Like Boeing either isn't playing ball, or is trying to pass the buck.
Probably a disagreement over who pays. -
#1325 by SWGlassPit on 06 Mar, 2020 15:59
-
First reaction, as a taxpayer, I expect a better and more concrete line-by-line summary of those 61 issues, preferably on a few pages of paper. Like a GAO report. Would be appreciated if NASA can make that.
For folks who are accustomed to previous NASA programs, this one is a tricky beast, because the CCtCap contracts allow the contractors to hold data much closer than the development and sustaining contracts for Shuttle and ISS did. Even if NASA has rights to examine them for their purposes, they don't necessarily have rights to disclose them, per the contracts. That's why, as they said in the telecon, there has to be a conversation. -
#1326 by SWGlassPit on 06 Mar, 2020 16:02
-
What is going on with this call?
Doug and Kathy from NASA spend multiple minutes describing why a decision on an OFT re-fly cannot be made yet, and will not be made without a thorough review of Boeing's forthcoming proposal. They fall over themselves explaining to the press why a decision is not yet warranted.
Following this, Jim Chilton of Boeing jumps in with a curt, "We're ready to re-fly, just need a decision from NASA."
Something about this contrast didn't feel right. Like Boeing either isn't playing ball, or is trying to pass the buck.
Probably a disagreement over who pays.
Maybe we could abstain from what amounts to a total WAG during this? -
#1327 by SoftwareDude on 06 Mar, 2020 16:02
-
What is going on with this call?
It's obvious that NASA is on the hook to pay for a large part if not all of an OFT refly.
Doug and Kathy from NASA spend multiple minutes describing why a decision on an OFT re-fly cannot be made yet, and will not be made without a thorough review of Boeing's forthcoming proposal. They fall over themselves explaining to the press why a decision is not yet warranted.
Following this, Jim Chilton of Boeing jumps in with a curt, "We're ready to re-fly, just need a decision from NASA."
Something about this contrast didn't feel right. Like Boeing either isn't playing ball, or is trying to pass the buck. -
#1328 by FutureSpaceTourist on 06 Mar, 2020 16:06
-
Lovarro: Two specific things: one thing in IRT we had a routine where there might have been four logical conditions, four ways the software could have run. Didn't check all four conditions, was a designer/tester choice. Not unusual you don't test all logical condition, do need to recognize we do need to test all logical conditions now. Process side, delegated too much authority to software board to approve changes and actions as it applies to software, software/hardware integration change should have been brought up all the way to engineering review board.
Wow. These are not subtle mistakes/omissions, but really basic ones. I cannot understand how a mature organisation could make such mistakes. This is what I might expect for an organisation with little experience that doesn’t understand the significance of what it is doing, in terms of criticality, and/or the importance of hardware & software integration. -
#1329 by SWGlassPit on 06 Mar, 2020 16:11
-
What is going on with this call?
It's obvious that NASA is on the hook to pay for a large part if not all of an OFT refly.
Doug and Kathy from NASA spend multiple minutes describing why a decision on an OFT re-fly cannot be made yet, and will not be made without a thorough review of Boeing's forthcoming proposal. They fall over themselves explaining to the press why a decision is not yet warranted.
Following this, Jim Chilton of Boeing jumps in with a curt, "We're ready to re-fly, just need a decision from NASA."
Something about this contrast didn't feel right. Like Boeing either isn't playing ball, or is trying to pass the buck.
I'm not seeing that. I think the issues are more complicated.
Obviously, you don't want to just say "go refly" before you've taken a close look at everything. Keep in mind they are also experiencing some schedule pressure to stop depending on Soyuz for crew transport, and they have to make sure they can maintain crewed status, while also protecting for the possibility that something goes wrong with one of the other crewed vehicles. Simply saying "it's obvious they're going to pay" is massively oversimplifying, if not wrong altogether. -
#1330 by SWGlassPit on 06 Mar, 2020 16:13
-
Lovarro: Two specific things: one thing in IRT we had a routine where there might have been four logical conditions, four ways the software could have run. Didn't check all four conditions, was a designer/tester choice. Not unusual you don't test all logical condition, do need to recognize we do need to test all logical conditions now. Process side, delegated too much authority to software board to approve changes and actions as it applies to software, software/hardware integration change should have been brought up all the way to engineering review board.
Wow. These are not subtle mistakes/omissions, but really basic ones. I cannot understand how a mature organisation could make such mistakes. This is what I might expect for an organisation with little experience that doesn’t understand the significance of what it is doing, in terms of criticality, and/or the importance of hardware & software integration.
If you're suggesting it's "basic" to do 100% code coverage in a hardware integration lab, I'm not sure I buy that. That's nowhere near as cheap or easy as people in this thread are making it out to be. You'd be talking millions of test cases at the absolute least. -
#1331 by SoftwareDude on 06 Mar, 2020 16:27
-
Boeing said go refly, they would be glad to. Only conclusion possible is Boeing's lawyers are better than NASA's lawyers.What is going on with this call?
It's obvious that NASA is on the hook to pay for a large part if not all of an OFT refly.
Doug and Kathy from NASA spend multiple minutes describing why a decision on an OFT re-fly cannot be made yet, and will not be made without a thorough review of Boeing's forthcoming proposal. They fall over themselves explaining to the press why a decision is not yet warranted.
Following this, Jim Chilton of Boeing jumps in with a curt, "We're ready to re-fly, just need a decision from NASA."
Something about this contrast didn't feel right. Like Boeing either isn't playing ball, or is trying to pass the buck.
I'm not seeing that. I think the issues are more complicated.
Obviously, you don't want to just say "go refly" before you've taken a close look at everything. Keep in mind they are also experiencing some schedule pressure to stop depending on Soyuz for crew transport, and they have to make sure they can maintain crewed status, while also protecting for the possibility that something goes wrong with one of the other crewed vehicles. Simply saying "it's obvious they're going to pay" is massively oversimplifying, if not wrong altogether. -
#1332 by SWGlassPit on 06 Mar, 2020 16:35
-
I'll just leave it at this: just because it's the only conclusion you can think of doesn't mean it's the only conclusion possible.
Boeing said go refly, they would be glad to. Only conclusion possible is Boeing's lawyers are better than NASA's lawyers.What is going on with this call?
It's obvious that NASA is on the hook to pay for a large part if not all of an OFT refly.
Doug and Kathy from NASA spend multiple minutes describing why a decision on an OFT re-fly cannot be made yet, and will not be made without a thorough review of Boeing's forthcoming proposal. They fall over themselves explaining to the press why a decision is not yet warranted.
Following this, Jim Chilton of Boeing jumps in with a curt, "We're ready to re-fly, just need a decision from NASA."
Something about this contrast didn't feel right. Like Boeing either isn't playing ball, or is trying to pass the buck.
I'm not seeing that. I think the issues are more complicated.
Obviously, you don't want to just say "go refly" before you've taken a close look at everything. Keep in mind they are also experiencing some schedule pressure to stop depending on Soyuz for crew transport, and they have to make sure they can maintain crewed status, while also protecting for the possibility that something goes wrong with one of the other crewed vehicles. Simply saying "it's obvious they're going to pay" is massively oversimplifying, if not wrong altogether. -
#1333 by FutureSpaceTourist on 06 Mar, 2020 16:35
-
Wow. These are not subtle mistakes/omissions, but really basic ones. I cannot understand how a mature organisation could make such mistakes. This is what I might expect for an organisation with little experience that doesn’t understand the significance of what it is doing, in terms of criticality, and/or the importance of hardware & software integration.
If you're suggesting it's "basic" to do 100% code coverage in a hardware integration lab, I'm not sure I buy that. That's nowhere near as cheap or easy as people in this thread are making it out to be. You'd be talking millions of test cases at the absolute least.
Condition coverage is not code coverage, nor do I think this has to (all) be in a hardware integration lab.
I would expect there to be tests at some level for all conditions that make up critical functions (e.g. functions that could contribute to loss of mission or loss of crew). As I understand it the OFT MET issue resulted from a whole condition in the requirement being omitted from the code and yet no testing identified that the condition was missing ... ?!
Shouldn’t need any flight hardware to verify the presence of the required condition in the code (unless the interface concerned was too hard to emulate / stub in a harness but I’d be very surprised if that was the case). -
#1334 by leovinus on 06 Mar, 2020 16:39
-
Lovarro: Two specific things: one thing in IRT we had a routine where there might have been four logical conditions, four ways the software could have run. Didn't check all four conditions, was a designer/tester choice. Not unusual you don't test all logical condition, do need to recognize we do need to test all logical conditions now. Process side, delegated too much authority to software board to approve changes and actions as it applies to software, software/hardware integration change should have been brought up all the way to engineering review board.
Wow. These are not subtle mistakes/omissions, but really basic ones. I cannot understand how a mature organisation could make such mistakes. This is what I might expect for an organisation with little experience that doesn’t understand the significance of what it is doing, in terms of criticality, and/or the importance of hardware & software integration.
If you're suggesting it's "basic" to do 100% code coverage in a hardware integration lab, I'm not sure I buy that.
Agreed, 100% code coverage in testing is not basic.That's nowhere near as cheap or easy as people in this thread are making it out to be. You'd be talking millions of test cases at the absolute least.
Instead of O(10^6) test cases, which does not work for several reasons, there are other procedures to guarantee maximum code coverage. In a previous life we used to do something like
(1) make testcases for important "do not fail" situations,
(2) do code fuzzing, i.e., provide random inputs to check code paths that were not exercised by (1),
(3) fix the crashes, identify code paths not checked at all via compiler and profiling tools,
go back to (1) until satisfied with code coverage and robustness.
From the information we get via these press conferences and discussions, it seems that (1) and (3) were done, but not such a closed loop iteration (1,2,3) to maximize the coverage.
Just my 2pc. -
#1335 by gaballard on 06 Mar, 2020 16:40
-
Lovarro: Two specific things: one thing in IRT we had a routine where there might have been four logical conditions, four ways the software could have run. Didn't check all four conditions, was a designer/tester choice. Not unusual you don't test all logical condition, do need to recognize we do need to test all logical conditions now. Process side, delegated too much authority to software board to approve changes and actions as it applies to software, software/hardware integration change should have been brought up all the way to engineering review board.
Wow. These are not subtle mistakes/omissions, but really basic ones. I cannot understand how a mature organisation could make such mistakes. This is what I might expect for an organisation with little experience that doesn’t understand the significance of what it is doing, in terms of criticality, and/or the importance of hardware & software integration.
If you're suggesting it's "basic" to do 100% code coverage in a hardware integration lab, I'm not sure I buy that. That's nowhere near as cheap or easy as people in this thread are making it out to be. You'd be talking millions of test cases at the absolute least.
Yet SpaceX manages to do it. This is human spaceflight. If you can’t commit to 100% code coverage, get out of the business. If you think it’s too expensive or difficult to do so, get out of the business. This isn’t a joke. If they can’t do what needs to be done they have no business attempting to send humans to space. -
#1336 by SWGlassPit on 06 Mar, 2020 16:41
-
Wow. These are not subtle mistakes/omissions, but really basic ones. I cannot understand how a mature organisation could make such mistakes. This is what I might expect for an organisation with little experience that doesn’t understand the significance of what it is doing, in terms of criticality, and/or the importance of hardware & software integration.
If you're suggesting it's "basic" to do 100% code coverage in a hardware integration lab, I'm not sure I buy that. That's nowhere near as cheap or easy as people in this thread are making it out to be. You'd be talking millions of test cases at the absolute least.
Condition coverage is not code coverage, nor do I think this has to (all) be in a hardware integration lab.
I would expect there to be tests at some level for all conditions that make up critical functions (e.g. functions that could contribute to loss of mission or loss of crew). As I understand it the OFT MET issue resulted from a whole condition in the requirement being omitted from the code and yet no testing identified that the condition was missing ... ?!
Shouldn’t need any flight hardware to verify the presence of the required condition in the code (unless the interface concerned was too hard to emulate / stub in a harness but I’d be very surprised if that was the case).
It's hard to say what exactly the decision was from that question. I get the sense that that discussion was simplified for the sake of the audience.
Problem is you have four conditions here, three conditions there, another half dozen over here, and they all multiply, and, sure you can run in an emulator that can run faster than real time, but how accurate is your emulator? I don't think that answer gave enough context to be able to make a value judgment about what happened, other than some folks here seem to be under the impression that this kind of testing is free (or even cheap) and takes no time whatsoever. When you get into specialized hardware-software integration, that's not really the case. -
#1337 by SWGlassPit on 06 Mar, 2020 16:43
-
Lovarro: Two specific things: one thing in IRT we had a routine where there might have been four logical conditions, four ways the software could have run. Didn't check all four conditions, was a designer/tester choice. Not unusual you don't test all logical condition, do need to recognize we do need to test all logical conditions now. Process side, delegated too much authority to software board to approve changes and actions as it applies to software, software/hardware integration change should have been brought up all the way to engineering review board.
Wow. These are not subtle mistakes/omissions, but really basic ones. I cannot understand how a mature organisation could make such mistakes. This is what I might expect for an organisation with little experience that doesn’t understand the significance of what it is doing, in terms of criticality, and/or the importance of hardware & software integration.
If you're suggesting it's "basic" to do 100% code coverage in a hardware integration lab, I'm not sure I buy that. That's nowhere near as cheap or easy as people in this thread are making it out to be. You'd be talking millions of test cases at the absolute least.
Yet SpaceX manages to do it. This is human spaceflight. If you can’t commit to 100% code coverage, get out of the business. If you think it’s too expensive or difficult to do so, get out of the business. This isn’t a joke. If they can’t do what needs to be done they have no business attempting to send humans to space.
[citation needed]
I'm not sure even shuttle flight software got 100% code coverage, though they got close, and that was enormously expensive. -
#1338 by Vettedrmr on 06 Mar, 2020 16:46
-
Lovarro: Two specific things: one thing in IRT we had a routine where there might have been four logical conditions, four ways the software could have run. Didn't check all four conditions, was a designer/tester choice. Not unusual you don't test all logical condition, do need to recognize we do need to test all logical conditions now. Process side, delegated too much authority to software board to approve changes and actions as it applies to software, software/hardware integration change should have been brought up all the way to engineering review board.
Wow. These are not subtle mistakes/omissions, but really basic ones. I cannot understand how a mature organisation could make such mistakes. This is what I might expect for an organisation with little experience that doesn’t understand the significance of what it is doing, in terms of criticality, and/or the importance of hardware & software integration.
If you're suggesting it's "basic" to do 100% code coverage in a hardware integration lab, I'm not sure I buy that. That's nowhere near as cheap or easy as people in this thread are making it out to be. You'd be talking millions of test cases at the absolute least.
This is a good summary of what leapt out at me as well. You didn't test software logic branches?? Are you kidding? That was required when I was working on Fortran projects in my freshman programming course.
You absolutely do 100% code coverage for safety-critical software. It doesn't have to be done in a hardware lab, in fact it may be impossible to do it in a hardware lab (some failure conditions that defensive coding protects against don't happen in non-failed hardware, for example), but you DO EXERCISE THEM.
You also don't test combinations of logic; that runs to infinity very quickly. But you check ALL of the code.
That's not bean counters driving the ship, that's just lazy. -
#1339 by SWGlassPit on 06 Mar, 2020 16:48
-
Lovarro: Two specific things: one thing in IRT we had a routine where there might have been four logical conditions, four ways the software could have run. Didn't check all four conditions, was a designer/tester choice. Not unusual you don't test all logical condition, do need to recognize we do need to test all logical conditions now. Process side, delegated too much authority to software board to approve changes and actions as it applies to software, software/hardware integration change should have been brought up all the way to engineering review board.
Wow. These are not subtle mistakes/omissions, but really basic ones. I cannot understand how a mature organisation could make such mistakes. This is what I might expect for an organisation with little experience that doesn’t understand the significance of what it is doing, in terms of criticality, and/or the importance of hardware & software integration.
If you're suggesting it's "basic" to do 100% code coverage in a hardware integration lab, I'm not sure I buy that.
Agreed, 100% code coverage in testing is not basic.That's nowhere near as cheap or easy as people in this thread are making it out to be. You'd be talking millions of test cases at the absolute least.
Instead of O(10^6) test cases, which does not work for several reasons, there are other procedures to guarantee maximum code coverage. In a previous life we used to do something like
(1) make testcases for important "do not fail" situations,
(2) do code fuzzing, i.e., provide random inputs to check code paths that were not exercised by (1),
(3) fix the crashes, identify code paths not checked at all via compiler and profiling tools,
go back to (1) until satisfied with code coverage and robustness.
From the information we get via these press conferences and discussions, it seems that (1) and (3) were done, but not such a closed loop iteration (1,2,3) to maximize the coverage.
Just my 2pc.
I couldn't say what did or did not happen, but yes, this is the general process for software from my experience. Trouble comes when you run into corner cases IRL that your test plan didn't cover, either because (a) you didn't think the triggering conditions were possible, (b) you thought they were so unlikely that with limited time and budget to test, this one didn't get tested, (c) real life hardware, for any number of reasons, didn't respond the way your hardware model did, or (d, et seq) any number of other causes not thought of ahead of time.
