-
#940
by
Lee Jay
on 08 Feb, 2020 00:47
-
I once visited a place that wrote the code that runs inside jet engine controllers for commercial airliners. IIRC, they told me that their staff of 750 could write around 4,000 lines of code a week, including all the necessary verification and testing. Commercial code is written faster than that by a factor of 1,000 or so. Human-safety-critical code requires a heck of a lot of scrutiny and testing.
-
#941
by
Vettedrmr
on 08 Feb, 2020 00:51
-
I once visited a place that wrote the code that runs inside jet engine controllers for commercial airliners. IIRC, they told me that their staff of 750 could write around 4,000 lines of code a week, including all the necessary verification and testing. Commercial code is written faster than that by a factor of 1,000 or so. Human-safety-critical code requires a heck of a lot of scrutiny and testing.
Assuming that same staff could review/re-test 8000 lines a week, that's 125 weeks. I still think about a year, but it's for sure not a 2-3 month task for a team of 10-20.
-
#942
by
Lee Jay
on 08 Feb, 2020 00:58
-
I once visited a place that wrote the code that runs inside jet engine controllers for commercial airliners. IIRC, they told me that their staff of 750 could write around 4,000 lines of code a week, including all the necessary verification and testing. Commercial code is written faster than that by a factor of 1,000 or so. Human-safety-critical code requires a heck of a lot of scrutiny and testing.
Assuming that same staff could review/re-test 8000 lines a week, that's 125 weeks. I still think about a year, but it's for sure not a 2-3 month task for a team of 10-20.
Sure it is. Just just have to do a half-assed job.
-
#943
by
Rocket Science
on 08 Feb, 2020 00:59
-
Whatever it takes to get it right, get it right... "Better to be on the ground wishing you were flying, rather to be flying wishing you were on the ground"...
-
#944
by
Vettedrmr
on 08 Feb, 2020 01:03
-
Sure it is. Just just have to do a half-assed job.
Not now. NASA is going to be involved in the reviews, which will slow things down and make sure the t's are crossed and the i's are dotted.
Have a good one,
Mike
-
#945
by
Stan-1967
on 08 Feb, 2020 01:29
-
Regarding the poll of should NASA require an OFT2? Shouldn't the poll have included a "Wait for the findings of the investigation" as an option?
I'd bet a reasonable % of people would go along with the existing NASA/Boeing path of finishing the investigation, even if they are disgusted with what is known so far.
-
#946
by
Lee Jay
on 08 Feb, 2020 02:02
-
Regarding the poll of should NASA require an OFT2? Shouldn't the poll have included a "Wait for the findings of the investigation" as an option?
I'd bet a reasonable % of people would go along with the existing NASA/Boeing path of finishing the investigation, even if they are disgusted with what is known so far.
I wouldn't. The test flight ended in failure to accomplish the ISS rendezvous, almost resulted in catastrophic failure, and the call said there were multiple process escapes. Fix all that and try again unless the conclusion of the investigation is that none of that ever happened.
-
#947
by
ulm_atms
on 08 Feb, 2020 02:21
-
We are all talking about NASA making Boeing redo the test and whatnot....but with everything that has come to light today....I think the ISS partners may have some say so too at this point.
Don't the ISS partners have to sign off on anything even getting close to the ISS?
-
#948
by
IdeallyNotOnImpact
on 08 Feb, 2020 02:44
-
To my mind, these are the critical paragraphs from today’s article:
“Mr. Loverro continued, saying Boeing and NASA did not have to disclose the second issue to the media or the U.S. taxpayers because “We fixed it. You wouldn’t want us talking about something that didn’t happen.”
The comments from Mr. Loverro came minutes after Jim Chilton of Boeing admitted they would never have found the second software issue had the first issue with the Mission Elapsed Timer not occurred.”
These reek management and culture problems. Firstly, “you don’t need to know” is quite the take for a potential LOM scenario. Secondly, they never would have found it, and only spoke about it because their hands were forced.
This goes directly to the culture issues that many have said plague both Boeing and NASA. They didn’t find it before, and this is a critical process problem. For all their claims to history and institutional knowledge, this is a (another) existence proof that it is not so. As a post mortem, it is insultingly deficient.
Like many here, I work on production software and run a business. If I went to a client with a story like this after two near misses, I’d deserve to be fired.
They didn’t find the defects in time.
They hid the defects.
They don’t admit to serious issues.
So they won’t FIX the serious issues.
And that’s the culture problem. There’s nothing to see here. Trust us.
Yeah, no.
These are the two things we know about. So far. Given the history, this lack of taking responsibility, you think we’re done? That there aren’t more undiscovered issues that we don’t need to know about because THEY don’t?
Me neither.
They don’t need to refly OFT.
They need to stop. Take responsibility. Change practices. Change management. Review everything. Become accountable.
And then - and only then - do it again.
-
#949
by
wolfpack
on 08 Feb, 2020 02:44
-
Don't the ISS partners have to sign off on anything even getting close to the ISS?
Now that’s a good question.
-
#950
by
mgeagon
on 08 Feb, 2020 03:03
-
During the presentation, it was stated that the communication issue was not hardware related, but was caused by too much "noise" on Earth (e.g. cellphone traffic). What wasn't asked by the press is why wasn't this expected and would a higher initial orbit mitigate the interference?
-
#951
by
Steven Pietrobon
on 08 Feb, 2020 03:14
-
I'm feeling very lonely, being only one of the three people who voted No on the reflight poll. I think a reflight is not necessary because the previous flight found the major problems that needed to be fixed, two software and one communications, as well as the process errors that led to those errors slipping through. In the past, NASA also had problems that were discovered with test flights.
For example, Mercury Redstone 2 on 31 January 1961 had a number of problems, although the spacecraft was safely recovered. However, Von Braun insisted on another test flight, which went perfectly with Mercury Redstone BD on 24 March 1961. However, the additional delay resulted in the USSR being the first in space on 12 April 1961.
The second example were the problems experienced from pogo and on the S-II and S-IVB stages of the Saturn V during Apollo 6 on 4 April 1968. In this case, NASA took the gamble and put crew on the next Saturn V flight, Apollo 8 on 21 December 1968. This allowed the US to be first around the Moon, beating the Russians with their Proton-Zond program.
NASA also flew crew after Challenger and Columbia. NASA could have modified the Shuttle to do an uncrewed test flight, but had enough confidence that the corrective measures taken would be successful.
Of course, the US is no longer in a geopolitical battle with Russia for dominance in space, as the US is so far ahead in terms of achievements, technology and money being spent. It can afford to take its time, especially since Dragon 2 is nearly ready to fly. So what would another test flight achieve and what are the risks to the crew? The last test showed that the spacecraft is basically sound. The two software bugs are easily fixed. The process errors that led to those bugs will take much longer to fix, but will result in the software being much safer. I am confident the communications problem will be fixed as well. So, I would personally feel confident in flying on CST-100, except for one thing.
That one thing is an inflight-abort. I have no confidence that will work since it hasn't been tested. I would rather Boeing spend money on that, instead of a repeat of OFT. NASA could provide a Peacekeeper first stage for the test as government supplied equipment to help reduce overall costs.
-
#952
by
wolfpack
on 08 Feb, 2020 03:25
-
The two software bugs are easily fixed.
There are only two?
I think that’s the problem.
-
#953
by
CJ
on 08 Feb, 2020 03:45
-
Of course, the US is no longer in a geopolitical battle with Russia for dominance in space, as the US is so far ahead in terms of achievements, technology and money being spent. It can afford to take its time, especially since Dragon 2 is nearly ready to fly. So what would another test flight achieve and what are the risks to the crew? The last test showed that the spacecraft is basically sound. The two software bugs are easily fixed. The process errors that led to those bugs will take much longer to fix, but will result in the software being much safer. I am confident the communications problem will be fixed as well. So, I would personally feel confident in flying on CST-100, except for one thing.
That one thing is an inflight-abort. I have no confidence that will work since it hasn't been tested. I would rather Boeing spend money on that, instead of a repeat of OFT. NASA could provide a Peacekeeper first stage for the test as government supplied equipment to help reduce overall costs.
I would have agreed with your position (it was the same as mine, until today) except for two things; fixing the bugs will be a long, slow process (a million lines of code to go through and verify), but the bigger issue IMHO is that Boeing cannot be trusted. Why? Because they were quite happy to lie after the landing, including, when asked directly, replying "No" when asked if there were anomalies beside the timer issue. I'm willing to give them a pass on the comms issue because maybe they didn't understand it at the time (and might have been truthful when they said it was due to orientation) but they sure as heck knew they'd had to patch the software after discovering the second bug. So, they lied.
The question I now have is, are they only lying to the public and taxpayers, or are they lying to NASA as well? For example, did they fudge a lot of the cert tests, and not just software? Was the missing parachute pin in the pad abort test a one-off quality control issue, or was it (along with other things, like software testing) indicative of a systemic problem, leaving many things to fix?
I now, reluctantly, support a re-flight (due to the large number of issues so far), but only after a throughout review and investigation. (my opinion may, of course, change once the full investigation has been completed and made public.).
Boeing's hardware workmanship needs a good looking-at too, IMHO, given the issues other Boeing divisions have had (such as the rejected tankers for the air force, 737 MAX, etc). I do not mean just Starliner, but any other Boeing business for NASA as well.
I will be utterly delighted if I am proven wrong, because IMHO any delays to Commercial Crew are bad.
-
#954
by
Torbjorn Larsson, OM
on 08 Feb, 2020 03:47
-
SpaceX: CEO smokes marijuana, company gets Organizational Safety Assessment.
Boeing: Craft gets nearly smoked, company gets Organizational Safety Assessment.
This is the company that did not do the In Flight Abort test since they claim to know software modeling. They botched the Pad Abort Test on a safety and quality issue. And they had two Loss Of Vehicle risks discovered during Orbital Flight Test.
Yes, let's do an assessment (and a prudent OFT, and a possible IFA).
-
#955
by
Lemurion
on 08 Feb, 2020 03:54
-
I'm feeling very lonely, being only one of the three people who voted No on the reflight poll. I think a reflight is not necessary because the previous flight found the major problems that needed to be fixed, two software and one communications, as well as the process errors that led to those errors slipping through. In the past, NASA also had problems that were discovered with test flights.
For example, Mercury Redstone 2 on 31 January 1961 had a number of problems, although the spacecraft was safely recovered. However, Von Braun insisted on another test flight, which went perfectly with Mercury Redstone BD on 24 March 1961. However, the additional delay resulted in the USSR being the first in space on 12 April 1961.
The second example were the problems experienced from pogo and on the S-II and S-IVB stages of the Saturn V during Apollo 6 on 4 April 1968. In this case, NASA took the gamble and put crew on the next Saturn V flight, Apollo 8 on 21 December 1968. This allowed the US to be first around the Moon, beating the Russians with their Proton-Zond program.
NASA also flew crew after Challenger and Columbia. NASA could have modified the Shuttle to do an uncrewed test flight, but had enough confidence that the corrective measures taken would be successful.
Of course, the US is no longer in a geopolitical battle with Russia for dominance in space, as the US is so far ahead in terms of achievements, technology and money being spent. It can afford to take its time, especially since Dragon 2 is nearly ready to fly. So what would another test flight achieve and what are the risks to the crew? The last test showed that the spacecraft is basically sound. The two software bugs are easily fixed. The process errors that led to those bugs will take much longer to fix, but will result in the software being much safer. I am confident the communications problem will be fixed as well. So, I would personally feel confident in flying on CST-100, except for one thing.
That one thing is an inflight-abort. I have no confidence that will work since it hasn't been tested. I would rather Boeing spend money on that, instead of a repeat of OFT. NASA could provide a Peacekeeper first stage for the test as government supplied equipment to help reduce overall costs.
The problem I have is that no matter what they may have fixed, and what process escapes they may have found I have absolutely zero faith that they will catch them all. Given their recent history I think a physical reflight is the only way to prove the testing process.
I don't trust their process enough to believe they will catch all the errors and process escapes. If I trusted their test processes then yes, it would be enough to fix the errors and escapes they have identified. Without that trust, nothing short of a reflight will really satisfy me.
-
#956
by
dlapine
on 08 Feb, 2020 03:57
-
I'm feeling very lonely, being only one of the three people who voted No on the reflight poll. I think a reflight is not necessary because the previous flight found the major problems that needed to be fixed, two software and one communications, as well as the process errors that led to those errors slipping through. In the past, NASA also had problems that were discovered with test flights.
...
Of course, the US is no longer in a geopolitical battle with Russia for dominance in space, as the US is so far ahead in terms of achievements, technology and money being spent. It can afford to take its time, especially since Dragon 2 is nearly ready to fly. So what would another test flight achieve and what are the risks to the crew? The last test showed that the spacecraft is basically sound. The two software bugs are easily fixed. The process errors that led to those bugs will take much longer to fix, but will result in the software being much safer. I am confident the communications problem will be fixed as well. So, I would personally feel confident in flying on CST-100, except for one thing.
That one thing is an inflight-abort. I have no confidence that will work since it hasn't been tested. I would rather Boeing spend money on that, instead of a repeat of OFT. NASA could provide a Peacekeeper first stage for the test as government supplied equipment to help reduce overall costs.
I don't think that vehicle flight software can now be classified as being a verifiable state with simply some small errors that need correction. Their entire verification and testing regime has been called into question.
If the point of the flight was to demonstrate automated, unattended flight of an uncrewed capsule for a docking with the international space station, the last information we were given seems to have proven that it would not have succeeded at that task. It was only the first timer failure that allowed them determine that there was another failure in the coding for service module thruster mapping/control, and without addressing that issue would have resulted in a LOM under standard mission execution. I mean, give them full credit for finding the thruster issue and patching it in realtime, but that's kinda the opposite of an automated mission. The point, they weren't aware of any potential issues in these areas until some minutes into the flight.
Given those previously undiscovered issues, and fully 1/3 of the programmed scripts not exercised for some critical flight areas, why would it be reasonable to make the assumption that having crew onboard would be safe, especially after having the previous communication interruption issue which prevented control or software updates from the ground?
I do agree that an IFA is necessary for them, but I believe that they also need to redo the OFT.
-
#957
by
clongton
on 08 Feb, 2020 03:58
-
I'm feeling very lonely, being only one of the three people who voted No on the reflight poll. I think a reflight is not necessary because the previous flight found the major problems that needed to be fixed, two software and one communications, as well as the process errors that led to those errors slipping through. <snip>
The issue this time Steven is not the problems that need to be fixed. It's Boeing's apparent lack of capability or lack of competence, or both, to bite the bullet and do the testing to the extent deserving of a spacecraft that will carry human souls inside. Maybe all the problems have been identified. Maybe there are others that simply haven't yet seen the test regimes that would flush them out. We just don't know and can no longer trust Boeing to tell the unvarnished truth about this. It appears that even NASA has lost their confidence in Boeing's ability and/or willingness to execute on the extreme vetting of the software that quite literally holds the lives of the crew in it's grip. Boeing needs to go thru its coding with a fine tooth comb and test it every which way from Sunday and show that there are no more bugs. Thru their ineptness, they brought this situation on themselves. At this point the schedule doesn't matter. They need to take whatever time is needed to do this, no matter how long it takes. It needs to be done. THEN, refly OFT to prove that they have truly fixed the problems and the corporate culture that caused this abject failure.
-
#958
by
IdeallyNotOnImpact
on 08 Feb, 2020 04:02
-
The two software bugs are easily fixed.
There are only two?
I think that’s the problem.
Right. If your focus is on the bugs, you're missing the point.
-
#959
by
Comga
on 08 Feb, 2020 05:39
-
“Mr. Loverro continued, saying Boeing and NASA did not have to disclose the second issue to the media or the U.S. taxpayers because “We fixed it. You wouldn’t want us talking about something that didn’t happen.”
Besides the outright absurdity of this, why is the NASA guy saying “we fixed it”?
Who’s “we”?
How is this “Commercial Crew”?
It’s like they lease a ($4.2B) car but have to repair their own brake lines at a highway rest area.
I forget who said it first upthread, but I agree that this was just a CYA sideshow after they got caught with their pants down. And they are still dissembling, if not outright lying.
And then Bridenstine goes and tells Boeing that no matter how much more they mess up or increase their costs NASA won’t terminate them because NASA absolutely needs “dissimilar redundancy”.
(The first concept of a commercial deal is that the buyer could walk away. Otherwise it’s like a unique and patented lifesaving drug, and we’ve seen what happens there.)