-
#920
by
Lemurion
on 07 Feb, 2020 21:18
-
Doug's position that another OFT may not be needed because we don't fly to test these types of issues, and that they should be found during testing is insanely scary given that the issues demonstrated during the OFT were not detected during testing and could have resulted in a loss to the vehicle.
Yep, to me the whole takeaway here is that Boeing's testing is suspect, more than the vehicle itself. I expect problems to crop up in projects of this magnitude--I also expect testing to catch them before it hits the pad.
How do you have confidence in "the vehicle itself" if the testing is suspect?
I think it is valid and necessary to think of software as an integral part of "the vehicle itself" as we discuss and analyze the functionality and safety of the vehicle. With that in mind, NO the takeaway is that the vehicle has failed several key areas, and is suspect in others.
While we did not discuss this in the presser, there may be problems with the hardware and/or the design that have not been disclosed yet. Comms problem root cause? Thrusters failing after minimal usage? Lack of enough reserve propellent?
I didn't say that I had confidence in the vehicle--I said that the testing is more suspect than the vehicle--there is a difference.
In other words I believe that the fundamental design of CST-100 is sound and that Boeing engineers are capable of producing a flightworthy craft. I don't believe that the current testing regime is capable of enabling that. With a functional testing regime, Boeing would make mistakes and catch them at the appropriate time so that they could be remedied before flight.
-
#921
by
groknull
on 07 Feb, 2020 21:21
-
One of my "favorite" lines during the call (forgive me I do not remember who said it, in reference to the separation "bug") was:
"We know the software patch we uploaded worked because the craft landed safely"
You uploaded a software patch that without it would have potentially caused catastrophic LOV and your primary indicator of whether it worked or not was to fly the re-entry sequence and cross your fingers?
That's a whole lot of YIKES.
Agreed. Causality and statistics do not work that way.
If you have a problem that results in a 10% success rate, and a fix that brings it up to 100% success rate, you get your spacecraft back.
If you have a problem that results in a 10% success rate, a fix that brings it up to 50% success rate, and you get lucky, you get your spacecraft back.
If you have a problem that results in a 10% success rate, a fix that doesn't improve anything, and you get really lucky, you get your spacecraft back.
The trick is how to tell the difference.
-
#922
by
freddo411
on 07 Feb, 2020 21:21
-
Earlier NASA said that clearly their oversight of Boeing had been insufficient and that they intended to address that going forward.
Ouch. IMO Kathy Lueders will have some explaining to do. Not good for her and the CCP office.
Possibly but it may be wider than that - is the issue that NASA (wrongly) assumed that didn’t need that level of oversight and so didn’t provide the resources needed? Kathy did say that the things they did look at were fine, while acknowledging that they could only look at so much.
It shouldn't have taken much in the way of NASA resources to catch the fact that Boeing's test plan would miss such blatant and easily-found bugs. A single days of meetings by one good software person should have flagged the problem.
Also, I'd fault every level of Boeing's management for this.
I liked this post because it's a good take.
But let me also put in a counter point:
NASA has hired Boeing as a commercial provider. NASA should expect Boeing to do the work, and get all the details correct without oversight at the level of "did the code pull the right timer value". If NASA is operating at that level, then it's not well functioning commercial contract ... it's a badly run government program.
I believe that NASA should require and review *successful* ground and flight tests sufficient to demonstrate the system works. Similar to what has happened so far. However, Boeing has failed the tests so far ... it's up to NASA to hold them accountable and have them run them until they get them right.
-
#923
by
SoftwareDude
on 07 Feb, 2020 21:22
-
A $4.2 billion contract to build and verify Starliner got them what?
I don't exactly know how SpaceX develops its software, but it appears they use iterative development at all levels. Boeing and NASA do not understand this. Here, is what iterative development should look like:
Engineers must check-in software at least once a week.
Automated software unit tests at check-in by an automated build.
Every night, deploy all software to a simulated test-bed and undergo automated full mission tests.
Regularly, maybe once a week, deploy it to a hardware test-bed and manually test it.
Rinse and repeat until all mission objectives are met.
-
#924
by
freddo411
on 07 Feb, 2020 21:25
-
Doug's position that another OFT may not be needed because we don't fly to test these types of issues, and that they should be found during testing is insanely scary given that the issues demonstrated during the OFT were not detected during testing and could have resulted in a loss to the vehicle.
Yep, to me the whole takeaway here is that Boeing's testing is suspect, more than the vehicle itself. I expect problems to crop up in projects of this magnitude--I also expect testing to catch them before it hits the pad.
How do you have confidence in "the vehicle itself" if the testing is suspect?
I think it is valid and necessary to think of software as an integral part of "the vehicle itself" as we discuss and analyze the functionality and safety of the vehicle. With that in mind, NO the takeaway is that the vehicle has failed several key areas, and is suspect in others.
While we did not discuss this in the presser, there may be problems with the hardware and/or the design that have not been disclosed yet. Comms problem root cause? Thrusters failing after minimal usage? Lack of enough reserve propellent?
I didn't say that I had confidence in the vehicle--I said that the testing is more suspect than the vehicle--there is a difference.
In other words I believe that the fundamental design of CST-100 is sound and that Boeing engineers are capable of producing a flightworthy craft. I don't believe that the current testing regime is capable of enabling that. With a functional testing regime, Boeing would make mistakes and catch them at the appropriate time so that they could be remedied before flight.
Apologies, I may have misunderstood your position.
Agree that testing is the first, most obvious problem here.
-
#925
by
saliva_sweet
on 07 Feb, 2020 21:40
-
NASA has hired Boeing as a commercial provider. NASA should expect Boeing to do the work, and get all the details correct without oversight at the level of "did the code pull the right timer value". If NASA is operating at that level, then it's not well functioning commercial contract ... it's a badly run government program.
I very much agree. And fear that NASA has learned the wrong lesson.
Earlier NASA said that clearly their oversight of Boeing had been insufficient and that they intended to address that going forward.
Boeing loves cost plus, and NASA insight. Because NASA insight in other words is... free testing. They are not used to, and have lost the capability for, doing their own testing.
-
#926
by
dsmillman
on 07 Feb, 2020 21:42
-
1. I noticed that Bill Harwood was the only reporter to get to the heart of the software problems and ask why the program executing the flight plan did not detect the MET timer mistake.
2. This issue is more than a coding issue or a QA issue. It suggests that the people designing this software did not understand that when you are controlling a system in real time(a car, a machine tool, an elevator) anything can go wrong. And you had better write you programs to handle ALL of those unexpected inputs and events.
-
#927
by
kendalla59
on 07 Feb, 2020 21:46
-
Is there a recording of the call somewhere? I managed to get my time zone conversion mixed up and missed it
The irony here is not lost on me.
-
#928
by
Rocket Science
on 07 Feb, 2020 21:48
-
Boeing has the depth of talent to produce and fly a spacecraft. Look at the amazing work on the X-37B. They could internally have the teams that worked that program overview the CST-100 procedures and process. They are just making bad decisions with respect to resources...
-
#929
by
rcoppola
on 07 Feb, 2020 21:49
-
Where was Chris Ferguson in this? Wouldn't he have been in the simulators going through each mission procedure, event & working through potential anomaly recovery checklists? I mean orbital insertion, coms & SM jettison seem like fairly important mission milestones.
-
#930
by
Rocket Science
on 07 Feb, 2020 21:50
-
Earlier NASA said that clearly their oversight of Boeing had been insufficient and that they intended to address that going forward.
Ouch. IMO Kathy Lueders will have some explaining to do. Not good for her and the CCP office.
Possibly but it may be wider than that - is the issue that NASA (wrongly) assumed that didn’t need that level of oversight and so didn’t provide the resources needed? Kathy did say that the things they did look at were fine, while acknowledging that they could only look at so much.
It shouldn't have taken much in the way of NASA resources to catch the fact that Boeing's test plan would miss such blatant and easily-found bugs. A single days of meetings by one good software person should have flagged the problem.
Also, I'd fault every level of Boeing's management for this.
I liked this post because it's a good take.
But let me also put in a counter point:
NASA has hired Boeing as a commercial provider. NASA should expect Boeing to do the work, and get all the details correct without oversight at the level of "did the code pull the right timer value". If NASA is operating at that level, then it's not well functioning commercial contract ... it's a badly run government program.
I believe that NASA should require and review *successful* ground and flight tests sufficient to demonstrate the system works. Similar to what has happened so far. However, Boeing has failed the tests so far ... it's up to NASA to hold them accountable and have them run them until they get them right.
And a correctly functioning system should hold NASA accountable...
-
#931
by
Lee Jay
on 07 Feb, 2020 22:30
-
When does the poll thread for if they have to repeat the test start?
-
#932
by
robertross
on 07 Feb, 2020 22:35
-
When does the poll thread for if they have to repeat the test start?
I added one.
Having just read the excellent article by our NSF team, I am just flabbergasted at all this.
-
#933
by
jamesh9000
on 07 Feb, 2020 22:35
-
What I don't understand about all of this (and this goes for Boeing and SpaceX), is the ongoing narrative of both companies being eager to fly crew, with NASA acting like "dad" who won't let the kids take the Corvette out without appropriate oversight.
Wouldn't it be far, far more in Boeing and SpaceX's interest to be the ones doing the emphasis on safety? Once they start flying crew, if they have an LOV people die. And there's no coming back from that, they're always dead. They get their name on that wall and every year its brought back up again and there's video of their grieving children and it never, ever, ever goes away. When the DM-1 capsule exploded all I could think is "imagine if that happened with people onboard".
I think the real prize here is flying people to space and back safely right through the contract. If they lose this 'race' to recapture the flag its a week or two of feeling disappointed, a conciliatory tweet through gritted teeth and then everyone gets on with it. If they lose a crew its infinite regret forever.
Why would Boeing want to press on with CFT if its just for bragging rights? Why would they want to take the risk? (SpaceX too for their issues) I don't get it.
EDIT: Arrgh, I thought this was in the discussion thread, can a mod please move it for me? Sorry.
Edit zubenelgenubi: Done!
-
#934
by
Lee Jay
on 07 Feb, 2020 22:43
-
When does the poll thread for if they have to repeat the test start?
I added one.
Having just read the excellent article by our NSF team, I am just flabbergasted at all this.
Thank you and, likewise.
-
#935
by
WannaWalnetto
on 07 Feb, 2020 22:50
-
-
#936
by
SoftwareDude
on 07 Feb, 2020 23:58
-
What I don't understand about all of this (and this goes for Boeing and SpaceX), is the ongoing narrative of both companies being eager to fly crew, with NASA acting like "dad" who won't let the kids take the Corvette out without appropriate oversight.
Wouldn't it be far, far more in Boeing and SpaceX's interest to be the ones doing the emphasis on safety? Once they start flying crew, if they have an LOV people die. And there's no coming back from that, they're always dead. They get their name on that wall and every year its brought back up again and there's video of their grieving children and it never, ever, ever goes away. When the DM-1 capsule exploded all I could think is "imagine if that happened with people onboard".
I think the real prize here is flying people to space and back safely right through the contract. If they lose this 'race' to recapture the flag its a week or two of feeling disappointed, a conciliatory tweet through gritted teeth and then everyone gets on with it. If they lose a crew its infinite regret forever.
Why would Boeing want to press on with CFT if its just for bragging rights? Why would they want to take the risk? (SpaceX too for their issues) I don't get it.
EDIT: Arrgh, I thought this was in the discussion thread, can a mod please move it for me? Sorry.
Edit zubenelgenubi: Done!
Isn't letting Boeing take care of safety how we got here? Isn't that the problem with 737 Max? Corporation are about profits, not safety. Sure, you say, in the long run safety is important to profits. Corporations only react to short term ideas.
-
#937
by
PreferToLurk
on 08 Feb, 2020 00:11
-
Corporations only react to short term ideas.
I mean that's just obviously not true, right? There are plenty of corporations, public and private, that have very long term strategies and don't give much hoot to quarterly results.
If you want to accuse Boeing of only caring about short term ideas, then fine, that is probably a valid criticism of that corporation right now. But I just really detest this meme that all corporations are heartless, soulless, profit machines that cant see beyond the next fiscal report.
-
#938
by
SoftwareDude
on 08 Feb, 2020 00:24
-
Corporations only react to short term ideas.
I mean that's just obviously not true, right? There are plenty of corporations, public and private, that have very long term strategies and don't give much hoot to quarterly results.
If you want to accuse Boeing of only caring about short term ideas, then fine, that is probably a valid criticism of that corporation right now. But I just really detest this meme that all corporations are heartless, soulless, profit machines that cant see beyond the next fiscal report.
People invest money in corporations for a return on investment, ROI. They are not investing for their health. Venture Capitalists, all of them, require a cash event within 5 years. However, publicly traded companies require quarterly results. The reasons are several but the main one is the magnitude of the capitalization of the corporation depends to a great degree on their quarterly results. If the capitalization falls, then the cost of their debt, ie corporate bonds rating will go down and force servicing of the debt to go up. This affects their cash flow.
Elon Musk and certain other entrepreneurs are an exception. I trust them for a future vision. In addition, some companies have a very patient Angel Investor, but these are exceptions.
Boeing borrows a lot of money because they have terms on aircraft. They borrow money to build airplanes until customers pay them. The profits of selling airplanes depend a lot on the company's capitalization. Therefore, Boeing is a very short term view of the world.
-
#939
by
Vettedrmr
on 08 Feb, 2020 00:32
-
Mulholland: All the software will be verified to check whether the requirements are being met. About a million lines of code.
A million lines is not that huge considering the project size. Your video player has about a million lines of code. The Linux bare kernel has 27.8 million lines of code, including nothing else in the OS.
A million lines of safety-critical control system software is quite a bit. To perform peer reviews of that many lines of code will take, based on how many qualified resources Boeing and NASA can put on the task, about a year or so. That includes the low level testing that will have to be run.
