-
#300
by
Markstark
on 23 Dec, 2019 13:33
-
I don't know how pertinent this may be, but the Boeing CEO has just resigned:
https://www.cnn.com/2019/12/23/business/boeing-dennis-muilenburg/index.html
Boeing CEO Dennis Muilenburg has stepped down after a tumultuous year, in which the 737 Max airplane and Starliner spacecraft encountered numerous issues and delays.
My guess is that his resignation was already planned around the holiday timeframe due to the Max debacle. I think the cake was baked already and Starliner issues didn’t play much of a role in the decision.
-
#301
by
DaveJ576
on 23 Dec, 2019 13:54
-
Unfortunately software errors like this are not unheard of. The control software developed by Northrup for the guided missile destroyer USS McCain left the ship in situations were the ship was uncontrollable costing 10 lives when it crashed in the Singapore Straits in 2017.
https://features.propublica.org/navy-uss-mccain-crash/navy-installed-touch-screen-steering-ten-sailors-paid-with-their-lives/
From the article:
'The NTSB put it plainly: “The design of the John S McCain’s touch-screen steering and thrust control system,” the board found, “increased the likelihood of the operator errors that led to the collision.”
The Navy investigators, for their part, determined that the system’s “known vulnerabilities” and risks had not been “clearly communicated to the operators on ships with these systems.”'
Fortunately this error with Starliner was found with no one onboard. Hopefully this problem can be fixed quicker and better than the problem on the USS McCain. The real world always exposes in my experience problems that were not apparent in using simulation no matter how much you do. Management doesn't have an easy task in determining if testing software used in critical situations is sufficiently rigorous and complete.
My reply is slightly off topic, but I feel it is pertinent to the discussion as a whole, especially when you are trying to establish a root cause.
I have personally operated the Propulsion and Auxiliaries Control Console on USN Mine Countermeasures vessels and the Electrical Plant Control Console on destroyers. I have also personally witnessed the operation of the helm console on both types of ships. The problem lays not with the software code, but with the INTERFACE. The way the touchscreen is designed and the sensitivity of the controls itself are the real issues. Large guys with big fingers sometimes have trouble with the touch buttons, and the ergonomics of the screens (which are not intuitive) are an issue. It is overly complicated and easy to misinterpret what mode you are operating in. These were some of the contributing factors in the McCain collision, and having spent a lot of time at sea on USN ships I can tell you it is easy to screw it up. Only ruthless procedural compliance, minute detail attention, and constant training on behalf of the crew can mitigate it, and that apparently was a problem on the McCain.
Wayne Hale said it best when referring to accident investigations: "You must ask "Why" at least ten times before you arrive at the root cause." It will be interesting to see what the 10th Why will uncover in this case, if we even get to that point.
-
#302
by
marsbase
on 23 Dec, 2019 14:08
-
The way the touchscreen is designed and the sensitivity of the controls itself are the real issues. Large guys with big fingers sometimes have trouble with the touch buttons, and the ergonomics of the screens (which are not intuitive) are an issue. It is overly complicated and easy to misinterpret what mode you are operating in.
A friend of mine who worked for General Dynamics did HFE(Human Factors Engineering) for an aircraft carrier control screen. He was good but the complexity was almost overwhelming, as you state. Have you compared the control panels for Dragon and Starliner? I have seen photos of the Dragon touchscreen but it's hard to tell much. My understanding is that the Starliner control panel is a modified panel from the cockpit of a Boeing 787, with LOTS of buttons and switches which are equally confusing. Any thoughts on this?
-
#303
by
abaddon
on 23 Dec, 2019 14:15
-
My guess is that his resignation was already planned around the holiday timeframe due to the Max debacle. I think the cake was baked already and Starliner issues didn’t play much of a role in the decision.
Or he was already on thin ice and the Starliner issue was the last straw.
I don't have any information to support
my guess, do you?
-
#304
by
OM72
on 23 Dec, 2019 14:16
-
The way the touchscreen is designed and the sensitivity of the controls itself are the real issues. Large guys with big fingers sometimes have trouble with the touch buttons, and the ergonomics of the screens (which are not intuitive) are an issue. It is overly complicated and easy to misinterpret what mode you are operating in.
A friend of mine who worked for General Dynamics did HFE(Human Factors Engineering) for an aircraft carrier control screen. He was good but the complexity was almost overwhelming, as you state. Have you compared the control panels for Dragon and Starliner? I have seen photos of the Dragon touchscreen but it's hard to tell much. My understanding is that the Starliner control panel is a modified panel from the cockpit of a Boeing 787, with LOTS of buttons and switches which are equally confusing. Any thoughts on this?
Starliner does not have any touch screen interface for the control of the vehicle.
-
#305
by
OM72
on 23 Dec, 2019 14:22
-
My guess is that his resignation was already planned around the holiday timeframe due to the Max debacle. I think the cake was baked already and Starliner issues didn’t play much of a role in the decision.
Or he was already on thin ice and the Starliner issue was the last straw.
I don't have any information to support my guess, do you?
I don't think the Starliner played a part. The flight was, in large part, a success. It was a test flight after all and a lot was learned. About 90% of flight objectives will be achieved. The vehicle executed a very successful EDL.
A CEO does not just "get fired". It's a bit of a process. Today also starts the traditional Boeing two-week shut down. This has the air of being in the cards for some time
-
#306
by
abaddon
on 23 Dec, 2019 14:26
-
A CEO does not just "get fired".
No, they "resign" with a golden parachute. But that's a different complaint for a different forum.
It's a bit of a process. Today also starts the traditional Boeing two-week shut down. This has the air of being in the cards for some time
Fair enough, that seems reasonable.
-
#307
by
dondar
on 23 Dec, 2019 14:29
-
Very relevant quote to the whole discussion from the update audio conference during this launch.
https://www.nasa.gov/sites/default/files/atoms/audio/oft_20191221.mp3time mark 17.58: "To the operator who assembled this thing it appeared that pin was in but our design allowed that to be misread".
.......
In these few words I believe is the story and the fate of this program.
-
#308
by
FutureSpaceTourist
on 23 Dec, 2019 14:46
-
My understanding is that the Starliner control panel is a modified panel from the cockpit of a Boeing 787, with LOTS of buttons and switches which are equally confusing.
See below
https://twitter.com/trevormahlmann/status/1207409799706664961
Lots o’ buttons on the #Starliner control panel. Interesting to see the different design choices with each @Commercial_Crew providers.
-
#309
by
Markstark
on 23 Dec, 2019 15:02
-
My guess is that his resignation was already planned around the holiday timeframe due to the Max debacle. I think the cake was baked already and Starliner issues didn’t play much of a role in the decision.
Or he was already on thin ice and the Starliner issue was the last straw.
I don't have any information to support my guess, do you?
No. Totally a guess
Sent from my iPhone using Tapatalk Pro
-
#310
by
launchwatcher
on 23 Dec, 2019 16:25
-
I'm wondering if a decent methodology would be to develop a simulation of each hardware component right along with the software designed to control it. So you'd be unit testing and integrating your sim in parallel with unit testing and integrating the actual iron.
The buzzword du jour for what you are referencing is "digital twin." However, the emulators that I've dealt with don't allow mocks of direct address reading, only bus and protocol mocks.
I have worked on ASIL D/ISO 26262 software in automotive applications. The main testing environment consisted of software-in-the-loop (SIL) and hardware-in-the-loop (HIL) tools. SIL runs the program against a software simulation either from a quick build on your own computer or on a real controller on your desk. HIL offers much more fidelity and essentially consists of all the hardware in car laid out across a room with all the real controllers running real software. You'd plug the controller you were testing in at the appropriate place and could then see the real hardware as the program runs, like watching the throttle move when running a WLTP cycle. In terms of simulation quality, you get real physical simulation of everything down to stuff like valve open/close latency. In terms of test frequency, I would sometimes spend multiple hours per day in a HIL and test small changes. It's a perfectly normal debugging tool that you are expected to use liberally. Ironically enough, one of my first tasks in the simulator was to track down a timing issue that could prevent the controller from taking the proper actions.
The teams also work slightly differently form other software projects. There is a single, tight group of about six people responsible for the whole software. They know everything that's going on and define the architecture in terms of signals (pretty much global variables) and modules (compact pieces of code that act upon and modify those variables). Although it all compiles down to C at some point, most of the development work is actually done in a graphical environment that handles metadata like units on the signals. I also wrote static analysis tools that could visualize the data flow and autonomously reposition modules while keeping all constraints in terms of read/write sequencing and access to operating system function provably equal. Every change is requested and orchestrated by the central team. In the case of small changes, they might change the code themselves, anything past a few lines is specified and given to a programmer. A copy of the high level specifications is given to a second team in a separate building that develops a corresponding "level 2" component. This component is run after all the modules, but before any actions are committed, and performs all sorts of bounds and sanity checking. In the case of automotive controllers, this is the only code required to run on a replicated processor core as that should be enough to cover most hardware failures.
While this methodology is somewhat antiquated by todays standards, it results in plenty safe software. I can't imagine how the MET error could have gotten past a simple test even with a simulated rocket that just replays a launch recording. All I can come up with is that they tried to emulate the rocket somehow which lead to them testing against their own, wrong, preconception about how the signaling should function.
Emphasis mine.
In other words: under your scenario Boeing tested not against the real-deal Atlas V EDS but against a stub.
Let's just hope that is NOT what actually went wrong because that would make the MET issue even more ridiculous than it already is.
I suspect that they did tests with a freshly-booted real-deal Atlas V EDS -- where the difference between boot time and launch time was small enough that the MET initialization error didn't cause any trouble for the simulation and so wasn't noticed..
There's a tension between getting all your test objectives coverered and making the tests highly realistic; it takes a lot longer to execute your test plan if every run has to start with "start Atlas, wait 11 hours".
-
#311
by
envy887
on 23 Dec, 2019 17:02
-
There's a tension between getting all your test objectives coverered and making the tests highly realistic; it takes a lot longer to execute your test plan if every run has to start with "start Atlas, wait 11 hours".
Not every test, for sure, but they should have done at least one full mission duration rehearsal, no?
-
#312
by
ThomasGadd
on 23 Dec, 2019 17:06
-
Come on folks enough 737MAX and Muilenburg leaving put it in a different thread.
-
#313
by
JEF_300
on 23 Dec, 2019 18:24
-
Again
The most important goal of launching is to get to orbit.
DUH!
The most important goal is to not endanger your crew.
Boeing decided that not stranding the crew in orbit was more important to them than reaching orbit. We can sit here back seat engineering all we want, but let's not act like the decision they made was totally nonsensical.
-
#314
by
happyflower
on 23 Dec, 2019 19:29
-
I am wondering if anybody here can give me a simple explanation of how difficult it would have been in this particular case for the human crew to have taken over and actually do the mission (if the human crew were on-board)?
When NASA says that if the astronauts where in that capsule that one they would have been OK, and two would have taken over and completed the mission do they have actually test case of a human bringing a capsule under control under these circumstances, or they are just confident of their astronauts being able to handle "this sort" of issue?
My civilian feeling about this, is that humans would have a hard time doing something a computer is doing. Can they do it? Maybe..., but NASA seemed awfully confident that it would have been OK.
-
#315
by
DigitalMan
on 23 Dec, 2019 20:05
-
I am wondering if anybody here can give me a simple explanation of how difficult it would have been in this particular case for the human crew to have taken over and actually do the mission (if the human crew were on-board)?
When NASA says that if the astronauts where in that capsule that one they would have been OK, and two would have taken over and completed the mission do they have actually test case of a human bringing a capsule under control under these circumstances, or they are just confident of their astronauts being able to handle "this sort" of issue?
My civilian feeling about this, is that humans would have a hard time doing something a computer is doing. Can they do it? Maybe..., but NASA seemed awfully confident that it would have been OK.
No need to ask here, you can listen to the post-launch press conference and listen to the astronauts tell you. Summary: not a problem.
-
#316
by
AnnK
on 23 Dec, 2019 20:28
-
I am wondering if anybody here can give me a simple explanation of how difficult it would have been in this particular case for the human crew to have taken over and actually do the mission (if the human crew were on-board)?
When NASA says that if the astronauts where in that capsule that one they would have been OK, and two would have taken over and completed the mission do they have actually test case of a human bringing a capsule under control under these circumstances, or they are just confident of their astronauts being able to handle "this sort" of issue?
My civilian feeling about this, is that humans would have a hard time doing something a computer is doing. Can they do it? Maybe..., but NASA seemed awfully confident that it would have been OK.
Computers are only as good as their programming. Humans on the other hand are able to come up with solutions that defy the book. There are many examples of human aircrew that have saved their aircraft and have flown them to safety. While a computer would fly it into the ground. This is the mistake of current aeronautical and space engineers. Computers are not able to do rational thinking nor come to unique solutions to problems. It is the true reason for the 737 Max crashes, allowing a flight computer to override the commands of the pilots.
Oh there were no crew on board, so what does the panel design have to do with anything?
-
#317
by
Rocket Science
on 23 Dec, 2019 20:35
-
The way the touchscreen is designed and the sensitivity of the controls itself are the real issues. Large guys with big fingers sometimes have trouble with the touch buttons, and the ergonomics of the screens (which are not intuitive) are an issue. It is overly complicated and easy to misinterpret what mode you are operating in.
A friend of mine who worked for General Dynamics did HFE(Human Factors Engineering) for an aircraft carrier control screen. He was good but the complexity was almost overwhelming, as you state. Have you compared the control panels for Dragon and Starliner? I have seen photos of the Dragon touchscreen but it's hard to tell much. My understanding is that the Starliner control panel is a modified panel from the cockpit of a Boeing 787, with LOTS of buttons and switches which are equally confusing. Any thoughts on this?
Interesting article about the USS McCain crash touchscreen might lend some insight...
https://features.propublica.org/navy-uss-mccain-crash/navy-installed-touch-screen-steering-ten-sailors-paid-with-their-lives/?utm_source=pocket-newtab
-
#318
by
meekGee
on 23 Dec, 2019 20:44
-
We'll see if NASA has any gonads or caves to Boeing pressure to allow this contractual requirement to be ignored.
Anything less than an OFT-2 would be scandalous in my opinion. Boeing should pay for the capsule and the expedited Atlas without charging the taxpayer, too. They got almost twice the money and achieved half the result. Enough.
Preferably, Boeing would actually own up and lead this move without being forced to.
Odds of that happening are.... low. I'd love to be pleasantly surprised.
-
#319
by
Steve G
on 23 Dec, 2019 20:59
-
The schedule is too tight to fly an OFT-2. It proved itself a safe vehicle, there still is a manned test flight before it goes operational, but nothing occurred that (we know about) would have endangered the crew. They flew Apollo 8 after a nearly disastrous Apollo 6 because they were confident they identified the issues that plagued the 6 launch. (Also, a tight timetable . . .) There will be a thorough review, of course, and the right people will make the decision.