Author Topic: SLS General Discussion Thread 2  (Read 601526 times)

Online jabe

  • Regular
  • Full Member
  • ****
  • Posts: 1223
  • Liked: 179
  • Likes Given: 12
Re: SLS General Discussion Thread 2
« Reply #1300 on: 02/24/2018 12:31 am »
got into a side discussion about this in another thread (my bad)
sort of relevant...yikes if true..
https://twitter.com/NASAWatch/status/967175843356270592
Quote
SLS Software Problems Continue
"....The SLS software team at MSFC is having great difficulty in hiring people to replace those who have quit. There is a lot of internal concern as a result of issues already raised with regard to SLS software safety to date that MSFC will literally have to go back to square one on software so as to verify it for use on human missions."
jb
not a good thing...
jb

Offline Eric Hedman

  • Senior Member
  • *****
  • Posts: 2313
  • The birthplace of the solid body electric guitar
  • Liked: 1953
  • Likes Given: 1142
Re: SLS General Discussion Thread 2
« Reply #1301 on: 02/24/2018 03:41 am »
got into a side discussion about this in another thread (my bad)
sort of relevant...yikes if true..
https://twitter.com/NASAWatch/status/967175843356270592
Quote
SLS Software Problems Continue
"....The SLS software team at MSFC is having great difficulty in hiring people to replace those who have quit. There is a lot of internal concern as a result of issues already raised with regard to SLS software safety to date that MSFC will literally have to go back to square one on software so as to verify it for use on human missions."
jb
not a good thing...
jb
I'm thinking there is a good chance that this is now going to be what pushes EM-1 into 2021 or later.  Losing core people in software in my experience is devastating.  On large complex projects like this it takes a while to get new people up to speed.  Just wait for the next schedule delays to be announced.

Offline cebri

  • Full Member
  • **
  • Posts: 245
  • Spain
  • Liked: 291
  • Likes Given: 181
Re: SLS General Discussion Thread 2
« Reply #1302 on: 02/24/2018 08:54 am »
And let's not forget this http://nasawatch.com/archives/2017/12/this-is-how-nas-1.html

Quote
A letter was sent to NASA MSFC management last week by Ben Samouha, a 30+ year veteran in software safety whose career reaches back to the Challenger era. As has been noted previously on NASAWatch there has been a significant amount of internal controversy over safety and software being developed for SLS. Clearly these safety issues remain. People are quitting instead of trying to fight the system, or in some cases, they leave after having been forced out for speaking up about their concerns. As Samouha notes:

"These people have been for a long time (and still are) continuously ignoring or not properly addressing FSW Safety related observations and findings and unethically do not disclose issues to the upper management in order to show a virtual progress in order to keep their jobs. Anyone with years of experience and integrity to Safety can see through these imposters just like I did."
"It's kind of amazing that a window of opportunity is open for life to beyond Earth, and we don't know how long this window is gonna be open" Elon Musk
"If you want to see an endangered species, get up and look in the mirror." John Young

Offline ncb1397

  • Senior Member
  • *****
  • Posts: 3497
  • Liked: 2310
  • Likes Given: 29
Re: SLS General Discussion Thread 2
« Reply #1303 on: 02/24/2018 03:22 pm »
Other than the Europa mission, which could fly on an existing launcher....

By the way, is everyone here aware that the administration's proposed FY 2019 NASA budget recommends moving Europa Clipper from SLS to a commercial launch vehicle?

The president's budget proposal is DOA. When US Congress is finished with it ISS will stay in orbit until (at least) 2028 and Europa Clipper will still be assigned to launch on SLS.

This budget won't determine how long ISS stays. And a launch for Europa Clipper by 2025 with a 7 year cruise puts reaching Europa at 2032. John Culberson would be 76 or dead. I don't think he is going to like this.

Offline AncientU

  • Senior Member
  • *****
  • Posts: 6257
  • Liked: 4164
  • Likes Given: 6078
Re: SLS General Discussion Thread 2
« Reply #1304 on: 02/24/2018 05:28 pm »
got into a side discussion about this in another thread (my bad)
sort of relevant...yikes if true..
https://twitter.com/NASAWatch/status/967175843356270592
Quote
SLS Software Problems Continue
"....The SLS software team at MSFC is having great difficulty in hiring people to replace those who have quit. There is a lot of internal concern as a result of issues already raised with regard to SLS software safety to date that MSFC will literally have to go back to square one on software so as to verify it for use on human missions."
jb
not a good thing...
jb
I'm thinking there is a good chance that this is now going to be what pushes EM-1 into 2021 or later.  Losing core people in software in my experience is devastating.  On large complex projects like this it takes a while to get new people up to speed.  Just wait for the next schedule delays to be announced.

What vintage of hardware/processors are they using on SLS (and Orion)?  Is the software state-of-the-art or generations old?
This will make a huge difference in how readily available are qualified programmers and how professionally rewarding/attractive are these positions.  Another recruitment factor is where these positions are located?  Alabama?
« Last Edit: 02/24/2018 05:31 pm by AncientU »
"If we shared everything [we are working on] people would think we are insane!"
-- SpaceX friend of mlindner

Offline Jim

  • Night Gator
  • Senior Member
  • *****
  • Posts: 37440
  • Cape Canaveral Spaceport
  • Liked: 21450
  • Likes Given: 428
Re: SLS General Discussion Thread 2
« Reply #1305 on: 02/26/2018 12:51 pm »

What vintage of hardware/processors are they using on SLS (and Orion)?  Is the software state-of-the-art or generations old?


Orion uses 787 type avionics.

Offline AncientU

  • Senior Member
  • *****
  • Posts: 6257
  • Liked: 4164
  • Likes Given: 6078
Re: SLS General Discussion Thread 2
« Reply #1306 on: 02/26/2018 04:29 pm »

What vintage of hardware/processors are they using on SLS (and Orion)?  Is the software state-of-the-art or generations old?


Orion uses 787 type avionics.

Quote
The Orion spacecraft is no smarter than your phone
Running 12-year-old processors*, next-gen spaceship’s tech is built for reliability, not to be state-of-the-art
* 15-year-old now

Quote
The computers are running IBM's PowerPC 750FX single-core processors, which were first launched in 2002.

NASA fit two of the processors into each flight computer, setting them up to run identical software and monitor each other. If the processors don't do the exact same thing, the system will stop giving commands and reset itself.

"The processors are obsolete already but they have the property of just getting upset by radiation, instead of being permanently damaged," said Lemke, noting that NASA has been using the processors for more than 10 years. "You could do it with something newer, but all the engineering that would go into making it work right would make it a lot more expensive for us to build it."
https://www.computerworld.com/article/2855604/the-orion-spacecraft-is-no-smarter-than-your-phone.html


Still cannot find anything about SLS computers, but software articles are available:

Article on SLS software/system from two years ago:
Quote
NASA launch system software upgrade now 77% over budget
Quote
As it builds the Space Launch System rocket, NASA is updating this Spaceport Command and Control System software for the Kennedy Space Center. However, a new report by the space agency's inspector general, Paul Martin, finds this decade-long software development effort has fallen behind schedule and is on track to exceed its initial budget of $117.3 million by 77 percent, with cost estimates now increased to $207.4 million. Moreover, the inspector general criticized NASA for not adopting cheaper, commercially available launch software already used by Orbital ATK and SpaceX to launch their rockets.

To develop its new launch software, NASA has essentially kluged together a bunch of different software packages, Martin noted in his report. "The root of these issues largely results from NASA’s implementation of its June 2006 decision to integrate multiple products or, in some cases, parts of products rather than developing software in-house or buying an off-the-shelf product," the report states. "Writing computer code to 'glue' together disparate products has turned out to be more complex and expensive than anticipated. As of January 2016, Agency personnel had developed 2.5 million lines of 'glue-ware,' with almost two more years of development activity planned."
https://arstechnica.com/science/2016/03/nasas-glue-ware-for-computer-launch-systems-over-budget-behind-schedule/

Note: It's been 'two more years' and they are nowhere near complete.

And this nicely sums up why they are having trouble finding software developers.  Also makes one wonder how they can rationalize that a NASA development project is 'safer' and not needing 'certification'... seem like they are taking the most hazardous and expensive ($300M for a rocket's software???????) route on the software/computing side.  Having all of this bureaucracy and 'oversight' appears to have created a monster.
« Last Edit: 02/26/2018 04:36 pm by AncientU »
"If we shared everything [we are working on] people would think we are insane!"
-- SpaceX friend of mlindner

Offline AncientU

  • Senior Member
  • *****
  • Posts: 6257
  • Liked: 4164
  • Likes Given: 6078
Re: SLS General Discussion Thread 2
« Reply #1307 on: 02/26/2018 05:31 pm »
Some pics of the hardware and managers in charge:
Quote
Oh, the places they'll go with the Space Launch System computers NASA just displayed in Huntsville (photos) (video)
Quote
You've heard that there's more computing power in your smartphone than what NASA used in sending Apollo to the moon? Take a look at the computing power displayed at Marshall Space Flight Center Thursday and imagine where American astronauts might go with it on board.
http://blog.al.com/breaking/2014/01/nasa_shows_off_the_brains_of_i.html

Echo chamber.
"If we shared everything [we are working on] people would think we are insane!"
-- SpaceX friend of mlindner

Offline Khadgars

  • Full Member
  • ****
  • Posts: 1750
  • Orange County, California
  • Liked: 1132
  • Likes Given: 3156
Re: SLS General Discussion Thread 2
« Reply #1308 on: 02/26/2018 08:42 pm »
@AncientU.  I don't really get your point.  The name of the game for manned flight isn't necessarily computing power, but reliability and resistance to radiation.  The flight computer for Orion/SLS are well designed for this. 
Evil triumphs when good men do nothing - Thomas Jefferson

Offline A_M_Swallow

  • Elite Veteran
  • Senior Member
  • *****
  • Posts: 8906
  • South coast of England
  • Liked: 500
  • Likes Given: 223
Re: SLS General Discussion Thread 2
« Reply #1309 on: 02/26/2018 09:03 pm »
The PowerPC 750FX first came out in 2002. IBM and Motorola (Freescale) no longer make the civilian version.

The PowerPC 750FX was replaced by the PowerPC e500 which in turn has been replaced by the faster e5500. The radiation hard version is called the RAD5500.

Offline AncientU

  • Senior Member
  • *****
  • Posts: 6257
  • Liked: 4164
  • Likes Given: 6078
Re: SLS General Discussion Thread 2
« Reply #1310 on: 02/26/2018 09:47 pm »
@AncientU.  I don't really get your point.  The name of the game for manned flight isn't necessarily computing power, but reliability and resistance to radiation.  The flight computer for Orion/SLS are well designed for this.

SLS software development is on the rocks and they're having a difficult time recruiting software developers. 
Quote
SLS Software Problems Continue
"....The SLS software team at MSFC is having great difficulty in hiring people to replace those who have quit. There is a lot of internal concern as a result of issues already raised with regard to SLS software safety to date that MSFC will literally have to go back to square one on software so as to verify it for use on human missions."

My point is that NASA is using old hardware and a kluged* up system of software, all being built in Alabama -- since 2006.  They are hiring because everyone is quitting...  There is no reason in the world that a sharp software developer would be professionally interested in this project.

The flight computer might be well designed for a Mars rover, but can 2.5 million lines of 'glue ware' code as of two years ago, plus all that has come since and remains to be written run successfully on it?  Can that software mess be reliable/bug free?  Can anyone justify a $300M software development program for a rocket (does it even need rad-hard electronics)?

* Inspector General's word, not mine.
« Last Edit: 02/26/2018 09:52 pm by AncientU »
"If we shared everything [we are working on] people would think we are insane!"
-- SpaceX friend of mlindner

Offline Kansan52

  • Full Member
  • ****
  • Posts: 1488
  • Hutchinson, KS
  • Liked: 570
  • Likes Given: 539
Re: SLS General Discussion Thread 2
« Reply #1311 on: 02/26/2018 10:04 pm »

The flight computer might be well designed for a Mars rover, but can 2.5 million lines of 'glue ware' code as of two years ago, plus all that has come since and remains to be written run successfully on it?  Can that software mess be reliable/bug free?  Can anyone justify a $300M software development program for a rocket (does it even need rad-hard electronics)?


Pardon the question, it's been a long time so I've done any coding but memory says that if the code is written properly, then the same computer (Mars rover flight computer) won't mind more lines of code. Unless, the processing speed is too slow to run the program quickly enough to do the job in the time window necessary.

Which prompts a question in my mind, are you saying the flight computer clock speed can't handle the work load?

Sorry, but my worry part of the brain is trying to understand.

Offline Zed_Noir

  • Senior Member
  • *****
  • Posts: 5490
  • Canada
  • Liked: 1809
  • Likes Given: 1302
Re: SLS General Discussion Thread 2
« Reply #1312 on: 02/26/2018 10:15 pm »
@AncientU.  I don't really get your point.  The name of the game for manned flight isn't necessarily computing power, but reliability and resistance to radiation.  The flight computer for Orion/SLS are well designed for this.

SLS software development is on the rocks and they're having a difficult time recruiting software developers. 
Quote
SLS Software Problems Continue
"....The SLS software team at MSFC is having great difficulty in hiring people to replace those who have quit. There is a lot of internal concern as a result of issues already raised with regard to SLS software safety to date that MSFC will literally have to go back to square one on software so as to verify it for use on human missions."

My point is that NASA is using old hardware and a kluged* up system of software, all being built in Alabama -- since 2006.  They are hiring because everyone is quitting...  There is no reason in the world that a sharp software developer would be professionally interested in this project.

The flight computer might be well designed for a Mars rover, but can 2.5 million lines of 'glue ware' code as of two years ago, plus all that has come since and remains to be written run successfully on it?  Can that software mess be reliable/bug free?  Can anyone justify a $300M software development program for a rocket (does it even need rad-hard electronics)?

* Inspector General's word, not mine.

You forget that all the SLS flight avionics are in the upper stage unlike more recent designs.

NASA is not going to find too many replacement software personnel. Non-standard kluged software for antiqued hardware means who ever NASA hires will have to be trained from scratch requiring more budget and time.

The fun part will be figuring out how well the software functions in a full up test.

« Last Edit: 02/26/2018 10:47 pm by Zed_Noir »

Offline AncientU

  • Senior Member
  • *****
  • Posts: 6257
  • Liked: 4164
  • Likes Given: 6078
Re: SLS General Discussion Thread 2
« Reply #1313 on: 02/26/2018 10:25 pm »
@AncientU.  I don't really get your point.  The name of the game for manned flight isn't necessarily computing power, but reliability and resistance to radiation.  The flight computer for Orion/SLS are well designed for this.

SLS software development is on the rocks and they're having a difficult time recruiting software developers. 
Quote
SLS Software Problems Continue
"....The SLS software team at MSFC is having great difficulty in hiring people to replace those who have quit. There is a lot of internal concern as a result of issues already raised with regard to SLS software safety to date that MSFC will literally have to go back to square one on software so as to verify it for use on human missions."

My point is that NASA is using old hardware and a kluged* up system of software, all being built in Alabama -- since 2006.  They are hiring because everyone is quitting...  There is no reason in the world that a sharp software developer would be professionally interested in this project.

The flight computer might be well designed for a Mars rover, but can 2.5 million lines of 'glue ware' code as of two years ago, plus all that has come since and remains to be written run successfully on it?  Can that software mess be reliable/bug free?  Can anyone justify a $300M software development program for a rocket (does it even need rad-hard electronics)?

* Inspector General's word, not mine.

You forget that all the SLS flight avionics are in the upper stage unlike more recent designs.

NASA is not going to find too many replacement software personnel. None standard kluged software for antiqued hardware means who ever NASA hires will have to be trained from scratch requiring more budget and time.

The fun part will be figuring out how well the software functions in a full up test.

Good point on the upper stage.

They could probably start from scratch with off the shelf hardware and modern software and get done faster, better, cheaper.  Rad harden in software like more recent designs.  Might find replacement software people easier, too.
"If we shared everything [we are working on] people would think we are insane!"
-- SpaceX friend of mlindner

Offline AncientU

  • Senior Member
  • *****
  • Posts: 6257
  • Liked: 4164
  • Likes Given: 6078
Re: SLS General Discussion Thread 2
« Reply #1314 on: 02/26/2018 10:41 pm »

The flight computer might be well designed for a Mars rover, but can 2.5 million lines of 'glue ware' code as of two years ago, plus all that has come since and remains to be written run successfully on it?  Can that software mess be reliable/bug free?  Can anyone justify a $300M software development program for a rocket (does it even need rad-hard electronics)?


Pardon the question, it's been a long time so I've done any coding but memory says that if the code is written properly, then the same computer (Mars rover flight computer) won't mind more lines of code. Unless, the processing speed is too slow to run the program quickly enough to do the job in the time window necessary.

Which prompts a question in my mind, are you saying the flight computer clock speed can't handle the work load?

Sorry, but my worry part of the brain is trying to understand.

I have zero idea if the computer clock speed can handle the code/work load in steady state -- but there isn't much of that during a rocket launch.  The real issue is can the bugs ever be cleaned up?  Can the system be made to reset/reboot fast enough that it will be robust?  Can the old hardware do fast enough self-checking to be reliable? 
I know that whenever we use wrappers around old code, it's buggy as hell. 

Would like to see an independent team of professional software developers evaluating the original plans for the system, how much code was written vs how much was planned, how the system performs in upset conditions, etc.

It doesn't appear that the project can heal itself by continuing to do what hasn't worked so far.
« Last Edit: 02/26/2018 10:48 pm by AncientU »
"If we shared everything [we are working on] people would think we are insane!"
-- SpaceX friend of mlindner

Offline Kansan52

  • Full Member
  • ****
  • Posts: 1488
  • Hutchinson, KS
  • Liked: 570
  • Likes Given: 539
Re: SLS General Discussion Thread 2
« Reply #1315 on: 02/27/2018 11:59 am »
Thanks!

Offline SWGlassPit

  • I break space hardware
  • Full Member
  • ****
  • Posts: 845
  • Liked: 893
  • Likes Given: 142
Re: SLS General Discussion Thread 2
« Reply #1316 on: 02/27/2018 03:02 pm »
They could probably start from scratch with off the shelf hardware and modern software and get done faster, better, cheaper.  Rad harden in software like more recent designs.  Might find replacement software people easier, too.

This kills the program.

A large part of the difficulty hiring people is that the overwhelming majority of software folks do not work in the field of high-reliability, real-time, hardware-interactive computing.  Computer Science programs around the country churn out web and app developers by the gross, but it's a totally different skill set.

Online oldAtlas_Eguy

  • Senior Member
  • *****
  • Posts: 5304
  • Florida
  • Liked: 5005
  • Likes Given: 1444
Re: SLS General Discussion Thread 2
« Reply #1317 on: 02/27/2018 05:10 pm »
They could probably start from scratch with off the shelf hardware and modern software and get done faster, better, cheaper.  Rad harden in software like more recent designs.  Might find replacement software people easier, too.

This kills the program.

A large part of the difficulty hiring people is that the overwhelming majority of software folks do not work in the field of high-reliability, real-time, hardware-interactive computing.  Computer Science programs around the country churn out web and app developers by the gross, but it's a totally different skill set.


Yes. The skill set is usually that of dual track high experienced personnel with EE and Software Engineering (includes extensive programming experience all the way down to assembler code) which currently are being offered high salaries by all the car companies doing self driving cars. The software for the cars is even more challenging than for a rocket. These highly capable people go where there is a high challenge not necessarily at much higher salaries. Plus these car software projects are likely to last decades vs 1 decade for the initial development with small fixes upgrades made by a very small team (< 1/4 of the development team size). Meaning the SLS software development is a "temporary job".

The other item is contending with all the conflicting government standards for software that make it difficult to write "tight" code. This creates "kludge" software more than anything else. Because the more exceptions handling added the more "kludges" are added to maintain execution speed. As more "kludges" are added more exption handling is added. At some point the software becomes so massive it is no longer debuggable or correctable by the programmers since it takes a very long time to discover what is causing the software failure in order to fix it. This is a exact example of a over-stressed environment for these high experienced personnel.

Look to the original software/hardware designers at not including all the possible inclusions the software would have to handle at the program start. Meaning no reuse of old software but clean sheet designs. Use of a modern RealTime-OS and modern high speed computers enable much faster and easier code development. The initial decisions to cuts costs by using existing software and hardware in the design is costing a lot more to patch in new software around hardware resource limitations (speed, memory, instruction set, thread switching [this is something that modern computer processors have worked on extensively to increase the effective execution speed], etc).

Again it is the cost cutting initial decisions that are ending costing more in time and money.

Offline AncientU

  • Senior Member
  • *****
  • Posts: 6257
  • Liked: 4164
  • Likes Given: 6078
Re: SLS General Discussion Thread 2
« Reply #1318 on: 02/27/2018 06:52 pm »
They could probably start from scratch with off the shelf hardware and modern software and get done faster, better, cheaper.  Rad harden in software like more recent designs.  Might find replacement software people easier, too.

This kills the program.

A large part of the difficulty hiring people is that the overwhelming majority of software folks do not work in the field of high-reliability, real-time, hardware-interactive computing.  Computer Science programs around the country churn out web and app developers by the gross, but it's a totally different skill set.

You over estimate the height of the wall involved with doing 'high-reliability, real-time, hardware-interactive computing' relative to many/most of the sophisticated software apps out there.  Certainly there are tonnes of generic web and app developers just as there are lots of kitty videos on line... but there is serious scientific and technical work being done in many fields that use more sophisticated software than that needed for rocketry.  OldAtlasGuy points out one, autonomous vehicles, where a 'super-computer' (NVIDIA Drive PV 2 AI computer*) is typical of the core processor.  The new constellations that employ Tb data rates among a network of inter-communicating satellites is another frequently discussed here.
* https://www.nvidia.com/en-us/self-driving-cars/drive-px/

I believe that if your 2018 software can run on a 2002 processor, you are not anywhere near the cutting edge software-wise.
"If we shared everything [we are working on] people would think we are insane!"
-- SpaceX friend of mlindner

Offline SWGlassPit

  • I break space hardware
  • Full Member
  • ****
  • Posts: 845
  • Liked: 893
  • Likes Given: 142
Re: SLS General Discussion Thread 2
« Reply #1319 on: 02/28/2018 03:08 pm »
I believe that if your 2018 software can run on a 2002 processor, you are not anywhere near the cutting edge software-wise.

In aerospace, you almost never want cutting edge.  You want reliability in unforgiving environments (radiation, vacuum, temperature extremes), with manageable power and thermal constraints.  New isn't necessarily good, and what folks use on the ground won't necessarily cut it in space.

Tags:
 

Advertisement NovaTech
Advertisement Northrop Grumman
Advertisement
Advertisement Margaritaville Beach Resort South Padre Island
Advertisement Brady Kenniston
Advertisement NextSpaceflight
Advertisement Nathan Barker Photography
0