ISS suffers external coolant loop issue - contingency spacewalks possible

[Targeteer]

(Chris Edit):

New thread for the ISS issue.

I'm editing in the great article on the situation by Pete Harding!

http://www.nasaspaceflight.com/2013/12/iss-suffers-coolant-loop-issue-spacewalks-possible/

Below are the first notes of the problem, thanks to Targeteer.

===

Evening DPC is detailing numerous system shutdowns due a flow control valve in ETCS (External Thermal Control System) string Alpha not closing properly.  The loop is too cold and loads are being shifted to string Bravo which is struggling to handle the load.  Shutdowns include powerdown of half of the systems in the JEM, Columbus, and Node 2.  The Node 2 MDM will be shut down.  The crew asked about the long term outlook if the valve continues to misbehave and the answer was that an R&R would be the eventual result.

[Targeteer]

Can the experts out there confirm an R&R requires an EVA? Seems obvious with the term External but maybe this component is inside?

[Targeteer]

Additional details on systems effected. Half of the smoke detectors in the modules will be in-op and caution/warning indications will be down but warning tones will remain--I think I have this last one correct but not totally sure.  Various computers and systems are being powered down in the JEM gracefully.  I believe I heard confirmation that the DDCU in Node 2 was also being powered down.

[Space Pete]

The ETCS FCV is located within the Pump and Control Valve Package (PCVP), which in itself is located inside the Pump Module (PM) ORU. The FCV regulates the temperature of ammonia coolant, by mixing cool ammonia exiting the radiators with warm ammonia that has bypassed the radiators.

As Targeteer said, the loop (A) is too cold, so it looks like the FCV would be either adding too much cool ammonia into the loop (which would fit with the valve not closing properly), or not mixing enough warm ammonia into the loop.

So, with the Loop A ETCS FCV down, we'd be looking at a Loop A ETCS PM R&R. For those that remember the previous Loop A ETCS PM R&R in August 2010, that turned out to be a three EVA job due to the very difficult QDs. There are currently three spare PMs outside the ISS - one on ESP-3, one on ELC-1, and one on ELC-2.

You can see here on ISSLive! that the Loop A NH3 outlet temperature is currently -32 degrees centigrade, whereas the Loop B NH3 outlet temperature is 4 degrees centigrade - big difference.
http://spacestationlive.jsc.nasa.gov/displays/spartanDisplay2.html

[NavySpaceFan]

From NBC Facebook Page:

BREAKING NEWS: NASA: Urgent situation aboard the International Space Station after cooling system fails; astronauts may have to perform a spacewalk in the next 24-48 hours for repairs.

[rdale]

The strange part is that the ISS status report issued on nasa.gov and Facebook about an hour ago has no problems at all. Disconnect somewhere?

[rdale]

Evening DPC with the problem details at http://issaudio.host-ed.me/iss/iss_20131211-140000-005.wav

[NavySpaceFan]

More from NBC:

http://www.nbcnews.com/science/nasa-reports-serious-problem-space-station-coolant-2D11729733

[jcm]

The ETCS FCV is located within the Pump and Control Valve Package (PCVP), which in itself is located inside the Pump Module (PM) ORU. The FCV regulates the temperature of ammonia coolant, by mixing cool ammonia exiting the radiators with warm ammonia that has bypassed the radiators.

As Targeteer said, the loop (A) is too cold, so it looks like the FCV would be either adding too much cool ammonia into the loop (which would fit with the valve not closing properly), or not mixing enough warm ammonia into the loop.

So, with the Loop A ETCS FCV down, we'd be looking at a Loop A ETCS PM R&R. For those that remember the previous Loop A ETCS PM R&R in August 2010, that turned out to be a three EVA job due to the very difficult QDs. There are currently three spare PMs outside the ISS - one on ESP-3, one on ELC-1, and one on ELC-2.

You can see here on ISSLive! that the Loop A NH3 outlet temperature is currently -32 degrees centigrade, whereas the Loop B NH3 outlet temperature is 4 degrees centigrade - big difference.
http://spacestationlive.jsc.nasa.gov/displays/spartanDisplay2.html

So wait, there's only one PM per Loop right? So the newly failed unit is the "new" one that was installed in 2010,
namely S/N 04  (PM-1)  currently installed on S1 , formerly on ESP-2 and launched on the STS-121 ICC. Is that correct?

[Targeteer]

From NBC Facebook Page:

BREAKING NEWS: NASA: Urgent situation aboard the International Space Station after cooling system fails; astronauts may have to perform a spacewalk in the next 24-48 hours for repairs.

The ISS story was second on the NBC evening news--following coverage of the NTSB hearings on the Asiana 777 crash at San Francisco.  Important timing because it was portrayed as a critical situation.  The discussion of possible a spacewalk was in the "weeks" not days time-frame and gave the impression the electronics were not being cooled enough as opposed to the system running too cold.  There was quick mention on CBS with few details and no input from Bill Harwood.  ABC's coverage was particularly alarmist and inaccurate starting with possible systems shutdowns--which have already occurred, a possible spacewalk with a clip of Sandra Bullock doing her movie spacewalk, and raising the specter of ISS de-manning.

[robertross]

The ETCS FCV is located within the Pump and Control Valve Package (PCVP), which in itself is located inside the Pump Module (PM) ORU. The FCV regulates the temperature of ammonia coolant, by mixing cool ammonia exiting the radiators with warm ammonia that has bypassed the radiators.

As Targeteer said, the loop (A) is too cold, so it looks like the FCV would be either adding too much cool ammonia into the loop (which would fit with the valve not closing properly), or not mixing enough warm ammonia into the loop.

So, with the Loop A ETCS FCV down, we'd be looking at a Loop A ETCS PM R&R. For those that remember the previous Loop A ETCS PM R&R in August 2010, that turned out to be a three EVA job due to the very difficult QDs. There are currently three spare PMs outside the ISS - one on ESP-3, one on ELC-1, and one on ELC-2.

You can see here on ISSLive! that the Loop A NH3 outlet temperature is currently -32 degrees centigrade, whereas the Loop B NH3 outlet temperature is 4 degrees centigrade - big difference.
http://spacestationlive.jsc.nasa.gov/displays/spartanDisplay2.html

and to note as a baseline, here is some info from an L2 manual:

"The default loop temperature setpoint in the PCVP firmware is 2.8 ± 1.1° C
(37° ± 2° F)."

So loop B is obviously right on the upper limit for thermal control.

[Space Pete]

So wait, there's only one PM per Loop right? So the newly failed unit is the "new" one that was installed in 2010,
namely S/N 04  (PM-1)  currently installed on S1 , formerly on ESP-2 and launched on the STS-121 ICC. Is that correct?

You are correct on all counts. Loop A PMs are bad boys.

[rdale]

Official NASA statement: Update on Space Station Cooling System
Dec. 11, 2013
Earlier Wednesday, the pump module on one of the space station’s two external cooling loops automatically shut down when it reached pre-set temperature limits. These loops circulate ammonia outside the station to keep both internal and external equipment cool. The flight control teams worked to get the cooling loop back up and running, and they suspect a flow control valve actually inside the pump itself might not be functioning correctly.
At no time was the crew or the station itself in any danger, but the ground teams did work to move certain electrical systems over to the second loop. Some non-critical systems have been powered down inside the Harmony node, the Kibo laboratory and the Columbus laboratory while the teams work to figure out what caused the valve to not function correctly and how to fix it. The crew is safe and preparing to begin a normal sleep shift while experts on the ground collect more data and consider what troubleshooting activities may be necessary.

[Space Pete]

Just in case anyone has forgot: EVAs are technically currently "grounded" due to the EMU water leak back in July. So this could get interesting.

Note to Mars mission planners: Re-design ISS coolant systems (as well as ELCSS).

[Malderi]

Yeah, considering how many critical items need EVAs for repairs, I wonder how many risk analyses took into account that EVAs themselves could be not available.

What's the risk/possibility of a single-person EVA using the good suit? I don't know enough about EVA protocols to know if there are safety requirements or procedures that absolutely require two people, or if the R&R is possible and would just go a lot slower.

[Targeteer]

Official NASA statement: Update on Space Station Cooling System
Dec. 11, 2013
Earlier Wednesday, the pump module on one of the space station’s two external cooling loops automatically shut down when it reached pre-set temperature limits. These loops circulate ammonia outside the station to keep both internal and external equipment cool. The flight control teams worked to get the cooling loop back up and running, and they suspect a flow control valve actually inside the pump itself might not be functioning correctly.
At no time was the crew or the station itself in any danger, but the ground teams did work to move certain electrical systems over to the second loop. Some non-critical systems have been powered down inside the Harmony node, the Kibo laboratory and the Columbus laboratory while the teams work to figure out what caused the valve to not function correctly and how to fix it. The crew is safe and preparing to begin a normal sleep shift while experts on the ground collect more data and consider what troubleshooting activities may be necessary.

Interesting that half of the systems in the three modules (including smoke detectors) count as "some" and "non-critical" 

[manboy]

Yeah, considering how many critical items need EVAs for repairs, I wonder how many risk analyses took into account that EVAs themselves could be not available.

What's the risk/possibility of a single-person EVA using the good suit? I don't know enough about EVA protocols to know if there are safety requirements or procedures that absolutely require two people, or if the R&R is possible and would just go a lot slower.

I thought they had three good suits?

[mtakala24]

I thought they had three good suits?

All suits are suspect pending root cause analysis. But they did find that debris/cause and subsequently successfully tested the failed suit. It all depends on the confidence of the analysis. Does anyone know if the failed part and debris samples from the suit got back on the ground already?

[sdsds]

the Loop B NH3 outlet temperature is 4 degrees centigrade

"The default loop temperature setpoint in the PCVP firmware is 2.8 ± 1.1° C
[...]
So loop B is obviously right on the upper limit for thermal control.

It is now outside the default setpoint range, at 4.14° C.

[erioladastra]

Yeah, considering how many critical items need EVAs for repairs, I wonder how many risk analyses took into account that EVAs themselves could be not available.

What's the risk/possibility of a single-person EVA using the good suit? I don't know enough about EVA protocols to know if there are safety requirements or procedures that absolutely require two people, or if the R&R is possible and would just go a lot slower.

Virtually zero chance of going out by self.  Plus there is still work ongoing to see if the suit up there is good.