Moving The Cloud to orbit

[sdsds]

[...] we *measured* 7kHz variations at the MW scale

This was on the AC side? That's insanity. Was the datacenter providing AC to each server in the cluster? My recollection of Cisco AGS routers is a bit fuzzy but I think back in the day when they were sometimes used in telco environments supplying the chassis with 24 or 48V DC was a configurable option. Is that not an option for AI servers these days?

[Lee Jay]

[...] we *measured* 7kHz variations at the MW scale

This was on the AC side? That's insanity.

Indeed.

Was the datacenter providing AC to each server in the cluster? My recollection of Cisco AGS routers is a bit fuzzy but I think back in the day when they were sometimes used in telco environments supplying the chassis with 24 or 48V DC was a configurable option. Is that not an option for AI servers these days?

It went transmission level - substation - 480V three-phase - single phase (can't remember if it was 120/208 or 277) - power supply (10-15kW each) - rack.

[Robotbeat]

Capacitors. Energy density of like 3J/gram for electrolytic ones. 15000Watts means you can smooth out the power flow for a second (ie plenty of time for a inverter to respond… inverters can respond within about a millisecond) with just 5kg of capacitors, one or two of magnitude more than needed to smooth out a 7kHz ripple to nothing. Heck, 100grams is probably enough. 10kg for MW scale.

Seriously. This is electricity 101. Not only is it not a showstopper, but given the ease of addressing this, I question your judgement for even thinking it's relevant to the conversation.

(And sounds like a problem with a switching power supply, not the actual load…)

[Lee Jay]

Capacitors. Energy density of like 3J/gram for electrolytic ones. 15000Watts means you can smooth out the power flow for a second (ie plenty of time for a inverter to respond… inverters can respond within about a millisecond) with just 5kg of capacitors, one or two of magnitude more than needed to smooth out a 7kHz ripple to nothing. Heck, 100grams is probably enough. 10kg for MW scale.

Seriously. This is electricity 101. Not only is it not a showstopper, but given the ease of addressing this, I question your judgement for even thinking it's relevant to the conversation.

(And sounds like a problem with a switching power supply, not the actual load…)

We think it's a problem with the power supply input magnetics resonating with the line parasitics after being stimulated by the extremely rapid load changes in the servers (which act like impulses - broad band forcing frequencies).  It's quite a hard problem to solve and a lot of people are working on it.  It's complex system with multiple layers of transformers, switches, and lines of different lengths in cable trays, and the switching noise of the power supplies all interacting.

[Robotbeat]

That makes a lot more sense. Also kind of off topic.

[Lee Jay]

That makes a lot more sense. Also kind of off topic.

Well, you could argue that the use of different electrical infrastructure would prevent the resonance, but remember what triggers is - multi-MW changes in load that happen in milliseconds, at random, unpredictable intervals.  Such impulses would be a challenge to handle for any power supply system even if it were based on DC and all solid-state because the changes in load include frequencies that are both higher and lower than both the closed-loop control frequency and the switching frequency of the power converters.

Some of these centers have on the order of 100,000 power supplies, effectively all in parallel.  Analyzing that from a network point of view is beyond our capabilities if you have to include the asynchronous switching frequencies involved as well as the closed-loop control inside each one.  It's like having a rocket with 100,000 engines, all running at different throttle settings with the ability to throttle from 0.001% to 100% in less than a millisecond, independently and at random times.  It's going to be a structural dynamics challenge to be sure.  At data centers, they rely on the stiff grid with all of its inertia to handle this.  In a DC-only system, there is no inertia and no infinite-bus.  Batteries are strong but, relative to the grid, they are not if they are sized for the local (data center) load.

[Robotbeat]

In a DC only bus, YOU CAN USE SIMPLE CAPACITORS. Seriously, the faster the change, the easier it is to smooth using capacitors.

[Lee Jay]

In a DC only bus, YOU CAN USE SIMPLE CAPACITORS. Seriously, the faster the change, the easier it is to smooth using capacitors.

It's not that simple.

You have a very large battery somewhere "far away" (electrically).  The distance is not uniform between it and the power supplies.  The power supplies have capacitors on the input, a switching system and capacitors on the output.  The battery and the lines have both inductance and capacitance as parasitics.  That means you have the opportunity for multiple LC oscillators at different frequencies and, because of the very rapid load variations, they will have a forcing function at their resonant frequencies.  That makes it complicated to stabilize.  Further, these load variations have a very high delta-I.  Put though the impedance of the system, you will have very large delta-V and thus very high EMI.  Further, the large changes will cause magnetostriction and the lenz effect creating both large physical forces and a noise.

Remember what you're switching.  Ultimately, it's single-digit volts at tens of megawatts.  Even at the power supplies, it might be low-double-digit volts.  You're still looking at on the order of a million amps changing in a fraction of a millisecond.  That's on the same order as lightning.  It's not a trivial problem to solve.

[Robotbeat]

Ok, but you literally claimed, incorrectly: “In a DC-only system, there is no inertia and no infinite-bus.”

In fact, it’s EASIER to provide “inertia” in a DC-only system as you just need to provide capacitors.

You’re doing this thing where you introduce a whole bunch of higher level concepts to try to inject more uncertainty to the discussion than is necessary to understand the primary constraints of the system, WHILE ignoring some of the basics of electric circuits.

It’s a good way to sound like you know what you’re talking about when you’re just kind of making up concern-troll objections and seeing what sticks. People who don’t have a solid technical background can be tricked by this, but it doesn’t work on people with a solid technical background.

And it’s unnecessary. There are straightforward challenges to making datacenters in orbit feasible, and they don’t rely on appealing to a half-remembered power supply failure from a completely different architecture.

[Robotbeat]

Remember what you're switching.  Ultimately, it's single-digit volts at tens of megawatts.  Even at the power supplies, it might be low-double-digit volts.  You're still looking at on the order of a million amps changing in a fraction of a millisecond.  That's on the same order as lightning.  It's not a trivial problem to solve.

It literally is. DC-only bus is FAR simpler than our AC grid. Put capacitors on the bus. DONE.

[Asteroza]

Terrestrial datacenters rely on the grid "spinning reserve" to deal with some of this. I mentioned in the SPS thread, but why not the large ribbon flywheel energy storage system proposed for SDI wheelguns, as a way of bringing similar capability, as flywheels essentially don't have a cycle life issue per se.

[Lee Jay]

Ok, but you literally claimed, incorrectly: “In a DC-only system, there is no inertia and no infinite-bus.”

What I said is correct .

In fact, it’s EASIER to provide “inertia” in a DC-only system as you just need to provide capacitors.

You’re doing this thing where you introduce a whole bunch of higher level concepts to try to inject more uncertainty to the discussion than is necessary to understand the primary constraints of the system, WHILE ignoring some of the basics of electric circuits.

It’s a good way to sound like you know what you’re talking about when you’re just kind of making up concern-troll objections and seeing what sticks. People who don’t have a solid technical background can be tricked by this, but it doesn’t work on people with a solid technical background.

And it’s unnecessary. There are straightforward challenges to making datacenters in orbit feasible, and they don’t rely on appealing to a half-remembered power supply failure from a completely different architecture.

Your simplistic understanding of electrical systems has led you to think this is trial when it's really not.  You're ignoring parasitics, which are not ignorable, you're ignoring that the voltage changes many times between source and use and those assumptions, which are false, lead you to the wrong conclusion that capacitors fix everything.

Why is it that the a laptop, powered by either a DC power supply (which has capacitors on the output) or a battery (which has a large output capacitance), has both bulk electrolytic capacitors and hundreds of tantalum capacitors on the board?  If you don't truly understand the reasons for that, then you don't understand the problem you're trying to solve, except th e actual problem is 10 million times larger, literally.

[edzieba]

Remember what you're switching.  Ultimately, it's single-digit volts at tens of megawatts.  Even at the power supplies, it might be low-double-digit volts.  You're still looking at on the order of a million amps changing in a fraction of a millisecond.  That's on the same order as lightning.  It's not a trivial problem to solve.

It literally is. DC-only bus is FAR simpler than our AC grid. Put capacitors on the bus. DONE.

WHICH bus? The variable DC bus from the solar panels to the distribution SMPS (because the voltage from PV panels in practice is far from stable or constant)? The bus from that inverter to the second inverter (because power distribution does not occur at the ~1V that sink devices need, because conversion and resistive losses would be astronomical) and the individual stepdowns on the device boards? The busses between all the stepdown power supplies and the individual rails of the low-voltage-high-current dies (which are ~1V but can be hundreds to thousands of amps)? How about the rails between all the power conditioning and buffering devices in-between?
And then how much capacitance are you applying? If you don't want to experience voltage dip as your capacitors need to discharge to cover a 10MW power dip over a few milliseconds, you need very large capacitors. And then you need to be able to handle a 10MW spike that followed that dip, so you also need active management of your capacitor banks to ensure your capacitors are not all charged all the time so you have somewhere for that power spike to end up. Those massive capacitor banks also have impedance and even inductance (try wrapping a few megawatts around a wrapped foil sheet in a few milliseconds and see if you don't get a magnetic field forming and collapsing!) to deal with. Is it capacitors all the way down? How do those banks deal with multiple parallel but independent busses to handle redundancy and failover as well as normal switching (e.g. as an array is self-shaded passing through a singularity once per orbit)?

If you're taking rules that work for basic uC circuitry where you can approximate with resistive loads perfectly adequately, and applying them to megawatt scale isolated microgrids, then you're applying the wrong rules. This should be self-evident from reality: The datacentres experiencing these issues in the real world, today already use DC distribution internally to the racks (48VDC is 'standard', but hyperscalers often do their own thong because they design their own hardware).
If "just use capacitors, idiots!" worked, then they would already be doing that and there would be no problem.

[Robotbeat]

Everyone IS doing it. Every power supply (including DC-DC) has a bunch of capacitors. The problem Lee Jay described was due to a faulty power supply.

We’re inventing fake problems where there are already plenty of real ones. Take an electricity or electronics course.

[Lee Jay]

Everyone IS doing it. Every power supply (including DC-DC) has a bunch of capacitors. The problem Lee Jay described was due to a faulty power supply.

There were over 10,000 of them.  And the "problem" is caused by the extremely rapid and large load variations from these server farms.  Having load vary by 100MW in sub-millisecond times is a problem.  It's worse that a steel mill arc furnace.

We’re inventing fake problems where there are already plenty of real ones. Take an electricity or electronics course.

I have two degrees in electrical engineering.

[Robotbeat]

Inexcusable, then!

[Lee Jay]

Inexcusable, then!

Please explain why I have seen publications and presentations by industry, government and academic experts, including those running high-MW field tests and collecting actual data, describing this problem in detail including how difficult it is to solve.  Some of the solves involve changing the software to stop the load from varying so much, but that costs energy (essentially when the actual load drops, you give the servers dummy tasks to keep the load from dropping to zero all at once).

It's a hard problem which you are over simplifying because you haven't thought through it or studied the issue.

[Paul451]

The problem Lee Jay described was due to a faulty power supply.

Wow, you didn't read a single thing he wrote, did you?

[Robotbeat]

Inexcusable, then!

Please explain why I have seen publications and presentations by industry, government and academic experts, including those running high-MW field tests and collecting actual data, describing this problem in detail including how difficult it is to solve.  Some of the solves involve changing the software to stop the load from varying so much, but that costs energy (essentially when the actual load drops, you give the servers dummy tasks to keep the load from dropping to zero all at once).

It's a hard problem which you are over simplifying because you haven't thought through it or studied the issue.

Because they’re all attaching to an AC grid dominated by slow response thermal power plants.

Additionally, the fact that a bunch of people are studying secondary effects from the load profile of AI training as almost no bearing on the actual discussion, as clearly these problems are solvable or tolerable because we HAVE these models and these models have been trained successfully again and again on the ground, even using The more complicated power system of the grid.

it’s pretty clear you’re not arguing in good faith as you think we should pursue some sort of energy degrowth policy and have, in the past, expressed hostility to the idea of human expansion into space. The thing about being highly educated is you can produce a whole bunch of highly technical sounding arguments that a lot of observers might find plausible… if you don’t actually look at the issues being discussed from first principles or apply basic logic. Such as:

Is AI load profile a showstopper for orbital datacenters? No more so than for Earth, where it clearly isn’t, and in fact, probably much less so due to the fact you have full control over the power distribution architecture.

[Robotbeat]

The problem Lee Jay described was due to a faulty power supply.

Wow, you didn't read a single thing he wrote, did you?

he literally confirmed my suspicion that it was likely due to faulty power supply