Virtual reality

[KelvinZero]

aerostationary orbit is about 17,000 km above mars's surface.

That would put a delay of 56ms.  A bit too much for comfortable VR (it is believed that delays become acceptable below 20ms), but usable I guess.

Orbit closer, take frequent coffee breaks..

[gosnold]

20 ms is motion to photon latency for looking around, if your rover/lander has a panoramic set of cameras, it can record and transmit a 360° view to the orbiter, and then the latency is only determined by the computer you put in the orbiter. If you want to interact with your environment however (with robotic arms for instance), the communication lag becomes an issue but a higher latency (>20ms) could be acceptable since it will probably not make you as sick as visual latency.

[grondilu]

20 ms is motion to photon latency for looking around, if your rover/lander has a panoramic set of cameras, it can record and transmit a 360° view to the orbiter, and then the latency is only determined by the computer you put in the orbiter. If you want to interact with your environment however (with robotic arms for instance), the communication lag becomes an issue but a higher latency (>20ms) could be acceptable since it will probably not make you as sick as visual latency.

Yes it's probably possible to do tricks like this.  But instead of a record of a panoramic view, I'd rather imagine that a computer would create a 3D model of the scene and display this instead of a real footage.  To some degree it could also simulate the physics of your behavior so that you can have a near-zero latency feedback.  The real data from the surface would only be used to correct the VR environment whenever the computer simulation was inaccurate.

For instance, if you want to see what's below a rock, you could lift the rock in the VR environment, the computer would send the corresponding command to the surface, and while it waits for the feedback, it would simulate it by guessing what could be below the rock.  You would stare at the rock a few seconds, and during this time the computer would have received the data from the robot on the ground so it could update the picture.  So during half a second or something, you would see nothing interesting but then suddenly the image would be updated as the real data would show up.

[KelvinZero]

Online multiplayer games attempt to deal with network lag with a combination of extrapolation and interpolation.. I guess we have all seen the artifacts of that.

I suppose for teleoperation you would have add a bit of paralysis so movements you thought you were applying to one situation are not suddenly applied to the world as it was a second earlier. I can imagine this being very mentally tiring. You are doing some action like screwing a bolt into a hole and a "waiting" icon pops up, and you have to review the situation to realize you are no longer grasping the bolt or some such. I think I would end up thinking in 2 second chunks: pick up  the bolt, did I really? No. Pick up the bolt. Did I really? put it in the hole, did I really? etc..

[aero]

Online multiplayer games attempt to deal with network lag with a combination of extrapolation and interpolation.. I guess we have all seen the artifacts of that.

I suppose for teleoperation you would have add a bit of paralysis so movements you thought you were applying to one situation are not suddenly applied to the world as it was a second earlier. I can imagine this being very mentally tiring. You are doing some action like screwing a bolt into a hole and a "waiting" icon pops up, and you have to review the situation to realize you are no longer grasping the bolt or some such. I think I would end up thinking in 2 second chunks: pick up  the bolt, did I really? No. Pick up the bolt. Did I really? put it in the hole, did I really? etc..

Hopefully your avatar would have some basic work related preprogrammed trained ability to aid in carrying out rudimentary steps, "grasping at the bolt" --> pick up the indicated bolt and look at it, "shove the bolt toward the hole" --> place the bolt in the hole and look at it, "grasp and twist the bolt --> screw the bolt into the hole and look at it.  Look at it until receipt of the next step directions.

Or likely more. It depends on how autonomous you want the avatar to be.

[grondilu]

You are doing some action like screwing a bolt into a hole and a "waiting" icon pops up, and you have to review the situation to realize you are no longer grasping the bolt or some such. I think I would end up thinking in 2 second chunks: pick up  the bolt, did I really? No. Pick up the bolt. Did I really? put it in the hole, did I really? etc.

VR would be absolutely unusable if there was anything close to one second delay between a movement of your body and a visual feedback.  Again, it is believed that an acceptable delay must be below 20ms.

20ms is physically unfeasible if you include in the pipeline the back-and-forth communication between the avatar on the ground and the computer on the spacecraft.

As I see it, one way to solve this is to simulate all your actions and rely on communication with the ground only in order to improve the VR model as more knowledge is acquired about the martian surface.

So when you screw a bolt, the VR system simulates what happens when you screw a bolt.  The computer does not need much knowledge about Mars in order to do that.  It does instruct the avatar on the ground to screw the real, physical bolt, but it uses the feedback only to check that it is conform to what it has simulated a second before.   It will usually be.  Hopefully the cases when the simulation was not conform would be either rare or slight enough so that corrections do not impend the user interaction too much.

PS.  One reason why this could be possible is that Mars is a fairly static environment.  The ground and atmosphere have very little activity (dust storms happens but they are quite rare and you don't want to do anything outside then, anyway).   Thus the only thing that would move on mars would basically be human artifacts, whose behavior is well known (since we either built or command them) and can thus be relatively easily simulated.  It's not like the surface of mars was populated by lots of living things moving all around in an unpredictable way.

[KelvinZero]

Hopefully your avatar would have some basic work related preprogrammed trained ability to aid in carrying out rudimentary steps, "grasping at the bolt" --> pick up the indicated bolt and look at it, "shove the bolt toward the hole" --> place the bolt in the hole and look at it, "grasp and twist the bolt --> screw the bolt into the hole and look at it.  Look at it until receipt of the next step directions.

Or likely more. It depends on how autonomous you want the avatar to be.

We are getting there, but I think basic agility/dexterity have proven to be very difficult problems. Look how slowly we have progressed on things like walking robots and object recognition, and we have had many reasons to bash our heads against these repeatedly. Operating a remote avatar with significant fraction of a second's lag might only be a critical problem for space.. and perhaps online gamers living in NZ

[KelvinZero]

You are doing some action like screwing a bolt into a hole and a "waiting" icon pops up, and you have to review the situation to realize you are no longer grasping the bolt or some such. I think I would end up thinking in 2 second chunks: pick up  the bolt, did I really? No. Pick up the bolt. Did I really? put it in the hole, did I really? etc.

VR would be absolutely unusable if there was anything close to one second delay between a movement of your body and a visual feedback.  Again, it is believed that an acceptable delay must be below 20ms.

Yes thats understood, and thats what I was describing. Im assuming instantaneous feedback because you are operating on a virtual environment, however Im also assuming that robot on mars will frequently mess up. Even humans repeatedly fumble when doing simple tasks like picking up a bolt, we dont even think about it and often begin retrying mid-fumble. The mars robot will probably fumble even more often, diverging from the virtual simulation, and retrying cannot begin until real information is returned to the human operator. (or as Aero mentions, if whatever is on Mars has some high degree of autonomy where it can invent new strageties to keep the virtual and mars environments in sync)

[grondilu]

Yes thats understood, and thats what I was describing. Im assuming instantaneous feedback because you are operating on a virtual environment, however Im also assuming that robot on mars will frequently mess up. Even humans repeatedly fumble when doing simple tasks like picking up a bolt, we dont even think about it and often begin retrying mid-fumble.

Well, teleoperation can only be as good as robotic avatars are, that's for sure.

Also, possibly a very powerful computer would be able to predict the avatar's failures and include them in the simulation.  After all the VR model must include not only mars's features, but also the robotic avatar itself.

[grondilu]

The considerations above made me think of a VSauce video:



In a nutshell the point is that the brain only process sensory data with a 80 ms delay already.  Yet we don't see that, and it's probably because the brain does lots of tricks that could be analogous to what would be done by a VR system for teleoperation.

[Hop_David]

Laser Link to Moon Trumped NASA and MIT Engineers’ Expectations

« In October of last year, a team from NASA and MIT’s Lincoln Laboratory made space communications history by beaming data, via laser, at speeds reaching 622 megabits per second, to Earth from a spacecraft orbiting the moon. »

This means that a robot on the surface of mars could transmit (presumably also receive) data from an orbiting spacecraft at very decent bitrate.   That's quite significant for the prospect of a VR-based teleoperated mission.

Just checked:  the aerostationary orbit is about 17,000 km above mars's surface.

That would put a delay of 56ms.  A bit too much for comfortable VR (it is believed that delays become acceptable below 20ms), but usable I guess.

Eh? 2 * (17,000 km) / (300,000 km/s) = .11 seconds

You would want the tele operators to have adequate radiation shielding. The easiest way to do this bury them on a Mars moon. But Deimos is even further than aerostationary. Light lag for Phobos to Mars surface and back would be .04 seconds, about twice what you say is acceptable.

To get light round trip down to .02 seconds, you would need an orbital altitude of about 3000 km. Which would have a period of 4.3 hours. To keep constant line of sight with a mars surface worksite, you'd need a constellation of sats.

For the moon, a 3000 km altitude orbit would have a period of 8 hours. Since the moon has a smaller radius, a sat would enjoy line of sight for a greater fraction of the orbit. Still, it'd require a constellation for the moon also.

[grondilu]

Eh? 2 * (17,000 km) / (300,000 km/s) = .11 seconds

Yeah I forgot the return time.

To keep constant line of sight with a mars surface worksite, you'd need a constellation of sats.

That would not do since it would increase the light path.

[KelvinZero]

You would want the tele operators to have adequate radiation shielding. The easiest way to do this bury them on a Mars moon. But Deimos is even further than aerostationary. Light lag for Phobos to Mars surface and back would be .04 seconds, about twice what you say is acceptable.

It probably is the easiest way to get huge shielding but it would also introduce other issues that we currently have no experience with. floating grit, tiny but not ignorable gravity, whatever operations are involved in such digging so far from home..

My guess is version 1.0 would be as ISS-like as possible. Stick near as possible to mars and I guess that would roughly halve cosmic radiation, similar to ISS in LEO? Perhaps additional shielding could be seen as similar to the asteroid capture mission: grab a sealed bag full of regolith robotically. Or perhaps the ISS-like base is firstly used to explore Deimos (without landing) and later it is moved to a low mars orbit for teleoperation. I think Deimos ISRU is meant to be more promising in any case.

[Hop_David]

You would want the tele operators to have adequate radiation shielding. The easiest way to do this bury them on a Mars moon. But Deimos is even further than aerostationary. Light lag for Phobos to Mars surface and back would be .04 seconds, about twice what you say is acceptable.

It probably is the easiest way to get huge shielding but it would also introduce other issues that we currently have no experience with. floating grit, tiny but not ignorable gravity, whatever operations are involved in such digging so far from home..

My guess is version 1.0 would be as ISS-like as possible. Stick near as possible to mars and I guess that would roughly halve cosmic radiation, similar to ISS in LEO? Perhaps additional shielding could be seen as similar to the asteroid capture mission: grab a sealed bag full of regolith robotically. Or perhaps the ISS-like base is firstly used to explore Deimos (without landing) and later it is moved to a low mars orbit for teleoperation. I think Deimos ISRU is meant to be more promising in any case.

I think the floating grit would be an issue, yes. Seeing how there's no jostling of dust particles by wind or water, I would expect Phobos dust to be very abrasive, like our lunar dust.

I still fantasize about a base on Stickney crater. The crater walls would afford a lot of protection. As would Mars, it takes up a big portion of the sky. If I remember right, Buzz Aldrin was advocating bulding Mars infra-structure using tele-operators on Phobos. I'll see if I can dig up his recent book.

Grondilu, in that NOW video they mentioned how subjects becoming accustomed to an 80 millisecond delay had the illusion of the light coming on before they pressed the button when the delay was shortened. This leads me to wonder: Can people grow accustomed to a longer latency than 20 ms? The necessity of keeping teleoperators within 3000 km of the worksite sure eliminates a lot of options.

[RonM]

I think we have an issue between frame rate, time delay, and good enough. 20 ms or better would give you visual perfection for VR, something that is not required to teleoperate a robot. Now, does this 20 ms number refer to frame rate or time delay. You can still have a high frame rate and a high time delay.

Movies are 24 fps, about 40 ms, and while a quick pan looks choppy, it's fine for the average person. Slow pans and stationary views look great. 20 ms is 50 fps, but once again, what does that have to do with time delay?

Typical human reaction time is 100 ms. That's for instinctive reactions and anticipated events (wait for it . . .). Unexpected events where you have to make a snap decision, say while driving a car and you have to avoid a hazard that just appeared, take about 300 ms for the average person.

As long as the robot doesn't move quickly, a delay of under a second will be acceptable.

The US operates combat drones halfway around the world from bases in the US. Using satellites, that's at least a 250 ms delay, assuming using only one satellite. 500 ms if you include the round trip.

[grondilu]

The 20ms refer to what at Oculus they call the "motion to photon" delay.  That is the time between the moment you move your body (typically your neck) and the moment your eyes receives the photons that result from this change of position of your body.

The time your brain needs to react to an external stimulus has very little to do with this : it's a very different issue.  Your organism is not prepared at all to withstand any substantial delay between your movements and the resulting visual input, because it is always pretty much zero IRL.  When such a delay happens, the mismatch between your proprioception, your vestibular system and your eyes can create motion sickness, which is powerful enough to make you through up.

[gosnold]

20ms is the time delay Oculus wants for for vision, the framerate they target is around 90Hz. Now for proprioception you might be able to get away with a higher time delay, this remains to be investigated.
If even for proprioception the maximum acceptable time delay is 20ms, you have a maximum range of 3000km, so if you orbit low around mars (with an orbit having a 19 000 km length passing over where your teleoperated robot is), you can use it 30% of the time (3000km before passing over, 3000km after). This requires a constellation of data relay satellites in low orbit with a high datarate (something like 32 bit color 90Hz*720p per eye, around 5Gbps uncompressed).

[RonM]

Then it sounds like VR is not the solution for operating robots from orbit. More conventional controls would work well.

VR would be a great solution from a base or ground vehicle. VR control of a nearby robots would mean astronauts would not have to suit up. They could have the same experience as being in a spacesuit, but from the comfort and safety of their habitat.

[aero]

It sounds to me like we are talking about the delay in receiving a visual when you glance over your shoulder? If the view is not there until after you look, then it causes motion sickness. Is that right?

If so, what is to prevent you from seeing the view over your shoulder from the same time-frame as the front view? That is, the avatar visual sensors capture and transmit a 360 degree view continuously and you look at what ever section of the synchronized visual information you are interested in. It will all be delayed by some large number of ms, but so what?

It may take more bandwidth but I must ask, " Is that more expensive than a constellation of low orbiting relay satellites with a high data rate?"

[gosnold]

It sounds to me like we are talking about the delay in receiving a visual when you glance over your shoulder? If the view is not there until after you look, then it causes motion sickness. Is that right?

If so, what is to prevent you from seeing the view over your shoulder from the same time-frame as the front view? That is, the avatar visual sensors capture and transmit a 360 degree view continuously and you look at what ever section of the synchronized visual information you are interested in. It will all be delayed by some large number of ms, but so what?

It may take more bandwidth but I must ask, " Is that more expensive than a constellation of low orbiting relay satellites with a high data rate?"

That is what I pointed out in post 41, but if your move your arm and see the robotic arm you control move much later, it can also be disturbing.
If you accept a 70ms delay  you can control your robot very simply through motion capture from anywhere in low Mars orbit with a constellation of low orbiting relay satellites. You would need to stream the whole 360° field of view though, so you would have something like 4 times more data to stream.
If you accept 120ms delay you can do the same from a Mars synchronous orbit just above your robot.