Virtual reality

[grondilu]

Here is a recent article from arstechnica:

How gaming tech is making for better interplanetary exploration

"My dream in this area is that, someday, when we put human boots on the surface of Mars, I want there to be millions of people in attendance for that event," Jeff Norris, Mission Operations lead at NASA's Jet Propulsion Laboratory told Ars in a recent interview. "I want them not just sitting in their living room watching a television screen; I want them standing on Mars in their own holodecks right there beside the astronauts."

I think this is not just a dream.   It's what is going to happen.  In case you don't know, Virtual Reality is coming.  The technology is now here, and is currently demonstrated with thousands of units of the Oculus rift prototype, which has been sold all around the world, and whose consumer, final version should be available next year.



By the way, the Oculus rift is not unknown in the space industry.  It was mentioned by Elon Musk on the SpaceX channel, for instance.  They use it for CAD experimentations:



Next month, the Gaia mission will be launched.  After a few years, it will have given us a map of a billion stars and other celestial bodies.   3D programs  like Celestia or spaceengine, once adapted to VR, will make it possible for the regular Joe to explore the galaxy as never before.




Space is a terrible place.  It's dangerous, and just staying alive there is extremely costly.   Yet we want to know and see what's up there, which is quite a natural consequence of our human curiosity.  But , do you think we have to physically be in space in order to satisfy this curiosity?


Moreover, when an astronaut is in space, he already looks around him through a helmet.  He doesn't touch anything with his own skin.  Were he standing on mars, he would not breathe martian air, nor would he feel martian wind.   So, there is already quite a thick layer of technology between his senses and the place he explores.    Is it so much different with VR?

[Oli]

Absolutely, could not agree more.

What is still holding VR back is the lack of screen resolution/quality (when you have 110° field of view HD is nothing), but that will change.

Of course you won't be able to replicate zero gravity, but apart from that why would you want to go to space when you can get almost the same experience here on earth with VR?

Robotics and VR may very well eliminate the need for HSF altogether.

And guess what, both technologies will progress at a much faster rate than rocket technology. You cannot beat Moore's Law.

SpaceX and Skylon are already obsolete before they even started.

[grondilu]

SpaceX and Skylon are already obsolete before they even started.

Well, not so much.  We'll still have to send probes and robots in order to get the data we'll need to feed our VR models.

Unless people are satisfied with purely fictional VR environments and get disinterested in what actually exists outer space.   This is a real possibility though, at least according to the Transcension hypothesis.

[Oli]

^

I have the appropriate image for that below

Of course we are technologically very far from that, but as you said, for virtual spaceflight it could become acceptable in the next 10 years or so.

And no, we will never get disinterested in what actually exists in outer space 

Edit: Btw the graphics in the videos above are horrible

[KelvinZero]

Robotics and VR may very well eliminate the need for HSF altogether.

Its a personal value judgement. To me HSF has always been about eventual space settlement and exponential growth of the human race out into the universe. That something else could duplicate the experience is besides the point. The experience will probably be something like claustrophobia and being sea sick a lot of the time.. why spend effort to duplicate that?

Having this clear though, Im also not at all afraid of using robots and VR to further this goal of space settlement. There are hundreds of questions to answer and technologies to develop. Sometimes robots are the best way to answer or develop these.

Just wanted to insert a recent hobby horse also: combine COLBERT-like treadmill, a teleoperation booth and VR immersive video game environment to allow people to live together in closet sized areas for months on end without losing too much bone mass or going nuts from claustrophobia or interpersonal conflicts.

[Nilof]

I don't think that VR will make HSF obsolete. It may however make spacesuits obsolete.

[grondilu]

To me HSF has always been about eventual space settlement and exponential growth of the human race out into the universe.

That's an unrealistic point of view.

[QuantumG]

To me HSF has always been about eventual space settlement and exponential growth of the human race out into the universe.

That's an unrealistic point of view.

That's your opinion.

[grondilu]

And no, we will never get disinterested in what actually exists in outer space 

Well, never completely, sure.   But maybe enough to estimate that it is not worth the efforts and resources required for space exploration.   I'm not saying it's going to be so, but it is a possibility.   To some extend, the lack of public interest is already a factor that impacts space agencies budgets quite a bit, I guess.


On an other note, I just watched the latest video update on Atlas:



It looks good, and it makes the idea of teleoperating an android robot more promising:

[KelvinZero]

To me HSF has always been about eventual space settlement and exponential growth of the human race out into the universe.

That's an unrealistic point of view.

Hi grondilu, let's not derail your thread to a "Whats HSF for" debate, but you really are holding a minority viewpoint here.   It doesn't mean the majority is right, but It does mean you need to explain your point of view before people can even guess what it is. At least notice that you are dismissing pretty much every big name in popular science from Elon Musk to Stephen Hawking.

[Oli]

^

Well at present its certainly a totally unrealistic point of view. But of course, if humanity discovers a nice planet in another solar system, you can expect humans to settle there (if environmental regulation does not forbid it )

To some extend, the lack of public interest is already a factor that impacts space agencies budgets quite a bit, I guess.

Space science is doing fine. HSF is not about exploration, that can be done by robots far cheaper/easier.

[KelvinZero]

Well at present its certainly a totally unrealistic point of view. But of course, if humanity discovers a nice planet in another solar system, you can expect humans to settle there (if environmental regulation does not forbid it )

There are totally realistic technologies and projects we can start on right now, we don't even need to get humans to LEO to begin them. It is only bad sci-fi movies that suggest interstellar travel will become easy while life support and ISRU will remain hard.

Lets try to pull this back on topic. VR is totally relevant without suggesting HSF is just a spectacle, no pun intended.

[sanman]

Well, just imagine that you're actually on Mars - you could still make tremendous use of telepresence technologies (I wouldn't necessarily call them "virtual reality", since the display could be showing a live video feed)

Instead of always having people don spacesuits to leave their habitat, they could put on a pair of VR goggles to pilot a robot that will do the hard work outside for them. That minimizes the risks, and yet still allows a personal hands-on approach to doing the tasks at hand. The robot would always be outside, ready to be activated and piloted.

Remember that Bruce Willis movie with the android avatars? Likewise, you could have entire work crews gathering outside to do hard work in robot form. Your job might be a construction worker, and yet all you would do is sit in your chair all day, using telepresence to pilot a robot that does that work in the dangerous outdoor Martian environment.

Does anyone object to that view of the future, or find it unrealistic?

[QuantumG]

Remember that Bruce Willis movie with the android avatars?

Unfortunately yes.

[Oli]

It is only bad sci-fi movies that suggest interstellar travel will become easy while life support and ISRU will remain hard.

Its hard to speculate because we have no clue how interstellar travel will look like

Lets try to pull this back on topic. VR is totally relevant without suggesting HSF is just a spectacle, no pun intended.

Who said VR will be just a spectacle? If VR is as good as the real world, why should we move physically? However how "real" it will be depends on the application. A view from LEO will be easier to replicate (with high FOV/res video feed or CGI), than walking around on Mars.

[QuantumG]

This sounds like a discussion from when I was a teenager (20 or so years ago).

It's not like there's been some revolutionary breakthrough in VR.. they're just attacking the latency problem.

If they succeed, I expect a few thousand units to ship before the hype wears off any everyone rediscovers that resolution is still king.

[Oli]

It's not like there's been some revolutionary breakthrough in VR.. they're just attacking the latency problem.

No there has not, but the resolution of affordable screens may now be good enough for consumer VR.

If they succeed, I expect a few thousand units to ship before the hype wears off any everyone rediscovers that resolution is still king.

Resolution is reaching a point where improvements are not noticeable anymore. Yes, we'll probably see 4k smartphones and 8k TV, but at some point you can only gain from higher res. by increasing field of view. In particular for gaming this is highly desirable (not so much for movies).

[QuantumG]

Gaming is what this latest rekindling of interest in VR is about. There will be other applications for this hardware, but they're sidelines. If they don't make the gaming sales, they'll be sunk and fade into the background like every other VR company has, selling an occasional high end system to a military or medical outfit.

To me, the outcome seems clear: they'll sell a bunch of units on the novelty factor, then gamers will start demanding resolution upgrades. If they can do that, affordably, they'll get a much bigger market and decades of sales. If not, they'll fade away as a passing fad.

That's just my opinion, though.

[Oli]

^

First of all, I'm not talking about that particular company. What they did was nothing special from a technical POV and I doubt they can patent it, but they did it at a point in time where the experience is good enough to trigger interest.

Secondly, I would not generally limit VR to head mounted displays only, however increasing screen sizes has its limits so I think it will go towards HMD in the long term.

Finally, its not only gaming. Imagine having a pair of augmented reality glasses which provide you with a huge 180° screen estate everywhere you go. That's a killer application (good see-through glasses are really hard to do however, so very long term).

[QuantumG]

I can imagine all sorts of things.. what's actually being done is what you should be interested in.

[Oli]

I can imagine all sorts of things.. what's actually being done is what you should be interested in.

You say that in the advanced concepts section of a spaceflight forum 

[KelvinZero]

Well, just imagine that you're actually on Mars - you could still make tremendous use of telepresence technologies (I wouldn't necessarily call them "virtual reality", since the display could be showing a live video feed)

Instead of always having people don spacesuits to leave their habitat, they could put on a pair of VR goggles to pilot a robot that will do the hard work outside for them. That minimizes the risks, and yet still allows a personal hands-on approach to doing the tasks at hand. The robot would always be outside, ready to be activated and piloted.

That is my view, Short range teleoperation will probably replace spacesuits entirely at some point. The list of advantages is huge. Its not clear when this will happen but I personally expect this technology to make leaps and bounds. Teleoperation can potentially surpass human dexterity, whereas a human in a suit can only approach it.

[grondilu]

Hi grondilu, let's not derail your thread to a "Whats HSF for" debate, but you really are holding a minority viewpoint here.   It doesn't mean the majority is right, but It does mean you need to explain your point of view before people can even guess what it is. At least notice that you are dismissing pretty much every big name in popular science from Elon Musk to Stephen Hawking.

I'm fine with holding a minority viewpoint here.   I'm not a big fan of argumentum ad populum, anyway.

[grondilu]

This sounds like a discussion from when I was a teenager (20 or so years ago).

It's not like there's been some revolutionary breakthrough in VR.. they're just attacking the latency problem.

VR didn't need a revolutionary breakthough.  It's just that the technology is now ready, mostly thanks to advancements in consumer electronics.   Now there is enough processing power to generate visually convincing 3D environments.  There are also cheap accelerometers, and high-resolution displays coming from the smartphone industry.

Sometimes a technology emerges not because some genius inventor cracked it (though Palmer Luckey was indeed kind of smart), but because the time is right.

[QuantumG]

Ya know it doesn't exist right?

All HMD systems suck. Oculus Rift will suck differently. Quite apart from the fact that output is only half the problem.

Don't talk about it like it has arrived. It hasn't.

[grondilu]

Ya know it doesn't exist right?

All HMD systems suck. Oculus Rift will suck differently. Quite apart from the fact that output is only half the problem.

Don't talk about it like it has arrived. It hasn't.

It has been tried by thousands of people already (lots of them being regular dudes).   I actually don't remember having heard anyone saying anything like "it sucks".

[QuantumG]

Huh? Every single person has said "yeah, it's great, are you working on better resolution?"

Not to mention the fact that first person shooter geeks are not "regular dudes".

[grondilu]

Huh? Every single person has said "yeah, it's great, are you working on better resolution?"

Yes, it needs a better resolution.   And positional tracking.  And the user interface needs to be carefully designed.   It's still in development.

This is the "advanced concept" section.

Not to mention the fact that first person shooter geeks are not "regular dudes".

First person shooter geeks were the people who bought it.   But then they had it tried by all kinds of people.

[QuantumG]

Who cares what Grandma thinks.

If it's the "advanced concepts" section, why are you talking about stuff from the 80s?

Every time VR is hot we hear the same nonsense. It's not magic, it's just a crappy screen really close to your face.

[grondilu]

Who cares what Grandma thinks.

I showed you the granma only to illustrate the point I was making that not just FPS geeks have tried the Oculus rift.

If it's the "advanced concepts" section, why are you talking about stuff from the 80s?

Because you did first, when you mentioned discussions you had twenty years ago or something?

Every time VR is hot we hear the same nonsense. It's not magic,

Nobody said it was magic.  You're imagining things.  You seem to have difficulties separating this actual thread from what you thought about VR from your past experiences and discussions.

it's just a crappy screen really close to your face.

You know the lenses focus the image at infinite, right?

[sanman]

Well, just imagine that you're actually on Mars - you could still make tremendous use of telepresence technologies (I wouldn't necessarily call them "virtual reality", since the display could be showing a live video feed)

Instead of always having people don spacesuits to leave their habitat, they could put on a pair of VR goggles to pilot a robot that will do the hard work outside for them. That minimizes the risks, and yet still allows a personal hands-on approach to doing the tasks at hand. The robot would always be outside, ready to be activated and piloted.

That is my view, Short range teleoperation will probably replace spacesuits entirely at some point. The list of advantages is huge. Its not clear when this will happen but I personally expect this technology to make leaps and bounds. Teleoperation can potentially surpass human dexterity, whereas a human in a suit can only approach it.

When I think about humans settling and colonizing Mars, there's obviously going to have to be huge amounts of construction, mining, etc done. If you imagine humans in spacesuits doing all that work in the unforgiving Martian environment, then it implies a huge amount of risk. Musk previously made comparisons to settling the American Frontier, and admitted that people would die, etc.

By keeping the humans indoors, it keeps them safer. They'd then be piloting their outdoor robots from the safety of the indoor sheltered environment, and we can then go at mining and construction full-tilt without worrying about all those on-the-job accidents and deaths, malfunctioning spacesuits, etc, etc.

The robot-driven mining and construction industries would be able to expand human living spaces on Mars, while minimizing the suffering and anguish. Human pilots all happily living indoors while operating all sorts of robots to expanding the envelope of human living space would then be free to transform the naked Martian surface into the human-inhabited environments that give us the room to freely live out our lives in.

The more sophisticated and capable our tele-operated robot fleet becomes, then the greater the scope and ambition of our mining, construction operations and overall transformation of the place. This forcing function continues and eventually culminates in our terraforming the planet, in the inestimably far, far, distant future.

[michaelwy]

Here is a recent article from arstechnica:

How gaming tech is making for better interplanetary exploration

"My dream in this area is that, someday, when we put human boots on the surface of Mars, I want there to be millions of people in attendance for that event," Jeff Norris, Mission Operations lead at NASA's Jet Propulsion Laboratory told Ars in a recent interview. "I want them not just sitting in their living room watching a television screen; I want them standing on Mars in their own holodecks right there beside the astronauts."

I think this is not just a dream.   It's what is going to happen.  In case you don't know, Virtual Reality is coming.  The technology is now here, and is currently demonstrated with thousands of units of the Oculus rift prototype, which has been sold all around the world, and whose consumer, final version should be available next year.


By the way, the Oculus rift is not unknown in the space industry.  It was mentioned by Elon Musk on the SpaceX channel, for instance.  They use it for CAD experimentations:


Next month, the Gaia mission will be launched.  After a few years, it will have given us a map of a billion stars and other celestial bodies.   3D programs  like Celestia or spaceengine, once adapted to VR, will make it possible for the regular Joe to explore the galaxy as never before.


Space is a terrible place.  It's dangerous, and just staying alive there is extremely costly.   Yet we want to know and see what's up there, which is quite a natural consequence of our human curiosity.  But , do you think we have to physically be in space in order to satisfy this curiosity?


Moreover, when an astronaut is in space, he already looks around him through a helmet.  He doesn't touch anything with his own skin.  Were he standing on mars, he would not breathe martian air, nor would he feel martian wind.   So, there is already quite a thick layer of technology between his senses and the place he explores.    Is it so much different with VR?

The holy grail is the holodeck from Star Trek Next Generation. The trouble with a holodeck in space is the lack of gravity for the omnidirectional treadmill. The combination of oculus rift and the Virtuix Omni treadmill may bring, however, a virtual reality capability into many homes soon. Transmissions from space expeditions may be broadcast to these, for example from a mars rover. 3d television may also be exploited.
Also, google recently stated that the technology of holograms is just around the corner. That means that by the time a mars landing comes underway in 2030, holograms may be broadcast to many homes allowing a level of immersion not yet previousy attained.

[Hop_David]

That is my view, Short range teleoperation will probably replace spacesuits entirely at some point. The list of advantages is huge. Its not clear when this will happen but I personally expect this technology to make leaps and bounds. Teleoperation can potentially surpass human dexterity, whereas a human in a suit can only approach it.

In someways robots are already improving on human dexterity.

The Da Vinci surgical robot has binocular cameras that give depth perception. Dr. Davis, a surgeon who uses the robot, says the telepresence is so immersive he sometimes forgets he's not inside the patient's body cavity.

The robotic hands are operated by motion capture. The motion capture records wrist motion as well as index finger and thumb motion. The robot hands are primitive, they're more like a crab's pinchers than a human hand.

Where they beat human dexterity is their very fine control. Davis says as he toggles the camera zoom, the hand's motion becomes more minute. He showed a video where peeling a grape was more like peeling a watermelon.

[dchill]

The holy grail is the holodeck from Star Trek Next Generation...

The LM CHIL that shows up around 43 seconds into this video is much more like a holodeck: 

www.youtube.com/embed/perMGY0oajs

The system tracks exactly where the user's eyeballs are, so they can walk around a virtual spacecraft and see if from different perspectives.  It's all at human scale, so it's used to pathfind assembly clearances, etc.  It's already been used for a couple years on JSF and GPS-III, to eliminate potential problems before production, and it's being used on Orion and most other LM-SSC programs now too.  I've been in it for a couple different demos and it's definitely cool.

You can google Lockheed CHIL to see more videos and info about that facility, but the video I linked puts it into more of an overall context, since just looking at something isn't even half the benefit...

[cordwainer]

Life support while not hard does require a great deal of extra mass to be put into space and a good deal of reliability to maintain human life. ISRU requires a great deal of labor as well as an initial investment in infrastructure sent into space to make it happen. There is little profit incentive for ISRU and beyond tourism and scientific curiosity there is not much point in sending humans into space. Telepresence and more advanced robotics may give us the tools to make ISRU, other space operations and launch infrastructure cheap enough to make manned spaceflights more feasible in the far future. Bad sci-fi is sci-fi that does not adequately explain human's presence or reason for being in space in the first place.

[grondilu]

beyond tourism and scientific curiosity there is not much point in sending humans into space

You can restrict it to tourism, imho.   Scientific curiosity can be fulfilled with telepresence or robotics if advanced enough.  But whatever the level of realism you can put in a mars-simulation VR system, there will always be rich people that will want the real deal and will be willing to pay a lot for that.

[grondilu]

An article in engadget.com.

Time Machines: NASA goes virtual at CES

It's mostly about the past, but around the end there is this:

« Although public interest in VR cooled off, space was still the perfect deployment arena, and NASA's virtual reality research continued. Its 1997 Robonaut project focused on creating a humanoid robot to perform tasks in place of human astronauts, either autonomously or controlled by a virtual interface. By 2011, the project reached its second stage of development and NASA deployed Robonaut 2 for duty on the International Space Station. The technology used to interact with Robonaut has drastically improved since VIVED was developed, with interface devices like Sensics' piSight display providing up to 6 million pixels per eye (with tiled optics) along with a panoramic field of view up to 166 degrees. »

[Hop_David]

Here is a recent article from arstechnica:

How gaming tech is making for better interplanetary exploration

"My dream in this area is that, someday, when we put human boots on the surface of Mars, I want there to be millions of people in attendance for that event," Jeff Norris, Mission Operations lead at NASA's Jet Propulsion Laboratory told Ars in a recent interview. "I want them not just sitting in their living room watching a television screen; I want them standing on Mars in their own holodecks right there beside the astronauts."

I think this is not just a dream.   It's what is going to happen.  In case you don't know, Virtual Reality is coming.  The technology is now here, and is currently demonstrated with thousands of units of the Oculus rift prototype, which has been sold all around the world, and whose consumer, final version should be available next year.

Not only is telepresence improving but also robotics. Besides being better able to perceive remote environments, we are growing in our ability to interact with and change remote environments.

Miniaturization of electronics was a trend picking up speed In the late 50's and early 60's. NASA and the military invested heavily in this trend. I believe that's why the U.S. led the electronics revolution. If not for their investment, Moore's law might have been known as Deng's Law.

Now we're at the cusp of a robotics revolution. The military is investing in this, DARPA is playing a part. But NASA? Robonaut and other robotic development is a shrinking piece of the pie while BFRs (big ******* rockets) continue to hog funding.

I was disappointed that the Ars article touted VR as a way to experience Mars. The moon and low lunar orbit are 200 to 700 times closer, so the signal strength is 40000 to 490000 times stronger. Much better bandwidth is possible. Also light lag latency is 3 seconds vs tens of minutes. I am hoping to see telerobots on the moon and on asteroids in lunar orbit.

[grondilu]

NASA JPL creates a more immersive way to control a space robot with the Oculus Rift and the Kinect 2

[grondilu]

Laser Link to Moon Trumped NASA and MIT Engineers’ Expectations

« In October of last year, a team from NASA and MIT’s Lincoln Laboratory made space communications history by beaming data, via laser, at speeds reaching 622 megabits per second, to Earth from a spacecraft orbiting the moon. »

This means that a robot on the surface of mars could transmit (presumably also receive) data from an orbiting spacecraft at very decent bitrate.   That's quite significant for the prospect of a VR-based teleoperated mission.

Just checked:  the aerostationary orbit is about 17,000 km above mars's surface.

That would put a delay of 56ms.  A bit too much for comfortable VR (it is believed that delays become acceptable below 20ms), but usable I guess.

[KelvinZero]

aerostationary orbit is about 17,000 km above mars's surface.

That would put a delay of 56ms.  A bit too much for comfortable VR (it is believed that delays become acceptable below 20ms), but usable I guess.

Orbit closer, take frequent coffee breaks..

[gosnold]

20 ms is motion to photon latency for looking around, if your rover/lander has a panoramic set of cameras, it can record and transmit a 360° view to the orbiter, and then the latency is only determined by the computer you put in the orbiter. If you want to interact with your environment however (with robotic arms for instance), the communication lag becomes an issue but a higher latency (>20ms) could be acceptable since it will probably not make you as sick as visual latency.

[grondilu]

20 ms is motion to photon latency for looking around, if your rover/lander has a panoramic set of cameras, it can record and transmit a 360° view to the orbiter, and then the latency is only determined by the computer you put in the orbiter. If you want to interact with your environment however (with robotic arms for instance), the communication lag becomes an issue but a higher latency (>20ms) could be acceptable since it will probably not make you as sick as visual latency.

Yes it's probably possible to do tricks like this.  But instead of a record of a panoramic view, I'd rather imagine that a computer would create a 3D model of the scene and display this instead of a real footage.  To some degree it could also simulate the physics of your behavior so that you can have a near-zero latency feedback.  The real data from the surface would only be used to correct the VR environment whenever the computer simulation was inaccurate.

For instance, if you want to see what's below a rock, you could lift the rock in the VR environment, the computer would send the corresponding command to the surface, and while it waits for the feedback, it would simulate it by guessing what could be below the rock.  You would stare at the rock a few seconds, and during this time the computer would have received the data from the robot on the ground so it could update the picture.  So during half a second or something, you would see nothing interesting but then suddenly the image would be updated as the real data would show up.

[KelvinZero]

Online multiplayer games attempt to deal with network lag with a combination of extrapolation and interpolation.. I guess we have all seen the artifacts of that.

I suppose for teleoperation you would have add a bit of paralysis so movements you thought you were applying to one situation are not suddenly applied to the world as it was a second earlier. I can imagine this being very mentally tiring. You are doing some action like screwing a bolt into a hole and a "waiting" icon pops up, and you have to review the situation to realize you are no longer grasping the bolt or some such. I think I would end up thinking in 2 second chunks: pick up  the bolt, did I really? No. Pick up the bolt. Did I really? put it in the hole, did I really? etc..

[aero]

Online multiplayer games attempt to deal with network lag with a combination of extrapolation and interpolation.. I guess we have all seen the artifacts of that.

I suppose for teleoperation you would have add a bit of paralysis so movements you thought you were applying to one situation are not suddenly applied to the world as it was a second earlier. I can imagine this being very mentally tiring. You are doing some action like screwing a bolt into a hole and a "waiting" icon pops up, and you have to review the situation to realize you are no longer grasping the bolt or some such. I think I would end up thinking in 2 second chunks: pick up  the bolt, did I really? No. Pick up the bolt. Did I really? put it in the hole, did I really? etc..

Hopefully your avatar would have some basic work related preprogrammed trained ability to aid in carrying out rudimentary steps, "grasping at the bolt" --> pick up the indicated bolt and look at it, "shove the bolt toward the hole" --> place the bolt in the hole and look at it, "grasp and twist the bolt --> screw the bolt into the hole and look at it.  Look at it until receipt of the next step directions.

Or likely more. It depends on how autonomous you want the avatar to be.

[grondilu]

You are doing some action like screwing a bolt into a hole and a "waiting" icon pops up, and you have to review the situation to realize you are no longer grasping the bolt or some such. I think I would end up thinking in 2 second chunks: pick up  the bolt, did I really? No. Pick up the bolt. Did I really? put it in the hole, did I really? etc.

VR would be absolutely unusable if there was anything close to one second delay between a movement of your body and a visual feedback.  Again, it is believed that an acceptable delay must be below 20ms.

20ms is physically unfeasible if you include in the pipeline the back-and-forth communication between the avatar on the ground and the computer on the spacecraft.

As I see it, one way to solve this is to simulate all your actions and rely on communication with the ground only in order to improve the VR model as more knowledge is acquired about the martian surface.

So when you screw a bolt, the VR system simulates what happens when you screw a bolt.  The computer does not need much knowledge about Mars in order to do that.  It does instruct the avatar on the ground to screw the real, physical bolt, but it uses the feedback only to check that it is conform to what it has simulated a second before.   It will usually be.  Hopefully the cases when the simulation was not conform would be either rare or slight enough so that corrections do not impend the user interaction too much.

PS.  One reason why this could be possible is that Mars is a fairly static environment.  The ground and atmosphere have very little activity (dust storms happens but they are quite rare and you don't want to do anything outside then, anyway).   Thus the only thing that would move on mars would basically be human artifacts, whose behavior is well known (since we either built or command them) and can thus be relatively easily simulated.  It's not like the surface of mars was populated by lots of living things moving all around in an unpredictable way.

[KelvinZero]

Hopefully your avatar would have some basic work related preprogrammed trained ability to aid in carrying out rudimentary steps, "grasping at the bolt" --> pick up the indicated bolt and look at it, "shove the bolt toward the hole" --> place the bolt in the hole and look at it, "grasp and twist the bolt --> screw the bolt into the hole and look at it.  Look at it until receipt of the next step directions.

Or likely more. It depends on how autonomous you want the avatar to be.

We are getting there, but I think basic agility/dexterity have proven to be very difficult problems. Look how slowly we have progressed on things like walking robots and object recognition, and we have had many reasons to bash our heads against these repeatedly. Operating a remote avatar with significant fraction of a second's lag might only be a critical problem for space.. and perhaps online gamers living in NZ

[KelvinZero]

You are doing some action like screwing a bolt into a hole and a "waiting" icon pops up, and you have to review the situation to realize you are no longer grasping the bolt or some such. I think I would end up thinking in 2 second chunks: pick up  the bolt, did I really? No. Pick up the bolt. Did I really? put it in the hole, did I really? etc.

VR would be absolutely unusable if there was anything close to one second delay between a movement of your body and a visual feedback.  Again, it is believed that an acceptable delay must be below 20ms.

Yes thats understood, and thats what I was describing. Im assuming instantaneous feedback because you are operating on a virtual environment, however Im also assuming that robot on mars will frequently mess up. Even humans repeatedly fumble when doing simple tasks like picking up a bolt, we dont even think about it and often begin retrying mid-fumble. The mars robot will probably fumble even more often, diverging from the virtual simulation, and retrying cannot begin until real information is returned to the human operator. (or as Aero mentions, if whatever is on Mars has some high degree of autonomy where it can invent new strageties to keep the virtual and mars environments in sync)

[grondilu]

Yes thats understood, and thats what I was describing. Im assuming instantaneous feedback because you are operating on a virtual environment, however Im also assuming that robot on mars will frequently mess up. Even humans repeatedly fumble when doing simple tasks like picking up a bolt, we dont even think about it and often begin retrying mid-fumble.

Well, teleoperation can only be as good as robotic avatars are, that's for sure.

Also, possibly a very powerful computer would be able to predict the avatar's failures and include them in the simulation.  After all the VR model must include not only mars's features, but also the robotic avatar itself.

[grondilu]

The considerations above made me think of a VSauce video:



In a nutshell the point is that the brain only process sensory data with a 80 ms delay already.  Yet we don't see that, and it's probably because the brain does lots of tricks that could be analogous to what would be done by a VR system for teleoperation.

[Hop_David]

Laser Link to Moon Trumped NASA and MIT Engineers’ Expectations

« In October of last year, a team from NASA and MIT’s Lincoln Laboratory made space communications history by beaming data, via laser, at speeds reaching 622 megabits per second, to Earth from a spacecraft orbiting the moon. »

This means that a robot on the surface of mars could transmit (presumably also receive) data from an orbiting spacecraft at very decent bitrate.   That's quite significant for the prospect of a VR-based teleoperated mission.

Just checked:  the aerostationary orbit is about 17,000 km above mars's surface.

That would put a delay of 56ms.  A bit too much for comfortable VR (it is believed that delays become acceptable below 20ms), but usable I guess.

Eh? 2 * (17,000 km) / (300,000 km/s) = .11 seconds

You would want the tele operators to have adequate radiation shielding. The easiest way to do this bury them on a Mars moon. But Deimos is even further than aerostationary. Light lag for Phobos to Mars surface and back would be .04 seconds, about twice what you say is acceptable.

To get light round trip down to .02 seconds, you would need an orbital altitude of about 3000 km. Which would have a period of 4.3 hours. To keep constant line of sight with a mars surface worksite, you'd need a constellation of sats.

For the moon, a 3000 km altitude orbit would have a period of 8 hours. Since the moon has a smaller radius, a sat would enjoy line of sight for a greater fraction of the orbit. Still, it'd require a constellation for the moon also.

[grondilu]

Eh? 2 * (17,000 km) / (300,000 km/s) = .11 seconds

Yeah I forgot the return time.

To keep constant line of sight with a mars surface worksite, you'd need a constellation of sats.

That would not do since it would increase the light path.

[KelvinZero]

You would want the tele operators to have adequate radiation shielding. The easiest way to do this bury them on a Mars moon. But Deimos is even further than aerostationary. Light lag for Phobos to Mars surface and back would be .04 seconds, about twice what you say is acceptable.

It probably is the easiest way to get huge shielding but it would also introduce other issues that we currently have no experience with. floating grit, tiny but not ignorable gravity, whatever operations are involved in such digging so far from home..

My guess is version 1.0 would be as ISS-like as possible. Stick near as possible to mars and I guess that would roughly halve cosmic radiation, similar to ISS in LEO? Perhaps additional shielding could be seen as similar to the asteroid capture mission: grab a sealed bag full of regolith robotically. Or perhaps the ISS-like base is firstly used to explore Deimos (without landing) and later it is moved to a low mars orbit for teleoperation. I think Deimos ISRU is meant to be more promising in any case.

[Hop_David]

You would want the tele operators to have adequate radiation shielding. The easiest way to do this bury them on a Mars moon. But Deimos is even further than aerostationary. Light lag for Phobos to Mars surface and back would be .04 seconds, about twice what you say is acceptable.

It probably is the easiest way to get huge shielding but it would also introduce other issues that we currently have no experience with. floating grit, tiny but not ignorable gravity, whatever operations are involved in such digging so far from home..

My guess is version 1.0 would be as ISS-like as possible. Stick near as possible to mars and I guess that would roughly halve cosmic radiation, similar to ISS in LEO? Perhaps additional shielding could be seen as similar to the asteroid capture mission: grab a sealed bag full of regolith robotically. Or perhaps the ISS-like base is firstly used to explore Deimos (without landing) and later it is moved to a low mars orbit for teleoperation. I think Deimos ISRU is meant to be more promising in any case.

I think the floating grit would be an issue, yes. Seeing how there's no jostling of dust particles by wind or water, I would expect Phobos dust to be very abrasive, like our lunar dust.

I still fantasize about a base on Stickney crater. The crater walls would afford a lot of protection. As would Mars, it takes up a big portion of the sky. If I remember right, Buzz Aldrin was advocating bulding Mars infra-structure using tele-operators on Phobos. I'll see if I can dig up his recent book.

Grondilu, in that NOW video they mentioned how subjects becoming accustomed to an 80 millisecond delay had the illusion of the light coming on before they pressed the button when the delay was shortened. This leads me to wonder: Can people grow accustomed to a longer latency than 20 ms? The necessity of keeping teleoperators within 3000 km of the worksite sure eliminates a lot of options.

[RonM]

I think we have an issue between frame rate, time delay, and good enough. 20 ms or better would give you visual perfection for VR, something that is not required to teleoperate a robot. Now, does this 20 ms number refer to frame rate or time delay. You can still have a high frame rate and a high time delay.

Movies are 24 fps, about 40 ms, and while a quick pan looks choppy, it's fine for the average person. Slow pans and stationary views look great. 20 ms is 50 fps, but once again, what does that have to do with time delay?

Typical human reaction time is 100 ms. That's for instinctive reactions and anticipated events (wait for it . . .). Unexpected events where you have to make a snap decision, say while driving a car and you have to avoid a hazard that just appeared, take about 300 ms for the average person.

As long as the robot doesn't move quickly, a delay of under a second will be acceptable.

The US operates combat drones halfway around the world from bases in the US. Using satellites, that's at least a 250 ms delay, assuming using only one satellite. 500 ms if you include the round trip.

[grondilu]

The 20ms refer to what at Oculus they call the "motion to photon" delay.  That is the time between the moment you move your body (typically your neck) and the moment your eyes receives the photons that result from this change of position of your body.

The time your brain needs to react to an external stimulus has very little to do with this : it's a very different issue.  Your organism is not prepared at all to withstand any substantial delay between your movements and the resulting visual input, because it is always pretty much zero IRL.  When such a delay happens, the mismatch between your proprioception, your vestibular system and your eyes can create motion sickness, which is powerful enough to make you through up.

[gosnold]

20ms is the time delay Oculus wants for for vision, the framerate they target is around 90Hz. Now for proprioception you might be able to get away with a higher time delay, this remains to be investigated.
If even for proprioception the maximum acceptable time delay is 20ms, you have a maximum range of 3000km, so if you orbit low around mars (with an orbit having a 19 000 km length passing over where your teleoperated robot is), you can use it 30% of the time (3000km before passing over, 3000km after). This requires a constellation of data relay satellites in low orbit with a high datarate (something like 32 bit color 90Hz*720p per eye, around 5Gbps uncompressed).

[RonM]

Then it sounds like VR is not the solution for operating robots from orbit. More conventional controls would work well.

VR would be a great solution from a base or ground vehicle. VR control of a nearby robots would mean astronauts would not have to suit up. They could have the same experience as being in a spacesuit, but from the comfort and safety of their habitat.

[aero]

It sounds to me like we are talking about the delay in receiving a visual when you glance over your shoulder? If the view is not there until after you look, then it causes motion sickness. Is that right?

If so, what is to prevent you from seeing the view over your shoulder from the same time-frame as the front view? That is, the avatar visual sensors capture and transmit a 360 degree view continuously and you look at what ever section of the synchronized visual information you are interested in. It will all be delayed by some large number of ms, but so what?

It may take more bandwidth but I must ask, " Is that more expensive than a constellation of low orbiting relay satellites with a high data rate?"

[gosnold]

It sounds to me like we are talking about the delay in receiving a visual when you glance over your shoulder? If the view is not there until after you look, then it causes motion sickness. Is that right?

If so, what is to prevent you from seeing the view over your shoulder from the same time-frame as the front view? That is, the avatar visual sensors capture and transmit a 360 degree view continuously and you look at what ever section of the synchronized visual information you are interested in. It will all be delayed by some large number of ms, but so what?

It may take more bandwidth but I must ask, " Is that more expensive than a constellation of low orbiting relay satellites with a high data rate?"

That is what I pointed out in post 41, but if your move your arm and see the robotic arm you control move much later, it can also be disturbing.
If you accept a 70ms delay  you can control your robot very simply through motion capture from anywhere in low Mars orbit with a constellation of low orbiting relay satellites. You would need to stream the whole 360° field of view though, so you would have something like 4 times more data to stream.
If you accept 120ms delay you can do the same from a Mars synchronous orbit just above your robot.

[aero]

Train yourself to deliberately blink your eyes when you turn your head. And other times as needed.

[grondilu]

I think we need to clarify a few things, here.  Most notably:  VR is not teleoperation.  Both concepts provide a similar experience to the user, but they are yet very different.

In VR, the events resulting from the user's commands are determined by a computer model.

In teleoperation, the events resulting from the user's commands are determined by the corresponding actions of the physical avatar in the environment.

As a consequence, excessive communication delays make teleoperation impossible or at least impractical.  But as I tried to explain earlier in this thread, this is not the case for VR.   VR *can* be used to explore an other planet, regardless of the communication delays, because the presence of robots on the surface is only needed to improve the computer model.

[grondilu]

Now that I think about it, the distinction I made above between VR and teleoperation brings up an interesting question:  is data acquisition necessarily part of exploration?

I mean, imagine we manage to communicate with an alien civilization and they give us a very accurate and complete model of the whole galaxy.  The exact position and chemical composition of any small rock on any planet or moon would be enclosed in this data.   That would represent a huge amount of data but at least conceptually it would not be impossible to store and process such a database.  Would that mean that humanity would be done with exploring the galaxy?  I don't think so.  It would just mean that the exploration of the galaxy by mankind would consist in exploring the "alien catalog", and this would probably most conveniently be done by using a VR environment based on this database.

Is that completely far fetched?   Well, think about what Gaia is and what it does.  It will generate petabytes of data, so much that once this database is available, more time will probably be consecrated to analyse it than it was needed to acquire it.   And lots of people (not scientists though, but does it really matter?) will "explore" the Gaia catalog with 3D astronomical software such as spaceengine, whose support of VR is currently being worked on:



So yeah, I think it's not absurd to think that VR is the future of space exploration, because data acquisition will not be done by human bodies.

[michaelwy]

When i think about virtual reality and space, the holodeck in star trek immediately springs to mind. Some will be achived by the combination of display technology and treadmills. However, i think the great leap comes with holograms. Hopefully, we will soon be able to create life size holograms of humans and endow these with artificial intelligence. Then you could have virtual concerts, virtual plays and holographic discussons with a computer on any given subject.

[Adaptation]

Vr headset, on a robot arm with a 3d treadmill, the future of emersion.

[grondilu]

This is something I often wondered.  How convincingly can you reproduce or at least simulate microgravity or any arbitrary acceleration field with a controlled trajectory in a constant gravitational field?

I guess it depends of the available space of your device.  After all, if you have an infinite space, you can reproduce microgravity with a free fall.   But in a bounded sphere of any given diameter, how well can it work?

[Adaptation]

This is something I often wondered.  How convincingly can you reproduce or at least simulate microgravity or any arbitrary acceleration field with a controlled trajectory in a constant gravitational field?

I guess it depends of the available space of your device.  After all, if you have an infinite space, you can reproduce microgravity with a free fall.   But in a bounded sphere of any given diameter, how well can it work?

If your small they can do it quite well.  Simply spin you across all three axis evenly, works almost perfectly for one point and pritty good for small things like eggs.  Bigger things like people sized things get too much of an effect from centripetal force.

You can also be placed in neutral buoyancy conditions but this wont fool your inner ear.  Not that that matters much, you trust your eyes more than your ears anyhow.

[Jim]

If your small they can do it quite well.  Simply spin you across all three axis evenly, works almost perfectly for one point and pritty good for small things like eggs. 

wrong.  Free fall is the only way.  Spinning would not do it. It is still in a one g field.  If spinning could do it then there would be a whole industry based on it and there would be no need to fly microgravity experiments.

[Adaptation]

wrong.  Free fall is the only way.  Spinning would not do it. It is still in a one g field.  If spinning could do it then there would be a whole industry based on it and there would be no need to fly microgravity experiments.

meow

There are like tons of these things and loads of papers studying how well they work. 
http://descsite.nl/Publications/Other/RPM-FS-MG-R00-017.pdf
Also a more common thing called rotating wall vesicle. 

They work, just not not perfectly and only for little things.  Stuff also has to either be physically attached or in a fluid, higher viscosity is better.   

[KelvinZero]

...the future of emersion.

Add elastic bands pulling the body down to simulate gravity, and that is just the sort of thing I imagine for making living in a tiny box in zero-g for six months bearable and healthy. I would probably prefer scenic walks over six months of zombie killing though.

How does that treadmill thing work in multiple directions? I thought maybe you could have ball bearings on the shoes with variable friction, but ideally the floor would actually keep moving in any direction. Tilting could also be useful to simulate mild inclines.