Toward webXR (POOS 2017)

Toward WebXR
Thomas Bourgeau
CTO BlumenLab

What are we talking about ??
What is WebXR : From real to « yet »
undefined Reality
From 360° to Augmented reality :
−Devices
−Experiences
−Interactions
Can we anticipate:
−Technologies
−Experiences
−Interactions
Session UX/UI - Toward WebXR
thomas@blumenlab.com

What is XR ?
Superset of various types of “Reality”, with “X”
as the variable.
Milgram’s Reality-Virtuality Continuum
− Tangible User Interface (TUI) : Interaction with real objects.
− Augmented Reality (AR) : Adding information on the real worl.
− Augmented Virtuality (AV) : Real information to computer environment.
− Virtual Reality (VR) : Immersion in complete computer environment.
R
X

What is XR ?
Reality not limited to one sense :
− Sound (Binaural, ambisonic, etc.)
− Touch (haptic, proprioception, etc.)
− Smell (Odor synthesizers, etc.)
− Etc.
We are just at the beginning !!
− Technology is becoming mature
− Network and Cloud offers computing
− Almost XReady!!

What is the Web ?
The most accessible medium worldwide
−Multi-Canal, Multi-Support, Multi-Content, etc.
−Built on top of the Internet (IP-Network)

What is the Web ?
Built on open standards:
−Allows Interoperability among devices.
−Specification are open (W3C), only browsers have
to integrate them.

What should be webXR ?
Open Standard:
−New form of “Reality” should be accessible:
●For consumers: Through web interfaces and APIs.
●For developers: Through web languages and
specifications (Next HTML X, Javascript releases).
Interoperability:
−Cross devices deployment
−Avoiding One SDK = One App = One experience
Scalability:
−XR should be accessible to everybody worldwide.
−XR should be integrated in most browsers.

Panoramic and 360°
Expose space at 360°
−Can turn around an object.
−Can look around an environment.
−Can interact with elements (click/keyboard)
Interesting for experiences like:
−Virtual tour/visit.
−Visualization of an object
Not really interactive!

Web360° : Requirements
No need of a special browser.
Addition of simple lightweight javascript library
Panoramic pictures:
−Need a special camera and/or a “stitching”
framework to create a 360 content.
−Picture exposure can be tricky (360°)
−Pictures may have different format and resolution:
equirectangular, cubic, cylindrical, etc.

Web360° : Experiences

Web360° : Interaction
Click/Keyboard: Select objects/div, explore
360°
Mobile Swipe: Let turn around a 360 scene.
Mostly based on captured images with overlay
Hotspots: Add interactivity in the 360° page:
○ Launch new panoramic/hyperlinks
○ Select feature to interact with the content.
○ Access to additional information (expanding
a div)

Web360° : Open Source
Javascript based:
−Marzipano: http://www.marzipano.net
−Reel: http://www.reel360.org
−Leanorama: https://github.com/leandigo/leanorama
−
Panoramic Image manipulation:
−Panotools: https://wiki.panotools.org
−Hugin: http://hugin.sourceforge.net/
−Gimp: https://www.gimp.org

3D
3D elements are extensively used in:
○Game/Special effects/Representation
3D modeling is quite expensive
(time/knowledge/computation):
○Rendering/Meshes/polygons.
○Needs computation power to render.
When 3D environment/Characters done:
○Easy to change and adapt.
○Can move the “virtual camera”.
○Physical engines to simulate real environment.

Web3D: Requirements
Today, WebGL allows to render 2D and 3D
objects in the browser.
WebGL allows to use OpenGL ES 2.0 API.
−Inside an HTML <canvas> element
Most modern browser including in smartphone
have webGL enabled.
Will use the GPU of graphical cards.

Web3D : Experiences

Web3D : Interactivity
Mouse/keyboard: Easy desktop control
Joypad/Gamepad: Propose extended
functionalities and buttons.
Other devices: Myo, Magic Leap, etc.
Ability to enter in the 3D scene:
−Create 3D world that you can explore.
−Physic engine to render gravity, physical
boundaries, etc.
Embedding real time content (webRTC)

Web3D : Open Source
Javascript 3D Libraries:
− three.js: https://threejs.org/
− babylon.js: https://www.babylonjs.com/
Creating 3D assets:
− https://www.blender.org/
− https://3dc.io/
Share 3D content:
− Sketchfab: https://sketchfab.com/
− Archive3D: https://archive3d.net/
WebGL: Khronos: https://www.khronos.org/

Virtual Reality (VR)
Immersive medium:
−Need a special headset to enter in VR.
Technology:
−Not new: Based on stereoscopic display.
−New: Gyroscope, displays, smartphone.
Can look all around inside the immersive
content: stereoscopic images/video/3D
−Stereoscopic video/images/3D.

Virtual Reality (VR)
Image/video: Equirectangular projection Stereoscopic view in headset
VR views and projections

VR : Devices
A large choice of devices/headsets:
−Fully integrated: Oculus, HTC Vive,
−With smart phones: Google Cardboard, daydream,
Samsung Gear VR.
Google: Daydream and cardboard

WebVR : Devices
Each headset has it’s own SDK/Store/Feature
Need a high level interface for developers:
−WebVR.

WebVR : Experience

WebVR : Interaction
Through the eyes:
− Gaze: An element is activated when fixing it during a certain
duration.
− Hover: Same as with mouse, can activate element when
looking over.
Through standard controllers:
− Buttons, remotes, gamepads, etc
− Activate an action when pressed.
New haptic controllers:
− Gesture: Myo - https://www.myo.com/
− Virtual hands: Leap Motion - https://www.leapmotion.com/

WebVR : open source
Need to have a compliant browser:
−See available browser: https://webvr.rocks/
−A-frame: https://aframe.io/

Mixed Reality (MR)
Immersive and “experiencing” environment.
Immersion with the addition of:
○Tracking movements in space.
○Evolve, manipulate, Explore the
environment.
Additional feature can be added:

WebMR : Devices

WebMR : Experience

WebMR : Interaction
Traditional controllers:
○Remotes, gamepads, wireless pads.
Need additional hardware for interaction:
○Gesture Space sensors
○Depth camera/sensors
Can be mix with other evolved controllers:
○Gesture (Myo), LeapMotion (handtracking).
○Specific installation (Birdy simulator, etc.)

WebMR : Open source
Basically the same requirements than for
webVR:
−See available browser: https://webvr.rocks/
−A-frame: https://aframe.io/
−BabylonJS: https://www.babylonjs.com/

Augmented Reality (AR)
Add a layer on top of what you see:
○Can be a 3D assets/scene, a picture,
animation, etc.
Need to see through a device:
○Can be a very specific hardware
○Commodity hardware for light experience
AR can be applyed to other senses:
○Mainly Sound and connected objects.

Marker Based (AR)
Detection of a pretrained marker

Markerless (AR)
Anchor point detection Plane detection

webAR Devices.
Microsoft Hololens
(no web support yet:
HoloJS)
Google Project Tango
ARCore
iPhone 11
ARKit

WebAR : Experiences

WebAR : Interactivity
Need a camera to see the environment
○ Can change the projected image.
Manipulation of “superposed” objects:
○ Swipe, Hover, Finger, etc.
Need a device to communicate:
○ IoT devices, sensors, actuators, etc.
Other Augmented senses to be added.

WebAR : Frameworks
Basically javascript framework adaptation:
−ArToolkit - Marker based: https://webxr.io/
−ArCore: https://github.com/google-ar/
−WebARonARKit: https://github.com/google-ar/WebARonARKit
−threejs-ar: https://github.com/google-ar/three.ar.js
−A-frame ar: https://github.com/autonome/aframe-ar

WebXR : UI/UX ?
Can we anticipate next Interfaces and usage:
○Holograms
○Tactile interfaces
○Smell interfaces
○Speech control, etc.
Can we merge existing interfaces with new one?
Can we integrate it with the Web?
What are going to be ne next button?
Is it the end of smart phones?

Toward webXR (POOS 2017)

More Related Content

Similar to Toward webXR (POOS 2017)

Recently uploaded

Toward webXR (POOS 2017)