3D as the New Modality

The third dimension is coming to computing.

• Capture 3D natively. Spatial video can be captured on the iPhone 15. Higher quality video and photos that can map to 3D settings.

• From 2D to 3D. Techniques like Gaussian Splatting, Photogrammetry, Neural Radiance Fields (NeRF) predict complex 3D scenes from 2D objects.

• Display 3D easily on 2D devices. Rendering and interacting with 3D objects in the browser has become easier and easier over the years. There’s three.js, which has been around since 2010 alongside WebGL, but continues to grow in popularity with things like WebGPU and bindings for modern application frameworks like React.

• New Devices for 3D. There’s the Meta Quest headsets and the upcoming Apple Vision headset. Mark Zuckerberg’s conversation with Lex Friedman happened with photorealistic 3D avatars and spatial audio (video, the results are impressive). The new Apple Watch has a gesture where you can “Double Tap” with a pinch gesture in mid-air. Not quite augmented reality but a precursor to the gestures we might use to interact with it.

There’s clearly a convergence of technology (some accelerated by companies like Meta and Apple, but others occurring independently). It’s not entirely obvious to me how it manifests as consumer or enterprise products. There will be applications that are near-direct ports from their 2D equivalents – video meetings will shift to 3D meetings (2D video will become the new phone call). Individual screens might be replaced with more mobile headsets (e.g., in offices). Consumer entertainment will become more immersive (e.g., we might scroll a version of TikTok or YouTube with 3D video, even on 2D devices). But there’s a set of new experiences that are unique to the 3D modality that haven’t been explored yet.

Of course, it could fail to reach a critical activation threshold – the expressiveness is not good enough to make us switch to a new modality. The review on spatial audio is mixed to positive – it’s not a selling point for most consumers.