← STEAM Lessons Fabrication · Lesson 02 Fabrication

Fabrication · Lesson 02 · Immersive Technology

Virtual Reality
How It Works & How to Build It

Explore the neuroscience behind VR headsets, interact with a live binaural spatial audio simulator, see how stereoscopic rendering tricks your eyes, then write real code to add VR to your own games.

Binaural Audio Stereoscopic Vision Head Tracking Game Dev

From Sci‑Fi to Silicon

Virtual Reality didn't happen overnight. It took over a century of engineering to catch up to the initial concept, moving from Victorian optical tricks to the digital headsets we build with today.

👓 The Optical Illusion (1838)

Wheatstone’s stereoscopy

Sir Charles Wheatstone presented two slightly offset drawings—one to each eye—proving the brain can perceive depth from flat images. This binocular disparity trick is still the optical foundation of every VR display.

📖 The Sci‑Fi Dream (1930s)

Pygmalion’s Spectacles

Stanley Weinbaum’s 1935 story imagined goggles that simulated sight, sound, smell, and touch—describing the blueprint for immersive VR decades before computers could render it.

⚙️ The Mechanical Imitation (1960s)

Sensorama + Sword of Damocles

Morton Heilig’s Sensorama delivered 3D film with sound, wind, and scent. Ivan Sutherland’s ceiling‑mounted Sword of Damocles (1968) became the first true head‑mounted display.

🚀 The Real Thing (2010s–Present)

Modern low‑latency VR

The Oculus Rift prototype (2012) combined high‑resolution phone displays with fast mobile IMUs, proving lightweight, low‑latency VR could finally scale beyond labs and arcades.

Tricking Your Brain into Believing a Fake World

Virtual Reality is a technology that replaces your real sensory environment with a computer-generated one so convincingly that your brain treats it as real. Every VR experience rests on three illusions working simultaneously:

👁️

Stereoscopic Vision

tap to reveal

Stereoscopic Vision

Your eyes sit ~63 mm apart. Each sees a slightly different angle — binocular disparity. VR renders two offset images, one per eye, so your brain extracts the depth difference exactly as it does in the real world.

🔊

Binaural Audio

tap to reveal

Binaural Audio

Sound reaches each ear at a slightly different time (ITD) and volume (ILD). Your brain decodes these sub-millisecond differences — plus HRTF filtering from your ear shape — to place sounds anywhere in 3D space.

🧠

Presence & Immersion

tap to reveal

Presence & Immersion

Presence is the brain's acceptance of a virtual space as physical reality. It needs <20 ms motion-to-photon latency, >90° FOV, and coherent audio-visual coupling. The parietal cortex is the key region fooled.

📡

Head Tracking (6DoF)

tap to reveal

Head Tracking (6DoF)

Six Degrees of Freedom: 3 rotational (pitch, yaw, roll) + 3 translational (X, Y, Z). IMUs handle rotation at 1000+ Hz. Inside-out cameras or external beacons track room-scale position — all fused in real time.

🖥️

Displays & Refresh Rate

tap to reveal

Displays & Refresh Rate

VR needs ≥90 Hz (ideally 120 Hz) — each frame rendered twice in <11 ms. Fresnel lenses warp the image, so the GPU pre-distorts a pincushion frame that the lens corrects back to straight geometry.

🌀

Motion Sickness & Latency

tap to reveal

Motion Sickness & Latency

Visual motion without matching vestibular signals triggers cybersickness — the brain suspects poisoning. Fix: keep latency <20 ms, avoid artificial locomotion, maintain locked frame rates with no stuttering.

Move the Sound Source — Hear the Difference

Drag the sound source dot around the head below (or use the angle slider), then click Play Tone to hear binaural panning in real time. Watch the ear-level meters show how the signal shifts between left and right. This is exactly what a VR headset does with every sound in its world.

45°
Left Ear
Right Ear

Use headphones for the clearest binaural effect

How VR Headsets Render Depth

Below is a live stereoscopic renderer. Two virtual cameras are placed 63 mm apart (default human IPD). Each renders the same 3D scene from its slightly different vantage point. The merged view at the bottom simulates what a VR lens system combines into perceived depth — notice how the parallax shift between left and right eyes creates the illusion of objects floating at different distances.

Left Eye
Right Eye
Merged Stereo View (what the brain sees)
63 mm
What to notice: as you increase IPD, the left and right images diverge further. Very high IPD creates uncomfortable depth cues — this is why real headsets have an IPD adjustment knob (usually 58–72 mm range). Mismatched IPD is one of the leading causes of VR eye strain.

From Flat Screen to Virtual World

Every major game engine supports VR through the OpenXR standard — a unified API so the same code runs on Meta Quest, PlayStation VR2, SteamVR (Valve Index, HTC Vive), and Windows Mixed Reality. Below are real code patterns for three popular engines used by students. Use headphones while developing to test binaural audio.

// Unity — Add VR in 4 steps
// 1. Window → Package Manager → Install "XR Plugin Management" + "OpenXR Plugin"
// 2. Edit → Project Settings → XR Plug-in Management → enable OpenXR
// 3. Add an XR Origin (Action-Based) to your scene from the XR menu
// 4. Attach this script to any AudioSource for binaural positioning

using UnityEngine;
using UnityEngine.XR;

public class VRSetup : MonoBehaviour
{
    // Attach to XR Camera Offset — keeps the player grounded
    void Start()
    {
        XRSettings.eyeTextureResolutionScale = 1.4f;   // super-sample for clarity
        Application.targetFrameRate = 90;             // lock to 90 Hz
    }
}

// Binaural audio — Unity uses HRTF (Head-Related Transfer Function) automatically
// when Spatialize is enabled on an AudioSource
public class BinauralSound : MonoBehaviour
{
    void Start()
    {
        AudioSource src = GetComponent<AudioSource>();
        src.spatialize         = true;   // enable HRTF binaural
        src.spatialBlend      = 1.0f;  // 1 = fully 3D
        src.rolloffMode       = AudioRolloffMode.Logarithmic;
        src.maxDistance       = 20f;
        src.dopplerLevel      = 1f;   // Doppler shift when moving
        src.Play();
    }
}

// Head tracking is handled automatically by XR Origin.
// To read the headset pose manually:
void Update()
{
    InputDevice head = InputDevices.GetDeviceAtXRNode(XRNode.Head);
    head.TryGetFeatureValue(CommonUsages.deviceRotation, out Quaternion rot);
    head.TryGetFeatureValue(CommonUsages.devicePosition,  out Vector3    pos);
    // rot and pos now give you the exact head orientation in world space
}
# Godot 4 — OpenXR VR setup
# 1. Project → Project Settings → XR → Enable OpenXR
# 2. Add XROrigin3D → XRCamera3D → XRController3D (left + right) to your scene
# 3. Attach this script to XROrigin3D

extends XROrigin3D

func _ready() -> void:
    var interface: XRInterface = XRServer.find_interface("OpenXR")
    if interface and interface.is_initialized():
        print("OpenXR initialised")
        get_viewport().use_xr = true
    else:
        push_warning("OpenXR not available — running flat")

# Binaural audio — attach to any AudioStreamPlayer3D in the scene
extends AudioStreamPlayer3D

func _ready():
    # Godot uses its built-in HRTF when the listener is an XRCamera3D
    attenuation_model = AudioStreamPlayer3D.ATTENUATION_INVERSE_DISTANCE
    unit_size        = 1.0
    max_distance     = 20.0
    doppler_tracking = AudioStreamPlayer3D.DOPPLER_TRACKING_PHYSICS_STEP
    play()

# Read head pose manually if needed
func _process(_delta):
    var camera: XRCamera3D = $XRCamera3D
    var head_pos: Vector3    = camera.global_position
    var head_rot: Quaternion = camera.global_transform.basis.get_rotation_quaternion()
    # head_pos and head_rot update every frame from the headset IMU
// Three.js + WebXR — VR in the browser (no install needed for users!)
// Requires HTTPS. Works on Meta Quest Browser, Chrome with WebXR flag.

import * as THREE from 'three';
import { VRButton } from 'three/addons/webxr/VRButton.js';
import { XRControllerModelFactory } from 'three/addons/webxr/XRControllerModelFactory.js';

// ── 1. Renderer ──────────────────────────────────
const renderer = new THREE.WebGLRenderer({ antialias: true });
renderer.xr.enabled = true;   // turn on WebXR
renderer.setPixelRatio(window.devicePixelRatio);
renderer.setSize(window.innerWidth, window.innerHeight);
document.body.appendChild(renderer.domElement);

// ── 2. VR Button ─────────────────────────────────
document.body.appendChild(VRButton.createButton(renderer));

// ── 3. Scene ──────────────────────────────────────
const scene  = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(70, window.innerWidth / window.innerHeight, 0.01, 100);

// ── 4. Binaural Audio (Web Audio API HRTF) ────────
const listener = new THREE.AudioListener();
camera.add(listener);                     // attach to camera = headset ears

const sound = new THREE.PositionalAudio(listener);
const loader = new THREE.AudioLoader();
loader.load('sounds/ambient.mp3', function(buffer) {
    sound.setBuffer(buffer);
    sound.setRefDistance(1);     // full volume within 1 m
    sound.setRolloffFactor(2);
    sound.setLoop(true);
    sound.play();
});

// Attach the sound to any mesh — it will spatialise automatically
const sphere = new THREE.Mesh(
    new THREE.SphereGeometry(0.1),
    new THREE.MeshStandardMaterial({ color: 0x6f4cff })
);
sphere.add(sound);          // sound follows the mesh in 3D space
scene.add(sphere);

// ── 5. Render loop ────────────────────────────────
renderer.setAnimationLoop(function(time) {
    sphere.position.x = Math.sin(time * 0.001) * 2;   // orbit the player
    renderer.render(scene, camera);
});

Breakout — A VR Arcade Classic

Play a polished A-Frame Breakout mini-game inside the lesson. It works on desktop with keyboard or mouse, and in a headset with WebXR controls, so students can connect the VR concepts they just learned to a real interactive scene.

Desktop: Move with ← →, A/D, or by sliding your mouse across the game window. VR: Press Enter VR, then move the paddle with the right thumbstick. The ball launches from the center and speeds up slightly as rallies get longer.

The Science Behind VR

HRTF — Head-Related Transfer Function
A mathematical model of how your ear shape, head, and shoulders filter sound differently depending on direction. All modern VR engines (Unity, Godot, Three.js, Unreal) use HRTF lookup tables to pan audio binaurally. The brain learns to decode these filters from childhood — which is why binaural audio sounds uncannily real when done correctly.
Lens Distortion Pre-Correction
Wide-angle fresnel lenses in VR headsets introduce barrel distortion. The GPU renders a pincushion-distorted frame that the lens then stretches back to a natural-looking image. If you ever see a raw VR frame buffer, it looks like a warped fishbowl — this is intentional and corrects to straight lines through the optic.
Reprojection & ATW
Asynchronous TimeWarp (Meta) and Reprojection (SteamVR) are techniques that take the last rendered frame and warp it to match the headset's latest position — inserting a "fake" frame if the GPU misses its deadline. This keeps the experience smooth even if frame rate dips, at the cost of slight blur around edges of moving objects.

Quick Quiz

1. What does ITD stand for in binaural audio?

2. What refresh rate is considered the minimum for comfortable VR?

3. Which open standard lets the same VR code run on Meta Quest, PlayStation VR2, and SteamVR?

4. What is the average human IPD (inter-pupillary distance)?