Learn Creative Coding (#111) - Creative Projection Mapping

Last time we made a plain grey box glow. We warped a sketch onto its faces with corner pinning, dragged the corners into place in a dark room, and got that little jolt of joy when the projection wrapped around a real object and made it look alive. But here's the thing that nagged at me for days after I first pulled that off: it was a monologue. The box performed, and I watched. Beautiful, sure - but it didn't care that I was there. Wave at it, walk past it, put your hand on it: nothing. It just kept doing its thing to an empty room.

Today we fix that. We're going to turn projection from a display into an installation - something that notices you, reacts to you, and pulls you inside the artwork instead of leaving you standing outside it. This is the leap from "look what my code can do" to "come play with what my code can do", and honestly it's my favourite kind of creative coding there is. Allez, let's teach the light to see.

Display versus installation

Let me pin down the difference, because it's the whole soul of this episode. A projection display is one-way: pixels go out, that's it. A projection installation is a loop: the projector throws light out, a sensor reads the room back in, and your code closes the circle by changing the light based on what it saw. Out, in, respond, out again. Thirty times a second.

// the whole shape of an interactive projection, in one comment:
//
//   PROJECTOR --light--> ROOM (with people in it)
//        ^                        |
//        |                    a SENSOR watches
//        |                        |
//    your CODE <--data-- (webcam / depth cam / mic)
//
//  it's a FEEDBACK LOOP. the room is now an input device.
//  last episode was the top arrow only. today we wire up the rest.

The moment you close that loop, the room becomes an input device. And the cheapest sensor for it is one you almost certainly already own - a webcam. We don't need fancy hardware to get the first real "whoa". We just need to teach the code to spot the difference between "empty room" and "someone's here".

Seeing people with an empty-room trick

I teased this at the very end of last episode: background subtraction. The idea is gorgeously simple. Take one photo of your empty scene and remember it. Then every frame, compare the live camera to that stored empty frame. Anything that differs is something new in the room - a person, a hand, a moved chair. Everything that matches is just... furniture.

First, grab and freeze that empty reference frame. You do this once, when nobody's standing in front of the camera.

// capture the empty room ONCE. call this when the scene is clear.
// we copy the pixels out so the reference doesn't keep updating.
let bg = null;               // our frozen "empty room" snapshot

function captureBackground(cam) {
  cam.loadPixels();
  bg = new Uint8ClampedArray(cam.pixels);   // a copy, not a reference!
}

That new Uint8ClampedArray(cam.pixels) matters - if you just wrote bg = cam.pixels, you'd be pointing at the live buffer and your "empty room" would keep changing every frame, which defeats the whole point. Copy it, freeze it, forget it. Now the comparison.

// compare each live pixel to the frozen background.
// big colour difference = foreground (a person). returns a 0/1 mask.
function foregroundMask(cam, bg, threshold = 45) {
  cam.loadPixels();
  const px = cam.pixels;
  const mask = new Uint8Array(cam.width * cam.height);
  for (let i = 0, p = 0; i < px.length; i += 4, p++) {
    const diff = Math.abs(px[i]   - bg[i])       // red
               + Math.abs(px[i+1] - bg[i+1])     // green
               + Math.abs(px[i+2] - bg[i+2]);    // blue
    mask[p] = diff > threshold ? 1 : 0;          // 1 = "someone here"
  }
  return mask;
}

This is the exact same pixel-reading muscle we built way back in episode 10, when we first learned to poke at raw image data. Nothing new under the hood - a loop over pixels, four bytes at a time, some subtraction. See how much of this whole "physical world" chapter is just old tools pointed somewhere new? The threshold is your one dial: too low and camera noise lights up the whole frame, too high and a person in dark clothes vanishes. You tune it by eye, in the room, on the day.

Cleaning up the mess

Here's the reality nobody warns you about: a raw background-subtraction mask is speckly. Camera sensors are noisy, light flickers, and you'll get lonely stray pixels flickering "person!" all over the empty parts of the frame. If you drive your art straight off that, it looks jittery and cheap. The fix is a classic bit of image processing called erosion - throw away any "person" pixel that doesn't have enough "person" neighbours. Lonely pixels are almost always noise; real bodies come in solid clumps.

// erosion: keep a foreground pixel only if most of its neighbours agree.
// kills the lonely speckle of camera noise, keeps solid body blobs.
function erode(mask, w, h, need = 5) {
  const out = new Uint8Array(mask.length);
  for (let y = 1; y < h - 1; y++) {
    for (let x = 1; x < w - 1; x++) {
      const i = y * w + x;
      if (!mask[i]) continue;
      let n = 0;
      for (let dy = -1; dy <= 1; dy++)
        for (let dx = -1; dx <= 1; dx++)
          n += mask[(y + dy) * w + (x + dx)];   // count nearby friends
      out[i] = n >= need ? 1 : 0;               // lonely? drop it.
    }
  }
  return out;
}

This is a neighbourhood operation, exactly like the blur kernels from our image-processing days and the neighbour-counting in Conway's Game of Life back in episode 48. Same shape of code, different job. Run erosion once (or twice for a really clean result) and your mask goes from TV-static to a crisp, confident silhouette. Makes sense, right? A real person is a big solid shape; noise is scattered loners.

The silhouette as a void

Now the fun part - doing something with that silhouette. My favourite first effect, the one that makes people gasp, is turning the person into a void in the visuals. Imagine a wall covered in drifting particles (our friends from episode 11). When you step in front of it, the particles can't occupy where your body is - so they flow around you, leaving a perfect you-shaped hole in the swarm. Your shadow becomes a physical presence the artwork respects.

// particles avoid the person: if a particle lands on a "1" in the mask,
// shove it away. the crowd flows around the body, leaving a silhouette.
function repelFromBody(p, mask, w, h) {
  const mx = Math.floor(p.x / width  * w);   // canvas coords -> mask coords
  const my = Math.floor(p.y / height * h);
  const i = my * w + mx;
  if (mask[i]) {                 // this particle is "inside" a person
    p.vx += (Math.random() - 0.5) * 4;   // kick it sideways...
    p.vy += (Math.random() - 0.5) * 4;   // ...and away it scatters
  }
}

Wire that into any particle loop you already have and project the result onto a wall, and suddenly the wall knows where you are. People will stand there waving their arms for ten minutes, watching the particles part around them like they're wading through a glowing river. That's the magic of interaction: the artwork stops being a picture and becomes a mirror that plays.

You can flip the idea, too - instead of a void, make the silhouette the only place things happen. Paint particles only inside the body outline and you become a walking window into another world, your shape filled with fire or stars or rain while everything around you stays dark.

Going 3D: depth cameras

The webcam trick is brilliant and free, but it only knows where in the flat image something changed - not how far away it is. For that you want a depth camera: an Intel RealSense, a Microsoft Azure Kinect, or even the LiDAR sensor tucked into a recent iPhone. These give you a value per pixel that is distance, not colour. Every pixel says "the nearest thing along this ray is 1.7 metres away". That changes everything.

// a depth frame isn't colour - each pixel is a DISTANCE in millimetres.
// 0 usually means "no reading". small = close, big = far.
//
//   depth[i] = 1700  -> something 1.7m away along that ray
//   depth[i] = 800   -> something much closer (a reaching hand!)
//   depth[i] = 0     -> sensor couldn't measure (too shiny, too far)
//
// suddenly "how far" is a thing your art can react to.

With depth you don't even need an empty-room reference - you just say "anything closer than 2 metres is a person" and you get a clean silhouette instantly, immune to lighting changes. Background subtraction cares about colour, so it breaks the moment someone turns a lamp on. Depth doesn't care about light at all, because it's measuring geometry.

// segment people out of a depth frame by a simple distance gate.
// no empty-room capture needed - geometry doesn't care about lighting.
function depthMask(depth, near = 500, far = 2500) {
  const mask = new Uint8Array(depth.length);
  for (let i = 0; i < depth.length; i++) {
    const d = depth[i];
    // valid reading, AND within our "stage" zone
    mask[i] = (d > near && d < far) ? 1 : 0;
  }
  return mask;
}

That near/far gate is like drawing an invisible box in the room - only things standing inside that slab of space count. It's how installations ignore the far wall and the ceiling and lock onto just the people on the "stage". And because you have real distance, you can do things a flat camera never could: make the visuals bloom brighter the closer someone leans in, or push a ripple outward at exactly the depth of a reaching hand.

Your body as a brush

Depth blobs are great for whole-body presence, but sometimes you want precision - you want to know where the hands are, the head, the feet. That's pose detection, and we already met it back in episode 93 when we put ml5.js to work reading the human body. The exciting bit is that everything we learned there plugs straight into a projector now. Instead of drawing skeletons on a screen, we paint with the joints onto a wall.

// pose detection (ml5, from episode 93) feeding a projection.
// we don't draw a skeleton - we use the wrist as a paintbrush.
let bodyPose, poses = [];

function setup() {
  createCanvas(1280, 800);
  const cam = createCapture(VIDEO);
  bodyPose = ml5.bodyPose();
  bodyPose.detectStart(cam, r => poses = r);   // keep the latest poses
}

Once you've got the joints streaming in, you map them to whatever visual you like. The simplest joyful thing: leave a glowing trail behind a wrist so people can draw in the air with their hand. Combine it with the projector and you've got a wall people paint on by dancing.

// paint a fading trail wherever the right wrist goes.
// this is "body as brush" - your movement IS the drawing tool.
let trail = [];

function paintWithWrist() {
  if (poses.length > 0) {
    const wrist = poses[0].right_wrist;
    if (wrist.confidence > 0.3) {            // only trust confident joints
      trail.push({ x: wrist.x, y: wrist.y, life: 1 });
    }
  }
  for (const t of trail) {
    noStroke();
    fill(255, 120, 40, t.life * 255);        // warm glow, fading out
    circle(t.x, t.y, 30 * t.life);
    t.life -= 0.02;                          // each dab slowly dies
  }
  trail = trail.filter(t => t.life > 0);     // forget the dead ones
}

That confidence > 0.3 check is a small thing I learned the hard way: pose models are guessing, and a low-confidence joint can teleport to the corner of the frame and drag an ugly streak across your whole piece. Trust only the joints the model is sure about, and everything gets calmer. When I first ran this at a little gallery night, a kid spent twenty minutes "conducting" the wall with both arms. That's the whole reward, right there.

Touching the surface

Here's a sneaky one that feels like proper wizardry: turning a plain table or wall into a touchscreen using nothing but a projector and a camera. Project onto a surface, point a camera at it, and watch for the moment a finger actually makes contact. With a depth camera it's clean - contact happens when a fingertip's distance matches the known distance of the surface itself.

// touch = something at (almost) exactly the surface's own depth.
// if the surface is 1200mm away, a finger touching it reads ~1200 too.
function touchPoints(depth, w, h, surfaceMM = 1200, slop = 25) {
  const hits = [];
  for (let y = 0; y < h; y++) {
    for (let x = 0; x < w; x++) {
      const d = depth[y * w + x];
      if (d > 0 && Math.abs(d - surfaceMM) < slop) {
        hits.push({ x, y });     // a point where something meets the surface
      }
    }
  }
  return hits;                   // cluster these into "fingers" next
}

A hand hovering above the table reads closer than the surface; only when a fingertip lands does its depth snap to the surface's depth (give or take a little slop for sensor jitter). Cluster those hit points into blobs and each blob is a finger. Now you've got multi-touch on a coffee table, from a projector and a sensor, no touchscreen hardware anywhere. Project ripples that spread from each touch and people will not believe it's just light.

Content that belongs to the place

Now let me put my art hat on, because interaction without good content is just a tech demo. The single biggest thing that separates a forgettable projection from one people remember is this: the content should belong to the surface it's on. Don't just splash a generic loop everywhere. Look at what you're projecting onto and make visuals that could only live there.

// site-specific: content that obeys the real geometry.
// water POOLS on flat tops and DRIPS down vertical faces.
// the surface's orientation drives the behaviour.
function flowByGravity(particle, surfaceTilt) {
  if (surfaceTilt === "horizontal") {
    particle.vy *= 0.85;          // on a table-top, water settles and pools
    particle.vx *= 0.9;
  } else {                        // on a wall, gravity wins - it runs down
    particle.vy += 0.3;           // steady downward pull
  }
}

A brick wall gets projected mortar that cracks along the real grout lines. A staircase gets water that genuinly cascades step to step. A tree in a courtyard gets autumn leaves that fall from its actual branches. When the light respects the thing it's landing on - gravity, edges, texture, the way a real surface would behave - people stop seeing "a projection" and start seeing an object that has come alive. That respect is the entire art of it, far more than any clever shader.

Making it heard as well as seen

One quick multi-sensory trick before we wrap the techniques: pair the visuals with spatial audio. If a ripple bursts on the left wall, the sound should come from the left. Our brains fuse sight and sound so eagerly that even rough stereo panning makes an installation feel dramatically more real and present.

// pan a sound to match where the visual event happened on-screen.
// x runs 0..width; map that to a stereo pan of -1 (left) .. +1 (right).
function playAt(sound, x) {
  const pan = map(x, 0, width, -1, 1);   // screen position -> ear position
  sound.pan(pan);
  sound.play();                          // splash on the left = sound on the left
}

It's a tiny amount of code for a huge lift in immersion. We first played with sound reacting to visuals back in episode 19; here we run it the other way, letting a visual event place a sound in the room. The senses reinforce each other and the whole thing tips over from "screen on a wall" into "place you're standing inside".

The unglamorous practicalities

Serious face for a minute, like every episode in this chapter, because the physical realities will decide whether your idea survives contact with a real room.

// femdev's interactive-projection reality check (notes-as-code, not a program :-)
const installTips = {
  lighting:   "your projector wants DARK, but your camera wants LIGHT. fight this: use a depth cam (ignores light) or infrared.",
  cameraSpot: "mount the camera near the projector's viewpoint, or the person's shadow and their mask won't line up.",
  daylight:   "normal projectors wash out in daylight. go 4000+ lumens, or rear-project, or embrace the evening.",
  latency:    "keep the loop under ~50ms or the reaction feels laggy and the magic dies. cheap sensors add delay.",
  calibrate:  "the camera sees the world in ITS coordinates, the projector in ITS own. you must map between them.",
  scale:      "building facades need 20,000+ lumen projectors, edge-blended in teams. start with a wall, not a cathedral.",
};

That calibrate one is the sneaky killer and worth a sentence more. Your camera has its own pixel grid; your projector has another; and the two are almost never aligned. So a person the camera sees at pixel (300, 200) is not at (300, 200) in your projected image. The standard fix is to reuse the exact tool from last episode: project a grid of dots, find where the camera sees each one, and compute a homography that translates camera-space into projector-space. Yes - the same corner-pinning maths from episode 110, now doing double duty to line up your sensor and your light. I told you these tools keep coming back :-).

Your exercise: a wall that notices you

Time to build the thing that got me hooked: a reactive projected wall. Point a projector at a blank wall, point a webcam at the same wall, and run a swarm of particles that parts around anyone who steps in front of it. Here's the whole loop stitched together from the pieces above.

// the complete reactive wall: capture, subtract, clean, react, draw.
// step in front of it and the particles flow around your silhouette.
let cam, particles = [];

function setup() {
  createCanvas(1280, 800);
  cam = createCapture(VIDEO);
  cam.size(160, 100);          // small = fast; the mask needn't be huge
  cam.hide();
  for (let i = 0; i < 1500; i++) particles.push(newParticle());
}

function draw() {
  background(0, 40);                              // gentle trails
  let mask = foregroundMask(cam, bg, 45);         // where are the people?
  mask = erode(mask, cam.width, cam.height);      // clean the speckle
  for (const p of particles) {
    repelFromBody(p, mask, cam.width, cam.height); // flow around bodies
    p.x += p.vx; p.y += p.vy;
    p.vx *= 0.96; p.vy *= 0.96;                    // gentle drag (episode 18!)
    wrap(p);                                       // reappear on the far edge
    stroke(120, 200, 255); point(p.x, p.y);
  }
}

function keyPressed() {
  if (key === 'b') captureBackground(cam);        // press b for empty-room snapshot
}

The ritual: run it, clear everyone out of frame, press b to grab the empty background, then step in front. If it's twitchy, nudge the threshold up or add a second erode pass. If your silhouette is offset from your body, that's the camera-projector calibration talking - line the camera up closer to the projector's eye.

Stretch goals, in rising order of "whoa":

• Fill the silhouette instead of voiding it - become a walking window full of fire.
• Swap the webcam for pose detection (episode 93) and paint trails from your wrists.
• Add spatial audio so movement on the left is heard on the left.
• Map the wall with corner pinning from last episode so the reactive art wraps a real 3D object, not just a flat wall.

Do at least the basic parting-swarm if you do nothing else. There is a very specific, giddy joy the first time a wall of light notices you walk in and rearranges itself around your shadow. It's the same lineage of joy as your first plot, your first laser cut, your first LED strip - but this one looks back at you. The artwork and the room are finally having a conversation.

't Komt erop neer...

• Interactive projection turns a one-way display into a two-way installation: the projector throws light out, a sensor reads the room back in, and your code closes the loop by changing the light. The room becomes an input device
• The cheapest sensor is a webcam plus background subtraction: freeze one empty-room frame, then every frame flag the pixels that differ. That's your people-mask, built on the same pixel-reading from episode 10. Copy the reference frame, don't alias it
• Raw masks are speckly with camera noise, so clean them with erosion - drop any foreground pixel without enough foreground neighbours. Same neighbourhood trick as blur kernels and Game of Life (episode 48)
• Turn the silhouette into a void (particles flow around your body) or a window (visuals only inside your outline). Either way the wall suddenly knows where you are, and people can't stop playing
• Depth cameras (RealSense, Kinect, iPhone LiDAR) give distance per pixel instead of colour. That means a lighting-proof silhouette from a simple near/far distance gate, plus reactions that depend on how close someone leans in
• Pose detection from episode 93 plugs straight into a projector: use joints as paintbrushes so people draw on the wall by moving. Trust only confident joints or they teleport and smear
• You can even build multi-touch on a plain table: with depth, a touch is a fingertip whose distance matches the surface's own distance. Cluster the hits into fingers
• The art is in site-specific content: make visuals that obey the real surface - water pools on tops and drips down walls, mortar cracks along real grout. When light respects the object, the object comes alive
• Pair it with spatial audio (episode 19, run backwards) so a splash on the left is heard on the left - the senses reinforce and immersion jumps
• Practicalities decide it: projector wants dark, camera wants light (depth cams dodge this), keep latency low, and calibrate camera-space to projector-space with - you guessed it - the same homography from last episode

So that's projection grown up: not just painting objects with light, but painting objects that paint back. And did you catch the pattern one more time? Almost nothing today was brand new - pixel diffing, neighbour counting, pose detection, homographies, panning audio - all tools we already owned, wired into a feedback loop with a room full of people. That's been the quiet lesson of this whole physical-world chapter: the ideas stay, only where the light lands keeps changing.

But everything we've done so far still needs the light on. Pull the plug and the magic vanishes. What if your generative shapes could become solid things with weight and edges - objects you can pick up, hold, hand to a friend, keep on a shelf long after the projector's switched off? That's a very different kind of machine, and it's exactly where we're headed next. Start saving your favourite sketches - some of them are about to become real :-).

Sallukes! Thanks for reading.

X

Hive account@femdev