← [ BACK_TO_BLOG ]

How Fun Balls Works: Real-Time Physics, Webcam Blob Detection, and Canvas Rendering in React

Fun Balls is an interactive browser-based physics playground built as a single React component inside a Next.js portfolio app. At first glance it looks like a simple particle effect but under the hood it orchestrates five distinct systems running in a single requestAnimationFrame loop: a 2D physics engine, webcam capture, computer-vision-style blob detection, canvas rendering, and optional video recording. This post breaks down every system in detail.

Architecture Overview

The entire experience lives in one 'use client' component — src/app/fun-balls/page.tsx and a small App.css stylesheet. There are no server components, no API routes, and no external data sources. Everything runs client-side in a continuous animation loop.

The systems are:

  1. Matter.js Physics Engine — simulates gravity, collisions, and restitution for balls and static obstacles.

  2. Canvas 2D Rendering — draws the webcam background, glowing fireball particles, hand-drawn lines, and detection overlays.

  3. Webcam Capture + Minimap — captures video via getUserMedia, mirrors it, and downscales it into a 240×180 debug minimap.

  4. Red Blob Detection — performs pixel-level color masking, flood-fill connected-component analysis, and oriented bounding box fitting on the minimap each frame.

  5. Stability Tracking — matches detected blobs across frames, determines when a red object is stationary, and locks it as a physics obstacle.

  6. Canvas Recording — captures the main canvas stream at 60fps via MediaRecorder and offers a downloadable .webm file.

1. React Refs and State Setup

The component uses five React refs to hold direct references to DOM elements that the animation loop needs to access without triggering re-renders:

const canvasRef = useRef<HTMLCanvasElement>(null)    // main full-screen canvas
const minimapRef = useRef<HTMLCanvasElement>(null)  // 240×180 debug view
const videoRef = useRef<HTMLVideoElement>(null)      // hidden <video> for webcam
const mediaRecorderRef = useRef<MediaRecorder | null>(null)
const chunksRef = useRef<Blob[]>([])
const [isRecording, setIsRecording] = useState(false)

This pattern is critical for high-performance canvas apps in React. By storing the actual DOM element references in refs, the animation loop can read and write pixels at 60fps without causing React reconciliation or re-renders. The only state variable that triggers a re-render is isRecording, which only changes when the user clicks the record button.

2. Constants and Configuration

At the top of the file, all tunable parameters are defined as module-level constants:

const BALL_RADIUS = 10
const BALLS_PER_SECOND = 15
const BALL_COLOR = '#ff00ff'
const SPAWN_VX = 4
const SPAWN_VY = 3.5
const MINIMAP_WIDTH = 240
const MINIMAP_HEIGHT = 180
const STABILITY_THRESHOLD_MS = 2000   // red object must be still for 2s
const POSITION_TOLERANCE = 30          // max minimap-px movement to count as 'still'

These values control the density of particles, the speed they launch at, the size of the debug minimap, and the sensitivity of the blob stability tracker. Changing BALLS_PER_SECOND from 15 to 30, for example, doubles the visual density of the particle shower.

3. The useEffect — One Giant Setup Function

Because this is a 'use client' component, all browser-dependent code runs inside a single useEffect with an empty dependency array []. This means it runs once after the component mounts and tears everything down on unmount. The return cleanup function handles:

return () => {
  cancelAnimationFrame(rafId)
  window.removeEventListener('resize', resize)
  canvas.removeEventListener('mousedown', onMouseDown)
  canvas.removeEventListener('mousemove', onMouseMove)
  canvas.removeEventListener('mouseup', onMouseUp)
  Matter.Engine.clear(engine)
  const stream = video.srcObject as MediaStream | null
  stream?.getTracks().forEach((t) => t.stop())
  if (mediaRecorderRef.current && isRecording) {
    mediaRecorderRef.current.stop()
  }
}

This is important because forgetting to stop the webcam tracks or cancel the animation frame causes memory leaks and background CPU usage even after navigating away.

4. Matter.js Physics Engine

The physics engine is initialized with stronger-than-default settings:

const engine = Matter.Engine.create({
  gravity: { x: 0, y: 1.2 },
  positionIterations: 30,
  velocityIterations: 30,
})

positionIterations and velocityIterations both default to 6. Setting them to 30 means the solver runs 5× more substeps per frame, producing much more stable collisions — especially important because balls are spawned at high frequency and user-drawn walls can create tight geometric constraints.

Ball Spawning

Every frame, the loop checks whether enough time has elapsed since the last spawn:

if (time - lastSpawnTime >= spawnInterval) {
  lastSpawnTime = time
  const x = canvas.width / 2
  const ball = Matter.Bodies.circle(x, -BALL_RADIUS * 2, BALL_RADIUS, {
    restitution: 0.6,
    friction: 0,
    frictionAir: 0,
    frictionStatic: 0,
    inertia: Infinity,
    label: 'ball',
    collisionFilter: { category: 0x0002, mask: 0x0001 },
  })
  Matter.Body.setVelocity(ball, { x: SPAWN_VX, y: SPAWN_VY })
  Matter.Composite.add(engine.world, ball)
  balls.push(ball)
}

Key details:

  • restitution: 0.6 — balls retain 60% of their velocity on bounce, creating a lively but not infinite bouncing effect.

  • friction: 0 and frictionAir: 0 — balls glide without drag, simulating a frictionless vacuum.

  • inertia: Infinity — prevents balls from rotating, keeping the visual clean.

  • collisionFilter — uses a category/mask bitmask system. Balls are category 0x0002 and can only collide with category 0x0001 (walls and detection bodies). This means balls pass through each other but bounce off walls — exactly what you want for a stream of particles.

Ball Removal

Balls that fall off-screen are removed from both the Matter.js world and the tracking array:

for (let i = balls.length - 1; i >= 0; i--) {
  const b = balls[i]
  if (b.position.y > canvas.height + 60 || ...) {
    Matter.Composite.remove(engine.world, b)
    balls.splice(i, 1)
  }
}

This prevents unbounded memory growth. Without it, the balls array and physics world would grow indefinitely.

Fixed-Timestep Integration

The engine is updated using a fixed-timestep accumulator pattern:

while (delta > 0) {
  const step = Math.min(FIXED_STEP, delta)
  Matter.Engine.update(engine, step)
  delta -= step
}

FIXED_STEP = 1000 / 60 (approximately 16.67ms). This means the physics simulation advances in fixed 60Hz increments regardless of the actual frame rate, preventing physics instability when frames take longer than expected.

5. User-Drawn Walls

The canvas listens for mousedown, mousemove, and mouseup events. When the user clicks and drags:

const onMouseUp = (e: MouseEvent) => {
  if (!drawStart) return
  const x1 = drawStart.x, y1 = drawStart.y
  const x2 = e.clientX, y2 = e.clientY
  drawStart = null

  const dx = x2 - x1, dy = y2 - y1
  const len = Math.sqrt(dx * dx + dy * dy)
  if (len < 15) return  // ignore tiny accidental clicks

  const cx = (x1 + x2) / 2, cy = (y1 + y2) / 2
  const angle = Math.atan2(dy, dx)
  const lineBody = Matter.Bodies.rectangle(cx, cy, len, 8, {
    isStatic: true,
    angle,
    restitution: 1,
    friction: 0,
    label: 'wall',
    collisionFilter: { category: 0x0001, mask: 0x0002 },
  })
  Matter.Composite.add(engine.world, lineBody)
  lines.push({ x1, y1, x2, y2, body: lineBody })
}

Each drawn line becomes a thin (8px high) static rectangle in the physics world, placed at the midpoint with the correct angle. The restitution: 1 means balls bounce off walls with 100% energy retention — creating a pinball-like feel.

Preview Line

While dragging, a dashed preview line is drawn between the start point and the current mouse position:

ctx.setLineDash([8, 5])
ctx.beginPath()
ctx.moveTo(drawStart.x, drawStart.y)
ctx.lineTo(mousePos.x, mousePos.y)
ctx.stroke()
ctx.setLineDash([])

This gives immediate visual feedback before the wall is committed.

6. Canvas Rendering Pipeline

Each frame follows this rendering order:

Layer 1: Webcam Background

If the webcam is available, the video frame is drawn cover-fit and mirrored horizontally:

ctx.translate(canvas.width, 0)
ctx.scale(-1, 1)
ctx.drawImage(video, -offsetX - drawW + canvas.width, offsetY, drawW, drawH)

The mirror transform creates a natural "selfie" view. If the webcam is unavailable, a solid dark background (#0a0a0a) is rendered instead.

Layer 2: Committed Lines

All user-drawn walls are rendered with a glowing white stroke:

ctx.strokeStyle = 'rgba(255, 255, 255, 0.9)'
ctx.shadowColor = 'rgba(255, 255, 255, 0.6)'
ctx.shadowBlur = 10

The shadowBlur creates a soft glow effect around each line, making them visually distinct from the background.

Layer 3: Fireball Particles

Each ball is rendered as a two-pass radial gradient to create a "fireball" glow effect:

// Outer glow
const gradient = ctx.createRadialGradient(x, y, BALL_RADIUS * 0.1, x, y, BALL_RADIUS * 1.8)
gradient.addColorStop(0, 'rgba(255, 0, 255, 0.95)')
gradient.addColorStop(0.45, 'rgba(255, 0, 255, 0.8)')
gradient.addColorStop(1, 'rgba(255, 0, 255, 0)')  // transparent edge
ctx.fillStyle = gradient
ctx.beginPath()
ctx.arc(x, y, BALL_RADIUS * 1.8, 0, Math.PI * 2)
ctx.fill()

// Solid core
ctx.fillStyle = BALL_COLOR  // '#ff00ff'
ctx.beginPath()
ctx.arc(x, y, BALL_RADIUS * 0.85, 0, Math.PI * 2)
ctx.fill()

The outer gradient spans 1.8× the ball radius with decreasing opacity, while the inner solid circle is 0.85× the radius. The combination creates a convincing neon-glow particle effect with zero texture assets.

Layer 4: Detection Overlay

Yellow bounding boxes are drawn on the main canvas for any tracked red objects (more on this below).

7. The Minimap — Webcam Debug View

A small 240×180 canvas sits in the bottom-left corner as a debug visualization. The minimap serves two purposes:

  1. Visual feedback — shows the user what the webcam sees and what the blob detector is detecting.

  2. Processing pipeline — the minimap canvas is where all the image processing happens (read via getImageData, processed, then written back via putImageData).

The willReadFrequently: true option on the minimap context is important — it tells the browser to keep the canvas data in CPU-accessible memory rather than GPU memory, significantly speeding up getImageData calls.

8. Red Blob Detection Pipeline

This is the most technically interesting part of the application. Each frame, the minimap undergoes a multi-step computer vision pipeline:

Step 1: Color Mask Generation

Every pixel in the minimap is tested against red dominance criteria:

const RED_MIN = 90
const RED_DOMINANCE = 40
for (let i = 0, p = 0; i < data.length; i += 4, p++) {
  const r = data[i], g = data[i + 1], b = data[i + 2]
  if (r >= RED_MIN && r - g >= RED_DOMINANCE && r - b >= RED_DOMINANCE) {
    mask[p] = 1
  }
}

A pixel is classified as "red" if:

  • The red channel is at least 90 (prevents dark noise from triggering).

  • The red channel exceeds both green and blue by at least 40 units (ensures genuine red, not just bright white or pink).

Step 2: Grayscale Conversion + Red Tint

After masking, the entire frame is converted to grayscale for display clarity. Masked pixels are then tinted red:

const gray = (r * 0.299 + g * 0.587 + b * 0.114) | 0
// ...
if (mask[p]) {
  data[di] = Math.min(255, (data[di] * 0.6 + 140) | 0)
  data[di + 1] = (data[di + 1] * 0.35) | 0
  data[di + 2] = (data[di + 2] * 0.4) | 0
}

This creates a clear visual distinction: the minimap shows a grayscale video feed with red-highlighted detected regions — making it easy to see what the detector is picking up.

Step 3: Flood-Fill Connected Components

The mask is scanned pixel by pixel. When an unvisited red pixel is found, a BFS flood-fill expands to find all connected red pixels:

for (let sy = 0; sy < h; sy++) {
  for (let sx = 0; sx < w; sx++) {
    const sIdx = sy * w + sx
    if (visited[sIdx]) continue
    if (!mask[sIdx]) continue

    // BFS flood-fill
    queue[qTail++] = sIdx
    while (qHead < qTail) {
      const p = queue[qHead++]
      // ... check 4 neighbors, add if red and unvisited
    }

The flood-fill uses a pre-allocated Int32Array queue to avoid garbage collection pressure — critical when running at 60fps.

After flood-filling, each blob is validated:

  • Minimum pixel count (MIN_BLOB_PIXELS = 120) — filters out noise.

  • Maximum fraction (MAX_BLOB_FRACTION = 0.5) — ignores blobs covering more than 50% of the frame (likely a false positive from ambient red lighting).

  • Border touch — blobs touching the minimap edge are discarded since they're likely background bleed.

Step 4: Oriented Bounding Box Fitting

For each valid blob, the code computes:

  1. Centroid — the average position of all pixels in the blob.

  2. Covariance matrix — sxx, syy, sxy over the blob's pixel positions.

  3. Principal angle — via 0.5 * Math.atan2(2 * sxy, sxx - syy), giving the rotation of the blob's longest axis.

  4. Rotated extents — each pixel is projected onto the rotated axes (u, v) to find the tight bounding box along the principal orientation.

const angle = 0.5 * Math.atan2(2 * sxy, sxx - syy)
const cos = Math.cos(angle)
const sin = Math.sin(angle)
// Project onto rotated axes
const u = dx * cos + dy * sin
const v = -dx * sin + dy * cos

This produces an oriented bounding box (cx, cy, hw, hh, angle) that tightly wraps the red blob regardless of its rotation — meaning a tilted red card is detected accurately.

9. Cross-Frame Stability Tracking

Blob detection is noisy — red pixels flicker in and out as the webcam adjusts exposure and the user moves slightly. The stability tracker solves this by matching blobs across frames:

Matching Algorithm

For each newly detected box, the code finds the closest existing tracked object within a distance threshold:

for (const b of boxes) {
  let bestIdx = -1
  let bestDist = Infinity
  for (let i = 0; i < trackedPostIts.length; i++) {
    if (matched.has(i)) continue
    const t = trackedPostIts[i]
    const dx = b.cx - t.cx
    const dy = b.cy - t.cy
    const dist = Math.sqrt(dx * dx + dy * dy)
    if (dist < bestDist) {
      bestDist = dist
      bestIdx = i
    }
  }
}

If the closest tracked object is within POSITION_TOLERANCE * 3 (90 minimap-pixels), it's considered a match. Otherwise, a new tracked object is created.

Locking Mechanism

Each tracked object has a lastMoveTime and a locked flag:

if (bestDist > POSITION_TOLERANCE) {
  t.lastMoveTime = now  // reset the stability timer
}
if (now - t.lastMoveTime >= STABILITY_THRESHOLD_MS) {
  t.locked = true  // 2 seconds of stability → lock
}

A red object must remain within 30 minimap-pixels of the same position for at least 2 seconds before it becomes a physics obstacle. This prevents brief flickers or hand movements from creating phantom obstacles.

Cleanup

Tracked objects that lose their matching blob are kept alive for 3 extra seconds (if locked) before being removed:

trackedPostIts = trackedPostIts.filter((t, i) => {
  if (matched.has(i)) return true
  if (t.locked && (now - t.lastMoveTime < 3000)) return true
  return false
})

This prevents visual flicker when a hand briefly passes over a red card.

10. Coordinate Mapping — Minimap to Main Canvas

The detected blobs live in minimap-space (240×180), but the physics bodies need to exist in main-canvas-space (full viewport). Both canvases render the same mirrored webcam feed using cover-fit scaling, so the transformation is a simple linear mapping:

const ratio = scaleMain / scale
const mainCx = (t.cx - offsetX) * ratio + offsetXMain
const mainCy = (t.cy - offsetY) * ratio + offsetYMain
const mainHw = t.hw * ratio
const mainHh = t.hh * ratio

The angle is preserved because both views use the same mirror + cover-fit transformation (uniform scale + translate, no shear).

11. Canvas Recording

The recording system uses the browser's native MediaRecorder API:

const stream = canvas.captureStream(60) // 60 fps
const recorder = new MediaRecorder(stream, {
  mimeType: 'video/webm;codecs=vp9',
})

When recording starts:

  1. A MediaStream is captured from the canvas at 60fps.

  2. A MediaRecorder encodes it as VP9 WebM.

  3. Data chunks are collected in chunksRef.

When recording stops:

  1. Chunks are combined into a single Blob.

  2. A temporary object URL is created.

  3. An invisible <a> element is programmatically clicked to trigger a download.

  4. The object URL and temporary element are cleaned up.

const blob = new Blob(chunksRef.current, { type: 'video/webm' })
const url = URL.createObjectURL(blob)
const a = document.createElement('a')
a.href = url
a.download = `canvas-recording-${Date.now()}.webm`
document.body.appendChild(a)
a.click()
document.body.removeChild(a)
URL.revokeObjectURL(url)

12. Performance Considerations

  • No React re-renders in the hot path — the entire animation loop runs outside React's state system, using refs for DOM access.

  • Pre-allocated arrays — the flood-fill queue uses Int32Array(W × H) allocated once, avoiding per-frame garbage collection.

  • Fixed timestep — physics runs at a consistent 60Hz regardless of frame drops, preventing simulation blowup.

  • willReadFrequently: true — tells the browser to optimize the minimap for CPU reads.

  • Separate collision filters — balls and walls use different bitmask categories so balls don't collide with each other (only with walls and detection bodies), reducing the O(n²) collision check overhead.

  • Off-screen cleanup — balls are removed when they leave the viewport, keeping the physics world small.

13. Potential Improvements

While the current implementation is functional and performant, there are several directions for enhancement:

  • Multi-color detection — extend the color mask to detect blue, green, or arbitrary hue ranges using HSV conversion.

  • Touch support — add touchstart, touchmove, touchend handlers for mobile drawing.

  • WASM blob detection — move the flood-fill and bounding box computation to a WebAssembly module for an order-of-magnitude speedup on the minimap processing.

  • OffscreenCanvas — move the minimap processing to an OffscreenCanvas on a Web Worker to free up the main thread.

  • Audio reactivity — use the Web Audio API to modulate ball spawn rate or color based on microphone input.

  • Undo/redo for drawn walls — store wall history in a stack to support Ctrl+Z removal.

Conclusion

Fun Balls demonstrates how far you can push a single React component when you combine the right browser APIs: Matter.js for physics, Canvas 2D for rendering, getUserMedia for webcam input, ImageData manipulation for computer vision, and MediaRecorder for export. The key architectural decisions — using refs instead of state for the hot path, pre-allocating buffers, and using fixed-timestep physics — make it performant enough to run smoothly even with hundreds of physics bodies and per-frame image processing.

The project is open in the portfolio repo at src/app/fun-balls/page.tsx. Try holding up a red card to your webcam and watching the physics bodies appear.

Published with LeafPad

[ END_OF_POST ]