Real-time Face Tracking in Minecraft

How Minecraft Webcam transforms your face into a blocky avatar.

The Concept

Your webcam sees your face. Minecraft Webcam tracks it and renders a Minecraft character that mirrors your expressions and head movements. The result: a Minecraft version of yourself, animated in real-time.

How Face Tracking Works

MediaPipe Face Mesh detects 468 facial landmarks at 30 FPS. From these landmarks, the system calculates:

• Head pitch — Looking up/down (from nose position relative to eyes)
• Head yaw — Looking left/right (from nose position relative to face center)
• Head roll — Tilting (from line between eyes)
• Blink detection — Eye aspect ratio from landmark distances
• Mouth openness — Vertical distance between lip landmarks

Rendering the Avatar

Minecraft skins are 64x64 PNG files. The renderer:

Minecraft Webcam

Real-time face tracking that transforms your webcam feed into a Minecraft character with facial animations. Eyes blink when you blink, mouth opens when you talk.

Features

• Real-time Face Tracking — 30 FPS with 468 facial landmarks via MediaPipe
• Head Movement — Pitch (up/down), yaw (left/right), roll (tilt)
• Facial Animations — Eyes blink automatically, mouth opens when talking
• Custom Skins — Supports any standard Minecraft skin (64x64 format)
• Animated Skin Support — Create custom facial expressions stored in unused texture space
• Virtual Camera Output — Use the avatar in video calls or streaming (via OBS)
• System Tray Integration — Runs minimized while active

Technical Highlights

• MediaPipe Face Mesh — 468 3D facial landmarks
• Minecraft Skin Parser — Exact specification compliance (64x32 and 64x64)
• Perspective-correct rendering — Smooth 60 FPS UI updates
• Unused texture space — Rows 32-40 store custom animations
• Depth sorting — Painter’s algorithm for proper rendering order

Tech Stack

How It Works

1. MediaPipe detects 468 facial landmarks at 30 FPS
2. Head rotation calculated from nose and eye positions
3. Minecraft skin texture mapped to 3D quad with perspective correction
4. Avatar output sent to virtual camera for use in other apps

Pet Face Recognition

A local-first pet registration and identification system using face recognition. No cloud AI dependencies — all processing happens locally with deterministic embeddings.

Features

• Local-Only Architecture — No external cloud APIs; all processing done locally
• Facial Embedding Pipeline — Extracts 256-dimensional feature vectors using classical computer vision:
  • Local Binary Patterns (LBP)
  • Histogram of Oriented Gradients (HOG)
  • Discrete Cosine Transform (DCT)
  • Color histograms (HSV)
  • Multi-scale grid pooling
• pgvector Similarity Search — Efficient vector storage and retrieval in PostgreSQL
• Pet Registration — Upload multiple images per pet with name and location
• Identification — Find matching pets from uploaded images with similarity scoring

Technical Highlights

• Deterministic embeddings — Same image always produces identical features
• Multi-criteria matching — Similarity score + margin between matches + minimum images
• YOLO11 for classification, segmentation, and pose estimation
• Docker Compose for easy local development
• FastAPI backend with React frontend

Tech Stack

Why Local-First?

Running AI locally means no API costs, no privacy concerns, and deterministic results. The system extracts facial features using classical computer vision techniques that are reproducible and don’t depend on external services.