Overview
chesshacker lets the person wearing Meta glasses play over-the-board chess with the best move in their ear. The glasses video-call a Mac on WhatsApp and share their view. The Mac works out when the opponent has moved, asks Stockfish for the reply, and speaks it back into the call.
Two rules keep it honest:
- The vision model only reports what it sees. Code matches that to a legal move.
- Stockfish picks the move, never the language model.
Tested end to end on glasses recordings. Not yet tried on a live WhatsApp call.
Setup
You need a Mac with Node.js 22 or newer (brew install node).
Setup downloads Stockfish, asks for your keys, and checks everything. You need an OpenAI API key; a Jev key from TypeSafe is optional and makes games cheaper. Keys are kept in your login Keychain.
Also install WhatsApp from the Mac App Store and BlackHole 2ch (brew install blackhole-2ch), and give your terminal app Screen Recording permission. Then check a real call once with npx chesshacker test-call.
Playing a game
- From the glasses, video-call this Mac’s WhatsApp and answer it on the Mac.
- Double-tap the glasses’ capture button to share their view. In the call’s ⋯ menu, set the microphone to BlackHole 2ch.
- Start chesshacker with your color:
npx chesshacker run --color white
npx chesshacker run --color black --fen "<position>"Look at the board. You’ll hear “Ready”, then the best move after each of your opponent’s moves. The command center opens in your browser at 127.0.0.1:8000, with the live frame, the board and its arrow, and controls to pause, undo, enter a move, or say it again.
Spending pauses at $10 by default (--max-cost).
How a move is read
About once a second, a frame gets two looks:
- A quick check. Jev skips it if the board isn’t readable (under 0.8), a hand is over it (over 0.5), or nothing changed (under 0.1).
- A full read. The vision model lists the squares that look different from the last confirmed frame.
Code compares that with the board after every legal move, and after every move plus a reply. A move is committed once it’s the best fit on 2 frames in a row, by 3 or more squares in total.
| Vision model | Moves | Per frame |
|---|---|---|
gpt-6.1-sol (default) | 3 of 3 | 1.4¢ · 4–4.6 s |
claude-opus-5-5 | 3 of 3 | 4¢ · 3.6–6.3 s |
About 10 seconds pass from a move to the spoken reply, mostly the two model reads.
When something’s wrong
Not sure what’s missing
Run npx chesshacker doctor. It checks Stockfish, your keys, WhatsApp, BlackHole and Screen Recording, and prints the exact fix for each problem.
It watches the wrong window
npx chesshacker windows lists WhatsApp’s windows. Pick one with --window.
“Lost track of the board”
Fix it from the command center: undo, enter the move yourself, or start a new game from a FEN.
No speech in the glasses
Set the call’s microphone to BlackHole 2ch in WhatsApp’s ⋯ menu. test-call checks this for you.