Transcribing a board by hand is slow and error-prone.
If you want to analyse a position you saw over the board — in a book, on a friend's set, mid-game — you first have to type it in. That means reading 64 squares, remembering algebraic notation, and not fumbling a rank. It is exactly the kind of mechanical task a camera should be able to do for you.
The goal was narrow and honest: point a phone at a board, get back a
FEN string you can paste into any engine, plus a
PGN you can replay. No app store, no accounts — just an
upload box and a result.
The classifier was easy. The geometry was the project.
Three things it had to get right.
Work from a single, imperfect photo.
No fixed rig, no top-down mount. A hand-held photo at an angle, in ordinary light, has to be usable — which makes perspective correction non-optional.
Always be correctable.
An OCR that is 95% right is still wrong on three squares. The user must be able to fix a misread piece before the FEN is finalised, so the board review step is part of the product, not a debug view.
Emit standard formats only.
The output is plain FEN and PGN via
python-chess— the notation every engine and database already speaks. No bespoke schema.
Four stages, one endpoint.
The whole system is a single POST /api/ocr that accepts an
image and returns JSON. Inside, the request passes through four stages:
detect the board, warp it flat, classify each cell, then assemble the
notation.
The model was trained largely on synthetic data — tens of thousands of board positions rendered programmatically with official piece sprites, then jittered with rotation, brightness, and occlusion so it wouldn't overfit to one clean render.
Bad crops poison everything downstream.
The classifier reached strong validation accuracy quickly. The failures in practice came almost entirely from the board-detection step: a missed corner or a slightly-off quadrilateral shears the whole grid, and then every one of the 64 cells is cropped a few pixels wrong. A great piece classifier fed a sheared board produces confident nonsense.
The fix was to treat detection as the first-class problem and give the user the last word — the review screen overlays the detected 8×8 grid so a wrong square is a one-click correction, not a re-shoot.
Upload, review, done.
The React client is three screens on purpose — an upload with instant preview, a review canvas where the detected grid sits over the photo and any square is a dropdown away from correction, and a result view with the FEN in a monospace block and the PGN ready to copy.
The review step is the one that earns trust. It admits, out loud, that the model can be wrong — and makes being wrong cheap.
Three things, next time.
Invest in corner detection before the CNN.
A dedicated corner/keypoint model would have removed the single biggest source of real-world failure earlier than any accuracy gain on the classifier.
Close the synthetic-to-real gap deliberately.
Synthetic renders train fast but miss real sets, shadows, and worn pieces. A small labelled set of real photos for fine-tuning would pay for itself.
Return confidence per square.
Surfacing low-confidence cells in the review UI would point the user straight at the squares worth checking, instead of asking them to scan all 64.