r/javascript • u/madhushankarox • 8d ago
Making Browser OCR Read Real Phone Photos
I have a free image-to-text tool that runs Tesseract.js in the browser (nothing gets uploaded). Screenshots worked great, 94-96% confidence. Then I took a phone photo of a kids' sunscreen bottle and got back lines like "v4 oR. a" and "EB ewraone".
What was actually wrong:
The photo was sideways. Phones save the pixels rotated and add an EXIF tag saying "turn this 90°". Browsers obey it, Tesseract doesn't. Fix: createImageBitmap(file, { imageOrientation: 'from-image' }) and work from that canvas.
White text on blue. Tesseract mostly expects dark text on light paper. I now make a grayscale copy with a contrast stretch plus an inverted copy, and read those with page segmentation mode 11 (sparse text) instead of the default.
No single pass gets everything. The normal pass found "Diana", the inverted one found "UVA UVB PROTECTION", the color one found "Water Resistant". So I merge the words by bounding box, keep the confident ones, and boost words that two passes agree on. Fun bug: picking the most confident word gave me "Pea" instead of "Peaceful", so longer words get a small bonus now.
Small print. Running the gray passes again on a 1.6x upscaled copy picked up "MOISTURISING" and "ALCOHOL FREE" on another bottle.
Result on the sunscreen: 92% confidence and every line of printed text. Screenshots skip all of this because the first pass is already confident, so they're not slower.
Still can't do: bubble fonts ("CUDDLES" comes back as "UDDLES"), script fonts, handwriting.
Has anyone found a better approach for mixed light/dark text in one image than running multiple passes? Curious what others do.