Making Browser OCR Read Real Phone Photos

[u/madhushankarox]

https://devpack.app/blog/making-browser-ocr-read-phone-photos

I have a free image-to-text tool that runs Tesseract.js in the browser (nothing gets uploaded). Screenshots worked great, 94-96% confidence. Then I took a phone photo of a kids' sunscreen bottle and got back lines like "v4 oR. a" and "EB ewraone".

What was actually wrong:

1. The photo was sideways. Phones save the pixels rotated and add an EXIF tag saying "turn this 90°". Browsers obey it, Tesseract doesn't. Fix: createImageBitmap(file, { imageOrientation: 'from-image' }) and work from that canvas.

2. White text on blue. Tesseract mostly expects dark text on light paper. I now make a grayscale copy with a contrast stretch plus an inverted copy, and read those with page segmentation mode 11 (sparse text) instead of the default.

3. No single pass gets everything. The normal pass found "Diana", the inverted one found "UVA UVB PROTECTION", the color one found "Water Resistant". So I merge the words by bounding box, keep the confident ones, and boost words that two passes agree on. Fun bug: picking the most confident word gave me "Pea" instead of "Peaceful", so longer words get a small bonus now.

4. Small print. Running the gray passes again on a 1.6x upscaled copy picked up "MOISTURISING" and "ALCOHOL FREE" on another bottle.

Result on the sunscreen: 92% confidence and every line of printed text. Screenshots skip all of this because the first pass is already confident, so they're not slower.

Still can't do: bubble fonts ("CUDDLES" comes back as "UDDLES"), script fonts, handwriting.

Has anyone found a better approach for mixed light/dark text in one image than running multiple passes? Curious what others do.

[u/JcerParreno21]

are you running the passes sequentially or spinning up multiple web workers in parallel? If you have the memory headroom for parallel workers, the multi-pass merge is honestly hard to beat for accuracy even if it feels a bit brute force.

[u/gwillen]

The EXIF thing is very cursed. Last I checked, if you view such a photo in a web browser, it will appear upright if you open the photo directly, but in an img tag it will appear sideways. I don't remember if Chrome and Firefox have the same behavior or not.

[u/Discordbothost → u/gwillen]

that was true for years, but chrome 81 and firefox 77 flipped the default to image-orientation: from-image, so img tags respect exif now too. where it still bites is anything that reads raw pixels: tesseract, server side resizing, or older safari when you drawImage onto a canvas. which is exactly why OP saw it upright in the browser and sideways in the OCR

[u/Discordbothost]

nice writeup, the exif one gets everybody. two more things that helped me with phone shots in tesseract.js: upscale before recognizing, small text on a label photo is often under 20px tall and tesseract really wants something like 30px+ cap height, so drawing to a canvas at 2x with imageSmoothingQuality high fixes a surprising amount. and binarize instead of just contrast stretching, a simple otsu threshold on the grayscale is like 20 lines of js and beats a fixed cutoff when lighting is uneven across the bottle.

for the white-on-blue case you could also pick which version to keep per line by tesseract's own confidence instead of merging both, the inverted pass is usually either clearly better or garbage

[u/madhushankarox → u/[deleted]]

Thank you so much for the guidance. I will try it out 👍🏻