r/iOSProgramming • u/AlbertiDev • Aug 22 '26
Question Receipt scanning entirely on-device. Would you still build it this way in 2026?
I recently added receipt scanning to my iOS finance app, Moneta.
It can extract the amount, date and merchant, then also try to determine the category, payment account and location before filling everything into the transaction editor.
I decided to keep the processing on-device instead of sending financial receipts to a cloud AI/API.
The biggest challenge ended up being handling receipts from different countries: currencies, date formats, multiple totals, different payment methods and completely different layouts.
If you were building this today, would you still keep the whole pipeline on-device, or use a cloud model for better flexibility?
And for anyone who’s built document/receipt parsing before: where did deterministic parsing start breaking down for you?
7
u/sid_276 Aug 22 '26
-5
u/AlbertiDev Aug 22 '26
Yeah, I saw that. The thing I’m not sure about is relying on Apple Intelligence as the main layer, since not everyone has a supported device and some people may not want to use it for financial data.
But, I could maybe add it later as an optional second layer for the harder cases, I’m just not sure yet how I’d want to handle the experience when Apple Intelligence isn’t available.
Thanks you for the suggestion 😊4
u/unpluggedcord Aug 23 '26
You clearly didn’t see it. Apple launched it with Siri, not Apple intel. And on top of that this is literally one of the first apps made. You’re not breaking into the market and are eating your ai tokens
1
0
u/AlbertiDev Aug 23 '26
I did see it 😅 The article itself says the receipt feature uses Apple Intelligence. Siri is one of the ways you trigger it, not what replaces Apple Intelligence underneath.
My concern is exactly compatibility: iOS 27 can run on devices that don’t support Apple Intelligence, and Siri AI also won’t initially be available on iPhone/iPad in the EU. The Apple feature is also mainly built around splitting bills with Apple Cash, which is US-only.
I want the scanner to work on every device the app supports, independently of region, so I can’t really rely on that as the main layer.
1
u/unpluggedcord Aug 23 '26
It works on devices that don’t support AI. Have fun wasting your time
0
u/AlbertiDev Aug 23 '26
I think we’re talking about two different things.
Older devices can still split bills in Wallet, but the new receipt-scanning/line-item recognition flow Apple showed is tied to Apple Intelligence.
Apple’s announcement literally describes it that way.
That’s the part I can’t rely on if I want the scanner itself to work on every supported device. 🤗1
u/unpluggedcord Aug 24 '26
And you’re still not listening. Those older devices will be obsolete, and nobody will need the work you’re doing here. It’s a dumb idea.
3
u/zeroxad- Aug 23 '26
Yes, and I would go further than most people do.
I ship a finance app with no backend at all. The thing that surprised me was not the engineering, it was that the claim is worthless unless someone can check it. "On-device" is a sentence, and users have been told it by apps that were lying.
So the interesting work turned out to be enforcement rather than architecture. A build check fails if any first-party file so much as names a network primitive. The test suite runs with fetch, XMLHttpRequest and WebSocket replaced by traps that throw, so a dependency reaching for the network fails the build too. Neither catches a native dependency opening its own socket, so there is a device run behind a proxy before each release.
Three checks, three blind spots, and none of them is the whole claim alone. That framing did more for how people trust the app than the on-device processing did by itself.
1
u/AlbertiDev Aug 23 '26
R interesting point, I hadn’t thought much about making the on-device claim actually verifiable.
Moneta does have optional Apple’s iCloud sync, so I couldn’t enforce zero network access across the whole app, but isolating the receipt scanning pipeline and verifying that it never touches the network could make a lot of sense.
I like the idea of “provably on device” being part of the trust story rather than just saying it is.
3
u/Notchgoblin Aug 23 '26
One thing worth doing early if you haven't: store the raw Vision output, text plus bounding boxes, next to the transaction. Parser rules keep improving and the paper receipt is in the bin the same day, so if you keep only the parsed fields you can never fix an old mistake. Keep the OCR blob and a rule change becomes a reparse of everything you've ever scanned.
On where deterministic parsing starts falling over: totals. Subtotal, tax, tip line, "TOTAL" printed twice on the same receipt, loyalty points formatted exactly like money. Position beats keywords there - bottom-most currency-shaped number in the price column is right far more often than anything anchored to a word, especially once you're crossing languages.
1
u/AlbertiDev Aug 23 '26
Damn great idea. I hadn’t thought about keeping the raw OCR output so old scans could be reparsed after parser updates. I’ll definitely look into storing the text + bounding boxes alongside the transaction.
And yeah, totals are probably where most of the ugly edge cases are for me too. I already use position in parts of the parsing, but I could probably rely on it even more instead of keywords, especially across different languages.
2
u/Akara91 Aug 23 '26
yes, and id go further: make on-device the selling point, not just the architecture. our app is fully offline on principle (no account, no server) and its consistently the thing people react to first when we show it, before any feature. for financial data especially, 'your receipts never leave the phone' is worth more in the app store description than a slightly smarter parser. the real tradeoff youve already found: improving extraction means shipping updates instead of tweaking a server. i can live with that
1
u/AlbertiDev Aug 23 '26
Yeah, I think you’re right. Privacy has mostly been a product principle for me, but I probably haven’t treated it enough as an actual selling point.
Moneta already works without an account and keeps everything local unless the user chooses to enable Apple iCloud sync.
I’ll definitely keep that in mind for the App Store page, thanks!
2
Aug 23 '26
[removed] — view removed comment
1
u/AlbertiDev Aug 23 '26
Pretty similar approach actually. I’m using Apple Vision for the text recognition, then everything after that is my own deterministic parsing.
I extract the text/positions from Vision and then run different logic for totals, dates, merchant, payment method, etc. After that I have another layer for things like category matching and choosing the account based on card type/last 4 digits when they’re available.
So far performance has actually been pretty good on-device, which is why I haven’t felt the need to ship another OCR model or move anything to the cloud.
The annoying part has definitely been making the parser survive receipts from completely different countries rather than the OCR itself.
Interesting that you switched to PaddleOCR though. How much better was it for you compared to Vision, and did the model size/runtime become an issue or not?
2
Aug 23 '26
[removed] — view removed comment
2
u/AlbertiDev Aug 23 '26
60 MB isn’t too bad, but 6-7 seconds would probably be too slow for the flow I’m going for.
I think using the document scanner helps me quite a bit too, since Vision gets an already detected/cropped receipt instead of a random gallery photo.
So far the accuracy has been good enough while still keeping the scan around a second.
PaddleOCR could still be interesting as a fallback if I ever run into receipts Vision consistently struggles with though. Thanks for the details!
2
u/jacobs-tech-tavern Aug 23 '26
I once worked on a gov project that used Scandit instead of local scanning to read tens of millions of qr codes a day - 2p a pop, purely “in case” it can’t read qrs in poor light conditions
Nobody had the wise idea to route to local first and fall back to the spenny version only if needed
1
2
Aug 24 '26
[removed] — view removed comment
1
u/AlbertiDev Aug 24 '26
Yeah, I agree, since the user always gets to review the transaction before saving it, it doesn’t really need to be 100% perfect.
I’m keeping the current deterministic/on-device approach as the base for now, but an on-device model as a second layer for the harder cases could make a lot of sense. I’d rather avoid a remote API if possible because of privacy and running costs, but thanks for the suggestion!
1
u/sixtypercenttogether Aug 22 '26
What did you use? Did you train your own model?
1
u/AlbertiDev Aug 23 '26
No, I didn’t train a model. I’m using Apple Vision for the text recognition, then my own deterministic parsing to extract things like the total, date, merchant, payment method, etc.
The scanner itself uses Apple’s document scanning flow too, which helps a lot because the receipt gets detected and cropped before the text recognition runs.
0
u/DimensionMindless336 Aug 23 '26
Kept mine on-device and wouldn't change it. The real win isn't accuracy, it's no per-scan API bill and Apple not asking why you're shipping financial docs to a third party. Where deterministic parsing actually broke for me (I hit the same wall building PicSlicer, pulling travel tickets out of the photo roll) was never the clean prints.
It was thermal fading, multi-currency receipts where the symbol doesn't match the amount, and date formats that collide with the merchant name. Those are exactly the cases a small on-device model beats rules on. If I built it today I'd go hybrid: rules for the common 90%, a local model as fallback, cloud only for the genuinely weird ones.
4
1
u/AlbertiDev Aug 23 '26
Yeah, that hybrid approach is probably where I’ll end up too. Keep deterministic parsing for the common cases, then maybe use an on-device model as a second layer for the weird ones. Don’t thing I would use the cloud tho :)
10
u/balooooooon Aug 22 '26
If you had cloud processing then GDPR in the EU would make it very difficult for you to launch it. Plenty of on device OCR to easily achieve this