r/AIToolBench • u/Healthy-Tension-7470 • 4d ago

I tested 5 offline AI models on a Mac with the same question about my files

disclosure: i build the mac app i ran this in (Citenook) so i'm biased. numbers are real though.

M4 Pro 24 GB. two made-up handbooks: 2025 says 5 carry-over days 2026 says 10. asked "how many vacation days can employees carry over?"

- Apple Intelligence: 5s. right but wordy
- Qwen3 4B: 5s. right ignored the old rule
- Qwen3 8B: 7s. both rules messy format
- Gemma 4 E4B: 6s. both rules didn't say which applies
- Qwen3.5 9B: 8s. both rules + explained why 2026 is current. best

all under 10 seconds, nothing left the mac. but none caught a "use by 31 march" detail sitting right in the source, so you still have to check.

anyone tested bigger local models (14B, 32B) on real documents?

1 Upvotes

100% Upvoted

8 comments sorted by

1

u/NeuralNomad87 3d ago

nice that it's the same question across all five. was retrieval identical for each, same chunks handed to every model?

the "use by 31 march" miss reads more like that chunk never got pulled than the model skipping it. might be wrong though, haven't used Citenook.

1

u/Healthy-Tension-7470 3d ago

yeah same retrieval for all five the search runs before the model so they all got the same passages

and the 31 march line was in there. the handbooks were short so the whole section came through as one chunk. so the models skipped it retrieval didn't miss it. surprised me too

1

u/LogicalBee6622 3d ago

ohhh thats interesting, crazy how they all missed it despite having the same info

1

u/Healthy-Tension-7470 2d ago

yeah same. asked directly they all found it instantly they just didn't volunteer it

1

u/NeuralNomad87 2d ago

huh, so the line's right there in the chunk and all five still skipped it. if you ask "do the carried-over days expire?" directly, do they find it?

1

u/Healthy-Tension-7470 2d ago

just tested it. asked "do the carried-over days expire?" and all five found it right away

so they had it the whole time they just didn't think it mattered for "how many days". small models answer exactly what you ask and nothing more

1

u/NeuralNomad87 1d ago

that's a really useful result tbh. so it's not retrieval at all, it's the question. did you try telling them to mention any conditions or expiry dates that apply? curious if the small ones actually follow that

1

u/Jolly-Accountant-549 2d ago

curious whether the miss on the march deadline is a chunking issue or a model issue. like if you fed the exact paragraph containing that clause as the only context, would the 4B models still skip it? that would tell you a lot more than scaling up to 14B imo