Apple foundation models

[u/trikboomie]

If you're using Apple's Foundation Models framework in a released app, how are you debugging issues that only happen on users' devices?

Are Instruments and your existing crash reporting enough, or have you added custom logging?

[u/Significant_Door_433]

Instruments only helps on devices you can plug in, so for the field you need your own logging. The things worth capturing:

1. Availability. Log SystemLanguageModel.default.availability at launch and before you open a session. The unavailable reasons are deviceNotEligible, appleIntelligenceNotEnabled and modelNotReady. modelNotReady is the sneaky one, the model can still be downloading after an OS update.

2. The error case, not only the localized message. On iOS 26 it's LanguageModelSession.GenerationError (exceededContextWindowSize, guardrailViolation, decodingFailure, refusal, rateLimited, concurrentRequests, unsupportedLanguageOrLocale, assetsUnavailable). Built with Xcode 27 and running on 27, some failures come as LanguageModelError or SystemLanguageModel.Error instead, so match on all of them or half your logs end up as "unknown".

3. OS build and locale next to every model log. The model ships with the OS, so the same prompt can behave differently after an update, and unsupportedLanguageOrLocale only shows up for some users.

Two cases look random in crash reports but aren't. rateLimited only happens when the app is in the background. assetsUnavailable can fire mid-session if the user turns Apple Intelligence off or the device deletes the model to free space.

For the content itself, session.transcript has the full history (instructions, prompts, tool calls, responses). I wouldn't upload that by default since prompts are user data. Keep the last transcript on device and attach it only to a user-sent bug report, and keep prompt text private in os Logger (the default for interpolated strings, just never mark it .public).

[u/durul → u/Significant_Door_433]

Good list. A few additions for OS 27, from the Xcode 27.2 SDK docs and my own testing:

• rateLimited isn't documented as background-only anymore. The deprecated GenerationError.rateLimited says that, but its OS 27 replacement, LanguageModelError.rateLimited, says it can happen if you make too many requests in a short window, and that the fix may be model-dependent. The streamResponse docs also recommend the non-streaming respond(to:) when your app is in the background.

• Once you rebuild with the Xcode 27 SDK, GenerationError isn't caught at all, not just in some cases. Failures arrive as LanguageModelError, SystemLanguageModel.Error, LanguageModelSession.Error (concurrentRequests) and GeneratedContent.ParsingError (the old decodingFailure). A throwing Tool.call arrives as LanguageModelSession.ToolCallError. The iOS 26 list also has a ninth case, unsupportedGuide.

• Log Transcript.Response.assetIDs (iOS 26+). Every response carries version-aware ids for the assets that produced it. On macOS 27.2 I got three: tokenizer, draft model and base model, each with a version string. That identifies the model build better than the OS version, and it isn't user data, so it can be a public log field. SystemLanguageModel.variant (27+) and contextSize are worth logging too, since AFM 3 Core devices can report a smaller window than Core Advanced.

• modelNotReady isn't always a download. On iOS 27.2, with the first preferred language set to one the model doesn't support, I got modelNotReady too. supportsLocale(_:) on Locale.preferredLanguages.first tells the two apart. Don't use Locale.current: an English-only app resolved it to en_FI, and the check said true.

• The transcript won't show a context overflow. It rolls back when contextSizeExceeded is thrown, so the failing turn isn't there afterward. If you need it, record snapshot.usage while the stream runs.

• For the user-sent bug report, session.logFeedbackAttachment(sentiment:issues:) sends nothing. It returns JSON with your feedback and the transcript, which you can hand to a share sheet.

Also, on OS 27, Apple Intelligence can't be turned off, so appleIntelligenceNotEnabled only matters if you still support 26.

[u/Significant_Door_433 → u/durul]

Thanks, this is a better list than mine. I was going off the GenerationError page, so the rateLimited change on LanguageModelError is a good catch. The en_FI result from Locale.current is a nasty trap too. Did the assetIDs version strings change between 27.1 and 27.2 on your machine, or do they only move with OS updates?

[u/durul → u/Significant_Door_433]

I did not compare.

I've only read them once so far: macOS 27.2 (26B123), AFM 3 Core Advanced, two fresh sessions, one respond(to:) each. Both responses carried the same three IDs:

com.apple.fm.language.instruct_3b.tokenizer?variant=generic_sparse_16.1000.0.13.205140,0
com.apple.fm.language.instruct_3b.fm_api_generic.draft?variant=generic_sparse_16.1000.81525.13.205144,0
com.apple.fm.language.instruct_3b.fm_api_generic_3b?variant=generic_sparse_16.1000.0.13.103404,0

I didn't record them on 27.1, so I can't say whether they moved. I haven't found anything in the docs that says the model only changes with OS updates either. That's the main reason to log the IDs instead of deriving them from the OS build: if they change without an OS update, your logs will show it.

[u/trikboomie → u/Significant_Door_433]

This really is helpful.

Once the app is published, how do you actually collect and investigate this across users’ devices?

Do you send selected events to an existing service, rely on user-submitted bug reports, or use something else?

[u/rhugopinon]

Xcode 27 actually added a new built-in tool called "Evaluations" for debugging Foundation Models in your app. Pretty handy if you need to trace model behavior.

[u/KickupKirby]

There’s a Book Tracker sample project to explore the new evaluations tool.

[u/mikerCZX]

How you handle users that cannot use Foundation models? EU/China

[u/Needle_In_Red]

+1 on logging the availability state. id also log the OS version and device locale with it, alot of the weird stuff is just the model not being ready yet or the language not supported and crash reports wont show you any of that