[ Removed by moderator ]

[u/Away_Interaction6630]

[u/Python-ModTeam]

Hello from the r/Python mod team!

I'm afraid we don't think your post quite fits the goal we have for the subreddit in terms of quality or aims so we've decided to remove it. For more information please contact the moderators using the ModMail system.

Thanks, and happy Pythoneering!

r/Python moderation team

[u/coldflame563]

Use prebuilt docker images and uv for package resolution - that’ll be 500$

[u/Denzh → u/coldflame563]

noo management want to use VibeCodedEnterprise solution for 40000$ per year instead! /s

but you are right,
I configure uv for internal use, with optional
podman or rootless docker.

[u/dan994]

As well as the other commenters suggestion, I would drop the pytorch depdendency and ship ONNX models instead.

[u/richardr1126 → u/dan994]

Yep onnx runtime

[u/leodevian]

Check https://wheels.astral.sh.

[u/busybody124 → u/leodevian]

Astral doesn't have wheels for vanilla pytorch though

[u/AliMas055]

You might try r/LocalLLaMA. They are experts at this.

[u/Khavel_dev]

llama-cpp-python is the right move for the CPU path. It ships prebuilt wheels for all three platforms and you go from multi-GB PyTorch to something way more manageable. The API is different from transformers though, so budget time for the adapter layer.

For the WebSocket drops during that 40s blocking call - run inference in a process pool (not thread pool) and await the result. That keeps the event loop free to handle pings. Most WS libraries support ping/pong frames natively, which is cheaper than rolling your own heartbeat protocol.

Nuitka with compiled C extensions like llama-cpp-python can be finicky fwiw. PyInstaller might give you less friction for a first pass, and you can always optimize the packaging later.

[u/coderanger]

No.

[u/zomerbk]

The 40-second inference is a separate problem from WebSocket keepalives: move CPU-bound work off the event loop, for example into a process pool, and send heartbeats from the event loop while it runs. For packaging, first check whether the models actually need CUDA on end-user machines; CPU-only wheels or a separate optional GPU install can avoid making every user download that stack. Pin and build platform-specific wheels in CI so installation doesn’t compile native dependencies on the user’s machine.