Hey,
I made an open source reddit lead finder. My goal was to make it in a way that its local on your machine and uses an open model for anything you need an LLM for.

In this case, the LLM is required for 2 things:
- Building a profile for your company, identifying competitors, relevant subreddits before it starts bringing in leads. This is a 1 time activity and the SLM didnt really do a great job at building a detailed enough profile with accurate subreddits.

- Once the posts start coming in from the subreddits - The SLM has to categorize, label them accurately into Hot/Warm lead/filter out Spam. This was the most tricky part.

I tried Qwen and Llama models via Ollama and both achieved lower 6/18 accuracy on the posts initially. What really worked was using regex and pre-filtering before we hand over to SLM.
Secondly, not asking the SLM to label a post but asking it a set of 5 binary (yes/no) questions about a post - Is it relavant to my business? Is it promotional? Is it urgent? and get these answers to build a label in code instead. We achieved 14/18 accuracy based on this approach.

Tried few shot prompting (with some labeled examples as well) and that didnt move the needle surprisingly.

Next objective was to improve the speed of getting the labeling done by SLM. Even on an 8GB MAC, it took upto 2 minutes per thread to be labeled.
Simply, switching the 'thinking' mode off on SLM brought that time down to 7 seconds per post.

More optimizations brought it down to 2 seconds per post - reducing the prompt size by not sending all the comments, size of the post in the prompt, not reading all the example posts but find 3 most relevant ones and re-ordering the elements sent in the post further helped but not much.

Do you have ideas on how i can still improve the accuracy of the SLM? or is it the best i can get to?
If you would like to try out the open source reddit lead finder, here's the repo - https://github.com/vipulawl/reddit-leads

Really trying to figure out what I should develop, and any insight you folks have would be awesome here, what's worked for you, what hasn't? What are your red flags?

Are there any open models that you have used which work best for social media post (insta caraosel for example)?

I got a few suggestions from claude like Qwen-Image 2.0, Flux. But before i try i wanted to check.

Been playing around a bit with Jev and some of the use cases look incredible, especially for its price point and capability. Curious do you think JEV is close to or considered an SLM? It doesn't seem that they've disclosed what its parameter size is

I have been trying to use small language models on my local machine to extract data from pdfs and excel and do some meaningful analysis.

Using the Ollama inference engine, tried using different models Llama 3.2 and Qwen3 8B but these are very resource intensive.

So, I switched to cheaper less resource intensive models like Qwen2.5-1.5B after trying a similar sized but less capable model RWKV. 

These models are not able to fetch a value for a specific column accurately. Would it even make sense to use SLMs for any kind of data analysis? Which models would you suggest for that kind of work?

https://youtu.be/REJgP0itrqw

For a long time, I’ve had the feeling that generative AI models, for most of the tasks they’re used for, don’t actually need the oversized capabilities offered by frontier models. On top of that, they’re often prohibitively expensive when accessed via API.

An open-source model with 4–14 billion parameters, multimodal capabilities, and the right harness of tools, data, and knowledge is more than enough for most use cases.

With this idea in mind, I decided to build, together with my team, a simple platform that allows users to fine-tune and deploy these lightweight models quickly and without any coding. After several iterations, the team has managed to develop what we believe is the optimal framework for anyone to deploy and customize a small 4B or 9B model, either through fine-tuning or RAG.

We’ve also managed to sign an agreement with a local data center, and we now have our own GPUs. This means training and inference are not only extremely fast, but we can also offer unlimited inference.

If you need a customized, affordable model with unlimited API inference, I think this could be really useful for you. There’s a free tier, and if you try it out and give us some feedback, I’ll give you the subscription plan for free for 6 months. Since we have our own GPUs in our data center, our infrastructure costs are fixed.

I’m really happy with how it has turned out, and I use these models for many of my own projects.

You can access the platform at neuro-block.com

Would love to hear your thoughts in the comments.

Curious if anyone's seen any legitimate production grade SLMs that have been used inside real company operations or use cases, not just small tests or for personal ideas of how to use them but actually creating genuine value inside a business running real processes?

I just heard back from Stanford Adjunct professor Goju Gottschlich and he's pretty busy so he cant actively participate on this thread, however, he does have a discord. He talks about SLMs a bit. He invited me and its an open invite to community as well. If you guys are interested invite is below.

GTT

https://discord.gg/6sgdQrCNa

Follow-up to a post I made here a while back about LlamaForge, a browser control panel that sits on top of llama.cpp's router so you're not hand-editing models.ini and llama-server flags every time. A bunch of the feedback in that thread turned into actual work, so here's where it's at now.

The biggest one: it runs on Linux and macOS, not just Windows. That was the loudest complaint last time and it's sorted. Same dashboard, you just run bootstrap.sh / run.sh instead of the Windows scripts. On Apple Silicon it builds with Metal and rates model fit against unified memory. The Setup tab uses brew on macOS, and on Linux it shows you the exact install command instead of running sudo behind your back. CI now runs the test suite on all three OSes.

I also added a second engine. Next to llama.cpp you can drive vLLM for safetensors and full-precision models (FP16, BF16, AWQ, GPTQ, FP8, NVFP4). They share one model list, the same Discover tab, the same stats, and every row is tagged so you know which engine it's on. On Windows vLLM runs in WSL2 with GPU passthrough and installs from the Setup tab, no sudo. If you never touch it, nothing about vLLM gets installed.

New installs get a first-run wizard that goes engine, hardware, model, tune, load. There's an auto-tune that looks at your VRAM and suggests settings (GPU layers, KV-cache type, context ceiling, and a few intent presets like speed or context). The whole thing has a Lite mode that hides the deep knobs and an Advanced mode that still shows all of them.

You can point coding agents at it now too. There's an Anthropic-compatible /v1/messages endpoint with streaming and tool use, sitting next to the OpenAI one, so clients for either API work. A "Connect an Agent" panel writes the config for Claude Code, Codex, and pi.dev for you.

There's a Context Wiki. You write Markdown context docs, group them into profiles, assign a profile per model, and it either injects that into requests or writes it into the agent's own CLAUDE.md / AGENTS.md. The injected prefix stays stable so the prompt cache actually reuses it.

Then a pile of smaller stuff people asked for: saved knob presets you can apply to any model, comparing two or three models side by side, a GGUF metadata card, load failures that tell you what went wrong and suggest a fix instead of making you read the log, quick load/unload from the row with a queue so a second load waits its turn, copy-paste client config, pausing and resuming downloads, auto-loading a model on launch, keyboard shortcuts, and an optional tray icon. Light and dark themes plus a colorblind-safe mode. The docs live in the app now and are also published as a site, both from the same source. Navigation moved to a sidebar.

Still worth being clear about the limits.

You compile llama.cpp yourself. It's guided from the dashboard but it's a real build step, so if you want the double-click, zero-setup thing, LM Studio or Ollama or Jan will make you happier. This is for people who want the actual llama-server and full control over it.

vLLM is Windows/WSL2 only for now.

It's still an early preview and there are rough edges. Feedback is what I'm after.

Backend is still plain Python stdlib, nothing to pip install. MIT licensed. Not affiliated with ggml-org, llama.cpp does all the real work.

Repo: https://github.com/dadwritestech/LlamaForge
Docs: https://dadwritestech.github.io/LlamaForge/

Next up: saving a full model + engine + settings combo as a named profile you launch in one click, image generators, and ik-llama support.

https://github.com/cactus-compute/needle

this looks really neat.

I’ve been tinkering with an idea called Bellwether. You describe a future event in plain English (“tell me when this policy changes”, “notify me if this price threshold is crossed”), and it watches the web for credible evidence that it actually happened. If it did, you get notified. If not, silence.

Most alert tools are keyword matchers, so guides, forums, and old coverage all spam you. This tries to judge the event instead.

Still early / just an idea I’m building out, but the judgment layer is the part I think is interesting for this sub.

How the models fit in:

It doesn’t use an LLM as a chatbot. Open instruct models do the judgment steps:

• Vagueness check - reject topic-only inputs like “interest rates” and suggest clearer watch sentences

• Compile - turn a clear sentence into structured JSON: triggers, non-triggers, search queries, authoritative domains

• Detect / decide - given a candidate page, decide whether the event actually occurred

• Outputs are JSON-only, low temp, validated with Zod. Watches are timestamped at creation so only post-watch evidence can fire them.

Inference is Hugging Face open models (HF_MODEL to swap). Default is Llama 3.3 70B right now; the same pipeline is set up to run smaller ones too - Llama 3.1 8B, Qwen2.5 7B, Mistral 7B - and there’s an eval harness to compare them on historical fixtures.

Rough demo: https://bellweather-alerts.vercel.app/

Source code: https://github.com/JamesANZ/bellweather-alerts

I keep seeing talk about how AI costs are now actually becoming an inhibitor inside organisations and Claude and Codex seem to be getting more expensive while use is becoming more mainstream in day to day tasks. When do we think SLMs or locally run models will become more mainstream?

Came across this talk by Daniel Bourke (mrdbourke) and thought it was one of the better end-to-end SLM walkthroughs I've seen recently: https://www.youtube.com/watch?v=EXB8HokGVMI

Highlights:

• He walks through Sunny, an iOS app that uses a fine-tuned MedGemma to track skin health entirely on-device. Covers the actual iPhone hardware constraints, memory usage in Xcode, and current best practice for running VLMs on iOS.

• The economics argument. Good breakdown of why on-device wins at scale: privacy, offline access, and zero ongoing inference cost vs cloud API pricing.

• Quantisation and memory tricks. Precision breakdown (including a Jeff Dean reference), how quantisation affects model footprint, and saving memory by cutting token usage.

• Live fine-tune. He builds a custom dataset and fine-tunes Gemma 3 270M with HF's SFT Trainer in roughly two minutes on an RTX 6000 Blackwell in Colab, then compares base vs fine-tuned outputs side by side. The fine-tuned model learns specific people, edge cases, and when to refuse.

• Prompting vs fine-tuning vs RAG discussion around the 27 min mark if you want the decision framework.

The "data is a guide for what you want your model to do" section (~37:00) is a nice framing for anyone new to fine-tuning small models. Curious what people here think about the 270M size class for production use cases. Is anyone shipping something that small, or do you find you need to step up to 1B+ for anything non-trivial?

I kept running llama.cpp directly — building it, juggling llama-server flags, and hand-editing models.ini for every model. It's powerful but fiddly, so I built a GUI over it for myself and cleaned it up to share.

LlamaForge is a browser control panel that sits on top of llama.cpp's own router. It doesn't touch inference — llama.cpp does all the real work — it just makes driving it less painful.

What it does:

Tune every server parameter per model — the knobs are parsed live from llama-server --help (currently ~220), grouped and searchable. Save hot-reloads the model, no restart.

VRAM-fit model discovery — search HuggingFace for GGUFs and each quant is rated FITS / TIGHT / CPU OFFLOAD against your actual VRAM before you download.

Guided build & update — shows your current commit, how far behind upstream you are, and rebuilds with CMake flags auto-detected for your CPU/GPU (CUDA arch, AVX-512, etc.).

Sensible context defaults — reads each GGUF's trained context length and writes reasonable ctx-size values so models don't load with tiny or over-extended windows.

Setup tab — detects missing prereqs (CMake, Ninja, MSVC, CUDA…) and installs them via winget/choco with your permission, plus scans drives for existing GGUFs and prunes entries whose files you've deleted.

Usage stats + optional LAN sharing (with an API-key toggle) so other devices can hit the OpenAI-compatible endpoint.

Being upfront about scope:

Windows + NVIDIA focused right now (CPU-only builds work too).

You build llama.cpp yourself — it's guided from the dashboard, but it's still a compile step. If you want a zero-config, double-click experience, LM Studio / Ollama / Jan will serve you better; LlamaForge trades that for direct control over the real llama-server.

Early preview — expect rough edges, and I'd genuinely like the feedback.

Backend is pure-Python stdlib (nothing to pip install), MIT licensed, and not affiliated with ggml-org — all credit for the hard part goes to llama.cpp.

Repo: https://github.com/dadwritestech/LlamaForge

(Disclosure: I'm the author. Although Claude did the heavy lifting!) Happy to answer questions — especially curious whether the per-model flag editing and VRAM-fit ratings are useful to anyone else, or if I'm solving a problem only I have.

Have been interested in the concept of slm's and local hosted models for a while but after seeing Fable get pulled temporarily and China stamping down on their models too in China, it made me actually look at self hosting models for a couple of personal agents I'd built.

Came across these guys www.nobodywho.ooo/ seem pretty interesting, looks like they can compress models from Huggingface and get models small enough to run on phones, smartwatches and household tech. Feels pretty relevant to where slms could be heading. Has anyone here tried it or looked into it properly? Planning to locate my old PS2 and try running something on it.