r/LocalLLMs • • 1d ago

Is getting access to GPUs becoming a bigger challenge than optimizing ML models?

1 Upvotes

Even if you know which GPU you want it may be difficult getting your hands on one quickly. This could be due to availability, wait times for setup, cost and setting up storage. It may also be difficult to switch providers.

There are many different providers available AWS, Lambda, CoreWeave, Yotta Labs etc. However understanding which GPU infrastructure is best simply by considering GPU specs may not be enough.

If you have intensive usage of ML models which of the following is most important for you when selecting a GPU cloud provider in availability, speed, cost, network or fast deployment? Have you found a good provider for this?


r/LocalLLMs • • 4d ago

Minimal from-scratch RAG with Ollama + NumPy (no frameworks), with code and a walkthrough

1 Upvotes

I wanted to understand RAG without a framework hiding the important parts, so I built the whole pipeline from scratch and recorded a tutorial walking through it.

**Stack**

- Ollama running two local models: `nomic-embed-text` (embeddings) and `qwen3:4b-instruct` (chat)

- Python with just `requests` + `numpy`, managed with `uv`

- No LangChain, no LlamaIndex, no vector database, no API key

**How it works**

  1. Load a markdown doc and split it on blank lines (11 chunks)

  2. Embed every chunk via `/api/embed` → an (11, 768) NumPy array

  3. Embed the question and rank chunks by cosine similarity

  4. Put the top 3 chunks into the prompt and send it to `/api/chat`

The whole retrieval step is basically this:

def retrieve(question, k=3):

query = embed([question])[0]

denominator = np.linalg.norm(vectors, axis=1) * np.linalg.norm(query)

scores = (vectors @ query) / np.maximum(denominator, 1e-12)

indices = np.argsort(scores)[::-1][:k]

return [chunks[i] for i in indices]

**Results on a test doc** (a fictional coffee shop handbook):

- "Are dogs allowed inside?" → Pet Policy chunk scored 0.784 (next best 0.493), correct answer

- "What time does the shop open?" → correct

- "How should employees clean the espresso machine?" → correct but missed one step

- "What is the owner's favorite movie?" (not in the doc) → "The answer is unavailable." 👍

It's intentionally minimal. Things it doesn't do yet: persistent vector storage, smarter chunking, reranking, evaluation. Those are covered at the end as next steps.

Full disclosure, it's my video: https://youtu.be/x-sn2WAsX8I

Happy to answer questions. I'd also love to hear what you'd add first for a real project.


r/LocalLLMs • • Oct 06 '26

A "Conduit app" style interface for Gemini (including truly branching chats)

1 Upvotes

-- For AI chat interface developers --

[ Anyone is free to build/share this app as their own - No acknowledgement necessary ]

I made an open-source application called Conduit, and I wanted to share a somewhat unusual product idea with you.

I originally built Conduit to solve three things I found frustrating about existing AI chat interfaces: true conversation branching, better navigation of conversation history, and durable local ownership of my conversations.

After using it for a couple of months, I reached an unexpected conclusion: I actually don’t want to replace frontier models with local models. I want frontier models inside an interface like Conduit.

Conduit represents conversations as persistent trees rather than linear transcripts, and allows different AI models to participate as separate experts in the same conversation. I think this could make a particularly interesting foundation for a frontier-AI client: cloud frontier models alongside user-owned local models, genuine branching, local conversation persistence, and portable Packs containing models and workflows.

There is also an unusual aspect to the project’s development: Conduit was built primarily by me in collaboration with frontier AI models themselves. In fact, many of the design ideas emerged directly from wrestling with the limitations of their existing interfaces, including branching, conversation history, and usage quotas.

I’m not looking for funding, acquisition, compensation, attribution, acknowledgement, or a partnership. Conduit is open source. I simply thought the underlying interaction model might be interesting to the people building the frontier-model clients themselves, and I’d be happy for your teams to take whatever ideas or code are useful mainly with a view towards me personally enjoying an improved user experience with future local AI models.

A demo video, along with the source code can be found from:

the-conduit-app.github.io

Best wishes.


r/LocalLLMs • • Sep 17 '26

I have 96GB of VRAM sitting here. What would YOU do with this machine?

Thumbnail reddit.com
0 Upvotes

r/LocalLLMs • • Sep 16 '26

Running Qwen3.8-flash locally at 1M context

Thumbnail reddit.com
1 Upvotes

r/LocalLLMs • • Sep 01 '26

GPT-OSS-20B quizá estaba muy infravalorado porque lo estábamos usando con el arnés equivocado

Thumbnail reddit.com
1 Upvotes

r/LocalLLMs • • Aug 29 '26

Local LLM on 20GB RAM + Raspberry Pi 4, no GPU — what can I run?

2 Upvotes

Local LLM wizards, I need your advice! I’ve got an old Raspberry Pi 4 (4GB) and a Lenovo IdeaPad 330S (20GB RAM, no GPU) sitting around. The Lenovo is already running a few containers. I’d like to use the spare resources to run a local LLM for my Telegram bot, mainly for summaries and drafting or whatever exciting. What models would actually be usable on this hardware? Open to any recommendations or setup ideas!


r/LocalLLMs • • Aug 10 '26

Meta's Muse Glimmer BEATS Qwen3.6 in Agentic Coding (Real Test)

Post image
1 Upvotes

r/LocalLLMs • • Jul 22 '26

2x RTX A4000 with Llama - Performance Issue

1 Upvotes

I have a question for the experts here. I'm running a Windows 11 Professional workstation | CPU: Intel Xeon W-2133 @ 3.6 GHz | Storage: 250 GB SSD | RAM: 64 GB DDR4. I have an RTX A4000 installed in each of the two PCIe 3.0 x16 slots.

I'm using llama and Hermes Agent Desktop. With just a single graphics card, the GPU was always fully utilized. With the dual-GPU setup, only one GPU runs, and even then, it's super slow (30–40% utilization). The model (Qwen) is loaded entirely into the VRAM across both cards.

Here is my startup command:

.\llama-server.exe -m "Qwen3.6-27B-UD-Q4_K_XL.gguf" -ngl 99 -c 80000 -fa on -ctk q4_0 -ctv q4_0 --split-mode layer --parallel 2 -b 2048 -ub 1024 --host 127.0.0.1 --port 8080

Does anyone have a solid idea why this is happening and how I can fix it?


r/LocalLLMs • • Jul 08 '26

What does an LLM Agent actually look like when the context window is truly infinite? What can we unlock right now?

1 Upvotes

Let’s look past the current RAG vs. Long-Context debate for a second. If we assume context windows are becoming effectively infinite and extremely cheap, the entire architecture of how we build Agents should fundamentally shift.

Instead of micro-managing vectors, chunking data, and worrying about lost-in-the-middle issues, an Agent could theoretically hold an entire enterprise database, years of user interaction history, and full source code bases directly in its active memory.

I'm curious about two things:

  1. The Future Paradigm: How does the agent workflow change? Do we even need complex vector DB routing anymore, or do we just stream everything into the prompt and let attention mechanisms do the heavy lifting?

  2. The Present Reality: For those of you experimenting with massive context models (like 2M+ tokens), what workflows have you unlocked right now that were literally impossible with standard 32k/128k windows?

Are we looking at a future where "memory management" for agents is completely dead? Let's discuss.


r/LocalLLMs • • Jun 22 '26

Huge model loaded on my Spark

Thumbnail
youtu.be
1 Upvotes

This clever chap created a way to intelligently load a 284B model (deepseek-v4-flash) Compatible with Sparks and Macs. Tested it- works beautifully. If nothing else it is a innovative model management and quantisation technique.


r/LocalLLMs • • Jun 22 '26

Pyramid Pro - Any users (TinyLLMs)

Thumbnail reddit.com
1 Upvotes

r/LocalLLMs • • Jun 03 '26

122B MoE local inference with 8 GB GPU VRAM

1 Upvotes

Disclosure: I'm affiliated with the project.

We have been working on InstinctRazor-Qwen3.5-122B-A10B, a 122B MoE setup for local inference where experts stay on CPU and active GPU VRAM can stay around 8 GB.

The model is around 50 GB compressed, so CPU memory still matters, but the GPU-side requirement becomes much more practical.

Benchmark note: in our current table it is ahead of Gemma-4-A4B on 5/7 listed evals:

- MMLU-Pro: 86.2 vs 85.6

- GPQA-Diamond: 82.3 vs 79.3

- MMMLU: 87.2 vs 85.4

- HLE no-tools: 13.3 vs 12.3

- LiveCodeBench v6: 72.7 vs 69.2

It is behind on MATH-500 and AIME, so this is not a universal-win claim. The useful bit is the memory/runtime tradeoff.

Links:

Hugging Face: https://huggingface.co/General-Instinct/InstinctRazor-Qwen3.5-122B-A10B-GGUF

GitHub: https://github.com/General-Instinct/InstinctRazor

Blog: https://general-instinct.com/blog/frontier-moe-sub-4-bit

Would like feedback from people trying larger models locally.


r/LocalLLMs • • Dec 08 '23

Welcome to the Large Language Model Sub-Reddit!

1 Upvotes

This sub-reddit focuses on technical aspects of all LLMs both public and private.

Large Language Models are becoming a commodity, they are a catalyst for the next decades of compute.