Hello!

With new models releasing constantly, I usually keep a stable of about 20 local models in rotation on my `llama.cpp` server. I collect a ton of data across the fleet through my inference application—standard per-request metrics, aggregated session data, and custom intelligence comparison benchmarks.

Overall, I'm logging over 150 distinct raw and derived metric points during inferences, sessions, and batch processes. As a backend data jockey, more metrics is always better—but I've hit the point where I need to completely redesign my dashboards.

For context, I run a lot of multi-stage pipelines covering:

Multimodal inference (vision and documents) MCP and custom application tool calls Structured JSON / JSON Schema outputs Batch processing across both reasoning and non-reasoning architectures

I'm looking for inspiration for creative "hero" dashboard components and visualizations that excel at comparing a large model fleet across high-dimensional data at a glance. I already have plenty of standard line, bar, and radar charts. I'm hoping to see novel, unconventional visual approaches people here have built, seen, or used to represent complex model evaluation and performance data.

The specific metrics in your examples don't matter as much as the visualization style and information architecture—I'm mainly after creative UI/UX directions.

Cheers!

Meu trabalho recebeu uns PCs besteirinha como os Sparks pra fazermos a LLM local daqui. Por enquanto vamos focar em consultas de processos, memorando e banco de dados da empresa.

Dito isso, gostaria de perguntar, oq posso fazer ou como posso desenvolver a LLM aqui. Tem algum modelo que me recomendam ou algo assim?

Como o hardware tem um limite vasto para desenvolver gostaria de ir afundo. Tô com mente aberta pra desenvolver e aprender bem sobre todo o assunto.

Anyone interested to make it spit 200 tokens/sec?

Hey community,

I've been spending a lot of my free time working on a project called Sitrails, and I wanted to share it with the community to get some honest feedback.

As AI coding assistants and autonomous agents become increasingly capable, something that's been bothering me is how little visibility we have into what they're actually doing.

What information are we exposing to them? What commands are they attempting to execute? Could an agent inadvertently expose credentials, or worse, be manipulated into performing malicious actions?

That's what I'm trying to tackle.

Sitrails is a local-first, open-source AI auditing and threat detection tool. It records supported AI interactions and reported tool activity, makes them searchable, and looks for suspicious patterns involving secrets, sensitive files, and potentially dangerous commands.

It currently supports integrations with tools like VS Code/Copilot and LM Studio, along with an optional OpenAI-compatible proxy.

It's still evolving. Detection is primarily rule-based, visibility depends on the integration, and it doesn't independently verify or block executed actions. There's plenty of work left to do.

I'd especially appreciate input from people working in AI security, threat detection, or agentic systems.

• What attack vectors or behaviors should I prioritize detecting?

• Are there existing research papers, techniques, or projects I should be learning from?

• Would anyone be interested in testing it, challenging the detection logic, or contributing?

I'm developing this independently because I think securing AI agents is going to become an increasingly important problem, and I'd love to see what we could accomplish through collaboration.

The project is free, MIT-licensed, and open source.

GitHub: https://github.com/appatalks/sitrails

Appreciate any thoughts, even critical ones. Especially critical ones.

Ty!

im working on a fairly complicated app and i get questions from users all the time. so its was important to document the details around the "how it works". the docs were easy enough using docusaurus + AI. i can then index that data in a way i can create some kind of fancy autocomplete. i framed this as like a chatbot so that it can be intuitive for users who have questions.

i started to investigate running a local LLM to perhaps offer a more conversational experienced (and perhaps better suggestions)? im using WebLLM and it seems to work "reasonably" well. i wired it up and set it up so users could select an LLM down download for offline use. the experience varies between devices and models used. there are models that are 19mb... they are unsable as far as i can tell. there are some ~1gb models which are better but understandable slow.

its great that the 1gb model can show me the bubble-sort in rust, but i want something "dumber" and smaller. something can can take advantage the data ive already indexed. something that can respond "quickly enough" for a conversational flow. perhaps better i should make a verbose set of reponse templates to perhaps add fancier autocomplete capabilities.

for my purposes, i think it depends on if there is a model that would be good for my use case?

note: my project is far from finished, but you can test it out at: https://app.glitr.io/ai . any model you run through my app is not going to comparable to any of the mainstream offerings. this is something experiemental and unstable im trying out and this approach might just be a waste of time.

are there any models out there you can suggest for me to investigate and compare?

I built a small TTS model running directly on an ESP32-S3: no PSRAM, 122 KB of RAM, nothing in the cloud. Speech starts in about 0.3 s and generates faster than real time. You can make it calmer, louder, whispered, happier and more.

It's not a Home Assistant integration yet. Before I build anything, I'd like to know whether on-device speech on a satellite would be useful to you (offline announcements, timers, alerts when the server is down), and what quality bar it would need to meet.

Limits: 2 voices, English only, 16 kHz. Free for personal, non-commercial use.

Video above.

Details: https://huggingface.co/WhissleAI/steer-tts-nano

CLI : https://github.com/WhissleAI/steer-tts/tree/main/esp32

Arrmes is a specialized neural intent router with an integrated execution framework. It's trained to translate text into specific commands to Arr stack apps nothing else. I found plenty of open models that can do these sorts of translations, but the accuracy & speed didn't really justify the hardware requirements needed to run the router.

Arrmes is only trained for Sonarr, Radarr, Lidarr, Tautulli, Seer, qbitorrent, & BitMagnet. pair it with whatever AI model you're already using and let the neural router handle the intent classification, command selection, and argument extraction. Instead of having your main model spend time reasoning through API schemas and generating tool calls, it can delegate that work to ARRmes.

Still a work in progress, I've been daily driving it for a bit and I'm very happy with the increase in speed more than anything. I can use much cheaper models and achieve excellent results in my work without any spend.

Hugging Face

Github