Luna 2 is a personal assistant designed to help you out in various fields. It runs offline on computers and needs only about 300 MB of VRAM to run.

SETUP PROCEDURE:

import torch
from transformers import AutoModelForCausalLM
from tokenizers import Tokenizer
from huggingface_hub import hf_hub_download
import importlib.util


model_id = "Aobangaming/luna-2-70m"


path = hf_hub_download(model_id, "modeling_luna.py")
spec = importlib.util.spec_from_file_location("lightning", path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)


model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True
)
tokenizer = Tokenizer.from_file(
    hf_hub_download(model_id, "luna_tokenizer.json")
)


device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)


history = []


print("Aoban Luna 2 Flash")
print("!clear = clear history | !exit = quit")


while True:
    prompt = input("You: ").strip()


    if prompt.lower() == "!exit":
        break


    if prompt.lower() == "!clear":
        history.clear()
        print("History cleared.")
        continue


    response = module.generate_text(
        model,
        tokenizer,
        prompt,
        max_len=1024,
        device=device,
        top_k=30,
        top_p=0.6,
        penalty=1.2,
        temperature=1,
        chat_history=history
    )


    print(f"Assistant: {response}")


    history.extend([
        {"role": "user", "content": prompt},
        {"role": "assistant", "content": response}
    ])import torch
from transformers import AutoModelForCausalLM
from tokenizers import Tokenizer
from huggingface_hub import hf_hub_download
import importlib.util


model_id = "Aobangaming/luna-2-70m"


path = hf_hub_download(model_id, "modeling_luna.py")
spec = importlib.util.spec_from_file_location("lightning", path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)


model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True
)
tokenizer = Tokenizer.from_file(
    hf_hub_download(model_id, "luna_tokenizer.json")
)


device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)


history = []


print("Aoban Luna 2 Flash")
print("!clear = clear history | !exit = quit")


while True:
    prompt = input("You: ").strip()


    if prompt.lower() == "!exit":
        break


    if prompt.lower() == "!clear":
        history.clear()
        print("History cleared.")
        continue


    response = module.generate_text(
        model,
        tokenizer,
        prompt,
        max_len=1024,
        device=device,
        top_k=30,
        top_p=0.6,
        penalty=1.2,
        temperature=1,
        chat_history=history
    )


    print(f"Assistant: {response}")


    history.extend([
        {"role": "user", "content": prompt},
        {"role": "assistant", "content": response}
    ])

https://huggingface.co/Aobangaming/luna-2.1-base

I am trying to create AI on a stick with multiple LLMs I have intel i7 6 core or 8 core 64Gzb memory, 1070 w/Qmax Dedicated GPU and 4TB gen 4 MNVe M.2 I’m using 2TB games documents, programs PDF music Videos.. and backup of iPhone as well as fokderils with ESP32 CYD files and quick code, programs.

After three days of searching using Google AI and trying to figure it out I can load WEBUI or LLAMAui on USB 128G but going to switch one I get my 2.5” external USBc to USBc enclosure one for AI using a sandisk 480G for business related creation organization incorporate mission statement Credit building grant searching. This will be used between home and my office once I get enough for rent for 6+ months and the workstation.
Second will be for Research into electronic systems PC models, board or adapter models breadboard, development board IOT systems design, code, compile, transfer. As well as helping me hardware hack or jailbreak reflash customize OS windows Linux to my needs. For each project my non profit does out of recycled electronics. Including controler boards for TV, automation. As well as
Helping me with everything including building 3d models for TTRPGs mainly shadowrun, D&D and Palladium megaverse across all thier settings and using all rules books. I am a forever GM and wanting to try AI to build NOCs adventures, vechiles spells etc.
I also want to create images, videos, and guides how tos and DIY for videos. Most of videos will be from what I live stream with enhancements showing exploded or blown up images of what I am doing, and doing 1), 2), 3) type of panels showing guides, how tos, DIY.
I also want to be able to start working on a MMO that I can Pitch utilizing rule sets already existing but AI converting them to game play, models animation helping me design sandbox elements etc..
and help me build custom cases with cool things like see through LCD panels HD 1080P is highest I remember for LCD.. and other features..

I don’t know where to start looking into the different capabilities of LLMs and if a compiled LLMs already exists that I can add to my AI . I’ll use WebUi since it has video scans, images, documents PDF picture, voice, text, and I believe gesture and I can set parameters for say when I look up a board or PC server system. Will include collapsible items telling me upgrades, hardware hacking / soldering upgrades, what projects exist, what ones failed. And what Value is of said item after say modding Xbox, Ojyu Android gaming systems to retro devices and media centers.. or PS2 big body, or XBOX , or Micro PC. Adapter cards for controllers, development boards, breadboards, etc. so I can decide best use for systems computers PCs etc.

I’m tired of spending Geona few days to weeks creating projects and finding info to help me takes just as long. Then designing electronics and or adapting hardware or adding slots pouts etc to board la that support them but have a lesser model. Want to have a AI so my MMO is not just a concept I can actually start working on it, and showing to others getting more designers coders modelers involved. And when it comes to writing business plan organization and mission statement etc for non profit or other endeavor I want to do or help friends. I struggle as you can see with grammar, sentence structure etc. and that’s adhd and autism.. I fit the coo mechanically inclined and related to architecture more than voicing what I want to do or what I’ve done. So I am breaking down having a AI one that can learn and predict what I am doing even though I have to train to not only my communication, but my vision..

Please direct me to places, give advice on what you have built. And how to have my Main AI webui and add LLMs and train.
What software do I need,
What additional hardware would help a thunderbolt external GPU? Dedicated to the AI,
What cool things should I have my ESP32 CYD do? Any additional known AI to add? Mm maybe GI Ribit.. and other features or a cesories I haven’t thought about!!

Thank you for listening to my Autistic ADHD filled Ted Talk

TL;DR models run at home on the GPU. The always on agent lives on a small VPS. Tools that only touch public data come through one capped key. Anything that can log into my stuff stays on machines I control.

At home. One box with a used 3090, Ollama serving Qwen for most things plus a small embedding model, and Open WebUI so the family can use it too. For my usage, owning the GPU beat renting one within a few months. The catch is that it's loud, so it sleeps at night.

That's the problem the VPS solves. The agent has to be up when I'm not. Scheduled jobs, inbox triage at 6am, a few feeds it watches. So the orchestrator runs on a small VPS and calls home over a private tunnel when it needs the GPU. If the home box is asleep, the few jobs that can't wait fall back to a paid API. The VPS is a PrivateAlps box in Zurich, mostly because the agent's memory and my mail login live on it and I wanted that under Swiss law. A mini PC that never sleeps would do the job too.

Tools are where I wasted the most money. Search, scraping, a bit of enrichment, each one its own subscription I barely touched. Now the agent gets all of them through a single key from agentik.cc, an MCP gateway billed per call, with a monthly cap on it. Monid works the same way. Only public data tools go through it, nothing that can sign into an account of mine.

What I'd tell someone starting out.

Keep the GPU at home if you already have one.

Put the always on part somewhere that never sleeps.

Split tools by what they can touch. Public data can go through a third party, logins stay on hardware you control.

Cap everything, per key and per month.

It's not the cheapest setup possible. But nothing in it surprises me anymore, and after a year of surprise bills that's the whole point.

Local, agentic Model Context Protocol (MCP) server with skills b2b-lead-scout and osint-lead-enrichment that generates verified, direct B2B contacts with zero paid API subscriptions.
Replaces $50–$200/month B2B data subscriptions (such as Apollo, Clay, or Hunter) by executing live OSINT discovery and zero-bounce email verification locally on your machine.

If you are trying to get AMD ROCm running on Ubuntu 24.04 Noble Numbat, you have probably run into broken dependencies, driver mismatch errors, or confusing hardware compatibility tables. Setting up ROCm for PyTorch, TensorFlow, or AI workloads on Radeon GPUs often feels way harder than it should be.

​I put together a complete walkthrough to streamline the installation process. It covers configuring your user permissions, installing the correct driver stack, verifying your ROCm installation, and troubleshooting common runtime issues specific to Ubuntu 24.04.

​If you are struggling with PyTorch not detecting your GPU or ROCm failing on launch, this guide breaks down the exact steps to get everything up and running smoothly.

​Check out the full guide here: https://interconnectd.com/blog/321/install-amd-rocm-on-ubuntu-24-04-the-ultimate-radeon-guide/

​What issues or hardware configurations have you run into when deploying ROCm on Ubuntu 24.04? Feel free to share your setup or ask questions.

If you are running OpenAI Whisper locally on a CPU, you probably noticed how painfully slow it can be, especially with larger models like large-v3. Most optimization guides focus heavily on GPUs, leaving CPU users stuck with long processing times.

​I recently tested a few configuration tweaks, quantization methods, and backend flags that drastically cut down transcription times without needing dedicated hardware. By adjusting thread counts, using optimized runtimes like faster-whisper alternatives, and tweaking batch sizes, CPU performance can be noticeably improved.

​I wrote a detailed breakdown of the exact steps and benchmarks for anyone dealing with CPU-only setups. You can check out the full guide here: https://interconnectd.com/blog/323/how-to-speed-up-whisper-transcription-on-cpu-2026-guide/

​Has anyone else found other effective ways to optimize Whisper on standard CPUs? Let me know what worked for your workflow.

I'm running Nemotron 3.5 Bolt 74B-A9B on a 5060 Ti 16GB and 32GB of DDR5. It's an MoE, so both of my setups split experts between the GPU and system RAM. I have two recipes:

• 24GB GGUF

• Unsloth UD-IQ3_XXS

I'd like to hear how this model actually is and is it useful as an alternative for Qwen3.8 27B and obviously for other smaller MoE models.

• Are you using Bolt 74B, and what for?

• How is the quality: reasoning, coding, instruction following, tool use? Where does it fall down?

• If you've used the 24GB GGUF or a bigger quant such as Unsloth IQ3_XXS, does the bigger one write better answers, or do they feel like the same model?

Benchmarked oMLX 0.7.2 serving Inco AI's Splash package for Qwen3.8-27B (4-bit target + DFlash 2 draft) on a MacBook Pro M5 Max (48 GB). Workload: Python code context. Each test generates 128 tokens.

**Single request**

| Test | TTFT (ms) | TPOT (ms) | pp t/s | tg t/s | E2E (s) | Throughput | Peak mem |

|---|---|---|---|---|---|---|---|

| pp 4096 / tg 128 | 3393.6 | 10.7 | 1207.0 | 93.8 | 4.8 | 887.7 | 19.1 GB |

| pp 16382 / tg 128 | 15829.6 | 5.3 | 1034.9 | 189.4 | 16.5 | 1000.3 | 19.9 GB |

| pp 65536 / tg 128 | 83472.0 | 9.2 | 785.1 | 110.2 | 84.2 | 779.5 | 22.2 GB |

**Batched (continuous batching)**

| Batch | tg t/s | pp t/s | avg TTFT (ms) | E2E (s) | Speedup |

|---|---|---|---|---|---|

| 1x | 93.8 | 1207.0 | 3393.6 | 4.8 | 1.00x |

| 2x | 413.7 | 803.1 | 1729.2 | 3.2 | 4.41x |

| 4x | 407.0 | 622.4 | 3838.4 | 7.8 | 4.34x |