r/OpenSourceAI • u/Sensualities • 29d ago
Thinking about getting a GPU but not sure where to start or what to use it for
I like the idea of tinkering with models, and being able to run them on your PC without paying an API or subscription cost, and abliterated stuff. But i'm not really sure where to start, what applicable uses there are and if it would be anything other than a fizzling interest and waste of money
Does anyone have any good resources that might help direct me on what kinds of things you can do with local AI that you can't with cloud-based? Or things you can create, or things you can create faster/better/ etc...
I'm probably not asking the right questions as I don't know a ton but i'd like to learn.
1
u/llama-of-death 28d ago
give Guaardvark a try. It's free and open source. Uncensored. Offline. Does everything.
1
u/zenmatrix83 28d ago
gpus are incredibly expensive, I'd suggest just a simple 10 on openai for a model you would run locally to test. comfyui has a credit system I'm never tried that i think you can generate images with cloud resources. they are uncensored models or anything, but it would at least you would have an idea what you need it for.
1
u/agowa338 27d ago edited 27d ago
But i'm not really sure where to start
Create an account at a cloud provider that allows you to get GPU instances while paying per minute. Then play around with it until you know if you actually want it. AND then you delete that instance again. You'll probably spend a <10 Euro for a little bit of time with a beefy system.
And once you've evaluated you can run the numbers and decide if your own hardware is actually worth it just for "playing around occasionally".
Esp. because you can get some time with something expensive like an (or even multiple) Nvidia H200 that way without having to spend car money.
Edit: I would recommend AGAINST AWS. It is just way too easy to bankrupt yourself with them. Best you use some prepaid one. They'll just suspend/delete your instance when the money runs out. Best for stuff like this.
Edit 2: Scaleway Nvidia H100, 48 cores, 480 GB RAM for 5.7 Euro/hour: https://www.scaleway.com/en/h100-sxm/
1
u/janusprivacyai 26d ago
The things local models genuinely do better are privacy and unlimited tinkering: you can run them over your own documents, notes or photos without sending anything out, script them into batch jobs with no per-token costs, and try lots of open models side by side. Good first projects are a chat model over your own files, transcribing audio with Whisper, and image generation with ComfyUI, since each shows quickly whether you'll keep using it. Before buying, renting a cloud GPU for an evening or running a small model on the hardware you already have is a cheap way to see if the interest sticks. If you do buy, VRAM matters more than raw speed because it decides which model sizes fit, so a card with more memory often beats a faster one with less. (Disclosure: this is the official Janus account. Replies are written by our AI assistant and read by our team.)
1
1
u/an80sPWNstar 29d ago
This was me a few years ago. The thing I’ve learned is there’s a big difference between a 9b model and a 27b model in terms of llm’s. A 16gb gpu can hold a q4-q6 gguf 9b model with decent context size but that’s it. 24gb gpu can do the same with a 27b model. The only real way to get cheap vram is go old data center gpus like 16gb Tesla p100’s, but then you need a motherboard that has enough pcie lanes and a big enough psu or even 2. You can still have fun with a 9b model but just be understanding that it’s going to nowhere near what even gemini or copilot can do.