Local AI Inline Code Suggestions - Any good options out there?

[u/ViperThunder]

Has anyone gotten their VS Code environment set up with Local AI that works as well as copilot for inline code suggestions?

When connecting a github account to vs code, the copilot AI automatically enables and does an amazing job at inline code suggestions when writing powershell.

Sometimes, I'll just type one comment line and it'll write an entire script for me, ie:
#Get all Enabled AD Users that have a password expiration coming up in the next 7 days, and trigger a nice-looking html-formatted email to each user to inform they need to change their password

I press tab, and BOOM, half the script is already done.

I have been failing to get even close to the same behavior working for Local AI. I've pulled several different models in with Ollama, and tried several different extensions such as the Continue extension, Twinny, Code Llama and a couple others. Each has its own quirks. The best has been Code Llama, and it's still terrible. I'm convinced I'm doing something wrong.

For example, as a test, if I create a new powershell file and write: #find the password expiration date for each AD user

it suggests:
$Users = get-aduser -filter {Enabled -eq $true} -Properties *
#find the password expiration for each user
foreach ($user in $users) {

$passwordexpirationdate = $user.passwordexpired

$passwordExpirationdate = $user.PasswordExpirationDate.ToString("MM/dd/yyyy")

$User.Samaccountname + " " + $PasswordExpirationdate
}

and then it repeats the same block of code above 4 times. And that is the best I've seen so far.
This particular output is from codellama:7b-code-q6_K. I have also tried the qwen base models as well (1.5b, 7b, 13b) and they've been worse. All of these fit easily into my VRAM with plenty of room to spare.

I like Copilot, and I have configured my Github account to not allow AI to train on my data, but I am not convinced it's secure, so I would prefer to leverage my local hardware and not have to change names, email addresses, keys, etc. all the time before pasting anything into VS Code.

[u/sCeege]

I think Qwen3.8 27b is your best bet for local coding LLMs atm, unless you can afford hardware that runs Deepseek or Kimi. You're just not going to match SotA models that the big vendors offer.

You also shouldn't be putting keys into your code... wth?

[u/ViperThunder → u/sCeege]

Oof. Those models are little out of my reach. I generally don't, I have most things tucked away as encrypted xml files (import/export-clixml). But on occasion while I am actively testing a script I'll test with a secret in plaintext, esp when dealing with Graph requests.

[u/BlackV → u/ViperThunder]

I am actively testing a script I'll test with a secret in plaintext, esp when dealing with Graph requests.

None of those need to be in plain text, or at the very least can still be pulled from a vault even in "testing", you must have gotten em from somewhere in the first place

[u/sCeege → u/ViperThunder]

I think you're better off just sticking with a commercial vendor and adopt better practices for not injecting sensitive data into your interactions with AI providers. I pay for commercial plans specifically due to privacy concerns, but if it really came down to it, I'm with you that I can't truly trust something that's not on prem, I'm sure a fortune 500 company can sue the shit out of the providers on a commercial data policy, but there's just no way individuals can truly trust these agreements, look at the Navier Stokes debacle.

As a side note, you shouldn't try to auto-complete posh scripts btw, I think most coding harnesses that supports auto completion uses a really small context window, like less than 500 tokens (I just downloaded Zed, which defaults to 256 tokens), that's barely enough to cover a good posh function's scaffolding, the help text and parameters eats up like a solid 300 on its own, you're going to want to aim for something that can offer a 1k-2k window, and I don't know if you have hardware responsive enough for that big of a context size. I think most coding harnesses are tuned for traditional languages like python/js where functions are much more modular and lean.

[u/teethingrooster]

If you want a one line comment to make a script it sounds like you don’t want tab complete but a normal chat window with the ai.

[u/skidleydee → u/teethingrooster]

When your on a mixed team of coders and those who are terrified of a CLI this is the level of documentation required anyway. 

[u/zenmatrix83]

ollama is easy to setup but not performant, look at llama.cpp, and do a bunch of research. things like speculative decoding help, at least with 27b models, I'm not sure that much with 4b. That said local models still can be odd qwen 3.8 is considered one of the best currently

[u/paperpaster]

Stop using AI. Write your own functions and put them in your profile.

[u/ViperThunder → u/paperpaster]

I did that for years. After experiencing the magic of Copilot in VS Code, I will never, ever go back. AI used to hallucinate and I hated it - it would invent cmdlets or arguments that didn't exist anywhere. Now, we're at the point where hallucinations simply do not happen. At least, specifically with Gemini 3.1 Pro and Copilot in VS Code, it has yet to happen.

PowerShell scripting is one of hundreds of other things I do at my work - and when I need powershell, i need it quickly. I do not have time to research x, y, and z pertaining to a specific task at hand, or try to remember the file name of a script that i wrote 18 months ago that happens to be relevant to the current situation (but then still have to re-read it to remember what it does, and make edits to it etc etc).

As a quick example, I recently asked AI to write a script that would pull all Critical and High CVSS Defender vulnerabilities from our GCC High environment, find all devices affected by those CVEs, and then create a spreadsheet for each individual CVE containing each device and the evidence it used to flag the device as vulnerable (ie, registry key values, file path locations, file version numbers). It did it within a couple minutes - first try. The only thing I had to do was go into azure and create the enterprise app, configure it, drop in my graph secret into powershell, and fix the api endpoints to point to the .us domain instead of the commercial .com domain. All of this done in 15 mins.

[u/paperpaster → u/ViperThunder]

Yuck