Are you using Private AI yet?

[CptKrf]

All the time now. I have played with locals under Jan and Ollama for a year or so for coding, but unless the request was kept fairly simple, the odds of a function not having errors was very low. But, with the addition of a decent GPU and the fairly new gpt-oss-20b, that has changed. I use it to code very complex scripts and they almost always work if I described my needs with sufficient clarity. A gotcha is that it can give back code that is very verbose in terms of comments unless it is told to keep them to a minimum. Many days I go without bothering to use the online versions at all.

The main drawback to a local LLM, unlike its online cousins, is that once you exit then it will not remember what you have done. But in all, I like the way that private AI is developing. Especially with all the news of on-line AI and their billions out and pennies in, almost assuring that considerable subscription fees are inevitable.

 

[CaffeineAddict]

Offline AI is useless compared to AI services unless you have super computer, e.g. 8 GB+ VRAM and 64 GB+ RAM and equivalently strong CPU.
Minimum 70B model for better results.

In any case AI for coding should be used only for 1-3 lines not for whole compilation units or scripts, and even then you need to verify what the output does and not blindly copy into your code.

Otherwise you end up with a bunch of unopimized bloat that is harder to fix and debug the more of it is AI generated.

 

[CptKrf]

I do have a fairly high end system. 64gb, 24gb vram and a Ryzen 7. I was about to change to a Ryzen 9 but I have discovered that only the GPU is significant. From my experiments it would appear that a 486 could be almost used if somehow it would drive a 7900 or 5090 GPU. Changing CPUs has made no difference. My next upgrade is a MB that will handle more ram so I can move upwards from a 20b LLM.

But, I have found that even the 20b level makes good and tight code - IF I am really specific in what I want produced. I see no indication of bloat (except in the comments) as long as it can follow what I want and I do far more than 1-3 lines. ChatGPT does fine also, of course, except for always suddenly presenting the message that "Your time is up. Come back in 6 hours."

All that being said, of all the local LLMs that I have installed, only the gpt-oss-xx version is consistently correct, unless the request is for really simple functions or the like. Over ask and they fall apart. Except for my version of Deepseek. It doesn't hallucinate as much as go on a meth-fulled bender. Really weird.

 

[Trynna3]

I wouldn't put absolute trust into AI. Always stay vigilant. On repeated occasions it advised me wrong, even basic laws of physics were a problem for it. It is built to please you with answers.
The other day I saw a meme where somebody asked AI what are dolphin nose potatoes. And the AI elaborated that they are potatoes in a shape of dolphin nose, outside crispy and fluffy inside.
Those frequent in the kitchen know where this went wrong LOL.

 

[Trynna3]

Offline AI is useless compared to AI services unless you have super computer, e.g. 8 GB+ VRAM and 64 GB+ RAM and equivalently strong CPU.
Minimum 70B model for better results.

In any case AI for coding should be used only for 1-3 lines not for whole compilation units or scripts, and even then you need to verify what the output does and not blindly copy into your code.

Otherwise you end up with a bunch of unopimized bloat that is harder to fix and debug the more of it is AI generated.

Yep, AI hallucination is a known issue. The other day, a few weeks ago I watched how AI is hammering developers with fake reports of vulnerabilities, over 80% of these were non-existent, but developers had to check each single one, the youtuber equating it to a DDOS attack.

 

[f33dm3bits]

 

[hasser33]

All the time now. I have played with locals under Jan and Ollama for a year or so for coding, but unless the request was kept fairly simple, the odds of a function not having errors was very low. But, with the addition of a decent GPU and the fairly new gpt-oss-20b, that has changed. I use it to code very complex scripts and they almost always work if I described my needs with sufficient clarity. A gotcha is that it can give back code that is very verbose in terms of comments unless it is told to keep them to a minimum. Many days I go without bothering to use the online versions at all.

The main drawback to a local LLM, unlike its online cousins, is that once you exit then it will not remember what you have done. But in all, I like the way that private AI is developing. Especially with all the news of on-line AI and their billions out and pennies in, almost assuring that considerable subscription fees are inevitable. I’ve tried managing AI projects in-house — it drains time, talent, and energy fast. Hiring a gen ai development services https://artjoker.net/services/generative-ai-development-services/ like those felt like the reset we needed. They handled everything: model creation, deployment, updates. No more firefighting. Much smoother.

I use it pretty much all the time now. I’ve run local models under Jan and Ollama for about a year, mostly for coding, and earlier the error rate was high unless requests were simple. But after adding a solid GPU and trying gpt-oss-20b, things changed a lot—complex scripts usually run correctly if I explain things clearly. Only downside: locals forget context once you exit. Still, I really like where private AI is heading.

 

[Aidell]

I've never tried using it, but it would be interesting. Is it really better?

 

[Thunderpants]

I think I'll stick with traditional intelligence, as limited as that may be, for now. I don't want to risk lapsing into a spiral of laziness and dependence.