LocalLLaMA

3742 readers

24 users here now

Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.

Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.

As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.

Rules:

Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.

Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.

Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.

Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.

founded 2 years ago

MODERATORS

pax@sh.itjust.works

noneabove1182@sh.itjust.works

Smokeydope@lemmy.world

MonsterBug@sh.itjust.works

Small models (lemdro.id)

submitted 1 day ago by mudkip@lemdro.id to c/localllama@sh.itjust.works

3 comments fedilink hide all child comments

The Apple M5 Pro chip is rumored to be announced later this week with improved GPU cores and faster inference. With consumer hardware getting better and better and AI labs squishing down models to fit in tiny amounts of vRAM, it's becoming increasingly more feasible to have an assistant which has absorbed the entirety of the internet's knowledge built right into your PC or laptop, all running privately and securely offline. The future is exciting everyone, we are closing the gap

you are viewing a single comment's thread
view the rest of the comments

[–] splendoruranium@infosec.pub 11 points 1 day ago* (last edited 1 day ago) (2 children)

it’s becoming increasingly more feasible to have an assistant which has absorbed the entirety of the internet’s knowledge

Yeah, that's never going to happen, I'm afraid. The models do get denser and better at transformative tasks, but you will simply never be able to ask that 22B 4-quant for the birthdates of obscure but historically important Bolivian politicians. That's simply about information density and that's not a useful application for models.

It's going to be irrelevant, of course, once there's a convenient 1-click way to integrate your local Kiwix-server into your model's Open-WebUI's knowledge base. There's no need to waste VRAM on Wikipedia and Stackoverflow knowledge.

[–] mudkip@lemdro.id 8 points 1 day ago (1 children)

Kiwix and privacy-respecting metasearch integration will be useful for this

[–] kata1yst@sh.itjust.works 4 points 1 day ago

Totally agreed. LLMs shouldn't be asked to know things, it's counterproductive. They should be asked to DO things, and use available tools to do that.