LocalLLaMA

3785 readers

27 users here now

Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.

Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.

As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.

Rules:

Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.

Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.

Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.

Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.

founded 2 years ago

MODERATORS

pax@sh.itjust.works

noneabove1182@sh.itjust.works

Smokeydope@lemmy.world

MonsterBug@sh.itjust.works

Best LLMs for PC/Tech Troubleshooting? (lemmy.ml)

submitted 3 weeks ago* (last edited 3 weeks ago) by FrankLaskey@lemmy.ml to c/localllama@sh.itjust.works

22 comments fedilink hide all child comments

I find myself really appreciating what LLMs can do when it comes to help with software and tech support. I am a pretty adept PC power user who is not a programmer and (until recently) has only had a modest amount of experience with GNU/Linux. However, I have started to get into self-hosting my own FOSS apps and servers (started with OpenWebUI, now Jellyfin/Sonarr via Docker compose etc). I’m also reading a book about the Linux command line and trying to decipher the wold of black magic that is networking etc myself.

I have found that LLMs can really help with comprehension and troubleshooting. That said, lately I am struggling to get good troubleshooting advice out of my LLMs. Specifically, for troubleshooting docker container setups and networking issues.

I had been using Qwen3 Coder 480b, but tried out Claude Sonnet 4 recently and both have let me down a bit. They don’t seem to think systematically when offering troubleshooting tips (Qwen at least). I was hoping Claude would be better since it is an order of magnitude more expensive on OpenRouter, but so far it has not seemed so.

So, what LLM do you use for this type of work? Any other tips for using models as a resource for troubleshooting? I have been providing access to full logs etc and being as detailed as possible and still struggling to get good advice lately. I’m not talking full vibe coding here but just trying to figure out why my docker container is throwing errors etc. Thanks!

Note: I did search and found a somewhat similar post from 6 months ago or so but it wasn’t quite as specific and because 6 months is half a lifetime in LLM development, I figured I’d post as well. Here’s the post in question in case anyone is curious to see that one.

you are viewing a single comment's thread
view the rest of the comments

[–] afk_strats@lemmy.world 1 points 3 weeks ago (1 children)

Totally. I think OSS is outright annoying with its verbosity. A system prompt will get around that

[–] FrankLaskey@lemmy.ml 1 points 3 weeks ago (1 children)

I tried that! I literally told it to be concise and to limit its response to a certain number of words unless strictly necessary and it seemed to completely ignore both.

[–] afk_strats@lemmy.world 1 points 2 weeks ago (1 children)

I don't know if this is still useful for you, but I tried this out, mostly because I wanted to make sure I wasn't crazy. Here's my gpt-oss setup running on cheap AMD Instinct VRAM:

./llama-server \
  --model {model}.gguf
  --alias "gpt-oss-120b-mxfp4" \
  --threads 16 \
  -fa on\
  --main-gpu 0 \
  --ctx-size 64000 \
  --n-cpu-moe 0 \
  --n-gpu-layers 999 \
  --temp 1.0 \
  -ub 1536 \
  -b 1536 \
  --min-p 0.0 \
  --top-p 1.0 \
  --top-k 0.0 \
  --jinja \
  --host 0.0.0.0 \
  --port 11343 \
  --chat-template-kwargs '{"reasoning_effort": "medium"}'

I trimmed the content because it wasn't relevant but left roughly the shape of the replies to give a sense of the verbosity.

Test 1: With default system message

user prompt: how do i calculate softmax in python

What is softmax
1 python + numpy
...
quick demo
...
2 SciPy
...
...
...
8 full script
...
running the script
...
results
...
TL;DR
...

followup prompt: how can i GPU-accelerate the function with torch

1 why pytorch is fast
...
...
**[Headers 2,3,4,5,6,7,8,9]**
...
...
TL;DR
...
Recap
...
Table Recap
...
Common pitfalls
...
Going beyond float32
...
10 Summary
...

Overall 6393 Tokens including reasoning

TEST 2 with this system prompt: You are a helpful coding assistant. Provide concise answers, to-the point answers. No fluff. Provide straightforward explanations when necessary. Do not add emoji and only provide tl;drs or summaries when asked.

user prompt: how do i calculate softmax in python

Softmax calculation in Python
...
Key points
...

followup prompt: how can i GPU-accelerate the function with torch

GPU‑accelerated Softmax with PyTorch
...
What the code does
...
Tips for larger workloads
...

Overall 1103 Tokens including reasoning

[–] FrankLaskey@lemmy.ml 1 points 2 weeks ago

Thanks. I may give an updated system prompt like this a shot. Not sure where mine went wrong other than maybe it wasn’t being honored or seen by OpenRouter (I’m not running 120b locally, it’s too large for my set up). I’m actually a bit confused on how to set parameters with OpenRouter.