Skip to content
← All problems
#27 by prioritySteadyAI

Running local models is slow / hardware-hungry

Local inference needs too much VRAM/RAM, is slow, or is painful to set up.

Priority (½-life 90d)
2.5
Total mentions
7
Distinct people
6
Last 30 days
0
Last 90 vs prior 90
2 / 2
Mentions per monthHover or focus a month for details
024Nov ’25DecJan ’26FebMarAprMay2JunJulAugSepOctNov
Show data table
Monthmentions
Nov 20250
Dec 20250
Jan 20260
Feb 20261
Mar 20261
Apr 20261
May 20260
Jun 20262
Jul 20260
Aug 20261
Sep 20260
Oct 20261
Nov 20260

Most often mentioned with

  • Local models 6
  • LLM 3
  • RAG 1
  • Claude Code 1
  • Coding agents 1

What people said · newest first · first seen Feb 24, 2026

  • Plugged it into OpenRouter and ~$.50 of Deepseek later I've installed llama.cpp and Llama 3.1. The local model doesn't work with Pi yet (and I know it will be bad and slow even if it does) but I'm curious to see what you can do on an 8GB consumer GPU these days...

    simpaticoder · Oct 1, 2026 · comment in “Pi 1.0”

  • Standard local RAG is surprisingly heavy. Running dense vector databases alongside an 8B+ generative LLM just to parse documents and maintain conversational context burns VRAM, chokes mid-range GPUs (like a GTX 1070), and still hallucinates when asked about missing or unverified information.

    roandejager5 · Aug 13, 2026 · comment in “Hillock – Local neuro-symbolic memory engine running in <1.2GB VRAM”

  • > I don't know about good, I use a lot of local models and they're still pretty painful to run locally

    locknitpicker · Jun 17, 2026 · comment in “Running local models is good now”

  • I don't know about good, I use a lot of local models and they're still pretty painful to run locally

    c0rruptbytes · Jun 16, 2026 · comment in “Running local models is good now”

  • After doing some tests, I found that with llama.cpp and IQ4_XS quant, the model and BF16 cache of the whole 262k context just fit on 32GB vram, which is impossible with full attention. In contrast, with gemma 4 31b IQ4_XS quant I have to use Q8_0 cache to fit 262k context on the vram, which is a little annoying (no offenses, thank you gemma team, too).

    RandyOrion · Apr 23, 2026 · comment in “Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model”

  • The experience of utilizing tiered storage (gpu vram, ram, and ssd) is generally poor for a lot of LLM inference engines out there, e.g., llama.cpp, sglang, vllm, etc.. My own experience shows that both weight and KV cache offload to ram on sglang and vllm is unavailable or unusable.

    RandyOrion · Mar 23, 2026 · comment in “Flash-MoE: Running a 397B Parameter Model on a Laptop”

  • Show HN: oMLX – coding agents on local LLMs without the painful reprefill

    jundot · Feb 24, 2026 · story in “Show HN: oMLX – coding agents on local LLMs without the painful reprefill”