Running a Local Vision LLM for Private, On-Device Work: Gemma 4 26B in LM Studio
Why I run a vision-capable LLM entirely on-device: Gemma 4 26B A4B in LM Studio, and the privacy reasoning behind keeping it off the cloud.
Why local at all?
Most people who use “AI at work” type their text into a cloud window and don’t think twice about where that data ends up. For me, that’s exactly the sticking point. The moment work-adjacent material is involved, I don’t want it reaching someone else’s server, showing up in a log, or becoming part of a training set.
So my model runs entirely locally on my Olares One. Nothing leaves the device. No cloud upload, no outbound API call, no question of who I’m trusting in the moment. This isn’t a feature I bolted on afterwards, it’s the whole reason for the setup.
I lean on it as a support tool for professional tasks that I’m deliberately not going into detail about here. The only thing that matters for this article is this: those tasks also involve visual material, and that’s what makes the model choice interesting.
Why a vision model
The key point is that the model understands not just text but images. A text-only LLM wouldn’t have helped me much. I need something that can work with visual content directly, without me laboriously translating it into words first.
And because those images can be sensitive, this closes the loop back to the first section. A multimodal model in the cloud would mean uploading exactly that visual material. Locally, it stays on the drive. Multimodal and on-device is the combination this is really about.
The model: Gemma 4 26B A4B
I currently run Gemma 4 26B A4B Instruct. A few things make it a good fit for this:
It’s a MoE model (mixture of experts): 26 billion parameters in total, but only around 4 billion are active per request (12 experts). In practice that means it answers about as quickly as a small model while carrying the knowledge of a large one. For a support tool where waiting is annoying, that’s exactly the right balance.
On top of that comes the large context window. The model supports up to 262,144 tokens; I run it at around 166,000, which is plenty for long documents in one pass.
And, of course, vision, the reason from the previous section.
Honestly, I didn’t spend long looking for alternatives. Gemma 4 struck me as the right model for the task from the start. The combination of vision, MoE speed, and a big context window fit, and I had no reason to keep comparing. No exhaustive benchmarking against Qwen and the rest, just: it worked, and I stuck with it.
The whole thing runs locally on my Olares One, where the roughly 24 GB footprint fits comfortably in the GPU’s VRAM.
My LM Studio configuration
I run the model in LM Studio, mostly because it’s easy to use and doesn’t need the command line. For an everyday tool I reach for often, that convenience matters more to me than squeezing out the last bit of performance a raw llama.cpp setup might give. The key settings, in case you want to reproduce it:
- GPU Offload: 30 layers on the GPU
- CPU Thread Pool: 12
- Context Length: ~166,000 tokens (of 262,144 possible)
- Evaluation Batch Size: 2048, Physical Batch: 512
- Unified KV Cache: on
- Offload KV Cache to GPU Memory: on
- Keep Model in Memory: on
- Flash Attention: on
- K/V Cache Quantization: Q8_0 (requires Flash Attention)
- Seed: 42 (for reproducible output)
Quantising the KV cache to Q8_0 saves a noticeable amount of memory without any quality loss I can perceive. LM Studio marks it as experimental, but it’s been stable for me. If you run into trouble, the first thing to try is setting the V cache back to F16.
The honest framing
I use this setup as a support tool, not a replacement for the actual work. It takes routine off my plate and gives me a first pass, but the responsibility and the final judgement stay with me. I’m deliberately not going into the specific work here; the point of this article is the setup and the reasoning behind it, not the content.
In daily use it feels good and makes the work noticeably easier, which is the whole point. The only limit I currently run into is very long documents, and I’m already working on that. Otherwise it runs smoothly and takes care of exactly the routine I use it for.
Conclusion
The real win isn’t “I use AI,” it’s how: a multimodal model that understands images, with a large context window and at usable speed, all without a single byte leaving the device. For work-adjacent tasks with sensitive material, that’s the difference between “handy but uncomfortable” and “handy and defensible.”
The hardware has the horsepower for it. And deliberately doing without the cloud isn’t a compromise I put up with, it’s the entire point of the exercise.