Getting started with local LLMs (part 1)

If you want to know the state of open weight LLM adoption, you need look no further than the recent price explosion for local compute supporting such workloads. For example, a single Nvidia DGX Spark went from $4700 to over $9400 in just fifteen days before settling back at around $8499 😬 And that’s after several months of stable pricing. What happened? Well, I think you’ll find that it’s finally worthwhile to try leveraging local language models for yourself.

Figure 1. DGX Spark prices from PriceHistory.app

Welcome to the era of Local Language Modes

On September 15, TypeSafe.ai introduced “System One” models such as Jev, and suddenly people realized they had the ability to quickly and cheaply run Agent automation with open weight variants such as Kev. Literally overnight agent workloads became a trivially scalable operation you can perform on your laptop. Pair that with larger models running on a half-decent graphics card like Nvidia 3090’s or 4090’s—or purpose-built hardware like Nvidia’s DGX Spark—and you can all-but-replace most of your use cases from the metered subscriptions sold by frontier AI labs.

A year ago I suggested that language models would get smaller and more capable, at the time citing a paper written by Nvidia that was originally published in June of 2025. It feels wild to me that things played out so quickly—but here we are. And honestly, you know what I don’t miss about relying on the frontier models? Being held back by their refusals and guardrails.

Offensive security researchers and defenders alike are quickly reaching for open weight models because the frontier labs are preventing them from fully realizing the potential of what these models can do. Defenders in particular need this technology now more than ever, as we saw with the Hugging Face breach—and as Caleb Sima repeated at [un]promptedCon II when his team shared the story about how Caleb surprised them with a “live fire” Agentic security assessment.

What can you run with the hardware you have?

For anyone who missed the window to procure hardware before prices went vertical, don’t fret—you still have options. There are many sites today that will tell you what models you can run on the hardware you currently have. The two I frequently use are willitrunai.com and canirun.ai; between these two sites, you should have all the information you need to determine what you can comfortably run.

But, perhaps a better question is what should you run—and the answer to that depends entirely on the problem(s) you’re trying to solve. Generally speaking, models with a lower number of parameters are going to either need more harness engineering to get the outputs you want—or you’re going to need more compute to run models that “just work” out of the box for your use case. Either way, you’ll want to identify and run benchmarks specific to your use case before you land on what models work best for the problems you’re trying to solve.

Personally, I run smaller models on my Intel NUC 11 Extreme boxes with their GeForce 3080 Ti’s. This includes Qwen-3.5-9B with 64k context, and Nemotron-Nano-9B for small coding and lightweight reasoning tasks. On my DGX Sparks I’m usually running Nemotron-Lightning-30B-A3B, Qwen3-Coder-Next, or Qwen3.8-Flash-Next respectively.

And if you’re like me and you’re leveraging local models to perform cyber activities, you can now make use of Matt Suiche’s recently released Weightless project to abliterate models using GGUF Layer Projection (GLP) without doing another training run. Matt make’s these lightweight modifications available directly on Hugging Face. Woot woot!

A quick start guide and what’s coming in Part 2

Along with this blog I’ve created a project where I’m sharing Markdown formatted guides that includes instructions to help you get started setting up your system(s) for local LLM deployments. In the next installment I’ll be sharing how to securely access your new local LLMs from anywhere, choosing the right harness, options for routing your workloads to the right model(s), as well as a tool I’ve designed for dynamically changing your model “load out” on the fly.

Stay tuned!

---

While I’m preparing the installment in this series you can git checkout other (usually off-topic) content I’m reading more regularly over at Instapaper. Until next time, remember to git commit && stay classy!

Cheers,

Keith