Getting started with local LLMs (part 2)

Welcome to Part 2 of the “Getting started with local LLMs” blog series. If you’re just arriving here for the first time, here’s a link to Part 1 of the series where you’ll learn how to determine which models you can run on your current hardware 😄 And for those returning to the series—now that you know what models you can run, it’s time to choose an inference engine. But which engine should you use?

Well, like any technical decision: it depends.

Oh, and before anyone tries to accuse me of using AI to write this blog post: these em-dashes, colons, semicolons, emojis, code snippets and blocks and so forth have been painstakingly crafted by human hands, TYVM 😜 And with that said…

Inference engines: Ollama or vLLM?

In simple terms, choosing between ollama and vllm comes down to how many parallel sessions you want (or need) to support. In my experience, I tend to use ollama with my GeForce RTX 3080 Ti machines, since I’m not targeting those systems for parallel workloads (although perhaps I should 🤔). Meanwhile, vllm runs exceptionally well inside a container, and is usually the example inference engine provided on a given model’s Hugging Face page; see Nvidia’s Qwen3.8-Flash-Next-FP4-Local model as just one example.

That said, the fastest and easiest way to get started is by simply using ollama; you can find the installation instructions on their website. One of the benefits of Ollama is that they offer many of the standard-issue open weights models, making it easy to download and run a model with the right parameters for your setup. You can find a list of the models they offer on their website; for example, Nvidia’s Nemotron 3.5 Bolt looks like an interesting small language model that offers 8-billion parameters with 4 billion effective parameters (8b-e4b). I was 🤏 this close to being nerd sniped when I saw this model while writing this.

Anyway, the downside to using ollama is that it’s intended for a single user runtime. This might be fine if you’re just running a local model on your laptop. And although ollama can support parallelism if you’re running it from a more powerful machine, it’s just not as efficient as vllm. So if you’re going to serve multiple users / multiple sessions simultaneously (hint: like when you want to run a swarm of Agents) you’re going to need vllm for that.

As I noted earlier, there are a lot of models that provide tailored vllm scripts to optimize their performance. There are also GitHub projects like eugr/spark-vllm-docker that provide pre-built recipes for running models on Nvidia’s DGX Spark. Regardless of your hardware, there’s almost certainly a vllm script for your use case. I’ve added a few of mine as examples in the getting-started-with-local-llms repo accompanying this blog.

Using your local LLMs from anywhere!

Note: if you’re running local LLMs from your laptop, then you can skip this section 👍

Once you’ve got your model(s) setup, it’s time to make them available from anywhere you’ve got a WiFi signal. If you’ve already followed my Linux Dev Server Setup Guide, you’re 90% of the way there. If you haven’t, I recommend pausing to go and cross-reference sections 5-7 to make sure you’ve got all the necessary software setup on your local LLM inference machine(s). Note that you’ll also need to add Tailscale to your laptop in addition to your desktop system(s).

For those who don’t already know about Tailscale, it’s a “Zero Trust identity-based connectivity platform that replaces your legacy VPN” with a free tier for personal use. This makes it an excellent choice for securely accessing your new LLM inference servers remotely. There are a few steps you’ll need to implement in order to get this up-and-running, so let’s walk through that.

After you’ve setup Tailscale, the first thing you’ll want to do is to ssh into your inference machine(s) and add a tag to the box with the following command: sudo tailscale up --advertise-tags=tag:inference. This will make it easier to limit how network traffic flows when you get around to setting up Access Control List (ACL) rules in the Tailscale console later. After you’ve tagged your inference machine(s), you’ll also want to run one of the the following commands to ensure you can access the inference engine remotely.

# If you're serving vLLM on port 8000:
sudo tailscale serve --bg --https=443 http://127.0.0.1:8000
# If you're serving Ollama on 11434
sudo tailscale serve --bg --https=443 http://127.0.0.1:11434

Once you’ve got that in place, it’s time to add some access controls.

I’ve added an example Access Control List (ACL) to the GitHub repo that you can copy if you’re using dev or inference tag names—and of course, feel free to modify accordingly if you chose something else. After saving the ACLs under the “Access control” section of the Tailscale console (you can use Tailscale’s “JSON editor” to copy/pasta the ACL linked above), be sure to test that you can still reach the systems from a terminal on your laptop with the following commands:

# Confirm that you can SSH to your box. I recommend using an SSH config file.
ssh user@your-tailnet.ts.net

# Confirm that your machine can "see" your inference server and model
curl -s https://your-tailnet.ts.net/v1/models | jq '.data[] | {id, max_model_len}'

If everything still works, try switching your laptop over to a cellular hotspot and repeat this step to double-confirm that everything still functions as expected. If it does—congratulations! You’ve now got local LLM access from anywhere you have a WiFi signal!

Using your local LLMs with a coding agent

Figure 1. Oh-my-pi logo: github.com/can1357/oh-my-pi

And now you’re finally ready to start using your local LLM setup. I’m going to go into more depth about this in the next blog, but lets get you started using your local language model(s) with a coding agent so that you can start playing around with things while I’m writing he next post in this series.

For now, let’s get you setup with oh-my-pi (AKA omp). I recently learned about this terminal-based agent harness thanks to Matt Suiche’s Weightless project, and I’ve been thoroughly impressed with how easy it was to get up-and-running. What I’ve enjoyed most about it is that it has built-in model routing based on an agent‘s role—so if you’re running multiple local LLMs like I am, it automatically routes a given task.

To install omp just follow the instructions on their website. After you’ve done that, you’re going to want to write your models.yaml file in your favorite text editor, which you then add to ~/.omp/agents/ directory (or wherever you’re pointing your omp install to). To give you an example of what this file should look like with Tailscale addresses, I’ve added an example to the getting-started-with-local-llms GitHub repo.

Once you’ve got that in place, spin-up omp and type in /models to review and setup your model roles. You can also pre-select that level of thinking each model uses before it starts to take action. The simplest analogy for thinking is that it tells the model how much time to “read the recipe” (meaning tokens from its model weights) before it starts cooking.

And there you have it. Now you’re thinking with portals! See what I did there? 😉 Although I should probably warn you now, some of these open weight models with “reasoning” capabilities really like to cook for a good long while if you set the thinking parameter too high. Experiment with this setting a bit to see what feels most comfortable for you in terms of timing vs. performance.

What’s coming up in Part 3

Woo! You made it! That was a lot to get through 😅 Okay, so I know I promised to cover things like choosing the right harness and routing your prompt to the right model, and I will absolutely cover these in the next blog post. I’ll also include guidance on how to enable web search with your local LLMs, because I really missed that when I first went down this path. Anyway, there was a lot more ground to cover in this post than I thought. Sorry about that! 🙇

For now, have fun experimenting with different model sizes and types (coding, vision, reasoning, etc.). Try having your model(s) build an omp extension for you 👀 I’ll warn you though: I’ve had mixed results.

Anyway, in the mean time—have fun!

---

While I’m preparing the next installment in this series you can git checkout other (usually off-topic) content I’m reading more regularly over at Instapaper.

Until next time, remember to git commit && stay classy!

Cheers,

Keith