My friend Kamila wrote a piece recently about why we overcomplicate agentic harnesses. It made me look at my own server closet.
Last week, I watched a 70B parameter model spend twelve seconds trying to format a response because the system prompt told it to dynamically choose between a bulleted list or a numbered list. It wasted thousands of context tokens on self-correction. And it got me thinking about waste.
We are currently building AI systems like we have infinite compute and zero latency. We don’t. Especially if we want things to run locally, privately, and reliably.
When I set out to build daily-relay — my local-first automation wrapper for compiling daily personal briefings — I set a hard constraint. It had to run in a low-power Docker container on a small 3B parameter model (llama3.2:3b).
At that size, you cannot afford runaway agentic loops. If your model gets stuck in a thinking loop, your CPU spikes, your latency goes to hell, and your briefing is late. Constraint forces architectural discipline.
Harness as the Core Differentiator (The “Sandwich” Sanitizer)
Kamila notes that a harness provides the structural boundaries to prevent silly or disastrous model mistakes. She is right. Reliability belongs to the wrapper code, not the weights.
With daily-relay, I implemented what I call the Sandwich Sanitizer — inbound and outbound boundaries that keep the model in a tight box.
On the way in, we do not let the LLM look at raw, noisy data or figure out its own prompt strategy. We scan the incoming text using regex tokens, strip the telemetry junk, and map it directly to pre-baked system prompts that explicitly ban conversational preambles. We don’t try to strip “Sure, here is your summary” with code; we command the model’s structural boundaries to never say it in the first place.
On the way out, we use code to handle what the prompt cannot guarantee. Smaller 3B models are notorious for slipping back into mechanical habits: wrapping text in accidental quotation marks, appending markdown formatting characters that make text-to-speech readers stutter, or leaking structural bullets.
Instead of wasting precious compute trying to write an essay-length prompt to fix syntax quirks, a standard O(1) Javascript string clean-up intercepts the raw output instantly:
// Post-LLM Cleanup: Strip accidental wrapper tokens and syntax artifactssummary = summary .replace(/^["']|["']$/g, '') // Removes leading/trailing quotes .replace(/^[-*•●○][ \t]+/, '') // Strips accidental list bullets and tabs .replace(/[\*\#\_\[\]]/g, '') // Cleans out markdown syntax for audio readability .trim();
The system prompt enforces the behavior. The code guarantees the schema. The model is freed to do what it does best: semantic compression.
Evals on Day One (The 56% Latency Win)
Evals and harnesses aren’t the polish you put on a finished AI app. They are the scaffolding required to build it in the first place.
I built tests/promptOptimizer.js concurrently with the initial runtime loop. It is a local-first testing utility that runs prompt variants through Ollama, measures latency with performance.now(), and scores them on a 1-to-10 automated scorecard.
We use a combination of deterministic regex checks (flagging forbidden build numbers or KB codes) and a local LLM-as-a-judge to grade formatting.
Because I had immediate telemetry, I noticed a huge bottleneck. Shifting the prompt constraint from paragraph-style summaries to structured, three-line bullet points sliced our execution latency by 56% — dropping from 4.28 seconds down to 1.87 seconds on my local Llama 3.2 instance.
Without immediate evals, I would have spent days tweaking model weights or upgrading hardware. With them, it was a five-minute prompt adjustment.
Pro Tip: If your local LLM is taking too long to think, look at your output constraints. Forcing a small model to write prose is slow. Forcing it to write structured bullets is fast. Let standard code handle the transitions.
The Minimal Functional Baseline (Bifurcated Routing)
We have been conditioned to believe that to achieve automation, we need massive agentic frameworks, vector graphs, or dynamic tool-calling layers.
We don’t. It’s just a complex pile of Legos when a simple wooden block will do.
In daily-relay, I use a Hard-Bifurcated Routing Layer. If data is structured and predictable — like Google Tasks, calendar boundaries, or house chores — we keep the LLM completely away from it. We use deterministic, zero-latency Javascript logic for the math and scheduling.
We isolate the AI path strictly to atomic, single-item text density summarization using a flat cache.json state lock.
Instead of writing a massive, brittle hard-coded JavaScript array directly into the execution loop, we extract those interest-based filters into a clean configuration file. To illustrate this “minimal baseline” approach for daily Wikimedia historical updates, we map it using our flat sources.yaml config file:
# sources.yaml- name: "Wikimedia History" type: "history" interests: - 'aerospace' - 'space' - 'orbit' - 'photography' - 'camera' - 'film' - 'seattle' - 'cascadia' - 'aviation'
Inside historyCollector.js, a simple low-overhead loop scans the day’s raw wiki events against this interest configuration before the model even wakes up:
// Inside historyCollector.js// A basic regex scans the raw events against your interest configurationconst interestRegex = new RegExp(`\\b(${interests.join('|')})\\b`, 'i');let filteredEvents = data.events.filter(event => interestRegex.test(event.text));
Instead of passing the entire, massive raw daily payload from the Wikimedia API to the LLM and asking it to “pick what I like,” we run a deterministic pre-filter. We load a flat YAML file, execute a standard array intersection in code, and only send the matching 3 or 4 high-signal historical events to Ollama. We save thousands of processing tokens before the model even wakes up.
It’s deterministic because we expect to either have one “On this day…” that matches our topics, or zero which is ok too. This application expects to gracefully degrade with something quirky like “Nothing interesting happened on this day.” LLMs are notorious for hallucinating in a panic when they don’t quickly find something to generate.
The system reads the cache, runs the summary, and writes it back. No state graphs. No memory stores. Just standard, file-based persistence.
The Architecture of Restraint
The industry is currently over-indexing on making models smarter. We should be indexing on making our harnesses more resilient.
True production reliability comes from knowing exactly when to step away from the LLM and let a basic regex handle the heavy lifting. Build boundaries. Guard your parameters. Write regular code.
Your CPU — and your budget — will thank you.
Leave a Reply