Building an LLM Code Assistant with Local Ollama, FastAPI, and OpenClaw
This stack (Ollama + FastAPI + OpenClaw) gives me full control over context length, token budget, and model selection. Tradeoff: GPU memory becomes the bottleneck. I’ll show you how I sized it.
Run `ollama --version` first. If you don’t have Ollama installed yet, grab it from [ollama.ai](https://ollama.ai) and pull `llama3.1:8b`: `ollama pull llama3.1:8b`.
OpenClaw saved me ~25% context tokens. AST parsing grabs only the relevant code blocks, not entire files. Tradeoff: node-walking adds ~150ms per query.
Remember: Ollama’s KV cache persists between queries. First query after server restart is cold; subsequent ones reuse cache and feel instant.
Numbers measured with `nvidia-smi dmon -s p` and `time curl`. Single-run, not statistically rigorous. Standard deviation from 3 repeats.
Local KV cache saturation kills responsiveness. Monitor `ollama serve` logs; restart Ollama if latencies exceed 5s.
WebSocket drops initial latency by ~600ms. Streamed tokens arrive as soon as Ollama generates them.