Running a Second Brain Entirely on a Local LLM

Why Local, and Why Now

In 2026, the shift toward a local LLM second brain is driven by hardware maturity rather than just privacy concerns. Apple Silicon M3 and M4 chips now provide sufficient unified memory bandwidth to hit 50+ tokens per second on 30B parameter models, making real-time interaction viable for professional workflows.

Open-weight releases from Meta (Llama 3.2), Alibaba (Qwen 3.5), and Google (Gemma 4) have narrowed the capability gap. For personal corpus tasks—such as synthesizing meeting notes or querying a private knowledge base—these models deliver 80-90% of the utility found in frontier cloud models.

While data sovereignty is a critical secondary driver, cost and latency are primary. Eliminating monthly API spend (often $300-500 for high-volume users) makes hardware like the Mac Studio or MacBook Pro with 128GB of unified memory a more economical substrate for long-term AI memory systems.

The Runtime Landscape

Selecting a runtime depends on the required balance between developer control and user interface. Ollama is the standard for CLI-driven automation and developers building RAG pipelines, providing a streamlined API that integrates easily with external memory systems.

For users requiring model inspection and GUI-based chat without configuration files, LM Studio provides an app-like experience. In contrast, llama.cpp serves as the lightweight backend for most of these tools, offering highly optimized 4-bit quantization to fit large models into consumer VRAM.

Hardware-specific optimization is handled by MLX (for Apple Silicon) and vLLM (for high-throughput Linux/NVIDIA setups). The primary trade-offs involve model format compatibility (GGUF vs. EXL2) and the availability of MCP server integrations for persistent memory.

Open-Weight Models Worth Running

Model selection for a local LLM second brain focuses on the balance between reasoning depth and inference speed. Llama 3.2 70B remains the benchmark for high-quality synthesis, while Qwen 3.5 32B is preferred for speed and efficiency on mid-range hardware.

Gemma 4 31B serves as a versatile default for general chat, whereas DeepSeek V3 is the primary choice for coding-heavy memory systems. For the retrieval layer, Nomic Embed provides high-performance embeddings that fit within small memory footprints.

Quantization is essential; Q4_K_M (4-bit) typically offers the optimal trade-off between perplexity and size. Typical performance on M3/M4 Max hardware follows these trends:

• Llama 3.2 8B: 100+ tokens/sec
• Gemma 4 31B: 30-50 tokens/sec
• Llama 3.2 70B: 15-25 tokens/sec

Wiring Local Models to a Second Brain via MCP

The integration of a local LLM second brain relies on the Model Context Protocol (MCP). In this architecture, the local runtime (Ollama or LM Studio) exposes an OpenAI-compatible endpoint for inference, while a separate MCP server manages data retrieval from a pgvector-backed database.

A lightweight MCP bridge connects the interface—such as Claude Desktop or open-source clients—to the local model and memory store. This allows the LLM to call tools that fetch specific documents from the vector store based on semantic similarity.

To register a local MCP server for memory retrieval, add the configuration to the client's settings file:

{
  "mcpServers": {
    "local-memory": {
      "command": "node",
      "args": ["/path/to/memory-server/index.js"],
      "env": {
        "DATABASE_URL": "postgresql://localhost:5432/second_brain"
      }
    }
  }
}

Performance Realities

Local hardware excels at specific memory tasks but faces hard limits on others. Semantic search over 10K to 500K chunks using pgvector is highly performant, enabling near-instant retrieval of relevant context for the LLM.

Chatting with local context and long-form writing assisted by RAG are stable workflows. However, processing extremely large contexts (above 128K tokens) often leads to memory exhaustion or severe degradation in tokens per second on consumer machines.

Vision-language models (VLMs) remain slower than text-only models and typically require significant VRAM overhead. Users should expect a slight increase in hallucination rates compared to trillion-parameter cloud models, requiring stricter grounding via the retrieval layer.

The Full Local Stack, Assembled

A production-ready local LLM second brain consists of four primary layers: a self-hosted Supabase instance for pgvector storage, Ollama for model inference, a custom MCP server for retrieval logic, and an interface like Cursor or Claude Desktop.

This stack eliminates recurring infrastructure costs and prevents data leakage. The system operates entirely offline once the models are pulled, ensuring total sovereignty over personal knowledge graphs.

For detailed implementation guides, refer to /build/ for hardware setup and /mcp/ for protocol configuration. Those seeking a pre-wired experience that maintains data sovereignty via their own Supabase instance can utilize the managed version at novcog.dev.