Neuromod

Neuromod is a composable, type-safe LLM inference library for Python. The library provides step-function pipelines, Pydantic-based tools, and optional thread persistence.

It was inspired by and draws heavily from nvms/threaded. The goal was to bring that approach to Python and add ergonomic improvements that fit the way I build AI applications.

Why I built it

Most AI projects I have built require two capabilities: switching models and providers, plus evaluation suites for comparing model performance on a specific task. Evaluations become especially important when a new model becomes available and a client needs evidence for a provider change.

The work has included raw SDKs from Anthropic, OpenAI, Google GenAI, and Vercel AI, plus larger frameworks such as LangChain and Mastra.

Raw SDKs offer direct control but require provider abstraction and pipeline logic on every project. Larger frameworks address part of the problem but introduce complex APIs, heavy dependencies, and opinionated architectural patterns. AI development remains experimental, so I need the freedom to rethink an approach without a framework’s design constraining the system.

Existing abstractions also did not match the application behavior I needed. Production AI systems rarely consist of a single model call followed by a raw response. The systems chain steps, route work between specialized agents, enforce intent guardrails for application-specific chat interfaces, tailor outputs, and run expert agents for tasks such as text-to-SQL.

Some libraries address parts of the requirement, but the APIs did not fit my preferred development model. Neuromod supports common agent patterns without unneeded complexity.

Neuromod emerged from several earlier iterations.

Python-native additions

Neuromod retains Threaded’s flexible primitive while adding a Python-first API for common application work:

• Agent provides generate(), async-iterator stream(), and token counting from one model configuration, while remaining usable as a step function in lower-level pipelines.
• Typed model definitions replace provider/model strings and carry each model’s input and output limits.
• Agent responses expose the final message, full history, accumulated usage, per-step results, duration, finish reason, and parsed structured output.
• Each tool declares its own call limit, approval requirement, and retry policy. Multiple tool calls execute concurrently to avoid unnecessary serial latency.
• The optional neuromod-sqlalchemy package provides a ready-to-use async thread store for durable conversation history in any SQLAlchemy-supported database.

Architecture

Neuromod uses a single core primitive: async (context) -> context. Agents, model calls, scoped sub-pipelines, tools, and threads use the primitive and compose through compose() and scope().

The primitive expresses several higher-level features:

• A pipeline consists of composed step functions.
• A sub-agent combines a step function with scope(), tools, and isolation rules.
• An agent provides a readable wrapper around a model() step while preserving access to lower-level composition.

The default API prioritizes readability while the underlying primitive preserves composition.

Design principles

Early versions exposed the cost of loose interface choices. The current design uses the following principles:

• Step functions form the primary abstraction: async (ctx) -> ctx.
• ConversationContext.messages serves as the single source of truth.
• Message.content always uses list[Content], never str | list.
• Tool calls live inside content[] rather than separate fields. Message roles are limited to system, user, and assistant.
• Providers represent API connections rather than model bindings. Each request selects a model.
• HTTP status categories define typed errors: authentication, rate limit, network, and API errors.
• Models and tools are immutable where practical. Context updates return new instances.

Production use

An earlier Neuromod version runs in production across several internal client projects. The current public version supports Cardlang and is published on PyPI.

Stack

The Python 3.11+ package uses pydantic and httpx as direct dependencies. The library supports Anthropic, OpenAI, Google Gemini, and xAI. Thread persistence is available through the optional neuromod-sqlalchemy package.