Memory Pipeline

Memories don't arrive one way. There are exactly three ingest paths, plus a class of server-generated memories that is deliberately not an ingest path. After ingest, every memory moves through the same stages — knowing them helps you debug "why didn't my agent recall this?" and "why does the stored memory look different from what I sent?".

The three ingest paths

Server-generated (not an ingest path): insight, outcome, and rule memories are produced by the platform itself — insights runs, memclaw_evolve, crystallization, Forge distillation. Agents cannot write these types directly; memclaw_write rejects them. If you're wondering why a type "won't write", this is why.

1. Ingest

Whichever path, core-api validates auth, enforces the caller's trust level and tenant quota, and persists the row. The HTTP response returns as soon as the row is committed; everything else below is asynchronous — with one deployment-dependent exception in stage 2.

2. Enrich

Two jobs run on every write:

• Embedding — vector generation via the configured embedding provider. Default is OpenAI; alternatives include local (TEI) and fake for development. Set with EMBEDDING_PROVIDER and OPENAI_API_KEY (or the matching keys for other providers).
• Entity extraction + classification — names, projects, repos, references; memory type, title, summary, tags. Driven by the configured entity-extraction provider (ENTITY_EXTRACTION_PROVIDER: openai / anthropic / gemini / openrouter). Vertex AI is the operator-managed platform-tier provider, used on the managed deployment.

Where they run depends on the deployment (the variance table has the full picture):

• Self-host (OSS, deployment_mode=inline) — core-api runs both inline on the write path. There is no core-worker container in the compose stack.
• Managed (deferred) — core-api publishes embed-requested / enrich-requested to the event bus; core-worker consumes, runs the providers, writes back through core-storage-api, and announces completion on embedded / enriched — which core-api itself subscribes to. That return edge is what triggers contradiction detection on the deferred path.

Until enrichment lands, the memory is queryable by id but won't show up in semantic recall. The exact provider matrix lives in the OSS .env.example header.

Contradiction detection runs after commit (fire-and-forget via track_task — see core-api/src/core_api/services/contradiction_detector.py). The write does not block on it; contradictions surface through memclaw_insights after the loop runs.

3. Govern

Independent of the write path, governance keeps the store useful. The scheduled pieces run as residents on the core-operations clock:

• Trust enforcement — every read and write checks the caller's trust_level. Operations beyond that level return 403 FORBIDDEN. See Trust levels.
• Karpathy Loop — agents report outcomes after acting on recalled memories via memclaw_evolve (success | failure | partial). The platform reinforces what works and may auto-generate preventive rule-type memories on failure. memclaw_insights surfaces the resulting reflection (contradictions, drift, stale entries); the nightly lifecycle-insights tick runs the same discovery per opted-in org.
• Memory Crystallizer — consolidates many small memories about the same entity into stronger, denser ones. Owned by the lifecycle-crystallize core-operations tick (daily, per active org, with a dedup gate in the consumer). POST /api/v1/crystallize remains the on-demand path for a single tenant.
• Lifecycle hygiene — the lifecycle-archive-expired, lifecycle-archive-stale, and lifecycle-purge-soft-deleted ticks age out expired, stale, and soft-deleted rows on each org's retention settings.

4. Recall

POST /api/v1/recall (or memclaw_recall) does hybrid retrieval: vector similarity + keyword full-text + entity-graph matches, blended into a single ranked list. Trust still applies at read — a level-1 agent cannot recall across fleets it doesn't own.

For non-semantic flows:

• memclaw_list — paginated browse by entity / type / time. Use when recall is too narrow.
• memclaw_manage op=read — read by id when you already know the memory_id.

Where each stage lives

This page and Architecture are a pair: this one follows a write, that one maps the machinery. For the live OpenAPI surface, see the API Reference.