Replicant — AI Agent & Contributor Guide

Copy Markdown View Source

How to work effectively in this repo. The Critical Rules are binding.

What this is

A framework-agnostic Elixir CDC consumer for Postgres logical replication (pgoutput). It consumes the logical stream, assembles committed transactions, and delivers them synchronously to a pluggable sink — advancing the replication slot only AFTER the sink has durably persisted. Tenant-blind, Ash-agnostic, classification-blind. Multitenancy and classification live in the sibling ash_replicant sink adapter, never here.

Capabilities (all shipped): initial snapshot + resumable incremental backfill, a lib-owned checkpoint store for non-transactional sinks (with bounded retry-then-halt), batched checkpointing, sink-owned atomic batch delivery, pgoutput proto-v2 in-progress-transaction streaming, consumer-side disk spill for oversized transactions, PG17+ forward-compat with failover slots, multi-publication per pipeline (publication: [p1, p2] — fail-closed on a missing pub), and logical-decoding messages (messages: true — pg_logical_emit_message payloads). A logical-decoding message's content and prefix are user bytes: Critical Rule 1 binds — never log them, surface them in an error, or emit them in telemetry.

Critical rules

1. No row value in an error, log, or telemetry event. Assume every value is PII or a secret. The vendored decoder raises on malformed WAL and embeds raw bytes; decode therefore runs behind Replicant.Decoder.decode/1, which catches raises and scrubs raw bytes into a value-free Replicant.Error. Column names stay strings — never String.to_atom (a wide or attacker-influenced schema would exhaust the atom table). Telemetry metadata is allowlisted (Replicant.Telemetry): LSNs, table names, counts, durations, error classes — never values.

2. Validate identifiers. Slot and publication names reach SQL; they go through Replicant.Identifier.validate/1 (a strict Postgres-identifier allowlist) before interpolation. A multi-publication list (publication: [p1, p2]) validates EVERY name and fails closed if any requested pub is absent (a START_REPLICATION that names a missing pub silently streams the subset). A failure carries the invalid-SHAPE fact only, never the offending string. This hardens walex's raw '#{publication}' interpolation.

3. Exactly-once is at-least-once + a transaction-watermark-idempotent sink. The watermark is the commit LSN at transaction granularity (every row in a pgoutput proto-v1 transaction shares one commit LSN). Skip any transaction whose commit_lsn <= checkpoint; upsert rows by table PK. There is no naked exactly-once without two-phase commit or an idempotent sink — do not claim one. A transactional logical-decoding message rides %Transaction.messages and inherits this effect-once dedup; a non-transactional message routes to handle_message/2 and is at-least-once (no dedup key, duplicates possible on reconnect) — state each guarantee honestly.

4. Unchanged TOAST is a sentinel, not a value. An UPDATE that does not touch a TOASTed column sends a sentinel. %Change{} surfaces it as a first-class unchanged: [col] list; the sentinel never appears in record. Sinks must leave those columns untouched on upsert. A change of replica identity or a dropped column classifies as :destructive and halts fail-closed.

5. Stay tenant-blind. No multitenancy, scope, or classification logic enters this lib. That boundary is the whole reason replicant and ash_replicant are separate.

Development workflow

mix deps.get
mix format
mix credo --strict
mix compile --warnings-as-errors
mix test
mix dialyzer
# or all quality gates at once:
mix quality

All gates must pass before a commit/PR. Update CHANGELOG.md under [Unreleased].

Testing

  • Unit + real-byte conformance tests (test/**/*_test.exs): no live server, no postgrex dependency. The decoder conformance suite decodes REAL captured pgoutput bytes (walex's MIT-licensed capture, inlined and credited) for every message type — including the unchanged-TOAST sentinel and all replica-identity modes. It never self-signs fixtures. An independent docker-PG16 capture (test/integration/pg16_conformance_test.exs) corroborates it against a live server.
  • Integration + crash-injection tests (test/integration/**): gate on REPLICANT_TEST_URL pointing at a live PG16 with wal_level=logical; skip when unset. Spin PG16 with docker run -e POSTGRES_HOST_AUTH_METHOD=trust -p 5599:5432 postgres:16 -c wal_level=logical -c max_wal_senders=10 -c max_replication_slots=10 then export REPLICANT_TEST_URL="postgres://postgres@localhost:5599/postgres".
  • PG17 forward-compat tests (test/integration/pg17_failover_test.exs, tagged :pg17): run against a PG17 server. Spin one alongside PG16 and point REPLICANT_TEST_URL at it: docker run -e POSTGRES_HOST_AUTH_METHOD=trust -p 5617:5432 postgres:17 -c wal_level=logical -c max_wal_senders=10 -c max_replication_slots=10 then export REPLICANT_TEST_URL="postgres://postgres@localhost:5617/postgres". The :pg17 tests are auto-excluded (skipped, never vacuously passed) when the server is < 17.
  • TDD: write the test first.

Docs & lifecycle-artifact policy

  • Tracked / published: AGENTS.md, README.md, CHANGELOG.md, CONTRIBUTING.md, usage-rules.md, LICENSE, NOTICE.
  • Never tracked: all brainstorm specs, plans, exec notes, reviews, and handoffs. They live under /docs/, which is gitignored. Do not move /docs/ artifacts to the repo root.
  • AI-tool state dirs (.claude/, .serena/, etc.) are gitignored.

graphify (code knowledge graph)

graphify-out/graph.json maps this repo (tree-sitter AST; rebuilt by the git post-commit hook; gitignored).

  • For orientation ("where is X handled", "what connects A to B", "explain module M"), prefer graphify query "<question>" / graphify explain "<Module>" / graphify path "<A>" "<B>" over grep/Read fan-outs — one call returns a scoped subgraph with file:line hits.
  • Graph output is NAVIGATION, never evidence. Edges reflect the last build, not the working tree, and cross-module call edges can be incomplete (Elixir: file-local only — alias-mediated calls are NOT resolved). Consumer sweeps and every load-bearing claim (review finding, plan anchor) still verify against live code: grep + file:line read.
  • After large uncommitted changes, graphify update . refreshes the graph (AST-only, no API cost, no key).