Post-mortem · IR-2026-04-29: Vector index reset on production

Severity: P1 · Customer impact: 4h17m of degraded retrieval · Root cause: missing index lock

Author

Incident Commander: Christos Despotakis

Published

April 30, 2026

ImportantTL;DR

A schema migration script ran during a maintenance window without acquiring the Qdrant write lock. A concurrent ingestion job re-keyed the collection, dropping 4.7M vector embeddings. Recovery from snapshot took 4.27 hours of degraded retrieval. Action items below; no customer data lost (vectors are derived from immutable source corpus).

Timeline

All times Athens (UTC+3).

Time Event
04:00 Scheduled maintenance window begins. Migration script migrate-v3.py starts.
04:02 Concurrent ingestion job (cron) fires for daily Slack archive ingest.
04:03 Migration script begins re-indexing without acquire_lock=True.
04:04 Ingestion job’s bulk-upsert collides with re-index. Collection state goes inconsistent.
04:11 Qdrant detects checksum mismatch, fails into safe mode (read-only).
04:12 Production retrieval requests start returning empty results.
04:18 First customer ticket. PagerDuty fires.
04:21 On-call (christos) acknowledges, opens warroom.
04:34 Decision: restore from 2026-04-29 03:30 snapshot (last-known-good).
04:42 Restore begins. Estimated 3h.
06:55 Restore completes. Validation passes.
06:57 Read traffic restored. Write traffic deferred 30m to verify.
08:29 Full operations resumed. Total degraded window: 4.27h.

Snapshots run every 24h at 03:30, retained 14 days. The window between 03:30 snapshot and 04:00 migration was 30 minutes — within tolerance, no data loss beyond ingest replay.

Root cause

The migration script was a recent addition that did not follow the team’s lock-acquisition protocol. The relevant block:

@@ -18,4 +18,6 @@ def migrate_collection_v3():     client = QdrantClient(host=PROD_HOST)+    # MISSING — should be:+    # with client.acquire_write_lock(collection="documents", timeout=600):     client.recreate_collection(         collection_name="documents",         vectors_config=VectorParams(size=1536, distance=Distance.COSINE),     )

The lock-acquisition protocol is documented in docs/runbooks/migrations.md, written 2025-08-14. The author of migrate-v3.py had not read it.

What we saw in the logs

A representative trace from the moment the collision happened:

[2026-04-29T04:03:11Z] migrate-v3 INFO  recreate_collection start
[2026-04-29T04:03:11Z] ingest-slack INFO  bulk_upsert begin (12,481 vectors)
[2026-04-29T04:03:14Z] qdrant       WARN  inconsistent state during write — segment_id=84372
[2026-04-29T04:03:14Z] qdrant       WARN  inconsistent state during write — segment_id=84373
[2026-04-29T04:03:15Z] qdrant       ERROR collection 'documents' checksum mismatch
[2026-04-29T04:03:15Z] qdrant       ERROR entering safe mode (read-only)
[2026-04-29T04:11:02Z] api-gateway  WARN  empty result set rate spike: 412/min (baseline 4)

When the war room opened, christos pulled the recent state directly:

$ kubectl exec -n production qdrant-0 -- /qdrant/qdrant cluster-infostatus: SAFE_MODEread_only: truelast_consistent_snapshot: 2026-04-29T03:30:01Z$ kubectl logs -n production qdrant-0 --since=30m | grep -E "ERROR|WARN" | head2026-04-29T04:03:14Z WARN  inconsistent state during write — segment_id=843722026-04-29T04:03:14Z WARN  inconsistent state during write — segment_id=843732026-04-29T04:03:15Z ERROR collection 'documents' checksum mismatch2026-04-29T04:03:15Z ERROR entering safe mode (read-only)2026-04-29T04:11:02Z ERROR retrieval request failed — collection in safe mode

Action items

WarningCritical (this week)
  1. Add lock acquisition lint check — migrate-* scripts must call acquire_write_lock or fail CI. Owner: @christos. ETA: 2026-05-02.
  2. Make recreate_collection require explicit force=True — Qdrant client wrapper. Owner: @ops. ETA: 2026-05-03.
  3. Document the migration runbook prominently in CODEOWNERS — anyone touching migrate-* files gets auto-assigned a runbook-reading reviewer. Owner: @christos. ETA: 2026-05-05.
NoteHigher leverage (this quarter)
  1. Migrate Qdrant client to a wrapper that always serializes destructive operations through a coordinator. Eliminates this class of bug structurally rather than via discipline.
  2. Snapshots → continuous WAL streaming for sub-minute RPO instead of 24h cadence.

The lock-acquisition protocol existed. The runbook existed. The author of the script had not read either. Discipline doesn’t scale; structural prevention does.

Acknowledgements

To the on-call team that kept the war room calm. To the customers who reported the degradation through quiet tickets rather than fire-emoji Twitter posts. And to the postgres dba who reminded us, mid-incident, that “snapshots are a tax on tomorrow that you pay today.”