Weekly status — Week 19, 2026
Platform team · 2026-05-04 → 2026-05-09
- Ingestion API throughput is up 31% on the new token-bucket limiter (PR-2841 shipped Wednesday)
- Vector index post-mortem (IR-2026-04-29) closed; 3 of 5 action items completed, the remaining 2 are queued for next week
- Hybrid retrieval RFC (RFC-2026-014) drafted; review window opens Monday
- One open risk: cognify ingestion compute cost trending 18% above projection
Numbers
| Metric | Last week | This week | Δ |
|---|---|---|---|
| Ingestion API requests/day | 4.2M | 4.5M | +7.1% |
| P95 ingestion latency | 31ms | 14ms | −55% |
| 5xx error rate | 0.4% |
0% |
−100% |
| Retrieval P95 | 1.42s | 1.41s | flat |
| Cognify ingestion cost (mtd) | $87 |
$102 |
+17% |
| Open P1 incidents | 1 | 0 | −1 |
The retrieval P95 hasn’t moved because the hybrid RFC isn’t shipped yet — that’s where we expect the 3× improvement. The 0.55× ingestion latency improvement is from the token-bucket smoothing, not the retrieval architecture change.
What landed this week
Token-bucket rate limiter (PR-2841)
Shipped Wednesday 2026-05-08. P99 latency dropped from 87ms to 32ms. Second-boundary 5xx errors eliminated entirely. See the PR explainer in examples/pr-explainer-rate-limiter.qmd for the diff and load-test results.
IR-2026-04-29 post-mortem
Closed Tuesday. Vector index reset on production cost 4.27h of degraded retrieval. Root cause: schema migration without lock acquisition. Action items:
Hybrid retrieval RFC
Drafted, ready for review. RFC-2026-014. Two open questions remain:
- Entity disambiguation when cognify NER doesn’t split distinct entities sharing a label
- Cold-start retrieval for documents ingested but not yet processed by cognify
Both have proposed answers in §5. Decision needed by 2026-05-20.
Open risks
The cognify ingestion compute is currently running at $102 month-to-date against a projected $87. That’s 17% over budget on a per-month basis; if the trend holds, end-of-month spend will be ~$161 vs. the $137 budget. Investigation: the H2SO4 citation in our chemistry pipeline corpus added a high-cardinality entity-relation set per document, multiplying the per-doc extraction cost. Considering: (a) cap relation extraction at depth 2 instead of full graph; (b) batch low-priority documents to off-peak compute. Decision next week.
Coming up next week
- W20-1: Land the Qdrant client wrapper (IR-2026-04-29 follow-up)
- W20-2: Open RFC-2026-014 for review; aim for ratification by
2026-05-20 - W20-3: Investigate the cognify cost overshoot; pick a mitigation
- W20-4: Begin scoping the hybrid retrieval migration (assumes RFC ratifies)
The post-mortem is closed but the lessons aren’t shipped yet. Discipline doesn’t scale; structural prevention does. W20 is the structural-prevention week.
Personal notes
This is week 7 on the platform team. The handoffs from the previous tech lead are mostly complete; only the runbook archeology remains. The team is shipping; the metrics are moving. The cognee architecture migration is the next big bet — RFC-2026-014 starts the conversation, but the conversation will take 4-6 weeks to settle.
File this under “things I wish I’d asked sooner”: the original Qdrant deployment was sized for 100k documents, not 4.7M. The cognify migration is partly a cost-control move, not just an architectural improvement.