How do you keep incremental RAG ingestion fresh, correct, and deletable?

1-minute summary

• Give each source and chunk a stable deterministic ID, then make every upsert safe to retry so duplicate delivery converges on the same state.
• Treat deletion and ACL revocation as first-class events: record a tombstone, remove chunks, vectors, and caches, and verify the content is no longer retrievable.
• Define separate freshness SLOs for updates, deletions, and permissions; measure end-to-end lag, reconcile source and index state, and retain a replay path.

Who this is for

• Teams operating RAG over documents that change after initial indexing
• Engineers building scheduled, event-driven, or hybrid ingestion pipelines
• Platform and security teams responsible for deletion, permission revocation, and audit evidence

Conclusion

A production RAG index is a derived store, not a one-time export. Its lifecycle must converge toward the source of truth after creates, updates, deletes, permission changes, retries, and partial failures.

The reliable pattern is:

1. identify each source document and derived chunk deterministically
2. persist the observed source version and the set of emitted chunk IDs
3. upsert the new version idempotently
4. tombstone removed sources before physical cleanup
5. delete obsolete chunks, vectors, enrichment artifacts, and semantic-cache entries
6. propagate ACL changes on their own clock
7. measure when the change becomes visible—or invisible—to retrieval
8. reconcile periodically and replay safely when drift appears

Vendor features can implement parts of this pattern, but their contracts differ. Azure AI Search indexers can use source-specific change and deletion detection, schedules, status history, resets, and index projections. Pinecone exposes namespace-scoped upsert and delete operations. Neither product choice removes the need for application-owned identity, manifests, SLOs, and verification.

Explanation

Incremental ingestion is a state-convergence problem

An ingestion job does more than copy text. It may extract files, split sections, enrich metadata, generate embeddings, write keyword and vector records, and warm caches. A run can fail between any two stages.

Therefore, “the job succeeded” is weaker than “retrieval matches the current authorized source state.” Track the lifecycle of each source version across every derived store.

Useful source-level state includes:

• immutable source_id
• observed source_version or monotonic change token
• content hash and ACL version
• active chunk-ID manifest
• lifecycle state such as active, tombstoned, or purged
• last successful stage and timestamps

Azure AI Search illustrates why the source contract matters. Its SQL high-water-mark policy requires inserts and updates to advance a column; Microsoft recommends rowversion for reliable synchronization under concurrent transactions. High-water-mark tracking alone cannot infer a row that was physically removed, so SQL deletion needs integrated change tracking or a configured soft-delete marker.

Stable IDs make retries and cleanup predictable

Use a stable source identifier that survives renames and ordinary edits. Derive chunk IDs from stable structure when possible—for example:

chunk_id = hash(tenant_id + source_id + section_anchor + chunk_variant)

Do not rely only on the chunk's current array position if inserting one paragraph can renumber every later chunk. If chunk boundaries change, compare the new manifest with the previous manifest and delete the IDs that disappeared.

A deterministic ID lets a retry target the same record. Pinecone documents that an upsert overwrites the entire record when the ID already exists; partial record changes use a different update operation. That is a Pinecone contract, not a universal vector-database guarantee, so confirm overwrite, merge, ordering, and consistency semantics for your chosen store.

Deletes must flow through every derived layer

Deleting the source object is not enough. A RAG pipeline can retain:

• extracted text and temporary artifacts
• chunk rows and keyword-search documents
• embeddings and vector records
• enrichment caches
• semantic caches and generated answers
• manifests, backups, exports, and evaluation fixtures

Record a tombstone before destructive cleanup so a delayed update cannot silently recreate deleted content. The tombstone should carry the source ID, deletion version, authorization scope, and retention decision. Workers can then remove derived objects idempotently and record proof of completion.

Azure's guidance is product-specific but instructive: reset plus run reprocesses content, yet does not remove orphaned search documents by itself. For Azure Blob indexers, a deletion policy must be present from the first indexer run; adding one later does not retroactively identify earlier deletions. Native blob soft-delete retention must also exceed the indexer interval long enough for the indexer to observe the deleted state.

ACL freshness is independent from content freshness

A document can keep identical text while its permitted users or groups change. Treat acl_version as separate from content_version, and trigger permission-only updates or resynchronization when the source supports them.

Azure AI Search's push API requires permission-filter fields in the schema and resolves repository permissions to identity IDs during ingestion. Its ADLS Gen2 guidance also notes that ACL changes might not update a blob's LastModified timestamp, so ordinary content change detection can miss them. The broader lesson is product-neutral: never assume a content timestamp is also a permission-change signal.

Freshness needs an observable deadline

Define SLOs from a source event to a retrieval-visible outcome:

• update freshness: source commit to the new authorized version appearing in retrieval
• deletion freshness: delete decision to all live retrieval and response caches returning no result
• ACL freshness: permission change to newly forbidden callers being denied and newly allowed callers seeing the correct version
• reconciliation freshness: maximum time drift may exist before a full comparison detects it

Measure percentiles and the oldest outstanding event, not only average job duration. A scheduler interval is just one component of lag. Azure AI Search allows scheduled indexer intervals from five minutes to 24 hours, but queueing, processing, failures, and downstream cache invalidation still affect end-to-end freshness.

Practical Guide

Step 1: define the source-of-truth contract

For every connector, document how it signals create, update, delete, restore, and permission change. Record whether its cursor is monotonic, whether events can arrive out of order, and how long delete markers remain observable.

If a connector cannot emit deletes, add a periodic full listing or inventory export. A high-water mark that sees only existing rows cannot prove that a missing row was deleted.

Step 2: create stable IDs and a source manifest

Persist one manifest per source version:

{
  "source_id": "tenant_7:policy_42",
  "source_version": "018f3d6a",
  "acl_version": "acl_109",
  "state": "active",
  "chunk_ids": ["c_31aa", "c_90bf", "c_a712"]
}

Generate the new chunk set, compare it with the last committed manifest, upsert current IDs, and delete the difference. Azure index projections similarly preserve a parent key while mapping one source into child search documents; follow its mapping rules if you use that feature rather than assuming generic parent-child behavior.

Step 3: make the handler idempotent and version-aware

The following is product-neutral pseudocode:

async function applySourceEvent(event: SourceEvent) {
  const current = await manifests.get(event.sourceId);

  if (current && compareVersion(event.version, current.version) <= 0) return;

  if (event.type === 'deleted') {
    await tombstones.put(event.sourceId, event.version);
    await removeDerivedData(current?.chunkIds ?? [], event.sourceId);
    await manifests.markPurged(event.sourceId, event.version);
    return;
  }

  if (await tombstones.blocks(event.sourceId, event.version)) return;

  const next = await buildChunks(event);
  await vectorStore.upsert(next.records);
  await deleteIds(difference(current?.chunkIds ?? [], next.chunkIds));
  await caches.invalidateBySource(event.sourceId);
  await manifests.commit(next.manifest);
}

Use compare-and-set, transactions, or an outbox where your infrastructure supports them. The exact atomicity mechanism is platform-specific. The invariant is that an older event must not overwrite a newer version, and repeating one event must not create extra records.

Step 4: design the tombstone lifecycle

Use explicit states such as:

active -> tombstoned -> derived-data-deleting -> purged

Keep tombstones long enough to cover delayed events, retries, connector lookback, and restore policy. Do not hard-code a universal duration; derive it from the source system and retention obligations. A restore should be a new versioned event, not an unversioned removal of the tombstone.

Step 5: delete vectors, chunks, and caches together

Build a deletion fan-out keyed by source_id and its recorded chunk manifest. Pinecone's 2026-04 API can delete records in one namespace by IDs or by metadata filter, and also exposes a namespace-wide deleteAll option. Other stores differ, so test the actual API contract and choose the narrowest safe selector.

After deletion, query by exact source ID and representative content under every relevant tenant or ACL scope. Also verify keyword indexes, semantic caches, generated-answer caches, and enrichment artifacts. Azure's enrichment-cache documentation requires an indexer deletion policy for synchronized blob deletion from both cache and index.

Step 6: propagate permissions as first-class changes

Store acl_version on every chunk. When permissions change:

1. resolve the current source ACL from the trusted repository
2. write the new permission fields to all active chunks
3. invalidate caches keyed by the old authorization scope
4. test a revoked principal and a newly permitted principal
5. record propagation completion against the ACL freshness SLO

Fail closed while required ACL metadata is missing or inconsistent. A fast content pipeline with stale permissions is not fresh enough.

Step 7: operate schedules, monitoring, and retries

Choose event-driven ingestion for low latency, scheduled catch-up for resilience, or both. Prevent overlapping workers from racing on the same source by using version checks or source-scoped serialization.

Monitor counts and lag by stage: events received, sources parsed, chunks emitted, upserts accepted, deletes accepted, caches invalidated, and verification passed. Azure AI Search's status API exposes the latest result and recent execution history, including processed and failed item counts. Treat those as pipeline evidence, not proof that application retrieval is correct; add an end-to-end probe.

Step 8: reconcile and replay

Run a periodic reconciliation that compares:

• source inventory vs. active manifests
• manifest chunk IDs vs. keyword and vector IDs
• source content and ACL versions vs. indexed metadata
• tombstones vs. any retrievable derived records

Classify mismatches before repair: missing, stale, orphaned, ACL-divergent, or duplicate. Emit ordinary versioned repair events through the same idempotent handler instead of maintaining an untested side path.

Retain enough immutable event or snapshot data to replay a connector, tenant, source, or time range. Azure indexer reset is one product's full-reprocessing control, but Microsoft explicitly notes that reset must be followed by a run and does not itself clean up orphaned documents. Replay and deletion verification remain separate concerns.

Pitfalls

• generating random chunk IDs on every retry
• using chunk array positions as the only stable identity
• accepting an older out-of-order event after a newer version
• physically deleting a source before downstream workers can observe a delete marker
• assuming a reset or full reindex automatically removes orphaned records
• deleting vectors but leaving keyword documents, extracted text, or caches
• applying a deletion policy only after the initial index already contains stale data
• treating content timestamps as ACL-change timestamps
• measuring scheduler success instead of retrieval-visible freshness
• retrying upserts without confirming the store's overwrite or merge semantics
• replacing a manifest before obsolete IDs have been identified
• running reconciliation through a separate, non-idempotent repair path
• keeping no replay boundary by tenant, connector, or source
• treating accepted write responses as proof that retrieval has converged

Checklist

• [ ] Every source has an immutable, tenant-scoped source_id
• [ ] Every chunk ID is deterministic across retries
• [ ] Chunk identity does not depend only on its current list position
• [ ] Source content and ACL versions are tracked separately
• [ ] Older and duplicate events cannot overwrite newer state
• [ ] Upsert overwrite, merge, consistency, and error semantics are verified for the selected store
• [ ] Each source manifest records all current chunk IDs
• [ ] Removed chunk IDs are computed before the new manifest is committed
• [ ] Deletes create durable tombstones before derived-data cleanup
• [ ] Tombstone retention covers retries, delayed delivery, and restore policy
• [ ] Keyword records, vectors, extracted artifacts, enrichments, and caches are all in deletion scope
• [ ] Deletion is verified through the production retrieval path
• [ ] ACL-only changes trigger chunk metadata and cache updates
• [ ] Missing or inconsistent ACL metadata fails closed
• [ ] Update, delete, and ACL freshness have separate SLOs
• [ ] Dashboards show end-to-end lag and the oldest outstanding event
• [ ] Failed and partial runs are retryable without duplicate records
• [ ] Reconciliation detects missing, stale, orphaned, and ACL-divergent records
• [ ] Repair events use the normal idempotent ingestion path
• [ ] Replay can be scoped by connector, tenant, source, and time range
• [ ] Full rebuild drills include orphan cleanup and post-rebuild retrieval probes
• [ ] Backups and exports follow documented deletion and retention rules

FAQ

1. Is an upsert enough to update a document?

Only if you know the store's ID and overwrite semantics and you remove chunks that no longer exist. In Pinecone, upserting an existing ID overwrites the entire record. That does not delete different IDs emitted by an older chunking result, so keep a manifest and delete the obsolete set.

2. Can a full reindex replace explicit delete handling?

Not safely as a general rule. Some rebuild workflows overwrite records that still exist but leave orphans. Azure AI Search documents that reset and run do not remove source-less documents by themselves. Configure supported deletion detection from the beginning or issue explicit delete actions, then verify retrieval.

3. How should we choose a freshness SLO?

Start from user and security impact. Permission revocation and legal deletion may need a tighter deadline than ordinary content updates. Include connector delay, queue time, processing, index visibility, and cache invalidation; then measure the complete path rather than copying the scheduler interval.

4. Why keep tombstones after the vectors are deleted?

A delayed update, retry, or replay can otherwise recreate deleted content. The tombstone supplies the deletion version and blocks older events. Retention depends on the connector's maximum delay, replay window, restore behavior, and applicable retention requirements.

5. How do we know reconciliation is complete?

Define the expected source population and compare IDs and versions across source inventory, manifests, indexes, and tombstones. A successful job with zero reported errors is not enough if an omitted source was never presented to the job.

Sources

• Run or reset indexers, skills, or documents (Microsoft Learn)
• Azure SQL indexer: change and deletion detection (Microsoft Learn)
• Changed and deleted blobs in Azure AI Search (Microsoft Learn)
• Define index projections (Microsoft Learn)
• Configure an enrichment cache (Microsoft Learn)
• Upsert records (Pinecone Docs)
• Delete records, API version 2026-04 (Pinecone Docs)
• Index ACLs using the push REST API (Microsoft Learn)
• Schedule indexer execution (Microsoft Learn)
• Monitor indexer status and results (Microsoft Learn)
• Pinecone quickstart

Internal links

• Parent hub: AI development
• Related:
  • RAG chunking and indexing checklist
  • Multi-tenant RAG access control checklist
  • RAG retrieval evaluation checklist

Disclaimer

General engineering and security guidance only. Validate lifecycle behavior, API versions, consistency, retention, deletion, and permission semantics against your actual source systems, search platform, vector store, contracts, and regulatory obligations.