Sync Cursor / Watermark

The saved bookmark that tells the next sync where the last successful one stopped, so it can continue instead of starting over.

remember where the last sync stoppedbookmark for a data syncsave the last record we importedcontinue syncing from here next timewhere did last night's import stophigh water marksync check pointsync curser

See it

Live demo coming soon

What it is

A sync cursor or watermark is the saved bookmark for a data transfer. A provider may hand back an opaque cursor, or you may store a high-water mark such as the last (updated_at, id) pair processed. The next run starts after that position instead of reading the entire source again.

Reach for one in any recurring import, export, or mirror. Store it per account and object type, and advance it only after the corresponding page of records is durably committed. A timestamp alone is rarely a safe bookmark because many rows can share it; pair it with a unique ID and keep the sort order fixed.

The dangerous bug is moving the watermark before the data is saved. A crash in between permanently skips that page. Moving it afterward may repeat work, which is why writes behind the sync must be idempotent. Opaque cursors can also expire or be rejected, so define a full-resync path instead of treating a broken cursor as an impossible state.

Ask AI for it

Implement a resumable Salesforce Account sync using the tuple (SystemModstamp, Id) as the watermark. Query with SOQL ordered by SystemModstamp then Id, page through every result, and store the watermark per Salesforce org in PostgreSQL. Apply each page with idempotent writes, then advance the checkpoint in the same database transaction only after that page commits. Use an inclusive overlap on restart, deduplicate by Salesforce Id, and add a full-resync path for a missing or invalid checkpoint.

You might have meant

incremental syncpaginationpollingidempotency keyreconciliation