Scheduling data sync and ETL jobs with cron

Incremental sync/ETL runs with locking and idempotency — and where cron alone stops being enough.

The problem

ETL and sync jobs are the case where an overlapping-run bug does the most damage, because these jobs mutate state. A sync job that reads "records updated since last run" and writes a new watermark at the end will, if two instances overlap, have both instances read the same starting watermark and one of them overwrite the other's progress — silently reprocessing or skipping a window of records.

The second failure is partial-write visibility: a job that dies halfway through a multi-table load can leave downstream tables in an inconsistent state that looks fine to any single query but is wrong when joined.

Recommended schedule

*/15 * * * *

Every 15 minutes balances freshness against load on the source system for most incremental syncs. If a run can plausibly take longer than the interval, either widen the interval or move to a job queue with explicit run-tracking instead of relying on cron's clock alone.

Check any expression at the cron parser, or browse more at the cron guide.

Example crontab entry

# norc: etl-sync Incremental orders sync
*/15 * * * * /usr/bin/flock -n /var/lock/etl-sync.lock /opt/etl/sync.sh >> /var/log/etl-sync.log 2>&1

#!/usr/bin/env bash
set -euo pipefail
# write the new watermark only after the load fully commits,
# so a crash mid-run re-processes the same window next time (idempotent upsert)
python3 sync.py --since-watermark-file /var/lib/etl/watermark.txt --commit-watermark-on-success

Failure modes to watch for

• Unlocked overlapping runs racing to read and write the same watermark, causing silently reprocessed or skipped windows.
• A job that dies mid-load without transactional or idempotent writes, leaving destination tables inconsistent until the next successful run papers over it — or doesn't.
• Watermark written before the load is confirmed committed, so a crash after the watermark update permanently loses that window's records.
• Source-system rate limiting or connection-pool exhaustion when the sync interval is shorter than the source can reliably serve, which looks like an ETL bug but is actually a scheduling problem.

How norc helps

For sync jobs that run across multiple machines, norc gives one place to see whether every host's sync actually completed this cycle, rather than SSHing into each one to check its log.