BadgeprintingEvent registration · badge automation

Polling Eventbrite Web APIs Without Rate Limiting: Fixing 429 Exhaustion & Cursor Drift

Symptom Statement

The attendee ingestion loop stalls: badge generation queues stop draining, the poller process pins a core at 100% CPU, and every call to GET /v3/events/{id}/attendees/ comes back 429 Too Many Requests until the hourly token budget resets. On-site the failure is visible within minutes — check-in latency climbs, the attendee roster diverges from the Eventbrite ledger, and unprinted badges stall downstream settlement so payment-reconciliation gaps widen. This page addresses that exact symptom: a fixed-interval poller that exhausts the Eventbrite request budget and never recovers cleanly. It is part of the Form API Polling Strategies stage, the deterministic pull-based entry point that backstops payment webhook handling when real-time delivery drops, and it feeds validated records into the wider Registration Ingestion & Payment Reconciliation pipeline. The tell is always the same combination: a 429 storm, a cursor that never advances, and duplicate badge jobs appearing after every restart.

Root Cause Analysis

Rate-limit exhaustion against Eventbrite is architectural, not infrastructural — throwing more retries or a bigger box at it makes the storm worse. Four concrete root causes account for essentially every incident.

  1. Rate-limit blindness. The loop treats the API as stateless, ignores the response’s remaining-capacity signal, and fires on a fixed timer. Eventbrite’s documented budget is roughly 2,000 calls/hour and 48,000/day per token; a 1-second loop burns the hourly allowance in under 35 minutes and then hard-fails for the rest of the window.
  2. Cursor amnesia (cursor drift). The continuation pagination token and the changed_since high-water mark are held only in memory. Any restart resets ingestion to the epoch, so the poller re-scans the entire event from scratch — multiplying request volume by the page count on every deploy or crash.
  3. Retry-storm amplification. Naive retry logic reacts to a 429 or 5xx with an immediate, un-jittered retry. Multiple workers synchronize into a thundering herd that re-hammers the endpoint the instant the window opens, guaranteeing the next 429 before real work resumes.
  4. Missing idempotency. When a transient network partition forces a retry mid-page, the same attendee records are processed twice. Without a deduplication guard that survives the retry, duplicate badge jobs reach the print queue and corrupt the reconciliation count.
One adaptive poll cycle — persistent cursor, token-bucket gate, idempotent handoff, and backoff loops A persistent SQLite cursor store (changed_since plus continuation token) feeds a token-bucket gate. The gate blocks until the hourly window refills when empty, otherwise fetches a page from the Eventbrite v3 attendees endpoint. A 429 or 5xx routes through a Retry-After / backoff-with-jitter box back to the gate. A 200 page fans through a SHA-256 idempotency check that branches to SKIP (duplicate, no-op) or PROCESS (hand off and advance changed_since, persisting the continuation token back to the store). A page with no continuation drains the dataset, resets the continuation token, and sleeps to the next window before looping back to the gate. One adaptive poll cycle — persistent cursor, budget-gated dispatch, idempotent handoff advance changed_since · persist continuation token → cursor store SQLite cursor store changed_since continuation token WAL · durable Token-bucket gate ≤ 2000 calls / hour dispatch on remaining budget empty → wait for refill Eventbrite API GET …/attendees/ changed_since + continuation SHA-256 dedup sha256(id : changed) seen this cycle? PROCESS hand off → badge queue advance cursor SKIP duplicate → no-op 429 / 5xx backoff honor Retry-After exp. backoff + jitter dataset drained reset continuation sleep to next window resume fetch page 200 page new dup on error no next page retry next cycle

Symptom-to-Resolution Matrix

Rate-Limit Blindness Burns the Hourly Budget

  • Symptom: 429 Too Many Requests arrives in bursts; the Eventbrite error body reads RATE_LIMIT_EXCEEDED; capacity recovers only on the hour boundary, not gradually.
  • Root Cause: Dispatch velocity is decoupled from API capacity. A fixed time.sleep(1) loop has no notion of the remaining budget, so it keeps firing after the tenant allowance is spent.
  • Fix:
    1. Gate every request behind a token bucket sized to the tier limit: self.tokens = 2000, refilling on a 3600-second window.
    2. Block dispatch when the bucket empties — while self.tokens <= 0: wait_for_window_reset() — instead of issuing a doomed call.
    3. On any 429, honor the server’s Retry-After header verbatim: time.sleep(int(resp.headers.get("Retry-After", 60))).
    4. Add adaptive pacing between pages that scales with remaining capacity, so a near-full bucket runs fast and a near-empty one throttles itself.

Cursor Amnesia Re-Scans the Whole Event

  • Symptom: Request volume spikes after every deploy or restart; identical attendee pages are fetched repeatedly; the changed_since value in logs is always 2020-01-01.
  • Root Cause: Pagination and high-water state live only in process memory, so a crash discards them and ingestion restarts from the epoch.
  • Fix:
    1. Persist both the continuation token and the changed_since timestamp to a durable store before the next call — a local SQLite table is enough.
    2. Enable WAL mode (PRAGMA journal_mode=WAL) so state writes never block the read path.
    3. Advance changed_since monotonically from each record’s changed field, and write it immediately after each page rather than at end-of-run.
    4. On startup, read the persisted cursor first and resume from it; treat the epoch only as a cold-start default.

Retry Storms Re-Exhaust the Window

  • Symptom: After a brief recovery the endpoint returns 429 again instantly; multiple workers retry in lockstep; capacity never stabilizes above zero.
  • Root Cause: Un-jittered, unbounded retries synchronize across workers into a thundering herd that reopens the window and immediately re-closes it.
  • Fix:
    1. Use truncated exponential backoff for 5xx and network errors: delay = min(2 ** attempt, 60).
    2. Add uniform jitter to desynchronize workers: delay += random.uniform(0, delay * 0.25).
    3. Reset the attempt counter to 0 only on a clean 2xx, so backoff genuinely widens under sustained failure.
    4. Cap concurrency: run a single poller per event token and let it drain sequentially rather than fanning out parallel readers against one budget.

Missing Idempotency Duplicates Badge Jobs

  • Symptom: Duplicate confirmation records and identical badge print jobs appear after a network blip; the reconciliation count exceeds the paid-ticket count.
  • Root Cause: A retry after a partial page re-processes attendees already handed off downstream, and nothing detects the repeat.
  • Fix:
    1. Hash a stable identity per record — sha256(attendee_id + ":" + changed_timestamp) — before any downstream dispatch.
    2. Skip any hash already seen this cycle so a mid-page retry is a no-op.
    3. Bound the dedup cache with LRU eviction (cap at 50,000 entries) so a multi-day event never exhausts memory.
    4. Route anything that fails validation to the dead-letter queue instead of dropping it, keeping the paid-vs-printed ledger auditable.

Minimal Working Implementation

The poller below is self-contained — standard library plus requests. It persists the cursor to SQLite, gates dispatch through a header-aware token bucket, backs off with jitter on failure, and deduplicates every record before handoff. The changed_since parameter and continuation pagination match Eventbrite’s v3 attendees endpoint. The final block is the verification step: run it, kill it mid-drain, and restart — it resumes from the persisted cursor instead of re-scanning.

PYTHON
import os
import time
import json
import random
import hashlib
import sqlite3
import requests
from collections import OrderedDict
from typing import Optional, Dict, Any

EVENTBRITE_API_BASE = "https://www.eventbriteapi.com/v3"
RATE_LIMIT = 2000        # calls per window (Eventbrite standard hourly budget)
RESET_WINDOW = 3600      # seconds
MAX_HASH_CACHE = 50_000  # bounded idempotency footprint
COLD_START = "2020-01-01T00:00:00Z"


class EventbritePoller:
    def __init__(self, api_token: str, event_id: str, db_path: str = "poller_state.db"):
        self.event_id = event_id
        self.db_path = db_path
        self.session = requests.Session()
        self.session.headers.update({
            "Authorization": f"Bearer {api_token}",
            "Accept": "application/json",
            "User-Agent": "EventOps-BadgePipeline/1.0",
        })
        self.tokens = RATE_LIMIT
        self.last_refill = time.time()
        self.seen: "OrderedDict[str, bool]" = OrderedDict()
        self._init_db()

    def _init_db(self) -> None:
        with sqlite3.connect(self.db_path) as conn:
            conn.execute("PRAGMA journal_mode=WAL;")
            conn.execute(
                "CREATE TABLE IF NOT EXISTS poll_state (key TEXT PRIMARY KEY, value TEXT)"
            )

    def _get_state(self, key: str) -> Optional[str]:
        with sqlite3.connect(self.db_path) as conn:
            row = conn.execute(
                "SELECT value FROM poll_state WHERE key=?", (key,)
            ).fetchone()
            return row[0] if row else None

    def _set_state(self, key: str, value: str) -> None:
        with sqlite3.connect(self.db_path) as conn:
            conn.execute(
                "INSERT OR REPLACE INTO poll_state (key, value) VALUES (?, ?)",
                (key, value),
            )

    def _is_duplicate(self, record: Dict[str, Any]) -> bool:
        key = f"{record.get('id')}:{record.get('changed')}"
        digest = hashlib.sha256(key.encode()).hexdigest()
        if digest in self.seen:
            return True
        if len(self.seen) >= MAX_HASH_CACHE:
            self.seen.popitem(last=False)  # evict oldest
        self.seen[digest] = True
        return False

    def _wait_for_token(self) -> None:
        while self.tokens <= 0:
            if time.time() - self.last_refill >= RESET_WINDOW:
                self.tokens = RATE_LIMIT
                self.last_refill = time.time()
            else:
                time.sleep(1)

    def _consume_token(self, headers: Dict[str, str]) -> None:
        remaining = headers.get("X-RateLimit-Remaining")
        self.tokens = int(remaining) if remaining is not None else self.tokens - 1

    def _backoff(self, attempt: int) -> float:
        delay = min(2 ** attempt, 60)
        return delay + random.uniform(0, delay * 0.25)

    def fetch_attendees(self) -> None:
        continuation = self._get_state("continuation_token") or None
        changed_since = self._get_state("changed_since") or COLD_START
        attempt = 0

        while True:
            self._wait_for_token()
            params: Dict[str, Any] = {
                "status": "attending",
                "changed_since": changed_since,
                "expand": "ticket_class",
            }
            if continuation:
                params["continuation"] = continuation

            try:
                resp = self.session.get(
                    f"{EVENTBRITE_API_BASE}/events/{self.event_id}/attendees/",
                    params=params,
                    timeout=15,
                )
                resp.raise_for_status()
            except requests.exceptions.HTTPError as exc:
                status = exc.response.status_code
                if status == 429:
                    retry_after = int(exc.response.headers.get("Retry-After", 60))
                    time.sleep(retry_after)
                    continue
                if status >= 500:
                    time.sleep(self._backoff(attempt))
                    attempt += 1
                    continue
                raise
            except requests.exceptions.RequestException:
                time.sleep(self._backoff(attempt))
                attempt += 1
                continue

            attempt = 0
            self._consume_token(resp.headers)
            body = resp.json()

            for attendee in body.get("attendees", []):
                if self._is_duplicate(attendee):
                    continue
                self._process_attendee(attendee)
                changed = attendee.get("changed")
                if changed and changed > changed_since:
                    changed_since = changed
            self._set_state("changed_since", changed_since)

            continuation = body.get("pagination", {}).get("continuation")
            if continuation:
                self._set_state("continuation_token", continuation)
                # adaptive pacing: slow down as the bucket empties
                time.sleep(max(0.5, (1 - self.tokens / RATE_LIMIT) * 5))
                continue

            # dataset drained: clear pagination, advance window, wait for next cycle
            self._set_state("continuation_token", "")
            changed_since = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
            self._set_state("changed_since", changed_since)
            time.sleep(RESET_WINDOW)

    def _process_attendee(self, attendee: Dict[str, Any]) -> None:
        # Replace with the badge-queue push / reconciliation sync.
        print(f"process {attendee['id']} changed={attendee.get('changed')}")


if __name__ == "__main__":
    token = os.getenv("EVENTBRITE_API_TOKEN")
    event_id = os.getenv("EVENTBRITE_EVENT_ID")
    if not token or not event_id:
        raise EnvironmentError("Set EVENTBRITE_API_TOKEN and EVENTBRITE_EVENT_ID")
    EventbritePoller(token, event_id).fetch_attendees()

Memory & Performance Constraints

Component Constraint Mitigation
Token bucket Hourly budget (~2,000 calls) is a hard ceiling per token Gate every call; adaptive pacing scales sleep with remaining capacity so a near-empty bucket throttles itself
SQLite cursor store Write lock can block the read path under contention PRAGMA journal_mode=WAL; single-writer poller per event token
Idempotency cache Unbounded OrderedDict grows with a multi-day event and risks OOM LRU eviction capped at 50,000 hashes; state that must survive restart lives in SQLite, not the cache
Response handling Buffering a full attendee page holds redundant JSON in memory Iterate body["attendees"] record-by-record and discard each after dedup + handoff
Retry logic Synchronized retries re-exhaust the window Truncated exponential backoff with uniform jitter; attempt counter resets only on 2xx

Incident Triage & Rollback

Fast path — restore stability in under five minutes:

  1. Pause the poller. SIGSTOP the process or scale the container to zero so no further calls hit the spent budget.
  2. Check cursor integrity. sqlite3 poller_state.db "PRAGMA integrity_check;" and inspect the high-water mark: sqlite3 poller_state.db "SELECT * FROM poll_state;".
  3. Clear a stuck cursor. If the continuation token points at an expired Eventbrite page, drop it: sqlite3 poller_state.db "DELETE FROM poll_state WHERE key='continuation_token';" — the next cycle re-paginates from the persisted changed_since.
  4. Confirm the window is open. Issue one manual probe and read the response: curl -sS -H "Authorization: Bearer $EVENTBRITE_API_TOKEN" "https://www.eventbriteapi.com/v3/users/me/" -o /dev/null -w '%{http_code}\n' — a 200 means the budget has refilled.

Rollback: revert to the previous poller image or revision. The dedup cache is ephemeral by design, so a restart safely rebuilds it; only the SQLite cursor is authoritative, and it is append-safe. If a bad deploy corrupted the cursor, restore poller_state.db from the last known-good backup before restarting.

Post-rollback validation: confirm the loop resumes without a 429 and that the cursor advances — watch -n5 'sqlite3 poller_state.db "SELECT value FROM poll_state WHERE key=\"changed_since\";"' should show the timestamp climbing. Verify badge-queue depth tracks changed_since progression and that the reconciliation gap closes within one full poll cycle.

Frequently Asked Questions

Does Eventbrite send X-RateLimit-Remaining on every response? Not reliably across every endpoint. The poller reads the header when present and otherwise decrements the bucket locally, so capacity accounting degrades gracefully instead of assuming a signal that may be absent. The authoritative recovery signal is the 429 plus its Retry-After header.

Why persist changed_since to SQLite instead of just polling everything each cycle? Re-scanning the full event every cycle is exactly the behavior that burns the hourly budget. Persisting the high-water mark means each cycle fetches only records changed since the last run, so request volume tracks real registration activity rather than event size.

Should I run one poller per event or fan out parallel workers? One poller per event token. Multiple readers share a single hourly budget and synchronize into retry storms. A single sequential drainer with adaptive pacing is both faster in aggregate and far less likely to trip RATE_LIMIT_EXCEEDED.

Where do malformed attendee records go? Not into the badge queue. Validation belongs one stage over in the schema validation pipelines contract; anything that fails is routed to the dead-letter queue so the paid-vs-printed ledger stays auditable rather than silently short a badge.