PDF Routing Workflows: Deterministic Badge Dispatch & Fallback Chains

PDF routing is the stage that decides where a finished badge goes and guarantees it arrives exactly once. It is part of the Badge Generation & Template Sync pipeline, sitting at the strict boundary between asset generation and downstream fulfillment: once a badge payload clears composition, the routing engine assumes sole responsibility for deterministic delivery, transport-path resolution, and state persistence. This stage does not compose layouts, embed fonts, or rasterize glyphs — it enforces delivery contracts, manages transport protocols, and guarantees idempotent handoffs to print queues, email relays, or cloud storage.

The operational problem this stage solves is silent loss under load. At a 5,000-attendee conference, a routing engine that swallows a transport exception or double-delivers a badge does not fail loudly — it produces a queue of people at the registration desk holding no badge, or a print spooler churning out duplicates. The failure mode this layer exists to prevent is precisely that class of quiet, load-triggered delivery gap: a payload that validated upstream but never reached a printer, an AWS SES relay that timed out without escalating, or a presigned URL that expired between generation and dispatch. Everything below is engineered for partial-failure tolerance, explicit fallback chains, and strict schema validation so that no badge is lost between “rendered” and “delivered.”

Scope Boundary Link to this section#

The routing layer stays stateless and well-bounded by delegating everything that is not a delivery decision to an adjacent stage. Holding this line is what keeps the component horizontally scalable and free of template-drift or hardware coupling.

The engine treats the PDF as an opaque artifact: it never parses, mutates, or re-renders the asset. It reads the checksum, resolves the target, and dispatches.

Data Contract Link to this section#

The routing boundary expects a normalized, immutable payload. Deviating from this contract introduces the silent routing failures that cascade into lost badges during peak registration surges, so the contract is enforced synchronously — before any socket is opened — using Pydantic v2. Every field is load-bearing:

• registration_id — a UUID v4 that serves as the primary idempotency key. It must remain constant across every retry and fallback so the downstream ledger can suppress duplicates.
• asset_path — a presigned https:// or s3:// URL pointing at the finished PDF. The engine never accepts a local path or an unauthenticated scheme.
• target — an enum (print_queue, email_relay, s3_archive) that selects the primary transport adapter.
• checksum — a SHA-256 hex digest of the asset bytes, used for end-to-end integrity verification without re-fetching or re-encoding.
• correlation_id — the trace identifier propagated from ingestion, threaded through every log line so a single badge’s journey is reconstructable across services.

import hashlib
import re
from enum import Enum
from typing import Any, Optional

from pydantic import BaseModel, ConfigDict, Field, field_validator

UUID_V4 = re.compile(
    r"^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$"
)
SHA256 = re.compile(r"^[0-9a-f]{64}$")


class RoutingTarget(str, Enum):
    PRINT_QUEUE = "print_queue"
    EMAIL_RELAY = "email_relay"
    S3_ARCHIVE = "s3_archive"


class RoutingPayload(BaseModel):
    """Immutable delivery contract. Unknown fields are rejected outright."""

    model_config = ConfigDict(extra="forbid", frozen=True, str_strip_whitespace=True)

    registration_id: str
    asset_path: str
    target: RoutingTarget
    checksum: str
    correlation_id: Optional[str] = None
    metadata: dict[str, Any] = Field(default_factory=dict)

    @field_validator("registration_id", mode="before")
    @classmethod
    def _check_uuid(cls, v: Any) -> str:
        v = str(v).strip().lower()
        if not UUID_V4.match(v):
            raise ValueError(f"registration_id is not a UUID v4: {v!r}")
        return v

    @field_validator("asset_path", mode="before")
    @classmethod
    def _check_scheme(cls, v: Any) -> str:
        v = str(v).strip()
        if not v.startswith(("https://", "s3://")):
            raise ValueError(f"asset_path scheme not allowed: {v!r}")
        return v

    @field_validator("checksum", mode="before")
    @classmethod
    def _check_sha256(cls, v: Any) -> str:
        v = str(v).strip().lower()
        if not SHA256.match(v):
            raise ValueError("checksum is not a 64-char SHA-256 hex digest")
        return v

This contract aligns directly with upstream dynamic field mapping output, guaranteeing that all variable substitution and attendee-data hydration has resolved before routing begins. Because the model is frozen=True, a payload cannot be mutated in flight — a fallback dispatch constructs a new payload with the same registration_id, which is what makes duplicate suppression reliable.

Deterministic Dispatch & Fallback Architecture Link to this section#

Production routing requires explicit fallback chains rather than naive exception swallowing. The dispatcher routes on the target enum, applies tiered retries with exponential backoff and jitter, and — only when a primary transport is exhausted — escalates through a predefined fallback before surrendering the payload to a dead-letter queue for manual reconciliation. The engine never blocks the event loop and never discards a transport exception silently; each adapter returns a boolean status and emits structured telemetry keyed by correlation_id.

import logging

import requests
from tenacity import (
    retry,
    retry_if_exception_type,
    stop_after_attempt,
    wait_exponential_jitter,
)

logger = logging.getLogger("pdf_routing")


def route_to_print_queue(payload: RoutingPayload) -> bool:
    resp = requests.post(
        "https://print-api.internal/v1/jobs",
        json={
            "url": payload.asset_path,
            "priority": "high",
            "id": payload.registration_id,
            "checksum": payload.checksum,
        },
        timeout=10,
    )
    resp.raise_for_status()
    return True


def route_to_email_relay(payload: RoutingPayload) -> bool:
    resp = requests.post(
        "https://email-relay.internal/v1/send",
        json={
            "recipient": payload.metadata.get("email"),
            "attachment": payload.asset_path,
            "id": payload.registration_id,
        },
        timeout=15,
    )
    resp.raise_for_status()
    return True


def route_to_s3_archive(payload: RoutingPayload) -> bool:
    # Production: boto3 put_object with ServerSideEncryption + ContentMD5 guard.
    logger.info(
        "archive_write | rid=%s cid=%s", payload.registration_id, payload.correlation_id
    )
    return True


TRANSPORT_MAP = {
    RoutingTarget.PRINT_QUEUE: route_to_print_queue,
    RoutingTarget.EMAIL_RELAY: route_to_email_relay,
    RoutingTarget.S3_ARCHIVE: route_to_s3_archive,
}

# print_queue and email_relay both degrade to durable archival; archival has no
# fallback because it is already the terminal durable sink.
FALLBACK_CHAIN = {
    RoutingTarget.PRINT_QUEUE: RoutingTarget.S3_ARCHIVE,
    RoutingTarget.EMAIL_RELAY: RoutingTarget.S3_ARCHIVE,
}


@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential_jitter(initial=2, max=10),
    retry=retry_if_exception_type((requests.RequestException, ConnectionError)),
    reraise=True,
)
def execute_transport(payload: RoutingPayload) -> bool:
    """Primary transport with tenacity-backed retries (3 attempts, jittered backoff)."""
    adapter = TRANSPORT_MAP.get(payload.target)
    if adapter is None:
        raise ValueError(f"Unsupported routing target: {payload.target}")
    return adapter(payload)


def push_to_dlq(payload: RoutingPayload, reason: str) -> None:
    logger.critical(
        "dlq_route | rid=%s cid=%s target=%s reason=%s",
        payload.registration_id,
        payload.correlation_id,
        payload.target.value,
        reason,
    )
    # Production: LPUSH to Redis list `dlq:routing` or SQS DLQ topic, payload as JSON.


def already_delivered(registration_id: str) -> bool:
    # Production: SETNX on `ledger:delivered:{registration_id}` in Redis.
    return False


def dispatch_with_fallback(payload: RoutingPayload) -> bool:
    """Deterministic dispatcher: idempotency gate → primary → fallback → DLQ."""
    if already_delivered(payload.registration_id):
        logger.info(
            "duplicate_suppressed | rid=%s cid=%s",
            payload.registration_id,
            payload.correlation_id,
        )
        return True

    try:
        execute_transport(payload)
        logger.info(
            "routed | rid=%s cid=%s target=%s",
            payload.registration_id,
            payload.correlation_id,
            payload.target.value,
        )
        return True
    except Exception as exc:  # retries exhausted for the primary transport
        logger.error(
            "primary_exhausted | rid=%s cid=%s target=%s error=%s",
            payload.registration_id,
            payload.correlation_id,
            payload.target.value,
            exc,
        )

    fallback_target = FALLBACK_CHAIN.get(payload.target)
    if fallback_target is not None:
        fallback_payload = payload.model_copy(update={"target": fallback_target})
        logger.warning(
            "fallback_escalate | rid=%s cid=%s from=%s to=%s",
            payload.registration_id,
            payload.correlation_id,
            payload.target.value,
            fallback_target.value,
        )
        try:
            execute_transport(fallback_payload)
            logger.info(
                "fallback_ok | rid=%s cid=%s target=%s",
                payload.registration_id,
                payload.correlation_id,
                fallback_target.value,
            )
            return True
        except Exception as exc:
            logger.error(
                "fallback_exhausted | rid=%s cid=%s error=%s",
                payload.registration_id,
                payload.correlation_id,
                exc,
            )

    push_to_dlq(payload, reason="PRIMARY_AND_FALLBACK_EXHAUSTED")
    return False

The idempotency gate runs first, keying off registration_id against the delivered ledger, so a replayed job after a worker crash is a no-op rather than a duplicate print. The fallback dispatch uses model_copy(update=...) to preserve every field — most importantly the idempotency key and correlation ID — while swapping only the transport target.

Production Debugging & Observability Link to this section#

Every log line above is a single structured event with a stable prefix, so log aggregation can pivot on the event name and thread by identity. When shipping to Datadog or ELK, promote these fields: rid (registration_id, the idempotency key), cid (correlation_id, the trace key), target, and reason. A typical incident trace reconstructed from one badge looks like this:

WARNING  fallback_escalate | rid=7f3d... cid=req-8832 from=email_relay to=s3_archive
ERROR    primary_exhausted | rid=7f3d... cid=req-8832 target=email_relay error=HTTPSConnectionPool timeout
INFO     fallback_ok       | rid=7f3d... cid=req-8832 target=s3_archive

Recommended aggregation patterns:

• Datadog: facet on @target and @reason; alert on event:dlq_route count over a rolling window. A saved query of service:pdf_routing event:primary_exhausted surfaces transport-health regressions before they reach the DLQ.
• ELK: index the event prefix as a keyword field (routing.event) and rid/cid as keyword fields for exact-match trace reconstruction. A single cid query returns the full life of a badge across ingestion, mapping, and routing.
• DLQ routing: every dlq_route event carries the full serialized payload. Rehydration tooling extracts registration_id, re-fetches the asset, recomputes the SHA-256 checksum, and re-injects via dispatch_with_fallback — the idempotency gate makes replay safe even if the original delivery secretly succeeded.

Correlation IDs are the backbone of triage: because cid is propagated from the schema validation pipelines that first admitted the record, a routing failure can be traced all the way back to the source export without guesswork.

Performance & Memory Constraints Link to this section#

The routing engine is I/O-bound, not CPU-bound, so its ceilings are connection pools and queue depth rather than compute. Size the following before an event, not during it.

Incident Triage Checklist Link to this section#

Target a mean-time-to-resolution under 15 minutes. Work these in order during a badge-delivery incident.

1. Confirm the symptom class. Pull recent criticals: kubectl logs deploy/pdf-routing --since=15m | grep dlq_route. A rising dlq_route count means deliveries are exhausting both primary and fallback; a rising primary_exhausted without dlq_route means the fallback is absorbing the load (degraded, not down).
2. Inspect DLQ depth. redis-cli LLEN dlq:routing. If it is climbing, sample one entry — redis-cli LINDEX dlq:routing 0 — and read its reason and target to localize the failing transport.
3. Check the failing transport directly. For email_relay or print_queue, curl -sS -o /dev/null -w '%{http_code} %{time_total}\n' https://print-api.internal/healthz. A 5xx or a multi-second time_total points at downstream saturation or a rate limit, not at the router.
4. Validate asset freshness. For primary_exhausted on a presigned URL, confirm the TTL with a HEAD against asset_path; a 403 means the link expired and the fix is regeneration, not retry.
5. Verify idempotency isn’t misfiring. If operators report missing badges while logs show duplicate_suppressed, inspect the ledger: redis-cli EXISTS ledger:delivered:<registration_id>. A stale ledger key from a failed prior run blocks legitimate redelivery — delete the specific key and replay.
6. Replay from the DLQ. Once the downstream is healthy, drain: pop each entry, recompute the checksum, and call dispatch_with_fallback. The idempotency gate guarantees no double-delivery.
7. Rollback strategy. If a bad deploy widened the failure, kubectl rollout undo deploy/pdf-routing and confirm primary_exhausted rates fall before draining the DLQ — never replay into a still-broken transport.

Related Link to this section#

• Badge Generation & Template Sync — the parent pipeline that assembles the PDF/A assets this stage delivers, and where routing sits as the final software-to-hardware handoff.
• Automating PDF Badge Delivery via AWS SES — the email-relay transport in depth: SES configuration, attachment limits, and bounce handling for the email_relay target.
• QR Code Generation — the encoding stage upstream; cross-reference its checksums when a routed asset fails integrity verification at the boundary.
• Dynamic Field Mapping — the stateless resolver whose versioned output becomes this stage’s immutable input contract.
• Async Batch Processing — the dead-letter queue and worker fan-out mechanism this stage escalates to when a delivery exhausts every transport.