Submission Gateway Transport & Acknowledgement Handling

A signed, technically valid eCTD sequence is still just a directory of bytes until it crosses the gateway boundary. That crossing is where automation earns its keep or quietly fails: the transport protocol demands a specific envelope, the health authority’s acceptance is not returned in the same HTTP response that delivered the payload, and a retry sent for the wrong reason can turn one intended filing into two. This area owns that last, narrow, high-stakes stretch of the submission pipeline — packaging a sequence for delivery over the FDA Electronic Submissions Gateway, reading back what the gateway says happened, and tying every receipt to the exact transmission it answers.

The sequence arriving here has already passed through eCTD Sequence Assembly & Lifecycle Management and carries a Part 11 signature bound to its content hash. Nothing about transport should be allowed to second-guess that work — the transport layer’s job is to move bytes reliably and to make the gateway’s response legible, not to re-validate content. What it must own instead is protocol correctness, delivery idempotency, and a complete accounting of every acknowledgement, written to the same append-only audit log that already records assembly and signature.

Why transport is its own compliance surface #

Three properties make gateway transport distinct from an ordinary file upload, and each has a regulatory consequence if it is treated casually:

• The transport protocol is prescribed, not chosen. The FDA Electronic Submissions Gateway (ESG) requires AS2 (Applicability Statement 2): a signed and encrypted S/MIME message delivered over HTTPS. A submission that arrives over plain HTTP, or unsigned, is not a compliance judgment call — it is rejected by the gateway before any human sees it.
• Acceptance is asynchronous and multi-stage. The gateway returns a synchronous Message Disposition Notification (MDN) that confirms only that the envelope was received intact. Whether the Center actually accepted the filing arrives later, as a sequence of acknowledgement receipts. Code that treats a 200-status HTTP response as “submitted” is reporting a fact one layer short of the one that matters.
• A retry is indistinguishable from a new filing unless the code makes it distinguishable. Networks fail mid-transfer. Without a stable, idempotent message identifier carried through every retry, an automated retry after a timeout can register as a second, competing submission for the same sequence.

Decision flow #

The transport stage takes a signed sequence as input, builds and delivers the AS2 envelope, and then owns three further receipts before the transmission can be called complete. Every one of the five stages, plus both terminal outcomes, writes to the audit log.

The reconciliation stage is drawn as one box but is really three waits: ACK1 confirms the gateway itself accepted the envelope, ACK2 confirms the receiving Center picked it up, and ACK3 carries the outcome of the Center’s technical validation of the sequence content. A transmission is not resolved — accepted or rejected — until all three have arrived and been matched.

Library and tooling landscape #

Python has no single batteries-included AS2 client that clinical teams should treat as a drop-in dependency; the honest approach composes a small number of mature libraries around a thin, well-tested transport layer built in-house.

Whichever HTTP client is chosen, treat certificate handling as security-critical rather than incidental: the AS2 signing certificate and its private key are read from a path supplied through the environment, never embedded in source, and access to that path is itself an event worth recording in the audit log.

Step-by-step implementation #

The implementation below builds a transmission record, constructs and signs the AS2 payload, delivers it with an idempotent message id, captures the synchronous MDN, and then processes the three asynchronous acknowledgements against that same transmission.

1. Model the transmission #

A transmission is the unit that everything downstream reconciles against. Making its identifier a first-class, generated value — rather than an incidental HTTP artifact — is what makes a retry safe.

"""Typed model of one gateway transmission attempt."""
from __future__ import annotations

import os
import secrets
from dataclasses import dataclass, field
from datetime import datetime, timezone
from enum import Enum


class Gateway(str, Enum):
    FDA_ESG = "fda_esg"


class TransmissionState(str, Enum):
    BUILT = "built"
    SENT = "sent"
    MDN_RECEIVED = "mdn_received"
    ACK1_RECEIVED = "ack1_received"
    ACK2_RECEIVED = "ack2_received"
    ACCEPTED = "accepted"
    REJECTED = "rejected"


def new_message_id(sequence_id: str) -> str:
    """A stable, unguessable identifier reused across every retry of one delivery."""
    return f"{sequence_id}.{secrets.token_hex(8)}@submission.local"


@dataclass(frozen=True)
class Transmission:
    """One delivery attempt over the AS2 gateway.

    message_id is generated once and carried on every retry, so a network
    failure and a resend of the same bytes are never mistaken for two filings.
    """

    transmission_id: str
    sequence_id: str
    message_id: str
    gateway: Gateway
    endpoint: str
    state: TransmissionState = TransmissionState.BUILT
    sent_at: datetime | None = None

    @classmethod
    def for_sequence(cls, transmission_id: str, sequence_id: str) -> "Transmission":
        endpoint = os.environ.get("FDA_ESG_ENDPOINT")
        if not endpoint:
            raise RuntimeError("Missing required environment variable: FDA_ESG_ENDPOINT")
        return cls(
            transmission_id=transmission_id,
            sequence_id=sequence_id,
            message_id=new_message_id(sequence_id),
            gateway=Gateway.FDA_ESG,
            endpoint=endpoint,
        )

2. Build the signed, encrypted AS2 envelope #

The AS2 message is a MIME payload that is signed and then encrypted so that the gateway can verify origin and integrity before decrypting the content. The signing and encryption operations themselves should stay thin wrappers over cryptography’s CMS/PKCS7 primitives rather than a hand-rolled cryptographic routine — this is a place to lean on a reviewed library, not to demonstrate cleverness.

"""Build the signed, encrypted AS2 payload for one sequence."""
from __future__ import annotations

import os
from dataclasses import dataclass
from pathlib import Path

from cryptography.hazmat.primitives.serialization import pkcs12


@dataclass(frozen=True)
class As2Credentials:
    """Certificate and key material for signing and encrypting AS2 payloads.

    Loaded once from environment-configured paths; never embedded in code
    and never logged.
    """

    signing_cert_path: str
    signing_key_path: str
    partner_cert_path: str  # the gateway's public certificate, for encryption

    @classmethod
    def from_env(cls) -> "As2Credentials":
        def required(name: str) -> str:
            value = os.environ.get(name)
            if not value:
                raise RuntimeError(f"Missing required environment variable: {name}")
            return value

        return cls(
            signing_cert_path=required("AS2_SIGNING_CERT_PATH"),
            signing_key_path=required("AS2_SIGNING_KEY_PATH"),
            partner_cert_path=required("AS2_PARTNER_CERT_PATH"),
        )


def build_as2_payload(sequence_bytes: bytes, message_id: str, credentials: As2Credentials) -> bytes:
    """Return a signed-then-encrypted S/MIME message ready for HTTPS delivery.

    The actual CMS sign/encrypt calls are delegated to `cryptography`'s
    audited primitives; this function is responsible only for sequencing
    them correctly and attaching the message id as a MIME header.
    """
    signed = _sign_cms(sequence_bytes, credentials.signing_cert_path, credentials.signing_key_path)
    encrypted = _encrypt_cms(signed, credentials.partner_cert_path)
    return _wrap_mime(encrypted, message_id)


def _sign_cms(payload: bytes, cert_path: str, key_path: str) -> bytes:
    """Sign payload with the sponsor's AS2 certificate using CMS/PKCS7."""
    # Delegates to cryptography.hazmat.primitives.serialization and a CMS
    # signing helper; kept out of scope here to avoid duplicating a
    # security-reviewed routine that should live in one audited module.
    raise NotImplementedError("wire to the organization's reviewed CMS signing module")


def _encrypt_cms(signed_payload: bytes, partner_cert_path: str) -> bytes:
    """Encrypt the signed payload to the gateway's public certificate."""
    raise NotImplementedError("wire to the organization's reviewed CMS encryption module")


def _wrap_mime(encrypted_payload: bytes, message_id: str) -> bytes:
    """Attach AS2 MIME headers, including the idempotent Message-Id."""
    header = (
        f"Message-Id: <{message_id}>\r\n"
        "Content-Type: application/pkcs7-mime; smime-type=enveloped-data\r\n"
        "\r\n"
    ).encode("ascii")
    return header + encrypted_payload

Keeping _sign_cms and _encrypt_cms as explicit seams — rather than inlining cryptography calls throughout the transport code — means a security review only has to examine one small module when the signing certificate rotates or the CMS parameters change.

3. Deliver and capture the synchronous MDN #

The POST carries the message id from step 1 unchanged on every attempt. The synchronous response is the MDN: a signed statement that the gateway received an intact, correctly signed envelope. It is not an acceptance decision.

"""Deliver the AS2 payload and capture the synchronous MDN."""
from __future__ import annotations

import logging

import requests

logger = logging.getLogger("gateway.transport")


class TransportError(Exception):
    """Raised for a transport failure that should not be silently retried forever."""


def send_transmission(transmission: "Transmission", payload: bytes) -> "MdnReceipt":
    headers = {
        "Content-Type": "application/pkcs7-mime; smime-type=enveloped-data",
        "Message-Id": f"<{transmission.message_id}>",
        "AS2-To": "FDA-ESG",
    }
    try:
        response = requests.post(transmission.endpoint, data=payload, headers=headers, timeout=60)
    except requests.exceptions.Timeout as exc:
        raise TransportError(f"gateway timed out for {transmission.message_id}") from exc
    except requests.exceptions.ConnectionError as exc:
        raise TransportError(f"connection failed for {transmission.message_id}") from exc

    if response.status_code >= 500:
        raise TransportError(f"gateway returned {response.status_code} for {transmission.message_id}")
    if response.status_code >= 400:
        # A 4xx here is a rejected envelope, not a network problem; do not retry blindly.
        raise ValueError(f"gateway rejected envelope {transmission.message_id}: {response.status_code}")

    return parse_mdn(response.content, transmission.message_id)

Parsing the MDN itself, and everything downstream of it — ACK1, ACK2, and ACK3 — is where reconciliation happens; that logic is broken out in Submitting to the FDA ESG AS2 Gateway with Python and Parsing Regulatory Gateway Acknowledgement Receipts, which give the full, testable implementation of each half.

4. Reconcile every acknowledgement to its transmission #

The reconciliation rule is simple to state and easy to get wrong in practice: an acknowledgement without a matching transmission is an anomaly, not a message to discard.

"""Reconcile an inbound acknowledgement against known transmissions."""
from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class Acknowledgement:
    message_id: str
    ack_level: str   # "ack1" | "ack2" | "ack3"
    status: str      # "accepted" | "rejected"
    detail: str | None = None


def reconcile(ack: Acknowledgement, known_transmissions: dict[str, "Transmission"]) -> "Transmission":
    transmission = known_transmissions.get(ack.message_id)
    if transmission is None:
        raise LookupError(
            f"acknowledgement {ack.ack_level} references unknown message id {ack.message_id!r}"
        )
    return transmission

An unmatched acknowledgement should raise loudly and page a human rather than being logged and dropped — in a compliant pipeline it usually means either a message id was generated inconsistently between a retry and the original attempt, or the gateway is returning a receipt for a transmission this system does not believe it sent.

Validation and audit-trail integration #

Every state transition in the diagram above — envelope built, delivered, MDN received, each acknowledgement reconciled — writes one immutable event to the append-only audit log. That log is what turns “did the filing go through” from a question that requires calling the help desk into a query against a table.

"""Emit an audit event for a transmission state change."""
from __future__ import annotations

import json
import logging

logger = logging.getLogger("gateway.audit")


def record_transmission_event(transmission: "Transmission", event: str, detail: dict | None = None) -> None:
    payload = {
        "action": event,
        "transmission_id": transmission.transmission_id,
        "sequence_id": transmission.sequence_id,
        "message_id": transmission.message_id,
        "gateway": transmission.gateway.value,
        "detail": detail or {},
    }
    logger.info(json.dumps(payload, sort_keys=True))

Because the sequence being transmitted already carries the SHA-256 provenance chain from eCTD Sequence Assembly & Lifecycle Management, a completed reconciliation event ties an accepted filing all the way back through assembly and signature to the original ingested artifacts. That end-to-end traceability is exactly what an inspector reconstructing a submission’s history is looking for, and what Inspection Readiness & Audit-Trail Reporting turns into a formal export.

Error categorization and recovery #

Transport failures split cleanly into two classes, and conflating them is the most common transport bug: a permanent failure retried blindly wastes the filing clock, while a transient failure treated as permanent abandons a submission that would have succeeded on the next attempt.

• Permanent — envelope or content rejected. A 4xx response to the POST, a negative MDN, or an ACK3 carrying a high-severity technical rejection code all mean the gateway examined the payload and refused it. Retrying with the same bytes reproduces the same rejection. The correct response is to surface the specific reason, route the sequence back to assembly or signing for correction, and record the rejection as a terminal state for this transmission id.
• Permanent — malformed idempotency. An acknowledgement that cannot be reconciled to any known transmission (the case raised by reconcile above) is a defect in message id generation or storage, not something a retry fixes. Treat it as a page-a-human condition.
• Transient — network and gateway availability. Connection timeouts, DNS failures, and 5xx responses reflect the state of the network or the gateway, not the submission. These are retried with bounded exponential backoff, reusing the same message id so a successful retry cannot register as a second filing. The pattern is identical to the one covered in Fallback Routing for Portal Outages, and a durable queue with dead-letter handling is the production-grade way to implement the retry loop itself rather than looping in-process.
• Transient — acknowledgement delay. ACK1 through ACK3 can legitimately take hours to arrive; a transmission sitting in an intermediate state is expected behavior, not a failure, until it exceeds the service-level window the sponsor has agreed to monitor.

Compliance checklist #

Before a transmission is considered complete and the sequence is marked filed, confirm each of these holds:

• The AS2 envelope is both signed and encrypted before it leaves the building
• Every delivery attempt for one sequence carries the same message identifier
• The synchronous MDN is captured and recorded before the transmission is considered sent
• Every ACK1, ACK2, and ACK3 receipt reconciles to exactly one known transmission
• An unmatched acknowledgement raises an alert rather than being silently discarded
• A transient transport failure is retried with bounded backoff and the same message identifier
• A rejected transmission records the specific reason and routes the sequence back for correction
• Every state transition is written to the append-only audit log before the next stage begins

FAQ #

What is the difference between the MDN and the ACK receipts? #

The MDN is the synchronous, protocol-level receipt required by AS2: a signed statement that the gateway received an intact, correctly signed and encrypted envelope. ACK1, ACK2, and ACK3 arrive asynchronously afterward and carry the substantive outcome — that the gateway queued the message, that the receiving Center picked it up, and that the Center’s technical validation accepted or rejected the content.

Why does a retry need to reuse the same message identifier? #

If a retried delivery generated a fresh identifier, a successful retry after a timed-out first attempt would look, from the gateway’s perspective, like two independent submissions of the same sequence. Reusing the identifier lets the gateway and the sponsor’s own reconciliation logic both recognize the retry as the same transmission, so a network hiccup can never become a duplicate filing.

What should happen when an acknowledgement cannot be matched to a transmission? #

It should never be silently dropped. An unmatched acknowledgement usually indicates either an inconsistency in how message identifiers were generated or stored, or a receipt intended for a different system. Treating it as an anomaly that raises an alert is what keeps the acknowledgement ledger trustworthy enough to be inspection evidence.

Related #

• Up one level: Electronic Submission Gateway & E-Signature Automation
• Submitting to the FDA ESG AS2 Gateway with Python
• Parsing Regulatory Gateway Acknowledgement Receipts
• eCTD Sequence Assembly & Lifecycle Management
• Core Architecture & Regulatory Mapping for Clinical Trials