The Point

GCP Pub/Sub retry is ACK-based – no ACK means redeliver. With a Pusher in the middle, retry becomes two layers: the Pusher’s own retry (fast, finely configurable) + Pub/Sub redelivery (slow, last resort).

Explanation

Native Pub/Sub retry mechanism

After a subscriber pulls a message, it must return an ACK within the ACK deadline. Otherwise Pub/Sub treats it as a failure and automatically redelivers:

Pub/Sub
  ↓ deliver message
Subscriber
  ↓ success → ACK → message removed from subscription
  ↓ failure / timeout → no ACK → Pub/Sub redelivers

Redelivery continues until the message is ACKed or exceeds the retention period (default 7 days).

With a Pusher: two-layer retry

The Pusher sits in between, creating two independent retry layers:

Pub/Sub Subscription
    ↓ pull
  Pusher
    ↓ push → Target Service

Layer 1 – Pusher’s own retry (CRD config)

When the Pusher’s push to the target service fails, it retries internally first (count and backoff are configurable via CRD) without going back to Pub/Sub.

Layer 2 – Pub/Sub redelivery

• If the target service processes successfully -> Pusher ACKs to Pub/Sub -> message done
• If all of Pusher’s retries are exhausted and it still fails -> NACK to Pub/Sub -> Pub/Sub redelivers to the Pusher

So Pub/Sub retry is the last resort. Day-to-day transient failures are absorbed by the Pusher layer.

Key: when does the Pusher ACK to Pub/Sub?

The ACK timing determines whether the entire retry chain works correctly:

Knowledge Sugar

Dead Letter Topic

When a message fails repeatedly beyond the maximum retry count, instead of letting it loop forever, move it to a dedicated topic for isolation:

Normal:  Pub/Sub → Pusher → Target Service ✓ → ACK
Failure: Pub/Sub → Pusher → Target Service ✗ → NACK → retry N times
                                                        ↓ exceeds limit
                                               Dead Letter Topic

Three uses for a Dead Letter Topic:

• Unblock normal traffic: problematic messages are moved away, the rest keep flowing
• Post-mortem investigation: see which messages keep failing and why
• Manual replay: after fixing the bug, replay dead letter messages back into the normal flow

Benefits of two-layer retry

Pure Pub/Sub retry uses exponential backoff, which is slow. The Pusher layer can use faster, finer-grained retry strategies. Most transient failures get resolved at this layer without going through the full Pub/Sub redelivery cycle.

For the Pusher architecture background, see the gRPC Pusher Pattern post. For Pub/Sub Topic & Subscription basics, see the GCP Pub Sub Topic & Subscription post.