All core GCP Pub/Sub settings can be mapped to four stages of a message’s lifecycle: how long it lives, how many to pull at once, how long to wait for a response, and what to do on failure.

Explanation

The parameters scattered across the docs look like a lot, but each one guards a specific stage in a message’s journey from publish to final processing (or abandonment). Organizing them by lifecycle stage – instead of memorizing each one individually – lets you instantly locate what any new parameter controls. This mental model also ties together the responsibility boundaries of the three resource layers: Topic, Subscription, and Dead Letter.

Knowledge Sugar

1. Parameters by lifecycle stage

The relationship between the three flow control params: MaxOutstandingMessages is the number of seats in the waiting room, NumGoroutines is the number of exam rooms, and MaxOutstandingBytes is the physical space limit of the waiting room.

2. Topic vs Subscription: two-layer responsibilities

A Topic is a named channel responsible for fan-out: one published message gives each subscription its own complete copy. A Subscription handles load balancing: with N pods underneath, they act as competing consumers where the same message goes to only one pod. Each layer has its own job – the Topic decides “which systems receive this event,” and the Subscription decides “how this system distributes workload internally.”

3. Subscription-to-Topic binding is infrastructure

A Subscription binds to a Topic at creation time. After that, application code only uses client.Subscription("name") to retrieve the existing resource without referencing the topic. This is the same pattern as K8s Pod volumeMounts pointing to a PVC name – the infrastructure layer binds things up front, and the application layer just uses the name. In practice, this step is usually handled by Terraform, not written in application code.

4. Dead Letter Policy isolates poison messages

When a message gets nacked or times out repeatedly up to MaxDeliveryAttempts, the broker stops sending it to the original subscription and forwards it to a dead letter topic. A DLQ topic is just a regular topic that semantically collects “messages that could not be processed.” The core value is separating “cannot process” from “not yet processed,” preventing a single poison message from occupying MaxOutstandingMessages quota and dragging down overall throughput. This is even more critical with EnableMessageOrdering on – messages behind the same ordering key would all be blocked.

GCP Pub/Sub retry is ACK-based – no ACK means redeliver. With a Pusher in the middle, retry becomes two layers: the Pusher’s own retry (fast, finely configurable) + Pub/Sub redelivery (slow, last resort).

Explanation

Native Pub/Sub retry mechanism

After a subscriber pulls a message, it must return an ACK within the ACK deadline. Otherwise Pub/Sub treats it as a failure and automatically redelivers:

Pub/Sub
  ↓ deliver message
Subscriber
  ↓ success → ACK → message removed from subscription
  ↓ failure / timeout → no ACK → Pub/Sub redelivers

Redelivery continues until the message is ACKed or exceeds the retention period (default 7 days).

With a Pusher: two-layer retry

The Pusher sits in between, creating two independent retry layers:

Pub/Sub Subscription
    ↓ pull
  Pusher
    ↓ push → Target Service

Layer 1 – Pusher’s own retry (CRD config)

When the Pusher’s push to the target service fails, it retries internally first (count and backoff are configurable via CRD) without going back to Pub/Sub.

Layer 2 – Pub/Sub redelivery

• If the target service processes successfully -> Pusher ACKs to Pub/Sub -> message done
• If all of Pusher’s retries are exhausted and it still fails -> NACK to Pub/Sub -> Pub/Sub redelivers to the Pusher

So Pub/Sub retry is the last resort. Day-to-day transient failures are absorbed by the Pusher layer.

Key: when does the Pusher ACK to Pub/Sub?

The ACK timing determines whether the entire retry chain works correctly:

Knowledge Sugar

Dead Letter Topic

When a message fails repeatedly beyond the maximum retry count, instead of letting it loop forever, move it to a dedicated topic for isolation:

Normal:  Pub/Sub → Pusher → Target Service ✓ → ACK
Failure: Pub/Sub → Pusher → Target Service ✗ → NACK → retry N times
                                                        ↓ exceeds limit
                                               Dead Letter Topic

Three uses for a Dead Letter Topic:

• Unblock normal traffic: problematic messages are moved away, the rest keep flowing
• Post-mortem investigation: see which messages keep failing and why
• Manual replay: after fixing the bug, replay dead letter messages back into the normal flow

Benefits of two-layer retry

Pure Pub/Sub retry uses exponential backoff, which is slow. The Pusher layer can use faster, finer-grained retry strategies. Most transient failures get resolved at this layer without going through the full Pub/Sub redelivery cycle.

For the Pusher architecture background, see the gRPC Pusher Pattern post. For Pub/Sub Topic & Subscription basics, see the GCP Pub Sub Topic & Subscription post.

Topic is where messages are published to; subscription is how a consumer receives messages from that topic.

A topic is the publishing channel for messages. A subscription is how a consumer subscribes to that channel. Separating the two lets multiple subscribers independently consume from the same topic.

Explanation

Publisher → Topic → Subscription A → Service A
                 → Subscription B → Service B

• Topic: the target for publishing. The publisher just sends messages to the topic without caring who reads them.
• Subscription: a consumption channel attached to a topic. Each subscription receives an independent copy of every message on the topic.

Each subscription tracks its own progress – where Service A has read up to and where Service B has read up to are independent. Messages stay in a subscription until they are acknowledged or exceed the retention period.

Two subscription types

• Pull: the subscriber actively asks “any new messages?” – suited for backend services that want to control their own consumption rate
• Push: Pub/Sub proactively sends messages to a specified HTTP endpoint – suited for serverless or webhook scenarios

Knowledge Sugar

Why separate topic and subscription?

If there were only topics, all consumers would compete for the same messages – once read, they are gone. With subscriptions, the same message can be consumed independently by a logging service, an analytics service, and a notification service without interference. This is the fan-out pattern.

What happens if nobody consumes from a subscription?

Messages pile up until the retention period expires (default 7 days), then get auto-deleted. So if you create a subscription, make sure something is consuming from it – otherwise it is wasted resources.

Topic ownership should follow the domain

The topic should be created by the service that owns the domain. For example, if monolith ServiceA publishes events to a topic belonging to ServiceB’s domain, the topic should live in ServiceB’s GCP project, with IAM granting ServiceA publish permission:

ServiceA (monolith)
  │  roles/pubsub.publisher (IAM grant)
  ▼
Topic (owned by ServiceB)
  ▼
Subscription → ServiceC consume

After migration completes, you only need to remove ServiceA’s publish logic – everything else stays the same.

Subscription ownership design

The subscription owner does not have to match the topic owner. Two options:

During migration, Option 1 is recommended – ServiceB owns both the topic and subscription, then grants ServiceC consume permission. This gives ServiceB full visibility over the message flow.

ServiceC connects to the subscription via IAM:

1resource "google_pubsub_subscription_iam_member" "subscriber" {
2  subscription = google_pubsub_subscription.my_subscription.name
3  role         = "roles/pubsub.subscriber"
4  member       = "serviceAccount:service-c@service-c-project.iam.gserviceaccount.com"
5}

ServiceC’s code just needs to pull using its own service account. GCP verifies IAM and grants access – no extra configuration needed.

Creating Topic and Subscription with Terraform

Like buckets, these are GCP resources that can be managed directly with Terraform:

 1resource "google_pubsub_topic" "my_topic" {
 2  name    = "my-topic"
 3  project = "service-b-project"
 4}
 5
 6resource "google_pubsub_subscription" "my_subscription" {
 7  name  = "my-subscription"
 8  topic = google_pubsub_topic.my_topic.id
 9}
10
11# Grant ServiceA's service account publish permission
12resource "google_pubsub_topic_iam_member" "publisher" {
13  topic  = google_pubsub_topic.my_topic.id
14  role   = "roles/pubsub.publisher"
15  member = "serviceAccount:service-a@service-a-project.iam.gserviceaccount.com"
16}

An application deployed on GKE can use OIDC federation to exchange a Kubernetes-issued JWT token for temporary AWS credentials, eliminating the need to hardcode AWS Access Keys.

Explanation

What OIDC does in this scenario

OIDC (OpenID Connect) is essentially a protocol for “let me prove this identity is real.” In this use case, GKE acts as the OIDC Identity Provider (IdP), and AWS is the party that trusts what GKE says.

The overall flow looks like this:

GKE Pod
  │
  │ 1. Kubernetes automatically mounts a Service Account JWT token on the Pod
  │    (this token contains the workload's identity info, signed by GKE)
  │
  ▼
AWS STS (AssumeRoleWithWebIdentity)
  │
  │ 2. Submit the JWT token to AWS STS
  │    AWS validates the token against GKE's OIDC endpoint
  │
  ▼
Temporary AWS credentials (Access Key + Secret + Session Token)
  │
  │ 3. Use temporary credentials to operate on S3
  │
  ▼
S3 Bucket ✓

sequenceDiagram
    participant Pod as GKE Pod
    participant STS as AWS STS
    participant OIDC as GKE OIDC
    participant S3

    Note over Pod: K8s mounts JWT token
    Pod->>STS: AssumeRoleWithWebIdentity(JWT)
    STS->>OIDC: Verify signature
    OIDC-->>STS: OK
    STS-->>Pod: Temporary credentials (with TTL)
    Pod->>S3: Upload CSV
    S3-->>Pod: 200 OK

What needs to be configured in Terraform

1. AWS IAM OIDC Provider – tells AWS “I trust tokens issued by this GKE cluster”; requires the GKE OIDC issuer URL and thumbprint
2. AWS IAM Role + Trust Policy – defines which GKE service account can assume this role; the condition typically requires the sub claim to match system:serviceaccount:<namespace>:<ksa-name>
3. S3 Permission – attach the S3 read/write policy to this IAM Role

Go STS package upgrade

In the Go application, use AssumeRoleWithWebIdentity, passing in the JWT token path read from the Pod (typically /var/run/secrets/kubernetes.io/serviceaccount/token). After exchanging for temporary credentials, initialize the S3 client. The package upgrade was mainly to ensure compatibility with SDK v2’s credential provider interface.

Knowledge Sugar

Why not just use AWS Access Keys? Hardcoding keys has rotation issues, leakage risks, and makes it hard to audit which workload is accessing what. OIDC federation’s temporary credentials have a TTL and expire automatically, making them significantly more secure.

What’s inside the JWT token? The token issued by GKE is a standard JWT. The payload contains:

• iss: OIDC issuer (the GKE cluster’s URL)
• sub: system:serviceaccount:<namespace>:<name>
• exp: expiration time

When AWS STS receives the token, it fetches the public key from the iss URL’s /.well-known/openid-configuration endpoint to verify the signature.

How is the JWT token mounted into the Pod? This is a built-in Kubernetes mechanism – every Pod automatically gets a Service Account token mounted at creation, no extra config needed. However, the default token doesn’t specify an audience, which AWS STS won’t accept. So you need to mount a dedicated one using a projected volume in the Pod spec:

1volumes:
2  - name: aws-token
3    projected:
4      sources:
5        - serviceAccountToken:
6            audience: sts.amazonaws.com # here!
7            expirationSeconds: 3600
8            path: token

Terraform only handles the AWS side (OIDC Provider, IAM Role); the token mounting itself is done by Kubernetes.

The official name for this pattern: Workload Identity Federation GCP’s own Workload Identity uses the same principle, just with GCP resources. For cross-cloud scenarios (GKE -> AWS), OIDC serves as the universal standard that bridges them.