When converting a partner’s HTTP API to proto, standard HTTP headers (Content-Length, etc.) should not go into the message – the gRPC framework handles them automatically. Custom headers (like amzn-request-id) go either in a response message field or in gRPC trailing metadata, depending on their purpose.

Explanation

Background: converting an Excel API spec to proto

The partner’s API doc (Japanese-style Excel format) lists HTTP response headers and custom headers together. When converting to proto, you need to decide which ones belong in the message and which do not.

Standard HTTP Headers – do not put in proto

Transport-layer headers like Content-Length, Content-Type, and Transfer-Encoding are handled by the gRPC framework:

• Content-Type is always application/grpc
• Content-Length is determined by serialized message size, filled in automatically by the framework
• Putting them in proto only causes semantic confusion (is this a business field or an HTTP header?)

Custom Headers (e.g. amzn-request-id) – two approaches

Go server-side trailing metadata example

 1import "google.golang.org/grpc/metadata"
 2
 3func (s *Server) GetOrder(ctx context.Context, req *pb.GetOrderRequest) (*pb.GetOrderResponse, error) {
 4    // ... call partner API ...
 5
 6    // Put amzn-request-id in trailing metadata
 7    trailer := metadata.Pairs("amzn-request-id", partnerResp.Header.Get("amzn-request-id"))
 8    grpc.SetTrailer(ctx, trailer)
 9
10    return &pb.GetOrderResponse{Order: order}, nil
11}

Add proto comments to preserve traceability

Even if you do not put all headers into the message, you can document the mapping in proto comments for future reference:

1message GetOrderResponse {
2  // Maps to amzn-request-id response header in partner API.
3  // Passed as gRPC trailing metadata, not as a message field.
4  Order order = 1;
5}
6
7// Note: Standard HTTP headers (Content-Length, Content-Type) are intentionally
8// omitted — handled by gRPC framework automatically.

Knowledge Sugar

The “shallow mapping first” logic from a colleague

Map all fields first for traceability, then prune later. This makes sense from a traceability perspective, but once a proto is published there is a risk of breaking changes. A better compromise is to document the mapping in comments rather than putting every header into the message.

What is gRPC Metadata?

gRPC metadata is a set of key-value pairs, equivalent to HTTP headers in the gRPC world. It comes in two flavors:

• Header metadata: sent before the response starts (grpc.SendHeader)
• Trailing metadata: sent after the response ends (grpc.SetTrailer), more commonly used for carrying additional information

GCP Pub/Sub retry is ACK-based – no ACK means redeliver. With a Pusher in the middle, retry becomes two layers: the Pusher’s own retry (fast, finely configurable) + Pub/Sub redelivery (slow, last resort).

Explanation

Native Pub/Sub retry mechanism

After a subscriber pulls a message, it must return an ACK within the ACK deadline. Otherwise Pub/Sub treats it as a failure and automatically redelivers:

Pub/Sub
  ↓ deliver message
Subscriber
  ↓ success → ACK → message removed from subscription
  ↓ failure / timeout → no ACK → Pub/Sub redelivers

Redelivery continues until the message is ACKed or exceeds the retention period (default 7 days).

With a Pusher: two-layer retry

The Pusher sits in between, creating two independent retry layers:

Pub/Sub Subscription
    ↓ pull
  Pusher
    ↓ push → Target Service

Layer 1 – Pusher’s own retry (CRD config)

When the Pusher’s push to the target service fails, it retries internally first (count and backoff are configurable via CRD) without going back to Pub/Sub.

Layer 2 – Pub/Sub redelivery

• If the target service processes successfully -> Pusher ACKs to Pub/Sub -> message done
• If all of Pusher’s retries are exhausted and it still fails -> NACK to Pub/Sub -> Pub/Sub redelivers to the Pusher

So Pub/Sub retry is the last resort. Day-to-day transient failures are absorbed by the Pusher layer.

Key: when does the Pusher ACK to Pub/Sub?

The ACK timing determines whether the entire retry chain works correctly:

Knowledge Sugar

Dead Letter Topic

When a message fails repeatedly beyond the maximum retry count, instead of letting it loop forever, move it to a dedicated topic for isolation:

Normal:  Pub/Sub → Pusher → Target Service ✓ → ACK
Failure: Pub/Sub → Pusher → Target Service ✗ → NACK → retry N times
                                                        ↓ exceeds limit
                                               Dead Letter Topic

Three uses for a Dead Letter Topic:

• Unblock normal traffic: problematic messages are moved away, the rest keep flowing
• Post-mortem investigation: see which messages keep failing and why
• Manual replay: after fixing the bug, replay dead letter messages back into the normal flow

Benefits of two-layer retry

Pure Pub/Sub retry uses exponential backoff, which is slow. The Pusher layer can use faster, finer-grained retry strategies. Most transient failures get resolved at this layer without going through the full Pub/Sub redelivery cycle.

For the Pusher architecture background, see the gRPC Pusher Pattern post. For Pub/Sub Topic & Subscription basics, see the GCP Pub Sub Topic & Subscription post.

A gRPC Pusher is a message dispatch intermediary – it pulls messages from a Pub/Sub subscription and proactively pushes them to a target service’s gRPC endpoint. This solves the problem where workers cannot control which consumer processes which message.

Explanation

Why is this needed?

The testing pain point with worker-based services: a PRRC (PR Review Copy) environment and master share the same subscription. There is no way to guarantee that a test message will be consumed by the PRRC pod rather than a master worker – so every test requires manually deploying the commit image to master, which is cumbersome.

The gRPC Pusher solves this as an intermediary:

Pub/Sub Subscription
        ↓  pull
   gRPC Pusher (intermediary layer)
        ↓  push (can control which endpoint to route to)
  Target Service (specified gRPC endpoint)

The Pusher centrally pulls messages, then pushes them to a specified endpoint based on configuration – giving precise control over PRRC traffic without worrying about master workers stealing messages.

Configured via CRD

The Pusher’s behavior is managed through Kubernetes CRD config, which can set:

• Retry policy: retry strategy after failures (count, backoff)
• Traffic limit: rate limiting for pushes

Knowledge Sugar

Public implementations of this pattern

This “pull then push” message dispatch pattern is common in the industry:

• Knative Eventing: pulls from Broker/Channel, pushes to HTTP/gRPC sink endpoints, supports retry and dead letter sink, configured via CRD
• Dapr Pub/Sub: runs as a sidecar, pulls from various pub/sub backends, pushes to application endpoints via gRPC or HTTP
• GCP Push Subscription: GCP’s native push mode, sends messages as HTTP POST to a specified endpoint, supports exponential backoff retry

Core design concept: decoupling consume and process

Traditional workers couple “pulling messages from a subscription” and “processing messages” in the same process, making traffic routing hard to control. The Pusher separates the two:

• Pusher handles consume (single entry point)
• Target service only handles process (can be any endpoint)

This is also the core idea behind Dapr and Knative – extracting messaging infrastructure out of application logic.

gRPC is an RPC framework developed by Google. It uses HTTP/2 for transport and Protobuf for serialization, making it faster and more structured than traditional REST/HTTP+JSON – but less readable, so it is mainly used for internal microservice communication.

Explanation

Traditional HTTP (REST + JSON)

Client → POST /users HTTP/1.1
         Content-Type: application/json
         {"name": "Alice", "age": 30}

Server → 200 OK
         {"id": 1, "name": "Alice"}

• Operations are defined by URL + HTTP method
• Data format is JSON (human-readable text)
• Based on HTTP/1.1 (each request uses an independent connection)

How gRPC does it

Client → calls UserService.CreateUser(CreateUserRequest)
Server → returns CreateUserResponse

• Services and messages are defined in .proto files; calling them feels like calling a local function
• Data format is Protobuf (binary, not directly readable)
• Based on HTTP/2 (multiplexing – a single connection handles multiple requests)

Core differences

Knowledge Sugar

What is HTTP/2 multiplexing?

In HTTP/1.1, each request must wait for the previous response before sending the next one (or open a new connection). HTTP/2 can handle multiple request/response pairs in parallel over a single connection, significantly reducing latency.

Streaming

gRPC supports four communication modes, which are hard to do with REST:

Unary:              one request → one response (most common)
Server streaming:   one request → multiple responses (e.g. real-time push)
Client streaming:   multiple requests → one response (e.g. uploading chunked data)
Bidirectional:      multiple requests ↔ multiple responses (e.g. real-time chat)

Relation to Protobuf gRPC’s data format is Protobuf – see the earlier post on Protobuf Reserved Fields & API Versioning for more context.

When removing a response field from a protobuf message, you cannot simply delete it – you must mark it as reserved. Otherwise, if a future field reuses the same field number, old clients will misinterpret the new field’s value as the old field.

Explanation

Why is reserved needed?

Protobuf identifies data on the wire by field number, not field name. So when you:

1. Originally have string old_field = 3; in a response message
2. Integrate a new external service, the backend logic changes, this field is no longer populated, so you delete it
3. Later add int32 new_field = 3; (reusing number 3)

At this point, clients still using the old proto definition will receive the response and try to read new_field’s value as old_field – the type mismatch causes a crash.

Correct approach: mark it as reserved

1message MyResponse {
2  reserved 3, 5;                    // reserve these field numbers from reuse
3  reserved "old_field", "another";  // also reserve the field names (prevents accidental reuse in code)
4
5  string active_field = 1;
6  int32 other_field = 2;
7}

With reserved in place, any attempt to reuse those numbers or names will cause a protoc compilation error.

What is the relationship between v2alpha and v2beta?

These are API version stability labels, following Google AIP (API Improvement Proposals) naming conventions, commonly seen in gRPC / proto package naming:

The typical progression is: v2alpha1 -> v2alpha2 -> v2beta1 -> v2beta2 -> v2

How it looks in proto files:

1// Experimental version
2package mycompany.myservice.v2alpha1;
3
4// Feature-complete but still being polished
5package mycompany.myservice.v2beta1;
6
7// Stable release
8package mycompany.myservice.v2;

Different versioned packages are completely independent namespaces that can coexist, allowing old and new clients to each use their corresponding version.

Knowledge Sugar

Two ways to write reserved, and they can be separate or combined:

1reserved 1 to 3;       // contiguous range
2reserved 1, 2, 3;      // individual listing
3reserved "foo", "bar"; // reserve names (recommended to do both, prevents typo reuse)

Difference from deprecated:

• deprecated = true is a human-readable warning telling developers “this field is not recommended anymore,” but it still works
• reserved is enforced at compile time – it completely blocks any reuse

Practical mindset when designing new proto contracts:

• A field number, once released, is forever – before deleting a field, decide whether it needs reserved
• Alpha/Beta versions give you room to experiment; once you reach a stable v2, removing or changing fields requires a formal deprecation process

Further reading:

• Google AIP-180: definition of breaking vs non-breaking changes
• oneof field numbers in protobuf also need reserved
• Buf (buf.build): a protobuf linter that can automatically detect breaking changes