1stop := context.AfterFunc(ctx, func() {
2    // cleanup logic here
3    conn.Close()
4})
5
6// Call stop() if you want to prevent the function from running
7defer stop()

This is cleaner than spinning up a goroutine with select on ctx.Done() – the stdlib handles the lifecycle for you.

Key things to note:

• The function runs in its own goroutine
• stop() returns true if it successfully prevented the call
• Multiple AfterFunc calls on the same context are independent

dbtpl (part of the xo ecosystem) connects to a database, introspects the schema, and generates type-safe structs and query methods through Go templates. Hand-written DB structs easily drift from the schema; dbtpl makes the schema the single source of truth.

Explanation

How it works

Database (PostgreSQL / MySQL / SQLite...)
    ↓ introspect schema (tables, columns, types, indexes, FK...)
dbtpl
    ↓ apply Go templates
Generated Go code (structs, query methods)

No need to maintain DB structs by hand – when the schema changes, re-run dbtpl to sync.

Two modes

Schema mode: generate from the entire DB schema

1dbtpl schema postgres://user:pass@host/dbname -o ./models

Generates a Go struct for each table, including methods related to primary keys, foreign keys, and indexes.

Query mode: generate type-safe result structs from custom SQL

1dbtpl query postgres://user:pass@host/dbname << ENDSQL
2SELECT a.name::varchar AS name, b.type::integer AS my_type
3FROM authors a
4JOIN authortypes b ON a.id = b.author_id
5WHERE a.id = %%authorID int%%
6ENDSQL

%%param type%% is dbtpl’s query parameter syntax, which generates a corresponding function signature.

What gets generated

• Go structs matching each table (field types aligned with DB schema)
• CRUD query methods (Insert, Update, Delete, Get by PK)
• Enum types
• Struct tags (db:"column_name" etc.)

Template customization

dbtpl’s generation logic uses Go text/template. You can dump the built-in templates and modify them:

1dbtpl dump --src base -o ./custom-templates
2# edit the .tpl files in custom-templates/
3dbtpl schema postgres://... --src ./custom-templates

Knowledge Sugar

Why not just hand-write structs?

dbtpl vs sqlc

Both generate Go code from a database, but they differ in approach:

• sqlc: SQL-query-centric – write SQL first, then generate the corresponding functions
• dbtpl: schema-centric – introspects the entire DB, with more template flexibility

Go’s context is the standard solution for propagating cancellation signals, timeouts, and request-scoped data across goroutines. It lets an entire call chain stop cleanly at a unified point, leaving no goroutine leaks.

Explanation

The problem

An HTTP handler often runs multiple goroutines underneath – hitting the DB, calling external APIs, doing computation. Three scenarios cause trouble:

1. Timeout: the request times out, and downstream work should stop too
2. Cancellation: the user closes the browser mid-request; continuing is wasted work
3. Request-scoped data: auth tokens and trace IDs need to flow down the call chain without adding a parameter to every function

context unifies all three into a single standard interface.

Basic usage

1ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
2defer cancel() // must call this, otherwise it leaks resources
3
4result, err := db.QueryContext(ctx, "SELECT ...")

When the timeout expires, ctx.Done() closes, and all context-aware functions stop automatically.

How to receive it in a function

Functions that accept a context take ctx context.Context as the first parameter – this is a mandatory Go convention.

1func doWork(ctx context.Context) error {
2    select {
3    case <-ctx.Done():
4        return ctx.Err() // context.DeadlineExceeded or context.Canceled
5    case result := <-longOperation():
6        return process(result)
7    }
8}

Four ways to create a context

Every new context is a child of its parent – cancellation only propagates from parent to child. When the parent is cancelled, all children are cancelled too.

Knowledge Sugar

Context must not be stored in a struct

Always pass it as a parameter, never put it in a struct field.

A context represents the lifetime of a request, not object state. Storing it in a struct makes the lifetime unclear.

WithValue is only for cross-cutting concerns

WithValue is for things like tracing IDs and auth tokens – data every request needs but that would pollute function signatures. Do not use it to pass regular business parameters – for those, just add a proper parameter.

Mental model

Context is a control signal for a tree. When the root says “stop,” every goroutine in the tree stops – cleanly, no leaks.

A shallow copy copies the “reference to the data” – the original and the copy still share the underlying data. A deep copy copies “the data itself” – the two are completely independent. Go’s assignment is shallow copy by default; whether the original is affected depends on whether the type is a value type or a reference type.

Explanation

Value type vs Reference type

Slice: the most common gotcha

1a := []int{1, 2, 3}
2b := a          // shallow copy: b and a share the same underlying array
3
4b[0] = 99
5fmt.Println(a)  // [99 2 3]  <- a was modified!

a → [ header: ptr → [99, 2, 3], len=3, cap=3 ]
b → [ header: ptr ↗ ]   ← same array

To deep copy a slice, use copy():

1c := make([]int, len(a))
2copy(c, a)      // deep copy
3
4c[0] = 0
5fmt.Println(a)  // [99 2 3]  <- a is not affected

Map: assignment also only copies the reference

 1m1 := map[string]int{"a": 1}
 2m2 := m1        // shallow copy
 3
 4m2["a"] = 99
 5fmt.Println(m1["a"])  // 99 <- m1 was also modified
 6
 7// deep copy a map by looping manually
 8m3 := make(map[string]int)
 9for k, v := range m1 {
10    m3[k] = v
11}

Struct: value copy by default, but watch out for pointer/slice fields

 1type Person struct {
 2    Name   string
 3    Scores []int   // slice is a reference type!
 4}
 5
 6p1 := Person{Name: "Alice", Scores: []int{90, 80}}
 7p2 := p1           // struct is value-copied, but Scores only copies the slice header
 8
 9p2.Name = "Bob"
10fmt.Println(p1.Name)      // "Alice" <- string is a value type, not affected
11
12p2.Scores[0] = 100
13fmt.Println(p1.Scores[0]) // 100 <- the underlying array of Scores is shared!

To deep copy a struct with slices, explicitly copy each reference field:

1p3 := Person{
2    Name:   p1.Name,
3    Scores: make([]int, len(p1.Scores)),
4}
5copy(p3.Scores, p1.Scores)

copy() limitation: only copies the outer layer

copy() is a built-in function, but it only copies element values – it does not recursively handle nested reference types.

 1// Elements are value type → copy() works as a deep copy
 2a := []int{1, 2, 3}
 3b := make([]int, len(a))
 4copy(b, a)  // fully independent, no problem
 5
 6// Elements are reference type → copy() only copies the outer layer, inner layer is still shared
 7a := [][]int{{1, 2}, {3, 4}}
 8b := make([][]int, len(a))
 9copy(b, a)  // b[0] and a[0] still point to the same underlying array!
10
11b[0][0] = 99
12fmt.Println(a[0][0])  // 99 <- still modified

Conclusion: copy() is sufficient when elements are value types; when elements contain reference types, you need to copy recursively by hand.

Knowledge Sugar

Go has no built-in deep copy

For complex nested structures, there is no silver bullet. Common approaches:

Interview essential: the underlying structure of slice assignment

A Go slice is essentially a struct:

1// Conceptually looks like this (not actual Go code)
2type SliceHeader struct {
3    Data uintptr  // pointer to the underlying array
4    Len  int
5    Cap  int
6}

b := a copies this header, not the underlying array – that is the root cause of shallow copy.

One-sentence test

If modifying the copy also changes the original, it is a shallow copy. If not, it is a deep copy.

When multiple goroutines share a value that gets updated periodically (like an auth token), use atomic.Value for lock-free reads and writes – lighter than a mutex. Combined with time.Ticker for periodic refresh and ctx.Done() for goroutine lifecycle control, this forms a complete concurrent token management pattern.

Explanation

The problem

10 goroutines publish events concurrently, each needing an AUTHZ token. Another goroutine refreshes the token every 40s (TTL 60s, refreshing 20s early as buffer).

How do you let the refresher (writer) and 10 publishers (readers) safely share the same token without a data race?

atomic.Value: lock-free shared value

1var token atomic.Value  // declared, zero value is nil
2
3// Write (refresher goroutine)
4token.Store("new-token-string")
5
6// Read (publisher goroutines, can read concurrently, no lock needed)
7t := token.Load().(string)  // type assertion to get the actual type

atomic.Value’s Store/Load are atomic operations – no mutex needed. Multiple goroutines reading concurrently is completely safe, and an occasional Store will not give any reader a corrupted value.

Complete pattern

 1var token atomic.Value
 2
 3// Refresher goroutine: periodically refresh the token
 4go func() {
 5    ticker := time.NewTicker(40 * time.Second)
 6    defer ticker.Stop()  // remember to release ticker resources
 7
 8    for {
 9        select {
10        case <-ticker.C:
11            newToken, err := fetchToken(ctx)
12            if err == nil {
13                token.Store(newToken)
14            }
15        case <-ctx.Done():
16            return  // exit when context is cancelled
17        }
18    }
19}()
20
21// Publisher goroutines: 10 running concurrently
22for i := 0; i < 10; i++ {
23    go func() {
24        for {
25            select {
26            case event := <-eventCh:
27                t := token.Load().(string)  // read current token
28                publish(ctx, t, event)
29            case <-ctx.Done():
30                return  // exit when context is cancelled
31            }
32        }
33    }()
34}

time.Ticker: periodically firing channel

1ticker := time.NewTicker(40 * time.Second)
2// ticker.C is a channel that receives a signal every 40s
3// must call ticker.Stop() to release the underlying timer resources
4defer ticker.Stop()

ctx.Done(): unified exit signal

ctx.Done() returns a channel that gets closed when the context is cancelled. All goroutines listening on it in a select will receive the signal and exit simultaneously.

1select {
2case <-ticker.C:
3    // periodic refresh
4case <-ctx.Done():
5    return  // stop when told to, no goroutine leak
6}

Knowledge Sugar

sync.Mutex: a lock to protect critical sections

When multiple goroutines read and write the same data, a mutex ensures only one goroutine can enter at a time:

 1var mu sync.Mutex
 2var sharedData map[string]int
 3
 4// Write
 5mu.Lock()
 6sharedData["key"] = 42
 7mu.Unlock()
 8
 9// Read (also needs lock because map is not concurrent-safe)
10mu.Lock()
11v := sharedData["key"]
12mu.Unlock()
13
14// Idiomatic: defer unlock to avoid forgetting or leaking on panic
15mu.Lock()
16defer mu.Unlock()
17sharedData["key"] = 42

sync.RWMutex is the upgraded version – allows multiple goroutines to read simultaneously (RLock), but writing is exclusive (Lock):

 1var mu sync.RWMutex
 2
 3// Multiple goroutines can hold RLock simultaneously
 4mu.RLock()
 5v := sharedData["key"]
 6mu.RUnlock()
 7
 8// Write is exclusive
 9mu.Lock()
10sharedData["key"] = 99
11mu.Unlock()

Why atomic.Value instead of mutex?

The token scenario: writes are rare (once every 40s), reads are frequent (10 goroutines on every publish) – atomic.Value is the better choice.

What makes atomic operations work?

“Atomic” means indivisible – the operation appears to others as either completed or not started, never “halfway done.”

A normal assignment token = newToken may compile to multiple machine instructions. On a multi-core CPU, another core might read in the middle and get a half-written value (data race).

Atomic operations use special CPU instructions (like x86’s CMPXCHG) to guarantee at the hardware level:

1. Indivisibility: Store is fully written before it becomes visible to other cores
2. Memory visibility: adds a memory barrier, ensuring all cores see the latest value instead of getting stuck on a CPU cache

Normal assignment: Core A writes halfway → Core B might read a corrupted value
Atomic:            Core A finishes writing before it is visible → Core B always sees a complete value

This is why atomic.Value is safe without a mutex – the guarantee is made at the CPU instruction level, not through software locks.

Why refresh every 40s with a 60s TTL?

The 20s buffer absorbs:

• Network latency when fetching the new token
• The time gap before all goroutines read the new token

If you wait until 59s to refresh, any slight delay could cause publishers to send requests with an expired token.

Common cause of goroutine leaks

If a goroutine has no exit mechanism, it runs until the process ends. ctx.Done() is the standard exit signal – once the context is cancelled, all listening goroutines exit cleanly.

Each request produces exactly one log entry, handled by middleware. Errors are only wrapped at boundaries with third-party code. Logging follows the OpenTelemetry standard so logs can plug into any observability backend.

Explanation

Four core principles

1. Every request gets exactly one log

Instead of logging in every handler or function, middleware outputs a single structured log entry when the request completes, containing status code, latency, request ID, and other fields. This avoids scattered log entries for the same request that are hard to correlate.

2. Log should be wrapped in a proper way

Logs should be structured (structured logging) – not fmt.Println("error:", err), but a format with fields:

1{
2  "level": "error",
3  "request_id": "abc-123",
4  "method": "POST",
5  "path": "/orders",
6  "latency_ms": 42,
7  "error": "calling payment sdk: connection refused"
8}

3. Only wrap errors on boundaries with 3rd party code

A boundary is the seam between your code and a system you do not own – calling a DB, calling a third-party SDK, calling an external API are all boundaries.

Errors from third parties lack your context, so wrap once at the boundary. Between internal functions, just pass the error up.

Given the chain: Handler -> getOrder() -> queryOrderFromDB():

 1// Layer 3: DB boundary, the only place to wrap
 2func queryOrderFromDB(ctx context.Context, id int) (*Order, error) {
 3    row, err := db.Query(ctx, "SELECT * FROM orders WHERE id = ?", id)
 4    if err != nil {
 5        // Wrap at the boundary, describe what operation was happening
 6        return nil, fmt.Errorf("querying order id=%d: %w", id, err)
 7    }
 8    // ...
 9}
10
11// Layer 2: internal function, pass up directly, do not re-wrap
12func getOrder(ctx context.Context, id int) (*Order, error) {
13    order, err := queryOrderFromDB(ctx, id)
14    if err != nil {
15        // Just return, no new wrap
16        return nil, err
17    }
18    return order, nil
19}
20
21// Layer 1: handler, also does not wrap -- let middleware handle the log
22func handleGetOrder(w http.ResponseWriter, r *http.Request) {
23    order, err := getOrder(r.Context(), id)
24    if err != nil {
25        // Hand off to middleware, do not log here
26        http.Error(w, "internal error", http.StatusInternalServerError)
27        return
28    }
29    // ...
30}

If every layer wraps, the error message becomes:

handleGetOrder: getOrder: queryOrderFromDB: querying order id=42: connection refused

The repeated function names add no new information – just noise.

Wrapping only at the boundary is much clearer:

querying order id=42: connection refused

The rule: wrap = add new information. No new information, just return err.

4. OTEL compliant logging in middleware

OTEL = OpenTelemetry, the CNCF observability standard that defines a unified format and API for traces, metrics, and logs.

When middleware outputs OTEL-compliant logs, you can plug into any backend (Grafana, Datadog, GCP Cloud Logging) without changing application code.

Request → Middleware (start timer, inject trace context)
           ↓
         Handler (process logic, pass errors up)
           ↓
         Middleware (at the end, log one entry with status / latency / error)

Knowledge Sugar

Why not let each handler log on its own?

Problems with scattered logging:

• The same request might get logged multiple times across functions – hard to correlate
• Inconsistent formats – hard to filter with queries
• Duplicate logs – more noise, higher cost

Middleware-centralized logging means: one request = one log entry = one trace. Searching and debugging become much more intuitive.

Error wrap decision guide

RBAC (Role-Based Access Control) is a permissions model where you first define what roles can do, then assign roles to people. K8s uses RBAC to control who can perform which operations on which resources. CD tools use RBAC to control who can deploy to which environment.

Explanation

Three core elements of RBAC

Subject (who)  →  RoleBinding (assignment)  →  Role (what they can do)

• Subject: a user, group, or service account (programmatic identity)
• Role: defines a set of permissions (which operations on which resources)
• RoleBinding: assigns a Role to a Subject

Intuitive analogy: a Role is a job title (engineer, manager), a RoleBinding is the “appointment letter,” and the Subject is the person being appointed.

RBAC in Kubernetes

K8s RBAC controls “who can do what against the K8s API.”

Resources & Verbs

1rules:
2- apiGroups: [""]
3  resources: ["pods", "services"]   # resource types
4  verbs: ["get", "list", "watch"]   # allowed operations

Common verbs: get list watch create update patch delete

Role vs ClusterRole

 1# Role: only effective in the production namespace
 2kind: Role
 3metadata:
 4  namespace: production
 5  name: pod-reader
 6rules:
 7- apiGroups: [""]
 8  resources: ["pods"]
 9  verbs: ["get", "list"]
10
11---
12# RoleBinding: assign this Role to alice
13kind: RoleBinding
14metadata:
15  namespace: production
16subjects:
17- kind: User
18  name: alice
19roleRef:
20  kind: Role
21  name: pod-reader

Service Account: identity for programs

Programs running in pods (like a CD tool agent) are not people. K8s uses Service Accounts to give them an identity, then uses RoleBinding to control what they can do:

CD tool agent (Pod)
  → uses ServiceAccount: octopus-agent
  → RoleBinding → ClusterRole: deploy-permissions
  → can apply manifests, update Deployments

RBAC in CD Tools (Octopus)

The CD tool has its own RBAC layer, controlling “who can operate which project / environment”:

Engineer alice
  → belongs to Team: backend-team
  → Team is assigned Role: deployer (can deploy, cannot change project settings)
  → restricted to Environment: dev, staging (cannot touch prod)

Typical role hierarchy:

Knowledge Sugar

Why does K8s need RBAC?

The K8s API can do a lot – delete Pods, modify Secrets, scale Deployments. Without access control, anyone or any program that can connect to the cluster can do anything. RBAC lets you:

• Restrict devs to only operate within their own namespace
• Give CI/CD agents only manifest apply permission, not Secret deletion
• Give SRE cluster-level read access without prod write access

Principle of Least Privilege

The design philosophy behind RBAC: give each subject only the minimum permissions needed to do its job. If a CD tool agent only needs apply, do not give it delete.

For CD tool background, see the WarpCD vs Octopus Deploy post.

WarpSpeedCD (internally called WarpCD) is a GitOps pull model – Git is the single source of truth, and a bot syncs changes to the cluster. Octopus is a push model – a pipeline actively triggers deploys, with approval and release management handled in the Octopus UI. The fundamental difference is who drives the deploy.

Explanation

WarpCD (GitOps / Pull model)

Code PR merged
    ↓
WarpCD bot opens a PR in the K8s repo (updates manifests)
    ↓
Manual review & approve PR
    ↓
PR merged → cluster auto-syncs (pull)

• Source of truth: K8s manifests live in a Git repo; cluster state always follows Git
• Audit trail: every deploy is a Git commit, giving you history for free
• Approval: through the PR review process, consistent with code review

Octopus Deploy (Push model)

CI build completes → artifact (image) pushed to registry
    ↓
Octopus detects new version, creates a Release
    ↓
Promotion flow: dev → staging → prod (each stage can have an approval gate)
    ↓
Octopus actively applies manifests to the cluster (push)

• Source of truth: Octopus’s Release and Project configuration
• Approval: approval gates configured in the Octopus UI, not through Git PRs
• Runbook: can run operational tasks (DB migration, rollback, smoke test)

Core differences

Knowledge Sugar

How to do a CD pipeline migration?

Switching from GitOps to Octopus is not just migrating YAML – it is replacing the entire deploy driver:

1. Artifacts stay the same: container images and Helm charts do not need changes
2. K8s manifests may be kept: Octopus can use kubectl/Helm steps to apply existing manifests
3. What actually migrates is the process:
   • Replace WarpCD bot’s PR flow with Octopus’s Release + Promotion
   • Replace Git PR approval with Octopus’s approval gates
   • Create corresponding Projects + Steps in Octopus for each service’s deploy config

Typical migration steps

1. Create environments in Octopus (dev / staging / prod)
2. Create an Octopus Project for each service, configure deploy steps
3. Set up approval gates to replace PR review
4. Run shadow mode (old and new in parallel) to verify results match
5. Once confirmed, turn off the WarpCD bot -- switchover complete

GCP Pub/Sub retry is ACK-based – no ACK means redeliver. With a Pusher in the middle, retry becomes two layers: the Pusher’s own retry (fast, finely configurable) + Pub/Sub redelivery (slow, last resort).

Explanation

Native Pub/Sub retry mechanism

After a subscriber pulls a message, it must return an ACK within the ACK deadline. Otherwise Pub/Sub treats it as a failure and automatically redelivers:

Pub/Sub
  ↓ deliver message
Subscriber
  ↓ success → ACK → message removed from subscription
  ↓ failure / timeout → no ACK → Pub/Sub redelivers

Redelivery continues until the message is ACKed or exceeds the retention period (default 7 days).

With a Pusher: two-layer retry

The Pusher sits in between, creating two independent retry layers:

Pub/Sub Subscription
    ↓ pull
  Pusher
    ↓ push → Target Service

Layer 1 – Pusher’s own retry (CRD config)

When the Pusher’s push to the target service fails, it retries internally first (count and backoff are configurable via CRD) without going back to Pub/Sub.

Layer 2 – Pub/Sub redelivery

• If the target service processes successfully -> Pusher ACKs to Pub/Sub -> message done
• If all of Pusher’s retries are exhausted and it still fails -> NACK to Pub/Sub -> Pub/Sub redelivers to the Pusher

So Pub/Sub retry is the last resort. Day-to-day transient failures are absorbed by the Pusher layer.

Key: when does the Pusher ACK to Pub/Sub?

The ACK timing determines whether the entire retry chain works correctly:

Knowledge Sugar

Dead Letter Topic

When a message fails repeatedly beyond the maximum retry count, instead of letting it loop forever, move it to a dedicated topic for isolation:

Normal:  Pub/Sub → Pusher → Target Service ✓ → ACK
Failure: Pub/Sub → Pusher → Target Service ✗ → NACK → retry N times
                                                        ↓ exceeds limit
                                               Dead Letter Topic

Three uses for a Dead Letter Topic:

• Unblock normal traffic: problematic messages are moved away, the rest keep flowing
• Post-mortem investigation: see which messages keep failing and why
• Manual replay: after fixing the bug, replay dead letter messages back into the normal flow

Benefits of two-layer retry

Pure Pub/Sub retry uses exponential backoff, which is slow. The Pusher layer can use faster, finer-grained retry strategies. Most transient failures get resolved at this layer without going through the full Pub/Sub redelivery cycle.

For the Pusher architecture background, see the gRPC Pusher Pattern post. For Pub/Sub Topic & Subscription basics, see the GCP Pub Sub Topic & Subscription post.

Topic is where messages are published to; subscription is how a consumer receives messages from that topic.

A topic is the publishing channel for messages. A subscription is how a consumer subscribes to that channel. Separating the two lets multiple subscribers independently consume from the same topic.

Explanation

Publisher → Topic → Subscription A → Service A
                 → Subscription B → Service B

• Topic: the target for publishing. The publisher just sends messages to the topic without caring who reads them.
• Subscription: a consumption channel attached to a topic. Each subscription receives an independent copy of every message on the topic.

Each subscription tracks its own progress – where Service A has read up to and where Service B has read up to are independent. Messages stay in a subscription until they are acknowledged or exceed the retention period.

Two subscription types

• Pull: the subscriber actively asks “any new messages?” – suited for backend services that want to control their own consumption rate
• Push: Pub/Sub proactively sends messages to a specified HTTP endpoint – suited for serverless or webhook scenarios

Knowledge Sugar

Why separate topic and subscription?

If there were only topics, all consumers would compete for the same messages – once read, they are gone. With subscriptions, the same message can be consumed independently by a logging service, an analytics service, and a notification service without interference. This is the fan-out pattern.

What happens if nobody consumes from a subscription?

Messages pile up until the retention period expires (default 7 days), then get auto-deleted. So if you create a subscription, make sure something is consuming from it – otherwise it is wasted resources.

Topic ownership should follow the domain

The topic should be created by the service that owns the domain. For example, if monolith ServiceA publishes events to a topic belonging to ServiceB’s domain, the topic should live in ServiceB’s GCP project, with IAM granting ServiceA publish permission:

ServiceA (monolith)
  │  roles/pubsub.publisher (IAM grant)
  ▼
Topic (owned by ServiceB)
  ▼
Subscription → ServiceC consume

After migration completes, you only need to remove ServiceA’s publish logic – everything else stays the same.

Subscription ownership design

The subscription owner does not have to match the topic owner. Two options:

During migration, Option 1 is recommended – ServiceB owns both the topic and subscription, then grants ServiceC consume permission. This gives ServiceB full visibility over the message flow.

ServiceC connects to the subscription via IAM:

1resource "google_pubsub_subscription_iam_member" "subscriber" {
2  subscription = google_pubsub_subscription.my_subscription.name
3  role         = "roles/pubsub.subscriber"
4  member       = "serviceAccount:service-c@service-c-project.iam.gserviceaccount.com"
5}

ServiceC’s code just needs to pull using its own service account. GCP verifies IAM and grants access – no extra configuration needed.

Creating Topic and Subscription with Terraform

Like buckets, these are GCP resources that can be managed directly with Terraform:

 1resource "google_pubsub_topic" "my_topic" {
 2  name    = "my-topic"
 3  project = "service-b-project"
 4}
 5
 6resource "google_pubsub_subscription" "my_subscription" {
 7  name  = "my-subscription"
 8  topic = google_pubsub_topic.my_topic.id
 9}
10
11# Grant ServiceA's service account publish permission
12resource "google_pubsub_topic_iam_member" "publisher" {
13  topic  = google_pubsub_topic.my_topic.id
14  role   = "roles/pubsub.publisher"
15  member = "serviceAccount:service-a@service-a-project.iam.gserviceaccount.com"
16}

Generics (Go 1.18+) let you write type-parameterized functions – the type is determined at call time, but type checking happens at compile time. This gives you both flexibility and safety.

Explanation

Without Generics vs with Generics

 1// Without generics: same logic duplicated for each type
 2func sumInts(nums []int) int {
 3    var total int
 4    for _, n := range nums { total += n }
 5    return total
 6}
 7func sumFloats(nums []float64) float64 {
 8    var total float64
 9    for _, n := range nums { total += n }
10    return total
11}
12
13// Or use any, but lose type safety -- errors only show up at runtime
14func sumAny(nums []any) any { ... }
15
16// With generics: one function, compile-time type safety
17func sum[T int | float64](nums []T) T {
18    var total T
19    for _, n := range nums { total += n }
20    return total
21}
22
23sum([]int{1, 2, 3})       // T = int
24sum([]float64{1.1, 2.2})  // T = float64

Syntax breakdown

1func ptr[T any](v T) *T { return &v }
2//       ^^^^^^  ^ ^
3//       |       | return type is also T
4//       |       parameter type is T
5//       type parameter: T is the name, any is the constraint

• T – type parameter name, convention is a single uppercase letter
• any – constraint, meaning T can be any type

Constraints can be narrowed

1// Only accept numbers
2func sum[T int | float64](nums []T) T { ... }
3
4// Only accept types that implement String()
5func print[T fmt.Stringer](v T) { ... }

Type inference: usually no need to specify explicitly

1ptr("hello")         // inferred: T = string
2ptr[string]("hello") // explicit, same result

Knowledge Sugar

Generics vs any vs interface

When to use Generics? When the same logic needs to work across multiple types and you want compile-time type error detection. Pointer helpers and collection operations (map, filter, reduce) are the most common examples.

Go 1.22+ introduced several language-level changes. go fix can automatically upgrade old patterns to modern ones: loop variable capture, interface{} to any, and inline pointer helper functions can all be removed or simplified.

Explanation

1. Remove unnecessary loop variable captures

Before Go 1.22, the for loop variable was shared across all iterations at the same memory address. Closures captured the same variable, causing a classic bug:

 1// Before Go 1.22: all goroutines print the same value (the last one)
 2for _, v := range items {
 3    go func() { fmt.Println(v) }()
 4}
 5
 6// Old workaround: redeclare inside the loop to force a new variable
 7for _, v := range items {
 8    v := v  // capture a new variable per iteration
 9    go func() { fmt.Println(v) }()
10}

Starting from Go 1.22, each iteration has an independent variable, making v := v redundant. go fix removes it automatically.

2. Replace interface{} with any

Go 1.18 introduced any as a type alias for interface{}. They are identical, but any is shorter:

1// Old
2func process(v interface{}) {}
3var m map[string]interface{}
4
5// New
6func process(v any) {}
7var m map[string]any

3. Remove inline pointer helper functions

Taking a pointer to a literal used to require a helper function (because Go does not allow &"hello" directly):

1// Old workaround
2func strPtr(s string) *string { return &s }
3func intPtr(i int) *int { return &i }
4
5name := strPtr("Alice")
6age  := intPtr(30)

Modern Go can replace all type-specific helpers with a single generic version, or just declare a variable and take its address:

1// One generic helper replaces all
2func ptr[T any](v T) *T { return &v }
3
4// Or just declare and reference
5name := "Alice"
6req.Name = &name

What is a literal?

A literal is a fixed value written directly in code – not a variable, not a computed result, just the value as written:

1"hello"        // string literal
242             // integer literal
3true           // boolean literal
4[]int{1, 2, 3} // slice literal

Go does not allow taking a pointer to a literal (&"hello" is a compile error) because literals have no memory address. You must assign to a variable first – that is why pointer helper functions existed.

For details on generics, see the Go Generics post.

Knowledge Sugar

What is go fix?

go fix is an official Go refactoring tool that applies mechanical fixes for each version’s breaking changes or language updates. Running it once when upgrading saves a lot of manual edits.

1go fix ./...

Why is the Go 1.22 loop variable change important?

This bug existed for over ten years and was one of Go’s most famous gotchas – nearly every Go developer hit it at least once. Go 1.22 changed the behavior to match intuition, which is a significant language-level fix.

A gRPC Pusher is a message dispatch intermediary – it pulls messages from a Pub/Sub subscription and proactively pushes them to a target service’s gRPC endpoint. This solves the problem where workers cannot control which consumer processes which message.

Explanation

Why is this needed?

The testing pain point with worker-based services: a PRRC (PR Review Copy) environment and master share the same subscription. There is no way to guarantee that a test message will be consumed by the PRRC pod rather than a master worker – so every test requires manually deploying the commit image to master, which is cumbersome.

The gRPC Pusher solves this as an intermediary:

Pub/Sub Subscription
        ↓  pull
   gRPC Pusher (intermediary layer)
        ↓  push (can control which endpoint to route to)
  Target Service (specified gRPC endpoint)

The Pusher centrally pulls messages, then pushes them to a specified endpoint based on configuration – giving precise control over PRRC traffic without worrying about master workers stealing messages.

Configured via CRD

The Pusher’s behavior is managed through Kubernetes CRD config, which can set:

• Retry policy: retry strategy after failures (count, backoff)
• Traffic limit: rate limiting for pushes

Knowledge Sugar

Public implementations of this pattern

This “pull then push” message dispatch pattern is common in the industry:

• Knative Eventing: pulls from Broker/Channel, pushes to HTTP/gRPC sink endpoints, supports retry and dead letter sink, configured via CRD
• Dapr Pub/Sub: runs as a sidecar, pulls from various pub/sub backends, pushes to application endpoints via gRPC or HTTP
• GCP Push Subscription: GCP’s native push mode, sends messages as HTTP POST to a specified endpoint, supports exponential backoff retry

Core design concept: decoupling consume and process

Traditional workers couple “pulling messages from a subscription” and “processing messages” in the same process, making traffic routing hard to control. The Pusher separates the two:

• Pusher handles consume (single entry point)
• Target service only handles process (can be any endpoint)

This is also the core idea behind Dapr and Knative – extracting messaging infrastructure out of application logic.

Foreign keys and JOINs solve two different problems: FK ensures data integrity at the write layer, while JOIN associates data at the read layer. Not having a FK does not prevent JOINs, but you lose the database-level safety net.

Explanation

Two core functions of a Foreign Key

Using shipping_fees (FK) -> shipping_classes (PK) as an example:

1. Referential integrity on insert: shipping_fees cannot contain a shipping_class_id that does not exist in shipping_classes
2. Cascading protection on delete: as long as any shipping_fees row references a shipping_class, that shipping_class row cannot be deleted

1-- With FK, the database blocks both of these:
2INSERT INTO shipping_fees (shipping_class_id, ...) VALUES (999, ...);
3-- ERROR: Cannot add or update a child row: foreign key constraint fails
4
5DELETE FROM shipping_classes WHERE id = 1;
6-- ERROR: Cannot delete or update a parent row: foreign key constraint fails

JOIN does something different

JOIN only associates data from two tables at query time. It does not care whether the data is valid:

1SELECT sf.*, sc.name
2FROM shipping_fees sf
3JOIN shipping_classes sc ON sf.shipping_class_id = sc.id;

You can JOIN without a FK, but if there are orphan records, the JOIN silently excludes them without raising an error.

Why do DBRE teams dislike FK?

FK causes trouble at the DB operations layer:

• Emergency data fixes via DML (direct INSERT / DELETE) can be blocked by FK constraints
• Large data migrations require temporarily disabling FK checks
• FK behavior gets more complex in replication setups

The compromise

Keep the JOIN design (no FK), but the DBRE team uses DML validation at the operations layer to ensure data correctness – essentially replacing FK’s two protections with a manual process.

Knowledge Sugar

FK vs Application-level validation vs DML validation

The nature of this tradeoff

FK means “let the database worry about data correctness.” No FK means “we worry about it ourselves.” The former is safer but sacrifices operational flexibility; the latter is more flexible but shifts responsibility to the application and operations layers.

For high-traffic systems or those with urgent on-call needs, DBRE teams tend to prefer the latter – they do not want to be stuck on a constraint during a critical moment.

gRPC is an RPC framework developed by Google. It uses HTTP/2 for transport and Protobuf for serialization, making it faster and more structured than traditional REST/HTTP+JSON – but less readable, so it is mainly used for internal microservice communication.

Explanation

Traditional HTTP (REST + JSON)

Client → POST /users HTTP/1.1
         Content-Type: application/json
         {"name": "Alice", "age": 30}

Server → 200 OK
         {"id": 1, "name": "Alice"}

• Operations are defined by URL + HTTP method
• Data format is JSON (human-readable text)
• Based on HTTP/1.1 (each request uses an independent connection)

How gRPC does it

Client → calls UserService.CreateUser(CreateUserRequest)
Server → returns CreateUserResponse

• Services and messages are defined in .proto files; calling them feels like calling a local function
• Data format is Protobuf (binary, not directly readable)
• Based on HTTP/2 (multiplexing – a single connection handles multiple requests)

Core differences

Knowledge Sugar

What is HTTP/2 multiplexing?

In HTTP/1.1, each request must wait for the previous response before sending the next one (or open a new connection). HTTP/2 can handle multiple request/response pairs in parallel over a single connection, significantly reducing latency.

Streaming

gRPC supports four communication modes, which are hard to do with REST:

Unary:              one request → one response (most common)
Server streaming:   one request → multiple responses (e.g. real-time push)
Client streaming:   multiple requests → one response (e.g. uploading chunked data)
Bidirectional:      multiple requests ↔ multiple responses (e.g. real-time chat)

Relation to Protobuf gRPC’s data format is Protobuf – see the earlier post on Protobuf Reserved Fields & API Versioning for more context.

Language preference does not need to be a request parameter. HTTP already has the Accept-Language header for this – the server reads it directly from the header, and the client does not need to pass an extra field.

Explanation

What is Accept-Language?

It is a request header defined by the HTTP standard (RFC 7231) that tells the server which languages the client prefers. Browsers set it automatically based on the user’s system language. API clients can set it manually:

Accept-Language: zh-TW,zh;q=0.9,en;q=0.8

• Multiple languages are separated by commas
• q is the quality factor, ranging from 0 to 1 (default 1.0) – higher means more preferred
• The example above: prefer zh-TW, then zh, then en

How does the server read it?

In Go, just get it from the header and parse:

1lang := r.Header.Get("Accept-Language")
2// "zh-TW,zh;q=0.9,en;q=0.8"
3
4// Simple approach: take the first one (highest priority)
5preferred := strings.Split(lang, ",")[0]  // "zh-TW"

In practice you would use an existing library to parse the full q-value ordering, match against the server’s supported languages, and fall back to a default if nothing matches.

Why is this better than a request parameter?

Knowledge Sugar

Content Negotiation

Accept-Language is part of HTTP content negotiation – the client tells the server “here’s what I can accept” and the server picks the best match from what it can provide. Related headers in the same family:

• Accept: preferred response format (application/json, text/html)
• Accept-Encoding: preferred compression (gzip, br)

Usually handled in middleware

No need for every handler to read the header itself. Parse it in middleware, put the result in the context, and let downstream handlers read from there:

1func LanguageMiddleware(next http.Handler) http.Handler {
2    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
3        lang := parseAcceptLanguage(r.Header.Get("Accept-Language"))
4        ctx := context.WithValue(r.Context(), "lang", lang)
5        next.ServeHTTP(w, r.WithContext(ctx))
6    })
7}

Marshal converts an in-memory data structure into a transmittable format (like JSON). Unmarshal does the reverse – it turns JSON back into a data structure. Most APIs use JSON because it is human-readable, natively supported by JavaScript, and universally available across languages.

Explanation

Marshal / Unmarshal

 1type ShippingClass struct {
 2    Name string `json:"name"`
 3    Lang string `json:"lang"`
 4}
 5
 6// Unmarshal: JSON string → struct
 7jsonStr := `{"name": "標準配送", "lang": "zh"}`
 8var obj ShippingClass
 9json.Unmarshal([]byte(jsonStr), &obj)
10// obj.Name == "標準配送"
11
12// Marshal: struct → JSON string
13data, _ := json.Marshal(obj)
14// data == {"name":"標準配送","lang":"zh"}

This is not limited to JSON – serialization for XML, YAML, and protobuf is also called marshal/unmarshal. Same concept, different formats.

Why do most APIs use JSON?

• Human-readable: you can read it directly when debugging; binary formats like protobuf cannot do this
• Native JS support: browsers parse JSON at zero cost, no extra handling on the frontend
• Universal: almost every language has a mature JSON library
• Less verbose than XML: XML’s opening and closing tags add a lot of overhead

Knowledge Sugar

What are Go struct tags?

The json:"name" tag tells Go’s JSON library which key name this field maps to in JSON. Without a tag, it defaults to the field name (case-sensitive):

1type Example struct {
2    DisplayName string `json:"display_name"`  // JSON key is display_name
3    Age         int    `json:"age,omitempty"` // omitempty: omit this key when value is zero
4    Internal    string `json:"-"`             // never output to JSON
5}

Downsides of JSON

JSON is not without problems. In high-performance scenarios (internal microservice communication), protobuf is the usual choice:

API request validation should happen on both sides, but with different roles: the client provides instant UX feedback, while the server enforces all business logic validation. The server side is mandatory because client validation can be bypassed.

Explanation

Why can’t you skip server-side validation?

Anyone can skip the frontend and hit the API directly. Client-side validation is just a UX optimization; the server is the real enforcement layer.

Cross-field Validation

When fields have logical dependencies (e.g. discount cannot be 0 when eligibility is true), handle it in a Validate() method on the server side:

 1type CreateDiscountRequest struct {
 2    Eligibility bool    `json:"eligibility"`
 3    Discount    float64 `json:"discount"`
 4}
 5
 6func (r CreateDiscountRequest) Validate() error {
 7    if r.Eligibility && r.Discount == 0 {
 8        return errors.New("discount cannot be 0 when eligibility is true")
 9    }
10    return nil
11}

Call it in the handler right after decoding:

1var req CreateDiscountRequest
2json.NewDecoder(r.Body).Decode(&req)
3
4if err := req.Validate(); err != nil {
5    http.Error(w, err.Error(), http.StatusBadRequest) // 400
6    return
7}

Knowledge Sugar

Should you use a validation library?

For simple cases, a hand-written Validate() is enough. When rules get complex or you have many structs, consider go-playground/validator, which supports struct tags for basic rules:

1type Request struct {
2    Name  string  `validate:"required"`
3    Email string  `validate:"required,email"`
4    Age   int     `validate:"gte=0,lte=130"`
5}

But cross-field logic (dependencies between fields) still needs a custom validator – no library handles your business logic for you.

Return 400 or 422?

• 400 Bad Request: malformed input, parse failure
• 422 Unprocessable Entity: valid format but failed business logic validation (like the cross-field error above)

Semantically 422 is more precise, but in practice many APIs use 400 for everything. Follow your team’s convention.

BFF (Backend For Frontend) is an architecture pattern where you build a dedicated backend layer for each frontend client, instead of having all clients share a single API.

Explanation

Why do you need BFF?

Different clients have very different data needs. Take an e-commerce platform as an example:

• Mobile app: small screen, limited bandwidth, needs only a few compact fields
• Web browser: can show richer data, needs more fields
• Third-party partners: yet another set of data format requirements

With a single shared API, two problems arise:

• Over-fetching: returning fields the client does not use
• Under-fetching: one request is not enough, so the client sends multiple requests

BFF adds a layer between clients and backend microservices:

Mobile App  →  Mobile BFF  ┐
Web App     →  Web BFF     ├─→  microservices
Partner API →  Partner BFF ┘

Each BFF handles:

1. Aggregating responses from multiple microservices (one request instead of many)
2. Trimming data to the format that specific client needs
3. Client-specific logic (e.g. mobile pagination)

Who maintains the BFF?

Usually the frontend team, so they can adjust the API format on their own without waiting for backend changes.

Knowledge Sugar

BFF vs API Gateway

These are easy to confuse, but they serve different purposes:

In practice they can coexist – the API Gateway sits in front handling common concerns, while BFFs sit behind it doing client-specific data aggregation.

When do you not need BFF?

• Only one client type (e.g. web only)
• The backend is a monolith with nothing to aggregate
• The team is small and maintaining multiple BFFs costs more than it saves

Running docker build inside GitLab CI does not work out of the box – the job itself already runs in a container with no Docker daemon. You need Docker-in-Docker (DinD) as a sidecar service to provide the daemon.

Explanation

Overall structure

1stages:
2  - test    # runs first
3  - build   # runs after (only if all test jobs pass)

Jobs in the same stage run in parallel; stages run sequentially.

run_tests job – typical job structure

1run_tests:
2  stage: test
3  image: python:3.11-slim  # set up container based on this image where job runs
4  before_script:
5    - apt-get update && apt-get install -y make gcc python3-dev
6  script:
7    - make test

• image: which Docker image to use as the execution environment for this job
• before_script: runs before every script, good for dependency installation
• script: the main commands to execute

build_image job – Docker-specific details

 1build_image:
 2  stage: build
 3  image: docker:20.10.16           # Docker CLI
 4  services:
 5    - docker:20.10.16-dind         # Docker daemon (sidecar)
 6  variables:
 7    DOCKER_TLS_CERTDIR: "/certs"   # enable TLS for secure communication
 8  before_script:
 9    - docker login -u $REGISTRY_USER -p $REGISTRY_PASS
10  script:
11    - docker build -t $IMAGE_NAME:$IMAGE_TAG .
12    - docker push $IMAGE_NAME:$IMAGE_TAG

Why do you need services: docker:dind?

Each GitLab CI job runs inside a container. That container has no Docker daemon by default, so docker build fails.

services is GitLab CI’s sidecar mechanism – it starts an extra container alongside the job container, both on the same network. docker:dind is an image with a built-in Docker daemon, designed for exactly this:

job container (docker:20.10.16, has CLI)
        ↕ TLS-encrypted communication
sidecar (docker:20.10.16-dind, has daemon)

What is DOCKER_TLS_CERTDIR: "/certs"?

Here “CLI” means the side that runs docker build – docker build itself just sends a request to the daemon via the Docker API, and the daemon does the actual build. TLS protects this API communication:

job container: docker build ...
     ↕ Docker API over TLS
sidecar: daemon actually runs the build

Setting DOCKER_TLS_CERTDIR: "/certs" makes DinD auto-generate TLS certificates. The CLI side reads them too, and the two complete a handshake before communicating. Setting it to an empty string "" disables TLS, but that is insecure and not recommended.

How are credentials passed in?

$REGISTRY_USER and $REGISTRY_PASS are CI/CD Variables configured in the GitLab project settings. They never appear in the YAML file, avoiding hardcoded secrets.

Knowledge Sugar

Top-level variables vs job-level variables

1variables:           # available to all jobs
2  IMAGE_NAME: alienmushroom/demo-app
3
4build_image:
5  variables:         # only this job; can override top-level
6    DOCKER_TLS_CERTDIR: "/certs"

Does before_script behave differently from script on failure? No – if any command returns a non-zero exit code, the job is marked as failed and subsequent stages do not run.

Keep versions aligned image: docker:20.10.16 and services: docker:20.10.16-dind must use the same version number. A mismatch between CLI and daemon versions can cause unexpected issues.

When removing a response field from a protobuf message, you cannot simply delete it – you must mark it as reserved. Otherwise, if a future field reuses the same field number, old clients will misinterpret the new field’s value as the old field.

Explanation

Why is reserved needed?

Protobuf identifies data on the wire by field number, not field name. So when you:

1. Originally have string old_field = 3; in a response message
2. Integrate a new external service, the backend logic changes, this field is no longer populated, so you delete it
3. Later add int32 new_field = 3; (reusing number 3)

At this point, clients still using the old proto definition will receive the response and try to read new_field’s value as old_field – the type mismatch causes a crash.

Correct approach: mark it as reserved

1message MyResponse {
2  reserved 3, 5;                    // reserve these field numbers from reuse
3  reserved "old_field", "another";  // also reserve the field names (prevents accidental reuse in code)
4
5  string active_field = 1;
6  int32 other_field = 2;
7}

With reserved in place, any attempt to reuse those numbers or names will cause a protoc compilation error.

What is the relationship between v2alpha and v2beta?

These are API version stability labels, following Google AIP (API Improvement Proposals) naming conventions, commonly seen in gRPC / proto package naming:

The typical progression is: v2alpha1 -> v2alpha2 -> v2beta1 -> v2beta2 -> v2

How it looks in proto files:

1// Experimental version
2package mycompany.myservice.v2alpha1;
3
4// Feature-complete but still being polished
5package mycompany.myservice.v2beta1;
6
7// Stable release
8package mycompany.myservice.v2;

Different versioned packages are completely independent namespaces that can coexist, allowing old and new clients to each use their corresponding version.

Knowledge Sugar

Two ways to write reserved, and they can be separate or combined:

1reserved 1 to 3;       // contiguous range
2reserved 1, 2, 3;      // individual listing
3reserved "foo", "bar"; // reserve names (recommended to do both, prevents typo reuse)

Difference from deprecated:

• deprecated = true is a human-readable warning telling developers “this field is not recommended anymore,” but it still works
• reserved is enforced at compile time – it completely blocks any reuse

Practical mindset when designing new proto contracts:

• A field number, once released, is forever – before deleting a field, decide whether it needs reserved
• Alpha/Beta versions give you room to experiment; once you reach a stable v2, removing or changing fields requires a formal deprecation process

Further reading:

• Google AIP-180: definition of breaking vs non-breaking changes
• oneof field numbers in protobuf also need reserved
• Buf (buf.build): a protobuf linter that can automatically detect breaking changes

When a MySQL lock is not properly released, other transactions trying to access the same data keep waiting indefinitely, causing batch jobs to get stuck (blocking).

Explanation

MySQL uses locks to protect data consistency – when a transaction is reading or writing a row, it acquires a lock on it first, forcing other transactions to queue up and wait.

Under normal circumstances, locks are automatically released after a transaction COMMITs or ROLLBACKs. But if a transaction gets stuck for some reason (e.g., a connection not properly closed, a process crash, or a long-running query), the lock persists and everything behind it piles up.

In this case, a transaction was holding a lock without finishing. The batch job kept waiting. The DBRE team manually terminated that session with a KILL command, which released the lock and allowed the batch job to resume.

Knowledge Sugar

Common lock types:

• Row lock: locks only the specific rows; fine-grained; InnoDB’s default
• Table lock: locks the entire table; coarse-grained; used by MyISAM
• Gap lock: locks the “gaps” between index ranges to prevent phantom reads

How to find out who is blocking whom:

1-- Check which locks are currently waiting
2SELECT * FROM information_schema.INNODB_LOCK_WAITS;
3
4-- View all active transactions
5SELECT * FROM information_schema.INNODB_TRX;
6
7-- Manually terminate a stuck session (what the DBRE team did)
8KILL <process_id>;

Why are batch jobs particularly prone to this? Batch jobs typically process large volumes of data with long-running transactions. If another transaction also needs the same set of rows, deadlocks or long blocking are more likely to occur.

Related concepts to explore further:

• Deadlock vs Blocking: deadlock is mutual waiting (both sides stuck); blocking is one-directional waiting
• InnoDB’s innodb_lock_wait_timeout setting (auto-abort on timeout)
• SELECT ... FOR UPDATE explicitly acquires a row lock

Fetching private Go modules directly from GitHub in CI/CD easily gets rate-limited (403) when there are too many requests. The fix is to set GOPROXY in your CD yaml to point at an internal Athens proxy server – Athens caches the modules, so subsequent builds never hit GitHub again.

CI / go build
      |
   Athens (internal proxy, with cache)
      | (only on first fetch)
   GitHub / VCS

Explanation

Go has three environment variables that control how modules are fetched:

GOPROXY fallback chain

GOPROXY supports comma-separated sources – Go tries them in order:

GOPROXY=https://internal-proxy.company.com,direct

direct is not another proxy – it means fetching directly from VCS (GitHub). If the internal proxy doesn’t have a particular public module, it falls back to direct.

Knowledge Sugar

Why can’t private modules be sent to sum.golang.org? sum.golang.org is a public service. By default, Go sends the module path there for checksum verification. If a private module’s path gets sent, it leaks internal repo information. Use GONOSUMDB or GOPRIVATE to exclude them.

GOPRIVATE vs setting GOPROXY + GONOSUMDB separately GOPRIVATE=company.com/* is equivalent to setting both GONOSUMDB=company.com/* and GONOPROXY=company.com/* (skip proxy, go straight to VCS). If you’re using an internal Athens proxy, you usually still need to explicitly set GOPROXY, because GOPRIVATE tells Go “don’t use a proxy” – which would bypass your Athens instance and hit GitHub directly.

How to pass a GitHub token in Docker builds? If your Dockerfile runs go build and needs to fetch private modules, don’t use ARG for the token – ARG values end up in the build history and can be recovered from image layers even after the file is deleted.

The correct approach is Docker BuildKit’s secret mount:

1RUN --mount=type=secret,id=github_token \
2    GITHUB_TOKEN=$(cat /run/secrets/github_token) \
3    go build ./...

The secret is only visible during this RUN step and is never written to any image layer.

An application deployed on GKE can use OIDC federation to exchange a Kubernetes-issued JWT token for temporary AWS credentials, eliminating the need to hardcode AWS Access Keys.

Explanation

What OIDC does in this scenario

OIDC (OpenID Connect) is essentially a protocol for “let me prove this identity is real.” In this use case, GKE acts as the OIDC Identity Provider (IdP), and AWS is the party that trusts what GKE says.

The overall flow looks like this:

GKE Pod
  │
  │ 1. Kubernetes automatically mounts a Service Account JWT token on the Pod
  │    (this token contains the workload's identity info, signed by GKE)
  │
  ▼
AWS STS (AssumeRoleWithWebIdentity)
  │
  │ 2. Submit the JWT token to AWS STS
  │    AWS validates the token against GKE's OIDC endpoint
  │
  ▼
Temporary AWS credentials (Access Key + Secret + Session Token)
  │
  │ 3. Use temporary credentials to operate on S3
  │
  ▼
S3 Bucket ✓

sequenceDiagram
    participant Pod as GKE Pod
    participant STS as AWS STS
    participant OIDC as GKE OIDC
    participant S3

    Note over Pod: K8s mounts JWT token
    Pod->>STS: AssumeRoleWithWebIdentity(JWT)
    STS->>OIDC: Verify signature
    OIDC-->>STS: OK
    STS-->>Pod: Temporary credentials (with TTL)
    Pod->>S3: Upload CSV
    S3-->>Pod: 200 OK

What needs to be configured in Terraform

1. AWS IAM OIDC Provider – tells AWS “I trust tokens issued by this GKE cluster”; requires the GKE OIDC issuer URL and thumbprint
2. AWS IAM Role + Trust Policy – defines which GKE service account can assume this role; the condition typically requires the sub claim to match system:serviceaccount:<namespace>:<ksa-name>
3. S3 Permission – attach the S3 read/write policy to this IAM Role

Go STS package upgrade

In the Go application, use AssumeRoleWithWebIdentity, passing in the JWT token path read from the Pod (typically /var/run/secrets/kubernetes.io/serviceaccount/token). After exchanging for temporary credentials, initialize the S3 client. The package upgrade was mainly to ensure compatibility with SDK v2’s credential provider interface.

Knowledge Sugar

Why not just use AWS Access Keys? Hardcoding keys has rotation issues, leakage risks, and makes it hard to audit which workload is accessing what. OIDC federation’s temporary credentials have a TTL and expire automatically, making them significantly more secure.

What’s inside the JWT token? The token issued by GKE is a standard JWT. The payload contains:

• iss: OIDC issuer (the GKE cluster’s URL)
• sub: system:serviceaccount:<namespace>:<name>
• exp: expiration time

When AWS STS receives the token, it fetches the public key from the iss URL’s /.well-known/openid-configuration endpoint to verify the signature.

How is the JWT token mounted into the Pod? This is a built-in Kubernetes mechanism – every Pod automatically gets a Service Account token mounted at creation, no extra config needed. However, the default token doesn’t specify an audience, which AWS STS won’t accept. So you need to mount a dedicated one using a projected volume in the Pod spec:

1volumes:
2  - name: aws-token
3    projected:
4      sources:
5        - serviceAccountToken:
6            audience: sts.amazonaws.com # here!
7            expirationSeconds: 3600
8            path: token

Terraform only handles the AWS side (OIDC Provider, IAM Role); the token mounting itself is done by Kubernetes.

The official name for this pattern: Workload Identity Federation GCP’s own Workload Identity uses the same principle, just with GCP resources. For cross-cloud scenarios (GKE -> AWS), OIDC serves as the universal standard that bridges them.

An optional external call should not share a cancellable context – if that call errors and triggers cancel, the entire context is cancelled, affecting operations that shouldn’t be impacted.

Explanation

Scenario from a PR review:

 1// ❌ Problematic version
 2func (s *Service) GetCampaign(ctx context.Context) (*Campaign, error) {
 3    ctx, cancel := context.WithCancel(ctx)
 4    defer cancel()
 5
 6    // campaign components are optional
 7    components, err := s.client.FetchComponents(ctx)
 8    if err != nil {
 9        return nil, err  // defer cancel() fires here, ctx is cancelled
10    }
11
12    // all subsequent operations using the same ctx are affected
13    result, err := s.doSomethingElse(ctx)
14    ...
15}

The problem: FetchComponents fails -> returns error -> defer cancel() executes -> this ctx is cancelled. If this ctx is shared elsewhere, everything blows up together.

Since campaign components are optional, the correct approach is to absorb the error from this call and not propagate it upward:

 1// ✅ Correct version: optional call errors don't affect the main flow
 2func (s *Service) GetCampaign(ctx context.Context) (*Campaign, error) {
 3    // don't create a cancel context here
 4
 5    components, err := s.client.FetchComponents(ctx)
 6    if err != nil {
 7        // optional -- log and continue, don't return error
 8        log.Warn("failed to fetch optional components", "err", err)
 9        components = nil
10    }
11
12    result, err := s.doSomethingElse(ctx)
13    ...
14}

Knowledge Sugar

Context cancellation propagation direction

Cancellation only propagates from parent to child, never in reverse. However, cancel() directly cancels the ctx you created – if other code also holds a reference to this ctx (or child contexts derived from it), they all get cancelled together.

If you need to fully isolate an optional call (unaffected by parent cancellation)

Go 1.21+ has context.WithoutCancel():

1// create a ctx that doesn't inherit the parent's cancel signal
2detachedCtx := context.WithoutCancel(ctx)
3components, err := s.client.FetchComponents(detachedCtx)

This way, even if the parent request is cancelled (e.g. client disconnects), this optional call still runs to completion.

When to use which approach:

• Languages: Go, PHP, Python
• Infrastructure: GCP, Terraform, Kubernetes
• Domain: Logistics / E-commerce

I’m currently based in Japan and open to backend engineering opportunities in New Zealand.

This blog collects my TIL notes – small discoveries worth remembering, and occasional longer technical posts.

When I am AFK, you can find me either bouldering or skiing or observing (real) bugs in the mountains.

What I’m trying to do more

• keeping my dopamine at a stable level
• being wary of structural inequality
• discovering mushrooms in the gap of the capitalist world.

What I’m trying to avoid

• inaccurate definitions
• an overdose of vocabulary and theory
• being satisfied with being a permanent dilettante