Part 1: Service connectivity — get endpoints splits the problem in two​

When you can't reach a Service, the single most useful first command is not describe pod. It's:

kubectl get endpoints <svc>

Endpoints is the table that says "which Pod IPs does this Service actually forward to." That one command cleaves every connectivity bug into two halves: can't find the Pods (endpoints empty) vs found them but can't reach them (endpoints populated). Each half points somewhere completely different.

Failure 1: selector doesn't match the Pod labels → endpoints empty​

kubectl expose deployment app --port=80 --target-port=80 --name=app-bad-selector
kubectl patch svc app-bad-selector -p '{"spec":{"selector":{"app":"WRONG"}}}'

• Symptom: can't connect; the Pods are all Running.
• Investigation: kubectl get endpoints app-bad-selector → empty (<none>). Then kubectl get svc app-bad-selector -o wide to read the Selector, and compare against kubectl get pods --show-labels.
• Cause: the Service selector matches no Pod labels, so the Endpoints controller can't assemble a backend list.
• Fix: correct the selector (or the Pod labels) so they line up.
• Concept: a Service doesn't know about Pods directly — it only knows labels. The "which Pod IPs" lookup lives in the Endpoints / EndpointSlice object. Wrong selector → that table is empty. Empty endpoints almost always means a selector problem.

Failure 2: targetPort points at a port nothing is listening on → connection refused​

kubectl expose deployment app --port=80 --target-port=8080 --name=app-bad-port

• Symptom: still can't connect — but it breaks differently than Failure 1.
• Investigation: kubectl get endpoints app-bad-port → this time there are IPs (the selector matches, so endpoints list podIP:8080). Then curl it and watch how it fails → connection refused, immediately, not a hang.
• Cause: the Service port is fine, but targetPort is 8080 while the container actually listens on 80. kube-proxy DNATs the packet to podIP:8080, nothing is listening there, and the kernel replies with a TCP RST.
• Fix: set targetPort to the port the container really listens on.
• Concept: endpoints populated + connection refused → it's a targetPort / wrong-container-port problem. "Refused" means the packet reached the Pod and got rejected — so it's not a routing problem, it's knocking on the wrong door.

Failure 3: NetworkPolicy silently drops the packet → timeout​

kubectl get netpol -A
kubectl describe netpol <name> -n <ns>

• Symptom: endpoints are populated, targetPort is correct, and curl times out (hangs with no response) — not "refused".
• Investigation: kubectl get netpol -A to see if any policy selects the source or destination Pod; describe to read its PodSelector / Ingress / Egress.
• Cause: a NetworkPolicy is dropping this traffic. Dropped packets get no reply (no RST), which is exactly why you see a timeout instead of a refusal.
• Fix: allow the source→destination flow in the relevant policy, or adjust its PodSelector.

Three things about NetworkPolicy that trip people up:

1. Whitelist flip. A Pod with no policy selecting it is "allow all." The moment any policy selects it — even one that only lists Ingress — that direction flips to "deny all except what's explicitly allowed." A common gotcha is a podSelector: {} default-deny someone added.
2. Two directions. Either the destination Pod's Ingress doesn't allow the source, or the source Pod's Egress doesn't allow the destination. Either one blocks the flow, so check policies in both namespaces.
3. The CNI has to enforce it. This is the big one for local labs:

Service decision tree​

Can't reach a Service → ① kubectl get endpoints <svc>
├─ endpoints empty ─────────────► selector doesn't match labels → fix selector
└─ endpoints populated → curl and watch HOW it fails
     ├─ connection refused (instant) ─► wrong targetPort / nothing listening
     └─ timeout (hangs) ─────────────► suspect NetworkPolicy dropping packets

The cheapest signal in the whole tree is refused vs timeout: refused means the packet arrived and was rejected; timeout means it was swallowed on the way.

Part 2: PVC stuck in Pending — read the Events, not the logs​

A PVC that never leaves Pending drags its Pod down with it (the Pod stays Pending too, it can't even schedule). There are no logs to read here — the answer is in kubectl describe pvc.

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: stuck-pvc
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: nonexistent-sc
  resources:
    requests:
      storage: 1Gi
EOF

• Symptom: kubectl get pvc stuck-pvc sits at Pending, never Bound.
• Investigation: kubectl describe pvc stuck-pvc (read the Events), then kubectl get storageclass to see what actually exists.
• Cause (this example): the requested storageClassName: nonexistent-sc doesn't exist, so no provisioner claims the PVC and it binds to nothing.
• Fix: use a StorageClass that exists (kubectl get sc — there's usually one marked (default)), or omit storageClassName to take the default.

The root causes worth memorizing as a checklist:

• StorageClass doesn't exist — Events say the class can't be found (this example). Fix the name.
• Static provisioning, no matching PV — with no dynamic provisioner you must pre-create a PV; if none has capacity ≥ the request with a matching accessMode and storageClassName, the PVC stays Pending. Create a matching PV, or switch to a class that supports dynamic provisioning.
• volumeBindingMode: WaitForFirstConsumer — this Pending is normal. The class deliberately waits until a Pod actually uses the PVC before binding, so the volume lands in the right node/zone. Not an error; create a Pod that mounts it and binding happens.

The one table to keep​

The thread running through both halves: when a Pod is healthy but unreachable or unschedulable, stop staring at the Pod. Walk the chain it depends on — Endpoints for networking, StorageClass and PV for storage — and let how it fails (empty vs refused vs timeout vs Pending) tell you which link to fix.

The realization: build-time vs runtime​

The stats bar already had four numbers, and they were all computed at build time by reading the repo:

• posts / tags — counted from blog frontmatter while Docusaurus builds.
• last post — diff between today and the newest post date.

Those are static: bake them into the HTML and they're correct until the next build. A visitor count is the opposite — it only exists at runtime, on a server that's recording traffic. The site itself records nothing. So any "visitors" or "online" number needs an external analytics source queried from the browser. That single distinction drove the whole design.

Why you can't call GA Realtime from the browser​

GA4's realtime data comes from the Analytics Data API (properties/{id}:runRealtimeReport, metric activeUsers). To call it you must authenticate as a service account — which means holding its private key and signing a JWT to exchange for an access token.

A static site's JavaScript is fully visible to anyone. Put that key in the bundle and you've handed out read access to your analytics. So the only safe shape is a tiny server in the middle:

browser (stats bar)  ──►  Cloudflare Worker  ──►  GA4 Realtime API
                          (holds the key,
                           caches ~60s,
                           returns { "active": N })

The Worker is the only thing that ever sees the key. The browser only ever talks to the Worker, which returns a harmless { "active": N } with permissive CORS.

Step 1 — page tracking (the easy, SEO part)​

This is the part most people mean by "add Google Analytics," and it's trivial. Docusaurus' classic preset has a built-in gtag plugin that also handles client-side route changes:

// docusaurus.config.ts
const GA_MEASUREMENT_ID = 'G-XXXXXXXXXX';

presets: [
  ['classic', {
    // ...
    ...(GA_MEASUREMENT_ID
      ? {gtag: {trackingID: GA_MEASUREMENT_ID, anonymizeIP: true}}
      : {}),
  }],
],

This alone gives you SEO/traffic analytics. It does not give you the realtime number — that's Steps 2–4.

Step 2 — service account + Data API​

1. In Google Cloud Console, enable the Google Analytics Data API.
2. Create a service account, then create and download a JSON key.
3. In GA Admin → Property access management, add the service account's client_email as a Viewer on the property. (Skipping this is Pitfall 3 below.)

Step 3 — the Worker: signing a Google JWT with WebCrypto​

Cloudflare Workers have no googleapis SDK, but they do have WebCrypto, so you build and sign the JWT yourself. The flow: make a JWT claiming the service-account identity, sign it RS256 with the private key, exchange it for an access token, then call the Realtime report. The interesting bits:

async function getAccessToken(env) {
  const now = Math.floor(Date.now() / 1000);
  const header = {alg: 'RS256', typ: 'JWT'};
  const claim = {
    iss: env.GA_CLIENT_EMAIL,
    scope: 'https://www.googleapis.com/auth/analytics.readonly',
    aud: 'https://oauth2.googleapis.com/token',
    iat: now, exp: now + 3600,
  };
  const unsigned = `${b64url(JSON.stringify(header))}.${b64url(JSON.stringify(claim))}`;

  const key = await importPrivateKey(env.GA_PRIVATE_KEY); // PKCS8 PEM -> CryptoKey
  const sig = await crypto.subtle.sign(
    {name: 'RSASSA-PKCS1-v1_5'}, key,
    new TextEncoder().encode(unsigned),
  );
  const jwt = `${unsigned}.${b64urlBytes(new Uint8Array(sig))}`;

  const res = await fetch('https://oauth2.googleapis.com/token', {
    method: 'POST',
    headers: {'content-type': 'application/x-www-form-urlencoded'},
    body: new URLSearchParams({
      grant_type: 'urn:ietf:params:oauth:grant-type:jwt-bearer',
      assertion: jwt,
    }),
  });
  return (await res.json()).access_token;
}

Two details that bite:

• The private key from the JSON is PKCS8 PEM. Strip the -----BEGIN/END----- lines and whitespace, base64-decode to bytes, then crypto.subtle.importKey('pkcs8', ...). Also normalize \n → real newlines, because secrets are often stored with escaped newlines.
• Cache both the access token (~55 min) and the active-users result (~60 s) in module scope. Otherwise every page load hits Google and you'll burn through quota.

The actual report call is small:

const res = await fetch(
  `https://analyticsdata.googleapis.com/v1beta/properties/${env.GA_PROPERTY_ID}:runRealtimeReport`,
  {method: 'POST',
   headers: {authorization: `Bearer ${token}`, 'content-type': 'application/json'},
   body: JSON.stringify({metrics: [{name: 'activeUsers'}]})},
);
const data = await res.json();
const active = Number(data?.rows?.[0]?.metricValues?.[0]?.value ?? 0);

Secrets are set with Wrangler (note the names must match env.X in the code):

npx wrangler secret put GA_CLIENT_EMAIL   # paste the client_email
npx wrangler secret put GA_PRIVATE_KEY    # paste the private_key value
npx wrangler deploy

GA_PROPERTY_ID and ALLOWED_ORIGIN are non-secret, so they live in wrangler.toml.

Step 4 — wire the frontend (poll every 60s)​

The homepage reads the Worker URL from config and polls it after hydration. During the static build there's no fetch, so it renders a placeholder, then fills in on the client:

function useOnline(apiUrl: string): string {
  const [value, setValue] = useState('···');
  useEffect(() => {
    if (!apiUrl) { setValue('—'); return; }
    let alive = true;
    const load = () =>
      fetch(apiUrl)
        .then((r) => (r.ok ? r.json() : Promise.reject()))
        .then((d) => alive && setValue(String(d.active ?? '—')))
        .catch(() => alive && setValue('—'));
    load();
    const id = setInterval(load, 60_000);
    return () => { alive = false; clearInterval(id); };
  }, [apiUrl]);
  return value;
}

A failed/unconfigured fetch falls back to —, so the page never shows a broken stat.

Step 5 — serve it from your own domain (optional, but cleaner)​

The Worker deploys to a *.workers.dev URL by default. If your domain's DNS is already managed by Cloudflare, you can serve it from your own subdomain instead — nicer to read, not tied to the workers.dev subdomain, and less likely to be tripped up by privacy extensions or networks that block *.workers.dev.

Add a custom-domain route to wrangler.toml and redeploy:

[[routes]]
pattern = "ga4-proxy.forklore.dev"
custom_domain = true

npx wrangler deploy

wrangler deploy provisions the DNS record and the TLS certificate automatically (the cert takes a minute or two). Then point the frontend at the new URL:

const ONLINE_API_URL = 'https://ga4-proxy.forklore.dev';

curl https://ga4-proxy.forklore.dev   # {"active":0}

Folding in the visitor count (and dropping GoatCounter)​

The realtime "online" number is only half the stats bar — I also wanted an all-time unique visitors count. My first version used GoatCounter: cookieless, zero-config, a one-line script plus a public /counter/TOTAL.json endpoint the page could fetch directly — no proxy needed. It worked.

Then I deleted it. The reasoning:

• GA was already in the page for SEO/analytics, and the Worker already existed for the realtime number. GoatCounter's headline advantage is being cookieless / privacy-first — but that edge evaporates the moment you're already running GA. I wasn't actually buying the one thing GoatCounter is best at.
• Keeping it meant two analytics sources to reconcile and a second third-party script on every page, for a number GA can already give me.

So I consolidated onto the Worker. GA's all-time count comes from a standard report (not realtime), so the Worker makes a second call and returns both numbers:

// alongside the realtime activeUsers call, a standard report for all-time users:
const data = await gaReport(env, token, 'runReport', {
  dateRanges: [{startDate: '2020-01-01', endDate: 'today'}],
  metrics: [{name: 'totalUsers'}],
});
const total = Number(data?.rows?.[0]?.metricValues?.[0]?.value ?? 0);
// the Worker response becomes: { active, total }

The frontend now reads active (online) and total (visitors) from a single fetch, and the GoatCounter script and account are gone.

• Concept: don't run two tools for one job unless each earns its place. GoatCounter would have earned it if I weren't already on GA — cookieless analytics is a real, distinct value. Here it was redundant, and a redundant dependency is pure maintenance cost. The cheapest integration is the one you don't keep.

The pitfalls (the part that actually cost time)​

Pitfall 1: npm audit fix --force wrecked the toolchain​

npm install for Wrangler succeeded, but it reported a few "vulnerabilities," so I reflexively ran npm audit fix --force. That made a major-version change and ended up downgrading Wrangler to a very old release, which dragged in an old better-sqlite3 that has no prebuilt binary for Node 24 — so it tried to compile from C++ source and exploded with C++20 or later required.

• Cause: --force is allowed to make breaking version changes to silence advisories.
• Fix: rm -rf node_modules package-lock.json, pin a current wrangler (^4), reinstall, and don't run audit fix here.
• Concept: an npm audit advisory describes a risk for some usage of a package. The ones here were in esbuild's dev server — code that never ships to the deployed Worker. The advisory being real doesn't mean my path is exposed. --force "fixing" a non-applicable advisory did far more damage than the advisory ever could.

Pitfall 2: wrangler secret put takes the NAME, not the value​

I ran wrangler secret put blog-ga-data-api@...iam.gserviceaccount.com, pasted a value at the prompt, and "succeeded" — having created a secret whose name was the email.

• Cause: the argument to secret put is the secret's key; the value is read from the interactive prompt afterward.
• Fix: wrangler secret delete "<the wrong name>", then wrangler secret put GA_CLIENT_EMAIL and paste the email at the prompt.
• Concept: the secret name must exactly match what the code reads (env.GA_CLIENT_EMAIL). A secret with the right value but the wrong name is invisible to the Worker.

Pitfall 3: GA 403 PERMISSION_DENIED​

First real request to the Worker returned:

{"active": null,
 "error": "GA API 403: ...User does not have sufficient permissions for this property..."}

• Cause: the JWT, token exchange, and the API call all worked — the service account simply hadn't been granted access to the GA property yet.
• Fix: GA Admin → Property access management → add the service account's client_email as Viewer, wait a minute, retry.
• Concept: a 403 here is reassuring — it means auth succeeded and you reached the property; it's purely an authorization (access-grant) gap, not a broken key. Read the error status: PERMISSION_DENIED (no access) is a different problem from UNAUTHENTICATED (bad token) or SERVICE_DISABLED (API not enabled).

Verify​

Curl the Worker (or just open it). Once the service account has access you should get:

curl https://<your-worker>.workers.dev
# {"active":0,"total":0}

0 is correct when nobody's currently on the site — null would mean something upstream is still wrong. A real number proves the whole chain.

Summary​

The mental model worth keeping: static numbers get baked at build; live numbers need a runtime source, and any source that requires a private key needs a proxy in front of it. The proxy is small, but it's the whole reason the secret never leaks.

Pitfall 1: command not found​

kubectl run crash-cmd --image=busybox --restart=Never -- /bin/sh -c "notacommand"

• Symptom: under restart=Never the STATUS stops at Error with RESTARTS at 0; switch to the default Always and you'll see CrashLoopBackOff with RESTARTS incrementing.
• Investigation: kubectl logs crash-cmd --previous → shows notacommand: not found.
• Cause: the entrypoint command itself doesn't exist, so the container exits the moment it starts.
• Fix: use a valid command.
• Concept: CrashLoop means "keeps restarting," not "can't be scheduled." The container did start — it just dies immediately — so you can see the application-level error in the logs.

Pitfall 2: missing env​

kubectl run crash-env --image=busybox --restart=Never -- /bin/sh -c 'test -n "$REQUIRED" || (echo "REQUIRED missing"; exit 2)'

• Symptom: fails the same way (Error under Never, CrashLoopBackOff under Always).
• Investigation: kubectl logs crash-env --previous → REQUIRED missing, exit code 2.
• Cause: the program self-checks for a required environment variable at startup and exits on its own when it's missing.
• Fix: supply the env var (--env=REQUIRED=x, or inject it via the Deployment's env: / a ConfigMap / a Secret).
• Concept: any non-zero exit code is treated as a failure and triggers a restart. For this kind of "the app decides to exit itself" error, the logs usually explain why — checking the logs first is the easiest path.

Pitfall 3: OOMKilled​

# set an extremely small memory limit, then run something that eats memory
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata: { name: crash-oom }
spec:
  restartPolicy: Never
  containers:
  - name: stress
    image: polinux/stress
    resources: { limits: { memory: "20Mi" } }
    command: ["stress"]
    args: ["--vm","1","--vm-bytes","100M","--vm-hang","1"]
EOF

• Symptom: STATUS shows OOMKilled, but kubectl logs --previous reveals nothing (it may even be empty).
• Investigation: kubectl describe pod crash-oom → look at Last State: Terminated, Reason: OOMKilled, exit code 137.
• Cause: the container's usage exceeded the memory limit and the kernel OOM killer killed it — the app didn't exit on its own, it was killed by an external force.
• Fix: raise resources.limits.memory, or fix the real memory leak / reduce usage.
• Concept: OOMKilled never makes it into the logs, because logs are the "container's stdout/stderr," whereas OOM is the kernel sending SIGKILL directly — the app never gets a chance to print anything. You have to look at Last State via describe. Exit code 137 = 128 + 9 (SIGKILL).

Debug Summary​

What I need: let the requirements pick the tool, not the features pick me​

I broke the requirements into four questions to decide between Kustomize and Helm:

• Do I need logic?
  • The moment a manifest needs a conditional or a loop, Kustomize can't do it — it has no if / range.
• Do I need a lifecycle?
  • install / upgrade / rollback, with release state stored in a namespace Secret in the cluster — this is the core of Helm.
  • Kustomize has no concept of a release; rolling back means git revert and re-apply.
• Do I need dependencies?
  • My existing redis and postgresql are both charts; integrating them as subcharts is directly supported by Helm.
• Do I need hooks?
  • Running a Job pre-install is something Helm has hooks for; Kustomize doesn't.

If my need were just "apply the same manifest to three environments, tweaking replicas and the image," I'd pick Kustomize — that's where it's at its cleanest.

The exercise: the chart I wrote myself​

I practiced by writing a chart that actually runs. The point isn't the few helm create commands (anyone can scaffold...).
The real learning is reshaping that "does everything" default chart into your own. I added:

• a ConfigMap
• a pre-install hook Job
• a values.schema.json to block bad input.

helm create gives you an all-capable skeleton that isn't yours; turning it into your own — the whole process is really about deciding what to throw away, what to add, and what to keep.

Repo: https://github.com/shoudevops/helm-practice

Removing a feature ≠ deleting a file: the tentacles of autoscaling​

The most eye-opening practice came from "deleting a feature."
I wanted to remove autoscaling; my instinct was to just delete hpa.yaml.
I thought I was done, but rendering blew up — that line in deployment.yaml, {{- if not .Values.autoscaling.enabled }}, was still referencing a value I thought no longer existed, straight to a nil pointer.
It turned out this feature's tentacles weren't only in hpa.yaml: values.yaml had its settings, NOTES.txt mentioned it, and the deployment's replicas logic was hooked into it too.
I chased them one by one, and only after I got to that line in the deployment was it truly clean.

What I learned from the process: deleting a feature means chasing every one of its references in values / other templates / NOTES; deleting the file is only step one.
Incidentally, I also stepped on two landmines along the way:

• never let secrets into git
• --reset-values wipes out the custom values you've accumulated.

When I'd switch to Kustomize​

This post isn't saying Helm is some cure-all.
For third-party apps I'll install with Helm;
but for purely environment-to-environment differences, I'll use Kustomize overlays — or even render with helm template first and finish with a Kustomize patch.

The criterion is still the same one:

• once I need conditionals or rollback, Kustomize is out; if it's just changing a few values, I don't need Helm.

Wrap-up​

Back to the question at the very start: why not Kustomize?
Because tool-selection skill was never about reciting which tool has which features — it's about honestly asking yourself "what do I need" first, then letting the requirements pick the tool.
Next I want to pick up secret management — SOPS or External Secrets Operator — which is yet another "let the requirements pick the tool" topic.

Instead, from the helm plugin install moment onward, I hit 4 pitfalls in a row — and the last one let me actually witness how "the diff shows no change, but the real deployment changes things" happens. Here's the whole process, written down.

Pitfall 1: the plugin won't install — you need --verify=false​

I got stuck on the very first step:

helm plugin install https://github.com/databus23/helm-diff
# Error: plugin source does not support verification. Use --verify=false to skip verification

The reason is that Helm v3.18+ requires GPG provenance signature verification on plugins by default, but most community plugins (helm-diff included) don't ship a signature file with their releases, so verification is guaranteed to fail. Adding the flag as the message suggests fixes it:

helm plugin install https://github.com/databus23/helm-diff --verify=false

What --verify does is supply-chain protection — confirming the plugin you downloaded hasn't been tampered with and really comes from the author. On a local lab, with the source being a well-known official repo, --verify=false is reasonable. But the right instinct to build is this: when production / CI installs plugins automatically, "skip verification" shouldn't be a casual habit. A safer approach is to clone first, review the contents, pin a specific tag/commit, then install from the local path. Plenty of supply-chain attacks start exactly with "it's just a tool, whatever."

Pitfall 2: the schema blocks the image tag​

After installing, the first diff spat out this:

Error: at '/image/tag': got number, want string

The day before, my practice had added a values.schema.json to the chart, which requires image.tag to be a string. The problem was in values.yaml:

image:
  tag: 1.27

Without quotes, YAML parses 1.27 as a number, and the schema blocks it on comparison. Adding quotes makes it pass:

image:
  tag: "1.27"

This is actually a hidden bug that schema validation caught for me: an image tag should always be quoted. If values like 1.20 or 1.0 get treated as numbers, 1.20 gets truncated to 1.2 and 1.0 becomes 1, and you pull the wrong image on deploy. One innocent-looking quote prevents production pulling the wrong version.

Pitfall 3: helm diff produces no output at all​

Once it passed the schema, the diff output stumped me — sometimes it printed -/+, sometimes it was completely blank. After sorting it out, two key points went into my notes:

- is the state of the existing release in the cluster (live), + is the state it will become after applying. It's not "file vs file," it's "current vs after-apply."

Value precedence (highest to lowest): --set > -f/--values files > the chart's values.yaml > subchart values. At one point I kept editing values.yaml and the diff didn't respond — it was being overridden by a --set on the command line. This is the most common "I changed it but nothing happened" trap.

The most useful investigation technique is to manually pull apart what helm-diff does under the hood:

# live: the manifest actually stored for the release on the cluster
helm get manifest demo -n demo | grep -E "^kind:|replicas:"

# new: the result of re-rendering with values.yaml
helm template demo ~/helm-lab/myfirst-chart | grep -E "^kind:|replicas:"

helm get manifest is the "current state," helm template is "recomputed from files" — compare the two sides and you know which object differs. Once I understood this layer, helm-diff stopped being a black box to me.

Pitfall 4: the diff shows "no change," but helm upgrade actually changes things​

This is the most important one today, and the one most worth writing into my notes.

The situation was this: values.yaml had replicaCount: 1, but the release on the cluster had been deployed and stored with replicaCount: 2. I ran:

helm diff upgrade demo ~/helm-lab/myfirst-chart -n demo
# (no output at all)

The diff showed "no change." But when I manually compared live and template, the Deployment's replicas were clearly 2 on one side and 1 on the other — there shouldn't be no difference, right?

Adding --reset-values and running it again, the truth came out:

helm diff upgrade demo ~/helm-lab/myfirst-chart -n demo --reset-values

demo, demo-myfirst-chart, Deployment (apps) has changed:
  spec:
-   replicas: 2
+   replicas: 1

The root of the problem is that the two commands have inconsistent default behavior:

• helm diff upgrade by default reuses the old values stored in the release (reuse), so it computes with replicaCount: 2, and comparing against live shows "no change," naturally.
• But Helm 3's helm upgrade defaults to reset — it uses the chart's values.yaml (i.e. 1) plus whatever overrides you give.

In other words: the diff lied to me. It said "no change," but if I'd actually run helm upgrade demo ., replicas would change from 2 to 1. The diff not matching the real deployment is exactly where real incidents come from — you see no change in the diff, merge with confidence, and the actual apply moves resources.

From this comes an iron rule: when you use helm-diff in CI, its value-resolution mode must match the command you actually deploy with. Three related flags to remember:

• --reuse-values: reuse the old release's values, only merging in this run's overrides
• --reset-values: discard the old values, use the chart's values.yaml entirely
• --reset-then-reuse-values (v3.14+): reset first, then layer the old overrides back on — usually the safest

One incidental finding: hooks (like the pre-install Job I added on Day 7, or the helm create default test-connection Pod) don't show up in the diff by default, because they aren't part of the persistent desired state, so comparing them is meaningless. So it's normal for helm template to show them while the diff doesn't mention them.

Wrap-up​

Installing a single plugin unexpectedly became the most rewarding lesson of the day. The four pitfalls strung together are really the same theme: a tool's defaults aren't necessarily the behavior you actually want.

• helm plugin install requires signature verification by default → it may not install
• YAML treats 1.27 as a number by default → the schema blocks it
• --set has the highest precedence by default → it overrides the file you edited
• helm diff defaults to reuse, helm upgrade defaults to reset → the diff lies to you

Use DevOps tools long enough and you find that what really bites you usually isn't "not knowing how to use it," but "assuming it does A when it actually defaults to B." In the next post, wrapping up the Helm chart, I'll put together a deep dive on the Helm vs Kustomize trade-off and include the chart I wrote over these two weeks.

1. helm template is not the same as helm install --dry-run​

helm template only does local rendering: it reads Chart.yaml and templates/, substitutes the values from values.yaml into the templates, and prints plain YAML — it never touches the cluster.

helm install --dry-run, on top of rendering, sends the result to the kube-apiserver for schema validation and adds Helm's own metadata. So dry-run can catch problems that template can't see (for example, a field with the wrong type). This is also why ArgoCD uses helm template and then applies the output itself, rather than helm install — it wants predictable plain YAML and doesn't want Helm taking over cluster state.

When paired with ArgoCD, ArgoCD prefers helm template over helm install, because it wants to be the single declarative state manager. It treats Helm as nothing more than a rendering tool, instead of handing control of cluster state over to Helm's release mechanism.

2. Values with a default / quote pipeline​

Reading a value directly with {{ .Values.image.repository }} renders to a blank when the value is missing — an easy trap. A pipeline makes it much safer:

replicas: {{ .Values.replicaCount | default 1 }}
image: {{ .Values.image.repository | default "nginx" | quote }}

A pipeline is evaluated left to right, passing the result down one stage at a time: default "nginx" supplies a fallback when the value is missing, and quote then turns the result into a quoted string. When a value is missing, default saves you from rendering an empty string that would break the whole YAML.

3. Helpers and the two dashes {{- -}}​

_helpers.tpl is where you put reusable snippets (DRY). Helpers are almost always written as {{- define "x" -}} rather than {{ define "x" }}; those leading and trailing dashes (-) trim the surrounding whitespace, including newlines, so the string produced by include stays clean and doesn't wreck your YAML indentation.

By contrast, the _helpers.tpl that helm create generates is very simple, because it only deals with basic objects like deployment / service / ingress. Bitnami's helpers are far more complex because it uses many labels and has many values that need to be referenced repeatedly.

Side note: the MDX traps I hit​

Writing this post in Docusaurus, I ran into two traps:

• HTML comments break the build. <!-- --> is invalid in MDX; you have to use the MDX comment {/* */} instead.
• Bare double braces get treated as JSX. A {{ .Values.xxx }} in the prose, if left unwrapped, makes MDX try to evaluate it as a JavaScript expression and error out — so wrap it in a code fence or inline code (backticks).

Wrap-up​

Getting these three things straight in week one — template vs install, the values pipeline, and helpers with their dashes — is essentially getting familiar with the skeleton of Helm's rendering flow. Next week I'll keep building on top: dependency subcharts, hooks, and then actually installing the chart into a cluster to run a round of install / upgrade / rollback.