helm diff? Installing one plugin, I hit 4 pitfalls in a row

Today's learning task started out simple: install the helm-diff plugin, try helm diff upgrade once, and understand why you'd want it in production.

Instead, from the helm plugin install moment onward, I hit 4 pitfalls in a row — and the last one let me actually witness how "the diff shows no change, but the real deployment changes things" happens. Here's the whole process, written down.

Pitfall 1: the plugin won't install — you need --verify=false

I got stuck on the very first step:

helm plugin install https://github.com/databus23/helm-diff
# Error: plugin source does not support verification. Use --verify=false to skip verification

The reason is that Helm v3.18+ requires GPG provenance signature verification on plugins by default, but most community plugins (helm-diff included) don't ship a signature file with their releases, so verification is guaranteed to fail. Adding the flag as the message suggests fixes it:

helm plugin install https://github.com/databus23/helm-diff --verify=false

What --verify does is supply-chain protection — confirming the plugin you downloaded hasn't been tampered with and really comes from the author. On a local lab, with the source being a well-known official repo, --verify=false is reasonable. But the right instinct to build is this: when production / CI installs plugins automatically, "skip verification" shouldn't be a casual habit. A safer approach is to clone first, review the contents, pin a specific tag/commit, then install from the local path. Plenty of supply-chain attacks start exactly with "it's just a tool, whatever."

Pitfall 2: the schema blocks the image tag

After installing, the first diff spat out this:

Error: at '/image/tag': got number, want string

The day before, my practice had added a values.schema.json to the chart, which requires image.tag to be a string. The problem was in values.yaml:

image:
  tag: 1.27

Without quotes, YAML parses 1.27 as a number, and the schema blocks it on comparison. Adding quotes makes it pass:

image:
  tag: "1.27"

This is actually a hidden bug that schema validation caught for me: an image tag should always be quoted. If values like 1.20 or 1.0 get treated as numbers, 1.20 gets truncated to 1.2 and 1.0 becomes 1, and you pull the wrong image on deploy. One innocent-looking quote prevents production pulling the wrong version.

Pitfall 3: helm diff produces no output at all

Once it passed the schema, the diff output stumped me — sometimes it printed -/+, sometimes it was completely blank. After sorting it out, two key points went into my notes:

- is the state of the existing release in the cluster (live), + is the state it will become after applying. It's not "file vs file," it's "current vs after-apply."

Value precedence (highest to lowest): --set > -f/--values files > the chart's values.yaml > subchart values. At one point I kept editing values.yaml and the diff didn't respond — it was being overridden by a --set on the command line. This is the most common "I changed it but nothing happened" trap.

The most useful investigation technique is to manually pull apart what helm-diff does under the hood:

# live: the manifest actually stored for the release on the cluster
helm get manifest demo -n demo | grep -E "^kind:|replicas:"

# new: the result of re-rendering with values.yaml
helm template demo ~/helm-lab/myfirst-chart | grep -E "^kind:|replicas:"

helm get manifest is the "current state," helm template is "recomputed from files" — compare the two sides and you know which object differs. Once I understood this layer, helm-diff stopped being a black box to me.

Pitfall 4: the diff shows "no change," but helm upgrade actually changes things

This is the most important one today, and the one most worth writing into my notes.

The situation was this: values.yaml had replicaCount: 1, but the release on the cluster had been deployed and stored with replicaCount: 2. I ran:

helm diff upgrade demo ~/helm-lab/myfirst-chart -n demo
# (no output at all)

The diff showed "no change." But when I manually compared live and template, the Deployment's replicas were clearly 2 on one side and 1 on the other — there shouldn't be no difference, right?

Adding --reset-values and running it again, the truth came out:

helm diff upgrade demo ~/helm-lab/myfirst-chart -n demo --reset-values

demo, demo-myfirst-chart, Deployment (apps) has changed:
  spec:
-   replicas: 2
+   replicas: 1

The root of the problem is that the two commands have inconsistent default behavior:

• helm diff upgrade by default reuses the old values stored in the release (reuse), so it computes with replicaCount: 2, and comparing against live shows "no change," naturally.
• But Helm 3's helm upgrade defaults to reset — it uses the chart's values.yaml (i.e. 1) plus whatever overrides you give.

In other words: the diff lied to me. It said "no change," but if I'd actually run helm upgrade demo ., replicas would change from 2 to 1. The diff not matching the real deployment is exactly where real incidents come from — you see no change in the diff, merge with confidence, and the actual apply moves resources.

From this comes an iron rule: when you use helm-diff in CI, its value-resolution mode must match the command you actually deploy with. Three related flags to remember:

• --reuse-values: reuse the old release's values, only merging in this run's overrides
• --reset-values: discard the old values, use the chart's values.yaml entirely
• --reset-then-reuse-values (v3.14+): reset first, then layer the old overrides back on — usually the safest

One incidental finding: hooks (like the pre-install Job I added on Day 7, or the helm create default test-connection Pod) don't show up in the diff by default, because they aren't part of the persistent desired state, so comparing them is meaningless. So it's normal for helm template to show them while the diff doesn't mention them.

Wrap-up

Installing a single plugin unexpectedly became the most rewarding lesson of the day. The four pitfalls strung together are really the same theme: a tool's defaults aren't necessarily the behavior you actually want.

• helm plugin install requires signature verification by default → it may not install
• YAML treats 1.27 as a number by default → the schema blocks it
• --set has the highest precedence by default → it overrides the file you edited
• helm diff defaults to reuse, helm upgrade defaults to reset → the diff lies to you

Use DevOps tools long enough and you find that what really bites you usually isn't "not knowing how to use it," but "assuming it does A when it actually defaults to B." In the next post, wrapping up the Helm chart, I'll put together a deep dive on the Helm vs Kustomize trade-off and include the chart I wrote over these two weeks.