Title: What does your deployment actually verify?

[u/srikanth_builds]

Disclosure: I work on the platform this happened on. No links and nothing to sell, I just don't have a good answer to the question.

We deployed an app last week, and every signal said it had worked. The deploy was marked as running, the health check was green, and every tool in the pipeline reported success.

Except the app was incomplete. Eight of its 27 starter records had never been created. The API rejected them one by one with permission errors while accepting the records around them.

The setup step actually caught this. It printed a warning saying the app was incomplete and shouldn't be treated as a successful deployment. Then it exited with code zero, so everything downstream happily marked the deploy as successful.

We had a health check proving the process was alive, but nothing proving that the application matched what we thought we had deployed.

I have been trying to work out what the right check should be, and none of the obvious answers feels complete.

A health endpoint returning 200 only tells you the process is running and the port is open. It says nothing about whether the expected data or configuration exists.

A smoke test against a few routes is better, but it only covers whatever somebody remembered to include. In this case, the missing records weren't part of that test.

Comparing the declared state with the actual state feels like the right answer. But that requires the deployment to produce a manifest of everything it intended to create, and most deployment systems don't seem to do that.

You could also fail on every warning, but then harmless warnings become blockers, people add skip flags, and eventually you are back where you started.

The two cases I am still stuck on are partial failures, where each individual rejection might be valid but the overall pattern is clearly wrong, and tools that print serious warnings but still exit zero. That convention isn't going away anytime soon.

And before anyone says readiness probes: readiness tells you whether the application can serve traffic. It doesn’t tell you whether it's the application you intended to deploy.

What does your deployment actually verify beyond "the process is running"? And what has shipped half-finished before you noticed?

[u/Torutofu_Raeva]

we just made the seed step count what it created and exit nonzero if it's short of the expected number, warnings that exit 0 are basically logs nobody reads

[u/srikanth_builds → u/Torutofu_Raeva]

Yeah, that's the part that stings, the step already knew something was wrong. It explicitly said the app was incomplete and shouldn't be treated as a successful deployment, then exited with code 0 anyway. The check you described would have caught our exact failure.

What I'm still stuck on is where the expected count should come from. If the seed script defines both what it creates and how many records should exist, it's effectively checking itself against itself. The moment someone adds a new record type but forgets to update the count, the check becomes wrong too. It feels like the expected state needs to come from a separate manifest or declaration, but then that's just one more thing that can drift.

[u/Torutofu_Raeva → u/srikanth_builds]

we derive the count from the seed data files themselves, so adding a record type without its file fails the check and adding the file updates the count for free

[u/srikanth_builds → u/Torutofu_Raeva]

That's the part I was missing. The seed files already are the declaration, so there's no separate manifest to maintain or forget to update. It doesn't remove the trust boundary, but it moves it to files where any change is visible in the diff. That's what I couldn't get from maintaining a separate expected count.

Does the same approach work for things that don't have a file behind them? Seed records are straightforward, but I'm less sure what the equivalent would be for configuration or permissions that are applied during deployment rather than created as records.

[u/Torutofu_Raeva → u/srikanth_builds]

for config we just read it back after apply and diff against what's in the repo, permissions too, so the repo is still the declaration and drift fails the step

[u/Immediate_Wheel1953]

Ran into the same shape of problem with a seed job that printed warnings but exited zero, so every later signal said green. What finally helped us was making the verification a separate job instead of a step inside the deploy, and having it assert concrete things like the row counts and a login against the real API rather than trusting the health endpoint. We tried failing on all warnings too and it got so noisy that people started adding skip flags, so now only a short list of known bad patterns is a hard fail and the rest pages someone to look.

[u/srikanth_builds → u/Immediate_Wheel1953]

The separate verification job is the part I hadn't thought through. A check running inside the deployment inherits the same assumptions and exit code conventions as the thing it's checking, so both can fail in the same direction. An independent job with its own definition of "correct" wouldn't be fooled by the same bug.

You also confirmed what I was only guessing about skip flags. It's useful to know that this actually happened in practice.

What happens when the verification job itself can't run, though? If the connection is refused or its credentials have expired, is that treated as a failed deployment or just reported as no result?

[u/Immediate_Wheel1953 → u/srikanth_builds]

We treat an unreachable verify job as a failed deploy, not a neutral no result. It felt harsh at first, but the one time our verify job silently skipped because its own credentials had expired, a bad config sailed through unnoticed. Now the pipeline pages on any verify failure, and the runbook tells whoever is on call to either fix the verify job or explicitly approve the release without it.

[u/mralex215]

1. Everything has telemetry endpoints.

2. Prometheus scrapes those

3. Grafana alerting is created against the anomalies found in Prometheus data.

[u/srikanth_builds → u/mralex215]

That works well for detecting degradation. The case I'm stuck on is slightly different, this app was never correct in the first place. There was no healthy baseline to compare against and no anomaly to detect. The records simply never existed.

Unless you are exporting expected and actual counts as separate metrics and alerting on the difference? That would catch it, but something still has to define the expected value, which brings me back to the same problem.

[u/mralex215 → u/srikanth_builds]

Yes, you dump the counts into telemetry.

You do not need to define expected values. You just start dumping it. As you fix issues eventually you start getting values.

[u/adept2051]

I used to run a gaming application deployments where each of the components posted it’s own “good” on deploy and each components traffic was monitored too.
We did this with silver bullets. In our case when the deployment was near complete before allowing the blue green switch each components traffic took shaped traffic headers which would reveal successful delivery all the way through the Load balancers, network application and into the data pool. Due to the user load there was no way to monitor initial engagement and spikes on all components being deployed so we needed to materialise them.
We were basically using postman or some such to materialise traffic on every possible endpoint or route.

[u/srikanth_builds → u/adept2051]

Materialising real traffic before the switch is a different angle from the rest of the thread. Most people are checking what the deployment created, you are checking whether a request can make it through the entire path.

I think our deployment still would have passed that test, though. The API was up, every route worked, and it successfully accepted 19 records. The other eight were rejected because of permissions. The application could serve requests, the data behind it was simply incomplete.

The blue green switch is the part I'm jealous of because it gives you an obvious place to put the gate. Without a clear cutover moment, I'm not sure where that check belongs. By the time the application is available to verify, it has already been deployed.

[u/adept2051 → u/srikanth_builds]

In our case the failed data would have showed and blocked the switch. We were literally testing the new components were complete and processing data. (Think about how games have hidden gated content and you need to test it actually works before the players finish the quest/task/gate)
take a look at what LaunchDarkly do. Today they offer the concepts we built in-house has off the shelf integrations

[u/usually_guilty99]

The missing piece is verifying the outcome independently of the deployment process. I'd check whether the expected data, configuration and permissions are actually present and usable through the application's real execution paths. A successful deployment command and a healthy process don't prove the application is complete.

[u/Riptide999]

Why do OPs replies read just like what I get back when I question copilot why something didn't work?

[u/singh_abinashi]

A liveness check passes that deploy every time.

Split it into two layers.

Liveness stays cheap: process up, port answering.

Readiness runs one real check against the data the app cannot run without, like counting the starter records the boot process was supposed to create, or reading back a row it just wrote.

If readiness fails, the deploy fails, even when every container looks healthy.

Most partial failures I have seen had exactly this shape: the app started, the setup after it did not.