Health and diagnostics

WorkloadHealth judges seven kinds — Pod, Deployment, StatefulSet, ReplicaSet, DaemonSet, Job, CronJob. Anything else is reported OK rather than guessed at, and the Workloads overview filters the nav category down to kinds a rule actually understands, because a kind with no rule would otherwise be reported as uniformly healthy — a claim rather than a measurement.

Four states rather than a boolean: OK, ATTENTION, SUSPENDED and IDLE. A suspended CronJob is an operator’s choice, and colouring it red beside genuine failures is how a health signal earns its way into being ignored.

ClusterObjectHealth judges the two cluster-scoped kinds on the cluster dashboard — Node and Namespace.

What is deliberately not a state here: the pressure conditions (MemoryPressure, DiskPressure, PIDPressure) — a node under disk pressure is still Ready, kubectl says so, and the Nodes list already has a Conditions column; a Cordoned state of kweblens' own, which would hide whether the node is also ready; and any judgement about what a Namespace contains, which would be a claim invented here about objects that are not in the Namespace object at all.

Events are not in this category, although they are the third kind in the Cluster nav section. An event’s Warning/Normal is its type — a field on a report about another object, not a verdict on the event — so there is no state to count, and the dashboard’s Warnings figure stays a plain count with the warnings table itself directly beneath it. The category is assembled by asking StatusVocabulary.covers which kinds have a verdict, so this falls out of the shared rule rather than being a special case.

The namespace filter is not passed to this category. Both kinds are cluster-scoped, so narrowing them is not a filter that returns fewer — it is a question the API does not answer, and the dashboard says so beside the cards rather than showing filtered and unfiltered numbers side by side.

One check: Services with nothing answering behind them. The Service exists, DNS resolves, kubectl shows it as normal, and every request to it fails.

It distinguishes the two causes, which have different fixes:

Both come out of one Endpoints list (notReadyAddresses is exactly that signal), so the whole check costs two list calls, not one per Service. ExternalName Services are always fine: they are a DNS CNAME with no endpoints by design.

PersistentVolumeClaims that are not bound. An unbound PVC is a pod that will never start, and the pod’s own status points at the claim rather than the cause — so a Pending claim’s reason names its StorageClass, which is nearly always the question.

Bound claims are also checked for fullness at 90%, but only where the reading is about the claim. See Metrics for why: kubelet reports the filesystem backing the volume, so on a provisioner with no per-volume quota every claim on a class reports the same size. A reading that fails the plausibility check is not used, and the claim is reported as bound rather than flagged on a number that describes a shared disk.

Capacity checks are simply absent when there is no metrics backend. The binding checks stand on their own.

ConfigMaps and Secrets that nothing in the namespace refers to. This is the one question the Kubernetes API cannot answer directly, and no other kweblens screen answers at all: nothing garbage-collects config objects and nothing warns about them.

It is framed as "not referenced", never as "safe to delete", and the wording matters. The scan covers pods, ServiceAccounts and Ingresses in one namespace, so an object referenced from a controller template whose pods do not exist yet, from another namespace, or from a CRD’s own spec will appear here while being very much in use. Findings are amber, because they are advisory.

Two Secret types are excluded outright rather than listed: service-account tokens (the cluster owns them) and Helm release records (helm.sh/release.v1), which look unused because nothing mounts them and are the exact opposite of disposable. Secrets are listed for their names only — the values are never read.

Each line of a card’s breakdown opens that kind’s list filtered to exactly the objects it counted, by way of the status: term. A state with nothing in it stays plain text, because opening an empty list to say "none" is a worse answer than the line.

Three of these verdicts cannot be recomputed from a list row, which is the whole reason the state is computed on the server and travels with the object: a Service’s needs the Endpoints collection, a claim’s fullness needs the metrics backend, and a ConfigMap’s needs a scan of the namespace. So a list request for one of those kinds opens a status context first — StatusContexts — which fetches that second collection once per request, before any row is looked at, and every row is then a lookup. It is the same fetch the card already makes, over the same scope:

Fetching per row instead would turn one list request into N, which is the bound #293 and #302 were spent establishing. If the second collection cannot be fetched at all, the rows come back with no state rather than the list failing: nothing is claimed, and no status: term selects them.

The live watch behind an open list re-opens its context at most once every 30 seconds. The inputs to these three verdicts are objects the list is not watching, so no event on the watched kind can report that they changed — a context opened once when the page loaded would still be asserting this morning’s Endpoints tonight, and re-opening per event would turn a busy list into a second stream of list calls.

Every check reports an error for a kind it could not list, instead of a zero. On a health screen those two must never look the same. The config check goes further: if the reference scan itself fails, both ConfigMaps and Secrets are reported as failed, because "0 unused" would read as a clean result.

This is the rule, and it is not a performance tweak:

The cache key is the findings, not the scope. A SHA-256 fingerprint is computed over every field of every finding, in order, with unambiguous separators. The deterministic findings are the model’s entire input, so a hit means the model would be shown exactly what it was shown before — and a cluster that has changed is a miss rather than a stale verdict. Cluster and namespace alone would serve yesterday’s opinion about a cluster that has since been rolled.

The summary endpoint is a POST because it is write-shaped, not because it writes: it spends money and ships cluster state to a third-party inference provider. That puts it behind the admin login in both security modes, and it is audited. Panel mounts and namespace switches used to buy an analysis each; now they cost nothing.

The read tells you what it is showing rather than letting an old reading pass as current:

The cache is in-memory and per-process, capped at 64 scopes with least-recently-read eviction. A restart empties it and a second replica has its own. That costs one more inference call, never a wrong answer. There is deliberately no endpoint to clear it: a summary is already discarded the moment the findings change, which is the only invalidation that means anything.

The LLM path requires kweblens.ai.enabled=true and a configured Anthropic API key. Without both, analyse() returns a 409 (ai-unavailable) — nothing broke and nothing was spent, and the caller could already tell in advance from aiAvailable on the read.

When it does run, the model is given the findings grouped by distinct problem, not one line per object, with exact per-group counts, up to 15 named objects per group and up to 4 distinct evidence messages. It is capped at 25 distinct problem groups, and when the cap bites the truncation is stated in the answer rather than happening silently. The model is asked for judgement only — one next action and at most three bullets, 90 words — because the reader can already see a table of every finding directly below it.

The model never invents state. It reasons over observed evidence, and if the call fails the deterministic findings are returned alone.

The preview is where "suggest → preview → confirm" is easy to overstate, so kweblens returns one of three distinct outcomes rather than flattening them into a string:

Which action gets which:

Saying "we did not check" is the design, not an omission. A hand-written sentence prefixed dry-run: describes what kweblens intends, which is a different claim from what the cluster would allow — and the difference is precisely what the operator is being asked to approve.

A new remediation action must pick one of those two shapes. Never narrate a third.