Monitoring Kubernetes

MetricHarbor reads a cluster through its API server — the same endpoints kubectl uses. It needs a read-only account with get and list on nodes and pods, and nothing else. No secrets, no configmaps, no exec, no write verbs anywhere.

Every pod is its own series, labelled by namespace and name. There is no "average pod CPU" in this product, because an average across pods answers no question anybody has.

Which side should the agent run on?#

Two arrangements, and the right one depends on whether you want a long-lived token on disk.

In-cluster is the better arrangement and the reason is the token. Kubernetes projects a ServiceAccount token into the pod and rewrites it roughly every hour; the agent re-reads it on every request, so no token is ever stored anywhere. Outside the cluster there is no kubelet doing that, so you create a token that does not expire and put it in the agent's credential store. That is a real tradeoff, not a formality.

This page covers the outside arrangement, because it needs nothing you do not already have. For in-cluster, see the chart's values.yaml (agent/deploy/helm/metricharbor-agent/); the RBAC below is the same either way.

1. Create the read-only account#

Run this with a kubeconfig that can create cluster roles — your own admin one.

kubectl apply -f - <<'YAML'
apiVersion: v1
kind: Namespace
metadata:
  name: metricharbor
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: mh-collector
  namespace: metricharbor
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: mh-collector
rules:
  - apiGroups: [""]
    resources: ["nodes", "pods", "persistentvolumeclaims"]
    verbs: ["get", "list"]
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets", "daemonsets"]
    verbs: ["get", "list"]
  - apiGroups: ["discovery.k8s.io"]
    resources: ["endpointslices"]
    verbs: ["get", "list"]
  - apiGroups: ["metrics.k8s.io"]
    resources: ["pods", "nodes"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: mh-collector
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mh-collector
subjects:
  - kind: ServiceAccount
    name: mh-collector
    namespace: metricharbor
---
apiVersion: v1
kind: Secret
metadata:
  name: mh-collector-token
  namespace: metricharbor
  annotations:
    kubernetes.io/service-account.name: mh-collector
type: kubernetes.io/service-account-token
YAML

That is the whole grant. get and list, on two resource types and their metrics. A monitoring account that can read secrets is a credential store with extra steps, and this one cannot read them, cannot write anything, and cannot exec into a pod.

The Secret at the end is what produces a long-lived token. Since Kubernetes 1.24 a ServiceAccount no longer gets one automatically — kubectl create token issues short-lived ones instead, which is the right default and the wrong fit for an agent outside the cluster that nothing is rotating for. Creating the Secret explicitly is the supported way to ask for a token that does not expire.

Check the grant is what you think it is:

kubectl -n metricharbor get secret mh-collector-token -o jsonpath='{.data.token}' | head -c 20
kubectl auth can-i --as=system:serviceaccount:metricharbor:mh-collector list pods -A
kubectl auth can-i --as=system:serviceaccount:metricharbor:mh-collector get secrets -A

The last one must print no. If it prints yes, something else in the cluster is granting more than this role does, and the account is not read-only.

2. Build a kubeconfig for it#

Run this where kubectl works, with the admin kubeconfig from step 1. That is usually your workstation or the control plane node — not the agent host, which is a different machine and the reason the steps below say which is which.

The agent takes a kubeconfig, so assemble one from the token and the cluster's CA.

SERVER=https://172.30.139.201:6443     # your API server, as the AGENT host can reach it

CA=$(kubectl -n metricharbor get secret mh-collector-token -o jsonpath='{.data.ca\.crt}')
TOKEN=$(kubectl -n metricharbor get secret mh-collector-token \
  -o jsonpath='{.data.token}' | base64 -d)

cat > /tmp/mh-kubeconfig.yaml <<YAML
apiVersion: v1
kind: Config
current-context: metricharbor
clusters:
  - name: cluster
    cluster:
      server: ${SERVER}
      certificate-authority-data: ${CA}
users:
  - name: mh-collector
    user:
      token: ${TOKEN}
contexts:
  - name: metricharbor
    context:
      cluster: cluster
      user: mh-collector
YAML
chmod 600 /tmp/mh-kubeconfig.yaml

The CA goes straight from the Secret into the kubeconfig: both want it base64-encoded, so decoding it to a file and encoding it again achieved nothing except a second file to clean up.

SERVER is the part that catches people. A k3s kubeconfig at /etc/rancher/k3s/k3s.yaml says https://127.0.0.1:6443, which is correct on the control plane node and useless anywhere else. Use the address the agent host can actually reach.

Prove it works before involving MetricHarbor at all:

kubectl --kubeconfig /tmp/mh-kubeconfig.yaml get nodes

If that lists your nodes, the agent will succeed. If it does not, nothing downstream will, and the error here is far more direct than the one you would get later.

3. Give it to the agent#

Two machines in this step. The kubeconfig is created where kubectl runs and used where the agent runs. Each cleans up its own copy; neither can shred a file on the other.

Copy it across:

scp /tmp/mh-kubeconfig.yaml root@AGENT-HOST:/tmp/mh-kubeconfig.yaml
shred -u /tmp/mh-kubeconfig.yaml          # on THIS machine, now that it is sent

Then, on the agent host, hand it over by file — never as an argument, since argv is world-readable in /proc and lands in shell history:

# The agent runs as the metricharbor user and this file arrived owned by root at mode 0600.
# Without this you get "permission denied" on a file you can see perfectly well.
chown metricharbor:metricharbor /tmp/mh-kubeconfig.yaml

runuser -u metricharbor -- metricharbor-agent cred add k8s-monitor \
    --type kubeconfig --kubeconfig-file /tmp/mh-kubeconfig.yaml

shred -u /tmp/mh-kubeconfig.yaml

The agent encrypts it under the machine key. Delete the plaintext afterwards — it has served its purpose and the copy in the store is the one that matters.

runuser -u metricharbor -- metricharbor-agent cred list

4. Create the target and add the check#

In the UI:

1. Credentials → Declare credential → Kubernetes kubeconfig, named exactly k8s-monitor. The name is the join key between MetricHarbor and the agent's own store; the secret stays on the agent.
2. Targets → Add target → Kubernetes cluster.
   • API server URL — the same SERVER as above.
   • Cluster credential — k8s-monitor.
   • Namespaces — leave empty for the whole cluster.
   • Collected by — the agent you installed the credential on.
3. On the target, Add a check → Kubernetes cluster.

The agent picks it up on its next config poll, within a minute.

What it reads#

Four GET requests against stable, versioned endpoints. The agent speaks plain HTTP to them rather than embedding client-go, which would add tens of megabytes to a binary installed on customer machines and tie the agent's releases to Kubernetes' own.

What you get#

Workloads answer what pods cannot. "Three of twelve pods are not ready" is a fact that has to be translated before anyone can act on it; "the payments Deployment is running on two of its four replicas" is the same event, already in the terms you deployed it in. A Deployment scaled to zero on purpose counts as complete, not degraded, because the comparison is against what it was asked for rather than against nothing.

A Service is included for one reason: a Service with no ready endpoints is a black hole. Requests to it fail and nothing about the Service object says so — it is a name and a selector, and it exists whether or not anything answers. The count comes from EndpointSlices, summed per Service, so several slices backing one Service give one series and not three.

There is no discovery step. Tools whose datapoints have no label dimension need one — a script that enumerates pods, then token substitution to create a datapoint per pod. Here one collection emits every pod as its own series in one pass, because the series identity already includes the labels. If you have used Active Discovery and ##WILDVALUE## elsewhere and are looking for the equivalent: there is nothing to configure, and that is not an omission.

Options on the check#

Scoping each resource separately#

The usual shape is not one namespace list but several: every Deployment in the cluster, because that is where the meaning is, and pods from one namespace only, because that is where the series count explodes. The per-resource fields exist for exactly that:

Deployments, DaemonSets, Services and claims fall through to the general list, so they are collected cluster-wide; pods come from prod and StatefulSets from databases.

There is no scoping expression — no deployments:*, statefulsets:prod. It would be shorter and it would also be a small language to parse, document and get wrong, which is the thing this product is deliberately not growing. Six dull fields need learning once, and only by people who need them.

When it is wrong#

A node that is NotReady is not an error — it is the collector working. k8s.nodes_ready drops below k8s.nodes_total, and k8s.node.ready is 0 for that node specifically.

Verified#

The RBAC, endpoints, metric names and labels on this page were read from the collector's source. The grant was checked with kubectl auth can-i against k3s v1.34.5.

Not yet verified end to end: no agent has collected a live cluster through this path at the time of writing. If something here is wrong, it is in the steps between a working kubectl --kubeconfig and a graph on screen.