Monitoring Linux hosts

MetricHarbor reads a Linux host over SSH. The agent logs in as an unprivileged user and runs a handful of read-only commands; nothing is installed on the monitored machine.

That includes the machine the agent itself is running on — see the host the agent runs on below, because it is the most common first thing people set up and the one with a wrinkle.

1. Give the agent an SSH key#

Generate it on the agent, so the private half never exists anywhere else — not in MetricHarbor, not in a browser, not in a paste buffer:

runuser -u metricharbor -- metricharbor-agent cred new-ssh ssh-monitor --username monitor

runuser rather than sudo, because sudo is not installed on a minimal Debian; sudo -u metricharbor … is equivalent where you have it. Not su - metricharbor either way — that account has a nologin shell by design, so su answers "This account is currently not available."

Declare a credential of kind ssh_key named ssh-monitor in Credentials → Declare credential first, and the page will show you this command already filled in.

The command prints a public key. Keep it; you need it in the next step.

Add --restrict to emit a command= forced-command prefix, which limits the key to the collector's own commands. Worth doing: it means a key stolen from the agent host cannot be used for a shell.

2. Create the login on each monitored host#

useradd --create-home --shell /bin/bash monitor
install -d -m 0700 -o monitor -g monitor /home/monitor/.ssh
echo 'ssh-ed25519 AAAA…  the key printed in step 1' \
  | tee /home/monitor/.ssh/authorized_keys
chown monitor:monitor /home/monitor/.ssh/authorized_keys
chmod 600 /home/monitor/.ssh/authorized_keys

The username must match the one you gave in step 1.

3. Create the target#

Targets → Add target → Linux host.

4. Add checks#

On the target page, Add a check. Four apply to a Linux host:

Start with System information. It needs no privileges beyond logging in, and it is the one that tells you the host is alive.

Per-filesystem, not averaged#

System information emits one series per mounted filesystem, labelled by mount point. This is deliberate and it is the reason the storage model works the way it does: averaging twelve filesystems into a single "disk usage" number is how a full /var hides behind eleven empty ones.

You will see this in Latest readings as several rows of the same metric with different mount= labels.

Verifying the host you connect to#

Leave Pinned SSH host key empty and the agent does not check the host's identity. It connects, authenticates and collects — and would do the same against anything that answered on that address.

To pin it, paste one line from the monitored host:

cat /etc/ssh/ssh_host_ed25519_key.pub

How much this matters depends on the credential. With an ssh_key credential, public-key authentication hands nothing to an impostor: the key proves itself without being transmitted, so the exposure is false metrics rather than a stolen credential. With an ssh_password credential, the password is sent to whatever answers, so pinning is not optional in any meaningful sense.

Being honest about the current state: there is no learn-on-first-use mode and the interface does not yet show which targets are unpinned. Both are known gaps.

When you need sudo#

Only SMART really needs it. smartctl talks to the disk device directly, which an unprivileged user cannot do.

Tick Use sudo on the target, and on the monitored host:

# /etc/sudoers.d/metricharbor  — install with: visudo -f /etc/sudoers.d/metricharbor
monitor ALL=(root) NOPASSWD: /usr/sbin/smartctl, /usr/sbin/mdadm

Restrict it to those two commands. A blanket NOPASSWD: ALL for a monitoring account turns a read-only credential into a root one, which undoes the reason for having a separate user.

NOPASSWD is required, not merely convenient. The agent runs sudo with -n, which never prompts. A sudo configured to ask for a password does not hang — it fails immediately — but the check then reports a command failure rather than anything about sudoers, which sends people looking in the wrong place.

If you do not want sudo at all, leave it off and do not bind the SMART check. Everything else works without it.

The host the agent runs on#

This surprises people, so it is worth stating plainly:

Installing the agent on a machine does not monitor that machine.

The agent collects and ships metrics. What it collects is a separate decision, and linux_host is always read over SSH — including when the host in question is the one the agent is running on. There is no local shortcut: the agent SSHes to itself.

So monitor it exactly as you would any other host. Give the agent an SSH key (step 1), authorise it for a login on that same machine (step 2), and create the target:

Hostname or IP:  127.0.0.1
SSH port:        22
SSH credential:  ssh-monitor
Collected by:    the agent on that host

It is more ceremony than it ought to be for reading /proc on the local machine, and it is a rough edge rather than a deliberate design. It is on the list.

Which address: 127.0.0.1#

Use 127.0.0.1, not localhost and not the machine's own LAN address.

• 127.0.0.1 is unambiguous. It needs no DNS and no /etc/hosts, the traffic never leaves the loopback interface, and it keeps working when the machine is readdressed — which on DHCP it will be.
• localhost usually resolves to ::1 first, before 127.0.0.1. If sshd is not listening on IPv6 — which happens whenever somebody sets AddressFamily inet or disables IPv6 — the connection either fails or stalls until it falls back. It works on most machines and fails confusingly on the rest, which is the worst combination.
• The LAN address works, but routes out through the network stack and whatever firewall sits in front of SSH, so a host firewall that allows SSH only from a management subnet will refuse its own agent. It also breaks the day the lease changes.

Agent health is not a substitute#

There is a separate Agent health check that reports the agent process — its uptime, memory and shipping backlog. It is useful, and it tells you nothing about the machine. For load, memory and filesystems you need a linux_host target as above.

What you should see#

On the target page, Recent collections shows success with a sample count, and Latest readings shows memory, load, uptime and one row per filesystem.

More in When nothing is collecting.