Metrics tell you something is wrong; logs tell you why. In a homelab with a hypervisor, a few virtual machines, a dozen containers, a router and a NAS, logs are scattered everywhere: in each Linux system's journal, in each application's files, in the router's limited memory. When a service stops at three in the morning, piecing together what happened by jumping from one machine to another is slow, and sometimes impossible because the log has been overwritten.
Why centralise
Collecting logs in one place brings concrete benefits:
- a single search to find an error across every machine;
- time correlation: seeing that a container restart coincides with a disk error on the NAS;
- data survival: if a machine hangs or is reinstalled, its last logs remain elsewhere;
- devices without storage: routers, switches and access points keep few logs and lose them on reboot.
The basic building blocks
syslog
The syslog protocol is the common denominator: almost every network device, NAS and Unix system can send messages to a remote syslog server, traditionally on UDP port 514. Implementations such as rsyslog and syslog-ng can receive those messages and write them to files, split by host.
It is the simplest solution: a receiving server, a folder per device, file rotation with logrotate. For many homelabs it is enough.
journald
On distributions with systemd, logs go to the journal, which stores structured fields such as unit, priority and process ID. The journal can forward to syslog or be read directly by a collection agent. Commands such as journalctl -u servicename --since "1 hour ago" remain invaluable for local analysis.
Docker
By default Docker keeps container output in JSON files on the host. Without a limit they grow forever: set the log driver's max-size and max-file options in the daemon configuration file. An agent can then read those files and send them to the central system.
Loki: logs indexed by labels
Grafana Loki is a log storage system designed to sit alongside Prometheus. Its key idea is not to index the full text of messages, only a small set of labels such as host, service and level. The content is compressed and stored in chunks. The result is a light system that suits modest hardware.
The typical architecture:
- an agent on each machine reads the journal, files and container logs and sends them to Loki; historically Promtail was used, and Grafana now points to its Alloy agent as the successor;
- Loki receives, compresses and stores;
- Grafana queries Loki with the LogQL language and shows the results next to the metric graphs.
An example LogQL query:
{host="pve1", unit="pveproxy.service"} |= "error"
selects the logs of the Proxmox web service on one host and filters the lines containing «error».
The same rules as for metrics apply to labels: they must have few possible values. Using the client IP address or a request ID as a label multiplies the streams and weighs the system down; that data should be searched in the text.
Alternatives
- Graylog: a full search interface, streams and processing rules; it needs a search engine such as OpenSearch and more memory.
- Elastic or OpenSearch stack: full-text indexing and very powerful searches, but heavy RAM and disk use for a homelab.
- rsyslog only: the bare minimum, no interface, searching with grep.
In my work, for a home network, I tend to start with rsyslog for network devices and Loki for servers and containers: it fits into existing dashboards and needs few resources.
Retention and privacy
Logs contain sensitive information: usernames, IP addresses, file paths, sometimes request data. Some common-sense rules:
- set a retention period, for example 30 or 90 days, and enforce it with the Loki configuration or file rotation;
- restrict access to the viewing interface to an administration network;
- do not send logs outside the home without encryption; between sites use TLS or a VPN;
- stop applications writing passwords or tokens to logs by checking the configured verbosity.
Useful queries to save
Some queries deserve a permanent place on a dashboard:
- failed SSH login attempts across all hosts;
- disk I/O errors in the kernel log;
- containers restarting repeatedly;
- authentication errors on the hypervisor interface;
- router messages about line disconnections.
By tying some of these to alerting rules, logs become a source of warnings as well as an investigation tool.
In short
- Centralising logs allows single searches, correlation and retention after a failure.
- syslog collects network devices; journald and container logs need an agent.
- Loki indexes only a few labels and stays light; you query it from Grafana with LogQL.
- Size limits on container logs and a retention period prevent full disks.
- Logs contain sensitive data: restrict access and encrypt transmission.
