Journalctl is a part of systemd (systemctl). It's great way t view status, and logs.
You know, systemd seems to be a touchy subject for some, but it's light years ahead of sysVinit and rc.unit.
You can do things sysVinit never dreamed of. I've given a few examples of this over the last few days.
But also, sysVinit is almost gone, only a handful of distro's still use it. As the borg would say "Prepare to be assimulated".
If you're running a systemd-based Linux distribution (which is most of them these days), you're using journalctl whether you realize it or not. It's the primary tool for viewing and querying logs from systemd's journal, and while many admins stick to the basics, journalctl has a lot more power under the hood than most people use.
The most common usage you'll see is checking on a specific service when something goes wrong:
Let's break down what those flags do. The -x flag adds explanatory help text from the systemd catalog when available - basically inline documentation for common errors. The -e flag jumps to the end of the log, which is usually where you want to be when troubleshooting. The -u flag filters to a specific systemd unit (service, timer, socket, etc.).
This combination is great for quick service troubleshooting, but journalctl can do much more.
One of the most useful features is time-based filtering. Instead of scrolling through thousands of log entries trying to find when something broke, you can narrow it down:
You can combine this with service filtering:
For watching logs in real-time, similar to tail -f on traditional log files:
Or for a specific service:
When you're dealing with critical issues, priority filtering helps cut through the noise. Systemd uses the standard syslog priority levels:
The numeric priorities are: 0=emerg, 1=alert, 2=crit, 3=err, 4=warning, 5=notice, 6=info, 7=debug.
You can combine priority with service and time filters:
One feature that saves a lot of headaches is boot-specific log viewing. When a system crashes or hangs, you need to see what happened during that boot, not the current one:
This shows all recorded boots with their boot IDs, timestamps, and offsets. Then you can view logs from a specific boot:
Or use the specific boot ID:
This is invaluable for troubleshooting "the server crashed overnight" scenarios.
For kernel-specific messages, replacing the old dmesg command:
The output format can be changed depending on what you're doing with the logs:
The json-pretty format is particularly useful when you need to parse logs programmatically or send them to log analysis tools. The cat format is great for piping to grep or other text processing tools.
You can filter on specific fields beyond just the service name:
To see what fields are available, use the verbose output format and look at the field names.
The journal can consume a lot of disk space over time. To see how much space it's using:
To clean up old logs:
You can also configure persistent limits in /etc/systemd/journald.conf:
For following multiple services simultaneously, you can specify multiple -u flags:
When you're troubleshooting issues that span multiple services (like a web application stack), this lets you see the interactions in real-time.
One thing that catches people off guard: by default, journal logs might not persist across reboots on some distributions. To ensure persistence, make sure /var/log/journal exists:
Or set Storage=persistent in /etc/systemd/journald.conf.
A particularly useful combination for debugging services that fail to start:
The -n 100 flag shows the last 100 lines, which is often enough context to see what led up to the failure without drowning in earlier unrelated messages.
For identifying patterns of repeated errors:
Or to see the frequency:
This pipeline strips the metadata, sorts the messages, counts duplicates, and shows the most frequent errors first.
When you're correlating issues across multiple systems, the --output-fields option combined with json format can be helpful:
The journal also captures stdout and stderr from services, which is different from traditional syslog. If a service writes to stdout or stderr, those messages appear in the journal automatically. This means you don't need separate log file configurations for many services.
For debugging systemd itself:
If you're seeing "journal corruption" messages or missing logs, these are the places to start.
One final tip: if you're grep-ing through journal output and want to preserve the useful metadata like timestamps, use the short-precise output format:
This keeps the timestamp with microsecond precision, which is helpful for correlating events.
The journal's biggest advantage over traditional log files is its structured data. Every log entry has metadata fields that you can query and filter on. Once you get comfortable with field-based filtering and time ranges, you'll find it much faster than grep-ing through text files. The learning curve is a bit steeper than tail and grep, but the payoff in troubleshooting speed is worth it.
You know, systemd seems to be a touchy subject for some, but it's light years ahead of sysVinit and rc.unit.
You can do things sysVinit never dreamed of. I've given a few examples of this over the last few days.
But also, sysVinit is almost gone, only a handful of distro's still use it. As the borg would say "Prepare to be assimulated".
If you're running a systemd-based Linux distribution (which is most of them these days), you're using journalctl whether you realize it or not. It's the primary tool for viewing and querying logs from systemd's journal, and while many admins stick to the basics, journalctl has a lot more power under the hood than most people use.
The most common usage you'll see is checking on a specific service when something goes wrong:
Code:
journalctl -xeu servicename.serviceLet's break down what those flags do. The -x flag adds explanatory help text from the systemd catalog when available - basically inline documentation for common errors. The -e flag jumps to the end of the log, which is usually where you want to be when troubleshooting. The -u flag filters to a specific systemd unit (service, timer, socket, etc.).
This combination is great for quick service troubleshooting, but journalctl can do much more.
One of the most useful features is time-based filtering. Instead of scrolling through thousands of log entries trying to find when something broke, you can narrow it down:
Code:
journalctl --since "2 hours ago"
journalctl --since "2025-01-29 14:00" --until "2025-01-29 15:00"
journalctl --since today
journalctl --since yesterdayYou can combine this with service filtering:
Code:
journalctl -u nginx.service --since "10 minutes ago"For watching logs in real-time, similar to tail -f on traditional log files:
Code:
journalctl -fOr for a specific service:
Code:
journalctl -u nginx.service -fWhen you're dealing with critical issues, priority filtering helps cut through the noise. Systemd uses the standard syslog priority levels:
Code:
journalctl -p err # errors only
journalctl -p warning # warnings and higher
journalctl -p 0..3 # emergency through errorThe numeric priorities are: 0=emerg, 1=alert, 2=crit, 3=err, 4=warning, 5=notice, 6=info, 7=debug.
You can combine priority with service and time filters:
Code:
journalctl -u mariadb.service -p err --since todayOne feature that saves a lot of headaches is boot-specific log viewing. When a system crashes or hangs, you need to see what happened during that boot, not the current one:
Code:
journalctl --list-bootsThis shows all recorded boots with their boot IDs, timestamps, and offsets. Then you can view logs from a specific boot:
Code:
journalctl -b # current boot only
journalctl -b -1 # previous boot
journalctl -b -2 # two boots agoOr use the specific boot ID:
Code:
journalctl -b 3e4f5a6b2c1dThis is invaluable for troubleshooting "the server crashed overnight" scenarios.
For kernel-specific messages, replacing the old dmesg command:
Code:
journalctl -k # kernel messages, current boot
journalctl -k -b -1 # kernel messages from previous bootThe output format can be changed depending on what you're doing with the logs:
Code:
journalctl -o json-pretty # JSON format, readable
journalctl -o json # JSON format, one line per entry
journalctl -o verbose # show all available fields
journalctl -o cat # just the message text, no metadataThe json-pretty format is particularly useful when you need to parse logs programmatically or send them to log analysis tools. The cat format is great for piping to grep or other text processing tools.
You can filter on specific fields beyond just the service name:
Code:
journalctl _PID=1234
journalctl _UID=1000
journalctl SYSLOG_IDENTIFIER=sshd
journalctl _COMM=nginxTo see what fields are available, use the verbose output format and look at the field names.
The journal can consume a lot of disk space over time. To see how much space it's using:
Code:
journalctl --disk-usageTo clean up old logs:
Code:
journalctl --vacuum-time=2weeks # keep only last 2 weeks
journalctl --vacuum-size=500M # limit total size to 500MB
journalctl --vacuum-files=5 # keep only 5 archived journal filesYou can also configure persistent limits in /etc/systemd/journald.conf:
Code:
SystemMaxUse=500M
SystemKeepFree=1G
SystemMaxFileSize=100MFor following multiple services simultaneously, you can specify multiple -u flags:
Code:
journalctl -f -u nginx.service -u mariadb.service -u php-fpm.serviceWhen you're troubleshooting issues that span multiple services (like a web application stack), this lets you see the interactions in real-time.
One thing that catches people off guard: by default, journal logs might not persist across reboots on some distributions. To ensure persistence, make sure /var/log/journal exists:
Code:
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journaldOr set Storage=persistent in /etc/systemd/journald.conf.
A particularly useful combination for debugging services that fail to start:
Code:
journalctl -xeu servicename.service -n 100The -n 100 flag shows the last 100 lines, which is often enough context to see what led up to the failure without drowning in earlier unrelated messages.
For identifying patterns of repeated errors:
Code:
journalctl -p err --since today | grep "specific error message" | wc -lOr to see the frequency:
Code:
journalctl -p err --since "1 hour ago" -o cat | sort | uniq -c | sort -rnThis pipeline strips the metadata, sorts the messages, counts duplicates, and shows the most frequent errors first.
When you're correlating issues across multiple systems, the --output-fields option combined with json format can be helpful:
Code:
journalctl -o json --output-fields=MESSAGE,PRIORITY,_HOSTNAMEThe journal also captures stdout and stderr from services, which is different from traditional syslog. If a service writes to stdout or stderr, those messages appear in the journal automatically. This means you don't need separate log file configurations for many services.
For debugging systemd itself:
Code:
journalctl -u systemd-journald.service
journalctl -u systemd-logind.serviceIf you're seeing "journal corruption" messages or missing logs, these are the places to start.
One final tip: if you're grep-ing through journal output and want to preserve the useful metadata like timestamps, use the short-precise output format:
Code:
journalctl -o short-precise | grep "error"This keeps the timestamp with microsecond precision, which is helpful for correlating events.
The journal's biggest advantage over traditional log files is its structured data. Every log entry has metadata fields that you can query and filter on. Once you get comfortable with field-based filtering and time ranges, you'll find it much faster than grep-ing through text files. The learning curve is a bit steeper than tail and grep, but the payoff in troubleshooting speed is worth it.
.
