systemd, journalctl, and suspend/resume (sleep/wake): marked by default in the logs? Or how to add? And scripting around suspend/resume in general?

[dwawlyn]

(
First of all, I'm not completely sure what the official terms are.
I think it's "suspend"/"resume",
with "sleep"/"wake" being synonyms
(and I'm not sure if there are any other common synonyms).

And there's other, related-but-different stuff, namely "hibernate",
which is much more rarely used because hardware support for it is trickier and often not worth trying to mess with (I believe?).

(There's also "suspend-to-ram". I'm less clear whether that's a synonym for plain "suspend", or for "hibernate", or what.)

However, it's only "sleep"/"wake" that I'm interested in here.
)

---
So I wanted to look back through my system's journal to see what's been happening right after I wake the system from sleep.

(I have a USB keyboard that seems to not be coming back properly until I unplug-and-replug it, and I think it's a software issue, and I'm trying to figure it out.)

However, I'm not confident that I'm consistently finding the correct spots in the system journal.

Someone told me to manually run
(before sleep):
su root -c 'echo "Sleep started" > /dev/kmsg'
(after sleep):
su root -c 'echo "Sleep ended" > /dev/kmsg'

Which seemed odd to me,
because I would've thought that something already logged to the journal by default would mark sleep/wake?

If so, what?

---
And if not, how should I add something to automatically log sleep/wake times?

And just in general,
how should I do scripting around sleep/wake?

eg, if I did want to take
su root -c 'echo "Sleep started" > /dev/kmsg'
and
su root -c 'echo "Sleep ended" > /dev/kmsg'
and make them automatically run before and after sleep respectively
(as close as possible to the last/first thing before/after that sleep)?

I found this file:
/etc/systemd/sleep.conf
should I be working with that?

(
Like, I guess I'm looking for the equivalent of an init-script,
but for wake/sleep rather than just boot/shutdown.
...
Although in the case of sleep, I note it's possible to just have a single script run,
with a command to sleep in the middle of the script,
and then, without needing to do anything else,
the rest of the script will continue running after you wake the computer back up.

Not sure whether that's the right way of doing things, but it's certainly in principle possible.
)

---
Also, I only just started using journalctl like a week ago honestly.

Before that, I only used dmesg occasionally (with the --ctime/-T flag)
-- just whenever I wanted to check if something had recently OOM'd or segfaulted or whatever.

However, dmesg --ctime has a longstanding bug
where it just counts seconds since boot,
and then naively turns that into a "human readable" timestamp,
but completely ignores sleep/wake
so the timestamps actually drift further and further back out of sync from the actual real time the longer have your computer running,
if you're sleeping it every night but not rebooting.

So do the timestamps for journalctl still have that bug,
or is that fixed?

 

[z7vl7abxc]

this may depend on what distribution you are running. i believe mx linux 21 uses pm-utils to suspend the system whereas linux mint 20 uses systemd-sleep so it may help to know which distro you are using.

 

[dwawlyn]

Oh, I assumed it would be the same for all distos using systemd?
(I'm on opensuse tumbleweed.)

(I do know MX-linux is not systemd. Apparently they use sysvinit.)

 

[z7vl7abxc]

i believe some systemd distros can also be set up to run with pm-utils, but part of the reason i asked was that mx can use either systemd or sysV. from the manual page for systemd-sleep on mx linux (man systemd-sleep:

Immediately before entering system suspend and/or hibernation systemd-suspend.service (and the other mentioned units, respectively) will run all executables in /lib/systemd/system-sleep/ and pass two arguments to them. The first argument will be "pre", the second either "suspend", "hibernate", "hybrid-sleep", or "suspend-then-hibernate" depending on the chosen action. Immediately after leaving system suspend and/or hibernation the same executables are run, but the first argument is now "post". All executables in this directory are executed in parallel, and execution of the action is not continued until all executables have finished.

an example of a script in that directory:

cat /usr/lib/systemd/system-sleep/tlp 
#!/bin/sh

# tlp - systemd suspend/resume hook
#
# Copyright (c) 2022 Thomas Koch <linrunner at gmx.net> and others.
# This software is licensed under the GPL v2 or later.

case $1 in
    pre)  tlp suspend ;;
    post) tlp resume  ;;
esac

 

[dwawlyn]

Oh cool, thanks, that seems like probably everything I need to know!

Hmm, and $systemctl suspend -h
says there's a --dry-run flag,
which may help with testing things are functioning as you expect without needing to actually sleep/wake for every single test.

(I'll make sure to come back and add some notes here on whatever else I figure out after messing around with it all more.)

 

[dwawlyn]

Oh wait, I forgot I also need to figure out the journal stuff...

There's a flag like
$journalctl -t systemd-sleep
which I think is the only thing I need.

I'm not sure how to check if there's anything other than systemd-sleep I should also use...

The manpage says:

 -t, --identifier=SYSLOG_IDENTIFIER
     Show messages for the specified syslog identifier SYSLOG_IDENTIFIER.

     This parameter can be specified multiple times.

although I'm not sure exactly how to find a full list of possible SYSLOG_IDENTIFIERs...

There is a man systemd.journal-fields, but that doesn't have a list...

...

Apparently you can search like
$man -w -K 'SYSLOG_IDENTIFIER'
#=>

 /usr/share/man/man1/journalctl.1.gz
 /usr/share/man/man5/org.freedesktop.LogControl1.5.gz
 /usr/share/man/man7/systemd.journal-fields.7.gz
 /usr/share/man/man7/systemd.directives.7.gz

I'll have to look into it more.

(
Like, I guess ideally there would simply be a flag you could pass to journalctl to get it to tell you all of the SYSLOG_IDENTIFIERs it knows about,
but I dunno if it'd actually have anything like that...
)


---
I think ideally I would want to know (if journalctl has all the functionality I'm thinking about) how to...

(I'm mostly just thinking outloud here.)

Get output from the journal:

• starting with any USB-related messages before a sleep, or a certain number of entries right before a sleep
• then continuing to the wake and any USB-related messages then, or a certain number of entries right after a wake

And I guess I want to be aware of any USB events specifically about plugging/unplugging...

Also, I know journalctl has a flag

 -g, --grep=
     Filter output to entries where the MESSAGE= field matches the specified regular expression.
     PERL-compatible regular expressions are used,
     see pcre2pattern(3) for a detailed description of the syntax.

     If the pattern is all lowercase, matching is case insensitive.
     Otherwise, matching is case sensitive.
     This can be overridden with the --case-sensitive option, see below.

     When used with --lines=, --reverse is implied.

but I don't think there's any way to get the more advanced functionality of a grep-replacement program like ag and its --passthrough/--passthru flag
(ie, show all lines, but highlight occurrences of what you're searching for)...


---
...Oh yeah, and I still have to figure out whether that "lying timestamps" bug from dmesg -T is still a problem even with journalctl.

cuz $man journalctl just says:

 -k, --dmesg
     Show only kernel messages. This implies -b and adds the match "_TRANSPORT=kernel".

 

[z7vl7abxc]

my mint 20.3 system also uses sytemd-sleep. the quickest way i saw to get a rough estimate of those messages was with

journalctl -b | grep systemd-sleep

then you could use journalctl's -S (since) and -U (until) flags to choose a time frame a little before and after the first messages shown by the above.

 

[z7vl7abxc]

(I have a USB keyboard that seems to not be coming back properly until I unplug-and-replug it, and I think it's a software issue, and I'm trying to figure it out.)

when i had a touchpad that didn't work right after suspend, what worked was to remove and then add its module or driver just after suspend. i believe that acted as a software version of turning it off then back on again. if that is of interest, you may be able to find your keyboard's driver with either dmesg -w or udevadm.