Why is so important for a Linux admin to master the sos command?

[linuxjedi]

I’m a senior developer and 3th level support engineer for a Linux based appliance system for a few years now. When I started in this position, I couldn’t believe how customer technical support was handled (an eternal send me the output of this command, email-cycle. Every ticket lasted weeks open.) so I built a script to collect diagnostic data and a web interface to exploit the data and share it with the team. It improved the support service dramatically from weeks to hours.

At that time I didn’t know about sos command (it was called sosreport back then) but it was very much the same concept. Based on that experience, I soon realized that a tool capable of managing, sharing and analyzing sosreports was missing, so I built one without me realizing how little known the sos command is through the Linux community. So I wrote this article hoping to pick-up your curiosity and encourage people to take advantage of the sos command.

If you make a living by troubleshooting or diagnosing Linux systems whether in large production environments, or in small business with desktop computers, I think that you will find this article extremely useful.

In this article, I provide a comprehensive overview of the sos command and its many features and at the end I will tell you why is so important to have it in your list of tools.

To keep the article concise and easy to read, I limited the depth of each topic and include only brief examples where appropriate but provide links to other articles if you’d like to dive deeper on a specific feature.

Hope you'll find this article useful and interesting.

 

[f33dm3bits]

You're sharing the same link to "your" article 4 times. Also before self-promoting it's usually better to be take part of the forums first. I'll allow it, but just hope you links won't change into spam links in a few weeks as we have seen this happen.

 

[linuxjedi]

You're sharing the same link to "your" article 4 times. Also before self-promoting it's usually better to be take part of the forums first. I'll allow it, but just hope you links won't change into spam links in a few weeks as we have seen this happen.

thnks

 

[linuxjedi]

You're sharing the same link to "your" article 4 times. Also before self-promoting it's usually better to be take part of the forums first. I'll allow it, but just hope you links won't change into spam links in a few weeks as we have seen this happen.

I removed 3 links for your peace of mind.

 

[f33dm3bits]

I removed 3 links for your peace of mind.

Thanks and welcome!

 

[beanburrito]

Welcome to the forums, linuxjedi.

 

[linuxjedi]

Welcome to the forums, linuxjedi.

Thanks!

 

[wizardfromoz]

...for your peace of mind.

It is not about peace of mind, it is about complying with a legitimate and reasonable request from Staff.

Your article is deficient in a small (so far as I have looked) number of ways.

• sos command is only available in RedHat. False. sos is included in the sos package on most Linux distributions.

Well, it is not. It is only native to RHEL-based systems, including Fedora and Ultramarine and so on.

Under Debian-based, it is still sosreport

chris@Xia-Cinnamon-HDD:~$ apt policy sosreport
sosreport:
  Installed: (none)
  Candidate: 4.10.2-0ubuntu0~24.04.1

and in Arch and Arch-based distros, likewise, but it is only available as a Snap.

In Arch, snaps through the AUR are impractical, as the AUR has had significant breaches in recent days.

Welcome to linux.org

Chris Turner
wizardfromoz

 

[linuxjedi]

It is not about peace of mind, it is about complying with a legitimate and reasonable request from Staff.

Your article is deficient in a small (so far as I have looked) number of ways.



Well, it is not. It is only native to RHEL-based systems, including Fedora and Ultramarine and so on.

Under Debian-based, it is still sosreport

chris@Xia-Cinnamon-HDD:~$ apt policy sosreport
sosreport:
  Installed: (none)
  Candidate: 4.10.2-0ubuntu0~24.04.1

and in Arch and Arch-based distros, likewise, but it is only available as a Snap.

In Arch, snaps through the AUR are impractical, as the AUR has had significant breaches in recent days.

Welcome to linux.org

Chris Turner
wizardfromoz

thanks for pointing this deficiency. It is corrected now.

 

[piorunz]

View attachment 32330

I’m a senior developer and 3th level support engineer for a Linux based appliance system for a few years now. When I started in this position, I couldn’t believe how customer technical support was handled (an eternal send me the output of this command, email-cycle. Every ticket lasted weeks open.) so I built a script to collect diagnostic data and a web interface to exploit the data and share it with the team. It improved the support service dramatically from weeks to hours.

At that time I didn’t know about sos command (it was called sosreport back then) but it was very much the same concept. Based on that experience, I soon realized that a tool capable of managing, sharing and analyzing sosreports was missing, so I built one without me realizing how little known the sos command is through the Linux community. So I wrote this article hoping to pick-up your curiosity and encourage people to take advantage of the sos command.

If you make a living by troubleshooting or diagnosing Linux systems whether in large production environments, or in small business with desktop computers, I think that you will find this article extremely useful.

In this article, I provide a comprehensive overview of the sos command and its many features and at the end I will tell you why is so important to have it in your list of tools.

To keep the article concise and easy to read, I limited the depth of each topic and include only brief examples where appropriate but provide links to other articles if you’d like to dive deeper on a specific feature.

Hope you'll find this article useful and interesting.

Please show example of the sos use if you are talking about it so extensively.

Anyway, why use sos when we have inxi? Anything else sos can do what inxi cannot? You did not explain.

 

[linuxjedi]

Please show example of the sos use if you are talking about it so extensively.

Anyway, why use sos when we have inxi? Anything else sos can do what inxi cannot? You did not explain.

Sorry the article is not clear enough!

Any way here is what inxi cannot do that sos can:

• inxi does not allow you troubleshoot the system when you do not have access to it (typical L2/L3 support scenario).
• inxi cannot deliver the output encrypted to a central analysis repository.
• inxi cannot retrieve the report from a cluster or server swarm from a single invocation.
• inxi does not preserve the data in an structured way.
• inxi does not include logs nor configuration files.
• Inxi does not allow you to include your own logs and your own commands.
• inxi cannot retrieve information from things like: mysql, ansible, openstack, kubernetes, docker, apparmor, apt, rmp, pam, and many, many more.

That is mainly the difference.

Maybe I'm wrong but I believe that you were expecting some output like inxi and that's why you say that there is no example of sos use? If that is the case, then let me clarify: that is not how sos works. The output of sos is a file like this:

/tmp/secured-sosreport-ganimide-ISSUE001-2025-04-20-jflhnxa.tar.xz.gpg

as you can see it is encrypted with GPG and if you know the decryption key then you can extract the contents and exploit the data usually in a different box.

Please let me know if that is what you meant by "show example of sos use" or is it something else?.

thanks for your comments

 

[piorunz]

Yes, I wonder how it works. I wonder what can it share on top of what inxi can generate already.

$ apt show sos
Package: sos
Version: 4.9.1-1
Priority: optional
Section: admin
Maintainer: sosreport Dev Team <[email protected]>
Installed-Size: 2,202 kB
Depends: python3-magic, python3-packaging, python3-pexpect, python3-yaml, python3:any
Recommends: e2fsprogs, lsof, mount, python3-boto3
Breaks: sosreport (<< 4.8.2-4~)
Replaces: sosreport (<< 4.8.2-4~)
Homepage: https://github.com/sosreport/sos
Download-Size: 365 kB
APT-Sources: http://deb.devuan.org/merged excalibur/main amd64 Packages
Description: Set of tools to gather troubleshooting data from a system
 Sos is a set of tools that gathers information about system
 hardware and configuration. The information can then be used for
 diagnostic purposes and debugging. Sos is commonly used to help
 support technicians and developers.

Output being encrypted? For debugging and sharing? Doesn't sound very user-friendly.