Linux utility /tool to display historical performance data (like sar) but along with the command/process

[linuxgeek]

Is there any Linux utility /tool to display historical performance data (like sar) but along with the command/process ? All i need is for a previous day want to see cpu, memory, disk usage process wise along with the command . So basically a tool that consolidates ps, sar information. Please share any such tool details that can be safer to install on Enterprise linux servers.

 

[f33dm3bits]

GitHub - grafana/grafana: The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.

The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many mo...

github.com

Grafana: The open and composable observability platform | Grafana Labs

Grafana is the open source analytics & monitoring solution for every database.

grafana.com

 

[linuxgeek]

GitHub - grafana/grafana: The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.

The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many mo...

github.com

Grafana: The open and composable observability platform | Grafana Labs

Grafana is the open source analytics & monitoring solution for every database.

grafana.com

Thank you for the response. Will grafana show historical details along with the command/process that was running/consuming?

 

[f33dm3bits]

Only stats which has been collected and forwarded to Grafana, so not commands. You can try it out here to get an idea of what kind of stats you can see but you can probably add more depending on what stats you configure your client to forward to the Grafana server.

 

[linuxgeek]

Just for the monitoring of metrics alone we already have various tools in place. What we are looking for is a tool that also captures the commands/job wise cpu, memory , disk space utilization . We are basically trying to find out who is the culprit when system goes out of resources not on real time basis but for previous/ past days

 

[f33dm3bits]

I don't know if any tools look what commands or jobs are running only metrics on the system, maybe one of these alternatives?

https://alternativeto.net/software/grafana/

https://alternativeto.net/software/prometheus/

At where I work we use nagios/nrpe to monitor system memory and load usuage, when we get an alert that the system is getting low on memory we just login and see what process is using the most memory what user the process belongs to. If that is not an option you could write a script that writes that information on a regular interval to a log file on the sysystem.

 

[f33dm3bits]

I just came across atop maybe that can do what you are looking for?

Atoptool.nl

An advanced interactive monitor for Linux-systems to view the load on system-level and process-level.

www.atoptool.nl

Atop is an ASCII full-screen performance monitor for Linux that is capable of reporting the activity of all processes (even if processes have finished during the interval), daily logging of system and process activity for long-term analysis, highlighting overloaded system resources by using colors, etc. At regular intervals, it shows system-level activity related to the CPU, memory, swap, disks (including LVM) and network layers, and for every process (and thread) it shows e.g. the CPU utilization, memory growth, disk utilization, priority, username, state, and exit code.

 

[linuxgeek]

I just came across atop maybe that can do what you are looking for?

Atoptool.nl

An advanced interactive monitor for Linux-systems to view the load on system-level and process-level.

www.atoptool.nl

This looks interesting and I'm now testing this. Thanks for your responses. I also need to analyze the impact of this tool and disk log usage over the period of time.

 

[denmon]

atop is very useful, and does not require any gui or webserver/browser. In Debian it's just "apt install atop". In interactive realtime mode it shows you current statistics every 10 seconds, similar to many other "top" variants.

But its real value is the logging... by default it keeps 28 days of logs, with historical data samples every 10 minutes. So if you want to know what was going on at 4:20pm 5 days ago, you can do "atop -R /var/log/atop_20250319" to look at that day, skip forward/backward every 10 minute sample with "t" and "T", or jump to a specific time with "b".

Once you are at the timepoint you want, you can change what all information is displayed to be the full command line of each process, real & virtual memory info, etc.

This is part of our default install on headless servers, and has been essential for figuring out performance issues that happened and resolved before we could look at them at the time.

 

[KGIII]

Hmm... top, atop, and htop have been around for ages.

I had no idea that atop had this function. I looked and it doesn't appear that htop has this functionality.

 

[osprey]

atop is very useful, and does not require any gui or webserver/browser. In Debian it's just "apt install atop". In interactive realtime mode it shows you current statistics every 10 seconds, similar to many other "top" variants.

But its real value is the logging... by default it keeps 28 days of logs, with historical data samples every 10 minutes. So if you want to know what was going on at 4:20pm 5 days ago, you can do "atop -R /var/log/atop_20250319" to look at that day, skip forward/backward every 10 minute sample with "t" and "T", or jump to a specific time with "b".

Once you are at the timepoint you want, you can change what all information is displayed to be the full command line of each process, real & virtual memory info, etc.

This is part of our default install on headless servers, and has been essential for figuring out performance issues that happened and resolved before we could look at them at the time.

It's probably worth pointing out a difference between atop and i7z in relation to cpu frequency output. For example the following shows first a selection from the output of i7z and second shows a selection from the output of atop. Both programs were run as root and simultaneously:

i7z:

Cpu speed from cpuinfo 2495.00Mhz
cpuinfo might be wrong if cpufreq is enabled. To guess correctly try estimating via tsc
Linux's inbuilt cpu_khz code emulated now
True Frequency (without accounting Turbo) 2495 MHz
  CPU Multiplier 25x || Bus clock frequency (BCLK) 99.80 MHz

Socket [0] - [physical cores=14, logical cores=20, max online cores ever=14]
  TURBO ENABLED on 14 Cores, Hyper Threading ON
  Max Frequency without considering Turbo 2594.80 MHz (99.80 x [26])
  Max TURBO Multiplier (if Enabled) with 1/2/3/4/5/6 Cores is  48x/48x/47x/47x/45x/45x
  Real Current Frequency 798.12 MHz [99.80 x 8.00] (Max of below)
        Core [core-id]  :Actual Freq (Mult.)      C0%   Halt(C1)%  C3 %   C6 %  Temp      VCore
        Core 1 [0]:       793.29 (7.95x)        20.1      85       0    8.63    25      0.7721
        Core 2 [2]:       791.84 (7.93x)        1.68    80.5       0      19    25      0.7721
        Core 3 [4]:       793.78 (7.95x)        24.5    34.7       0    57.5    25      0.7714
        Core 4 [6]:       793.98 (7.96x)        15.9    69.7       0    25.2    26      0.7764
        Core 5 [8]:       791.46 (7.93x)        8.71    68.8       0    28.4    25      0.7764
        Core 6 [10]:      795.17 (7.97x)        8.46    39.7       0    57.6    28      0.7814
        Core 7 [12]:      797.96 (8.00x)        3.86    5.87       0    92.9    28      0.7814
        Core 8 [13]:      797.73 (7.99x)        1.75    1.53       0    97.9    28      0.7783
        Core 9 [14]:      798.12 (8.00x)           1    2.19       0    97.5    28      0.7733
        Core 10 [15]:     796.78 (7.98x)           1    2.02       0    97.8    28      0.7717
        Core 11 [16]:     797.42 (7.99x)           1    2.65       0      97    26      0.7767
        Core 12 [17]:     797.91 (8.00x)        1.61    13.2       0    86.3    26      0.7717
        Core 13 [18]:     797.97 (8.00x)           1    1.75       0    98.1    26      0.7767
        Core 14 [19]:     797.18 (7.99x)           1    1.87       0    97.9    26      0.7767
  Ctrl+C to exit

atop:

PRC | #exit     20 |
CPU | curf  796MHz |
cpu | curf  800MHz |
cpu | curf  800MHz |
cpu | curf  800MHz |
cpu | curf  800MHz |
cpu | curf  749MHz |
cpu | curf  789MHz |
cpu | curf  800MHz |
cpu | curf  800MHz |
cpu | curf  800MHz |
cpu | curf  800MHz |
cpu | curf  799MHz |
cpu | curf  800MHz |
cpu | curf  790MHz |
cpu | curf  800MHz |
cpu | curf  800MHz |
cpu | curf  800MHz |
cpu | curf  800MHz |
cpu | curf  800MHz |
cpu | curf  800MHz |
cpu | curf  800MHz |

Both apps can log to logfiles as described in the manpages.