var scripting

[ilovelinux]

Hi,

I would like to seek for your assistance on my ongoing ls bash scripting.

I need to create a script to get the content of my var folder. Please help me on this.
This will run every minute.

Here is my current script:

#!/bin/bash

ls -lar /var


Or please suggest any script that I could put on the server.. Really appreciate all your help. Thank you.

 

[JasKinasis]

Offhand, one program that might help is monit:
https://mmonit.com/monit

Monit is a free, open source program which allows you to deal with monitoring system services and other files (such as log-files) - that might be worth looking into. It should be available in the repos of most, if not all Linux distros.

The company that develops monit also develop a paid version called mmonit, which is supposedly better than monit. So this might also be worth looking into if this is for your employers servers, or if monit doesn't quite do what you want.

So monit would definitely be worth taking a look at. I haven't used it myself, but from what I know about it - it should be able to do what you need.

 

[JasKinasis]

Regarding writing a script, would using the -t switch help?
That would list files by modification time. So the most recently accessed files would be listed first.

Also why are you using the -r switch (reverse sort). Did you mean -R (recursive)?
If so, recursively listing everything in /var/ would end up listing thousands of files.
Wouldn't it make more sense to watch a particular sub-directory in var?

Typically, the places where most activity will be taking place would be /var/tmp/ and /var/log/.
However, IMO - the most relevant one to be monitoring would probably be /var/log/.

You could perhaps try listing all of the files as you are doing (but with the addition of the -t switch) and store the output in a variable and compare the results against the results from the previous minutes run.

Here you go, this is something called monitorscript, which I knocked up earlier in about 15-20 minutes:
(see also monitorscript in attached .zip)

#!/usr/bin/env bash
# monitorscript by Jason Trunks a user at linux.org
# < https://notabug.org/JasKinasis/ >
# This is free software licensed under the GPLv3
# See https://www.gnu.org/licenses/gpl.html for full details

function die
{
   echo "ERROR: monitorscript takes a single, optional parameter"
    echo "which MUST be a positive number (in minutes)"
   echo
   echo "Usage: 'monitorscript 5' - run the script for 5 minutes"
   echo "OR: 'monitorscript' - run the script perpetually"
   echo "Note: Running monitorscipt without specifying a duration will require you to manually kill the process"
   echo
   exit 1;
}

# This will only be populated if the user specifies
# how long the script should run for
requiredDuration=

# Ensure we only have no more than a single parameter
# which is a number greater than zero
if [[ $# == 1 ]]; then
   val=$(echo "$1" | awk '/^[0-9]+$/') 
   if [[ ! $val ]] || [[ $val == 0 ]]; then
       die
   fi
    requiredDuration=("$val"*60) # convert minutes to seconds
elif [[ $# -ge 1 ]]; then
   die
fi

# Path to logfile - change this to reflect where you want
# the logfile to be stored
logfile="$HOME/varlogmonitorfile"

# current duration
duration=0;

# listing from previous minute
prev=""

# set up an infinite loop
while true
do
    # list files and store in a variable
    cur=$(ls -lARt /var/log 2> /dev/null )

    # if prev not empty
    if [[ $prev ]]; then
       # Diff the content of the two variables
        diffs=$( diff --suppress-common-lines --suppress-blank-empty <(echo "$prev") <(echo "$cur") )
     
       # Log any diffs
        if [[ $diffs ]]; then
         echo "$(date +%F@%T) - Some files have been changed:" >> "$logfile"
          echo "$diffs" >> "$logfile"
         printf "\n\n" >> "$logfile"
     fi
    fi

   # if user specified a duration
   # check how long we have been running for
   if [[ ${requiredDuration} ]]; then
       if [[ ${duration} -ge ${requiredDuration} ]]; then
           #time is up - end the script
           exit 0
       fi
    fi

   # set prev to cur ready for the next pass
    prev=$cur

   # sleep for another 60 seconds
    sleep 60

   # increment duration by 60
   if [[ ${requiredDuration} ]]; then
        duration=$duration+60
   fi
done

To run the script:

./monitorscript

Which will run forever, or until you manually stop it (or until the log file ends up taking up all the remaining space on your hard-drive!)

The best way to run it is to pass it the number of minutes it should run for:

./monitorscript 5

The above example would run the script for 5 minutes. If you want it to run for an hour, pass in 60. For 2 hours, 120 etc etc.

As it stands it would write a file called varlogmonitorfile to the user running the scripts home directory. So if you were running the script as you, it would be in /home/yourusername/, or as root it would be in /root/. To change the path/filename for the logfile change the value of the variable $logfile in the script.

You could set that up as a cron job to run for a specified amount of time at particular times.
e.g. to do your 3AM on Thursday for 1 hour:

0 3 * * 4 /path/to/monitorscript 60

Then the monitorscript will run for 60 minutes at 3AM on Thursday. Every minute in that hour, it will wake up and list the content of /var/log, logging any differences and will end when the hour is up.

In the log-file - the output from monitorscript looks like this:

2018-04-12@13:04:40 - Some files have been changed:
2a3,6
> -rw-r----- 1 root              adm       1234467 Apr 12 13:03 kern.log
> -rw-r----- 1 root              adm       1280608 Apr 12 13:03 messages
> -rw-r----- 1 root              adm        489660 Apr 12 13:03 syslog
> -rw-r----- 1 root              adm        724138 Apr 12 13:03 ufw.log
4,7d7
< -rw-r----- 1 root              adm        489262 Apr 12 13:02 syslog
< -rw-r----- 1 root              adm       1234069 Apr 12 13:01 kern.log
< -rw-r----- 1 root              adm       1280210 Apr 12 13:01 messages
< -rw-r----- 1 root              adm        723740 Apr 12 13:01 ufw.log

The above output is from one cycle of the script. So it woke up at 13.04 and found that some files had been written to while it was sleeping between 13.03 and 13.04.
The first 4 entries are from the current minute and the last 4 are from the previous minute. We can see that kern.log, messages, syslog and ufw.log have all been written to . Their file-sizes have all increased and that kern.log was most recently written to.

You may need to write an additional script to analyse the content of the log-file and determine which files have been changed and how often.
Also, thinking logically - any time the log-rotation system automatically rotates the logs - you could end up with a huge block of diffs written.
log-rotation occurs when logs get to a certain size. The numbers in the file-names of any current gzipped logs are increased by one, starting with the one with the highest number
e.g.
log.2.gz becomes log.3.gz,
log.1.gz becomes log.2.gz etc.
The current log is then gzipped and called log.1.gz and a new log is started.
So in the minute after a log-rotation, you could see a lot of files listed as changed.

If the output from my script is not quite what you are looking for by all means change it, or look into using monit! Heh heh! XD

To understand how the script works:
Lets assume a user has started the script with a duration of 2 minutes.
e.g. they ran the script like this

./monitorscript 2

We'll ignore all of the stuff at the start that deals with checking the parameters etc.
Once we're past all of that, the variable $requiredDuration is set to 120 seconds (or 2 minutes) and the variable $duration is set to 0;

The main part of the script we're interested in is the while loop.
First we list the content of /var/log using 'ls -lARt' and store it in the variable $cur.

The -l flag tells ls to use the long list output format.
The -A flag lists ALMOST ALL files. Meaning it will list everything except . and ..
The -R flag is the recursive flag, so it will recursively list the content of the current directory and all sub-directories.
The -t flag sorts the output by the modification date, so the most recently modified files are listed first.

Then we check to see if the variable $prev has been populated yet. In the first minute, $prev will not be populated with anything, so no diffs are performed and nothing is logged.
We check to see if it is time to quit - duration is 0 and requiredDuration is 120 - so it is not!
$prev is then given the value of $cur and the script sleeps for 60 seconds.

Upon waking, the script increments $duration from 0 to 60 and the while loop re-iterates.

The second time through the while loop - we get a fresh listing of /var/log and store it in $cur
This time, $prev WILL have something in it (from the previous minute), so we diff $prev against $cur and see if there are any differences.
If there are any differences, they are written to the log-file along with a date stamped message.
We check if it is time to quit yet - duration is 60, requiredDuration is 120 - so nope, onwards!
$prev is then given the value of $cur and the script sleeps for another 60 seconds.

Upon waking, the script increments $duration from 60 to 120 - so our two minutes have now passed.
The while loop reiterates one final time. Gets the list, performs the diff - writing any diffs to the log, then we check to see if it is time to quit - it is - and the script ends.

And at the end, we will have a file in our home directory called varlogmonitorfile with the diffs in it, so we can see which files were changed and when.

And that is that.
I've had quite good fun writing this script. I'm currently off work, in bed with the lurg ATM. So this has at least allowed me to fill some time. If nothing else, I hope it at least gives you some ideas!

As always - any questions, fire away.
Regards,
Jas.