Issue with the Log Archival and Removal using Bash script

[iamsaicharanburle]

Hi Team,

I prepared one shell script to find a file and move and zip and archive. But while i execute the script it is throwing error for the files dated with 2019 as no such file or directory file name will be something like server.log.2019-07-10* : No Such file or Directory. However it is moving zipping into the archival for the file names server.log.2020*.Below is my Script

#!/bin/bash
############################
# LOGS HOUSEKEEPING SCRIPT
# CREATED BY: Saicharan Burle
# LAST MODIFIED BY: Saicharan Burle : Added Archiving

############################
TIME=`date +%m%d%Y%H%M`
SCRIPTHOME="/home/ec2-user/scripts"

SCRIPT_LOCK=${SCRIPTHOME}/$0.lock

##### Directories to search (also include in the for loop below) ######
LOC1="/home/ec2-user/scripts/log"

##### Verify if script is already running #####
if [ -s ${SCRIPT_LOCK} ]
then
echo -e "Script is already running... Aborting!"
exit 1
fi

##### Create Lock file for the Script #####
echo $$ > ${SCRIPT_LOCK}


##### Archive logs older than 1 days, gzip older than 1 days and delete them if older than 365 days ######
for LOC in $LOC1
do
ls $LOC/archive > /dev/null
if [ "$?" != "0" ]
then
mkdir $LOC/archive
else
echo "archive directory is already created to move logs files older than 1 day"
fi
cd $LOC
if [ "$?" == "0" ]; then
ls -F .|grep -v /| xargs -I {} find ./{} -mtime +1 -exec mv {} archive \;
fi
cd $LOC/archive
if [ "$?" == "0" ]; then
ls -F .|grep -v /| xargs -I {} find ./{} -mtime +1 -exec gzip {} \;
ls -F .|grep -v /| xargs -I {} find ./{} -mtime +365 -exec rm {} \;
fi
done

##################### Realease Lock file for the script ####################################
cd $SCRIPTHOME
rm logrotate.sh.lock


Please correct me if i need to change in the above script

 

[wizardfromoz]

g'day @iamsaicharanburle and welcome to linux.org

i can't help with your question, but i moving this thread to Command Line, where questions on scripting are answered.

good luck

chris turner
wizardfromoz

 

[iamsaicharanburle]

Hi There,

Can i get any assistance in this regard.

 

[wizardfromoz]

you are dealing with volunteers from around the world on different timezones.

be patient please and someone will come along.

thank you

wizard

 

[JasKinasis]

Hmmm, I'm not sure what's going on there.
It may have something to do with where you are not quoting your variables when you dereference them e.g. using $LOC instead of "$LOC"

But from taking a quick look, your script could be simplified a little bit.
Things like checking the archive directory exists.
Instead of this:

ls $LOC/archive > /dev/null
if [ "$?" != "0" ]
then
mkdir $LOC/archive
else
echo "archive directory is already created to move logs files older than 1 day"
fi

You could simply do this:

if [ ! -d "$LOC/archive" ]
then
    mkdir "$LOC/archive"
else
    echo "$LOC/archive is already created to move logs files older than 1 day"
fi

And also where you are using cd and ls, with xargs with find, I think you can just get away with using find on it's own.

This is a slightly modified version of your script:

#!/bin/bash
############################
# LOGS HOUSEKEEPING SCRIPT
# CREATED BY: Saicharan Burle
# LAST MODIFIED BY: JasKinasis (from linux.org)

############################
TIME="$(date +%m%d%Y%H%M)" # FYI - this is currently unused!
SCRIPTHOME="/home/ec2-user/scripts"

# Create a temporary directory in /tmp/
TMPHOME=$(mktemp -d)

# Lock-file
SCRIPT_LOCK="$SCRIPTHOME/$0.lock"

##### Directories to search (also include in the for loop below) ######
LOC1="$SCRIPTHOME/log/"

##### Verify if script is already running #####
if [ -s "$SCRIPT_LOCK" ] ; then
    echo -e "Script is already running... Aborting!"
    exit 1
fi

##### Create Lock file for the Script #####
echo "$$" > "$SCRIPT_LOCK"


##### Archive logs older than 1 days, gzip older than 1 days and delete them if older than 365 days ######

# identify sub-directories in
find "$LOC1" -maxdepth 1 -type d | grep -v "archive" > "$TMPHOME/LOCS"

while read -r LOC
do
    if [ -d "$LOC" ] ; then
        if [ ! -d "$LOC/archive" ] ; then
            mkdir "$LOC/archive"
        else
            echo "$LOC/archive is already created to move logs files older than 1 day"
        fi
        find "$LOC" -maxdepth 1 -type f -mtime +1 -exec gzip {} \; -exec mv {}.gz "$LOC/archive" \;
        find "$LOC/archive" -mtime +365 -delete
    fi
done < "$TMPHOME/LOCS"


##################### Release Lock file for the script ####################################
rm "$SCRIPT_LOCK"
# remove temporary dir
rm -r "$TMPHOME"

The initial part of the script is more or less the same.
But the biggest difference is that it uses find to create a list of top level sub-directories in $LOC1 and stores it in a temporary file in a directory in /tmp/ (created at the start of the script via mktemp).
The grep is to exclude "$LOC/archive" - if it already exists.
We don't want to end up making an archive sub-directory in there!

And because we've constrained the search with a maxdepth of 1 - the first find command would find locations like:
$LOC1
$LOC1/script1
$LOC1/script2
$LOC1/script3
...

So those locations are written out to a temporary file in our temp directory ($TEMPHOME), which will be a sub-directory in /tmp/.
Then we use a "while read" loop to read the list of locations from the temporary file.

For each location/subdirectory ($LOC)- we check to see if it has an archive directory ($LOC/archive). If there is no archive directory - it is created.

So for each location identified above, there will be an archive sub-directory created (if one doesn't already exist):
e.g.
$LOC1/archive
$LOC1/script1/archive
$LOC1/script2/archive
$LOC1/script3/archive
...

Next, we use find to search the current "$LOC" for any files that are over 1 day old.
For any files that meet the criteria - we use two -exec commands. - The first zips the file using gzip.
- The second moves the zipped file into "$LOC/archive".

Then we search the current "$LOC/archive" directory for any files that are over 365 days old and delete them using the -delete option, rather than using -exec with rm.

That process is repeated for all locations that are listed in our temporary file.

Finally, we remove the lock-file and the temporary directory we created (which will also remove the temporary file!).

I hope these changes fit your needs!
Any problems, or questions - by all means post them here!

 

[iamsaicharanburle]

Hi Jas,

Thank you for your response. Just to give some update to my initial post. I've found the root cause like why it is throwing the error and its because the file names with permissions 755. Whatever the find command that i used with the combination of ls and xargs are ignoring the files which has permission 755 and considering only 664 . So i went ahead and modified my script something like below added one more find command which will change the permissions to 664 and then followed by the ls commands.

#!/bin/bash
############################
# LOGS HOUSEKEEPING SCRIPT
# CREATED BY: Saicharan Burle
# LAST MODIFIED BY: Saicharan Burle : Added Archiving

############################
TIME=`date +%m%d%Y%H%M`
SCRIPTHOME="/home/ec2-user/scripts"

SCRIPT_LOCK=${SCRIPTHOME}/$0.lock

##### Directories to search (also include in the for loop below) ######
LOC1="/home/ec2-user/scripts/log"

##### Verify if script is already running #####
if [ -s ${SCRIPT_LOCK} ]
then
echo -e "Script is already running... Aborting!"
exit 1
fi

##### Create Lock file for the Script #####
echo $$ > ${SCRIPT_LOCK}


##### Archive logs older than 1 days, gzip older than 1 days and delete them if older than 365 days ######
for LOC in $LOC1
do
ls $LOC/archive > /dev/null
if [ "$?" != "0" ]
then
mkdir $LOC/archive
else
echo "archive directory is already created to move logs files older than 1 day"
fi
cd $LOC
if [ "$?" == "0" ]; then
find . -type f -name "*.log" -exec chmod 664 {} \;
ls -F .|grep -v /| xargs -I {} find ./{} -mtime +1 -exec mv {} archive \;
fi
cd $LOC/archive
if [ "$?" == "0" ]; then
ls -F .|grep -v /| xargs -I {} find ./{} -mtime +1 -exec gzip {} \;
ls -F .|grep -v /| xargs -I {} find ./{} -mtime +365 -exec rm {} \;
fi
done

##################### Realease Lock file for the script ####################################
cd $SCRIPTHOME
rm logrotate.sh.lock



Also, Thank you for your script i will test it again using your script.

One more thing i wanted is if i want to add one more log location to the same script can i just simply give another location of the logs

something like
LOC2="/home/ec2-user/scripts/log2"

 

[JasKinasis]

With my script, you could add LOC2, with the rest of the variables in your script and then add another find immediately after the first one and before the read loop that will append results to the temporary list of locations.
e.g.

find "$LOC2" -maxdepth 1 -type d | grep -v "archive" >> "$TMPHOME/LOCS"

Or in your script, you could perhaps turn the block of code which deals with the locations into a function and then call it twice passing $LOC1 the first time and $LOC2 the second.

e.g.

##### Archive logs older than 1 days, gzip older than 1 days and delete them if older than 365 days ######
function processLogs()
{
    for LOC in $1
    do
        ls "$LOC/archive" > /dev/null
        if [ "$?" != "0" ]
        then
            mkdir "$LOC/archive"
        else
            echo "archive directory is already created to move logs files older than 1 day"
        fi
        cd "$LOC"
        if [ "$?" == "0" ]; then
            ls -F .|grep -v /| xargs -I {} find ./{} -mtime +1 -exec mv {} archive \;
        fi
        cd "$LOC/archive"
        if [ "$?" == "0" ]; then
        ls -F .|grep -v /| xargs -I {} find ./{} -mtime +1 -exec gzip {} \;
        ls -F .|grep -v /| xargs -I {} find ./{} -mtime +365 -exec rm {} \;
        fi
    done
}

processLogs "$LOC1"
processLogs "$LOC2"

That's definitely an option!

You could even change my version to use a function like this:

##### Archive logs older than 1 days, gzip older than 1 days and delete them if older than 365 days ######
function processLogs()
{
    find "$1" -maxdepth 1 -type d | grep -v "archive" > "$TMPHOME/LOCS"

    while read -r LOC
    do
        if [ -d "$LOC" ] ; then
            if [ ! -d "$LOC/archive" ] ; then
                mkdir "$LOC/archive"
            else
                echo "$LOC/archive already exists for logs older than 1 day"
            fi
            find "$LOC" -maxdepth 1 -type f -mtime +1 -exec gzip {} \; -exec mv {}.gz "$LOC/archive" \;
            find "$LOC/archive" -mtime +365 -delete
        fi
    done < "$TMPHOME/LOCS"
}

processLogs "$LOC1"
processLogs "$LOC2"

Each time the function is called, it would re-write the temporary list of locations and then process those locations.

I haven't got time to test ANY of the above snippets because I have work soon.
But it should work, in theory! :/

 

[iamsaicharanburle]

Hi Jas,

Basically i tried this way with your script and it worked too

find "$LOC1" "$LOC2" -maxdepth 1 -type d | grep -v "archive" > "$TMPHOME/LOCS"

Added LOC2 variable and used the same in the find command above.

Thanks Much for your quick assistance in this regard.

Will come back to you in case of any additional requirement

 

[JasKinasis]

Hi Jas,

Basically i tried this way with your script and it worked too

find "$LOC1" "$LOC2" -maxdepth 1 -type d | grep -v "archive" > "$TMPHOME/LOCS"

Added LOC2 variable and used the same in the find command above.

Thanks Much for your quick assistance in this regard.

Will come back to you in case of any additional requirement

Good point! I forgot you could put multiple searches in a single find command!
That’s an even better option!

And no worries! Glad to have helped.
I’m mainly here to answer programming and scripting related questions. I like a good programming puzzle.