useful commands for apache logs

T

tomfmason

Guest
These assume you use a standard log format. If not you may have to adjust them to suit your format

Most viewed pages(top ten).
Code:
awk '{print $7}' /path/to/log |sort |uniq -c |sort -rn |head -10

Top ten referrers:
Code:
awk '{print $11}' /path/to/log |sort |uniq -c |sort -rn |head -10

Search logs:
Code:
grep /path/to/log query |awk '{print $8}'|tail -n+5|sort|uniq|tr -d [1]

And here is a script I wrote a long time ago that incorparates these and few others

Code:
#!/bin/bash
# usage 
#    ./this_script search pattern log_file
#       all hosts with 5 or more matches of the given pattern will be banned
#    ./this_script ban_from_log log_file
#       all hosts that appear more than 5 times in the given log file will be banned
#    ./this_script ban_rfi log_file
#       bans all hosts that match the rfi pattern(rfi's and proxy requests)
#    ./this_script most_viewed log_file
#       shows the top ten viewed pages with the number of views
#    ./this_script statuses response log_file
#       shows the top ten viewed pages for the given response header e.g. 404, 200 ect
#    ./this_script referrers log_file 
#       shows the top ten referrers and page views for each
#
# author tomfmason
ban_file=/etc/hosts.deny

function ban_ip() {
  exists=`grep ${1} $ban_file`
  if [ ! "$exists" ]; then
     echo "ALL: ${1}" >> $ban_file
  fi
}

function search() {
  ret=`grep ${1} ${2} |awk '{print $8}'|tail -n+5|sort|uniq|tr -d [1]`
  for r in $ret; do
      ban_ip $r
  done
}

function ban_from_log() {
  ret=`awk '{print $8}' ${1}|tail -n+5|sort|uniq|tr -d [1]`
  for r in $ret; do
     ban_ip $r
  done
}

function ban_rfi() {
  ret=`awk '{print $1 " " $7}' ${1} |grep -iE '(http|https|ftp)'|awk '{print $1}'|tail -n+5|sort|uniq`
  for r in $ret; do
    ban_ip $r
  done
}

function most_viewed() {
    awk '{print $7}' ${1} |sort |uniq -c |sort -rn |head -10
}

function statuses() {
   awk '{print $7 " " $9}' ${2} |grep -iE '${1}' |sort | uniq -c | sort -rn |awk '{print $1 "  " $2 }' |head -10
}

function referrers() {
    awk '{print $11}' ${1} |sort |uniq -c |sort -rn |head -10
}

if type "$1" | grep -qF "$1 is a function"; then "[email protected]"; fi

I wrote that a long time ago and now I would not suggest using hosts.deny to ban an ip. Iptables would be a much better choice. I am just to lazy to fix it ;)
 


OP
M

MustangV10

Guest
Some interesting commands. I currently run LSWS (litespeed) on my server. Would these commands still apply for that? I know litespeed is httpd, but just curious.
 
OP
T

tomfmason

Guest
Some interesting commands. I currently run LSWS (litespeed) on my server. Would these commands still apply for that? I know litespeed is httpd, but just curious.

I am sure it would work but you will have to adjust for the different log format.
 
MALIBAL Linux Laptops

Linux Laptops Custom Built for You
MALIBAL is an innovative computer manufacturer that produces high-performance, custom laptops for Linux.

For more info, visit: https://www.malibal.com

Members online


Latest posts

Top