useful commands for apache logs

[tomfmason]

These assume you use a standard log format. If not you may have to adjust them to suit your format

Most viewed pages(top ten).

awk '{print $7}' /path/to/log |sort |uniq -c |sort -rn |head -10

Top ten referrers:

awk '{print $11}' /path/to/log |sort |uniq -c |sort -rn |head -10

Search logs:

grep /path/to/log query |awk '{print $8}'|tail -n+5|sort|uniq|tr -d [1]

And here is a script I wrote a long time ago that incorparates these and few others

#!/bin/bash
# usage 
#    ./this_script search pattern log_file
#       all hosts with 5 or more matches of the given pattern will be banned
#    ./this_script ban_from_log log_file
#       all hosts that appear more than 5 times in the given log file will be banned
#    ./this_script ban_rfi log_file
#       bans all hosts that match the rfi pattern(rfi's and proxy requests)
#    ./this_script most_viewed log_file
#       shows the top ten viewed pages with the number of views
#    ./this_script statuses response log_file
#       shows the top ten viewed pages for the given response header e.g. 404, 200 ect
#    ./this_script referrers log_file 
#       shows the top ten referrers and page views for each
#
# author tomfmason
ban_file=/etc/hosts.deny

function ban_ip() {
  exists=`grep ${1} $ban_file`
  if [ ! "$exists" ]; then
     echo "ALL: ${1}" >> $ban_file
  fi
}

function search() {
  ret=`grep ${1} ${2} |awk '{print $8}'|tail -n+5|sort|uniq|tr -d [1]`
  for r in $ret; do
      ban_ip $r
  done
}

function ban_from_log() {
  ret=`awk '{print $8}' ${1}|tail -n+5|sort|uniq|tr -d [1]`
  for r in $ret; do
     ban_ip $r
  done
}

function ban_rfi() {
  ret=`awk '{print $1 " " $7}' ${1} |grep -iE '(http|https|ftp)'|awk '{print $1}'|tail -n+5|sort|uniq`
  for r in $ret; do
    ban_ip $r
  done
}

function most_viewed() {
    awk '{print $7}' ${1} |sort |uniq -c |sort -rn |head -10
}

function statuses() {
   awk '{print $7 " " $9}' ${2} |grep -iE '${1}' |sort | uniq -c | sort -rn |awk '{print $1 "  " $2 }' |head -10
}

function referrers() {
    awk '{print $11}' ${1} |sort |uniq -c |sort -rn |head -10
}

if type "$1" | grep -qF "$1 is a function"; then "[email protected]"; fi

I wrote that a long time ago and now I would not suggest using hosts.deny to ban an ip. Iptables would be a much better choice. I am just to lazy to fix it

 

[MustangV10]

Some interesting commands. I currently run LSWS (litespeed) on my server. Would these commands still apply for that? I know litespeed is httpd, but just curious.

 

[tomfmason]

Some interesting commands. I currently run LSWS (litespeed) on my server. Would these commands still apply for that? I know litespeed is httpd, but just curious.

I am sure it would work but you will have to adjust for the different log format.