useful commands for apache logs

Discussion in 'Command Line' started by tomfmason, Nov 3, 2011.

  1. tomfmason

    tomfmason New Member Staff Writer

    Messages:
    22
    Likes Received:
    5
    Trophy Points:
    0
    These assume you use a standard log format. If not you may have to adjust them to suit your format

    Most viewed pages(top ten).
    Code:
    awk '{print $7}' /path/to/log |sort |uniq -c |sort -rn |head -10
    Top ten referrers:
    Code:
    awk '{print $11}' /path/to/log |sort |uniq -c |sort -rn |head -10
    Search logs:
    Code:
    grep /path/to/log query |awk '{print $8}'|tail -n+5|sort|uniq|tr -d [1]
    And here is a script I wrote a long time ago that incorparates these and few others


    Code:
    #!/bin/bash
    # usage 
    #    ./this_script search pattern log_file
    #       all hosts with 5 or more matches of the given pattern will be banned
    #    ./this_script ban_from_log log_file
    #       all hosts that appear more than 5 times in the given log file will be banned
    #    ./this_script ban_rfi log_file
    #       bans all hosts that match the rfi pattern(rfi's and proxy requests)
    #    ./this_script most_viewed log_file
    #       shows the top ten viewed pages with the number of views
    #    ./this_script statuses response log_file
    #       shows the top ten viewed pages for the given response header e.g. 404, 200 ect
    #    ./this_script referrers log_file 
    #       shows the top ten referrers and page views for each
    #
    # author tomfmason
    ban_file=/etc/hosts.deny
    
    function ban_ip() {
      exists=`grep ${1} $ban_file`
      if [ ! "$exists" ]; then
         echo "ALL: ${1}" >> $ban_file
      fi
    }
    
    function search() {
      ret=`grep ${1} ${2} |awk '{print $8}'|tail -n+5|sort|uniq|tr -d [1]`
      for r in $ret; do
          ban_ip $r
      done
    }
    
    function ban_from_log() {
      ret=`awk '{print $8}' ${1}|tail -n+5|sort|uniq|tr -d [1]`
      for r in $ret; do
         ban_ip $r
      done
    }
    
    function ban_rfi() {
      ret=`awk '{print $1 " " $7}' ${1} |grep -iE '(http|https|ftp)'|awk '{print $1}'|tail -n+5|sort|uniq`
      for r in $ret; do
        ban_ip $r
      done
    }
    
    function most_viewed() {
        awk '{print $7}' ${1} |sort |uniq -c |sort -rn |head -10
    }
    
    function statuses() {
       awk '{print $7 " " $9}' ${2} |grep -iE '${1}' |sort | uniq -c | sort -rn |awk '{print $1 "  " $2 }' |head -10
    }
    
    function referrers() {
        awk '{print $11}' ${1} |sort |uniq -c |sort -rn |head -10
    }
    
    if type "$1" | grep -qF "$1 is a function"; then "$@"; fi
    
    I wrote that a long time ago and now I would not suggest using hosts.deny to ban an ip. Iptables would be a much better choice. I am just to lazy to fix it ;)
  2. MustangV10

    MustangV10 New Member

    Messages:
    87
    Likes Received:
    2
    Trophy Points:
    0
    Some interesting commands. I currently run LSWS (litespeed) on my server. Would these commands still apply for that? I know litespeed is httpd, but just curious.
  3. tomfmason

    tomfmason New Member Staff Writer

    Messages:
    22
    Likes Received:
    5
    Trophy Points:
    0
    I am sure it would work but you will have to adjust for the different log format.

Share This Page