Purchase Linux CDs / DVDs / Flash Drives at OSDisc.com

Welcome to Our Community

While Linux.org has been around for a while, we recently changed management and had to purge most of the content (including users). If you signed up before April 23rd please sign up again. Thanks!

useful commands for apache logs

Discussion in 'Command Line' started by tomfmason, Nov 3, 2011.

  1. tomfmason

    tomfmason Guest

    These assume you use a standard log format. If not you may have to adjust them to suit your format

    Most viewed pages(top ten).
    Code:
    awk '{print $7}' /path/to/log |sort |uniq -c |sort -rn |head -10
    Top ten referrers:
    Code:
    awk '{print $11}' /path/to/log |sort |uniq -c |sort -rn |head -10
    Search logs:
    Code:
    grep /path/to/log query |awk '{print $8}'|tail -n+5|sort|uniq|tr -d [1]
    And here is a script I wrote a long time ago that incorparates these and few others



    Code:
    #!/bin/bash
    # usage 
    #    ./this_script search pattern log_file
    #       all hosts with 5 or more matches of the given pattern will be banned
    #    ./this_script ban_from_log log_file
    #       all hosts that appear more than 5 times in the given log file will be banned
    #    ./this_script ban_rfi log_file
    #       bans all hosts that match the rfi pattern(rfi's and proxy requests)
    #    ./this_script most_viewed log_file
    #       shows the top ten viewed pages with the number of views
    #    ./this_script statuses response log_file
    #       shows the top ten viewed pages for the given response header e.g. 404, 200 ect
    #    ./this_script referrers log_file 
    #       shows the top ten referrers and page views for each
    #
    # author tomfmason
    ban_file=/etc/hosts.deny
    
    function ban_ip() {
      exists=`grep ${1} $ban_file`
      if [ ! "$exists" ]; then
         echo "ALL: ${1}" >> $ban_file
      fi
    }
    
    function search() {
      ret=`grep ${1} ${2} |awk '{print $8}'|tail -n+5|sort|uniq|tr -d [1]`
      for r in $ret; do
          ban_ip $r
      done
    }
    
    function ban_from_log() {
      ret=`awk '{print $8}' ${1}|tail -n+5|sort|uniq|tr -d [1]`
      for r in $ret; do
         ban_ip $r
      done
    }
    
    function ban_rfi() {
      ret=`awk '{print $1 " " $7}' ${1} |grep -iE '(http|https|ftp)'|awk '{print $1}'|tail -n+5|sort|uniq`
      for r in $ret; do
        ban_ip $r
      done
    }
    
    function most_viewed() {
        awk '{print $7}' ${1} |sort |uniq -c |sort -rn |head -10
    }
    
    function statuses() {
       awk '{print $7 " " $9}' ${2} |grep -iE '${1}' |sort | uniq -c | sort -rn |awk '{print $1 "  " $2 }' |head -10
    }
    
    function referrers() {
        awk '{print $11}' ${1} |sort |uniq -c |sort -rn |head -10
    }
    
    if type "$1" | grep -qF "$1 is a function"; then "$@"; fi
    
    I wrote that a long time ago and now I would not suggest using hosts.deny to ban an ip. Iptables would be a much better choice. I am just to lazy to fix it ;)
     
  2. MustangV10

    MustangV10 Guest

    Some interesting commands. I currently run LSWS (litespeed) on my server. Would these commands still apply for that? I know litespeed is httpd, but just curious.
     
  3. tomfmason

    tomfmason Guest

    I am sure it would work but you will have to adjust for the different log format.
     

Share This Page