过滤访问日志里的无效ip和robot

过滤访问日志里的无效ip和robot

 

定期更新ip的脚本: 

  www.2cto.com  

#!/bin/sh

#定时更新公司IP, 以用来过滤

#author: Felix Zhang

#date: 2012-12-29

 

filedir=/opt/logdata/companyip

 

adate=$(date -d "today" +"%Y%m%d")

filename="${filedir}/ip.${adate}"

ip=`/usr/bin/host yourcompany.3322.org|awk '{print $4}'`

if [ '' != "`grep $ip ${filename}`" ]; then

    exit 0

fi

 

echo "$ip" >> ${filename}

 

 

#Set how long you want to save

save_days=30

 

#delete 30 days ago nginx log files

find ${filedir} -mtime +${save_days} -exec rm -rf {} \; 

 

分析日志的脚本:  www.2cto.com  

 

#!/bin/sh

ipdir=/opt/logdata/companyip

adate=$(date -d "today" +"%Y%m%d")

ipfile="${ipdir}/ip.${adate}"

ipreg="127.0.0.1"

if [ -e ${ipfile} ]; then

        ipreg=`cat ${ipfile} |sed ':a N;s/\n/|/;ta'`

        echo "1"

fi

if [ "${ipreg}" = ""  ]; then

   ipreg="127.0.0.1"

   echo "2"

fi

echo ${ipreg}

#cat ip.test |grep -E -v  '127.0.0.1|126.23.23.44'

fileName=$1;

echo '分析文件'$fileName

 

cat $fileName | egrep  -v ${ipreg} |awk '{print $7}'

 

这样分析日志时就可以过滤掉自己公司的IP了. 当然根据机器人的特征也可以过滤机器人, 此处不在多说了, 此处仅给出几个机器人

 

cat ${logfile} |grep -E -v ${ipreg} |grep -E -v "DNSPod-monitor|bot.htm|spider.htm|webmasters.htm" >${cleanlogfile}