Hello, I wanted to share my experience how I have rather successfully eliminated intensive bot traffic from many ASNs and subnets using free Cloudflare account + I would like your feedback on what that traffic is and if i could do it better way, maybe even without Cloudflare...
My vbulletin4 forum website went down due to a CPU limit reached on a shared hosting account.
According to a logs, the reason was many visitors, sometimes several per second.
They did not come from the same IP or User-Agent, nor similar subnets /16 or /24 but many ASNs, most visits (according to Webalizer stats - i have pulled these into Calc app and sorted) I have noticed from:
HostPapa
RackNerd
Web2Objects
I had no better budget friendly idea than setting Interactive challenge (captcha) for visits which seemed resource intensive and rarely used by regular visitors. These rules were set at https://dash.cloudflare.com/idhere/mysite.com/security/security-rules
like this:
"When incoming requests match…"
Field = URI Path, Operator = wildcard, Value = /tags.php*
Field = AS Num, Operator = equals, Value = 36352
Full expression:
at that page /security/security-rules, i have also set another rule for Interactive challenge for IPs/subnets listed in https://dash.cloudflare.com/idhere/configurations/lists/idhere :
Field = IP Source Address, Operator = is in list, Value = blockedips
(blockedips is name of my list)
what really made a change in my logs (in terms of a traffic reduction) was that last rule: http.request.uri wildcard r"/.php*s="
Sample traffic before applying the rule:
Note that also as a next measure to reduce bot flood I am using .htaccess firewalls:
perishablepress.com
and
perishablepress.com
and having set crawl-delay in robots.txt (most bots does not respect, but such bots may be at least reported)
My vbulletin4 forum website went down due to a CPU limit reached on a shared hosting account.
According to a logs, the reason was many visitors, sometimes several per second.
They did not come from the same IP or User-Agent, nor similar subnets /16 or /24 but many ASNs, most visits (according to Webalizer stats - i have pulled these into Calc app and sorted) I have noticed from:
HostPapa
RackNerd
Web2Objects
I had no better budget friendly idea than setting Interactive challenge (captcha) for visits which seemed resource intensive and rarely used by regular visitors. These rules were set at https://dash.cloudflare.com/idhere/mysite.com/security/security-rules
like this:
"When incoming requests match…"
Field = URI Path, Operator = wildcard, Value = /tags.php*
Field = AS Num, Operator = equals, Value = 36352
Full expression:
at that page /security/security-rules, i have also set another rule for Interactive challenge for IPs/subnets listed in https://dash.cloudflare.com/idhere/configurations/lists/idhere :
Field = IP Source Address, Operator = is in list, Value = blockedips
(blockedips is name of my list)
what really made a change in my logs (in terms of a traffic reduction) was that last rule: http.request.uri wildcard r"/.php*s="
Sample traffic before applying the rule:
am I blocking legitimate traffic using that s= rule? I have used it because when I am browsing the site as a human, i do not see these s= in URLs nor in logs near my visits.
Note that also as a next measure to reduce bot flood I am using .htaccess firewalls:
8G Firewall | Perishable Press
After more than a year of beta testing, 8G Firewall is ready for use on production sites. So you can benefit from the powerful protection provided by the...
perishablepress.com
Ultimate Block List to Stop AI Bots | Perishable Press
More than you might think, AI (Artificial Intelligence) and ML (Machine Learning) bots are crawling your site and scraping your content. They are...
perishablepress.com
and having set crawl-delay in robots.txt (most bots does not respect, but such bots may be at least reported)
Last edited:
