Block Bad Bots and Spiders using .htaccess


Below is a useful code block for blocking a lot of the known bad bots and site rippers currently out there.
Simply add the code to your /public_html/.htaccess file:

# Bad bot
SetEnvIfNoCase User-Agent "^abot" bad_bot
SetEnvIfNoCase User-Agent "^aipbot" bad_bot
SetEnvIfNoCase User-Agent "^asterias" bad_bot
SetEnvIfNoCase User-Agent "^EI" bad_bot
SetEnvIfNoCase User-Agent "^libwww-perl" bad_bot
SetEnvIfNoCase User-Agent "^LWP" bad_bot
SetEnvIfNoCase User-Agent "^lwp" bad_bot
SetEnvIfNoCase User-Agent "^MSIECrawler" bad_bot
SetEnvIfNoCase User-Agent "^nameprotect" bad_bot
SetEnvIfNoCase User-Agent "^PlantyNet_WebRobot" bad_bot
SetEnvIfNoCase User-Agent "^UCmore" bad_bot
SetEnvIfNoCase User-Agent "Alligator" bad_bot
SetEnvIfNoCase User-Agent "AllSubmitter" bad_bot
SetEnvIfNoCase User-Agent "Anonymous" bad_bot
SetEnvIfNoCase User-Agent "Asterias" bad_bot
SetEnvIfNoCase User-Agent "autoemailspider" bad_bot
SetEnvIfNoCase User-Agent "Badass" bad_bot
SetEnvIfNoCase User-Agent "Baiduspider" bad_bot
SetEnvIfNoCase User-Agent "BecomeBot" bad_bot
SetEnvIfNoCase User-Agent "Bitacle" bad_bot
SetEnvIfNoCase User-Agent "bladder\ fusion" bad_bot
SetEnvIfNoCase User-Agent "Blogshares\ Spiders" bad_bot
SetEnvIfNoCase User-Agent "Board\ Bot" bad_bot
SetEnvIfNoCase User-Agent "Board\ Bot" bad_bot
SetEnvIfNoCase User-Agent "Convera" bad_bot
SetEnvIfNoCase User-Agent "ConveraMultiMediaCrawler" bad_bot
SetEnvIfNoCase User-Agent "c-spider" bad_bot
SetEnvIfNoCase User-Agent "DA" bad_bot
SetEnvIfNoCase User-Agent "DnloadMage" bad_bot
SetEnvIfNoCase User-Agent "Download\ Demon" bad_bot
SetEnvIfNoCase User-Agent "Download\ Express" bad_bot
SetEnvIfNoCase User-Agent "Download\ Wonder" bad_bot
SetEnvIfNoCase User-Agent "dragonfly" bad_bot
SetEnvIfNoCase User-Agent "DreamPassport" bad_bot
SetEnvIfNoCase User-Agent "DSurf" bad_bot
SetEnvIfNoCase User-Agent "DTS Agent" bad_bot
SetEnvIfNoCase User-Agent "EBrowse" bad_bot
SetEnvIfNoCase User-Agent "eCatch" bad_bot
SetEnvIfNoCase User-Agent "edgeio" bad_bot
SetEnvIfNoCase User-Agent "Email\ Extractor" bad_bot
SetEnvIfNoCase User-Agent "EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "EmailWolf" bad_bot
SetEnvIfNoCase User-Agent "EmeraldShield" bad_bot
SetEnvIfNoCase User-Agent "ESurf" bad_bot
SetEnvIfNoCase User-Agent "Exabot" bad_bot
SetEnvIfNoCase User-Agent "ExtractorPro" bad_bot
SetEnvIfNoCase User-Agent "FileHeap!\ file downloader" bad_bot
SetEnvIfNoCase User-Agent "FileHound" bad_bot
SetEnvIfNoCase User-Agent "Forex" bad_bot
SetEnvIfNoCase User-Agent "Franklin\ Locator" bad_bot
SetEnvIfNoCase User-Agent "FreshDownload" bad_bot
SetEnvIfNoCase User-Agent "FrontPage" bad_bot
SetEnvIfNoCase User-Agent "FSurf" bad_bot
SetEnvIfNoCase User-Agent "Gaisbot" bad_bot
SetEnvIfNoCase User-Agent "Gamespy_Arcade" bad_bot
SetEnvIfNoCase User-Agent "genieBot" bad_bot
SetEnvIfNoCase User-Agent "GetBot" bad_bot
SetEnvIfNoCase User-Agent "GetRight" bad_bot
SetEnvIfNoCase User-Agent "Gigabot" bad_bot
SetEnvIfNoCase User-Agent "Go!Zilla" bad_bot
SetEnvIfNoCase User-Agent "Go-Ahead-Got-It" bad_bot
SetEnvIfNoCase User-Agent "GOFORITBOT" bad_bot
SetEnvIfNoCase User-Agent "heritrix" bad_bot
SetEnvIfNoCase User-Agent "HLoader" bad_bot
SetEnvIfNoCase User-Agent "HooWWWer" bad_bot
SetEnvIfNoCase User-Agent "HTTrack" bad_bot
SetEnvIfNoCase User-Agent "iCCrawler" bad_bot
SetEnvIfNoCase User-Agent "ichiro" bad_bot
SetEnvIfNoCase User-Agent "iGetter" bad_bot
SetEnvIfNoCase User-Agent "imds_monitor" bad_bot
SetEnvIfNoCase User-Agent "Industry\ Program" bad_bot
SetEnvIfNoCase User-Agent "Indy\ Library" bad_bot
SetEnvIfNoCase User-Agent "InetURL" bad_bot
SetEnvIfNoCase User-Agent "InstallShield\ DigitalWizard" bad_bot
SetEnvIfNoCase User-Agent "IRLbot" bad_bot
SetEnvIfNoCase User-Agent "IUPUI\ Research\ Bot" bad_bot
SetEnvIfNoCase User-Agent "Java" bad_bot
SetEnvIfNoCase User-Agent "jeteye" bad_bot
SetEnvIfNoCase User-Agent "jeteyebot" bad_bot
SetEnvIfNoCase User-Agent "JoBo" bad_bot
SetEnvIfNoCase User-Agent "JOC\ Web\ Spider" bad_bot
SetEnvIfNoCase User-Agent "Kapere" bad_bot
SetEnvIfNoCase User-Agent "Larbin" bad_bot
SetEnvIfNoCase User-Agent "LeechGet" bad_bot
SetEnvIfNoCase User-Agent "LightningDownload" bad_bot
SetEnvIfNoCase User-Agent "Linkie" bad_bot
SetEnvIfNoCase User-Agent "Mac\ Finder" bad_bot
SetEnvIfNoCase User-Agent "Mail\ Sweeper" bad_bot
SetEnvIfNoCase User-Agent "Mass\ Downloader" bad_bot
SetEnvIfNoCase User-Agent "MetaProducts\ Download\ Express" bad_bot
SetEnvIfNoCase User-Agent "Microsoft\ Data\ Access" bad_bot
SetEnvIfNoCase User-Agent "Microsoft\ URL\ Control" bad_bot
SetEnvIfNoCase User-Agent "Missauga\ Locate" bad_bot
SetEnvIfNoCase User-Agent "Missauga\ Locator" bad_bot
SetEnvIfNoCase User-Agent "Missigua Locator" bad_bot
SetEnvIfNoCase User-Agent "Missouri\ College\ Browse" bad_bot
SetEnvIfNoCase User-Agent "Mister\ PiX" bad_bot
SetEnvIfNoCase User-Agent "MovableType" bad_bot
SetEnvIfNoCase User-Agent "Mozi!" bad_bot
SetEnvIfNoCase User-Agent "Mozilla/3.0 (compatible)" bad_bot
SetEnvIfNoCase User-Agent "Mozilla/5.0 (compatible; MSIE 5.0)" bad_bot
SetEnvIfNoCase User-Agent "MSIE_6.0" bad_bot
SetEnvIfNoCase User-Agent "MSIECrawler" badbot
SetEnvIfNoCase User-Agent "MVAClient" bad_bot
SetEnvIfNoCase User-Agent "MyFamilyBot" bad_bot
SetEnvIfNoCase User-Agent "MyGetRight" bad_bot
SetEnvIfNoCase User-Agent "NASA\ Search" bad_bot
SetEnvIfNoCase User-Agent "Naver" bad_bot
SetEnvIfNoCase User-Agent "NaverBot" bad_bot
SetEnvIfNoCase User-Agent "NetAnts" bad_bot
SetEnvIfNoCase User-Agent "NetResearchServer" bad_bot
SetEnvIfNoCase User-Agent "NEWT\ ActiveX" bad_bot
SetEnvIfNoCase User-Agent "Nextopia" bad_bot
SetEnvIfNoCase User-Agent "NICErsPRO" bad_bot
SetEnvIfNoCase User-Agent "NimbleCrawler" bad_bot
SetEnvIfNoCase User-Agent "Nitro\ Downloader" bad_bot
SetEnvIfNoCase User-Agent "Nutch" bad_bot
SetEnvIfNoCase User-Agent "Offline\ Explorer" bad_bot
SetEnvIfNoCase User-Agent "OmniExplorer" bad_bot
SetEnvIfNoCase User-Agent "OutfoxBot" bad_bot
SetEnvIfNoCase User-Agent "P3P" bad_bot
SetEnvIfNoCase User-Agent "PagmIEDownload" bad_bot
SetEnvIfNoCase User-Agent "pavuk" bad_bot
SetEnvIfNoCase User-Agent "PHP\ version" bad_bot
SetEnvIfNoCase User-Agent "playstarmusic" bad_bot
SetEnvIfNoCase User-Agent "Program\ Shareware" bad_bot
SetEnvIfNoCase User-Agent "Progressive Download" bad_bot
SetEnvIfNoCase User-Agent "psycheclone" bad_bot
SetEnvIfNoCase User-Agent "puf" bad_bot
SetEnvIfNoCase User-Agent "PussyCat" bad_bot
SetEnvIfNoCase User-Agent "PuxaRapido" bad_bot
SetEnvIfNoCase User-Agent "Python-urllib" bad_bot
SetEnvIfNoCase User-Agent "RealDownload" bad_bot
SetEnvIfNoCase User-Agent "RedKernel" bad_bot
SetEnvIfNoCase User-Agent "relevantnoise" bad_bot
SetEnvIfNoCase User-Agent "RepoMonkey\ Bait\ &\ Tackle" bad_bot
SetEnvIfNoCase User-Agent "RTG30" bad_bot
SetEnvIfNoCase User-Agent "SBIder" bad_bot
SetEnvIfNoCase User-Agent "script" bad_bot
SetEnvIfNoCase User-Agent "Seekbot" bad_bot
SetEnvIfNoCase User-Agent "SiteSnagger" bad_bot
SetEnvIfNoCase User-Agent "SmartDownload" bad_bot
SetEnvIfNoCase User-Agent "sna-" bad_bot
SetEnvIfNoCase User-Agent "Snap\ bot" bad_bot
SetEnvIfNoCase User-Agent "SpeedDownload" bad_bot
SetEnvIfNoCase User-Agent "Sphere" bad_bot
SetEnvIfNoCase User-Agent "sproose" bad_bot
SetEnvIfNoCase User-Agent "SQ\ Webscanner" bad_bot
SetEnvIfNoCase User-Agent "Stamina" bad_bot
SetEnvIfNoCase User-Agent "Star\ Downloader" bad_bot
SetEnvIfNoCase User-Agent "Teleport" bad_bot
SetEnvIfNoCase User-Agent "TurnitinBot" bad_bot
SetEnvIfNoCase User-Agent "UdmSearch" bad_bot
SetEnvIfNoCase User-Agent "URLGetFile" bad_bot
SetEnvIfNoCase User-Agent "User-Agent" bad_bot
SetEnvIfNoCase User-Agent "UtilMind\ HTTPGet" bad_bot
SetEnvIfNoCase User-Agent "WebAuto" bad_bot
SetEnvIfNoCase User-Agent "WebCapture" bad_bot
SetEnvIfNoCase User-Agent "webcollage" bad_bot
SetEnvIfNoCase User-Agent "WebCopier" bad_bot
SetEnvIfNoCase User-Agent "WebFilter" bad_bot
SetEnvIfNoCase User-Agent "WebReaper" bad_bot
SetEnvIfNoCase User-Agent "Website\ eXtractor" bad_bot
SetEnvIfNoCase User-Agent "WebStripper" bad_bot
SetEnvIfNoCase User-Agent "WebZIP" bad_bot
SetEnvIfNoCase User-Agent "Wells\ Search" bad_bot
SetEnvIfNoCase User-Agent "WEP\ Search\ 00" bad_bot
SetEnvIfNoCase User-Agent "Wget" bad_bot
SetEnvIfNoCase User-Agent "Wildsoft\ Surfer" bad_bot
SetEnvIfNoCase User-Agent "WinHttpRequest" bad_bot
SetEnvIfNoCase User-Agent "WWWOFFLE" bad_bot
SetEnvIfNoCase User-Agent "Xaldon\ WebSpider" bad_bot
SetEnvIfNoCase User-Agent "Y!TunnelPro" bad_bot
SetEnvIfNoCase User-Agent "YahooYSMcm" bad_bot
SetEnvIfNoCase User-Agent "Zade" bad_bot
SetEnvIfNoCase User-Agent "ZBot" bad_bot
SetEnvIfNoCase User-Agent "zerxbot" bad_bot

<Limit GET POST>
order allow,deny
allow from all
deny from env=bad_bot
</Limit>

Was dit antwoord nuttig?

 Print dit artikel

Gerelateerde artikelen

.htaccess RewriteRule Examples

Example 1 Original URL: http://www.domain.com/product.php?id=15 Rewritten URL:...

Redirect non-www to www over http and https

Use the following .htaccess code to redirect non-www URLs to www over http and https...

Set Timezone

Sometimes, when you using date or mktime function in php, it will show you a funny message...