apache、iis6、ii7独立ip主机屏蔽拦截蜘蛛抓取（适用vps云主机服务器）

大哥大 · 发表于 2025-7-12 15:21:05

亲注册登录道勤网-可以查看更多帖子内容哦！（包涵精彩图片、文字详情等）请您及时注册登录-www.daoqin.net

您需要登录才可以下载或查看，没有账号？立即注册

x

如果是正常的搜索引擎蜘蛛访问，不建议对蜘蛛进行禁止，否则网站在百度等搜索引擎中的收录和排名将会丢失，造成客户流失等损失。可以优先考虑升级虚拟主机型号以获得更多的流量或升级为云服务器（不限流量）。更多详情请访问： http://www.west.cn/faq/list.asp?unid=626

linux下规则文件.htaccess(手工创建.htaccess文件到站点根目录)

<IfModule mod_rewrite.c>
RewriteEngine On
#Block spider
RewriteCond %{HTTP_USER_AGENT} "Bytespider|Amazonbot|YisouSpider|ClaudeBot|GPTBot|meta-externalagent|SemrushBot|DotBot|BLEXBot|SMTBot|PetalBot|Apache-HttpClient|SemrushBot|Webdup|AcoonBot|AhrefsBot|Ezooms|EdisterBot|EC2LinkFinder|jikespider|Purebot|MJ12bot|WangIDSpider|WBSearchBot|Wotbox|xbfMozilla|Yottaa|YandexBot|Jorgee|SWEBot|spbot|TurnitinBot-Agent|mail.RU|curl|perl|Python|Wget|Xenu|ZmEu" [NC]
RewriteRule !(^robots\.txt$) - [F]
</IfModule>

复制代码

windows2003下规则文件httpd.conf

#Block spider
RewriteCond %{HTTP_USER_AGENT} (Bytespider|Amazonbot|YisouSpider|ClaudeBot|GPTBot|meta-externalagent|SemrushBot|DotBot|BLEXBot|SMTBot|PetalBot|Apache-HttpClient|SemrushBot|Webdup|AcoonBot|AhrefsBot|Ezooms|EdisterBot|EC2LinkFinder|jikespider|Purebot|MJ12bot|WangIDSpider|WBSearchBot|Wotbox|xbfMozilla|Yottaa|YandexBot|Jorgee|SWEBot|spbot|TurnitinBot-Agent|mail.RU|curl|perl|Python|Wget|Xenu|ZmEu) [NC]
RewriteRule !(^/robots.txt$) - [F]

复制代码

windows2008下 web.config

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<system.webServer>
<rewrite>
<rules>
<rule name="Block spider">
<match url="(^robots.txt$)" ignoreCase="false" negate="true" />
<conditions>
<add input="{HTTP_USER_AGENT}" pattern="Bytespider|Amazonbot|YisouSpider|ClaudeBot|GPTBot|meta-externalagent|SemrushBot|DotBot|BLEXBot|SMTBot|PetalBot|Apache-HttpClient|SemrushBot|Webdup|AcoonBot|AhrefsBot|Ezooms|EdisterBot|EC2LinkFinder|jikespider|Purebot|MJ12bot|WangIDSpider|WBSearchBot|Wotbox|xbfMozilla|Yottaa|YandexBot|Jorgee|SWEBot|spbot|TurnitinBot-Agent|curl|perl|Python|Wget|Xenu|ZmEu" ignoreCase="true" />
</conditions>
<action type="AbortRequest" />
</rule>
</rules>
</rewrite>
</system.webServer>
</configuration>

复制代码

Nginx对应屏蔽规则

代码需添加到对应站点配置文件server段内

if ($http_user_agent ~* "Bytespider|Amazonbot|YisouSpider|ClaudeBot|GPTBot|meta-externalagent|SemrushBot|DotBot|BLEXBot|SMTBot|PetalBot|Apache-HttpClient|Bytespider|Java|PhantomJS|SemrushBot|Scrapy|Webdup|AcoonBot|AhrefsBot|Ezooms|EdisterBot|EC2LinkFinder|jikespider|Purebot|MJ12bot|WangIDSpider|WBSearchBot|Wotbox|xbfMozilla|Yottaa|YandexBot|Jorgee|SWEBot|spbot|TurnitinBot-Agent|mail.RU|perl|Python|Wget|Xenu|ZmEu|^$" )
{
return 444;
}

复制代码

注：规则中默认屏蔽部分不明蜘蛛，要屏蔽其他蜘蛛按规则添加即可

附各大蜘蛛名字：

google蜘蛛：googlebot

百度蜘蛛：baiduspider

百度手机蜘蛛：baiduboxapp

yahoo蜘蛛：slurp

alexa蜘蛛：ia_archiver

msn蜘蛛：msnbot

bing蜘蛛：bingbot

altavista蜘蛛：scooter

lycos蜘蛛：lycos_spider_(t-rex)

alltheweb蜘蛛：fast-webcrawler

inktomi蜘蛛：slurp

有道蜘蛛：YodaoBot和OutfoxBot

热土蜘蛛：Adminrtspider

搜狗蜘蛛：sogou spider

SOSO蜘蛛：sosospider

360搜蜘蛛：360spider

道勤主机提供365天*24小时全年全天无休、实时在线、零等待的售后技术支持。竭力为您免费处理您在使用道勤主机过程中所遇到的一切问题! 如果您是道勤主机用户，那么您可以通过QQ【792472177】、售后QQ【59133755】、旺旺【诠释意念】、微信：q792472177免费电话、后台提交工单这些方式联系道勤主机客服！如果您不是我们的客户也没问题，点击页面最右边的企业QQ在线咨询图标联系我们并购买后，我们为您免费进行无缝搬家服务，让您享受网站零访问延迟的迁移到道勤主机的服务！