会python真的可以为所欲为0(回)
396天前
这里还有人吗1(回)
581天前这里还有人吗0(回)
581天前
每天面对着电脑屏幕，敲打键盘。我所面对的并不只是代码，而是一种生活方式。0(回)
806天前到处都是羊，不想上班0(回)
886天前
鸽子
0(回)
890天前#----------------------------- |
use LWP::RobotUA; |
$ua = LWP::RobotUA->new ( 'websnuffler/0.1', 'me@wherever.com' ); |
#----------------------------- |
403 ( Forbidden ) Forbidden by robots.txt |
#----------------------------- |
#% GET http://www.webtechniques.com/robots.txt |
#User-agent: * |
# |
# Disallow: /stats |
# |
# Disallow: /db |
# |
# Disallow: /logs |
# |
# Disallow: /store |
# |
# Disallow: /forms |
# |
# Disallow: /gifs |
# |
# Disallow: /wais-src |
# |
# Disallow: /scripts |
# |
# Disallow: /config |
#----------------------------- |
#% GET http://www.cnn.com/robots.txt | head |
## robots, scram |
# |
## $I d : robots.txt,v 1.2 1998/03/10 18:27:01 mreed Exp $ |
# |
#User-agent: * |
# |
#Disallow: / |
# |
#User-agent: Mozilla/3.01 (hotwired-test/0.1) |
# |
#Disallow: /cgi-bin |
# |
#Disallow: /TRANSCRIPTS |
# |
#Disallow: /development |
#----------------------------- |
by: 发表于：2017-09-15 16:58:08 顶(0) | 踩(0) 回复
？？
回复评论