# www.ok4s.com # 按照robots.txt的标准写法,规定一些不允许爬虫爬的页面或目录。 # robots.txt 的写法参照 # Format is: # User-agent: # Disallow: | # ----------------------------------------------------------------------------- User-agent: * Disallow: /bin/ Disallow: /a_dmi_n/ Disallow: /adv/ Disallow: /common/ Disallow: /service/ Disallow: /DSSI/ Disallow: /editor/ Disallow: /img/ Disallow: /ali/news/ Disallow: /member/ Disallow: /IsapiRewrite/ Disallow: /inc/ Disallow: /template/ Disallow: /skins/ Disallow: /js/ Disallow: /blog_dir/ #圈子 Disallow: /club/edit/ Disallow: /club/editor/ #bbs Disallow: /bbs/post.asp Disallow: /bbs/postedit.asp Disallow: /bbs/reply.asp