image.png

1 欢迎爬虫

1.1 SEO-搜索引擎优化

比如谷歌 百度等搜索引擎，这种正规的爬虫会设置明显的User-Agent
谷歌 PC UA
Mozilla/5.0(compatible;Googlebot/2.1;+http://www.google.com/bot.html)

百度 PC UA
Mozilla/5.0(compatible;Baiduspider/2.0;+http://www.baidu.com/search/spider.html)Mozilla/5.0(compatible;Baiduspider-render/2.0;+http://www.baidu.com/search/spider.html)

必应：
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
image.png

1.2 robots.txt —拒绝搜索引擎

是一种存放于网站根目录下的ASCII编码的文本文件，它通常告诉网络搜索引擎的漫游器（又称网络蜘蛛），此网站中的哪些内容是不应被搜索引擎的漫游器获取的，哪些是可以被漫游器获取的

image.png
淘宝：全栈拒绝百度搜索
image.png
image.png
京东：拒绝一淘等比价相关商业类型爬虫
image.png
amazon: 设置的挺全面，最后也把一淘全屏蔽了
User-agent: EtaoSpider
Disallow: /

  1. User-agent: *
  2. Disallow: /exec/obidos/account-access-login
  3. Disallow: /exec/obidos/change-style
  4. Disallow: /exec/obidos/flex-sign-in
  5. Disallow: /exec/obidos/handle-buy-box
  6. Disallow: /exec/obidos/tg/cm/member/
  7. Disallow: /gp/aw/help/id=sss
  8. Disallow: /gp/cart
  9. Disallow: /gp/flex
  10. Disallow: /gp/product/e-mail-friend
  11. Disallow: /gp/product/product-availability
  12. Disallow: /gp/product/rate-this-item
  13. Disallow: /gp/sign-in
  14. Disallow: /gp/reader
  15. Disallow: /gp/sitbv3/reader
  16. Disallow: /gp/richpub/syltguides/create
  17. Disallow: /gp/gfix
  18. Disallow: /gp/associations/wizard.html
  19. Disallow: /gp/dmusic/order
  20. Disallow: /gp/legacy-handle-buy-box.html
  21. Disallow: /gp/aws/ssop
  22. Disallow: /gp/yourstore
  23. Disallow: /gp/gift-central/organizer/add-wishlist
  24. Disallow: /gp/vote
  25. Disallow: /gp/voting/
  26. Disallow: /gp/music/wma-pop-up
  27. Disallow: /gp/customer-images
  28. Disallow: /gp/richpub/listmania/createpipeline
  29. Disallow: /gp/content-form
  30. Disallow: /gp/pdp/invitation/invite
  31. Disallow: /gp/customer-reviews/common/du
  32. Disallow: /gp/customer-reviews/write-a-review.html
  33. Disallow: /gp/associations/wizard.html
  34. Disallow: /gp/music/clipserve
  35. Disallow: /gp/customer-media/upload
  36. Disallow: /gp/history
  37. Disallow: /gp/item-dispatch
  38. Disallow: /gp/dmusic/order/handle-buy-box.html
  39. Disallow: /gp/recsradio
  40. Disallow: /gp/slredirect
  41. Disallow: /dp/shipping/
  42. Disallow: /dp/twister-update/
  43. Disallow: /dp/manual-submit/
  44. Disallow: /dp/e-mail-friend/
  45. Disallow: /dp/product-availability/
  46. Disallow: /dp/rate-this-item/
  47. Disallow: /gp/registry/wishlist/*/reserve
  48. Disallow: /gp/structured-ratings/actions/get-experience.html
  49. Disallow: /gp/twitter/
  50. Disallow: /ap/signin
  51. Disallow: /gp/registry/wishlist/
  52. Disallow: /wishlist/
  53. Allow: /wishlist/universal*
  54. Allow: /wishlist/vendor-button*
  55. Allow: /wishlist/get-button*
  56. Disallow: /gp/wishlist/
  57. Allow: /gp/wishlist/universal*
  58. Allow: /gp/wishlist/vendor-button*
  59. Allow: /gp/wishlist/ipad-install*
  60. Disallow: /registry/wishlist/
  61. Disallow: /review/common/du
  62. Disallow: /gp/registry/search.html
  63. Disallow: /product-reviews/B0069IY63Y
  64. Disallow: /gp/orc/rml/
  65. Disallow: */gcrnsts
  66. Disallow: /gp/gc/widget
  67. Disallow: /gp/dmusic/mp3/player
  68. Disallow: /gp/entity-alert/external
  69. Disallow: /gp/customer-reviews/dynamic/sims-box
  70. Disallow: /review/dynamic/sims-box
  71. Disallow: /gp/redirect.html
  72. Disallow: /gp/twister/ajaxv2
  73. Disallow: /ss/twister/ajax
  74. Disallow: /b?*node=7454917011
  75. Disallow: /b?*node=7454927011
  76. Disallow: /b?*node=7454939011
  77. Disallow: /b?*node=7454898011
  78. Disallow: /gp/customer-media/actions/delete/
  79. Disallow: /gp/customer-media/actions/edit-caption/
  80. Disallow: /gp/dmusic/
  81. Allow: /gp/dmusic/promotions/PrimeMusic
  82. Allow: /gp/dmusic/promotions/AmazonMusicUnlimited
  83. Disallow: /gp/offer-listing/
  84. Disallow: /b?*node=9052533011
  85. Disallow: /lm/R1XIHQVKXSKBNJ
  86. Disallow: /lm/R3HQ5WJSZK6QSO
  87. Disallow: /surprise/
  88. Disallow: /local/ajax/
  89. Disallow: */B00M3E1NYI
  90. Disallow: */B00M3E1Q5Y
  91. Disallow: */B00M3E1TOM
  92. Disallow: */B00M3E1WYO
  93. Disallow: */B00M3E204K
  94. Disallow: */B00M3E236A
  95. Disallow: */B00M3E260I
  96. Disallow: */B00M3E28WO
  97. Disallow: */B00M3E2BC6
  98. Disallow: */B00M3E2DPQ
  99. Disallow: */B00M3E2GU8
  100. Disallow: */B00M3E2J14
  101. Disallow: */B00M3E2LOE
  102. Disallow: */B00M3E1HJY
  103. Disallow: /gp/socialmedia/giveaways
  104. Disallow: /gp/b2b-rd
  105. Disallow: /gp/aw/so.html
  106. Disallow: /gp/rentallist
  107. Disallow: /gp/video/dvd-rental/settings
  108. Disallow: /gp/rl/settings
  109. Disallow: /gp/video/settings
  110. Disallow: /gp/video/library
  111. Disallow: /gp/video/watchlist
  112. Disallow: /reviews/iframe
  113. Disallow: /gp/switch-language
  114. Disallow: /ga/p/
  115. Disallow: /gp/profile/
  116. Disallow: /giveaway/host/setup/
  117. Disallow: /ss/customer-reviews/lighthouse/
  118. Disallow: /ospublishing/story/*
  119. Disallow: /gp/aw/ol/
  120. Disallow: /gp/promotion/
  121. Disallow: /hz/leaderboard/top-reviewers/
  122. Disallow: /creatorhub
  123. Disallow: /creatorhub/*
  124. Disallow: /slp/s$
  125. Disallow: /-/
  126. Allow: /-/es/
  127. Disallow: /hz/help/contact/*/message/$
  128. Disallow: /gp/aw/shoppingAids/
  129. Disallow: /rss/people/*/reviews
  130. Disallow: /gp/pdp/rss/*/reviews
  131. Disallow: /gp/cdp/member-reviews/
  132. Disallow: /gp/aw/cr/
  133. Disallow: */sim/B001132UEE
  134. Allow: /gp/offer-listing/B000
  135. Allow: /gp/offer-listing/9000
  136. Disallow: /gp/aag
  137. Allow: /gp/aag/main?*seller=ABVFEJU8LS620
  138. Disallow: /gp/pdp/profile/
  139. Disallow: /gp/help/customer/express/c2c/
  140. Disallow: /slp/*/b$
  141. Disallow: /hz/contact-us/ajax/initiate-trusted-contact/
  143. User-agent: EtaoSpider
  144. Disallow: /

DHgate：比较依赖google搜索引擎，设置的较全面，把国内的一众搜索引擎全屏蔽了

  1. User-Agent: *
  2. Allow: /*?f=bm
  3. Allow: /*?utm_source
  4. Disallow: /sellerPromise.do
  5. Disallow: /search.do
  6. Disallow: /promoproduct.do
  7. Disallow: /disc.do
  8. Disallow: /dcp/
  9. Disallow: /mydhgate/
  10. Disallow: /loadShipList.do?
  11. Disallow: /logodata.do 
  12. Disallow: /viewcart.do 
  13. Disallow: /omniture/omniture.do
  14. Disallow: /im/dhtalk.do
  15. Disallow: /product/brandreport.do
  16. Disallow: /product/productdisplay.do
  17. Disallow: /product/shippingpayment.do
  18. Disallow: /product/mobileprict.do
  19. Disallow: /product/productguaranteedajax.do
  20. Disallow: /product/inventory.do
  21. Disallow: /product/getstprodinfo.do
  22. Disallow: /product/productreview.do
  23. Disallow: /product/displayorder.do
  24. Disallow: /product/getonlinestatus.do
  25. Disallow: /product/getqrcode.do
  26. Disallow: /cart/
  27. Disallow: /csmsg/
  28. Disallow: /dhrec/
  29. Disallow: /product-images/
  30. Disallow: /wholesale/store/
  31. Disallow: /w/
  32. Disallow: /promotionproduct/
  33. Disallow: /seller-feedback/
  34. Disallow: /!forums/
  35. Disallow: /*?
  38. #
  39. # Crawlers which we'd rather not have.
  40. # 
  42. User-Agent: Googlebot
  44. Allow: /*?f=
  45. Allow: /*?utm_source=
  46. Allow: /*?_escaped_fragment_=
  47. Allow: /*?dspm=
  48. Allow: /*?d1_page_num=
  49. Allow: /*?m=
  50. Allow: /*?yangfaninfo=
  51. Allow: /*?admsg=
  52. Allow: /*?shareToken=
  53. Allow: /*?invitorid=
  54. Allow: /*?from=
  55. Allow: /*?utm_terms=
  61. Disallow: /w/*?dspm=
  62. Disallow: /dcp/*?dspm=
  63. Disallow: /dhrec/*?dspm=
  64. Disallow: /disc.do*?dspm=
  68. Disallow: /sellerPromise.do
  69. Disallow: /search.do
  70. Disallow: /promoproduct.do
  71. Disallow: /disc.do
  72. Disallow: /dcp/
  73. Disallow: /mydhgate/
  74. Disallow: /loadShipList.do?
  75. Disallow: /logodata.do
  76. Disallow: /viewcart.do
  77. Disallow: /!forums/
  78. Disallow: /dhrec/
  79. Disallow: /w/
  80. Disallow: /product/shippingpayment.do
  81. Disallow: /product/mobileprict.do
  82. Disallow: /product/productguaranteedajax.do
  83. Disallow: /product/inventory.do
  84. Disallow: /product/getstprodinfo.do
  85. Disallow: /product/productreview.do
  86. Disallow: /product/displayorder.do
  87. Disallow: /product/getonlinestatus.do
  88. Disallow: /product/getqrcode.do
  89. Disallow: /*?
  92. User-Agent: YandexBot
  93. Allow: /*?f=bm
  94. Allow: /*?utm_source
  95. Allow: /*?_escaped_fragment_=
  96. Disallow: /sellerPromise.do
  97. Disallow: /search.do
  98. Disallow: /promoproduct.do
  99. Disallow: /disc.do
  100. Disallow: /dcp/
  101. Disallow: /mydhgate/
  102. Disallow: /loadShipList.do?
  103. Disallow: /logodata.do
  104. Disallow: /viewcart.do 
  105. Disallow: /!forums/
  106. Disallow: /dhrec/
  107. Disallow: /w/
  108. Disallow: /product/shippingpayment.do
  109. Disallow: /product/mobileprict.do
  110. Disallow: /product/productguaranteedajax.do
  111. Disallow: /product/inventory.do
  112. Disallow: /product/getstprodinfo.do
  113. Disallow: /product/productreview.do
  114. Disallow: /product/displayorder.do
  115. Disallow: /product/getonlinestatus.do
  116. Disallow: /product/getqrcode.do
  117. Disallow: /*?
  120. User-Agent: bingbot
  121. Allow: /*?f=bm
  122. Allow: /*?utm_source
  123. Allow: /*?_escaped_fragment_=
  124. Disallow: /sellerPromise.do
  125. Disallow: /search.do
  126. Disallow: /promoproduct.do
  127. Disallow: /disc.do
  128. Disallow: /dcp/
  129. Disallow: /mydhgate/
  130. Disallow: /loadShipList.do?
  131. Disallow: /logodata.do
  132. Disallow: /viewcart.do 
  133. Disallow: /!forums/
  134. Disallow: /dhrec/
  135. Disallow: /w/
  136. Disallow: /product/shippingpayment.do
  137. Disallow: /product/mobileprict.do
  138. Disallow: /product/productguaranteedajax.do
  139. Disallow: /product/inventory.do
  140. Disallow: /product/getstprodinfo.do
  141. Disallow: /product/productreview.do
  142. Disallow: /product/displayorder.do
  143. Disallow: /product/getonlinestatus.do
  144. Disallow: /product/getqrcode.do
  145. Disallow: /*?
  147. User-agent: ia_archiver
  148. Allow: /*?f=bm
  149. Allow: /*?utm_source
  150. Allow: /*?_escaped_fragment_=
  151. Disallow: /sellerPromise.do
  152. Disallow: /search.do
  153. Disallow: /promoproduct.do
  154. Disallow: /disc.do
  155. Disallow: /dcp/
  156. Disallow: /mydhgate/
  157. Disallow: /loadShipList.do?
  158. Disallow: /logodata.do
  159. Disallow: /viewcart.do 
  160. Disallow: /!forums/
  161. Disallow: /dhrec/
  162. Disallow: /w/
  163. Disallow: /product/shippingpayment.do
  164. Disallow: /product/mobileprict.do
  165. Disallow: /product/productguaranteedajax.do
  166. Disallow: /product/inventory.do
  167. Disallow: /product/getstprodinfo.do
  168. Disallow: /product/productreview.do
  169. Disallow: /product/displayorder.do
  170. Disallow: /product/getonlinestatus.do
  171. Disallow: /product/getqrcode.do
  172. Disallow: /*?
  175. #
  176. # Naverbot Yeti
  177. # 
  178. User-agent: NaverBot
  179. Allow: /*?f=bm
  180. Allow: /*?utm_source
  181. Allow: /*?_escaped_fragment_=
  182. Disallow: /sellerPromise.do
  183. Disallow: /search.do
  184. Disallow: /promoproduct.do
  185. Disallow: /disc.do
  186. Disallow: /dcp/
  187. Disallow: /mydhgate/
  188. Disallow: /loadShipList.do?
  189. Disallow: /logodata.do
  190. Disallow: /viewcart.do 
  191. Disallow: /!forums/
  192. Disallow: /dhrec/
  193. Disallow: /w/
  194. Disallow: /product/shippingpayment.do
  195. Disallow: /product/mobileprict.do
  196. Disallow: /product/productguaranteedajax.do
  197. Disallow: /product/inventory.do
  198. Disallow: /product/getstprodinfo.do
  199. Disallow: /product/productreview.do
  200. Disallow: /product/displayorder.do
  201. Disallow: /product/getonlinestatus.do
  202. Disallow: /product/getqrcode.do
  203. Disallow: /*?
  205. User-agent: Yeti
  206. Allow: /*?f=bm
  207. Allow: /*?utm_source
  208. Allow: /*?_escaped_fragment_=
  209. Disallow: /sellerPromise.do
  210. Disallow: /search.do
  211. Disallow: /promoproduct.do
  212. Disallow: /disc.do
  213. Disallow: /dcp/
  214. Disallow: /mydhgate/
  215. Disallow: /loadShipList.do?
  216. Disallow: /logodata.do
  217. Disallow: /viewcart.do 
  218. Disallow: /!forums/
  219. Disallow: /dhrec/
  220. Disallow: /w/
  221. Disallow: /product/shippingpayment.do
  222. Disallow: /product/mobileprict.do
  223. Disallow: /product/productguaranteedajax.do
  224. Disallow: /product/inventory.do
  225. Disallow: /product/getstprodinfo.do
  226. Disallow: /product/productreview.do
  227. Disallow: /product/displayorder.do
  228. Disallow: /product/getonlinestatus.do
  229. Disallow: /product/getqrcode.do
  230. Disallow: /*?
  232. User-Agent: Googlebot-Image
  233. Allow: /
  235. User-Agent: BaiduSpider
  236. Disallow: /
  238. User-Agent: Sosospider
  239. Disallow: /
  241. User-Agent: HaosouSpider
  242. Disallow: /
  244. User-Agent: Sogou web spider
  245. Disallow: /
  247. User-Agent: Sogou inst spider
  248. Disallow: /
  250. User-Agent: Sogou spider
  251. Disallow: /
  253. User-Agent: Sogou wap spider
  254. Disallow: /
  256. User-Agent: almaden
  257. Disallow: /
  259. User-Agent: ASPSeek
  260. Disallow: /
  263. User-Agent: Axmo
  264. Disallow: /
  266. User-Agent: BecomeBot
  267. Disallow: /
  269. User-Agent: CherryPicker
  270. Disallow: /
  272. User-Agent: Crescent Internet ToolPak
  273. Disallow: /
  275. User-Agent: DISCo Pump
  276. Disallow: /
  278. User-agent: Download Ninja
  279. Disallow: /
  281. User-Agent: DTS Agent
  282. Disallow: /
  284. User-Agent: EmailCollector
  285. Disallow: /
  287. User-Agent: EmailSiphon
  288. Disallow: /
  290. User-Agent: EmailWolf
  291. Disallow: /
  293. User-Agent: Expired Domain Sleuth
  294. Disallow: /
  296. User-Agent: Franklin Locator
  297. Disallow: /
  299. User-Agent: Gaisbot
  300. Disallow: /
  302. User-Agent: grub
  303. Disallow: /
  305. User-Agent: htdig
  306. Disallow: /
  308. User-Agent: HTTrack
  309. Disallow: /
  311. User-Agent: iaea.org
  312. Disallow: /
  314. User-Agent: IconSurf
  315. Disallow: /
  318. User-Agent: Iltrovatore-Setaccio
  319. Disallow: /
  321. User-Agent: Indy Library
  322. Disallow: /
  324. User-agent: InternetSeer.com
  325. Disallow: /
  327. User-Agent: IUPUI
  328. Disallow: /
  330. User-Agent: larbin
  331. Disallow: /
  333. User-agent: libwww
  334. Disallow: /
  336. User-Agent: LNSpiderGuy
  337. Disallow: /
  339. User-Agent: lwp-trivial
  340. Disallow: /
  342. User-Agent: MetaTagRobot
  343. Disallow: /
  345. User-Agent: Missigua Locator
  346. Disallow: /
  348. User-Agent: mozDex
  349. Disallow: /
  351. User-agent: MSIECrawler
  352. Disallow: /
  354. User-Agent: NaverBot
  355. Disallow: /
  357. User-Agent: NextGenSearch
  358. Disallow: /
  360. User-Agent: NPbot
  361. Disallow: /
  362. User-Agent: Nutch
  363. Disallow: /
  365. User-agent: Offline Explorer
  366. Disallow: /
  368. User-Agent: Oracle Ultra Search
  369. Disallow: /
  371. User-Agent: PictureOfInternet
  372. Disallow: /
  374. User-Agent: PlantyNet
  375. Disallow: /
  377. User-Agent: psbot
  378. Disallow: /
  380. User-Agent: QuepasaCreep
  381. Disallow: /
  383. User-Agent: RPT-HTTPClient
  384. Disallow: /
  386. User-agent: sitecheck.internetseer.com
  387. Disallow: /
  389. User-agent: SiteSnagger
  390. Disallow: /
  392. User-Agent: spider.acont.de
  393. Disallow: /
  395. User-Agent: Sqworm
  396. Disallow: /
  398. User-Agent: SSM Agent
  399. Disallow: /
  401. User-Agent: szukacz
  402. Disallow: /
  404. User-Agent: TAMU
  405. Disallow: /
  407. User-agent: Teleport
  408. Disallow: /
  410. User-agent: TeleportPro
  411. Disallow: /
  413. User-Agent: Telesoft
  414. Disallow: /
  416. User-Agent: TurnitinBot
  417. Disallow: /
  419. User-Agent: TutorGig
  420. Disallow: /
  422. User-Agent: Ultraseek
  423. Disallow: /
  425. User-Agent: WebCopier
  426. Disallow: /
  428. User-Agent: WebEMailExtractor
  429. Disallow: /
  431. User-Agent: WebReaper
  432. Disallow: /
  434. User-Agent: Webster Pro
  435. Disallow: /
  437. User-Agent: WebStripper
  438. Disallow: /
  440. User-Agent: WebZIP
  441. Disallow: /
  443. User-Agent: Wotbox
  444. Disallow: /
  446. User-Agent: Xenu
  447. Disallow: /
  449. User-agent: Zao
  450. Disallow: /
  452. User-agent: Zealbot
  453. Disallow: /
  455. User-Agent: ZipppBot
  456. Disallow: /
  458. User-agent: ZyBORG
  459. Disallow: /
  461. User-agent: AdsBot-Google
  462. Disallow: 
  465. #
  466. # Sitemap
  467. #
  469. Sitemap:https://www.dhgate.com/sitemapseo/product_sitemap__index.xml
  470. Sitemap:https://www.dhgate.com/sitemapseo/pc_item_sitemap_index.xml
  471. Sitemap:https://www.dhgate.com/sitemapseo/pc_p_sitemap_index.xml

2 拒绝爬虫

网站大部分都欢迎搜索引擎，但是不欢迎恶意的商业性质的爬虫行为

3 项目遇到的爬虫问题

3.1 爬虫对服务器的压力

爬虫导致日志过多

3.2 电商运营平台（大量产品/价格/排名等数据)-防爬

参考：

3.3 网盘搜索—防爬

IMG_3470的副本.jpg