拉勾网上争对反爬虫机制出现的问题

走钢丝

如何解决抓取信息时“{”状态“:false，“MSG”:“您操作过于频繁，请稍后再访问”，“clientip”:“182.129.38.91”，“state: 2408}”的问题?添加了所有请求条件，设置了动态UA，但仍然不能工作

1. DEFAULT_REQUEST_HEADERS = {
   
2.   # 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
   
3.   # 'Accept-Language': 'en',
   
4.    "accept": "application/json, text/javascript, */*; q=0.01",
   
5.    "accept-encoding": "gzip, deflate, br",
   
6.    "accept-language": "zh-CN,zh;q=0.9",
   
7.    "content-type": "application/x-www-form-urlencoded; charset=UTF-8",
   
8.    "cookie": "JSESSIONID=ABAAAECABIEACCA6B0B35CC82843AFC00F32AC6B45A76AE; WEBTJ-ID=20200520170246-172315226f3e9-0d022377fe6f86-366b4108-921600-172315226f6c9; RECOMMEND_TIP=true; _ga=GA1.2.1963165012.1589965367; _gid=GA1.2.1320686678.1589965367; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1589965367; user_trace_token=20200520170247-3f09971d-19df-4827-9bcf-74c99b5c9dcf; LGUID=20200520170247-da3d96d1-a445-42ae-a7f4-9374f320e5e4; index_location_city=%E5%85%A8%E5%9B%BD; sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%22172315288e8488-081e1b46a8d664-366b4108-921600-172315288eaac4%22%2C%22%24device_id%22%3A%22172315288e8488-081e1b46a8d664-366b4108-921600-172315288eaac4%22%7D; sajssdk_2015_cross_new_user=1; TG-TRACK-CODE=search_code; X_MIDDLE_TOKEN=fbb896af01a0adbc319581251f75b474; X_HTTP_TOKEN=8f6ad38bd41b517633076998513a00d575eaa5b241; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1589967033; LGRID=20200520173033-36acdbfa-3b82-4c5a-aa26-281603de6f11; SEARCH_ID=c6ae56ca6a0644a78264dbc3d61c1e4c",
   
9.    "referer":"https://www.lagou.com/jobs/list_python?labelWords=&fromSearch=true&suginput=",
   
10.    "x-anit-forge-code": 0,
    
11.    "x-anit-forge-token": None,
    
12.    "x-requested-with": "XMLHttpRequest"
    
13. }

复制代码

1. agent1='Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3947.100 Safari/537.36'
   
2. agent2='E808 SAMSUNG-SGH-E808/1.0*MzU0MTk0MDAwNTgzMDgx UP.Browser/6.2.2.6 (GUI) MMP/1.0'
   
3. agent3='D500C SAMSUNG-SGH-D500C/1.0 Profile/MIDP-2.0 Configuration/CLD\
   
4. C-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2'
   
5. agent4='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 \
   
6. (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362'
   
7. agent5=' E100A SAMSUNG-SGH-E100A/T2 UP.Browser/6.1.0.6 (GUI) MMP/1.0'
   
8. UserAgent=[
   
9.    agent1,
   
10.    agent2,
    
11.    agent3,
    
12.    agent4,
    
13.    agent5
    
14. ]

复制代码

碧瑶

控制访问速度，稍微慢一点。或者IP代理池

久韦

你被屏蔽了。估计代理池正常。