当前位置：首页 > news >正文

极速微网站建设cms网站推广和网站优化

news 2026/4/8 8:36:09

极速微网站建设cms,网站推广和网站优化,怎么制作网站建设,顺德品牌网站建设咨询题目链接爬虫往往不能在一个页面里面获取全部想要的数据，需要访问大量的网页才能够完成任务。这里有一个网站，还是求所有数字的和，只是这次分了1000页。思路找到调用接口可以看到后面有个参数page来控制页码代码实现 import reques…

题目
链接

爬虫往往不能在一个页面里面获取全部想要的数据，需要访问大量的网页才能够完成任务。
这里有一个网站，还是求所有数字的和，只是这次分了1000页。

思路

找到调用接口

可以看到后面有个参数page来控制页码

代码实现

import requests
import reurl = 'http://www.glidedsky.com/level/web/crawler-basic-2'headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36 Edg/89.0.774.54','Cookie': ''
}res = 0
for i in range(1, 1001):temp_url = url + '?page=' + str(i)response = requests.get(temp_url, headers=headers)html = response.textpattern = re.compile('<div class="col-md-1">.*?(\d+).*?</div>', re.S)n_list = re.findall(pattern, html)for n in n_list:res += int(n)print(f'Result: {res}')

使用多线程实现更快爬取：

import requests
import re
import threadingurl = 'http://www.glidedsky.com/level/web/crawler-basic-2'
total_threads = 10  # 设置线程数量
lock = threading.Lock()  # 创建一个锁，用于线程间的数据同步
res = 0def worker(thread_id):global resfor i in range(thread_id, 1001, total_threads):temp_url = url + '?page=' + str(i)response = requests.get(temp_url, headers=headers)html = response.textpattern = re.compile('<div class="col-md-1">.*?(\d+).*?</div>', re.S)n_list = re.findall(pattern, html)with lock:for n in n_list:res += int(n)threads = []
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36 Edg/89.0.774.54','Cookie': ''
}# 创建并启动线程
for i in range(total_threads):thread = threading.Thread(target=worker, args=(i,))thread.start()threads.append(thread)# 等待所有线程执行完成
for thread in threads:thread.join()print(f'Result: {res}')

使用异步函数

import aiohttp
import asyncio
import reurl = 'http://www.glidedsky.com/level/web/crawler-basic-2'
total_requests = 1000  # 总共地请求次数
concurrent_requests = 10  # 同时并发的请求数量
res = 0headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36 Edg/89.0.774.54','Cookie': ''
}async def fetch_url(session, temp_url):async with session.get(temp_url, headers=headers) as response:html = await response.text()pattern = re.compile('<div class="col-md-1">.*?(\d+).*?</div>', re.S)n_list = re.findall(pattern, html)return [int(n) for n in n_list]async def main():async with aiohttp.ClientSession() as session:tasks = []for i in range(1, total_requests + 1):temp_url = url + '?page=' + str(i)tasks.append(fetch_url(session, temp_url))if len(tasks) >= concurrent_requests or i == total_requests:results = await asyncio.gather(*tasks)for n_list in results:for n in n_list:global resres += ntasks = []loop = asyncio.get_event_loop()
loop.run_until_complete(main())print(f'Result: {res}')