[Python] 小说爬虫

906680082

小说网站的爬虫
想下载哪本小说，就进入小说第一章的网页，复制网页地址，作为输入参数就可以了

1. import requests
   
2. import re
   
3. from bs4 import BeautifulSoup
   
4. 
   
5. start_url = "https://www.aabook.xyz/read.php?id=68339"
   
6. start_url = input("请输入网址:")
   
7. headers = {
   
8.     "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36"
   
9. }
   
10. 
    
11. res = requests.get(start_url)
    
12. res.encoding = 'utf-8'
    
13. soup = BeautifulSoup(res.text, 'lxml')
    
14. # page_num = re.search(r'\d+', soup.find_all('li')[-1]('a')[0].text).group()
    
15. sumUrl = ["https://www.aabook.xyz/" + i.find('a').get('href') for i in soup.find('ul','section_list')('li')]
    
16. name = soup.find('p',["nav","pngFix"])('a')[-1].text
    
17. # res.encoding = "utf-8"
    
18. # &v=a52bT%2FFbmkRqe7W8hwFugqPbNG5OpnWcepYftiB1SNuFddrUcJKZ
    
19. def get_text(url):
    
20.     r = requests.get(url)
    
21.     v = re.search(r"(&v=.*)"", r.text).group(1)
    
22.     chapid = url.split("=")[1]
    
23.     true_url = "https://www.aabook.xyz/" + "_getcontent.php?id=" + chapid + v
    
24.     res2 = requests.get(true_url)
    
25.     res2.encoding = "utf-8"
    
26.     soup2 = BeautifulSoup(res2.text,"lxml")
    
27.     textList = [i.text for i in soup2('p')]
    
28.     return textList
    
29. 
    
30. if __name__ == "__main__":
    
31.     for id in sumUrl:
    
32.         text = get_text(id)
    
33.         print(text)
    
34.         with open(name + ".txt",'a',encoding='utf-8') as f:
    
35.             f.write(''.join(text))
    
36.             f.close()

复制代码

676940452

不错不错~~~~~

307483809

不错不错，，，

笑东风丶

感谢大佬分享，大牛因有你而更精彩。

NniAo

谢谢分享

wolf1111

多谢分享多谢分享

q69988458

感谢楼主分享

x757679776

6666

a90317

谢谢楼主分享，先收藏了。

li476188305

感谢分享，谢谢提供分享