SpringBoot中使用Jsoup爬取網(wǎng)站數(shù)據(jù)的方法
爬取數(shù)據(jù)
導(dǎo)入jar包
<properties> <java.version>1.8</java.version> <elasticsearch.version>7.6.1</elasticsearch.version> </properties> <dependencies> <dependency> <groupId>org.jsoup</groupId> <artifactId>jsoup</artifactId> <version>1.10.2</version> </dependency> <dependency> <groupId>com.alibaba</groupId> <artifactId>fastjson</artifactId> <version>1.2.62</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-redis</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-elasticsearch</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-thymeleaf</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-devtools</artifactId> <scope>runtime</scope> <optional>true</optional> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-configuration-processor</artifactId> <optional>true</optional> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <optional>true</optional> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> <exclusions><exclusion> <groupId>org.junit.vintage</groupId> <artifactId>junit-vintage-engine</artifactId></exclusion> </exclusions> </dependency> </dependencies>
新建實體類
@Data@NoArgsConstructor@AllArgsConstructorpublic class Content { private String title; private String img; private String price;}
編寫爬蟲工具類
public class HtmlParseUtil { public static void main(String[] args) throws Exception { new HtmlParseUtil().parseDDJJ('包').forEach(System.out::println); } public List<Content> parseDDJJ(String keywords) throws Exception { //爬取url地址 String url = 'https://search.xxxx.com/Search?keyword='+keywords; //解析網(wǎng)頁,30s內(nèi)未爬取成功,打印錯誤 Document document = Jsoup.parse(new URL(url),30000); //獲取每一本書籍的id Element element = document.getElementById('DJ_goodsList'); //獲取所有的li標(biāo)簽 Elements elements = element.getElementsByTag('li'); ArrayList<Content> goodsList = new ArrayList<>(); //遍歷li標(biāo)簽的內(nèi)容 for (Element el : elements) { String img = el.getElementsByTag('img').eq(0).attr('src'); String price = el.getElementsByClass('p-price').eq(0).text(); String title = el.getElementsByClass('p-name').eq(0).text(); Content content = new Content(); content.setTitle(title); content.setPrice(price); content.setImg(img); goodsList.add(content); } return goodsList; }}
可以看到內(nèi)容、圖片、價格系數(shù)爬取

到此這篇關(guān)于SpringBoot中使用Jsoup爬取網(wǎng)站數(shù)據(jù)的方法的文章就介紹到這了,更多相關(guān)SpringBoot Jsoup爬取內(nèi)容請搜索好吧啦網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持好吧啦網(wǎng)!
相關(guān)文章:
1. IntelliJ IDEA設(shè)置自動提示功能快捷鍵的方法2. 使用Python webdriver圖書館搶座自動預(yù)約的正確方法3. ASP.NET MVC使用jQuery ui的progressbar實現(xiàn)進(jìn)度條4. 從Python的字符串中剝離所有非數(shù)字字符(“。”除外)5. 在線php代碼縮進(jìn)、代碼美化工具:PHP Formatter6. PHP如何開啟Opcache功能提升程序處理效率7. Python3 json模塊之編碼解碼方法講解8. SpringBoot整合Redis的步驟9. PHP程序員簡單的開展服務(wù)治理架構(gòu)操作詳解(二)10. 詳解如何使用Net將HTML簡歷導(dǎo)出為PDF格式

網(wǎng)公網(wǎng)安備