爬蟲實踐–CBA歷年比賽數據
- 2019 年 11 月 15 日
- 筆記
閑來無聊,剛好有個朋友來問爬蟲的事情,說起來了CBA這兩年的比賽數據,做個分析,再來個大數據啥的。來了興趣,果然搞起來,下面分享一下爬蟲的思路。
1、選取數據源
這裡我並不懂CBA,數據源選的是中國某門戶網站的CBA專欄,下面會放鏈接地址,有興趣的可以去看看。
2、分析數據
經過查看頁面元素,發現頁面是後台渲染,沒辦法通過介面直接獲取數據。下面就要分析頁面元素,看到所有的數據都是存在表格裡面的,這下就簡單了很多。
3、確定思路
思路比較簡單,通過正則把所有行數據都提取出來,過濾掉無用的修飾資訊,得到的就是想要的數據。此處我把每行的列符合替換成了「,」方便用csv記錄數據。
經過過濾之後的數據如下:
球隊,第一節,第二節,第三節,第四節,總比分 廣州,33,37,36,27,133 北控,23,18,17,34,92 2019-01-1619:35:00輪次:31場序309開始比賽 比賽已結束 首發,球員,出場時間,兩分球,三分球,罰球,進攻,籃板,助攻,失誤,搶斷,犯規,蓋帽,得分 ,張永鵬,25.8,7-9,0-0,1-1,4,8,3,0,0,1,0,15 ,鞠明欣,19.1,2-4,1-2,0-0,2,5,2,2,0,1,0,7 ,西熱力江,25.5,1-1,4-8,0-0,1,2,4,1,3,1,0,14 ,郭凱,15.5,2-2,0-0,0-0,2,3,0,2,0,2,0,4 ,凱爾·弗格,38.1,5-9,5-9,11-11,0,10,12,2,2,4,0,36 ,姚天一,12.3,0-1,1-4,0-0,0,1,5,0,0,0,0,3 ,科里·傑弗森,24.0,4-4,2-4,3-4,0,6,0,1,0,1,1,17 ,陳盈駿,22.6,1-1,2-7,1-1,0,2,4,2,1,2,0,9 ,司坤,19.0,2-2,0-2,0-0,0,5,1,0,1,4,0,4 ,孫鳴陽,20.6,2-3,0-0,3-3,1,4,1,2,3,4,0,7 ,谷玥灼,7.4,1-1,1-2,0-0,0,0,2,0,0,0,0,5 ,鄭准,10.1,3-4,2-3,0-0,0,2,0,0,0,1,0,12 ,總計,240.0,30-41(73.2%),18-41(43.9%),19-20(95.0%),10,48,34,12,10,21,1,133 首發,球員,出場時間,兩分球,三分球,罰球,進攻,籃板,助攻,失誤,搶斷,犯規,蓋帽,得分 ,於梁,20.8,1-3,0-1,0-0,0,0,2,0,1,5,0,2 ,於澍龍,17.9,0-1,1-3,0-0,0,2,1,2,0,1,0,3 ,許夢君,46.2,1-3,5-12,0-0,1,6,2,1,0,3,0,17 ,托馬斯·羅賓遜,43.4,9-20,0-2,9-14,3,11,5,2,1,3,1,27 ,楊敬敏,16.0,3-4,0-3,0-0,0,0,0,2,0,1,0,6 ,孫賀男,2.8,0-0,0-0,0-0,0,0,0,1,0,1,0,0 ,劉大鵬,28.0,1-1,3-5,0-0,1,4,3,2,2,3,0,11 ,張銘浩,8.5,0-0,0-0,1-2,0,0,0,0,1,1,0,1 ,張帆,27.5,5-7,1-3,0-0,0,1,6,4,1,2,0,13 ,王征,23.3,3-3,0-0,6-8,0,2,0,0,1,1,1,12 ,常亞松,5.6,0-0,0-1,0-0,0,1,0,1,2,0,0,0 ,總計,240.0,23-42(54.8%),10-30(33.3%),16-24(66.7%),5,27,19,15,9,21,2,92
下面分享自己程式碼:
1package com.fun 2 3import com.fun.frame.Save 4import com.fun.frame.httpclient.FanLibrary 5import com.fun.utils.Regex 6import com.fun.utils.WriteRead 7 8class sd extends FanLibrary { 9 10 public static void main(String[] args) { 11 int i = 1 12 def total = [] 13 range(300, 381).forEach {x -> 14 total.addAll test(x) 15 } 16 Save.saveStringList(total, "total4.csv") 17 testOver() 18 } 19 20 21 static def test(int i) { 22 if (new File(LONG_Path + "${i}.csv").exists()) return WriteRead.readTxtFileByLine(LONG_Path + "${i}.csv") 23 String url = "http://cbadata.sports.sohu.com/game/content/2017/${i}" 24 25 def get = getHttpGet(url) 26 27 def response = getHttpResponse(get) 28 29 30 def string = response.getString("content").replaceAll("\s", EMPTY) 31// output(string) 32 def all = Regex.regexAll(string, "<tr.*?<\/tr>") 33 def list = [] 34 all.forEach {x -> 35 def info = x.replaceAll("</*?tr.*?>", EMPTY).replaceAll("</t(d|h)>", ",") 36 info = info.replaceAll("<.*?>", EMPTY) 37 38 info = info.charAt(info.length() - 1) == ',' ? info.substring(0, info.length() - 1) : info 39 if (info.startsWith("總計")) info = "," + info 40 list << info 41 output(info) 42 43 } 44 Save.saveStringList(list, "${i}.csv") 45 return list 46 } 47 48}