涓�鍒嗛挓浜嗚В浜掕仈缃戞暟鎹�鎸栨帢娴佺�?- 鐧惧害鏂囧簱

鏂板缓

涓婁紶

棣栭〉

鍔╂墜

鏈�杩?/div>

璧勬枡搴?/div>

宸ュ叿

涓�鍒嗛挓浜嗚В浜掕仈缃戞暟鎹�鎸栨帢娴佺�?/p>

 

 

 

1

銆佺埇铏�鎶撳彇缃戠粶鏁版�?/p>

 

鐪熷疄鐨勬暟鎹�鎸栨帢椤圭洰锛�

涓�瀹氭槸浠庤幏鍙栨暟鎹�寮�濮嬬殑锛?/p>

闄や簡閫氳繃涓�浜涙笭閬撹喘涔版垨鑰呬笅

杞戒笓涓氭暟鎹�澶栵�?/p>

甯稿父闇�瑕佸ぇ瀹惰嚜宸卞姩鎵嬬埇浜掕仈缃戞暟鎹�锛�

杩欎釜鏃跺�欙紝

鐖�铏�灏辨樉寰楁牸

澶栭噸瑕佷簡銆?/p>

 

Nutch

鐖�铏�鐨勪富瑕佷綔鐢ㄦ槸浠庣綉缁滀笂鎶撳彇缃戦〉鏁版嵁骞跺缓绔嬬储寮曘�?/p>

鎴戜滑鍙�闇�鎸囧畾缃戠珯

鐨勯《绾х綉鍧�锛屽��

taobao.com

锛岀埇铏�鍙�浠ヨ嚜鍔ㄦ帰娴嬪嚭椤甸潰鍐呭�归噷鏂扮殑缃戝潃锛屼粠鑰?/p>

杩涗竴姝ユ姄鍙栭摼鎺ョ綉椤垫暟鎹�銆?/p>

nutch

鏀�鎸佹妸鎶撳彇鐨勬暟鎹�杞�鍖栨垚鏂囨湰锛�

濡?/p>

锛?/p>

PDF

銆?/p>

WORD

銆?/p>

EXCEL

銆?/p>

HTML

銆?/p>

XML

绛夊舰寮忥級杞�鎹㈡垚绾�鏂囧瓧瀛楃�︺�?/p>

 

Nutch

涓?/p>

Hadoop

闆嗘垚锛屽彲浠ュ皢涓嬭浇鐨勬暟鎹�淇濆瓨鍒�

hdfs

锛岀敤浜庡悗缁�绂荤嚎鍒嗘瀽銆備娇

鐢ㄦ�ラ�や负锛?/p>

 

飩?/p>

 

鍚?/p>

hdfs

涓�瀛樺叆寰呮姄鍙栫殑缃戠�?/p>

url 

$ hadoop fs -put urldir urldir 

娉�锛�

 

绗�涓�涓?/p>

urldir

涓烘湰鍦版枃浠跺す锛屽瓨鏀句簡

url

鏁版嵁鏂囦欢锛屾瘡琛屼竴涓?/p>

url

鍦板潃

 

绗�浜屼�?/p>

urldir

涓?/p>

hdfs

鐨勫瓨鍌ㄨ矾寰勩�?/p>

 

飩?/p>

 

鍚�鍔�

nutch

锛屽湪

NUTCH_HONE

鐩�褰曚笅鎵ц�屼互涓嬪懡浠?/p>

 

$ bin/nutch crawlurldir 

鈥?/p>

dir crawl -depth 3 

鈥?/p>

topN 10 

鍛戒护鎴愬姛鎵ц�屽悗锛屼細鍦�

hdfs

涓�鐢熸�?/p>

crawl

鐩�褰曘�?/p>

 

 

 

2

銆?/p>

MapReduce

棰勫�勭悊鏁版�?/p>

 

瀵逛簬涓嬭浇鐨勫師濮嬫枃鏈�鏂囨。锛�

鏃犳硶鐩存帴杩涜�屽�勭悊锛?/p>

闇�瑕佸�规枃鏈�鍐呭�硅繘琛岄�勫�勭悊锛�

鍖?/p>

鎷�鏂囨。鍒囧垎銆佹枃鏈�鍒嗚瘝銆佸幓鍋滅敤璇嶏紙鍖呮嫭鏍囩偣銆佹暟瀛椼�佸崟瀛楀拰鍏跺畠涓�浜涙棤鎰忎箟鐨?/p>

璇嶏級銆佹枃鏈�鐗瑰緛鎻愬彇銆佽瘝棰?/p>

缁熻��

銆佹枃鏈�鍚戦噺鍖栫瓑鎿嶄綔銆?/p>

 

甯哥敤鐨勬枃鏈�棰勫�勭悊绠楁硶鏄?/p>

TF-IDF

锛屽叾涓昏�佹�濇兂鏄�锛屽�傛灉鏌愪釜璇嶆垨鐭�璇�鍦ㄤ竴绡囨枃

绔犱腑鍑虹幇鐨勯�戠巼楂橈�?/p>

骞朵笖鍦ㄥ叾浠栨枃绔犱腑寰堝皯鍑虹幇锛?/p>

鍒欒�や负姝よ瘝鎴栬�呯煭璇�鍏锋湁寰堝�?/p>

鐨勭被鍒�鍖哄垎鑳藉姏锛岄�傚悎鐢ㄦ潵鍋氬垎绫汇�?/p>

 

飩?/p>

 

杈撳叆鍘熷�嬫枃鏈�鍐呭�癸�?/p>