鍓嶈█
Spark Streaming
璇炵敓浜?/p>
2013
骞达紝鎴愪负
Spark
骞冲彴涓婃祦寮忓鐞嗙殑瑙ｅ喅鏂规锛屽悓鏃朵篃缁欏ぇ瀹舵彁
渚涢櫎
Storm
浠ュ鐨勫彟涓�涓�夋嫨銆傝繖绡囧唴瀹逛富瑕佷粙缁?/p>
Spark Streaming
鏁版嵁鎺ユ敹娴佺▼妯″潡涓笌
Kafka
闆嗘垚鐩稿叧鐨勫姛鑳姐�?/p>
Spark Streaming
涓?/p>
Kafka
闆嗘垚鎺ュ彈鏁版嵁鐨勬柟寮忔湁涓ょ锛?/p>
1.
Receiver-based Approach
2.
Direct Approach (No Receivers)
鎴戜滑浼氬杩欎袱绉嶆柟妗堝仛璇︾粏鐨勮В鏋愶紝鍚屾椂瀵规瘮涓ょ鏂规浼樺姡銆傞�夊瀷鍚庯紝鎴戜滑閽堝
Direct
Approach (No Receivers)
妯″紡璇︾粏浠嬬粛鍏跺浣曞疄鐜?/p>
Exactly Once Semantics
锛屼篃灏辨槸淇濊瘉鎺ユ敹
鍒扮殑鏁版嵁鍙澶勭悊涓�娆★紝涓嶄涪锛屼笉閲嶃�?/p>
Receiver-based Approach
瑕佹弿杩版竻妤?/p>
Receiver-based Approach
锛屾垜浠渶瑕佷簡瑙ｅ叾鎺ユ敹娴佺▼锛屽垎鏋愬叾鍐呭瓨浣跨敤锛屼互鍙?/p>
鐩稿叧鍙傛暟閰嶇疆瀵瑰唴瀛樼殑褰卞搷銆?/p>
*
鏁版嵁鎺ユ敹娴佺▼
*
鍚姩
Spark Streaming(
鍚庣画缂╁啓涓?/p>
SS)
鍚庯紝
SS
浼氶�夋嫨涓�鍙?/p>
Executor
鍚姩
ReceiverSupervisor,
骞朵笖鏍囪涓?/p>
Active
鐘舵�併�傛帴鐫�鎸夊涓嬫楠ゅ鐞嗭細
1.
ReceiverSupervisor
浼氬惎鍔ㄥ搴旂殑
Receiver(
杩欓噷鏄?/p>
KafkaReceiver)
2.
KafkaReceiver
浼氭牴鎹厤缃惎鍔ㄦ柊鐨勭嚎绋嬫帴鍙楁暟鎹紝鍦ㄨ绾跨▼涓皟
鐢?/p>
ReceiverSupervisor.store
鏂规硶濉厖鏁版嵁锛屾敞鎰忥紝杩欓噷鏄竴鏉′竴鏉″～鍏呯殑銆?/p>
3.
ReceiverSupervisor
浼氳皟鐢?/p>
BlockGenerator.addData
杩涜鏁版嵁濉厖銆?/p>
鍒扮洰鍓嶄负姝紝
鏁翠釜杩囩▼涓嶄細鏈夊お澶氬唴瀛樻秷鑰楋紝
姝ｅ父鐨勪竴涓嚎鎬ц皟鐢ㄣ�?/p>
鎵�鏈夊鏉傜殑鏁版嵁缁撴瀯閮?/p>
闅愬惈鍦?/p>
BlockGenerator
涓�?/p>
* BlockGenerator
瀛樺偍缁撴瀯
*
BlockGenerator
浼氬鏉備簺锛岄噸瑕佺殑鏁版嵁瀛樺偍缁撴瀯鏈夊洓涓細
1.
缁存姢浜嗕竴涓紦瀛?/p>
currentBuffer
锛屽氨鏄竴涓棤闄愰暱搴︾殑
ArrayBuffer
銆?
currentBuffer
骞朵笉浼氳澶嶇敤锛岃�屾槸姣忔閮戒細鏂板缓锛岀劧鍚庢妸鑰佺殑瀵硅薄鐩存帴灏佽鎴?/p>
Block
锛?/p>
BlockGenerator
浼氳礋璐ｄ繚璇?/p>
currentBuffer
鍙湁涓�涓�?/p>
currentBuffer
濉?/p>
鍏呯殑閫熷害鏄彲浠ヨ闄愬埗鐨勶紝浠ョ涓哄崟浣嶏紝閰嶇疆鍙傛暟涓?
spark.streaming.receiver.maxRate
銆傝繖涓槸
Spark
鍐呭瓨鎺у埗鐨勭涓�閬撻槻绾匡紝濉?/p>
鍏?/p>
currentBuffer
鏄樆濉炵殑锛屾秷璐?/p>
Kafka
鐨勭嚎绋嬬洿鎺ュ仛濉厖銆?/p>