大数据实时处理时代

// count the words in this stream as an aggregated table KTable<String, Long> counts = words.countByKey(”Counts”);
// write the result table to a new topic counts.to(”topic2”);
// create a stream processing instance and start running it
KafkaStreams streams = new KafkaStreams(builder, config);
streams.start();
}
27
Kafka Streams DSL
// count the words in this stream as an aggregated table KTable<String, Long> counts = words.countByKey(”Counts”);
// write the result table to a new topic counts.to(”topic2”);
// count the words in this stream as an aggregated table KTable<String, Long> counts = words.countByKey(”Counts”);
// write the result table to a new topic counts.to(”topic2”);
• consumer Powerful yet easy-to-use
•
Event-at-a-time, Stateful
•
Windowing with out-of-order handling
•
Highly scalable, distributed, fault
•
tolerant
22
Anywhere, anytime
// create a stream processing instance and start running it
KafkaStreams streams = new KafkaStreams(builder, config);
streams.start();
}
30
Kafka Streams DSL
time
10
Kafka: Streaming P• laPtufbolirsmh / Subscribe
• Move data around as online streams
• Store
• “Source-of-truth” continuous data
• • PReraoctc/epsroscess data in real-
}
14
DIY Stream Processing is Hard
• Ordering
• State Management
• Partitioning & Scalability
• Time,Window & Out-of-order Data
• Fault tolerance • Re-processing
15
Stream Processing with Kafka
• Option I: Do It Yourself !
• Option II: full-fledged stream processing • system
Storm, Spark, Flink, Samza, ..
16
MapReduce Heritage?
// create a stream processing instance and start running it
KafkaStreams streams = new KafkaStreams(builder, config);
streams.start();
}
30
Kafka Streams DSL
time
11
Stream Processing with Kafka
12
Stream Processing with Kafka
• Option I: Do It Yourself !
13
Stream Processing with Kafka
• Option I: Do It Yourself !
大数据实时处理时代
在这里填写您的姓名
适用于报告通用开题报告/课题汇报/述职等模板
1
Stream Processing isn’t (necessarily)
• Transient, approximate, lossy… • .. that you must have batch processing as safety
public static void main(String[] args) { // specify the processing topology by first reading in a stream from a topic KStream<String, String> words = builder.stream(”topic1”);
7
Why Kafka in Stream Processing?
8
Kafka: Streaming P• laPtefrosirstment Buffering
• Logical Ordering • Scalable “source-of-
truth”
10
Kafka: Streaming
P• laPteufrboslirsmthen/
• tMSBouvbeudsfafctearraibironeugnd as online Lstoregamicsal
•• OStrodrering
•
•
Se“Scoaurlcae-bofl-etru“tsho” cuonrtcineu-oousftdrautath”
• Proces
s • React / process data in real-
Ok.
Ok.
Ok.
Ok.
23
Anywhere, anytime
<dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-streams</artifactId> <version>0.10.0.0</version>
</dependency>
public static void main(String[] args) { // specify the processing topology by first reading in a stream from a topic KStream<String, String> words = builder.stream(”topic1”);
• Option II: full-fledged stream processing system
• Option III: lightweight stream processing
library
21
Kafka Streams (0.10+)
• New client library besides producer and
public static void main(String[] args) { // specify the processing topology by first reading in a stream from a topic KStream<String, String> words = builder.stream(”topic1”);
// create a stream processing instance and start running it
KafkaStreams streams = new KafkaStreams(builder, config);
streams.start();
}
28
Kafka Streams DSL
• Config Management
• ResouCracenMIajnuagsetmuesnet my own?!
• Deployment
• etc..
20
Example: Async. Micro-Services
20
Stream Processing with Kafka
• Option I: Do It Yourself !
while (isRunning) { // read some messages from Kafka inputMessages = consumer.poll();
// do some processing…
// send output messages back to Kafka
producer.send(outputMessages);
// create a stream processing instance and start running it
KafkaStreams streams = new KafkaStreams(builder, config);
streams.start();
}
31