最近我花了一些時間比較分析kafka,rabbitMQ,NATS何者適合Log Aggregation並符合公司系統要求
由於我公司的系統打算撤掉ServiceBus(目前系統一天產生約5百萬筆Log)，
之前我有使用過RabbitMQ來處理Indexing Databases with Lucene.NET相關document的Insert、delete和update經驗，
雖然RabbitMQ完全符合現有公司系統需求，但它吞吐量實在太低，
而比較後我們最終選擇Kafka，但我要強調一下，使用那套平台架構完全取決於你公司需求和系統特性，
我覺得沒有那一套平台架構可以完勝，下面是我最後的整理，可以參考看看。
Apache Kafka Architecture
NATS Architecture
RabbitMQ Architecture
|
feature |
Apache Kafka |
NATS |
RabbitMQ |
|
High Throughtput |
2 |
1(best) |
3 |
|
High concurrency |
2 |
1(best) |
3 |
|
Low latency |
2 |
1(best) |
3 |
|
protocol |
Binary protocol over TCP |
Text-based protocol over TCP |
several standardized protocols such as AMQP,SMTP… |
|
Support Complex routing scenarios |
no |
no |
Yes,(powerful routing features) |
|
Message system |
Basic pub/sub, queue routing |
Basic pub/sub, synthetic queuing |
Several pub/sub, queue routing |
|
Client library |
The Kafka Project Handles Clients,C/C++,Python,Go (AKA golang),.NET,Node.js,PHP... |
Java, .NET, node.js, Ruby, PHP.... |
|
|
Compare Microsoft |
Like Azure Event Hubs 1.Ordered sequence of events stored in partitions,read via offset. 2.biased towards velocity |
Unknow 1.Unique portability , performance is everything |
Like Azure Service Bus 1.AMQP support 2.Similar pub/sub constructs |
|
Support Load balancing |
yes |
yes |
yes |
|
Data Persisted |
Disk |
Disk,memory(via NATS Streaming) |
Disk,memory |
|
Support clustering(HA) |
replica (Use zookeeper) |
master/slave |
|
|
implementing Real time |
yes |
yes |
yes(not good) |
|
Support batch processing |
yes |
yes |
no |
|
lightweight |
yes |
yes |
no |
|
Powered by |
Reference
http://objectzen.com/2016/10/14/not-use-rabbitmq-real-time-messaging/
https://news.ycombinator.com/item?id=11284489
http://bravenewgeek.com/tag/kafka/
http://bravenewgeek.com/dissecting-message-queues/
https://content.pivotal.io/blog/understanding-when-to-use-rabbitmq-or-apache-kafka