ニュースレター

Hortonworks から最新情報をメールで受け取る

月に一度、ビッグデータに関する最新のインサイト、トレンド、分析情報、ナレッジをお届けします。

AVAILABLE NEWSLETTERS:

Sign up for the Developers Newsletter

月に一度、ビッグデータに関する最新のインサイト、トレンド、分析情報、ナレッジをお届けします。

行動喚起

始める

クラウド

スタートのご用意はできましたか?

Sandbox をダウンロード

ご質問はありませんか?

*いつでも登録を解除できることを理解しています。Hortonworks プライバシーポリシーのその他の情報も確認しています。
クローズクローズボタン
December 18, 2018
前のスライド次のスライド

Monitoring Kafka Streams Microservices with Hortonworks Streams Messaging Manager (SMM)

In last week’s blog Secure and Governed Microservices with HDF/HDP Kafka Streams Support, we walked through how to build microservices with the new Kafka Streams support in HDF 3.3 and HDP 3.1 that is fully integrated with Ranger, Schema Registry and other platform services. This blog is all about monitoring these microservices with Hortonworks Streams Messaging Manager (SMM).

Monitoring MicroServices with SMM

In a microservices architecture, you will see a proliferation of stand-alone decoupled services. Hence, monitoring and managing these services becomes extremely critical. SMM provides users a powerful tool to monitor and visualize their microservices and understand how data flows across these services.

Working off the trucking fleet use case example from the previous blog, you can view each of the three microservices as a consumer and producer as depicted in the below diagram.

Lets first focus on monitoring the stream between MicroService 1 and MicroService 2 where MicroService 1 is a Kafka producer into the driver-violation-events topic and MicroServce 2 is a consumer from that topic. The below video showcases how to monitor the interactions between these two Kafka Streams microservices.

 

As the above video showcased, SMM cured the Kafka blindness for the streams app between the two microservices shedding light on some important information including the rate at which MicroService 1 was producing data, the lag of MicroService 2 and understanding the details of the internal Kafka changelog topics created by the Kafka streams join operator.

Another common use case for monitoring Kafka streams application using SMM is the following:

  • Detect a Kafka Streams microservice that is lagging considerably behind in processing messages.
  • Scale out the Kafka Streams microservice by spinning up new instances of the app (adding new instances to the consumer group).
  • Validate that adding new instances has decreased the lag of the consumer group.

The below video showcases how SMM address this use case.

 

The addition of Kafka Streams to HDF 3.3 and HDP 3.1 integrated with platform services like Ranger, Schema Registry and other platform services provides app developers a comprehensive platform to build secure and governed microservices. With SMM, devops and platform operations teams have enterprise  tools to debug, monitor, and troubleshoot these microservices built using Kafka Streams.

What’s Next

In the next installment of the Kafka Analytics blog series, we walk through the new Hive and Kafka integration for the SQL access pattern.

コメントする

メールアドレスは公開されません。必須フィールドには * のマークが付いています。