ニュースレター

Hortonworks から最新情報をメールで受け取る

月に一度、ビッグデータに関する最新のインサイト、トレンド、分析情報、ナレッジをお届けします。

AVAILABLE NEWSLETTERS:

Sign up for the Developers Newsletter

月に一度、ビッグデータに関する最新のインサイト、トレンド、分析情報、ナレッジをお届けします。

行動喚起

始める

クラウド

スタートのご用意はできましたか?

Sandbox をダウンロード

ご質問はありませんか?

*いつでも登録を解除できることを理解しています。Hortonworks プライバシーポリシーのその他の情報も確認しています。
クローズクローズボタン
HDF > Develop Data Flow & Streaming Applications > 入門編の基本

NiFi in Trucking IoT on HDF

NiFi in Trucking IoT Use Case

クラウド スタートのご用意はできましたか?

SANDBOX をダウンロード

概要

The IoT Use Case

Visit the Storm tutorial to learn about the Trucking IoT Use Case.

What is NiFi?

What is NiFi’s role in this Stream Processing Application?

  • NiFi acts as the producer that ingests data from the truck and traffic IoT devices, does simple event processing on the data, so that it can be split into TruckData and TrafficData that can be sent as messages to two Kafka topics.

To learn about what NiFi is, visit What is Apache NiFi? from our Analyze Transit Patterns with Apache NiFi tutorial.

Architectural Overview

At a high level, our data pipeline looks as follows:


MiNiFi Simulator -----> NiFi ----> Kafka

There is a data simulator that replicates MiNiFi’s place in the data flow on IoT edge, MiNiFi is embedded on the vehicles, so the simulator generates truck and traffic data. NiFi ingests this sensor data. NiFi’s flow performs preprocessing on the data to prepare it to be sent to Kafka.

Benefits of NiFi

Flow Management

  • Guaranteed Delivery: Achieved by persistent write-ahead log and content repository allow for very high transaction rates, effective load-spreading, copy-on-write, and play to the strengths of traditional disk read/writes.

  • Data Buffering with Back Pressure and Pressure Release: If data being pushed into the queue reaches a specified limit, then NiFi will stop the process send data into that queue. Once data reaches a certain age, NiFi will terminate the data.

  • Prioritized Queuing: A setting for how data is retrieved from a queue based on largest, smallest, oldest or other custom prioritization scheme.

  • Flow Specific QoS: Flow specific configuration for critical data that is loss intolerant and whose value becomes of less value based on time sensitivity.

使い易さ

  • Visual Command and Control: Enables visual establishment of data flow in real-time, so any changes made in the flow will occur immediately. These changes are isolated to only the affected components, so there is not a need to stop an entire flow or set of flows to make a modification.

  • Flow Templates: A way to build and publish flow designs for benefitting others and collaboration.

  • Data Provenance: Taking automatic records and indexes of the data as it flows through the system.

  • Recovery/Recording a rolling buffer of fine-grained history: Provides click to content, download of content and replay all at specific points in an object’s life cycle.

セキュリティ

  • System to System: Offers secure exchange through use of protocols with encryption and enables the flow to encrypt and decrypt content and use shared-keys on either side of the sender/recipient equation.

  • User to System: Enables 2-Way SSL authentication and provides pluggable authorization, so it can properly control a user’s access and particular levels (read-only, data flow manager, admin).

  • Multi-tenant Authorization: Allows each team to manage flows with full awareness of the entire flow even parts they do not have access.

Extensible Architecture

  • Extension: Connects data systems no matter how different data system A is from system B, the data flow processes execute and interact on the data to create a uni-line or bidirectional line of communication.

  • Classloader Isolation: NiFi provides a custom class loader to guarantee each extension bundle is as independent as possible, so component-based dependency problems do not occur as often. Therefore, extension bundles can be created without worry of conflict occurring with another extension.

  • Site-to-Site Communication Protocol: Eases transferring of data from one NiFi instance to another easily, efficiently and securely. So devices embedded with NiFi can communicate with each other via S2S, which supports a socket based protocol and HTTP(S) protocol.

Flexible Scaling Model

  • Scale-out (Clustering): Clustering many nodes together. So if each node is able to handle hundreds of MB per second, then a cluster of nodes could be able to handle GB per second.

  • Scale-up & down: Increase the number of concurrent tasks on a processor to allow more processes to run concurrently or decrease this number to make NiFi suitable to run on edge devices that have limited hardware resources. View MiNiFi Subproject to learn more about solving this small footprint data challenge.

Next: NiFi in Action

We have become familiar with NiFi’s role in the use case, next let’s move onto seeing NiFi in action while the demo application runs.

ユーザーの評価

ユーザーの評価
2 1.5 out of 5 stars
5 Star 0%
4 Star 0%
3 Star 0%
2 Star 50%
1 Star 50%
チュートリアル名
NiFi in Trucking IoT on HDF

質問する回答を探す場合は、Hortonworks Community Connectionをご参照ください。

2 Reviews
評価する

登録

登録して評価をご記入ください

ご自身の体験を共有してください

例: 最高のチュートリアル

この欄に最低50文字で記入してください。

成功

ご意見を共有していただきありがとうございます!

Not completely missing, but still lacking
by test test on February 10, 2019 at 6:27 pm

Nathan, the page you were looking at was just the outline, clicking on the hyperlinks will direct to the actual contents for this chapter. Still, instructions could have been more explicit. Even after getting to the actual content, I found there was a lot of stuff missing. For example, all of section 3 has you build out the IOT trucking dataflow but doesn't write the instructions out for how to add each processor (you drag and drop the processor icon onto the canvas, in case you were wondering). Also some of the properties are out of… Show More

Nathan, the page you were looking at was just the outline, clicking on the hyperlinks will direct to the actual contents for this chapter. Still, instructions could have been more explicit.

Even after getting to the actual content, I found there was a lot of stuff missing. For example, all of section 3 has you build out the IOT trucking dataflow but doesn’t write the instructions out for how to add each processor (you drag and drop the processor icon onto the canvas, in case you were wondering). Also some of the properties are out of date. The delimiter that should actually be used is “|” whereas the instructions has you use a delimter of ” `”.

表示件数を減らす
Cancel

Review updated successfully.

WTF
by Nathan Maxfield on December 13, 2018 at 2:37 pm

How is this a tutorial for “Creating a NiFi DataFlow?” It doesn’t actually show you how to do anything.

How is this a tutorial for “Creating a NiFi DataFlow?” It doesn’t actually show you how to do anything.

表示件数を減らす
Cancel

Review updated successfully.