Efficiently transfers bulk data between Apache Hadoop and structured datastores
Apache Sqoop efficiently transfers bulk data between Apache Hadoop and structured datastores such as relational databases. Sqoop helps offload certain tasks (such as ETL processing) from the EDW to Hadoop for efficient execution at a much lower cost. Sqoop can also be used to extract data from Hadoop and export it into external structured datastores. Sqoop works with relational databases such as Teradata, Netezza, Oracle, MySQL, Postgres, and HSQLDB
Apache Sqoop does the following to integrate bulk data movement between Hadoop and structured datastores:
|Import sequential datasets from mainframe||Satisfies the growing need to move data from mainframe to HDFS|
|Import direct to ORCFiles||Improved compression and light-weight indexing for improved query performance|
|Data imports||Moves certain data from external stores and EDWs into Hadoop to optimize cost-effectiveness of combined data storage and processing|
|Parallel data transfer||For faster performance and optimal system utilization|
|Fast data copies||From external systems into Hadoop|
|Efficient data analysis||Improves efficiency of data analysis by combining structured data with unstructured data in a schema-on-read data lake|
|Load balancing||Mitigates excessive storage and processing loads to other systems|
YARN coordinates data ingest from Apache Sqoop and other services that deliver data into the Enterprise Hadoop cluster.
Sqoop provides a pluggable mechanism for optimal connectivity to external systems. The Sqoop extension API provides a convenient framework for building new connectors which can be dropped into Sqoop installations to provide connectivity to various systems. Sqoop itself comes bundled with various connectors that can be used for popular database and data warehousing systems.
The Apache Sqoop community is working on improvements around security, support for additional data platforms, and closer integration with other Hadoop components.
|Ease of Use||
Recent Sqoop releases extended data integration between Apache Hadoop and relational data stores.
|1.4.6 – July 2015||
|1.4.5 – August 2014||
|1.4.4 – July 2013||
|1.4.3 – March 2013||
Introduction R is a popular tool for statistics and data analysis. It has rich visualization capabilities and a large collection of libraries that have been developed and maintained by the R developer community. One drawback to R is that it’s designed to run on in-memory data, which makes it unsuitable for large datasets. Spark is […]
多くのお客様から非常によくいただくリクエストは、たとえばスキャンした PNG ファイルのテキストなど、画像ファイル中でテキストをインデックスすることです。このチュートリアルでは、それを SOLR を使って行う方法を段階的に説明します。前提条件：Hortonworks Sandbox がダウンロードされていること、「HDP Sandbox のコツを学ぶ」のチュートリアルを完了していること。ステップバイステップ・ガイド […]
Introduction In this tutorial, you will learn about the different features available in the HDF sandbox. HDF stands for Hortonworks DataFlow. HDF was built to make processing data-in-motion an easier task while also directing the data from source to the destination. You will learn about quick links to access these tools that way when you […]
はじめに：JReport は、Apache Hive の JDBC ドライバを使用して Hortonworks Data Platform 2.3 からデータを簡単に抽出し可視化することができる、組み込み BI レポーティングツールです。レポート、ダッシュボード、データ分析を作成することが可能で、後で自分のアプリケーションに組み込むこともできます。このチュートリアルでは、次のステップをご説明します[...]
Apache Zeppelin on HDP 2.4.2 Author: Vinay Shukla In March 2016 we delivered the second technical preview of Apache Zeppelin, on HDP 2.4. Meanwhile we and the Zeppelin community have continued to add new features to Zeppelin. These features are now available in the final technical preview of Apache Zeppelin. This technical preview works with […]
Introduction Hadoop has always been associated with BigData, yet the perception is it’s only suitable for high latency, high throughput queries. With the contribution of the community, you can use Hadoop interactively for data exploration and visualization. In this tutorial you’ll learn how to analyze large datasets using Apache Hive LLAP on Amazon Web Services […]
Introduction The Azure cloud infrastructure has become a common place for users to deploy virtual machines on the cloud due to its flexibility, ease of deployment, and cost benefits. Microsoft has expanded Azure to include a marketplace with thousands of certified, open source, and community software applications and developer services, pre-configured for Microsoft Azure. This […]
The Hortonworks Sandbox is delivered as a Dockerized container with the most common ports already opened and forwarded for you. If you would like to open even more ports, check out this tutorial.
Apache, Hadoop, Falcon, Atlas, Tez, Sqoop, Flume, Kafka, Pig, Hive, HBase, Accumulo, Storm, Solr, Spark, Ranger, Knox, Ambari, ZooKeeper, Oozie, Phoenix, NiFi, HAWQ, Zeppelin, Atlas, Slider, Mahout, MapReduce, HDFS, YARN, Metron and the Hadoop elephant and Apache project logos are either registered trademarks or trademarks of the Apache Software Foundation in the United States or other countries.