Apache Hadoop

Apache Hadoop Development Tools Eclipse Tutorial

1. Introduction

This is an in-depth article related to the Apache Hadoop Development Tools Eclipse. Eclipse is used for developing java applications. Apache Hadoop is used for analyzing and storing big data. Developers use eclipse versions like Indigo, Juno, Kepler, Oxygen, and Photon. Hadoop Eclipse tools work well with eclipse above or equal to version 3.6. You can manage multiple versions of Hadoop from this IDE with plugins.


2. Apache Hadoop Development Tools Eclipse

2.1 Prerequisites

Java 7 or 8 is required on the linux, windows or mac operating system. Maven 3.6.1 is required for building the spring and hibernate application. Eclipse Oxygen can be used for this example. Apache Hadoop 2.6 can be downloaded from Hadoop Website

2.2 Download

You can download Java 8 can be downloaded from the Oracle website. Apache Maven 3.6.1 can be downloaded from Apache site. Eclipse Oxygen can be downloaded from the eclipse web site. Apache Hadoop 2.6 can be downloaded from Hadoop Website.  The Hadoop Eclipse plugin can be downloaded from this site.

2.3 Setup

You can set the environment variables for JAVA_HOME and PATH. They can be set as shown below:


export JAVA_HOME
export PATH

The environment variables for maven are set as below:

Maven Environment

export M2_HOME=/users/bhagvan.kommadi/Desktop/apache-maven-3.6.1
export M2=$M2_HOME/bin
export PATH=$M2:$PATH

2.4 How to download and install Eclipse

2.4.1 Eclipse Oxygen Setup

The ‘eclipse-java-oxygen-2-macosx-cocoa-x86_64.tar’ can be downloaded from the eclipse website. The tar file is opened by double click. The tar file is unzipped by using the archive utility. After unzipping, you will find the eclipse icon in the folder. You can move the eclipse icon from the folder to applications by dragging the icon.

2.4.2 Launching IDE

Eclipse has features related to language support, customization, and extension. You can click on the eclipse icon to launch eclipse. The eclipse screen pops up as shown in the screenshot below:

Hadoop Tools Eclipse - Launching IDE
Launching IDE

You can select the workspace from the screen which pops up. The attached image shows how it can be selected.

Hadoop Tools Eclipse - IntelliJ vs Eclipse
Eclipse Workspace

You can see the eclipse workbench on the screen. The attached screenshot shows the Eclipse project screen.

Hadoop Tools Eclipse - Eclipse Workbench
Eclipse Workbench

Java Hello World class prints the greetings. The screenshot below is added to show the class and execution on the eclipse.

Hadoop Tools Eclipse - Java Hello
Java Hello

2.5 Hadoop Setup

You need to configure HADOOP_HOME as below:

Hadoop Home

export HADOOP_HOME=/users/bhagvan.kommadi/desktop/hadoop-2.6/

You need to configure $HADOOP_HOME/etc/hadoop/core-site.xml as below:

Core Site – Hadoop Configuration

<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--  Licensed under the Apache License, Version 2.0 (the "License");  you may not use this file except in compliance with the License.  You may obtain a copy of the License at     http://www.apache.org/licenses/LICENSE-2.0   Unless required by applicable law or agreed to in writing, software  distributed under the License is distributed on an "AS IS" BASIS,  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the License for the specific language governing permissions and  limitations under the License. See accompanying LICENSE file.--> <!-- Put site-specific property overrides in this file. --> 

You need to start running Hadoop by using the command below :

Hadoop Execution

cd hadoop-2.6/cd sbin./start-dfs.sh

The output of the commands is shown below :

Hadoop Execution

apples-MacBook-Air:sbin bhagvan.kommadi$ ./start-dfs.sh20/06/29 20:26:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicableStarting namenodes on [apples-MacBook-Air.local]apples-MacBook-Air.local: Warning: Permanently added the ECDSA host key for IP address 'fe80::4e9:963f:5cc3:a000%en0' to the list of known hosts.Password:apples-MacBook-Air.local: starting namenode, logging to /Users/bhagvan.kommadi/desktop/hadoop-2.9.1/logs/hadoop-bhagvan.kommadi-namenode-apples-MacBook-Air.local.outPassword:localhost: starting datanode, logging to /Users/bhagvan.kommadi/desktop/hadoop-2.9.1/logs/hadoop-bhagvan.kommadi-datanode-apples-MacBook-Air.local.outStarting secondary namenodes []Password: starting secondarynamenode, logging to /Users/bhagvan.kommadi/desktop/hadoop-2.9.1/logs/hadoop-bhagvan.kommadi-secondarynamenode-apples-MacBook-Air.local.out20/06/29 20:27:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

2.6 Hadoop Tools

Apache Hadoop Tools Eclipse has features related to map reduce projects, hadoop cluster configuration, hadoop wizards, wizards for mapper, reducer, driver, job listing, job execution, HDFS node inspection, Zoo keeper node inspection. Hadoop eclipse plugin jar file hadoop-eclipse-plugin-2.6.0.jar can be placed at eclipse dropins/plugins after creating the folder plugins under dropins. Eclipse needs to be started with clean option as shown in the command below:

Eclipse Execution

./eclipse -clean -vmargs -XstartOnFirstThread

Eclipse launches and you can select the workspace. You can select Windows -> Perspective-> Other. You can see the screenshot below where Map Reduce perspective is shown.

Hadoop Tools Eclipse - eclipse prespecive
Eclipse Perspective

Click on the Map reduce perspective and you can see the project as shown below:

Map reduce Project

You can create a new Mapreduce location as shown in the screenshot below:

Mapreduce location

You can set the hadoop location info in the UI as shown below:

Hadoop Configuration

You can create a new project or see other wizards by selecting New . The UI is shown below:

Hadoop Project

Click on New -> Other to see the different wizards as shown below:

Map reduce Wizards

3. Download the Source Code

You can download the full source code of this example here: Apache Hadoop Development Tools Eclipse Tutorial

Bhagvan Kommadi

Bhagvan Kommadi is the Founder of Architect Corner & has around 20 years’ experience in the industry, ranging from large scale enterprise development to helping incubate software product start-ups. He has done Masters in Industrial Systems Engineering at Georgia Institute of Technology (1997) and Bachelors in Aerospace Engineering from Indian Institute of Technology, Madras (1993). He is member of IFX forum,Oracle JCP and participant in Java Community Process. He founded Quantica Computacao, the first quantum computing startup in India. Markets and Markets have positioned Quantica Computacao in ‘Emerging Companies’ section of Quantum Computing quadrants. Bhagvan has engineered and developed simulators and tools in the area of quantum technology using IBM Q, Microsoft Q# and Google QScript. He has reviewed the Manning book titled : "Machine Learning with TensorFlow”. He is also the author of Packt Publishing book - "Hands-On Data Structures and Algorithms with Go".He is member of IFX forum,Oracle JCP and participant in Java Community Process. He is member of the MIT Technology Review Global Panel.
Notify of

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Inline Feedbacks
View all comments
Back to top button