1. Introduction
This is an in-depth article related to the Apache Hadoop Development Tools Eclipse. Eclipse is used for developing java applications. Apache Hadoop is used for analyzing and storing big data. Developers use eclipse versions like Indigo, Juno, Kepler, Oxygen, and Photon. Hadoop Eclipse tools work well with eclipse above or equal to version 3.6. You can manage multiple versions of Hadoop from this IDE with plugins.
Table Of Contents
2. Apache Hadoop Development Tools Eclipse
2.1 Prerequisites
Java 7 or 8 is required on the linux, windows or mac operating system. Maven 3.6.1 is required for building the spring and hibernate application. Eclipse Oxygen can be used for this example. Apache Hadoop 2.6 can be downloaded from Hadoop Website.
2.2 Download
You can download Java 8 can be downloaded from the Oracle website. Apache Maven 3.6.1 can be downloaded from Apache site. Eclipse Oxygen can be downloaded from the eclipse web site. Apache Hadoop 2.6 can be downloaded from Hadoop Website. The Hadoop Eclipse plugin can be downloaded from this site.
2.3 Setup
You can set the environment variables for JAVA_HOME and PATH. They can be set as shown below:
Setup
JAVA_HOME="/desktop/jdk1.8.0_73" export JAVA_HOME PATH=$JAVA_HOME/bin:$PATH export PATH
The environment variables for maven are set as below:
Maven Environment
JAVA_HOME=”/jboss/jdk1.8.0_73″ export M2_HOME=/users/bhagvan.kommadi/Desktop/apache-maven-3.6.1 export M2=$M2_HOME/bin export PATH=$M2:$PATH
2.4 How to download and install Eclipse
2.4.1 Eclipse Oxygen Setup
The ‘eclipse-java-oxygen-2-macosx-cocoa-x86_64.tar’ can be downloaded from the eclipse website. The tar file is opened by double click. The tar file is unzipped by using the archive utility. After unzipping, you will find the eclipse icon in the folder. You can move the eclipse icon from the folder to applications by dragging the icon.
2.4.2 Launching IDE
Eclipse has features related to language support, customization, and extension. You can click on the eclipse icon to launch eclipse. The eclipse screen pops up as shown in the screenshot below:
You can select the workspace from the screen which pops up. The attached image shows how it can be selected.
You can see the eclipse workbench on the screen. The attached screenshot shows the Eclipse project screen.
Java Hello World class prints the greetings. The screenshot below is added to show the class and execution on the eclipse.
2.5 Hadoop Setup
You need to configure HADOOP_HOME as below:
Hadoop Home
export HADOOP_HOME=/users/bhagvan.kommadi/desktop/hadoop-2.6/
You need to configure $HADOOP_HOME/etc/hadoop/core-site.xml as below:
Core Site – Hadoop Configuration
<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--> <!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://apples-MacBook-Air.local:8020</value>
</property>
</configuration>
You need to start running Hadoop by using the command below :
Hadoop Execution
cd hadoop-2.6/cd sbin./start-dfs.sh
The output of the commands is shown below :
Hadoop Execution
apples-MacBook-Air:sbin bhagvan.kommadi$ ./start-dfs.sh20/06/29 20:26:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicableStarting namenodes on [apples-MacBook-Air.local]apples-MacBook-Air.local: Warning: Permanently added the ECDSA host key for IP address 'fe80::4e9:963f:5cc3:a000%en0' to the list of known hosts.Password:apples-MacBook-Air.local: starting namenode, logging to /Users/bhagvan.kommadi/desktop/hadoop-2.9.1/logs/hadoop-bhagvan.kommadi-namenode-apples-MacBook-Air.local.outPassword:localhost: starting datanode, logging to /Users/bhagvan.kommadi/desktop/hadoop-2.9.1/logs/hadoop-bhagvan.kommadi-datanode-apples-MacBook-Air.local.outStarting secondary namenodes [0.0.0.0]Password:0.0.0.0: starting secondarynamenode, logging to /Users/bhagvan.kommadi/desktop/hadoop-2.9.1/logs/hadoop-bhagvan.kommadi-secondarynamenode-apples-MacBook-Air.local.out20/06/29 20:27:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
2.6 Hadoop Tools
Apache Hadoop Tools Eclipse has features related to map reduce projects, hadoop cluster configuration, hadoop wizards, wizards for mapper, reducer, driver, job listing, job execution, HDFS node inspection, Zoo keeper node inspection. Hadoop eclipse plugin jar file hadoop-eclipse-plugin-2.6.0.jar can be placed at eclipse dropins/plugins after creating the folder plugins under dropins. Eclipse needs to be started with clean option as shown in the command below:
Eclipse Execution
./eclipse -clean -vmargs -XstartOnFirstThread
Eclipse launches and you can select the workspace. You can select Windows -> Perspective-> Other. You can see the screenshot below where Map Reduce perspective is shown.
Click on the Map reduce perspective and you can see the project as shown below:
You can create a new Mapreduce location as shown in the screenshot below:
You can set the hadoop location info in the UI as shown below:
You can create a new project or see other wizards by selecting New . The UI is shown below:
Click on New -> Other to see the different wizards as shown below:
3. Download the Source Code
You can download the full source code of this example here: Apache Hadoop Development Tools Eclipse Tutorial
