HOW TO RUN MAPREDUCE PROGRAMS USING ECLIPSE

-
Hadoop provides us a plugin for Eclipse that helps us to connect our Hadoop cluster to Eclipse. We can then run MapReduce jobs and browse Hdfs, through the Eclipse itself. But it requires a few things to be done in order to achieve that. Normally, it is said that we just have to copy hadoop-eclipse-plugin-*.jar to the eclipse/plugins directory in order to get things going. But unfortunately it did not work for me. When I tried to connect eclipse to my Hadoop cluster it threw this error :
An internal error occurred during: "Map/Reduce location status updater".
org/codehaus/jackson/map/JsonMappingException

 You may face some different error, but it would be somewhat similar to this. This is because of the fact that some required jars are missing from the plugin that comes with Hadoop. Then, I tried a few things and it turned out to be positive.
So, I thought of sharing it, so that if anybody else is facing the same issue, can try it out. Just try the steps outlined below and let me know if it works for you.

First of all setup a Hadoop cluster properly on your machine. If you need some help on that just go here. Then download eclipse compatible with your environment from eclipse home. Also set your HADOOP_HOME to point to your hadoop folder. 

 Now, follow these steps :

 1- Go to your HADOOP_HOME/contrib folder. Copy the hadoop-eclipse-plugin-*.jar somewhere and extract it. This will give a folder named hadoop-eclipse-plugin-*

 2- Now, add following 5 jars from your HADOOP_HOME/lib folder to the hadoop-eclipse-plugin-*/lib folder, you have got just now after extracting the plugin :
    commons-configuration-1.6.jar
    commons-httpclient-3.0.1.jar
    commons-lang-2.4.jar
    jackson-core-asl-1.0.1.jar
    jackson-mapper-asl-1.0.1.jar
3- Now, modify the hadoop-eclipse-plugin-*/META-INF/MANIFEST.MF file and change the Bundle-ClassPath to :
Bundle-ClassPath: classes /, lib / hadoop-core.jar, lib/commons-cli-1.2.jar, lib/commons-httpclient-3.0.1.jar, lib/jackson-core-asl-1.0.1.jar , lib/jackson-mapper-asl-1.0.1.jar, lib/commons-configuration-1.6.jar, lib/commons-lang-2.4.jar

4- Now, re 'jar' the package and place this new jar inside eclipse/plugin directory and restart the eclipse. 

You are good to go now. Do let me know it it doesn't work for you.

 NOTE : For details you can visit the official home page.

  Thank you.

 EDIT : If you are not able to see the job status at the JobTracker port(50070) you might find this post of mine useful.

Comments

  1. me.busybodyAugust 19, 2012 at 2:25 PM

    Hi
    I have this error.
    An internal error occurred during: "Connecting to DFS hadoop-master01".
    org/apache/commons/configuration/Configuration

    Please advise.

    ReplyDelete
  2. UnknownAugust 28, 2012 at 10:47 AM

    hi i am using hadoop-1.0.1 on windows machine with cygwin...and i didnt found any hadoop-eclipse-plugin-*.jar in my hadoop installation i found an folder in my hadoop installation location like hadoop-1.0.1\src\contrib\eclipse-plugin but there was no jar found any where for the eclipse...what to do now?

    ReplyDelete
  3. TariqAugust 28, 2012 at 6:49 PM

    Hello DeviKiran, I had tested it long back using 0.20.203.0, and the jar was present in that distribution. I don't know the reason, but they have removed it from 1.x release. Try to build it using and creating the jar.

    ReplyDelete
    Replies
    1. Rohit KondekarNovember 27, 2012 at 1:15 AM

      hey i did build the jar and added those jackson jars u told to lib and rejared it... but its not getting detected in eclipse. What could be the reason? any specific way of jaring up?

      Delete
    2. TariqNovember 27, 2012 at 2:13 AM

      Hi Rohit, nothing special..just a usual jar..make sure you are meeting all the dependencies..perhaps some dependency is missing..give it a few more shots, if it still doesn't work just let me know..i'll send you the working jar, in case you need it.

      Delete
  4. AnonymousDecember 4, 2012 at 11:32 PM

    I'm trying to get this to work in 1.1.1. Build the plugin, unjar'd and added the specified jar files, but still can't get it to work. Returning the same error.

    ReplyDelete
  5. priyaMarch 21, 2013 at 11:29 AM

    Hi Tariq ... I have one issue regarding running hadoop program through eclipse. When I run Program through eclipse the web interface of hadoop doesnt show any progress neither there is any entry in log but it actully runs as the output can be seen in eclipse console. when I run the same program through terminal I dont find this problem. I didnt found any appropriate answer to this problm on google.....

    ReplyDelete
  6. TariqMarch 21, 2013 at 4:07 PM

    Hello ma'am, this is happening because eclipse is actually running all the daemons on its own instead of actually submitting the job to the JT. adding these 2 line in the code should do the trick :
    conf.set("fs.default.name", "YOUR_NN_HOSTNAME:9000");
    conf.set("mapred.job.tracker", "YOUR_JT_HOSTNAME:9001");

    ReplyDelete

Post a Comment

Popular posts from this blog

Fun with HBase shell

-
Image
HBase shell is great, specially while getting yourself familiar with HBase. It provides lots of useful shell commands using which you can perform trivial tasks like creating tables, putting some test data into it, scanning the whole table, fetching data from a specific row etc etc. Executing  help  on HBase shell will give you the list of all the HBase shell commands. If you need help on a specific command, type help "command" . For example, help "get" will give you a detailed explanation of the get command. But this post is not about the above said stuff. We will try to do something fun here. Something which is available, but less known. So, get ready, start your HBase daemons, open HBase shell and get your hands dirty. For those of us who are unaware, HBase shell is based on  JRuby , the Java Virtual Machine-based implementation of Ruby. More specifically, it uses the  Interactive Ruby Shell (IRB) , which is used to enter Ruby commands and get an immedia
Read more

BETWEEN OPERATOR IN HIVE

-
Hive is a wonderful tool for those who like to perform batch operations to process their large amounts of data residing on a Hadoop cluster and who are comparatively new to the NOSQL world. Not only it provides us warehousing capabilities on top of a Hadoop cluster, but also a superb SQL like interface which makes it very easy to use and makes our task execution more familiar. But, one thing which newbies like me always wanted to have is the support of BETWEEN operator in Hive. Since the release of version 0.9.0  earlier  this year, Hive provides us some new and very useful features. BETWEEN operator is one among those.
Read more

HOW TO SETUP AND CONFIGURE 'ssh' ON LINUX (UBUNTU)

-
SSH (Secure Shell) is a network protocol  secure data communication, remote shell services or command execution and other secure network services between two networked computers that it connects via a secure channel  over an insecure network. The ssh server runs on a machine (server) and ssh client runs on another machine (client). ssh has 2 main components : 1- ssh : T he command we use to connect to remote machines - the client.  2-  sshd : T he daemon that is running on the  server  and allows clients to connect to the server. ssh is pre-enabled on Linux, but in order to start sshd daemon , we need to install ssh first. Use this command to do that : $ sudo apt-get install ssh This will install ssh on your machine. In order to check if ssh is setup properly do this : $ which ssh It will throw this line on your terminal /usr/bin/ssh $ which sshd It will throw this line on your terminal /usr/bin/sshd SSH uses  public-key cryptography  to  authenticate  the remote co
Read more