MapReduce jobs running through Eclipse don't appear in the JobTracker Web UI at 50030

Hello all,

      In response to an earlier post of mine, that shows how to run a MapReduce job through Eclipse IDE, I quite frequently receive comments that the users are not able to see the status of their MapReduce job, which they are currently running, on the JobTracker Web UI.

The trick is very simple. Just add the following 2 lines in your code where you are doing all the configuration. Something like this :


Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://localhost:9000");
conf.set("mapred.job.tracker", "localhost:9001");

This should do the trick for you. After doing this just point your web browser to the JobTracker Web UI at localhost:50030.

**Modify the hostname and port address as per your configuration.

To know about Hadoop configuration and setup you can go to this link. It shows the entire process in detail.

HTH

Unable To Connect Your Phone In VirtualBox Through USB Cable??

Recently I came to know about the Premium Suite For Samsung Galaxy Note. And being a proud owner of this great device it was quite obvious that I wanted to take the pleasure of this. So, I thought of upgrading my phone through Samsung Kies. But I have been working of Linux since last couple of years, so got kinda stuck as Kies doesn't come for Linux. So, I installed Oracle VirtualBox on my Ubuntu Box and installed Windows 8 in it. After that I quickly installed Kies in it. But, to my surprise I was not able to connect my phone. After some diagnosis I found the error message shown below :


Failed to access the USB subsystem. VirtualBox is not currently allowed to access USB devices. You can change this by adding your user to the 'vboxusers' group. Please see the user manual for a more detailed explanation.

And the detail message was :


NS_ERROR_FAILURE (0x00004005)
Component: 
Host
Interface: 
IHost {dab4a2b8-c735-4f08-94fc-9bec84182e2f}
Callee: 
IMachine {5eaa9319-62fc-4b0a-843c-0cb1940f8a91}

If you are also facing something similar, don't worry. There is simple one line workaround for this problem. But before that let me put one thing. This is not a problem with VirtualBox. You just have to add the current user to the vboxusers group. Issue the following command to do this :
sudo usermod -aG vboxusers <your username>

*Replace <your username> with your user.

Do not forget  to log off and back in to finalize the change in permissions. After logging in restart your VM, open Kies, plug-in your phone and you are good to go.

HOW TO BENCHMARK HBASE USING YCSB

YCSB (Yahoo Cloud Serving Benchmark) is a popular tool for evaluating the performance of different key-value and cloud serving stores. You can use it to test the read/write performance of your Hbase cluster and trust me it's very effective. In this post i'll show you how to build and use YCSB for your particular version of Hbase. So, this is just about setting up and using YCSB and not about YCSB itself. For detailed info on YCSB you can go to the below specified links :

1- Github-YCSB page : https://github.com/brianfrankcooper/YCSB
2- The paper from ACM Symposium on Cloud Computing, "Benchmarking Cloud Serving Systems with YCSB" : http://research.yahoo.com/files/ycsb.pdf

So, let us get started...

Step1- Clone the YCSB git repository :

apache@hadoop:~$ git clone http://github.com/brianfrankcooper/YCSB.git

This will create a directory caleed YCSB inside your current directory. (It might take some time depending on your internet connection speed. So, be patient)

Step2- Go inside this newly created YCSB directory and move inside the hbase directory. You will find an xml file here named as pom.xml. Open this pom.xml file and edit it so that it looks like this :

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>com.yahoo.ycsb</groupId>
    <artifactId>root</artifactId>
    <version>0.1.4</version>
  </parent>
  <artifactId>hbase-binding</artifactId>
  <name>HBase DB Binding</name>
  <dependencies>
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase</artifactId>
      <!--<version>${hbase.version}</version>-->
      <version>0.94.4</version>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-core</artifactId>
      <!--<version>1.0.0</version>-->
      <version>1.0.4</version>
    </dependency>
    <dependency>
      <groupId>com.yahoo.ycsb</groupId>
      <artifactId>core</artifactId>
      <version>${project.version}</version>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-assembly-plugin</artifactId>
        <version>${maven.assembly.version}</version>
        <configuration>
          <descriptorRefs>
            <descriptorRef>jar-with-dependencies</descriptorRef>
          </descriptorRefs>
          <appendAssemblyId>false</appendAssemblyId>
        </configuration>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>single</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

      Pay attention to the lines in red. These are the changes that you have to make in order to build YCSB without any problem for your specific version of Hbase.

NOTE : As of this writing I am usign hadoop-1.04 and hbase-0.94.4, so I have mentioned these versions in the above shown file. You have to specify the versions which you are going to use.

Step3- Now, go back to your terminal and move inside the YCSB directory :
apache@hadoop:~$ cd YCSB

Step4- It's time to do the build now :
apache@hadoop: /YCSB/ mvn clean package
This will start the build process. You can see all the information as the build process continues. If everything goes fine then you will see something like this on your terminal :


NOTE: If multiple descriptors or descriptor-formats are provided for this project, the value of this file will be non-deterministic!
[WARNING] Replacing pre-existing project main-artifact file: /hadoop/projects/YCSB/voldemort/target/archive-tmp/voldemort-binding-0.1.4.jar
with assembly file: /hadoop/projects/YCSB/voldemort/target/voldemort-binding-0.1.4.jar
[INFO]                                                                      
[INFO] ------------------------------------------------------------------------
[INFO] Building YCSB Release Distribution Builder 0.1.4
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.3:clean (default-clean) @ ycsb ---
[INFO]
[INFO] --- maven-checkstyle-plugin:2.6:checkstyle (validate) @ ycsb ---
[INFO]
[INFO] --- maven-assembly-plugin:2.2.1:single (default) @ ycsb ---
[INFO] Reading assembly descriptor: src/main/assembly/distribution.xml
[INFO] Processing sources for module project: com.yahoo.ycsb:core:jar:0.1.4
[INFO] Processing sources for module project: com.yahoo.ycsb:cassandra-binding:jar:0.1.4
[INFO] Processing sources for module project: com.yahoo.ycsb:hbase-binding:jar:0.1.4
[INFO] Processing sources for module project: com.yahoo.ycsb:hypertable-binding:jar:0.1.4
[INFO] Processing sources for module project: com.yahoo.ycsb:dynamodb-binding:jar:0.1.4
[INFO] Processing sources for module project: com.yahoo.ycsb:elasticsearch-binding:jar:0.1.4
[INFO] Processing sources for module project: com.yahoo.ycsb:infinispan-binding:jar:0.1.4
[INFO] Processing sources for module project: com.yahoo.ycsb:jdbc-binding:jar:0.1.4
[INFO] Processing sources for module project: com.yahoo.ycsb:mapkeeper-binding:jar:0.1.4
[INFO] Processing sources for module project: com.yahoo.ycsb:mongodb-binding:jar:0.1.4
[INFO] Processing sources for module project: com.yahoo.ycsb:orientdb-binding:jar:0.1.4
[INFO] Processing sources for module project: com.yahoo.ycsb:redis-binding:jar:0.1.4
[INFO] Processing sources for module project: com.yahoo.ycsb:voldemort-binding:jar:0.1.4
[INFO] Processing sources for module project: com.yahoo.ycsb:ycsb:pom:0.1.4
[INFO] Building tar : /hadoop/projects/YCSB/distribution/target/ycsb-0.1.4.tar.gz
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] YCSB Root ......................................... SUCCESS [1.940s]
[INFO] Core YCSB ......................................... SUCCESS [23.149s]
[INFO] Cassandra DB Binding .............................. SUCCESS [7.421s]
[INFO] HBase DB Binding .................................. SUCCESS [15.638s]
[INFO] Hypertable DB Binding ............................. SUCCESS [2.805s]
[INFO] DynamoDB DB Binding ............................... SUCCESS [3.451s]
[INFO] ElasticSearch Binding ............................. SUCCESS [8.123s]
[INFO] Infinispan DB Binding ............................. SUCCESS [2:27.468s]
[INFO] JDBC DB Binding ................................... SUCCESS [18.235s]
[INFO] Mapkeeper DB Binding .............................. SUCCESS [10.011s]
[INFO] Mongo DB Binding .................................. SUCCESS [4.874s]
[INFO] OrientDB Binding .................................. SUCCESS [19.702s]
[INFO] Redis DB Binding .................................. SUCCESS [3.960s]
[INFO] Voldemort DB Binding .............................. SUCCESS [14.181s]
[INFO] YCSB Release Distribution Builder ................. SUCCESS [7.076s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4:48.305s
[INFO] Finished at: Mon Feb 04 01:13:00 IST 2013
[INFO] Final Memory: 107M/737M
[INFO] ------------------------------------------------------------------------

This shows that the build has been completed successfully and you are all set to go. 

Step5- Step4 will create a directory named target inside your /YCSB/distribution/ directory. You will find the YCSB tar file here, ycsb-0.1.4.tar.gz in my case. Copy this file to some location of your choice and extract it. This will give you the ycsb-1.0.4 directory which contains all the important and necessary stuff.

Step6- Move inside the ycsb-1.0.4 directory where you will find a directory called /hbase-binding. Go inside the /hbase-binding and open the /lib directory situated there. Copy the following jars from your /HBASE_HOME/lib into this /lib directory :

     1-slf4j-api-*.jar

     2-slf4j-log4j12-*.jar

     3-zookeeper-*.jar

Step7- You will find another directory named /conf inside /hbase-binding. You will find an xml file here named as hbase-site.xml file. Replace this hbase-site.xml file with the habse-site.xml present in your /HBASE_HOME/conf directory.

Step8- You are all set for testing your Hbase now. Start the Hadoop and Hbase processes and go inside ycsb-1.0.4. Now, issue the following command to load test your Hbase deployment :

apache@hadoop:/ycsb-0.1.4$ bin/ycsb load hbase -P workloads/workloada -p columnfamily=f1 -p recordcount=1000000 -p threadcount=4 -s | tee -a workloada.dat

This will start the load test and after sometime it will give you the result summary. Do not get overwhelmed by the great amount of information displayed on your terminal after this operation. For our convenience we have piped this ycsb command with the Linux tee command and written the entire output information to the terminal and the workloada.dat. You will find this file inside your ycsb-0.1.4

directory which contains the same content as your terminal has. You can extract useful insights from this file(or from your terminal) like :

The overall runtime in milliseconds

Throughput i.e. operations per second

Number of operations

AverageLatency etc etc

Here are some of the lines from my terminal :

[OVERALL], RunTime(ms), 73258.0

[OVERALL], Throughput(ops/sec), 13650.386305932458

[UPDATE], Operations, 4

[UPDATE], AverageLatency(us), 530564.25

[UPDATE], MinLatency(us), 65895

[UPDATE], MaxLatency(us), 1642179

I hope you found this post helpful. Stay connected for more :)

How to install Java 6 on Ubuntu

In one of the previous post of mine I have shown you how to install Sun(Oracle) Java on Ubuntu through its repository. This will, by default, install Java-7 on your machine as Ubuntu 12.04(and onwards) have Java-7 in their repository. But, sometimes you may come across a situation wherein you need some specific version of Java. For example, it is advisable to use Java-6 while trying to configure or use Apache Hadoop. In such a scenario you need to download the appropriate version of Java and install it manually. It is again a straightforward process. Just follow the steps below :

Note : Java-6 has been taken here, as an example, on a machine running Ubuntu 12.10

Step 1 : Download the required version of Java from the official download page. It will download  jdk-6u38-linux-x64.bin inside your Downloads directory.

Step 2 : Go to the directory where jdk was downloaded(Download here) and make it an executable file using this command :
apache@hadoop:~/Downloads$ sudo chmod +x jdk-6u38-linux-x64.bin

Step 3 : Run this executable .bin file using :
apache@hadoop:~/Downloads$ ./jdk-6u38-linux-x64.bin

Step 4 : Now move the file to the /usr/lib/jvm directory with the help of mv command :
apache@hadoop:~/Downloads$ sudo mv jdk1.6.0_38/ /usr/lib/jvm/

Step 5 : This is the step where you actually do the installation. Use the following commands to do that :
apache@hadoop:~/Downloads$ sudo update-alternatives  --install /usr/bin/javac javac /usr/lib/jvm/jdk1.6.0_38/bin/javac 1
apache@hadoop:~/Downloads$ sudo update-alternatives --install /usr/bin/java java /usr/lib/jvm/jdk1.6.0_38/bin/java 1
apache@hadoop:~/Downloads$ sudo update-alternatives --install /usr/bin/javaws javaws /usr/lib/jvm/jdk1.6.0_38/bin/javaws 1

Once you are done with these steps you are good to go. One step is actually still required though. The above procedure will just install Java-6 on your machine. In case you have multiple versions of Java installed on your machine, you need to set which version you want to use as your default Java. To do so issue the command shown below :
apache@hadoop:~/Downloads$ sudo update-alternatives --config java

This will show you all the Java versions installed on your machine. Choose the number corresponding to the version of your choice and you are all set.

Salesforce.com's Phoenix : SQL layer for your Hbase

Ever wished to have the ability to write SQL queries for your data stored in Hbase?I know your answer is gonna be Hive. But I am talking about something which doesn't incur heavy start-up costs and which is based on native HBase APIs rather than going through the MapRreduce framework. Need not worry. Salesforce.com comes to the rescue this time. Salesforce.com has recently announced Phoenix, an SQL layer over HBase. What do I meant by that???

Phoenix is an SQL layer over HBase delivered as a client-embedded JDBC driver targeting low latency queries over HBase data. Phoenix takes our SQL query, compiles them into a series of HBase scans, and orchestrates the running of those scans to produce regular JDBC result sets. Cool..Isn't it?

Phoenix doesn't depend on MapReduce, but that doesn't mean that it doesn't believe in the philosophy of bringing computation closer to data. It very well does that, through :
      Coprocessors : To perform operations on the server-side thus minimizing client/server data transfer
      Custom filters : to prune data as close to the source as possible

And the best part is that there is no adverse effect on the performance.

I am showing a couple a graphs below which present relative performance between Phoenix and some other related products (Courtesy : Phoenix Github page)

Phoenix vs Hive (running over HDFS and HBase)

Phoenix vs Impala (running over HBase)

The performance, as you can see from these graphs is quite good. For a detailed info you can visit this link.

Phoenix stores table metadata in an HBase table and keep it versioned, such that snapshot queries over prior versions will automatically use the correct schema. Direct use of the HBase API, along with coprocessors and custom filters, results in performance on the order of milliseconds for small queries, or seconds for tens of millions of rows.

Phoenix SQL Support

Phoenix supports all typical SQL query statement clauses, including SELECT, FROM, WHERE, GROUP BY, HAVING, ORDER BY, etc. It also supports a full set of DML commands as well as table creation and versioned incremental alterations through our DDL commands. We try to follow the SQL standards wherever possible. For a complete set of all the things which Phoenix supports you can visit the language reference page.

But, there are certain things which Phoenix doesn't support as of now. They include :

       Joins : Single table only currently.

       Derived tables : Nested queries along with TopN queries are coming soon.

       Relational operators : Union, Intersect, Minus.

       Miscellaneous built-in functions.

I don't feel it's bad considering that Phoenix has just born :)

For an in-depth info about Phoenix, you can visit Phoenix Wiki.

In the next post i'll try to write about building Phoenix with some hands-on. Stay connected till then.

Audi's Autonomous Piloted Parking : Real Life Transformers???

Ever wished to have a car which would make you feel yourself like 007. NO??? Need not worry. Soon you will be able to do that (hopefully ;) ). How?? Keep on reading...

At the Consumer Electronics Show (CES) which was held in Las Vegas few days ago Audi showed something which would definitely blow your mind. CES is considered as world’s most important electronics show. There, Audi had shown several new technological aspects and one of them is through this self-driven Audi A7 showing off the brand’s piloted parking system. It reminds of James Bond movie Tomorrow Never Dies where Bond would operate his BMW 750iL with a remote control. Here we have Annie Lien (Senior Engineer, Electronics Lab at Volkswagen Group of America), who shows us a cool demo of this awesome feature. She directs an Audi A7 using her phone and it parks itself at the Mandarin Oriental Hotel Las Vegas and on leaving hotel; she directs it off from the parking to her. They call it as Park Assist. And this what Audi has to say about its Park Assist technology :

"Audi’s automatic parking systems operate by means of either ultrasound or cameras, which display images via the onboard monitor. One   particularly convenient solution is park assist. When backing into a parking space, it performs all the necessary steering movements; it can handle both parallel parking and parking perpendicular to the street.

The system finds a parking space with ultrasound sensors that scan the roadside in two dimensions while driving at moderate speed. The system notifies the driver via a message in the display once the sensors have found a space which is large enough.

If the driver wishes to park in the space, he or she shifts into reverse and the park assist system takes over the steering. The driver must accelerate, shift gears, and brake. When parallel parking, the detected space is large enough if it is about 80 centimeters (2.62 ft) longer than the vehicle itself. Park assist can perform multi-point parking maneuvers and also offers support in leaving parallel parking spaces.

Another technology from Audi is the parking system plus with surround view cameras. Four small cameras – in the single-frame grille, at the rear and in the side mirror housings – record the vehicle’s immediate surroundings. The driver can call up a variety of views on the large onboard monitor, including a top-down virtual view. On corners or junctions with an obstructed view, the system can analyze cross-traffic otherwise invisible to the driver in front of or behind the vehicle."

To get a feel of that you can watch this video :

Hope you enjoyed this. Stay connected for more.

Google Spanner : The Future Of NoSQL

Quite often, while working with Hbase, I used to feel how cool it would be to have a database that can replicate my data to datacenters across the world consistently. So that I can take the pleasure of global availability and geographic locality. And also which will save my data even in case of some catastrophe or natural disaster. Which supports general-purpose transactions, and provides a SQL-based query language. And which has features of an SQL database as well. But it was only untill recently I found out that it is not an imagination anymore.

I was sitting with a senior+friend of mine at a Cafe Coffee Day nearby and having a casual chat on BigData stuff. During the discussion he told me about something called as SPANNER.
(You might be wondering, why the heck I have emphasized on the word spanner so much. Believe me, you will do the same after reading this post).

After that meeting I almost forgot about that incident. Out of the blue, the word spanner flashed back to my mind 2 days ago and I started googling about spanner and the search led me to this Google research page, which just blew my mind. Google has already been working extensively on something,which they call as Spanner.

Spanner is a scalable, globally-distributed database designed, built, and deployed at Google. At the highest level of abstraction, it is a database that shards data across many sets of Paxos state machines in datacenters spread all over the world. Replication is used for global availability and geographic locality; clients automatically failover between replicas. Spanner automatically reshards data across machines as the amount of data or the number of servers changes, and it automatically migrates data across machines (even across datacenters) to balance load and in response to failures. Spanner is designed to scale up to millions of machines across hundreds of datacenters and trillions of rows. Applications can use Spanner for high availability,even in the face of wide-area natural disasters, by replicating their data within or even across continents.

We can think of Spanner as globally-distributed database that may spread across the continents covering the planet. Spanner provides several very interesting features :
1 : The replication configurations for data can be controlled dynamically by the applications in a fine grained manner.
2 : It gives us the ability to control which datacenters contain which data.
3 : To control read latency it gives application the ability to decide how far data is from its users etc etc.

But there are 2 things which really stand out : externally consistent reads and writes, and globally consistent reads across the database at a timestamp. Both these things are really difficult to implement in a distributed database. These features enable Spanner to support consistent backups, consistent MapReduce executions, and atomic schema updates, all at global scale, and even in the presence of ongoing transactions.

Few words on the Structure :

A Spanner deployment is called a universe. Spanner is organized as a set of zones, where each zone is somewhat like a Bigtable deployment. Zones can be added to or removed from a running system as new datacenters are brought into service and old ones are turned off. The set of zones is also the set of locations across which data can be replicated. The figure drawn below shows the Spanner server organization :

A zone has one zonemaster and between one hundred and several thousand spanservers. The former assigns data to spanservers; the latter serve data to clients. The per-zone location proxies are used by clients to locate the spanservers assigned to serve their data. The universe master and the placement driver are currently singletons. The universe master is primarily a console that displays status information about all the zones for interactive debugging. The placement driver handles automated movement of data across zones on the timescale of minutes. The placement driver periodically communicates with the spanservers to find data that needs to be moved, either to meet updated replication constraints or to balance load.

For a detailed info you can download the original paper (used as the reference) from here.

I hope you enjoyed reading this post and knowing about Spanner as much as I did. Don't forget to provide me your comments and/or suggestions. Thank you.