HOW TO MOVE DATA INTO AN HBASE TABLE USING FLUME-NG

-
The first Hbase sink was commited to the Flume 1.2.x trunk few days ago. In this post we'll see how we can use this sink to collect data from a file stored in the local filesystem and dump this data into an Hbase table. We should have Flume built from the trunk in order to achieve that. If you haven't built it yet and looking for some help, you can visit my other post that shows how to build and use Flume-NG at this link :
http://cloudfront.blogspot.in/2012/06/how-to-build-and-use-flume-ng.html

 First of all we have to write the configuration file for our agent. This agent will collect data from the file and dump it into the Hbase table. A simple configuration file might look like this :


 hbase-agent.sources = tail
hbase-agent.sinks = sink1
hbase-agent.channels = ch1
hbase-agent.sources.tail.type = exec
hbase-agent.sources.tail.command = tail -F /home/mohammad/demo.txt
hbase-agent.sources.tail.channels = ch1
hbase-agent.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
hbase-agent.sinks.sink1.channel = ch1
hbase-agent.sinks.sink1.table = demo
hbase-agent.sinks.sink1.columnFamily = cf
hbase-agent.sinks.sink1.serializer = org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
hbase-agent.sinks.sink1.serializer.payloadColumn = col1
hbase-agent.sinks.sink1.serializer.keyType = timestamp
hbase-agent.sinks.sink1.serializer.rowPrefix = 1
hbase-agent.sinks.sink1.serializer.suffix = timestamp
hbase-agent.channels.ch1.type=memory

 Save this in a file called hbase-agent.conf inside the /conf directory of your Flume distribution. Now start your Hadoop and Hbase and create a table called demo with a column family called cf. Now open another terminal and change you directory to /conf inside your FlumeHome. Then start your agent using the below specified command :
$ bin/flume-ng agent -n hbase-agent -c conf/ -f conf/hbase-agent.conf

 Now go back to your Hbase shell and scan the demo table. If everything was ok then you will see something like this :

hbase(main):004:0> scan 'demo'
ROW                                    COLUMN+CELL                                                                                                     
 11339770815331                        column=cf:col1, timestamp=1339770818340, value=value1                                                           
 11339770815332                        column=cf:col1, timestamp=1339770818342, value=value6                                                           
2 row(s) in 0.0500 seconds


 NOTE : I have taken a small text file called demo.txt here which has following few lines in it 

value1
value2
value3
value4
value5
value6

Comments

  1. IltnOctober 11, 2012 at 7:04 PM

    Hi - thx for the post, is there a way to fill two or more HBase columns at once?
    Cheers.

    ReplyDelete
  2. TechzoneFebruary 18, 2013 at 1:16 PM

    Hi -thanks for the post, it is working fine but i have a dot that how it is going to store data from dump to hbase table...?

    Techzone

    ReplyDelete
  3. TariqFebruary 18, 2013 at 2:05 PM

    you are welcome. could you please tell me which dump are you talking about?is it the RDBMS dump?

    ReplyDelete

Post a Comment

Popular posts from this blog

Fun with HBase shell

-
Image
HBase shell is great, specially while getting yourself familiar with HBase. It provides lots of useful shell commands using which you can perform trivial tasks like creating tables, putting some test data into it, scanning the whole table, fetching data from a specific row etc etc. Executing  help  on HBase shell will give you the list of all the HBase shell commands. If you need help on a specific command, type help "command" . For example, help "get" will give you a detailed explanation of the get command. But this post is not about the above said stuff. We will try to do something fun here. Something which is available, but less known. So, get ready, start your HBase daemons, open HBase shell and get your hands dirty. For those of us who are unaware, HBase shell is based on  JRuby , the Java Virtual Machine-based implementation of Ruby. More specifically, it uses the  Interactive Ruby Shell (IRB) , which is used to enter Ruby commands and get an immedia
Read more

BETWEEN OPERATOR IN HIVE

-
Hive is a wonderful tool for those who like to perform batch operations to process their large amounts of data residing on a Hadoop cluster and who are comparatively new to the NOSQL world. Not only it provides us warehousing capabilities on top of a Hadoop cluster, but also a superb SQL like interface which makes it very easy to use and makes our task execution more familiar. But, one thing which newbies like me always wanted to have is the support of BETWEEN operator in Hive. Since the release of version 0.9.0  earlier  this year, Hive provides us some new and very useful features. BETWEEN operator is one among those.
Read more

HOW TO SETUP AND CONFIGURE 'ssh' ON LINUX (UBUNTU)

-
SSH (Secure Shell) is a network protocol  secure data communication, remote shell services or command execution and other secure network services between two networked computers that it connects via a secure channel  over an insecure network. The ssh server runs on a machine (server) and ssh client runs on another machine (client). ssh has 2 main components : 1- ssh : T he command we use to connect to remote machines - the client.  2-  sshd : T he daemon that is running on the  server  and allows clients to connect to the server. ssh is pre-enabled on Linux, but in order to start sshd daemon , we need to install ssh first. Use this command to do that : $ sudo apt-get install ssh This will install ssh on your machine. In order to check if ssh is setup properly do this : $ which ssh It will throw this line on your terminal /usr/bin/ssh $ which sshd It will throw this line on your terminal /usr/bin/sshd SSH uses  public-key cryptography  to  authenticate  the remote co
Read more