First trials with Hadoop

-

I followed the Hadoop Quickstart guide and the whole process is described below.

This post can be used as a reference for other people installing Hadoop.

My system is OpenSuse 10.3 and Java version is 1.6.0_03.

After downloading and installing the package I did the Standalone operation test:

$ mkdir input
$ cp conf/*.xml input
$ bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'
$ cat output/*

Here is the output (line feeds may be corrupted, sorry for that):

gtz2:/home/telzur/downloads/hadoop-0.14.4 # bin/hadoop jar hadoop-0.14.4-examples.jar grep input output 'dfs[a-z.]+' 08/01/05 15:47:13 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 08/01/05 15:47:13 INFO mapred.FileInputFormat: Total input paths to process : 3 08/01/05 15:47:13 INFO mapred.JobClient: Running job: job_local_1 08/01/05 15:47:13 INFO mapred.MapTask: numReduceTasks: 1 08/01/05 15:47:13 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/input/mapred-default.xml:0+180 08/01/05 15:47:13 INFO mapred.MapTask: numReduceTasks: 1 08/01/05 15:47:13 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/input/hadoop-default.xml:0+27489 08/01/05 15:47:13 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/input/hadoop-default.xml:0+27489 08/01/05 15:47:13 INFO mapred.MapTask: numReduceTasks: 1 08/01/05 15:47:13 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/input/hadoop-site.xml:0+178 08/01/05 15:47:13 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/input/hadoop-site.xml:0+178 08/01/05 15:47:14 INFO mapred.LocalJobRunner: reduce > reduce 08/01/05 15:47:14 INFO mapred.TaskRunner: Saved output of task 'reduce_3r6jh8' to file:/home/telzur/downloads/hadoop-0.14.4/grep-temp-346467784 08/01/05 15:47:14 INFO mapred.JobClient: Job complete: job_local_1 08/01/05 15:47:14 INFO mapred.JobClient: Counters: 9 08/01/05 15:47:14 INFO mapred.JobClient: Map-Reduce Framework 08/01/05 15:47:14 INFO mapred.JobClient: Map input records=940 08/01/05 15:47:14 INFO mapred.JobClient: Map output records=34 08/01/05 15:47:14 INFO mapred.JobClient: Map input bytes=27847 08/01/05 15:47:14 INFO mapred.JobClient: Map output bytes=942 08/01/05 15:47:14 INFO mapred.JobClient: Combine input records=34 08/01/05 15:47:14 INFO mapred.JobClient: Combine output records=33 08/01/05 15:47:14 INFO mapred.JobClient: Reduce input groups=33 08/01/05 15:47:14 INFO mapred.JobClient: Reduce input records=33 08/01/05 15:47:14 INFO mapred.JobClient: Reduce output records=33 08/01/05 15:47:14 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 08/01/05 15:47:14 INFO mapred.FileInputFormat: Total input paths to process : 1 08/01/05 15:47:14 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/input/mapred-default.xml:0+180 08/01/05 15:47:14 INFO mapred.JobClient: Running job: job_local_1 08/01/05 15:47:14 INFO mapred.MapTask: numReduceTasks: 1 08/01/05 15:47:14 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/grep-temp-346467784/part-00000:0+1279 08/01/05 15:47:14 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/grep-temp-346467784/part-00000:0+1279 08/01/05 15:47:14 INFO mapred.LocalJobRunner: reduce > reduce 08/01/05 15:47:14 INFO mapred.TaskRunner: Saved output of task 'reduce_h016y4' to file:/home/telzur/downloads/hadoop-0.14.4/output 08/01/05 15:47:14 INFO mapred.LocalJobRunner: reduce > reduce 08/01/05 15:47:14 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/input/hadoop-default.xml:0+27489 08/01/05 15:47:14 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/input/hadoop-site.xml:0+178 08/01/05 15:47:15 INFO mapred.LocalJobRunner: reduce > reduce 08/01/05 15:47:15 INFO mapred.JobClient: Job complete: job_local_1 08/01/05 15:47:15 INFO mapred.JobClient: Counters: 7 08/01/05 15:47:15 INFO mapred.JobClient: Map-Reduce Framework 08/01/05 15:47:15 INFO mapred.JobClient: Map input records=33 08/01/05 15:47:15 INFO mapred.JobClient: Map output records=33 08/01/05 15:47:15 INFO mapred.JobClient: Map input bytes=1193 08/01/05 15:47:15 INFO mapred.JobClient: Map output bytes=929 08/01/05 15:47:15 INFO mapred.JobClient: Reduce input groups=4 08/01/05 15:47:15 INFO mapred.JobClient: Reduce input records=66 08/01/05 15:47:15 INFO mapred.JobClient: Reduce output records=66 08/01/05 15:47:15 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/grep-temp-346467784/part-00000:0+1279 gtz2:/home/telzur/downloads/hadoop-0.14.4 # cat output/* 2 dfs. 1 dfs.block.size 1 dfs.blockreport.interval 1 dfs.client.block.write.retries 1 dfs.client.buffer.dir 1 dfs.data.dir 1 dfs.datanode.bind 1 dfs.datanode.dns.interface 1 dfs.datanode.dns.nameserver 1 dfs.datanode.du.pct 1 dfs.datanode.du.reserved 1 dfs.datanode.port 1 dfs.default.chunk.view.size 1 dfs.df.interval 1 dfs.heartbeat.interval 1 dfs.hosts 1 dfs.hosts.exclude 1 dfs.impl 1 dfs.info.bind 1 dfs.info.port 1 dfs.name.dir 1 dfs.namenode.handler.count 1 dfs.namenode.logging.level 1 dfs.network.script 1 dfs.replication 1 dfs.replication.consider 1 dfs.replication.max 1 dfs.replication.min 1 dfs.replication.min. 1 dfs.safemode.extension 1 dfs.safemode.threshold.pct 1 dfs.secondary.info.bind 1 dfs.secondary.info.port gtz2:/home/telzur/downloads/hadoop-0.14.4 #


Next step: Pseudo-Distributed Operation

Format a new distributed-filesystem:
gtz2:/home/telzur/downloads/hadoop-0.14.4 # bin/hadoop namenode -format
08/01/05 17:06:57 INFO dfs.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = gtz2/127.0.0.1
STARTUP_MSG: args = [-format]
************************************************************/
08/01/05 17:06:58 INFO dfs.Storage: Storage directory /tmp/hadoop-root/dfs/name has been successfully formatted.
08/01/05 17:06:58 INFO dfs.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at gtz2/127.0.0.1
************************************************************/
gtz2:/home/telzur/downloads/hadoop-0.14.4 #

gtz2:/home/telzur/downloads/hadoop-0.14.4 # bin/start-all.sh
starting namenode, logging to /home/telzur/downloads/hadoop-0.14.4/bin/../logs/hadoop-telzur-namenode-gtz2.out
localhost: starting datanode, logging to /home/telzur/downloads/hadoop-0.14.4/bin/../logs/hadoop-root-datanode-gtz2.out
localhost: starting secondarynamenode, logging to /home/telzur/downloads/hadoop-0.14.4/bin/../logs/hadoop-root-secondarynamenode-gtz2.out
starting jobtracker, logging to /home/telzur/downloads/hadoop-0.14.4/bin/../logs/hadoop-telzur-jobtracker-gtz2.out
localhost: starting tasktracker, logging to /home/telzur/downloads/hadoop-0.14.4/bin/../logs/hadoop-root-tasktracker-gtz2.out
gtz2:/home/telzur/downloads/hadoop-0.14.4 #


Browsing the web interface:






Run the examples:

gtz2:/home/telzur/downloads/hadoop-0.14.4 # bin/hadoop dfs -put conf input gtz2:/home/telzur/downloads/hadoop-0.14.4 # bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+' 08/01/05 17:21:39 INFO mapred.FileInputFormat: Total input paths to process : 10 08/01/05 17:21:40 INFO mapred.JobClient: Running job: job_200801051712_0001 08/01/05 17:21:41 INFO mapred.JobClient: map 0% reduce 0% 08/01/05 17:21:51 INFO mapred.JobClient: map 18% reduce 0% 08/01/05 17:21:52 INFO mapred.JobClient: map 36% reduce 0% 08/01/05 17:21:53 INFO mapred.JobClient: map 54% reduce 0% 08/01/05 17:21:54 INFO mapred.JobClient: map 63% reduce 0% 08/01/05 17:21:55 INFO mapred.JobClient: map 72% reduce 0% 08/01/05 17:21:56 INFO mapred.JobClient: map 90% reduce 0% 08/01/05 17:21:57 INFO mapred.JobClient: map 100% reduce 0% 08/01/05 17:22:06 INFO mapred.JobClient: map 100% reduce 27% 08/01/05 17:22:07 INFO mapred.JobClient: map 100% reduce 100% 08/01/05 17:22:08 INFO mapred.JobClient: Job complete: job_200801051712_0001 08/01/05 17:22:08 INFO mapred.JobClient: Counters: 12 08/01/05 17:22:08 INFO mapred.JobClient: Job Counters 08/01/05 17:22:08 INFO mapred.JobClient: Launched map tasks=11 08/01/05 17:22:08 INFO mapred.JobClient: Launched reduce tasks=1 08/01/05 17:22:08 INFO mapred.JobClient: Data-local map tasks=11 08/01/05 17:22:08 INFO mapred.JobClient: Map-Reduce Framework 08/01/05 17:22:08 INFO mapred.JobClient: Map input records=1153 08/01/05 17:22:08 INFO mapred.JobClient: Map output records=43 08/01/05 17:22:08 INFO mapred.JobClient: Map input bytes=34316 08/01/05 17:22:08 INFO mapred.JobClient: Map output bytes=1118 08/01/05 17:22:08 INFO mapred.JobClient: Combine input records=43 08/01/05 17:22:08 INFO mapred.JobClient: Combine output records=39 08/01/05 17:22:08 INFO mapred.JobClient: Reduce input groups=38 08/01/05 17:22:08 INFO mapred.JobClient: Reduce input records=39 08/01/05 17:22:08 INFO mapred.JobClient: Reduce output records=38 08/01/05 17:22:08 INFO mapred.FileInputFormat: Total input paths to process : 1 08/01/05 17:22:09 INFO mapred.JobClient: Running job: job_200801051712_0002 08/01/05 17:22:10 INFO mapred.JobClient: map 0% reduce 0% 08/01/05 17:22:18 INFO mapred.JobClient: map 100% reduce 0% 08/01/05 17:22:24 INFO mapred.JobClient: map 100% reduce 100% 08/01/05 17:22:25 INFO mapred.JobClient: Job complete: job_200801051712_0002 08/01/05 17:22:25 INFO mapred.JobClient: Counters: 10 08/01/05 17:22:25 INFO mapred.JobClient: Job Counters 08/01/05 17:22:25 INFO mapred.JobClient: Launched map tasks=1 08/01/05 17:22:25 INFO mapred.JobClient: Launched reduce tasks=1 08/01/05 17:22:25 INFO mapred.JobClient: Data-local map tasks=1 08/01/05 17:22:25 INFO mapred.JobClient: Map-Reduce Framework 08/01/05 17:22:25 INFO mapred.JobClient: Map input records=38 08/01/05 17:22:25 INFO mapred.JobClient: Map output records=38 08/01/05 17:22:25 INFO mapred.JobClient: Map input bytes=1330 08/01/05 17:22:25 INFO mapred.JobClient: Map output bytes=1026 08/01/05 17:22:25 INFO mapred.JobClient: Reduce input groups=3 08/01/05 17:22:25 INFO mapred.JobClient: Reduce input records=38 08/01/05 17:22:25 INFO mapred.JobClient: Reduce output records=38

Examine the output files:
gtz2:/home/telzur/downloads/hadoop-0.14.4 # bin/hadoop dfs -get output output gtz2:/home/telzur/downloads/hadoop-0.14.4 # cat output/* cat: output/output: Is a directory 2 dfs. 1 dfs.block.size 1 dfs.blockreport.interval 1 dfs.client.block.write.retries 1 dfs.client.buffer.dir 1 dfs.data.dir 1 dfs.datanode.bind 1 dfs.datanode.dns.interface 1 dfs.datanode.dns.nameserver 1 dfs.datanode.du.pct 1 dfs.datanode.du.reserved 1 dfs.datanode.port 1 dfs.default.chunk.view.size 1 dfs.df.interval 1 dfs.heartbeat.interval 1 dfs.hosts 1 dfs.hosts.exclude 1 dfs.impl 1 dfs.info.bind 1 dfs.info.port 1 dfs.name.dir 1 dfs.namenode.handler.count 1 dfs.namenode.logging.level 1 dfs.network.script 1 dfs.replication 1 dfs.replication.consider 1 dfs.replication.max 1 dfs.replication.min 1 dfs.replication.min. 1 dfs.safemode.extension 1 dfs.safemode.threshold.pct 1 dfs.secondary.info.bind 1 dfs.secondary.info.port gtz2:/home/telzur/downloads/hadoop-0.14.4 #

Re-check the web interfaces after the job ended:







Finally, stop the daemons when we are done:
gtz2:/home/telzur/downloads/hadoop-0.14.4 # bin/stop-all.sh stopping jobtracker localhost: stopping tasktracker stopping namenode localhost: stopping datanode localhost: stopping secondarynamenod

and that concludes the Hadoop Quitestart tutorial

Comments

Post a Comment

Popular posts from this blog

Cellular Automata - Part II, Using Condor

-
My post on Cellular Automata from January 12 was not put there by mistake. I want to use it as a starting point for a couple of exercises in my Parallel Processing course. In that post I gave a few drawings that differ only by the generating rule number. Today, I am going to show how using the Condor High-Throughput Computing system allows to handle in a very simple way large volume of computations. I used this simple Condor submit file : universe = vanilla executable = nks.py Error = err.$(Process) Output= out.$(Process) Log = log.$(Process) Arguments = $(Process) Queue 256 And was able to compute the whole set of 256 rules (jobs) with the same effort of computing a single rule. I submitted the task to my Personal Condor on my laptop and was not disappointed; After a while I got all the outputs happily waiting for post processing.
Read more

[mpich-discuss] Errors while testing MPICH2 on Vista

-
[mpich-discuss] Errors while testing MPICH2 on Vista Guy Tel-Zur gtelzur at gmail.com Sat Oct 24 14:39:26 CDT 2009 Previous message: [mpich-discuss] MPI-3 Fortran feedback Next message: [mpich-discuss] MPI function to test a machine alive or not. Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] I am trying to test a fresh installation of MPICH2 (version 1.2) on Vista32 and get the following errors: c:\temp>"c:\Program Files\MPICH2\bin\mpiexec.exe" -n 4 cpi.exe Please specify an authentication passphrase for smpd: <*I type here the passphrase, e.g. behappy*> Error while connecting to host, No connection could be made because the target machine actively refused it. (10061) Connect on sock (host=user-PC, port=8676) failed, exhaused all end points Unable to connect to 'user-PC:8676', sock error: Error = -1 I will appreciate any comment. --Gu
Read more

The End of Grid Computing?

-
Image
In the year 2003 the MIT Technology review ranked "Grid Computing" among the 10 Emerging Technologies That Will Change the World [1]. We are now four years later and something is not going well with "Grid Computing". An indication that there is a problem can easily be seen by looking at the "Google Trends" plot for the term "Grid Computing": (click on the image to get the current trend). This finding can be compared with another buzz word, "Virtualization", which is older than "Grid Computing" and yet is gaining more and more momentum: There is however one exception. The Academic Grid is still having lot's of glory thanks to the huge heavily funded European (EGEE) and other US projects. When LHC data will start to be taken at CERN it will reach it's top importance. But, it seems that for other scientific projects Grid Computing is not going to be such a success. It will remain as "Nice to have" but will never
Read more