First trials with Hadoop

-

I followed the Hadoop Quickstart guide and the whole process is described below.

This post can be used as a reference for other people installing Hadoop.

My system is OpenSuse 10.3 and Java version is 1.6.0_03.

After downloading and installing the package I did the Standalone operation test:

$ mkdir input
$ cp conf/*.xml input
$ bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'
$ cat output/*

Here is the output (line feeds may be corrupted, sorry for that):

gtz2:/home/telzur/downloads/hadoop-0.14.4 # bin/hadoop jar hadoop-0.14.4-examples.jar grep input output 'dfs[a-z.]+' 08/01/05 15:47:13 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 08/01/05 15:47:13 INFO mapred.FileInputFormat: Total input paths to process : 3 08/01/05 15:47:13 INFO mapred.JobClient: Running job: job_local_1 08/01/05 15:47:13 INFO mapred.MapTask: numReduceTasks: 1 08/01/05 15:47:13 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/input/mapred-default.xml:0+180 08/01/05 15:47:13 INFO mapred.MapTask: numReduceTasks: 1 08/01/05 15:47:13 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/input/hadoop-default.xml:0+27489 08/01/05 15:47:13 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/input/hadoop-default.xml:0+27489 08/01/05 15:47:13 INFO mapred.MapTask: numReduceTasks: 1 08/01/05 15:47:13 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/input/hadoop-site.xml:0+178 08/01/05 15:47:13 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/input/hadoop-site.xml:0+178 08/01/05 15:47:14 INFO mapred.LocalJobRunner: reduce > reduce 08/01/05 15:47:14 INFO mapred.TaskRunner: Saved output of task 'reduce_3r6jh8' to file:/home/telzur/downloads/hadoop-0.14.4/grep-temp-346467784 08/01/05 15:47:14 INFO mapred.JobClient: Job complete: job_local_1 08/01/05 15:47:14 INFO mapred.JobClient: Counters: 9 08/01/05 15:47:14 INFO mapred.JobClient: Map-Reduce Framework 08/01/05 15:47:14 INFO mapred.JobClient: Map input records=940 08/01/05 15:47:14 INFO mapred.JobClient: Map output records=34 08/01/05 15:47:14 INFO mapred.JobClient: Map input bytes=27847 08/01/05 15:47:14 INFO mapred.JobClient: Map output bytes=942 08/01/05 15:47:14 INFO mapred.JobClient: Combine input records=34 08/01/05 15:47:14 INFO mapred.JobClient: Combine output records=33 08/01/05 15:47:14 INFO mapred.JobClient: Reduce input groups=33 08/01/05 15:47:14 INFO mapred.JobClient: Reduce input records=33 08/01/05 15:47:14 INFO mapred.JobClient: Reduce output records=33 08/01/05 15:47:14 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 08/01/05 15:47:14 INFO mapred.FileInputFormat: Total input paths to process : 1 08/01/05 15:47:14 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/input/mapred-default.xml:0+180 08/01/05 15:47:14 INFO mapred.JobClient: Running job: job_local_1 08/01/05 15:47:14 INFO mapred.MapTask: numReduceTasks: 1 08/01/05 15:47:14 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/grep-temp-346467784/part-00000:0+1279 08/01/05 15:47:14 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/grep-temp-346467784/part-00000:0+1279 08/01/05 15:47:14 INFO mapred.LocalJobRunner: reduce > reduce 08/01/05 15:47:14 INFO mapred.TaskRunner: Saved output of task 'reduce_h016y4' to file:/home/telzur/downloads/hadoop-0.14.4/output 08/01/05 15:47:14 INFO mapred.LocalJobRunner: reduce > reduce 08/01/05 15:47:14 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/input/hadoop-default.xml:0+27489 08/01/05 15:47:14 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/input/hadoop-site.xml:0+178 08/01/05 15:47:15 INFO mapred.LocalJobRunner: reduce > reduce 08/01/05 15:47:15 INFO mapred.JobClient: Job complete: job_local_1 08/01/05 15:47:15 INFO mapred.JobClient: Counters: 7 08/01/05 15:47:15 INFO mapred.JobClient: Map-Reduce Framework 08/01/05 15:47:15 INFO mapred.JobClient: Map input records=33 08/01/05 15:47:15 INFO mapred.JobClient: Map output records=33 08/01/05 15:47:15 INFO mapred.JobClient: Map input bytes=1193 08/01/05 15:47:15 INFO mapred.JobClient: Map output bytes=929 08/01/05 15:47:15 INFO mapred.JobClient: Reduce input groups=4 08/01/05 15:47:15 INFO mapred.JobClient: Reduce input records=66 08/01/05 15:47:15 INFO mapred.JobClient: Reduce output records=66 08/01/05 15:47:15 INFO mapred.LocalJobRunner: file:/home/telzur/downloads/hadoop-0.14.4/grep-temp-346467784/part-00000:0+1279 gtz2:/home/telzur/downloads/hadoop-0.14.4 # cat output/* 2 dfs. 1 dfs.block.size 1 dfs.blockreport.interval 1 dfs.client.block.write.retries 1 dfs.client.buffer.dir 1 dfs.data.dir 1 dfs.datanode.bind 1 dfs.datanode.dns.interface 1 dfs.datanode.dns.nameserver 1 dfs.datanode.du.pct 1 dfs.datanode.du.reserved 1 dfs.datanode.port 1 dfs.default.chunk.view.size 1 dfs.df.interval 1 dfs.heartbeat.interval 1 dfs.hosts 1 dfs.hosts.exclude 1 dfs.impl 1 dfs.info.bind 1 dfs.info.port 1 dfs.name.dir 1 dfs.namenode.handler.count 1 dfs.namenode.logging.level 1 dfs.network.script 1 dfs.replication 1 dfs.replication.consider 1 dfs.replication.max 1 dfs.replication.min 1 dfs.replication.min. 1 dfs.safemode.extension 1 dfs.safemode.threshold.pct 1 dfs.secondary.info.bind 1 dfs.secondary.info.port gtz2:/home/telzur/downloads/hadoop-0.14.4 #


Next step: Pseudo-Distributed Operation

Format a new distributed-filesystem:
gtz2:/home/telzur/downloads/hadoop-0.14.4 # bin/hadoop namenode -format
08/01/05 17:06:57 INFO dfs.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = gtz2/127.0.0.1
STARTUP_MSG: args = [-format]
************************************************************/
08/01/05 17:06:58 INFO dfs.Storage: Storage directory /tmp/hadoop-root/dfs/name has been successfully formatted.
08/01/05 17:06:58 INFO dfs.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at gtz2/127.0.0.1
************************************************************/
gtz2:/home/telzur/downloads/hadoop-0.14.4 #

gtz2:/home/telzur/downloads/hadoop-0.14.4 # bin/start-all.sh
starting namenode, logging to /home/telzur/downloads/hadoop-0.14.4/bin/../logs/hadoop-telzur-namenode-gtz2.out
localhost: starting datanode, logging to /home/telzur/downloads/hadoop-0.14.4/bin/../logs/hadoop-root-datanode-gtz2.out
localhost: starting secondarynamenode, logging to /home/telzur/downloads/hadoop-0.14.4/bin/../logs/hadoop-root-secondarynamenode-gtz2.out
starting jobtracker, logging to /home/telzur/downloads/hadoop-0.14.4/bin/../logs/hadoop-telzur-jobtracker-gtz2.out
localhost: starting tasktracker, logging to /home/telzur/downloads/hadoop-0.14.4/bin/../logs/hadoop-root-tasktracker-gtz2.out
gtz2:/home/telzur/downloads/hadoop-0.14.4 #


Browsing the web interface:






Run the examples:

gtz2:/home/telzur/downloads/hadoop-0.14.4 # bin/hadoop dfs -put conf input gtz2:/home/telzur/downloads/hadoop-0.14.4 # bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+' 08/01/05 17:21:39 INFO mapred.FileInputFormat: Total input paths to process : 10 08/01/05 17:21:40 INFO mapred.JobClient: Running job: job_200801051712_0001 08/01/05 17:21:41 INFO mapred.JobClient: map 0% reduce 0% 08/01/05 17:21:51 INFO mapred.JobClient: map 18% reduce 0% 08/01/05 17:21:52 INFO mapred.JobClient: map 36% reduce 0% 08/01/05 17:21:53 INFO mapred.JobClient: map 54% reduce 0% 08/01/05 17:21:54 INFO mapred.JobClient: map 63% reduce 0% 08/01/05 17:21:55 INFO mapred.JobClient: map 72% reduce 0% 08/01/05 17:21:56 INFO mapred.JobClient: map 90% reduce 0% 08/01/05 17:21:57 INFO mapred.JobClient: map 100% reduce 0% 08/01/05 17:22:06 INFO mapred.JobClient: map 100% reduce 27% 08/01/05 17:22:07 INFO mapred.JobClient: map 100% reduce 100% 08/01/05 17:22:08 INFO mapred.JobClient: Job complete: job_200801051712_0001 08/01/05 17:22:08 INFO mapred.JobClient: Counters: 12 08/01/05 17:22:08 INFO mapred.JobClient: Job Counters 08/01/05 17:22:08 INFO mapred.JobClient: Launched map tasks=11 08/01/05 17:22:08 INFO mapred.JobClient: Launched reduce tasks=1 08/01/05 17:22:08 INFO mapred.JobClient: Data-local map tasks=11 08/01/05 17:22:08 INFO mapred.JobClient: Map-Reduce Framework 08/01/05 17:22:08 INFO mapred.JobClient: Map input records=1153 08/01/05 17:22:08 INFO mapred.JobClient: Map output records=43 08/01/05 17:22:08 INFO mapred.JobClient: Map input bytes=34316 08/01/05 17:22:08 INFO mapred.JobClient: Map output bytes=1118 08/01/05 17:22:08 INFO mapred.JobClient: Combine input records=43 08/01/05 17:22:08 INFO mapred.JobClient: Combine output records=39 08/01/05 17:22:08 INFO mapred.JobClient: Reduce input groups=38 08/01/05 17:22:08 INFO mapred.JobClient: Reduce input records=39 08/01/05 17:22:08 INFO mapred.JobClient: Reduce output records=38 08/01/05 17:22:08 INFO mapred.FileInputFormat: Total input paths to process : 1 08/01/05 17:22:09 INFO mapred.JobClient: Running job: job_200801051712_0002 08/01/05 17:22:10 INFO mapred.JobClient: map 0% reduce 0% 08/01/05 17:22:18 INFO mapred.JobClient: map 100% reduce 0% 08/01/05 17:22:24 INFO mapred.JobClient: map 100% reduce 100% 08/01/05 17:22:25 INFO mapred.JobClient: Job complete: job_200801051712_0002 08/01/05 17:22:25 INFO mapred.JobClient: Counters: 10 08/01/05 17:22:25 INFO mapred.JobClient: Job Counters 08/01/05 17:22:25 INFO mapred.JobClient: Launched map tasks=1 08/01/05 17:22:25 INFO mapred.JobClient: Launched reduce tasks=1 08/01/05 17:22:25 INFO mapred.JobClient: Data-local map tasks=1 08/01/05 17:22:25 INFO mapred.JobClient: Map-Reduce Framework 08/01/05 17:22:25 INFO mapred.JobClient: Map input records=38 08/01/05 17:22:25 INFO mapred.JobClient: Map output records=38 08/01/05 17:22:25 INFO mapred.JobClient: Map input bytes=1330 08/01/05 17:22:25 INFO mapred.JobClient: Map output bytes=1026 08/01/05 17:22:25 INFO mapred.JobClient: Reduce input groups=3 08/01/05 17:22:25 INFO mapred.JobClient: Reduce input records=38 08/01/05 17:22:25 INFO mapred.JobClient: Reduce output records=38

Examine the output files:
gtz2:/home/telzur/downloads/hadoop-0.14.4 # bin/hadoop dfs -get output output gtz2:/home/telzur/downloads/hadoop-0.14.4 # cat output/* cat: output/output: Is a directory 2 dfs. 1 dfs.block.size 1 dfs.blockreport.interval 1 dfs.client.block.write.retries 1 dfs.client.buffer.dir 1 dfs.data.dir 1 dfs.datanode.bind 1 dfs.datanode.dns.interface 1 dfs.datanode.dns.nameserver 1 dfs.datanode.du.pct 1 dfs.datanode.du.reserved 1 dfs.datanode.port 1 dfs.default.chunk.view.size 1 dfs.df.interval 1 dfs.heartbeat.interval 1 dfs.hosts 1 dfs.hosts.exclude 1 dfs.impl 1 dfs.info.bind 1 dfs.info.port 1 dfs.name.dir 1 dfs.namenode.handler.count 1 dfs.namenode.logging.level 1 dfs.network.script 1 dfs.replication 1 dfs.replication.consider 1 dfs.replication.max 1 dfs.replication.min 1 dfs.replication.min. 1 dfs.safemode.extension 1 dfs.safemode.threshold.pct 1 dfs.secondary.info.bind 1 dfs.secondary.info.port gtz2:/home/telzur/downloads/hadoop-0.14.4 #

Re-check the web interfaces after the job ended:







Finally, stop the daemons when we are done:
gtz2:/home/telzur/downloads/hadoop-0.14.4 # bin/stop-all.sh stopping jobtracker localhost: stopping tasktracker stopping namenode localhost: stopping datanode localhost: stopping secondarynamenod

and that concludes the Hadoop Quitestart tutorial

Comments

Post a Comment

Popular posts from this blog

The End of Grid Computing?

-
Image
In the year 2003 the MIT Technology review ranked "Grid Computing" among the 10 Emerging Technologies That Will Change the World [1]. We are now four years later and something is not going well with "Grid Computing". An indication that there is a problem can easily be seen by looking at the "Google Trends" plot for the term "Grid Computing": (click on the image to get the current trend). This finding can be compared with another buzz word, "Virtualization", which is older than "Grid Computing" and yet is gaining more and more momentum: There is however one exception. The Academic Grid is still having lot's of glory thanks to the huge heavily funded European (EGEE) and other US projects. When LHC data will start to be taken at CERN it will reach it's top importance. But, it seems that for other scientific projects Grid Computing is not going to be such a success. It will remain as "Nice to have" but will never...
Read more

Parallel Debugging with a Serial Debugger

-
Image
Parallel Debuggers are important tools when trying to eliminate bugs in parallel programs. Commercial parallel debuggers are expensive. In this blog post I will show you a free and open source alternative which is good enough for education purposes and for small codes. Suppose you want to debug the famous cpi.c code which computes an approximation to pi by a parallel numerical integration. This demo uses the free Data Display Debugger (ddd) which is based on gdb but has a nice GUI. Step 1, compile: mpicc -g -o cpi ./cpi.c Step 2, run (without a debugger): mpirun -np 2 ./cpi Step 3, run with the free Data Display Debugger (ddd) : mpirun -np 2 ddd ./cpi The last command will open two MPI processes (instances) of ddd each running cpi, see screen capture: Finally, I enclose here a short video which I hope is convincing: hi-res version:  https://youtu.be/N_J4NKJkuMs Happy (Free) MPI debugging!
Read more

[mpich-discuss] Errors while testing MPICH2 on Vista

-
[mpich-discuss] Errors while testing MPICH2 on Vista Guy Tel-Zur gtelzur at gmail.com Sat Oct 24 14:39:26 CDT 2009 Previous message: [mpich-discuss] MPI-3 Fortran feedback Next message: [mpich-discuss] MPI function to test a machine alive or not. Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] I am trying to test a fresh installation of MPICH2 (version 1.2) on Vista32 and get the following errors: c:\temp>"c:\Program Files\MPICH2\bin\mpiexec.exe" -n 4 cpi.exe Please specify an authentication passphrase for smpd: passphrase, e.g. behappy*> Error while connecting to host, No connection could be made because the target machine actively refused it. (10061) Connect on sock (host=user-PC, port=8676) failed, exhaused all end points Unable to connect to 'user-PC:8676', sock error: Error = -1 I will appreciate any comment. --Guy -------------- next...
Read more