手机版
你好,游客 登录 注册 搜索
背景:
阅读新闻

Hadoop集群安装&Hbase实验环境搭建

[日期:2013-04-28] 来源:Linux社区  作者:chessloveyou [字体: ]

10.MapReduce测试:
 
hduser@master:/usr/local/Hadoop$ mkdir /tmp/test
 hduser@master:/usr/local/hadoop$ cd /tmp/test
 hduser@master:/tmp/test$ rz
 rz waiting to receive.
 开始 zmodem 传输。  按 Ctrl+C 取消。
  100%    615 KB  615 KB/s 00:00:01      0 Errors
  100%    502 KB  502 KB/s 00:00:01      0 Errors
  100%    813 KB  813 KB/s 00:00:01      0 Errors
 
hduser@master:/tmp/test$ /usr/local/hadoop/bin/start-all.sh
 starting namenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hduser-namenode-master.out
 slave2: starting datanode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hduser-datanode-slave2.out
 slave1: starting datanode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hduser-datanode-slave1.out
 master: starting datanode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hduser-datanode-master.out
 localhost: starting secondarynamenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hduser-secondarynamenode-master.out
 starting jobtracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hduser-jobtracker-master.out
 slave1: starting tasktracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hduser-tasktracker-slave1.out
 slave2: starting tasktracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hduser-tasktracker-slave2.out
 master: starting tasktracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hduser-tasktracker-master.out
 

 

 


hduser@master:~$ /usr/local/hadoop/bin/hadoop dfs -copyFromLocal /tmp/test /user/hduser/test
 hduser@master:~$ /usr/local/hadoop/bin/hadoop dfs -ls /user/hduser
 Found 1 items
 drwxr-xr-x  - hduser supergroup          0 2013-04-09 08:06 /user/hduser/test
 hduser@master:~$ /usr/local/hadoop/bin/hadoop dfs -ls /user/hduser/test
 Found 3 items
 -rw-r--r--  2 hduser supergroup    630010 2013-04-09 08:06 /user/hduser/test/1.epub
 -rw-r--r--  2 hduser supergroup    514824 2013-04-09 08:06 /user/hduser/test/2.epub
 -rw-r--r--  2 hduser supergroup    832882 2013-04-09 08:06 /user/hduser/test/3.epub
 

hduser@master:~$ cd /usr/local/hadoop/
 hduser@master:/usr/local/hadoop$ bin/hadoop jar hadoop-examples-1.0.4.jar wordcount /user/hduser/test /user/hduser/test-output
 13/04/09 08:18:45 INFO input.FileInputFormat: Total input paths to process : 3
 13/04/09 08:18:45 INFO util.NativeCodeLoader: Loaded the native-hadoop library
 13/04/09 08:18:45 WARN snappy.LoadSnappy: Snappy native library not loaded
 13/04/09 08:18:45 INFO mapred.JobClient: Running job: job_201304090758_0001
 13/04/09 08:18:46 INFO mapred.JobClient:  map 0% reduce 0%
 13/04/09 08:19:10 INFO mapred.JobClient:  map 66% reduce 0%
 13/04/09 08:19:19 INFO mapred.JobClient:  map 100% reduce 0%
 13/04/09 08:19:31 INFO mapred.JobClient:  map 100% reduce 100%
 13/04/09 08:19:36 INFO mapred.JobClient: Job complete: job_201304090758_0001
 13/04/09 08:19:37 INFO mapred.JobClient: Counters: 29
 13/04/09 08:19:37 INFO mapred.JobClient:  Job Counters
 13/04/09 08:19:37 INFO mapred.JobClient:    Launched reduce tasks=1
 13/04/09 08:19:37 INFO mapred.JobClient:    SLOTS_MILLIS_MAPS=42658
 13/04/09 08:19:37 INFO mapred.JobClient:    Total time spent by all reduces waiting after reserving slots (ms)=0
 13/04/09 08:19:37 INFO mapred.JobClient:    Total time spent by all maps waiting after reserving slots (ms)=0
 13/04/09 08:19:37 INFO mapred.JobClient:    Launched map tasks=3
 13/04/09 08:19:37 INFO mapred.JobClient:    Data-local map tasks=3
 13/04/09 08:19:37 INFO mapred.JobClient:    SLOTS_MILLIS_REDUCES=20867
 13/04/09 08:19:37 INFO mapred.JobClient:  File Output Format Counters
 13/04/09 08:19:37 INFO mapred.JobClient:    Bytes Written=3216032
 13/04/09 08:19:37 INFO mapred.JobClient:  FileSystemCounters
 13/04/09 08:19:37 INFO mapred.JobClient:    FILE_BYTES_READ=3421949
 13/04/09 08:19:37 INFO mapred.JobClient:    HDFS_BYTES_READ=1978040
 13/04/09 08:19:37 INFO mapred.JobClient:    FILE_BYTES_WRITTEN=6930267
 13/04/09 08:19:37 INFO mapred.JobClient:    HDFS_BYTES_WRITTEN=3216032
 13/04/09 08:19:37 INFO mapred.JobClient:  File Input Format Counters
 13/04/09 08:19:37 INFO mapred.JobClient:    Bytes Read=1977716
 13/04/09 08:19:37 INFO mapred.JobClient:  Map-Reduce Framework
 13/04/09 08:19:37 INFO mapred.JobClient:    Map output materialized bytes=3421961
 13/04/09 08:19:37 INFO mapred.JobClient:    Map input records=14841
 13/04/09 08:19:37 INFO mapred.JobClient:    Reduce shuffle bytes=2449921
 13/04/09 08:19:37 INFO mapred.JobClient:    Spilled Records=64440
 13/04/09 08:19:37 INFO mapred.JobClient:    Map output bytes=3685555
 13/04/09 08:19:37 INFO mapred.JobClient:    CPU time spent (ms)=10830
 13/04/09 08:19:37 INFO mapred.JobClient:    Total committed heap usage (bytes)=496644096
 13/04/09 08:19:37 INFO mapred.JobClient:    Combine input records=36177
 13/04/09 08:19:37 INFO mapred.JobClient:    SPLIT_RAW_BYTES=324
 13/04/09 08:19:37 INFO mapred.JobClient:    Reduce input records=32220
 13/04/09 08:19:37 INFO mapred.JobClient:    Reduce input groups=31501
 13/04/09 08:19:37 INFO mapred.JobClient:    Combine output records=32220
 13/04/09 08:19:37 INFO mapred.JobClient:    Physical memory (bytes) snapshot=614944768
 13/04/09 08:19:37 INFO mapred.JobClient:    Reduce output records=31501
 13/04/09 08:19:37 INFO mapred.JobClient:    Virtual memory (bytes) snapshot=3845349376
 13/04/09 08:19:37 INFO mapred.JobClient:    Map output records=36177

hduser@master:/usr/local/hadoop$ mkdir /tmp/test-output
 hduser@master:/usr/local/hadoop$ bin/hadoop dfs -getmerge /user/hduser/test-output /tmp/test-output
 13/04/09 08:22:48 INFO util.NativeCodeLoader: Loaded the native-hadoop library
 

$head /tmp/test-output/test-output
 
查看内容~
 
接下来是HBase的安装:http://www.linuxidc.com/Linux/2013-04/83560p5.htm

linux
相关资讯       Hadoop集群  Hbase  Hadoop安装  Hadoop HBase 
本文评论   查看全部评论 (0)
表情: 表情 姓名: 字数

       

评论声明
  • 尊重网上道德,遵守中华人民共和国的各项有关法律法规
  • 承担一切因您的行为而直接或间接导致的民事或刑事法律责任
  • 本站管理人员有权保留或删除其管辖留言中的任意内容
  • 本站有权在网站内转载或引用您的评论
  • 参与本评论即表明您已经阅读并接受上述条款