手机版
你好,游客 登录 注册
背景:
阅读新闻

Linux下Hadoop集群安装详细步骤

[日期:2011-08-27] 来源:Linux社区  作者:liuyuanshijie [字体: ]

1. 环境的需求(这里虚拟机下CentOS6的安装就不说了)
          Centos6 + Hadoop-0.21.0.tar

2. 服务器的配置(我在这里均为2个CPU,2G内存,100硬盘)
   在这里,服务器IP最好是固定的,也就是说,能相互之间用ping命令ping通的IP

   建议在公司开发,因为公司有这个条件,我在这里配置了三台datanode,分别为 Datanode1,Datanode2,Datanode3(这些都是创建虚机的主机名)

      服务器名称                 IP地址(自己定)

   Namenode                            192.168.16.1

   Datanode1                            192.168.16.2

   Datanode2                            192.168.16.3

   Datanode3                            192.168.16.4

   2.1 修改服务器IP的方法如下:
                            vi  /etc/sysconfig/network-scripts/ifcfg-eth0

                           

                            DEVICE="eth0"

                            #这里是你网卡的物理地址,通常检测到的网卡你就不用输入了

                            #打开后该项已经存在,无需修改

                            HWADDR="00:0C:29:95:1D:A5"

                            BOOTPROTO="static"

                            ONBOOT="yes"

                            #这里是IP地址,不能重复,从253降序排列,被占用了的不能再次使用.

                            IPADDR=172.16.101.245

                            NETMASK=255.255.255.0

                            NETWORK=172.16.101.0

                            BROADCAST=172.16.101.255

                            GATEWAY=172.16.101.254

 

                            退出保存后,执行如下命令,使设置的网关马上生效:

                            shell>> ifdown eth0

                            shell>> ifup eth0

                            shell>> /etc/init.d/network restart

3. 安装JDK6(我用的是jdk-6u26-linux-x64-rpm.bin)
        3.1. 创建安装目录 mkdir /usr/java/
        3.2. 把jdk-6u26-linux-x64-rpm.bin移动到/usr/java/下然后执行
                          ./jdk-6u26-linux-x64-rpm.bin

                          运行中会要求输入,顺序yes和按回车就行.

                          执行后会看到一个文件夹名为:jdk1.6.0_26

        3.3. 设置环境变量
                            /etc/profile增加如下内容

                            #config java

                            JAVA_HOME=/usr/java/jdk1.6.0_26

                            CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar

                            PATH=$JAVA_HOME/bin:$HOME/bin:$PATH

                            export PATH JAVA_HOME CLASSPATH

        3.4. 让设置生效: source /etc/profile
4. 安装ssh服务器和客户端
        a. yum search ssh
        b. 找到要安装的server(这里拿openssh-server.x86_64)
        c. 安装server: yum install openssh-server.x86_64
        d. 安装client(这里拿openssh-clients.x86_64)
        e. 安装client: yum install openssh-clients.x86_64
5. 设置ssh进行Namenode和Datanode之间无密码访问
        a. 用 ssh-key-gen 在本地主机上创建公钥和密钥
                            [root@www.linuxidc.com ~]# ssh-keygen -t  rsa

                            Enter file in which to save the key (/home/jsmith/.ssh/id_rsa):[Enter key]

                            Enter passphrase (empty for no passphrase): [Press enter key]

                            Enter same passphrase again: [Pess enter key]

                            Your identification has been saved in /home/jsmith/.ssh/id_rsa.

                            Your public key has been saved in /home/jsmith/.ssh/id_rsa.pub.

                            The key fingerprint is: 33:b3:fe:af:95:95:18:11:31:d5:de:96:2f:f2:35:f9

                            root@www.linuxidc.com

        b. 用 ssh-copy-id 把公钥复制到远程主机上
                            [root@www.linuxidc.com ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub  root@Datanode1

                            root@Datanode1's password:

                            Now try logging into the machine, with ―ssh ?root@Datanode1‘‖, and check in:

                            .ssh/authorized_keys to make sure we haven‘t added extra keys that you weren‘t expecting.

                            [注: ssh-copy-id 把密钥追加到远程主机的 .ssh/authorized_key 上.]

   c. 直接登录远程主机
                            [root@www.linuxidc.com ~]# ssh Datanode1

                            Last login: Sun Nov 16 17:22:33 2008 from 192.168.1.2

                            [注: SSH 不会询问密码.]

                            [root@Datanode1 ~]

                            [注: 你现在已经登录到了远程主机上]

   d. 注意:在这里,执行都在Namenode上面,而且Namenode也需要对自己进行无密码操作即
      [root@www.linuxidc.com ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub  root@www.linuxidc.com操作,

      其他的,按照a-c重复操作Datanode2和Datanode3就行了

      一定要能无密码访问,否则不能集群Hadoop一定失败.

6. 安装Hadoop(这里,每一台服务器的JDK和Hadoop安装路径都相同)
        a. 创建安装目录 mkdir /usr/local/hadoop/
        b. 解压安装文件hadoop-0.21.0.tar放入到安装目录
tar -zxvf hadoop-0.21.0.tar

        c. 设置环境变量
                            /etc/profile增加如下内容

                            #config hadoop

                            export HADOOP_HOME=/usr/local/hadoop/

                            export PATH=$HADOOP_HOME/bin:$PATH

                            #hadoop的日志文件路径的配置

                            export HADOOP_LOG_DIR=${HADOOP_HOME}/logs

               让设置生效: source /etc/profile

        d. 设置主从配置
                     Namenode中/etc/hosts的配置如下:

                     192.168.16.1        Namenode

                     192.168.16.2        Datanode1

                     192.168.16.3        Datanode2

                     192.168.16.4        Datanode3

  

                     Namenode中/usr/local/hadoop/conf/masters的配置如下:

                     Namenode

                    

                     Namenode中/usr/local/hadoop/conf/slaves的配置如下:

                     Datanode1

                     Datanode2

                     Datanode3

                    

                    

                     Datanode1中/etc/hosts的配置如下:(/usr/local/hadoop/conf/中的masters和slaves的配置跟Namenode一样)

                     192.168.16.1        Namenode

                     192.168.16.2        Datanode1

                    

                     Datanode2中/etc/hosts的配置如下:(/usr/local/hadoop/conf/中的masters和slaves的配置跟Namenode一样)

                     192.168.16.1        Namenode

                     192.168.16.3        Datanode2

                    

                     Datanode3中/etc/hosts的配置如下:(/usr/local/hadoop/conf/中的masters和slaves的配置跟Namenode一样)

                     192.168.16.1        Namenode

                     192.168.16.4        Datanode3

   e. 修改配置文件/usr/local/hadoop/conf/ hadoop-env.sh
      把JAVA_HOME该为安装jdk的路径

                            # The java implementation to use.  Required.

                            export JAVA_HOME=/usr/java/jdk1.6.0_26/

   f. 修改配置文件 core-site.xml内容如下:
                            <configuration>

                               <property>

                                 <name>fs.default.name</name>

                                 <value>hdfs://Namenode:9000/</value>

                               </property>

                              <property>

                                <name>hadoop.tmp.dir</name>

                                <value>/usr/local/hadoop/tmp/</value>

                              </property>

                            </configuration>

   g. 修改配置文件 hdfs-site.xml内容如下:
                            <configuration>

                <property>

                  <name>dfs.replication</name>

                  #设置备份文件数

                  <value>1</value>

                </property>

                            </configuration>

   h. 修改配置文件 mapred-site.xml内容如下:
                            <configuration>

                 <property>

              <name>mapred.job.tracker</name>

              #一般jobtracker和namenode设置到同一台机器上,但是同样可以集群

              <value>Namenode:9001</value>

                 </property>

                            </configuration>

   i. 注意:上面讲的配置文件全部是在Namenode中配置的,只要把这三个配置文件拷贝复制到其他的Datanode上就行了
   j. 初始化Hadoop: cd /usr/local/hadoop/
               ./bin/hadoop namenode -format

               出现类似如下的信息:但是不能出现ERORR字段.

                            .2.jar:/usr/local/hadoop/bin/../lib/paranamer-generator-2.2.jar:/usr/local/hadoop/bin/../lib/qdox-1.10.1.jar:/usr/local/hadoop/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/local/hadoop/bin/../lib/slf4j-api-1.5.11.jar:/usr/local/hadoop/bin/../lib/slf4j-log4j12-1.5.11.jar:/usr/local/hadoop/bin/../lib/xmlenc-0.52.jar:/usr/local/hadoop/bin/../lib/jsp-2.1/*.jar:/usr/local/hadoop/hdfs/bin/../conf:/usr/local/hadoop/hdfs/bin/../hadoop-hdfs-*.jar:/usr/local/hadoop/hdfs/bin/../lib/*.jar:/usr/local/hadoop/bin/../mapred/conf:/usr/local/hadoop/bin/../mapred/hadoop-mapred-*.jar:/usr/local/hadoop/bin/../mapred/lib/*.jar:/usr/local/hadoop/hdfs/bin/../hadoop-hdfs-*.jar:/usr/local/hadoop/hdfs/bin/../lib/*.jar

                            STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21 -r 985326; compiled by 'tomwhite' on Tue Aug 17 01:02:28 EDT 2010

                            ************************************************************/

                            Re-format filesystem in /usr/local/hadoop/tmp/dfs/name ? (Y or N) y

                            Format aborted in /usr/local/hadoop/tmp/dfs/name

                            11/06/16 13:04:17 INFO namenode.NameNode: SHUTDOWN_MSG:

                            /************************************************************

                            SHUTDOWN_MSG: Shutting down NameNode at namenode/172.16.101.251

                            ************************************************************/

   k. 启动Hadoop ./bin/start-all.sh
                            This script is Deprecated. Instead use start-dfs.sh and start-mapred.sh

                            starting namenode, logging to /usr/local/hadoop//logs/hadoop-root-namenode-namenode.out

                            datanode1: starting datanode, logging to /usr/local/hadoop/bin/../logs/hadoop-root-datanode-datanode1.out

                            datanode2: starting datanode, logging to /usr/local/hadoop/bin/../logs/hadoop-root-datanode-datanode2.out

                            datanode3: starting datanode, logging to /usr/local/hadoop/bin/../logs/hadoop-root-datanode-datanode3.out

                            namenode: starting secondarynamenode, logging to /usr/local/hadoop/bin/../logs/hadoop-root-secondarynamenode-namenode.out

                            starting jobtracker, logging to /usr/local/hadoop//logs/hadoop-root-jobtracker-namenode.out

                            datanode3: starting tasktracker, logging to /usr/local/hadoop/bin/../logs/hadoop-root-tasktracker-datanode3.out

                            datanode2: starting tasktracker, logging to /usr/local/hadoop/bin/../logs/hadoop-root-tasktracker-datanode2.out

                            datanode1: starting tasktracker, logging to /usr/local/hadoop/bin/../logs/hadoop-root-tasktracker-datanode1.out

      启动后用命令JPS查看结果如下:

                            [root@www.linuxidc.com hadoop]# jps

                            1806 Jps

                            1368 NameNode

                            1694 JobTracker

                            1587 SecondaryNameNode

                            然后到Datanode1/2/3上去查看,执行JPS,结果如下:

                            [root@datanode2 hadoop]# jps

                            1440 Jps

                            1382 TaskTracker

                            1303 DataNode

                            [root@datanode2 hadoop]# jps

                            1382 TaskTracker

                            1303 DataNode

                            1452 Jps

                            说明你成功集群安装了Hadoop

7. HDFS操作
          运行bin/目录的hadoop命令,可以查看Haoop所有支持的操作及其用法,这里以几个简单的操作为例。

建立目录
          [root@www.linuxidc.com hadoop]# ./bin/hadoop  dfs  -mkdir  testdir

          在HDFS中建立一个名为testdir的目录

复制文件
          [root@www.linuxidc.com hadoop]# ./bin/hadoop  dfs  -put  /home/dbrg/large.zip  testfile.zip

          把本地文件large.zip拷贝到HDFS的根目录/user/dbrg/下,文件名为testfile.zip

查看现有文件
          [root@www.linuxidc.com hadoop]# ./bin/hadoop  dfs  -ls

linux
本文评论   查看全部评论 (0)
表情: 表情 姓名: 字数

       

评论声明
  • 尊重网上道德,遵守中华人民共和国的各项有关法律法规
  • 承担一切因您的行为而直接或间接导致的民事或刑事法律责任
  • 本站管理人员有权保留或删除其管辖留言中的任意内容
  • 本站有权在网站内转载或引用您的评论
  • 参与本评论即表明您已经阅读并接受上述条款