手机版
你好,游客 登录 注册 搜索
背景:
阅读新闻

Hadoop完全分布式模式安装步骤

[日期:2014-02-12] 来源:Linux社区  作者:lichangzai [字体: ]

Hadoop模式介绍

单机模式:安装简单,几乎不用作任何配置,但仅限于调试用途

伪分布模式:在单节点上同时启动namenodedatanodejobtrackertasktrackersecondary namenode5个进程,模拟分布式运行的各个节点

完全分布式模式:正常的Hadoop集群,由多个各司其职的节点构成

 

安装环境

操作平台:vmware2

操作系统:Oracle linux 5.6

软件版本:hadoop-0.22.0,jdk-6u18

集群架构:3 nodemaster node(gc),slave node(rac1,rac2)

 

安装步骤

1. 下载Hadoopjdk

http://mirror.bit.edu.cn/apache/hadoop/common/

如:hadoop-0.22.0

 

 

2. 配置hosts文件

所有的节点(gc,rac1,rac2)都修改/etc/hosts,使彼此之间都能把主机名解析为ip

[root@gc ~]$ cat /etc/hosts

# Do not remove the following line, or various programs

# that require network functionality will fail.

127.0.0.1 localhost.localdomain localhost

::1 localhost6.localdomain6 localhost6

192.168.2.101 rac1.localdomain rac1

192.168.2.102 rac2.localdomain rac2

192.168.2.100 gc.localdomain gc

 

 

3. 建立hadoop运行账号

在所有的节点创建hadoop运行账号

[root@gc ~]#groupadd hadoop

[root@gc ~]#useradd -g hadoop grid--注意此处一定要指定分组,不然可能会不能建立互信

[root@gc ~]# id grid

uid=501(grid) gid=54326(hadoop) groups=54326(hadoop)

[root@gc ~]#passwd grid

Changing password for user grid.

New UNIX password:

BAD PASSWORD: it is too short

Retype new UNIX password:

passwd: all authentication tokens updated successfully.

 

4. 配置ssh免密码连入

注意要以hadoop用户登录,在hadoop用户的主目录下进行操作。

每个节点做下面相同的操作

[hadoop@gc ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):

Created directory '/home/hadoop/.ssh'.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/hadoop/.ssh/id_rsa.

Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.

The key fingerprint is:

54:80:fd:77:6b:87:97:ce:0f:32:34:43:d1:d2:c2:0d hadoop@gc.localdomain

[hadoop@gc ~]$ cd .ssh

[hadoop@gc .ssh]$ ls

id_rsa id_rsa.pub

 

把各个节点的authorized_keys的内容互相拷贝加入到对方的此文件中,然后就可以免密码彼此ssh连入。

在其中一节点(gc)节点就可完成操作

[hadoop@gc .ssh]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

[hadoop@gc .ssh]$ssh rac1 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

The authenticity of host 'rac1 (192.168.2.101)' can't be established.

RSA key fingerprint is 19:48:e0:0a:37:e1:2a:d5:ba:c8:7e:1b:37:c6:2f:0e.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'rac1,192.168.2.101' (RSA) to the list of known hosts.

hadoop@rac1's password:

[hadoop@gc .ssh]$ssh rac2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

The authenticity of host 'rac2 (192.168.2.102)' can't be established.

RSA key fingerprint is 19:48:e0:0a:37:e1:2a:d5:ba:c8:7e:1b:37:c6:2f:0e.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'rac2,192.168.2.102' (RSA) to the list of known hosts.

hadoop@rac2's password:

[hadoop@gc .ssh]$scp ~/.ssh/authorized_keys rac1:~/.ssh/authorized_keys

hadoop@rac1's password:

authorized_keys 100% 1213 1.2KB/s 00:00

[hadoop@gc .ssh]$scp ~/.ssh/authorized_keys rac2:~/.ssh/authorized_keys

hadoop@rac2's password:

authorized_keys 100% 1213 1.2KB/s 00:00

[hadoop@gc .ssh]$ ll

总计 16

-rw-rw-r-- 1 hadoop hadoop 1213 10-30 09:18 authorized_keys

-rw------- 1 hadoop hadoop 1675 10-30 09:05 id_rsa

-rw-r--r-- 1 hadoop hadoop 403 10-30 09:05 id_rsa.pub

 

--分别测试连接

[grid@gc .ssh]$ ssh rac1 date

2012 11 18日星期日 01:35:39 CST

[grid@gc .ssh]$ ssh rac2 date

2012 10 30日星期二 09:52:46 CST

 

--可以看到这步和配置oracle RAC中使用 SSH建立用户等效性步骤是一样的。

 

 

5. 解压hadoop安装包

--可先一某节点解压配置文件

[grid@gc ~]$ ll

总计 43580

-rw-r--r-- 1 grid hadoop 44575568 2012-11-19 hadoop-0.20.2.tar.gz

[grid@gc ~]$ tar xzvf /home/grid/hadoop-0.20.2.tar.gz

[grid@gc ~]$ ll

总计 43584

drwxr-xr-x 12 grid hadoop 4096 2010-02-19 hadoop-0.20.2

-rw-r--r-- 1 grid hadoop 44575568 2012-11-19 hadoop-0.20.2.tar.gz

 

--在各节点安装jdk

[root@gc ~]# ./jdk-6u18-linux-x64-rpm.bin

 

相关阅读

Ubuntu 13.04上搭建Hadoop环境 http://www.linuxidc.com/Linux/2013-06/86106.htm

Ubuntu 12.10 +Hadoop 1.2.1版本集群配置 http://www.linuxidc.com/Linux/2013-09/90600.htm

Ubuntu上搭建Hadoop环境(单机模式+伪分布模式) http://www.linuxidc.com/Linux/2013-01/77681.htm

Ubuntu下Hadoop环境的配置 http://www.linuxidc.com/Linux/2012-11/74539.htm

单机版搭建Hadoop环境图文教程详解 http://www.linuxidc.com/Linux/2012-02/53927.htm

搭建Hadoop环境(在Winodws环境下用虚拟机虚拟两个Ubuntu系统进行搭建) http://www.linuxidc.com/Linux/2011-12/48894.htm

linux
相关资讯       Hadoop安装 
本文评论   查看全部评论 (0)
表情: 表情 姓名: 字数

       

评论声明
  • 尊重网上道德,遵守中华人民共和国的各项有关法律法规
  • 承担一切因您的行为而直接或间接导致的民事或刑事法律责任
  • 本站管理人员有权保留或删除其管辖留言中的任意内容
  • 本站有权在网站内转载或引用您的评论
  • 参与本评论即表明您已经阅读并接受上述条款