手机版
你好,游客 登录 注册 搜索
背景:
阅读新闻

Hadoop添加datanode或者tasktracker节点

[日期:2013-09-14] 来源:Linux社区  作者:wangxiaoyu [字体: ]

1 规划新添加的Hadoop节点的角色和配置 

1.1 角色

slave: 即datanode节点或者tasktracker节点
1.2 配置
主机名:hadoop03
IP: 192.168.88.173

hadoop用户:xiaoyu 

2. 部署新的节点 

2.1 安装操作系统
2.3 关闭不必要的服务
建议只保留以下服务:abrt-ccpp abrt-oops autofs crond haldaemon lvm2-monitor mdmonitor messagebus netfs network nfslock ntpd portreserve rsyslog sshd udev-post
2.4 网络配置
2.4.1 网卡地址
示例配置文件如下,请根据实际所处的网络配置。
# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
IPADDR=192.168.88.173
NETMASK=255.255.255.0
GATEWAY=192.168.88.2
DNS1=192.168.88.2
IPV6INIT=no
USERCTL=no
2.4.2 修改主机名
# sudo vim /etc/sysconfig/network
修改HOSTNAME值为主机名
HOSTNAME=hadoop03
# hostname hadoop03
2.4.3 配置密钥对
[xiaoyu@hadoop03 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/xiaoyu/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/xiaoyu/.ssh/id_rsa.
Your public key has been saved in /home/xiaoyu/.ssh/id_rsa.pub.
The key fingerprint is:
45:41:8d:17:3b:0c:20:e0:5d:3f:38:ed:1f:e6:b9:7a xiaoyu@hadoop03
The key's randomart image is:
+--[ RSA 2048]----+
| ... oo=+.. |
| . . o =.oo. |
| . . o =.+ |
| + . . |
| S . o |
| + o |
| + |
| E. |
| .o. |
+-----------------+
[xiaoyu@hadoop03 ~]$
可以把这个操作写成一行命令,甚至写到脚本中:
$ expect -c "spawn ssh-keygen ; set timeout 5; expect \":\"; send \"\r\n\"; set timeout 3; expect \":\"; send \"\r\n\";set timeout 3; expect \":\"; send \"\r\n\"; expect eof;"
 
2.4.4 配置ssh免密码认证
看到过许多大牛,都是远程scp把公钥拷贝来拷贝去的。其实openssh客户端应用程序包已经提供了专门的命令来处理这个操作。
[xiaoyu@hadoop03 ~]$ ssh-copy-id -i .ssh/id_rsa.pub 192.168.88.171
The authenticity of host '192.168.88.171 (192.168.88.171)' can't be established.
RSA key fingerprint is a8:24:3f:34:86:f3:46:67:c0:a6:b0:42:86:a2:f2:c9.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.88.171' (RSA) to the list of known hosts.
Address 192.168.88.171 maps to localhost, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
xiaoyu@192.168.88.171's password:
Now try logging into the machine, with "ssh '192.168.88.171'", and check in:
.ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.

[xiaoyu@hadoop03 ~]$

相关阅读:

Hadoop datanode重新加载失败 无法启动解决步骤 http://www.linuxidc.com/Linux/2013-07/87953.htm

Hadoop中的datanode起不起来 http://www.linuxidc.com/Linux/2013-05/83863.htm

Hadoop中datanode与NameNode之间的心跳机制 http://www.linuxidc.com/Linux/2013-03/81014.htm

HJadoop分布式环境部署之namenode或datanode启动失败 http://www.linuxidc.com/Linux/2012-12/75669.htm

Hadoop 1.0.0的分布式配置(一个namenode 两个个datanode) http://www.linuxidc.com/Linux/2012-12/76535.htm

 

更多详情见请继续阅读下一页的精彩内容http://www.linuxidc.com/Linux/2013-09/90096p2.htm

linux
本文评论   查看全部评论 (0)
表情: 表情 姓名: 字数

       

评论声明
  • 尊重网上道德,遵守中华人民共和国的各项有关法律法规
  • 承担一切因您的行为而直接或间接导致的民事或刑事法律责任
  • 本站管理人员有权保留或删除其管辖留言中的任意内容
  • 本站有权在网站内转载或引用您的评论
  • 参与本评论即表明您已经阅读并接受上述条款