这里通过在Linux系统下,搭建基于Hadoop + Eclipse plugin的开发环境,并在实际中使用。下面详述具体的配置和实践情况。
RHEL 5环境配置
(1)无密码验证配置
执行如下命令:
- [www.linuxidc.com @localhost .ssh]$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
- [www.linuxidc.com @localhost .ssh]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
- [www.linuxidc.com @localhost .ssh]$ ssh localhost
否则,如果出现需要输入密码进入,则需要检查一下你的.ssh目录的权限,包括.ssh目录下的authorized_keys和known_hosts是否具有读(r)的权限,如果问题出在这里,授权即可。
(2)环境变量配置
修改.bashrc文件,配置环境变量:
[www.linuxidc.com @localhost ~]$ vi .bashrc
- # .bashrc
- # Source global definitions
- if [ -f /etc/bashrc ]; then
- . /etc/bashrc
- fi
- # User specific aliases and functions
- export JAVA_HOME=/usr/java/jdk1.6.0_16
- export PATH=$JAVA_HOME/bin:$PATH
- export CLASSPATH=.:$JAVA_HOME/jre/lib/*.jar:$JAVA_HOME/jre/lib/*.jar
- export HADOOP_HOME=/home/shirdrn/eclipse/eclipse-3.5.2/hadoop/hadoop-0.20.0
- export PATH=$PATH:$HADOOP_HOME/bin
Hadoop准备
1、下载hadoop-0.20.0.tar.gz压缩包,解压缩到/home/shirdrn/eclipse/eclipse-3.5.2/hadoop/目录下面:
[www.linuxidc.com @localhost hadoop]$ tar -xzvf hadoop-0.20.0.tar.gz
2、配置Hadoop
(1)修改hadoop-0.20.0/conf/hadoop-env.sh,在该文件中增加如下三行:
- export JAVA_HOME=/usr/java/jdk1.6.0_16
- export HADOOP_HOME=/home/shirdrn/eclipse/eclipse-3.5.2/hadoop/hadoop-0.20.0
- export PATH=$PATH:$HADOOP_HOME/bin
(2)修改hadoop-0.20.0/conf/core-site.xml,内容如下所示:
- <?xml version="1.0"?>
- <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
- <!-- Put site-specific property overrides in this file. -->
- <configuration>
- <property>
- <name>fs.default.name</name>
- <value>hdfs://localhost:9000</value>
- </property>
- <property>
- <name>hadoop.tmp.dir</name>
- <value>/tmp/hadoop/hadoop-${user.name}</value>
- </property>
- </configuration>
(3)修改hadoop-0.20.0/conf/hdfs-site.xml,内容如下所示:
- <?xml version="1.0"?>
- <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
- <!-- Put site-specific property overrides in this file. -->
- <configuration>
- <property>
- <name>dfs.replication</name>
- <value>1</value>
- </property>
- </configuration>
(4)修改hadoop-0.20.0/conf/mapred-site.xml,内容如下所示:
- <?xml version="1.0"?>
- <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
- <!-- Put site-specific property overrides in this file. -->
- <configuration>
- <property>
- <name>mapred.job.tracker</name>
- <value>localhost:9001</value>
- </property>
- </configuration>
Eclipse plugin配置
这个配置比较容易一点:
解压缩eclipse-SDK-3.5.2-linux-gtk.tar.gz到/home/shirdrn/eclipse/目录下面,然后将hadoop-0.20.0/contrib/eclipse-plugin/hadoop-0.20.0-eclipse-plugin.jar插件拷贝到/home/shirdrn/eclipse/eclipse-3.5.2/eclipse/plugins/目录下面:
[www.linuxidc.com @localhost ~]$ cp /home/shirdrn/eclipse/eclipse-3.5.2/hadoop/hadoop-0.20.0/contrib/eclipse-plugin/hadoop-0.20.0-eclipse-plugin.jar /home/shirdrn/eclipse/eclipse-3.5.2/eclipse/plugins/
然后,就可以在RHEL 5下启动Eclipse开发工具了。