手机版
你好,游客 登录 注册 搜索
背景:
阅读新闻

使用Cloudera Manager安装Hadoop

[日期:2016-06-23] 来源:Linux社区  作者:Linux [字体: ]

Hadoop是由好多不同的服务(比如HDFS,Hive,HBase,Spark等等)构成的,这些服务之间还有些依赖关系,如果直接下Apache上的原始的包,需要下载多次,配置多次,显得比较麻烦。由此就产生了一些对Hadoop进行定制的公司,比如 ClouderaHortonworks ,还有 MapR . 这些公司都有自己的Hadoop发行版,Cloudera的发行版叫CDH。因为公司的系统用的就是CDH,这一段时间装了多次CDH5,做了一些实际操作,还看了些Cloudera的文档。

Cloudera使用Cloudera Manager,一个Web界面的Hadoop管理系统来进行Hadoop相关服务的安装,配置,和监控。

安装Cloudera Manager

官方指南在 这里 ,有几种方式,我这里说到的只是其中之一,不是最简单的,但是可以让你对Cloudera Manager有比较多的了解。

可以通过下面这张简单的图来大概了解下Cloudera Manager是如何去管理Hadoop服务的。

配置Cloudera Manager的YUM源

在/etc/yum.repos.d/下加一个文件CM.repo,内容如下.

 

[CM]
name=CM
baseurl=http://archive-primary.cloudera.com/cm5/RedHat/6/x86_64/cm/5.1.3/
gpgcheck=0

[ CM ]

name = CM

baseurl = http : //archive-primary.cloudera.com/cm5/redhat/6/x86_64/cm/5.1.3/

gpgcheck = 0

运行下面的命令查看Yum仓库配置成功与否,如果yum list能列出cloudera-manager-***那些包,就说明成功了。如果你是在实验室环境里安装,可能无法连接到外网,那么你可以在一台可以连接外网的机器上用reposync命令把外网的yum源clone到你本地,然后本地起个http服务给实验室的机器当yum服务器。也可以在实验室的CM.repo文件配置代理来连接外网,具体步骤Google一下就行了。

 

[root@bogon yum.repos.d]# yum clean all
Loaded plugins: fastestmirror, refresh-packagekit, security
Cleaning repos: CM base extras updates
Cleaning up Everything
Cleaning up list of fastest mirrors
[root@bogon yum.repos.d]# yum list | grep cloudera
cloudera-manager-daemons.x86_64        5.1.3-1.cm513.p0.155.el6          @CM    
cloudera-manager-server.x86_64         5.1.3-1.cm513.p0.155.el6          @CM    
cloudera-manager-agent.x86_64          5.1.3-1.cm513.p0.155.el6          CM     
cloudera-manager-server-db-2.x86_64    5.1.3-1.cm513.p0.155.el6          CM

[ root @ bogon yum . repos . d ] # yum clean all

Loaded plugins : fastestmirror , refresh - packagekit , security

Cleaning repos : CM base extras updates

Cleaning up Everything

Cleaning up list of fastest mirrors

[ root @ bogon yum . repos . d ] # yum list | grep cloudera

cloudera - manager - daemons . x86_64 5.1.3 - 1.cm513.p0.155.el6 @ CM

cloudera - manager - server . x86_64 5.1.3 - 1.cm513.p0.155.el6 @ CM

cloudera - manager - agent . x86_64 5.1.3 - 1.cm513.p0.155.el6 CM

cloudera - manager - server - db - 2.x86_64 5.1.3 - 1.cm513.p0.155.el6 CM

安装Cloudera Manager相关的包, 安装这个包时会自动把它依赖的cloudera-manager-daemons包也装上。

 

yum install cloudera-manager-server

yum install cloudera - manager - server

看看这两个包到底装些什么,可以看到server基本上就复制了一些配置文件.真正的管理网站是通过daemons装的,具体的文件都在/usr/share/cmf目录下,日志在/var/log/cloudera-scm-server目录下,而运行时的配置文件在/var/run/cloudera-scm-server

[root@bogon ~]# rpm -ql cloudera-manager-server
/etc/cloudera-scm-server
/etc/cloudera-scm-server/db.properties
/etc/cloudera-scm-server/log4j.properties
/etc/default/cloudera-scm-server
/etc/rc.d/init.d/cloudera-scm-server
/etc/security/limits.d/cloudera-scm.conf
/opt/cloudera/csd
/opt/cloudera/parcel-repo
/usr/sbin/cmf-server
/var/lib/cloudera-scm-server

[root@bogon ~]# rpm -ql cloudera-manager-daemons
...
/usr/share/cmf/yarn-fixtures/details_1.json
/usr/share/cmf/yarn-fixtures/details_2.json
/usr/share/cmf/yarn-fixtures/details_3.json
/usr/share/cmf/yarn-fixtures/details_4.json
/usr/share/cmf/yarn-fixtures/details_5.json
/usr/share/cmf/yarn-fixtures/details_6.json
/usr/share/cmf/yarn-fixtures/details_7.json
/usr/share/cmf/yarn-fixtures/details_8.json
/usr/share/cmf/yarn-fixtures/details_9.json
/var/log/cloudera-scm-server
/var/run/cloudera-scm-server

[ root @ bogon ~ ] # rpm -ql cloudera-manager-server

/ etc / cloudera - scm - server

/ etc / cloudera - scm - server / db . properties

/ etc / cloudera - scm - server / log4j . properties

/ etc / default / cloudera - scm - server

/ etc / rc . d / init . d / cloudera - scm - server

/ etc / security / limits . d / cloudera - scm . conf

/ opt / cloudera / csd

/ opt / cloudera / parcel - repo

/ usr / sbin / cmf - server

/ var / lib / cloudera - scm - server

[ root @ bogon ~ ] # rpm -ql cloudera-manager-daemons

. . .

/ usr / share / cmf / yarn - fixtures / details_1 . json

/ usr / share / cmf / yarn - fixtures / details_2 . json

/ usr / share / cmf / yarn - fixtures / details_3 . json

/ usr / share / cmf / yarn - fixtures / details_4 . json

/ usr / share / cmf / yarn - fixtures / details_5 . json

/ usr / share / cmf / yarn - fixtures / details_6 . json

/ usr / share / cmf / yarn - fixtures / details_7 . json

/ usr / share / cmf / yarn - fixtures / details_8 . json

/ usr / share / cmf / yarn - fixtures / details_9 . json

/ var / log / cloudera - scm - server

/ var / run / cloudera - scm - server

安装数据库和对应的driver,这是因为Cloudera Manager本身需要把配置保存数据库中,另外后面装Hive时也需要数据库来保存元数据。

 

yum install mysql-server
yum install mysql-connector-java.noarch

yum install mysql - server

yum install mysql - connector - java . noarch

接着配置一些数据库相关的参数,先在数据库添加数据库cmf,以及对应的用户cmf,还有密码。

 

mysql> create database cmf DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all on cmf.* TO 'cmf'@'localhost' IDENTIFIED BY '123456';
Query OK, 0 rows affected (0.00 sec)

mysql > create database cmf DEFAULT CHARACTER SET utf8 ;

Query OK , 1 row affected ( 0.00 sec )

mysql > grant all on cmf . * TO 'cmf' @ 'localhost' IDENTIFIED BY '123456' ;

Query OK , 0 rows affected ( 0.00 sec )

然后修改 /etc/cloudera-scm-server/db.properties文件,配置Cloudera manager要访问哪个数据库,对应的用户名,密码等。

 

# The database type
# Currently 'mysql', 'postgresql' and 'Oracle' are valid databases.
com.cloudera.cmf.db.type=mysql

# The database host
# If a non standard port is needed, use 'hostname:port'
com.cloudera.cmf.db.host=localhost

# The database name
com.cloudera.cmf.db.name=cmf

# The database user<a href="http://linmingren.me/blog/wp-content/uploads/2014/09/cloudera-welcome.png"><img src="http://linmingren.me/blog/wp-content/uploads/2014/09/cloudera-welcome-1024x375.png" alt="" title="cloudera welcome" width="584" height="213" class="alignnone size-large wp-image-479" /></a>
com.cloudera.cmf.db.user=cmf

# The database user's password
com.cloudera.cmf.db.password=123456

# The database type

# Currently 'mysql', 'postgresql' and 'oracle' are valid databases.

com . cloudera . cmf . db . type = mysql

# The database host

# If a non standard port is needed, use 'hostname:port'

com . cloudera . cmf . db . host = localhost

# The database name

com . cloudera . cmf . db . name = cmf

# The database user<a href="http://linmingren.me/blog/wp-content/uploads/2014/09/cloudera-welcome.png"><img src="http://linmingren.me/blog/wp-content/uploads/2014/09/cloudera-welcome-1024x375.png" alt="" title="cloudera welcome" width="584" height="213" class="alignnone size-large wp-image-479" /></a>

com . cloudera . cmf . db . user = cmf

# The database user's password

com . cloudera . cmf . db . password = 123456

现在就可以启动cloudera-scm-server服务了, 用netstat命令看下7180端口有没有启动(从服务启动到端口开启在我的机器上大概要半分钟),有的话就可以在浏览器中输入localhost:7180来访问Cloudera Manger的web界面了 (默认用户密码是admin/admin),没有的话请看/var/log/cloudera-scm-server/cloudera-scm-server.log中的异常是什么。

 

service cloudera-scm-server restart

service cloudera - scm - server restart

第一次登陆进去的首页是这

更多Hadoop相关信息见Hadoop 专题页面 http://www.linuxidc.com/topicnews.aspx?tid=13

本文永久更新链接地址http://www.linuxidc.com/Linux/2016-06/132598.htm

linux
本文评论   查看全部评论 (0)
表情: 表情 姓名: 字数

       

评论声明
  • 尊重网上道德,遵守中华人民共和国的各项有关法律法规
  • 承担一切因您的行为而直接或间接导致的民事或刑事法律责任
  • 本站管理人员有权保留或删除其管辖留言中的任意内容
  • 本站有权在网站内转载或引用您的评论
  • 参与本评论即表明您已经阅读并接受上述条款