首页 > 服务器 > Linux > 正文

centos7.1 安装Cloudera Manager 5.7
   来源:网络收集   评论:0 点击:

安装centos 7.1:参考Win7+Ubuntu+Centos7.1三系统安装centos7.1之卸载opjdk,安装oracle jdk安装mysql http://blog.sina.com.cn/s/blog_4...
   安装centos 7.1:参考
    Win7+Ubuntu+Centos7.1三系统安装
    centos7.1之卸载opjdk,安装oracle jdk
    安装mysql http://blog.sina.com.cn/s/blog_446be0430102wr3c.html
    下载Cloudera Manager 5.7:http://archive.cloudera.com/cm5/cm/5中的cloudera-manager-centos7-cm5.7.0_x86_64.tar.gz
     解压后 mv cm-5.7.0/ /opt/

配置免密码登录:

ssh-keygen -t rsa -P "" -f ~/.ssh/id_rsa


cd ~/.ssh #获取宿主机的公钥,方便宿主机登录 scp xx@11.111:~/.ssh/id_rsa.pub authorized_keys

cat id_rsa.pub >> authorized_keys
chmod 600 authorized_keys


为Cloudera Manager 5建立数据库

首先需要去MySql的官网下载JDBC驱动,http://dev.mysql.com/downloads/connector/j/,解压后,找到mysql-connector-java-5.1.33-bin.jar,放到/opt/cm-5.1.3/share/cmf/lib/中。

cp /home/work/soft/mysql-connector-java-5.1.39/mysql-connector-java-5.1.39-bin.jar  /opt/cm-5.7.0/share/cmf/lib/

在主节点初始化CM5的数据库:
/opt/cm-5.7.0/share/cmf/schema/scm_prepare_database.sh mysql  cm -hlocalhost -uroot -p*xx   --scm-host localhost scm scm scm   -v
JAVA_HOME=/usr/java/jdk1.8.0_91
Verifying that we can write to /opt/cm-5.7.0/etc/cloudera-scm-server
Database type: mysql
Database user: root
Executing: /usr/java/jdk1.8.0_91/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cm-5.7.0/share/cmf/schema/../libetc/cloudera-scm-agent/config.ini的server_host为主节点的主机名
zjl:我这里是loalhost:
server_host=localhost

# Port that the CM server is listening on.
server_port=7182


因为Cloudera Manage 5.7.0不会自动创建run目录下面的
cloudera-scm-agent目录,所以需要手动创建,这个真是一个坑啊
mkdir -p   /opt/cm-5.7.0/run/cloudera-scm-agent

将/opt/cm-5.*同步到所有节点

在namenode启动server和agent( 主节点也是要做到集群中的, 所以主节点也要开启Agent ):(注意必须以管理员权限启动)
/opt/cm-5.7.0/etc/init.d/cloudera-scm-server start
/opt/cm-5.7.0/etc/init.d/cloudera-scm-agent start

在所有datanode启动agent
/opt/cm-5.7.0/etc/init.d/cloudera-scm-agent start


Cloudera Manager Server和Agent都启动以后,就可以进行CDH5的安装配置了。
这时可以通过浏览器访问主节点的7180端口测试一下了(由于CM Server的启动需要花点时间,这里可能要等待一会才能访问),默认的用户名和密码均为admin
http://localhost:7180/cmf/login

有的需要“选择主机”, 选择即可:
cp /home/work/soft/mysql-connector-java-5.1.39/mysql-connector-java-5.1.39-bin.jar  /usr/share/java/mysql-connector-java.jar


主机安装过程报错::
安装失败。 无法接收 Agent 发出的检测信号。

    请确保主机的名称已正确配置。
    请确保端口 7182 可在 Cloudera Manager Server 上访问(检查防火墙规则)。
    请确保正在添加的主机上的端口 9000 和 9001 空闲。
    检查正在添加的主机上 /var/log/cloudera-scm-agent/ 中的代理日志(某些日志可在安装详细信息中找到)。
详细log:

Failed to connect to newly launched supervisor.

无法接收 Agent 发出的检测信号。 (当前步骤) 

无法接收到agent检测信号有几种可能:
1、Python文件不匹配;参考http://www.cnblogs.com/lion.net/archive/2014/09/02/3950619.html中_io的设置
2、日志文件不存在,在config.ini中把log_file放开
3、/etc/hosts/中主机和ip配置问题
4、防火墙是否关闭,ubuntu是ufw disable
5、端口配置,config.ini中端口是否配置的为7182
6、集群时间是否同步,安装ntp同步时间
7、ssh私钥的问题-----我现在正在查这个问题呢,前边都配完了,但是仍然无法检测到信号,我没有使用私钥,不知道是不是跟这个有关系 
>>ProtocolError:
>>[09/Jul/2016 11:13:11 +0000] 18231 MainThread agent INFO Trying to connect to newly launched supervisor (Attempt 5)
>>[09/Jul/2016 11:13:11 +0000] 18231 MainThread agent ERROR Failed! trying again in 1 second(s)
>>Traceback (most recent call last):
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.0-py2.7.egg/cmf/agent.py", line 2133, in connect_to_new_supervisor
>> self.get_supervisor_process_info()
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.0-py2.7.egg/cmf/agent.py", line 2155, in get_supervisor_process_info
>> self.identifier = self.supervisor_client.supervisor.getIdentification()
>> File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__
>> return self.__send(self.__name, args)
>> File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request
>> verbose=self.__verbose
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/xmlrpc.py", line 470, in request
>> '' )
>>ProtocolError:
>>[09/Jul/2016 11:13:11 +0000] 18231 MainThread agent ERROR Failed to connect to newly launched supervisor. Agent will exit
>>[09/Jul/2016 11:13:11 +0000] 18231 MainThread agent INFO Stopping agent...
>>[09/Jul/2016 11:13:11 +0000] 18231 MainThread agent INFO No extant cgroups; unmounting any cgroup roots
>>[09/Jul/2016 11:13:11 +0000] 18231 MainThread agent INFO Cleaning up daemon
>>[09/Jul/2016 11:13:11 +0000] 18231 Dummy-1 agent INFO Stopping agent...
>>[09/Jul/2016 11:13:11 +0000] 18231 Dummy-1 agent INFO No extant cgroups; unmounting any cgroup roots
>>[09/Jul/2016 11:13:11 +0000] 18231 Dummy-1 agent INFO Cleaning up daemon


[root@localhost cm-5.7.0]# ps aux | grep super
root     18949  0.0  0.1 223624 12996 ?        Ss   11:16   0:00 /opt/cm-5.7.0/lib64/cmf/agent/build/env/bin/python /opt/cm-5.7.0/lib64/cmf/agent/build/env/bin/supervisord
root     23390  0.0  0.0 112656   972 pts/0    S+   11:37   0:00 grep --color=auto super
[root@localhost cm-5.7.0]# kill   18949
/opt/cm-5.7.0/etc/init.d/cloudera-scm-agent stop
/opt/cm-5.7.0/etc/init.d/cloudera-scm-server stop

CentOS 7.0默认使用的是firewall作为防火墙,使用iptables必须重新设置一下

1、直接关闭防火墙

systemctl stop firewalld.service #停止firewall

systemctl disable firewalld.service #禁止firewall开机启动

zjl:我重启后再安装后没有出现该问题,具体原因还不太清楚。

作业一直是running

错误描述:使用Hadoop提交简单的作业–wordcount后,一直处于running状态。在作业界面查看,显示作业状态为“accepted”,“unassigned”。

问题原因:内存分配不合理。

解决办法:

zjl: http://blog.csdn.net/flyfish778/article/details/44755591
The solution therefore, is to increase the above mentioned properties in the cluster’s yarn-site.xml, and restart the YARN service. Cloudera Manager users would typically find the same configuration property names searchable under CM -> YARN -> Configuration.
For instance, the yarn.nodemanager.resource.memory-mb and yarn.nodemanager.resource.cpu-vcores are very small values, such that the MR2 job’s Application Master itself consumes all of it, leaving no room for the actual map tasks to get allocated for running after it.
改为8G
yarn.nodemanager.resource.memory-mb

Personally I like leaving these values for my 1-node clusters at their default values of 8 GB Memory and 8 CPU Cores。Cloudera docs also have a good guide on configuring resources on the cluster here.
zjl:修改上面两个参数为8G和8核后, job可以运行起来:


启动此 NFS Gateway     
启动角色
Supervisor 返回 FATAL。请参阅角色日志文件、Stderr 或 Stdout。
No portmap or rpcbind service is running on this host. Please start portmap or rpcbind service before attempting to start the NFS Gateway role on this host. 2016 07 09 16:46:07 CST JAVA_HOME=/usr/java/jdk1.8.0_91 using /usr/java/jdk1.8.0_91 as JAVA_HOME using 5 as CDH_VERSION using /run/cloudera-scm-agent/process/82-hdfs-NFSGATEWAY as CONF_DIR using as SECURE_USER using as SECURE_GROUP unlimited Cannot connect to port 111. No portmap or rpcbind service is running on this host. Please start portmap or rpcbind service before attempting to start the NFS Gateway role on this host.
解决方法:
http://stackoverflow.com/questions/23981795/cannot-start-nfs-gateway-hadoop
You need to have rpcbind running:
sudo service rpcbind start

http://desert3.iteye.com/blog/1675522
    yum  install  nfs-utils  portmap
    chkconfig rpcbind on #chkconfig:更新和查询各运行级别的系统服务
    chkconfig nfs on
    service rpcbind start
    service nfs start 
Http协议中的各种长度限制总结
重启IIS服务的几种方法
Windows平台的Apache各个版本区别
54款开源服务器软件
端口大全


手机页面二维码(扫一扫转给需要的人)

相关热词搜索:centos7.1 Manager 5.7

上一篇:如何在CentOS linux 6.x 安装Dell OpenManage管理软件
下一篇:最后一页

分享到: 收藏