你好,游客 登录 注册 搜索
背景:
阅读新闻

Oracle RAC Database 11.1.0.6监听故障案例

[日期:2014-11-26] 来源:Linux社区  作者:尛样儿 [字体: ]

接到电话,客户的一套核心Oracle RAC数据库连接不上,连接时报无监听程序,客户的Oracle RAC版本为11.1.0.6,平台为AIX 6.1.05,使用了IBM HACMP 5.5.0.8。

当我远程过去的时候,发现节点2已经没有任何oracle用户的进程,且concurrent的vg没有激活,HACMP的服务也offline。

另一个节点Oracle的实例是正常的,且有部分服务器进程依然在工作,但是本地监听器出现了故障,导致新的连接无法连接到实例,通过crs_stat -t看到两个实例的监听也都是OFFLINE状态。

在节点上并没有发现有LISTENER进程,且手动杀掉了所有的服务器进程,在oracle用户下启动监听时收到以下的报错:

$ lsnrctl start listener_cdfy740a

LSNRCTL for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production on 20-NOV-2014 20:09:09

Copyright (c) 1991, 2007, Oracle.  All rights reserved.

Starting /oracle/app/oracle/product/11.1.0/db_1/bin/tnslsnr: please wait...

TNSLSNR for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production
System parameter file is /oracle/app/oracle/product/11.1.0/db_1/network/admin/listener.ora
Log messages written to /oracle/app/oracle/diag/tnslsnr/cdfy740a/listener_cdfy740a/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.107.64.1)(PORT=1521)))
Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.107.64.1)(PORT=1521)(IP=FIRST)))
TNS-12542: TNS:address already in use
 TNS-12560: TNS:protocol adapter error
  TNS-00512: Address already in use
  IBM/AIX RISC System/6000 Error: 67: Address already in use

Listener failed to start. See the error message(s) above...


10.107.64.1是该节点的vip地址,下面是RAC环境的hosts配置:

10.107.64.1    vip1
10.107.64.2    vip2
10.107.64.3    cdfy740a
10.107.64.4    cdfy740b
172.201.201.1  prv1
172.201.201.2  prv2

手动停掉该节点的nodeapps服务:

cdfy740a@root[/oracle/app/11.1.0/crs/bin]./srvctl stop nodeapps -n cdfy740a

成功停止后,VIP在主机层面已经消失:

cdfy740a@root[/oracle/app/11.1.0/crs/bin]ifconfig -a | more
en0: flags=1e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
        inet 172.200.200.1 netmask 0xffffff00 broadcast 172.200.200.255
        tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
en1: flags=1e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
        inet 172.201.201.1 netmask 0xffffff00 broadcast 172.201.201.255
        tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
en4: flags=5e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>
        inet 10.107.64.3 netmask 0xffffff00 broadcast 10.107.64.255
        tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
lo0: flags=e08084b,c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,LARGESEND,CHAIN>
        inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255
        inet6 ::1%1/0
        tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1

再次启动节点nodeapps服务:

cdfy740a@root[/oracle/app/11.1.0/crs/bin]./srvctl start nodeapps -n cdfy740a
CRS-1006: No more members to consider
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:LSNRCTL for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production on 20-NOV-2014 20:13:07
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Copyright (c) 1991, 2007, Oracle.  All rights reserved.
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Starting /oracle/app/oracle/product/11.1.0/db_1/bin/tnslsnr: please wait...
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:TNSLSNR for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:System parameter file is /oracle/app/oracle/product/11.1.0/db_1/network/admin/listener.ora
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Log messages written to /oracle/app/oracle/diag/tnslsnr/cdfy740a/listener_cdfy740a/alert/log.xml
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.107.64.1)(PORT=1521)))
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.107.64.1)(PORT=1521)(IP=FIRST)))
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:TNS-12542: TNS:address already in use
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-12560: TNS:protocol adapter error
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:  TNS-00512: Address already in use
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:  IBM/AIX RISC System/6000 Error: 67: Address already in use
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Listener failed to start. See the error message(s) above...
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:LSNRCTL for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production on 20-NOV-2014 20:13:08
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Copyright (c) 1991, 2007, Oracle.  All rights reserved.
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=vip1)(PORT=1521)))
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:TNS-12541: TNS:no listener
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-12560: TNS:protocol adapter error
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:  TNS-00511: No listener
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:  IBM/AIX RISC System/6000 Error: 79: Connection refused
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.107.64.1)(PORT=1521)(IP=FIRST)))
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:TNS-12541: TNS:no listener
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-12560: TNS:protocol adapter error
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:  TNS-00511: No listener
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:  IBM/AIX RISC System/6000 Error: 79: Connection refused
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.107.64.3)(PORT=1521)(IP=FIRST)))
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:TNS-12541: TNS:no listener
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-12560: TNS:protocol adapter error
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:  TNS-00511: No listener
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:  IBM/AIX RISC System/6000 Error: 79: Connection refused
CRS-0215: Could not start resource 'ora.cdfy740a.LISTENER_CDFY740A.lsnr'.


之前使用lsnrctl status listener_cdfy740a查看监听器状态时也收到Connection refused的错误。

查看主机层面已经成功绑定了VIP地址:
cdfy740a@root[/oracle/app/11.1.0/crs/bin]ifconfig -a | more
en0: flags=1e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
        inet 172.200.200.1 netmask 0xffffff00 broadcast 172.200.200.255
        tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
en1: flags=1e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
        inet 172.201.201.1 netmask 0xffffff00 broadcast 172.201.201.255
        tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
en4: flags=5e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>
        inet 10.107.64.3 netmask 0xffffff00 broadcast 10.107.64.255
        inet 10.107.64.1 netmask 0xffffff00 broadcast 10.107.64.255
        tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
lo0: flags=e08084b,c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,LARGESEND,CHAIN>
        inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255
        inet6 ::1%1/0
        tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1

再次尝试手动启动本地监听器:

cdfy740a@root[/]su - oracle
$ lsnrctl start listener_cdfy740a

LSNRCTL for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production on 20-NOV-2014 20:18:37

Copyright (c) 1991, 2007, Oracle.  All rights reserved.

Starting /oracle/app/oracle/product/11.1.0/db_1/bin/tnslsnr: please wait...

TNSLSNR for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production
System parameter file is /oracle/app/oracle/product/11.1.0/db_1/network/admin/listener.ora
Log messages written to /oracle/app/oracle/diag/tnslsnr/cdfy740a/listener_cdfy740a/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.107.64.1)(PORT=1521)))
Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.107.64.1)(PORT=1521)(IP=FIRST)))
TNS-12542: TNS:address already in use
 TNS-12560: TNS:protocol adapter error
  TNS-00512: Address already in use
  IBM/AIX RISC System/6000 Error: 67: Address already in use

Listener failed to start. See the error message(s) above...

启动依然失败。

检查监听器配置文件:
$ cat listener.ora
# listener.ora.cdfy740a Network Configuration File: /oracle/app/oracle/product/11.1.0/db_1/network/admin/listener.ora.cdfy740a
# Generated by Oracle configuration tools.

LISTENER_CDFY740A =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = vip1)(PORT = 1521))
      (ADDRESS = (PROTOCOL = TCP)(HOST = 10.107.64.1)(PORT = 1521)(IP = FIRST))
      (ADDRESS = (PROTOCOL = TCP)(HOST = 10.107.64.3)(PORT = 1521)(IP = FIRST))
    )
  )

在监听配置文件中,vip1和10.107.64.1是两个重复的地址,手动将10.107.64.1所在行去掉之后,监听即可正常的启动。

之后恢复节点2的HACMP服务,Oracle RAC随即恢复正常。

另外,还发现客户的监听日志已经被填得很大,大概在1.6GB左右,过大的监听日志文件也会导致监听器��稳定,这里将两个节点的监听日志进行了重命名操作。

Oracle数据库监听非常慢,基本hang住故障处理 http://www.linuxidc.com/Linux/2014-08/104942.htm

Oracle监听之动态监听与静态监听特点 http://www.linuxidc.com/Linux/2013-09/89702.htm

Oracle 11g RAC 环境下单实例非缺省监听及端口配置 http://www.linuxidc.com/Linux/2013-08/88936.htm

Oracle 监听器日志配置与管理 http://www.linuxidc.com/Linux/2013-08/88935.htm

Oracle错误- ORA-12514:TNS:无监听程序 http://www.linuxidc.com/Linux/2013-06/85878.htm

ORA-12514 监听错误解决 http://www.linuxidc.com/Linux/2012-12/76049.htm

Oracle监听器出现的6种连接问题及其解决方法 http://www.linuxidc.com/Linux/2012-12/75568.htm

Oracle LISTENER 未监听到Oracle实例问题解决 http://www.linuxidc.com/Linux/2012-05/60910.htm

设置 Oracle 监听器密码(LISTENER) http://www.linuxidc.com/Linux/2011-07/39421.htm

更多Oracle相关信息见Oracle 专题页面 http://www.linuxidc.com/topicnews.aspx?tid=12

本文永久更新链接地址http://www.linuxidc.com/Linux/2014-11/109874.htm

linux
本文评论   查看全部评论 (0)
表情: 表情 姓名: 字数

       

评论声明
  • 尊重网上道德,遵守中华人民共和国的各项有关法律法规
  • 承担一切因您的行为而直接或间接导致的民事或刑事法律责任
  • 本站管理人员有权保留或删除其管辖留言中的任意内容
  • 本站有权在网站内转载或引用您的评论
  • 参与本评论即表明您已经阅读并接受上述条款