前言
我们在处理数据库的问题的时候,有时在数据库端看不到异常的等待,但是应用却报连接异常。
这种情况,有可能是在压力还没有传递到数据库,在网络这层已经发生问题了。如连接风暴,此时的数据库listener的进程已经忙不过来,将连接丢弃。(在主机层面可以看到tcpListeneDrop增加)。
我们如果去看普通的listener日志,看到的东西是这样的:
……24-FEB-2017 09:11:03 * (CONNECT_DATA=(SID=mynwdb)(CID=(PROGRAM=sqlplus@mynwdb)(HOST=mynwdb)(USER=appl))) * (ADDRESS=(PROTOCOL=tcp)(HOST=11.22.3.123)(PORT=65184)) * establish * mynwdb * 024-FEB-2017 09:11:06 * (CONNECT_DATA=(SID=mynwdb)(CID=(PROGRAM=STANDARD)(HOST=mynwdb)(USER=appl))) * (ADDRESS=(PROTOCOL=tcp)(HOST=11.22.3.123)(PORT=65186)) * establish * mynwdb * 024-FEB-2017 09:11:10 * (CONNECT_DATA=(SID=mynwdb)(CID=(PROGRAM=STANDARD)(HOST=mynwdb)(USER=appl))) * (ADDRESS=(PROTOCOL=tcp)(HOST=11.22.3.123)(PORT=65188)) * establish * mynwdb * 024-FEB-2017 09:11:10 * (CONNECT_DATA=(SID=mynwdb)(CID=(PROGRAM=STANDARD)(HOST=mynwdb)(USER=appl))) * (ADDRESS=(PROTOCOL=tcp)(HOST=11.22.3.123)(PORT=65189)) * establish * mynwdb * 0Fri Feb 24 09:11:14 201724-FEB-2017 09:11:14 * (CONNECT_DATA=(SID=mynwdb)(CID=(PROGRAM=INVMGR)(HOST=mynwdb)(USER=nusr))) * (ADDRESS=(PROTOCOL=tcp)(HOST=11.22.3.123)(PORT=65194)) * establish * mynwdb * 024-FEB-2017 09:11:14 * (CONNECT_DATA=(SID=mynwdb)(CID=(PROGRAM=INVMGR)(HOST=mynwdb)(USER=nusr))) * (ADDRESS=(PROTOCOL=tcp)(HOST=11.22.3.123)(PORT=65195)) * establish * mynwdb * 024-FEB-2017 09:11:15 * (CONNECT_DATA=(SID=mynwdb)(CID=(PROGRAM=INVMGR)(HOST=mynwdb)(USER=nusr))) * (ADDRESS=(PROTOCOL=tcp)(HOST=11.22.3.123)(PORT=65196)) * establish * mynwdb * 024-FEB-2017 09:11:15 * (CONNECT_DATA=(SID=mynwdb)(CID=(PROGRAM=INVMGR)(HOST=mynwdb)(USER=nusr))) * (ADDRESS=(PROTOCOL=tcp)(HOST=11.22.3.123)(PORT=65197)) * establish * mynwdb * 024-FEB-2017 09:11:15 * (CONNECT_DATA=(SID=mynwdb)(CID=(PROGRAM=INVMGR)(HOST=mynwdb)(USER=nusr))) * (ADDRESS=(PROTOCOL=tcp)(HOST=11.22.3.123)(PORT=65198)) * establish * mynwdb * 024-FEB-2017 09:11:15 * (CONNECT_DATA=(SID=mynwdb)(CID=(PROGRAM=INVMGR)(HOST=mynwdb)(USER=nusr))) * (ADDRESS=(PROTOCOL=tcp)(HOST=11.22.3.123)(PORT=65199)) * establish * mynwdb * 024-FEB-2017 09:11:15 * (CONNECT_DATA=(SID=mynwdb)(CID=(PROGRAM=INVMGR)(HOST=mynwdb)(USER=nusr))) * (ADDRESS=(PROTOCOL=tcp)(HOST=11.22.3.123)(PORT=65200)) * establish * mynwdb * 024-FEB-2017 09:11:15 * (CONNECT_DATA=(SID=mynwdb)(CID=(PROGRAM=INVMGR)(HOST=mynwdb)(USER=nusr))) * (ADDRESS=(PROTOCOL=tcp)(HOST=11.22.3.123)(PORT=65201)) * establish * mynwdb * 024-FEB-2017 09:11:16 * (CONNECT_DATA=(SID=mynwdb)(CID=(PROGRAM=INVMGR)(HOST=mynwdb)(USER=nusr))) * (ADDRESS=(PROTOCOL=tcp)(HOST=11.22.3.123)(PORT=65202)) * establish * mynwdb * 024-FEB-2017 09:11:16 * (CONNECT_DATA=(SID=mynwdb)(CID=(PROGRAM=INVMGR)(HOST=mynwdb)(USER=nusr))) * (ADDRESS=(PROTOCOL=tcp)(HOST=11.22.3.123)(PORT=65203)) * establish * mynwdb * 024-FEB-2017 09:11:16 * (CONNECT_DATA=(SID=mynwdb)(CID=(PROGRAM=INVMGR)(HOST=mynwdb)(USER=nusr))) * (ADDRESS=(PROTOCOL=tcp)(HOST=11.22.3.123)(PORT=65204)) * establish * mynwdb * 024-FEB-2017 09:11:16 * (CONNECT_DATA=(SID=mynwdb)(CID=(PROGRAM=INVMGR)(HOST=mynwdb)(USER=nusr))) * (ADDRESS=(PROTOCOL=tcp)(HOST=11.22.3.123)(PORT=65205)) * establish * mynwdb * 024-FEB-2017 09:11:16 * (CONNECT_DATA=(SID=mynwdb)(CID=(PROGRAM=INVMGR)(HOST=mynwdb)(USER=nusr))) * (ADDRESS=(PROTOCOL=tcp)(HOST=11.22.3.123)(PORT=65206)) * establish * mynwdb * 024-FEB-2017 09:11:17 * (CONNECT_DATA=(SID=mynwdb)(CID=(PROGRAM=INVMGR)(HOST=mynwdb)(USER=nusr))) * (ADDRESS=(PROTOCOL=tcp)(HOST=11.22.3.123)(PORT=65207)) * establish * mynwdb * 024-FEB-2017 09:11:17 * (CONNECT_DATA=(SID=mynwdb)(CID=(PROGRAM=INVMGR)(HOST=mynwdb)(USER=nusr))) * (ADDRESS=(PROTOCOL=tcp)(HOST=11.22.3.123)(PORT=65208)) * establish * mynwdb * 0……
这样我们会看的比较晕,不知道里面的连接数变化。
有一个listener日志的解析利器,可以分析到这种类型的问题。下载见文末。
用法:lsnr_analyzer.pl -t ‘time rexge' [-h] filename
如,我们要分析24-FEB-2017 09:30~09:59的连接情况,可以如下操作:
[ora@mynwdb trace]$ lsnr_analyzer.pl -t '24-FEB-2017 09:[3-5][0-9]' mynwdb_cut1.log24-FEB-2017 09:30 => Total:3724-FEB-2017 09:31 => Total:5024-FEB-2017 09:32 => Total:1724-FEB-2017 09:33 => Total:1624-FEB-2017 09:34 => Total:1924-FEB-2017 09:35 => Total:1624-FEB-2017 09:36 => Total:1724-FEB-2017 09:37 => Total:1624-FEB-2017 09:38 => Total:1324-FEB-2017 09:39 => Total:2924-FEB-2017 09:40 => Total:4324-FEB-2017 09:41 => Total:3824-FEB-2017 09:42 => Total:1724-FEB-2017 09:43 => Total:1624-FEB-2017 09:44 => Total:4224-FEB-2017 09:45 => Total:2124-FEB-2017 09:46 => Total:1624-FEB-2017 09:47 => Total:1524-FEB-2017 09:48 => Total:1224-FEB-2017 09:49 => Total:2524-FEB-2017 09:50 => Total:4724-FEB-2017 09:51 => Total:4124-FEB-2017 09:52 => Total:2124-FEB-2017 09:53 => Total:1224-FEB-2017 09:54 => Total:1724-FEB-2017 09:55 => Total:1624-FEB-2017 09:56 => Total:2024-FEB-2017 09:57 => Total:1724-FEB-2017 09:58 => Total:1724-FEB-2017 09:59 => Total:21[ora@mynwdb trace]$
可以看到,在09:31的时候,有个连接数的增加,到了一分钟50个连接。
我们可以加-h的参数,breakdown具体是哪些主机连接过来的:
[ora@mynwdb trace]$ lsnr_analyzer.pl -t '24-FEB-2017 09:[3-5][0-9]' -h mynwdb_cut1.log24-FEB-2017 09:30 => Total:3711.22.3.123 3724-FEB-2017 09:31 => Total:5022.3.4.111 111.22.3.123 4924-FEB-2017 09:32 => Total:1710.8.1.218 111.22.3.123 1624-FEB-2017 09:33 => Total:1611.22.3.123 1624-FEB-2017 09:34 => Total:1911.22.3.123 1924-FEB-2017 09:35 => Total:1611.22.3.123 1624-FEB-2017 09:36 => Total:1722.3.4.111 111.22.3.123 1624-FEB-2017 09:37 => Total:1612.5.6.221 111.22.3.123 1524-FEB-2017 09:38 => Total:1311.22.3.123 1324-FEB-2017 09:39 => Total:2911.22.3.123 2924-FEB-2017 09:40 => Total:4311.22.3.123 4324-FEB-2017 09:41 => Total:3822.3.4.111 111.22.3.123 3724-FEB-2017 09:42 => Total:1712.5.6.221 111.22.3.123 1624-FEB-2017 09:43 => Total:1611.22.3.123 1624-FEB-2017 09:44 => Total:4211.22.3.123 4224-FEB-2017 09:45 => Total:2111.22.3.123 2124-FEB-2017 09:46 => Total:1622.3.4.111 111.22.3.123 1524-FEB-2017 09:47 => Total:1512.5.6.221 111.22.3.123 1424-FEB-2017 09:48 => Total:1211.22.3.123 1224-FEB-2017 09:49 => Total:2511.22.3.123 2524-FEB-2017 09:50 => Total:4711.22.3.123 4724-FEB-2017 09:51 => Total:4122.3.4.111 111.22.3.123 4024-FEB-2017 09:52 => Total:2112.5.6.221 111.22.3.123 2024-FEB-2017 09:53 => Total:1211.22.3.123 1224-FEB-2017 09:54 => Total:1711.22.3.123 1724-FEB-2017 09:55 => Total:1611.22.3.123 1624-FEB-2017 09:56 => Total:2022.3.4.111 111.22.3.123 1924-FEB-2017 09:57 => Total:1712.5.6.221 111.22.3.123 1624-FEB-2017 09:58 => Total:1711.22.3.123 1724-FEB-2017 09:59 => Total:2111.22.3.123 21[ora@mynwdb trace]$
可以看到09:30的时候,是来自11.22.3.123有49个连接,来自22.3.4.111有1个连接。
注1:lsnr_analyzer.pl 是用来分析已经establish的连接,如果是其他连接,不统计在内。
注2:lsnr_analyzer.pl 的统计比较消耗cpu,30万行的listener日志分析大约1秒出结果,300万行的listener日志分析大约6秒出结果。所以如果listener日志比较大,建议tail剪切后统计。
对于已经出现tcpListenDrop的情况,我们分几种情况分析:
(1)如果是突然的连接风暴,需要控制前端的连接,如weblogic中间件逐台启动,而不是同时启动;
(2)如果一直连接数很高,导致listener无法处理,就需要拆分listener,如按照端口拆分。
点击下载:lsnr_analyzer.pl
总结
以上就是这篇文章的全部内容了,希望本文的内容对大家的学习或者工作能带来一定的帮助,如果有疑问大家可以留言交流,谢谢大家对武林网的支持。
新闻热点
疑难解答