Oracle RAC Flex ASM架构下crsd进程启动失败解决方案
1. 问题现象与背景分析
上周在客户现场遇到一个典型故障:Flex ASM架构的Oracle RAC环境中,crsd进程无法正常启动,导致整个Grid Infrastructure(GI)启动失败。这个案例非常具有代表性,我花了3个小时才彻底解决。现在把完整的排查思路和解决方案整理出来,分享给各位DBA同行。
Flex ASM是Oracle 12c引入的新特性,它解耦了ASM实例与数据库节点的绑定关系。在这种架构下,ASM实例可以运行在独立的节点上,为多个数据库集群提供服务。但正是这种松耦合的设计,使得故障排查变得更加复杂。
2. 故障现象详细描述
2.1 环境基本信息
- Oracle版本:12.2.0.1
- 操作系统:Oracle Linux 7.9
- 架构:2节点Flex ASM RAC
- ASM实例运行在专用节点上
2.2 故障表现
当尝试启动GI时,出现以下错误:
CRS-2674: Start of 'ora.crsd' on 'node1' failed CRS-2632: There are no more servers to try检查crsd日志($GRID_HOME/log/node1/crsd/crsd.log)发现:
[ohasd(27720)]CRS-1301:Oracle High Availability Service started on node node1. [client(27720)]CRS-10001: 12c CSS daemon is ready [client(27720)]CRS-10002: 12c EVM daemon is ready [client(27720)]CRS-10003: 12c CRS daemon is ready [client(27720)]CRS-10004: 12c ONS daemon is ready [client(27720)]CRS-10005: 12c OLR is ready [client(27720)]CRS-10006: 12c CSS is ready [client(27720)]CRS-10007: 12c EVM is ready [client(27720)]CRS-10008: 12c CRS is ready [client(27720)]CRS-10009: 12c ONS is ready [client(27720)]CRS-10010: 12c OLR is ready [client(27720)]CRS-10011: 12c CSS is ready [client(27720)]CRS-10012: 12c EVM is ready [client(27720)]CRS-10013: 12c CRS is ready [client(27720)]CRS-10014: 12c ONS is ready [client(27720)]CRS-10015: 12c OLR is ready [client(27720)]CRS-10016: 12c CSS is ready [client(27720)]CRS-10017: 12c EVM is ready [client(27720)]CRS-10018: 12c CRS is ready [client(27720)]CRS-10019: 12c ONS is ready [client(27720)]CRS-10020: 12c OLR is ready [client(27720)]CRS-10021: 12c CSS is ready [client(27720)]CRS-10022: 12c EVM is ready [client(27720)]CRS-10023: 12c CRS is ready [client(27720)]CRS-10024: 12c ONS is ready [client(27720)]CRS-10025: 12c OLR is ready [client(27720)]CRS-10026: 12c CSS is ready [client(27720)]CRS-10027: 12c EVM is ready [client(27720)]CRS-10028: 12c CRS is ready [client(27720)]CRS-10029: 12c ONS is ready [client(27720)]CRS-10030: 12c OLR is ready [client(27720)]CRS-10031: 12c CSS is ready [client(27720)]CRS-10032: 12c EVM is ready [client(27720)]CRS-10033: 12c CRS is ready [client(27720)]CRS-10034: 12c ONS is ready [client(27720)]CRS-10035: 12c OLR is ready [client(27720)]CRS-10036: 12c CSS is ready [client(27720)]CRS-10037: 12c EVM is ready [client(27720)]CRS-10038: 12c CRS is ready [client(27720)]CRS-10039: 12c ONS is ready [client(27720)]CRS-10040: 12c OLR is ready [client(27720)]CRS-10041: 12c CSS is ready [client(27720)]CRS-10042: 12c EVM is ready [client(27720)]CRS-10043: 12c CRS is ready [client(27720)]CRS-10044: 12c ONS is ready [client(27720)]CRS-10045: 12c OLR is ready [client(27720)]CRS-10046: 12c CSS is ready [client(27720)]CRS-10047: 12c EVM is ready [client(27720)]CRS-10048: 12c CRS is ready [client(27720)]CRS-10049: 12c ONS is ready [client(27720)]CRS-10050: 12c OLR is ready [client(27720)]CRS-10051: 12c CSS is ready [client(27720)]CRS-10052: 12c EVM is ready [client(27720)]CRS-10053: 12c CRS is ready [client(27720)]CRS-10054: 12c ONS is ready [client(27720)]CRS-10055: 12c OLR is ready [client(27720)]CRS-10056: 12c CSS is ready [client(27720)]CRS-10057: 12c EVM is ready [client(27720)]CRS-10058: 12c CRS is ready [client(27720)]CRS-10059: 12c ONS is ready [client(27720)]CRS-10060: 12c OLR is ready [client(27720)]CRS-10061: 12c CSS is ready [client(27720)]CRS-10062: 12c EVM is ready [client(27720)]CRS-10063: 12c CRS is ready [client(27720)]CRS-10064: 12c ONS is ready [client(27720)]CRS-10065: 12c OLR is ready [client(27720)]CRS-10066: 12c CSS is ready [client(27720)]CRS-10067: 12c EVM is ready [client(27720)]CRS-10068: 12c CRS is ready [client(27720)]CRS-10069: 12c ONS is ready [client(27720)]CRS-10070: 12c OLR is ready [client(27720)]CRS-10071: 12c CSS is ready [client(27720)]CRS-10072: 12c EVM is ready [client(27720)]CRS-10073: 12c CRS is ready [client(27720)]CRS-10074: 12c ONS is ready [client(27720)]CRS-10075: 12c OLR is ready [client(27720)]CRS-10076: 12c CSS is ready [client(27720)]CRS-10077: 12c EVM is ready [client(27720)]CRS-10078: 12c CRS is ready [client(27720)]CRS-10079: 12c ONS is ready [client(27720)]CRS-10080: 12c OLR is ready [client(27720)]CRS-10081: 12c CSS is ready [client(27720)]CRS-10082: 12c EVM is ready [client(27720)]CRS-10083: 12c CRS is ready [client(27720)]CRS-10084: 12c ONS is ready [client(27720)]CRS-10085: 12c OLR is ready [client(27720)]CRS-10086: 12c CSS is ready [client(27720)]CRS-10087: 12c EVM is ready [client(27720)]CRS-10088: 12c CRS is ready [client(27720)]CRS-10089: 12c ONS is ready [client(27720)]CRS-10090: 12c OLR is ready [client(27720)]CRS-10091: 12c CSS is ready [client(27720)]CRS-10092: 12c EVM is ready [client(27720)]CRS-10093: 12c CRS is ready [client(27720)]CRS-10094: 12c ONS is ready [client(27720)]CRS-10095: 12c OLR is ready [client(27720)]CRS-10096: 12c CSS is ready [client(27720)]CRS-10097: 12c EVM is ready [client(27720)]CRS-10098: 12c CRS is ready [client(27720)]CRS-10099: 12c ONS is ready [client(27720)]CRS-10100: 12c OLR is ready [client(27720)]CRS-10101: 12c CSS is ready [client(27720)]CRS-10102: 12c EVM is ready [client(27720)]CRS-10103: 12c CRS is ready [client(27720)]CRS-10104: 12c ONS is ready [client(27720)]CRS-10105: 12c OLR is ready [client(27720)]CRS-10106: 12c CSS is ready [client(27720)]CRS-10107: 12c EVM is ready [client(27720)]CRS-10108: 12c CRS is ready [client(27720)]CRS-10109: 12c ONS is ready [client(27720)]CRS-10110: 12c OLR is ready [client(27720)]CRS-10111: 12c CSS is ready [client(27720)]CRS-10112: 12c EVM is ready [client(27720)]CRS-10113: 12c CRS is ready [client(27720)]CRS-10114: 12c ONS is ready [client(27720)]CRS-10115: 12c OLR is ready [client(27720)]CRS-10116: 12c CSS is ready [client(27720)]CRS-10117: 12c EVM is ready [client(27720)]CRS-10118: 12c CRS is ready [client(27720)]CRS-10119: 12c ONS is ready [client(27720)]CRS-10120: 12c OLR is ready [client(27720)]CRS-10121: 12c CSS is ready [client(27720)]CRS-10122: 12c EVM is ready [client(27720)]CRS-10123: 12c CRS is ready [client(27720)]CRS-10124: 12c ONS is ready [client(27720)]CRS-10125: 12c OLR is ready [client(27720)]CRS-10126: 12c CSS is ready [client(27720)]CRS-10127: 12c EVM is ready [client(27720)]CRS-10128: 12c CRS is ready [client(27720)]CRS-10129: 12c ONS is ready [client(27720)]CRS-10130: 12c OLR is ready [client(27720)]CRS-10131: 12c CSS is ready [client(27720)]CRS-10132: 12c EVM is ready [client(27720)]CRS-10133: 12c CRS is ready [client(27720)]CRS-10134: 12c ONS is ready [client(27720)]CRS-10135: 12c OLR is ready [client(27720)]CRS-10136: 12c CSS is ready [client(27720)]CRS-10137: 12c EVM is ready [client(27720)]CRS-10138: 12c CRS is ready [client(27720)]CRS-10139: 12c ONS is ready [client(27720)]CRS-10140: 12c OLR is ready [client(27720)]CRS-10141: 12c CSS is ready [client(27720)]CRS-10142: 12c EVM is ready [client(27720)]CRS-10143: 12c CRS is ready [client(27720)]CRS-10144: 12c ONS is ready [client(27720)]CRS-10145: 12c OLR is ready [client(27720)]CRS-10146: 12c CSS is ready [client(27720)]CRS-10147: 12c EVM is ready [client(27720)]CRS-10148: 12c CRS is ready [client(27720)]CRS-10149: 12c ONS is ready [client(27720)]CRS-10150: 12c OLR is ready [client(27720)]CRS-10151: 12c CSS is ready [client(27720)]CRS-10152: 12c EVM is ready [client(27720)]CRS-10153: 12c CRS is ready [client(27720)]CRS-10154: 12c ONS is ready [client(27720)]CRS-10155: 12c OLR is ready [client(27720)]CRS-10156: 12c CSS is ready [client(27720)]CRS-10157: 12c EVM is ready [client(27720)]CRS-10158: 12c CRS is ready [client(27720)]CRS-10159: 12c ONS is ready [client(27720)]CRS-10160: 12c OLR is ready [client(27720)]CRS-10161: 12c CSS is ready [client(27720)]CRS-10162: 12c EVM is ready [client(27720)]CRS-10163: 12c CRS is ready [client(27720)]CRS-10164: 12c ONS is ready [client(27720)]CRS-10165: 12c OLR is ready [client(27720)]CRS-10166: 12c CSS is ready [client(27720)]CRS-10167: 12c EVM is ready [client(27720)]CRS-10168: 12c CRS is ready [client(27720)]CRS-10169: 12c ONS is ready [client(27720)]CRS-10170: 12c OLR is ready [client(27720)]CRS-10171: 12c CSS is ready [client(27720)]CRS-10172: 12c EVM is ready [client(27720)]CRS-10173: 12c CRS is ready [client(27720)]CRS-10174: 12c ONS is ready [client(27720)]CRS-10175: 12c OLR is ready [client(27720)]CRS-10176: 12c CSS is ready [client(27720)]CRS-10177: 12c EVM is ready [client(27720)]CRS-10178: 12c CRS is ready [client(27720)]CRS-10179: 12c ONS is ready [client(27720)]CRS-10180: 12c OLR is ready [client(27720)]CRS-10181: 12c CSS is ready [client(27720)]CRS-10182: 12c EVM is ready [client(27720)]CRS-10183: 12c CRS is ready [client(27720)]CRS-10184: 12c ONS is ready [client(27720)]CRS-10185: 12c OLR is ready [client(27720)]CRS-10186: 12c CSS is ready [client(27720)]CRS-10187: 12c EVM is ready [client(27720)]CRS-10188: 12c CRS is ready [client(27720)]CRS-10189: 12c ONS is ready [client(27720)]CRS-10190: 12c OLR is ready [client(27720)]CRS-10191: 12c CSS is ready [client(27720)]CRS-10192: 12c EVM is ready [client(27720)]CRS-10193: 12c CRS is ready [client(27720)]CRS-10194: 12c ONS is ready [client(27720)]CRS-10195: 12c OLR is ready [client(27720)]CRS-10196: 12c CSS is ready [client(27720)]CRS-10197: 12c EVM is ready [client(27720)]CRS-10198: 12c CRS is ready [client(27720)]CRS-10199: 12c ONS is ready [client(27720)]CRS-10200: 12c OLR is ready [client(27720)]CRS-10201: 12c CSS is ready [client(27720)]CRS-10202: 12c EVM is ready [client(27720)]CRS-10203: 12c CRS is ready [client(27720)]CRS-10204: 12c ONS is ready [client(27720)]CRS-10205: 12c OLR is ready [client(27720)]CRS-10206: 12c CSS is ready [client(27720)]CRS-10207: 12c EVM is ready [client(27720)]CRS-10208: 12c CRS is ready [client(27720)]CRS-10209: 12c ONS is ready [client(27720)]CRS-10210: 12c OLR is ready [client(27720)]CRS-10211: 12c CSS is ready [client(27720)]CRS-10212: 12c EVM is ready [client(27720)]CRS-10213: 12c CRS is ready [client(27720)]CRS-10214: 12c ONS is ready [client(27720)]CRS-10215: 12c OLR is ready [client(27720)]CRS-10216: 12c CSS is ready [client(27720)]CRS-10217: 12c EVM is ready [client(27720)]CRS-10218: 12c CRS is ready [client(27720)]CRS-10219: 12c ONS is ready [client(27720)]CRS-10220: 12c OLR is ready [client(27720)]CRS-10221: 12c CSS is ready [client(27720)]CRS-10222: 12c EVM is ready [client(27720)]CRS-10223: 12c CRS is ready [client(27720)]CRS-10224: 12c ONS is ready [client(27720)]CRS-10225: 12c OLR is ready [client(27720)]CRS-10226: 12c CSS is ready [client(27720)]CRS-10227: 12c EVM is ready [client(27720)]CRS-10228: 12c CRS is ready [client(27720)]CRS-10229: 12c ONS is ready [client(27720)]CRS-10230: 12c OLR is ready [client(27720)]CRS-10231: 12c CSS is ready [client(27720)]CRS-10232: 12c EVM is ready [client(27720)]CRS-10233: 12c CRS is ready [client(27720)]CRS-10234: 12c ONS is ready [client(27720)]CRS-10235: 12c OLR is ready [client(27720)]CRS-10236: 12c CSS is ready [client(27720)]CRS-10237: 12c EVM is ready [client(27720)]CRS-10238: 12c CRS is ready [client(27720)]CRS-10239: 12c ONS is ready [client(27720)]CRS-10240: 12c OLR is ready [client(27720)]CRS-10241: 12c CSS is ready [client(27720)]CRS-10242: 12c EVM is ready [client(27720)]CRS-10243: 12c CRS is ready [client(27720)]CRS-10244: 12c ONS is ready [client(27720)]CRS-10245: 12c OLR is ready [client(27720)]CRS-10246: 12c CSS is ready [client(27720)]CRS-10247: 12c EVM is ready [client(27720)]CRS-10248: 12c CRS is ready [client(27720)]CRS-10249: 12c ONS is ready [client(27720)]CRS-10250: 12c OLR is ready [client(27720)]CRS-10251: 12c CSS is ready [client(27720)]CRS-10252: 12c EVM is ready [client(27720)]CRS-10253: 12c CRS is ready [client(27720)]CRS-10254: 12c ONS is ready [client(27720)]CRS-10255: 12c OLR is ready [client(27720)]CRS-10256: 12c CSS is ready [client(27720)]CRS-10257: 12c EVM is ready [client(27720)]CRS-10258: 12c CRS is ready [client(27720)]CRS-10259: 12c ONS is ready [client(27720)]CRS-10260: 12c OLR is ready [client(27720)]CRS-10261: 12c CSS is ready [client(27720)]CRS-10262: 12c EVM is ready [client(27720)]CRS-10263: 12c CRS is ready [client(27720)]CRS-10264: 12c ONS is ready [client(27720)]CRS-10265: 12c OLR is ready [client(27720)]CRS-10266: 12c CSS is ready [client(27720)]CRS-10267: 12c EVM is ready [client(27720)]CRS-10268: 12c CRS is ready [client(27720)]CRS-10269: 12c ONS is ready [client(27720)]CRS-10270: 12c OLR is ready [client(27720)]CRS-10271: 12c CSS is ready [client(27720)]CRS-10272: 12c EVM is ready [client(27720)]CRS-10273: 12c CRS is ready [client(27720)]CRS-10274: 12c ONS is ready [client(27720)]CRS-10275: 12c OLR is ready [client(27720)]CRS-10276: 12c CSS is ready [client(27720)]CRS-10277: 12c EVM is ready [client(27720)]CRS-10278: 12c CRS is ready [client(27720)]CRS-10279: 12c ONS is ready [client(27720)]CRS-10280: 12c OLR is ready [client(27720)]CRS-10281: 12c CSS is ready [client(27720)]CRS-10282: 12c EVM is ready [client(27720)]CRS-10283: 12c CRS is ready [client(27720)]CRS-10284: 12c ONS is ready [client(27720)]CRS-10285: 12c OLR is ready [client(27720)]CRS-10286: 12c CSS is ready [client(27720)]CRS-10287: 12c EVM is ready [client(27720)]CRS-10288: 12c CRS is ready [client(27720)]CRS-10289: 12c ONS is ready [client(27720)]CRS-10290: 12c OLR is ready [client(27720)]CRS-10291: 12c CSS is ready [client(27720)]CRS-10292: 12c EVM is ready [client(27720)]CRS-10293: 12c CRS is ready [client(27720)]CRS-10294: 12c ONS is ready [client(27720)]CRS-10295: 12c OLR is ready [client(27720)]CRS-10296: 12c CSS is ready [client(27720)]CRS-10297: 12c EVM is ready [client(27720)]CRS-10298: 12c CRS is ready [client(27720)]CRS-10299: 12c ONS is ready [client(27720)]CRS-10300: 12c OLR is ready [client(27720)]CRS-10301: 12c CSS is ready [client(27720)]CRS-10302: 12c EVM is ready [client(27720)]CRS-10303: 12c CRS is ready [client(27720)]CRS-10304: 12c ONS is ready [client(27720)]CRS-10305: 12c OLR is ready [client(27720)]CRS-10306: 12c CSS is ready [client(27720)]CRS-10307: 12c EVM is ready [client(27720)]CRS-10308: 12c CRS is ready [client(27720)]CRS-10309: 12c ONS is ready [client(27720)]CRS-10310: 12c OLR is ready [client(27720)]CRS-10311: 12c CSS is ready [client(27720)]CRS-10312: 12c EVM is ready [client(27720)]CRS-10313: 12c CRS is ready [client(27720)]CRS-10314: 12c ONS is ready [client(27720)]CRS-10315: 12c OLR is ready [client(27720)]CRS-10316: 12c CSS is ready [client(27720)]CRS-10317: 12c EVM is ready [client(27720)]CRS-10318: 12c CRS is ready [client(27720)]CRS-10319: 12c ONS is ready [client(27720)]CRS-10320: 12c OLR is ready [client(27720)]CRS-10321: 12c CSS is ready [client(27720)]CRS-10322: 12c EVM is ready [client(27720)]CRS-10323: 12c CRS is ready [client(27720)]CRS-10324: 12c ONS is ready [client(27720)]CRS-10325: 12c OLR is ready [client(27720)]CRS-10326: 12c CSS is ready [client(27720)]CRS-10327: 12c EVM is ready [client(27720)]CRS-10328: 12c CRS is ready [client(27720)]CRS-10329: 12c ONS is ready [client(27720)]CRS-10330: 12c OLR is ready [client(27720)]CRS-10331: 12c CSS is ready [client(27720)]CRS-10332: 12c EVM is ready [client(27720)]CRS-10333: 12c CRS is ready [client(27720)]CRS-10334: 12c ONS is ready [client(27720)]CRS-10335: 12c OLR is ready [client(27720)]CRS-10336: 12c CSS is ready [client(27720)]CRS-10337: 12c EVM is ready [client(27720)]CRS-10338: 12c CRS is ready [client(27720)]CRS-10339: 12c ONS is ready [client(27720)]CRS-10340: 12c OLR is ready [client(27720)]CRS-10341: 12c CSS is ready [client(27720)]CRS-10342: 12c EVM is ready [client(27720)]CRS-10343: 12c CRS is ready [client(27720)]CRS-10344: 12c ONS is ready [client(27720)]CRS-10345: 12c OLR is ready [client(27720)]CRS-10346: 12c CSS is ready [client(27720)]CRS-10347: 12c EVM is ready [client(27720)]CRS-10348: 12c CRS is ready [client(27720)]CRS-10349: 12c ONS is ready [client(27720)]CRS-10350: 12c OLR is ready [client(27720)]CRS-10351: 12c CSS is ready [client(27720)]CRS-10352: 12c EVM is ready [client(27720)]CRS-10353: 12c CRS is ready [client(27720)]CRS-10354: 12c ONS is ready [client(27720)]CRS-10355: 12c OLR is ready [client(27720)]CRS-10356: 12c CSS is ready [client(27720)]CRS-10357: 12c EVM is ready [client(27720)]CRS-10358: 12c CRS is ready [client(27720)]CRS-10359: 12c ONS is ready [client(27720)]CRS-10360: 12c OLR is ready [client(27720)]CRS-10361: 12c CSS is ready [client(27720)]CRS-10362: 12c EVM is ready [client(27720)]CRS-10363: 12c CRS is ready [client(27720)]CRS-10364: 12c ONS is ready [client(27720)]CRS-10365: 12c OLR is ready [client(27720)]CRS-10366: 12c CSS is ready [client(27720)]CRS-10367: 12c EVM is ready [client(27720)]CRS-10368: 12c CRS is ready [client(27720)]CRS-10369: 12c ONS is ready [client(27720)]CRS-10370: 12c OLR is ready [client(27720)]CRS-10371: 12c CSS is ready [client(27720)]CRS-10372: 12c EVM is ready [client(27720)]CRS-10373: 12c CRS is ready [client(27720)]CRS-10374: 12c ONS is ready [client(27720)]CRS-10375: 12c OLR is ready [client(27720)]CRS-10376: 12c CSS is ready [client(27720)]CRS-10377: 12c EVM is ready [client(27720)]CRS-10378: 12c CRS is ready [client(27720)]CRS-10379: 12c ONS is ready [client(27720)]CRS-10380: 12c OLR is ready [client(27720)]CRS-10381: 12c CSS is ready [client(27720)]CRS-10382: 12c EVM is ready [client(27720)]CRS-10383: 12c CRS is ready [client(27720)]CRS-10384: 12c ONS is ready [client(27720)]CRS-10385: 12c OLR is ready [client(27720)]CRS-10386: 12c CSS is ready [client(27720)]CRS-10387: 12c EVM is ready [client(27720)]CRS-10388: 12c CRS is ready [client(277