Instance terminated : ORA-7445 in post-upgrade steps

it’s Maundy Thursday and i’m doing database upgrade from 11.1.0.7.0 to 11.2.0.4.1. Database is a production and it’s 9TB.
i always DBUA silent when doing upgrade to make my life easier.

completed all pre upgrade task, same task i’m doing for the past month. been upgrading 4-5 production DB per week since February.

Below is the output of DBUA on my console.

Log files for the upgrade operation are located at: /opt/app/oracle/cfgtoollogs/dbua/XMIDB/upgrade1
Performing Pre Upgrade
1% complete
7% complete
Upgrading Oracle Server
8% complete
9% complete
10% complete
10% complete
11% complete
12% complete
13% complete
13% complete
14% complete
15% complete
16% complete
16% complete
17% complete
18% complete
19% complete
20% complete
20% complete
21% complete
22% complete
23% complete
23% complete
24% complete
Upgrading JServer JAVA Virtual Machine
26% complete
26% complete
27% complete
Upgrading Oracle XDK for Java
29% complete
30% complete
Upgrading OLAP Analytic Workspace
31% complete
Upgrading OLAP Catalog
33% complete
Upgrading EM Repository
34% complete
35% complete
36% complete
36% complete
37% complete
38% complete
39% complete
40% complete
40% complete
41% complete
42% complete
43% complete
43% complete
44% complete
45% complete
46% complete
46% complete
47% complete
48% complete
49% complete
50% complete
50% complete
51% complete
52% complete
53% complete
53% complete
54% complete
55% complete
56% complete
56% complete
57% complete
58% complete
59% complete
Upgrading Oracle Text
60% complete
Upgrading Oracle XML Database
62% complete
63% complete
63% complete
64% complete
Upgrading Oracle Java Packages
66% complete
Upgrading Oracle interMedia
67% complete
68% complete
69% complete
Upgrading Spatial
70% complete
71% complete
72% complete
73% complete
73% complete
74% complete
75% complete
76% complete
76% complete
77% complete
78% complete
79% complete
80% complete
80% complete
81% complete
Upgrading Oracle Workspace Manager
83% complete
Upgrading Expression Filter
84% complete
Upgrading Rule Manager
Upgrading Oracle Application Express
Upgrading Oracle OLAP API
Performing Post Upgrade
100% complete
Check the log file "/opt/app/oracle/cfgtoollogs/dbua/logs/silent.log" for upgrade details.

since it says 100% complete, when i logged in database is not running. checking the alert show..

ORA-07445: exception encountered: core dump [pevm_icd_call_common()+755] [SIGSEGV] [ADDR:0xD8] [PC:0x98A5475] [Address not mapped to object] []
ORA-04067: not executed, package body "GGSADMIN.DDLREPLICATION" does not exist
ORA-06508: PL/SQL: could not find program unit being called: "GGSADMIN.DDLREPLICATION"
ORA-04067: not executed, package body "GGSADMIN.DDLREPLICATION" does not exist
ORA-06508: PL/SQL: could not find program unit being called: "GGSADMIN.DDLREPLICATION"
Incident details in: /opt/app/oracle/diag/rdbms/xmidb/XMIDB/incident/incdir_158756/PMIDB_ora_32473_i158756.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Wed Apr 16 23:41:56 2014
Dumping diagnostic data in directory=[cdmp_20140416234156], requested by (instance=1, osid=32473), summary=[incident=158756].
Wed Apr 16 23:41:56 2014
Sweep [inc][158756]: completed
Sweep [inc2][158756]: completed
System state dump requested by (instance=1, osid=32413 (PMON)), summary=[abnormal instance termination].
System State dumped to trace file /opt/app/oracle/diag/rdbms/xmidb/XMIDB/trace/XMIDB_diag_32424_20140416234157.trc
Wed Apr 16 23:41:57 2014
PMON (ospid: 32413): terminating the instance due to error 397
Dumping diagnostic data in directory=[cdmp_20140416234157], requested by (instance=1, osid=32413 (PMON)), summary=[abnormal instance termination].
Instance terminated by PMON, pid = 32413

checking the /opt/app/oracle/cfgtoollogs/dbua/logs/silent.log show database re

Performing Pre Upgrade
UPGRADE_PROGRESS : 1%
UPGRADE_PROGRESS : 7%
Upgrading Oracle Server
UPGRADE_PROGRESS : 8%
UPGRADE_PROGRESS : 9%
UPGRADE_PROGRESS : 10%
UPGRADE_PROGRESS : 10%
UPGRADE_PROGRESS : 11%
UPGRADE_PROGRESS : 12%
UPGRADE_PROGRESS : 13%
UPGRADE_PROGRESS : 13%
UPGRADE_PROGRESS : 14%
UPGRADE_PROGRESS : 15%
UPGRADE_PROGRESS : 16%
UPGRADE_PROGRESS : 16%
UPGRADE_PROGRESS : 17%
UPGRADE_PROGRESS : 18%
UPGRADE_PROGRESS : 19%
UPGRADE_PROGRESS : 20%
UPGRADE_PROGRESS : 20%
UPGRADE_PROGRESS : 21%
UPGRADE_PROGRESS : 22%
UPGRADE_PROGRESS : 23%
UPGRADE_PROGRESS : 23%
UPGRADE_PROGRESS : 24%
Upgrading JServer JAVA Virtual Machine
UPGRADE_PROGRESS : 26%
UPGRADE_PROGRESS : 26%
UPGRADE_PROGRESS : 27%
Upgrading Oracle XDK for Java
UPGRADE_PROGRESS : 29%
UPGRADE_PROGRESS : 30%
Upgrading OLAP Analytic Workspace
UPGRADE_PROGRESS : 31%
Upgrading OLAP Catalog
UPGRADE_PROGRESS : 33%
Upgrading EM Repository
UPGRADE_PROGRESS : 34%
UPGRADE_PROGRESS : 35%
UPGRADE_PROGRESS : 36%
UPGRADE_PROGRESS : 36%
UPGRADE_PROGRESS : 37%
UPGRADE_PROGRESS : 38%
UPGRADE_PROGRESS : 39%
UPGRADE_PROGRESS : 40%
UPGRADE_PROGRESS : 40%
UPGRADE_PROGRESS : 41%
UPGRADE_PROGRESS : 42%
UPGRADE_PROGRESS : 43%
UPGRADE_PROGRESS : 43%
UPGRADE_PROGRESS : 44%
UPGRADE_PROGRESS : 45%
UPGRADE_PROGRESS : 46%
UPGRADE_PROGRESS : 46%
UPGRADE_PROGRESS : 47%
UPGRADE_PROGRESS : 48%
UPGRADE_PROGRESS : 49%
UPGRADE_PROGRESS : 50%
UPGRADE_PROGRESS : 50%
UPGRADE_PROGRESS : 51%
UPGRADE_PROGRESS : 52%
UPGRADE_PROGRESS : 53%
UPGRADE_PROGRESS : 53%
UPGRADE_PROGRESS : 54%
UPGRADE_PROGRESS : 55%
UPGRADE_PROGRESS : 56%
UPGRADE_PROGRESS : 56%
UPGRADE_PROGRESS : 57%
UPGRADE_PROGRESS : 58%
UPGRADE_PROGRESS : 59%
Upgrading Oracle Text
UPGRADE_PROGRESS : 60%
Upgrading Oracle XML Database
UPGRADE_PROGRESS : 62%
UPGRADE_PROGRESS : 63%
UPGRADE_PROGRESS : 63%
UPGRADE_PROGRESS : 64%
Upgrading Oracle Java Packages
UPGRADE_PROGRESS : 66%
Upgrading Oracle interMedia
UPGRADE_PROGRESS : 67%
UPGRADE_PROGRESS : 68%
UPGRADE_PROGRESS : 69%
Upgrading Spatial
UPGRADE_PROGRESS : 70%
UPGRADE_PROGRESS : 71%
UPGRADE_PROGRESS : 72%
UPGRADE_PROGRESS : 73%
UPGRADE_PROGRESS : 73%
UPGRADE_PROGRESS : 74%
UPGRADE_PROGRESS : 75%
UPGRADE_PROGRESS : 76%
UPGRADE_PROGRESS : 76%
UPGRADE_PROGRESS : 77%
UPGRADE_PROGRESS : 78%
UPGRADE_PROGRESS : 79%
UPGRADE_PROGRESS : 80%
UPGRADE_PROGRESS : 80%
UPGRADE_PROGRESS : 81%
Upgrading Oracle Workspace Manager
UPGRADE_PROGRESS : 83%
Upgrading Expression Filter
UPGRADE_PROGRESS : 84%
Upgrading Rule Manager
Upgrading Oracle Application Express
Upgrading Oracle OLAP API
Performing Post Upgrade
ORA-03113: end-of-file on communication channel

ORA-03113: end-of-file on communication channel

Upgrade failed due to running the step "Upgrading Oracle Server"
UPGRADE_PROGRESS : 100%

since database is a production, i opened a Sev 1 ticket with oracle. but i continue working on it.
below steps what i did to start the database.
note : database is not able start in normal mode.

1. start database in upgrade mode.

SQL> startup upgrade
ORACLE instance started.

Total System Global Area 6413680640 bytes
Fixed Size                  2265224 bytes
Variable Size            3758100344 bytes
Database Buffers         2634022912 bytes
Redo Buffers               19292160 bytes
Database mounted.
Database opened.
SQL>

2. Check if all component is valid.

SQL> @/opt/app/oracle/product/11.2.0.4/db_1/rdbms/admin/utlu112s.sql
.
Oracle Database 11.2 Post-Upgrade Status Tool           04-17-2014 01:47:53
.
Component                               Current      Version     Elapsed Time
Name                                    Status       Number      HH:MM:SS
.
Oracle Server
.                                         VALID      11.2.0.4.0  00:14:47
JServer JAVA Virtual Machine
.                                         VALID      11.2.0.4.0  00:04:23
Oracle Workspace Manager
.                                         VALID      11.2.0.4.0  00:00:23
OLAP Analytic Workspace
.                                         VALID      11.2.0.4.0  00:00:32
OLAP Catalog
.                                         VALID      11.2.0.4.0  00:00:27
Oracle OLAP API
.                                         VALID      11.2.0.4.0  00:00:14
Oracle Enterprise Manager
.                                         VALID      11.2.0.4.0  00:02:14
Oracle XDK
.                                         VALID      11.2.0.4.0  00:00:18
Oracle Text
.                                         VALID      11.2.0.4.0  00:00:27
Oracle XML Database
.                                         VALID      11.2.0.4.0  00:01:38
Oracle Database Java Packages
.                                         VALID      11.2.0.4.0  00:00:10
Oracle Multimedia
.                                         VALID      11.2.0.4.0  00:01:57
Spatial
.                                         VALID      11.2.0.4.0  00:02:31
Oracle Expression Filter
.                                         VALID      11.2.0.4.0  00:00:09
Oracle Rules Manager
.                                         VALID      11.2.0.4.0  00:00:05
Oracle Application Express
.                                         VALID     3.2.1.00.12  00:07:34
Final Actions
.                                                                00:00:41
Total Upgrade Time: 00:38:48

PL/SQL procedure successfully completed.

SQL>

3.Since everything is valid execute utlrp.sql, i still have 5 invalid, before doing utlrp the ivalid object is more that 2000.
those invalid object are owned by other users.

SQL> @?/rdbms/admin/utlrp.sql

TIMESTAMP
-------------------------------------------------------------------------------------------------------------------------------------------------
COMP_TIMESTAMP UTLRP_BGN  2014-04-17 03:28:26

DOC>   The following PL/SQL block invokes UTL_RECOMP to recompile invalid
DOC>   objects in the database. Recompilation time is proportional to the
DOC>   number of invalid objects in the database, so this command may take
DOC>   a long time to execute on a database with a large number of invalid
DOC>   objects.
DOC>
DOC>   Use the following queries to track recompilation progress:
DOC>
DOC>   1. Query returning the number of invalid objects remaining. This
DOC>      number should decrease with time.
DOC>         SELECT COUNT(*) FROM obj$ WHERE status IN (4, 5, 6);
DOC>
DOC>   2. Query returning the number of objects compiled so far. This number
DOC>      should increase with time.
DOC>         SELECT COUNT(*) FROM UTL_RECOMP_COMPILED;
DOC>
DOC>   This script automatically chooses serial or parallel recompilation
DOC>   based on the number of CPUs available (parameter cpu_count) multiplied
DOC>   by the number of threads per CPU (parameter parallel_threads_per_cpu).
DOC>   On RAC, this number is added across all RAC nodes.
DOC>
DOC>   UTL_RECOMP uses DBMS_SCHEDULER to create jobs for parallel
DOC>   recompilation. Jobs are created without instance affinity so that they
DOC>   can migrate across RAC nodes. Use the following queries to verify
DOC>   whether UTL_RECOMP jobs are being created and run correctly:
DOC>
DOC>   1. Query showing jobs created by UTL_RECOMP
DOC>         SELECT job_name FROM dba_scheduler_jobs
DOC>            WHERE job_name like 'UTL_RECOMP_SLAVE_%';
DOC>
DOC>   2. Query showing UTL_RECOMP jobs that are running
DOC>         SELECT job_name FROM dba_scheduler_running_jobs
DOC>            WHERE job_name like 'UTL_RECOMP_SLAVE_%';
DOC>#

PL/SQL procedure successfully completed.


TIMESTAMP
-------------------------------------------------------------------------------------------------------------------------------------------------
COMP_TIMESTAMP UTLRP_END  2014-04-17 03:29:18

DOC> The following query reports the number of objects that have compiled
DOC> with errors.
DOC>
DOC> If the number is higher than expected, please examine the error
DOC> messages reported with each object (using SHOW ERRORS) to see if they
DOC> point to system misconfiguration or resource constraints that must be
DOC> fixed before attempting to recompile these objects.
DOC>#

OBJECTS WITH ERRORS
-------------------
                  5

DOC> The following query reports the number of errors caught during
DOC> recompilation. If this number is non-zero, please query the error
DOC> messages in the table UTL_RECOMP_ERRORS to see if any of these errors
DOC> are due to misconfiguration or resource constraints that must be
DOC> fixed before objects can compile successfully.
DOC>#

ERRORS DURING RECOMPILATION
---------------------------
                          5


Function created.


PL/SQL procedure successfully completed.


Function dropped.


PL/SQL procedure successfully completed.

SQL> 

4. Now restart the database is normal mode.

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area 6413680640 bytes
Fixed Size                  2265224 bytes
Variable Size            3758100344 bytes
Database Buffers         2634022912 bytes
Redo Buffers               19292160 bytes
Database mounted.
Database opened.
SQL>

database looks good, after few hour oracle updated my SR. with below information.


Oracle Support- 5+ hours ago[ODM Action Plan]
Hi

I went through the uploaded trace files and below is the workaround to fix the ORA-07445 errors.

Symptoms:
==========
ORA-07445: exception encountered: core dump [pevm_icd_call_common()+755] [SIGSEGV] [ADDR:0xD8] [PC:0x98A5475] [Address not mapped to object] []
ORA-04067: not executed, package body "GGSADMIN.DDLREPLICATION" does not exist
ORA-06508: PL/SQL: could not find program unit being called: "GGSADMIN.DDLREPLICATION"
ORA-04067: not executed, package body "GGSADMIN.DDLREPLICATION" does not exist
ORA-06508: PL/SQL: could not find program unit being called: "GGSADMIN.DDLREPLICATION"

Cause
======

A Golden Gate trigger is causing the upgrade failure

Solution
========

Set the following parameter in your parameter file and then try to open the database:

"_system_trig_enabled" =FALSE


When the database opens successfully then perform the following:

alter trigger sys.GGS_DDL_TRIGGER_BEFORE disable ;

Then shutdown the database and remove the "_system_trig_enabled" parameter and restart it again.

Regards
support

Thanks for checking

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s