SHUTDOWN: waiting for active calls to complete处理。遇到一个latch free的问题，kill掉了当时的session，报session的sid不存在

遇到一个latch free的问题，kill掉了当时的session，报session的sid不存在，并在os层也kill了一次，但是v$session中看到的session的状态依然是active的！

很无语的问题，，但是因为是测试环境。所以觉得停库。

Shutting down instance (immediate)
License high water mark = 705
Thu May 30 22:03:39 2013
Active call for process 20102 user 'otfmsstg' program 'Oracle@z4ah8020'
Active call for process 20272 user 'otfmsstg' program 'oracle@z4ah8020'
Active call for process 20387 user 'otfmsstg' program 'oracle@z4ah8020'
SHUTDOWN: waiting for active calls to complete.

一直卡在这个地方。

support上的10393.6，其内容如下：

Checked for relevance on 11-Dec-2008

Problem Description:
====================

You are attempting to shut down the database and the database hangs. The
alert log contains the following message:


SHUTDOWN: Waiting for active calls to complete


There are no other error messages in the alert log.

Solution Description:
=====================

--------------------------------------------------------
Caveat: This note is intended for Unix platforms primarily, but we should
address this issue on Windows platforms as well. On Windows, stopping the
service is a valid workaround for these errors.
--------------------------------------------------------

Locate and kill any client connections to the database at the Unix level, as
follows:

1. Locate any client connections to the database using ps, and grep for any
processes belonging to this.

Example: ps -ef | grep V733

2. Look for processes that include a 'Local=No' designation.

Example: osupport 6235 1 0 Nov 24 0:01 oracleV733 (LOCAL=NO)

3. Kill the Unix process(es) with the 'Local=No' designation.

Example: Kill -9 6235


Explanation:
============

The database is waiting for pmon to clean up processes, but pmon is unable to
clean them. The client connections to the server are causing the shutdown
immediate or normal to hang. Killing them allows pmon to clean up and release
the associated Oracle processes and resources.

What resources are we talking about?

1) Any non committed transactions must be rolled back
2) Any temporary space (sort segments / lobs / session temporary tables) must be freed
3) The session itself and any associated memory consumed by the session.
4) Internal locks / enqueues must be cleaned up

Often Oracle (SMON or PMON depending on whether Shared Server is used) will wait for the OS to terminate the process(es) associated with the session. I the OS never returns, or fails to terminate them, then the instance shutdown will hang with this message (Shutdown Waiting for Active Calls to Complete)

Other means exist to achieve a quick shutdown, as outlined inNote 3808.1- What Is The Fastest Way To Cleanly Shutdown An Oracle Database?

Note that in E-Business Environments, the same messages can be produced if the database shutdown process starts before the 原来是有些session无法被pmon进程清理，需要手动kill掉。

z4ah8020:fmsstag > ps -ef|grep otfmsstg|grep LOCAL=NO
otfmsstg 27651 1 0 Mar 17 ? 0:03 oraclefmsstag (LOCAL=NO)
otfmsstg 27523 1 0 Mar 17 ? 0:02 oraclefmsstag (LOCAL=NO)
otfmsstg 27603 1 0 Mar 17 ? 0:02 oraclefmsstag (LOCAL=NO)
otfmsstg 6931 1 0 21:35:06 ? 0:00 oraclefmsstag (LOCAL=NO)
otfmsstg 17581 1 0 Mar 18 ? 55:52 oraclefmsstag (LOCAL=NO)
otfmsstg 17553 1 0 Mar 18 ? 1:38 oraclefmsstag (LOCAL=NO)

kill -9 27651等进程后

Thu May 30 22:10:33 2013
ALTER DATABASE CLOSE NORMAL
Thu May 30 22:10:35 2013
SMON: disabling tx recovery
SMON: disabling cache recovery
Thu May 30 22:10:44 2013
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Thread 1 closed at log sequence 19859
Successful close of redo thread 1
Thu May 30 22:10:45 2013
Completed: ALTER DATABASE CLOSE NORMAL
Thu May 30 22:10:45 2013
ALTER DATABASE DISMOUNT
Completed: ALTER DATABASE DISMOUNT
ARCH: Archiving is disabled
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archiving is disabled
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Thu May 30 22:11:19 2013

数据库正常关闭了，重启后latch free消失。

latch free的问题后续研究一下，有经验的老大些多多指导。

谢谢