好久没做过安装和打PSU的活了，心里还有点小期待，本以为2个小时就可以结束的事情，结果轮到真正的打的时候，打了4个多小时才弄完，遇到了一些小问题，我这次打的环境是11.2.0.3.9，这里做个记录。废话就不说了，我们来先看下日志： root@crmdb1rac1:/#/orac

好久没做过安装和打PSU的活了，心里还有点小期待，本以为2个小时就可以结束的事情，结果轮到真正的打的时候，打了4个多小时才弄完，遇到了一些小问题，我这次打的环境是11.2.0.3.9，这里做个记录。废话就不说了，我们来先看下日志：

root@crmdb1rac1:/#/oracle/app/11.2.0/grid/OPatch/opatch auto /oraclelog/opatch_soft -ocmrf /oracle/app/grid/ocm0709.rsp
Executing /oracle/app/11.2.0/grid/perl/bin/perl /oracle/app/11.2.0/grid/OPatch/crs/patch11203.pl -patchdir /oraclelog -patchn opatch_soft -ocmrf /oracle/app/grid/ocm0709.rsp -paramfile /oracle/app/11.2.0/grid/crs/install/crsconfig_params
/oracle/app/11.2.0/grid/crs/install/crsconfig_params
/oracle/app/11.2.0/grid/crs/install/s_crsconfig_defs
This is the main log file: /oracle/app/11.2.0/grid/cfgtoollogs/opatchauto2014-07-10_00-18-14.log
This file will show your detected configuration and all the steps that opatchauto attempted to do on your system: /oracle/app/11.2.0/grid/cfgtoollogs/opatchauto2014-07-10_00-18-14.report.log
2014-07-10 00:18:14: Starting Clusterware Patch Setup
Using configuration parameter file: /oracle/app/11.2.0/grid/crs/install/crsconfig_params
patch /oraclelog/opatch_soft/17592127/custom/server/17592127 apply successful for home /oracle/app/oracle/product/11.2.0/db 
 patch /oraclelog/opatch_soft/18031683 apply successful for home /oracle/app/oracle/product/11.2.0/db 
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'crmdb1rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'crmdb1rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'crmdb1rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'crmdb1rac1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'crmdb1rac1'
CRS-2673: Attempting to stop 'ora.cvu' on 'crmdb1rac1'
CRS-2673: Attempting to stop 'ora.crmdb1rac1.vip' on 'crmdb1rac1'
CRS-2677: Stop of 'ora.cvu' on 'crmdb1rac1' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'crmdb1rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'crmdb1rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'crmdb1rac1'
CRS-2676: Start of 'ora.cvu' on 'crmdb1rac2' succeeded
CRS-2677: Stop of 'ora.crmdb1rac1.vip' on 'crmdb1rac1' succeeded
CRS-2672: Attempting to start 'ora.crmdb1rac1.vip' on 'crmdb1rac2'
CRS-2677: Stop of 'ora.scan1.vip' on 'crmdb1rac1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'crmdb1rac2'
CRS-2676: Start of 'ora.crmdb1rac1.vip' on 'crmdb1rac2' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'crmdb1rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'crmdb1rac2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'crmdb1rac2' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'crmdb1rac1' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'crmdb1rac2'
CRS-2676: Start of 'ora.oc4j' on 'crmdb1rac2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'crmdb1rac1'
CRS-2677: Stop of 'ora.ons' on 'crmdb1rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'crmdb1rac1'
CRS-2677: Stop of 'ora.net1.network' on 'crmdb1rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'crmdb1rac1' has completed
CRS-2677: Stop of 'ora.crsd' on 'crmdb1rac1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'crmdb1rac1'
CRS-2673: Attempting to stop 'ora.crf' on 'crmdb1rac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'crmdb1rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'crmdb1rac1'
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'crmdb1rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'crmdb1rac1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'crmdb1rac1' succeeded
CRS-2677: Stop of 'ora.crf' on 'crmdb1rac1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'crmdb1rac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'crmdb1rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'crmdb1rac1'
CRS-2677: Stop of 'ora.cssd' on 'crmdb1rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'crmdb1rac1'
CRS-2677: Stop of 'ora.gipcd' on 'crmdb1rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'crmdb1rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'crmdb1rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'crmdb1rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /oracle/app/11.2.0/grid
patch /oraclelog/opatch_soft/17592127 apply failed for home /oracle/app/11.2.0/grid
CRS-4123: Oracle High Availability Services has been started.

在打PSU进行到Successfully unlock /oracle/app/11.2.0/grid这一步的时候。这个时候屏幕的输出会非常非常的慢，我们可以观察后台的日志输出，我们发现在后面进行一个copy操作的时候出现了问题。如下所示：

 Patching component oracle.usm, 11.2.0.3.0...
 The following actions have failed:
 Copy failed from '/oracle/app/11.2.0/grid/.patch_storage/166198_Jun_26_2013_21_46_53/files/lib/libeons.so' to '/oracle/app/11.2.0/grid/lib/libeons.so'... 
 Copy failed from '/oracle/app/11.2.0/grid/.patch_storage/166198_Jun_26_2013_21_46_53/files/lib/libhasgen11.so' to '/oracle/app/11.2.0/grid/lib/libhasgen11.so'... 
 Copy failed from '/oracle/app/11.2.0/grid/.patch_storage/166198_Jun_26_2013_21_46_53/files/lib/libocr11.so' to '/oracle/app/11.2.0/grid/lib/libocr11.so'... 
 Copy failed from '/oracle/app/11.2.0/grid/.patch_storage/166198_Jun_26_2013_21_46_53/files/lib/libocrb11.so' to '/oracle/app/11.2.0/grid/lib/libocrb11.so'... 
 Copy failed from '/oracle/app/11.2.0/grid/.patch_storage/166198_Jun_26_2013_21_46_53/files/lib/libocrutl11.so' to '/oracle/app/11.2.0/grid/lib/libocrutl11.so'... 

我们可以看到在copy这些文件的时候是无法成功copy的，这是因为我们的Oracle的这些so文件在内存中被占用着，我们需要执行slibclean去清理这些缓存。于是我做了回退的操作，然后把crs完整的停下来，做了几次slibclean的操作，在正常的启动crs。做完这些操作后，我发现打PSU仍然在这一步失败了。于是我查了一些资料。我把这几个coyp失败的.so的文件做了一个备份，移动到了其他的目录里面，然后crs就无法成功启动了。crs无法启动还怎么打PSU呢？于是我又把文件拷贝回去，我仔细观察了下，在打PSU的时候，会自动把crs停掉，然后就会运行我们刚刚所说的Successfully unlock /oracle/app/11.2.0/grid。那我能不能在停掉CRS这个阶段执行slibclean去清理这些缓存呢？

这是在CRS还在启动状态下占用的lib文件:

root@crmdb1rac1:/oracle/app/11.2.0/grid/.patch_storage#genkld |grep oracle
 900000004d98000 b2c7 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libnio.a
 900000004d77000 203b1 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libnet.a
 900000004d5f000 17840 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libzip.a
 900000004d31000 21700 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9ute23.so
 900000004d1e000 12c7c /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libiverel23.so
 900000004cb6000 6792f /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libjclscar_23.so
 900000004c91000 2427d /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9vrb23.so
 900000004c5e000 32c3f /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9jvmti23.so
 900000004c2f000 2ed2c /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9dyn23.so
 900000004b97000 97a31 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9gc23.so
 900000004b7f000 179e1 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9bcv23.so
 9000000046cf000 4af269 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9jit23.so
 9000000046be000 10eef /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9trc23.so
 900000004681000 2ac84 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9dmp23.so
 9000000046ac000 11bab /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9zlib23.so
 9000000049000 37b9e /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libjava.a
 90000000091a000 1d41 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libwrappers.a
 900000004588000 6f0fe /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9vm23.so
 9000000045f8000 50434 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9prt23.so
 900000000918000 1d2f /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9hookable23.so
 90000000456c000 1b631 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/j9vm/libjvm.so
 900000004059000 bf6a /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9thr23.so
 900000000916000 1ff3 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libjsig.so
 900000004049000 f6aa /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/classic/libjvm.so
 90000000ff8e000 248bf1b /oracle/app/11.2.0/grid/lib/libclntsh.so
 900000004513000 5852a /oracle/app/11.2.0/grid/lib/libcell11.so
 90000000434a000 1b3f27 /oracle/app/11.2.0/grid/lib/libdb-4.6.so
 90000000428a000 bf3a1 /oracle/app/11.2.0/grid/lib/libcrf11.so
 9000000041d7000 ad7d /oracle/app/11.2.0/grid/lib/libeonsserver.so
 9000000041b1000 15c4e /oracle/app/11.2.0/grid/lib/libonsx.so
 900000004019000 2509a /oracle/app/11.2.0/grid/lib/libeons.so
 90000000123b000 2dd6039 /oracle/app/11.2.0/grid/lib/libttsh11.so
 900000004012000 6c16 /oracle/app/11.2.0/grid/lib/libskgxn2.so
 90000000406c000 1449e1 /oracle/app/11.2.0/grid/lib/libgns11.so
 90000000122e000 c5c5 /oracle/app/11.2.0/grid/lib/libocrutl11.so
 90000000103e000 1ef18b /oracle/app/11.2.0/grid/lib/libocrb11.so
 900000000f9f000 9e0a5 /oracle/app/11.2.0/grid/lib/libocr11.so
 900000000921000 67dc0c /oracle/app/11.2.0/grid/lib/libhasgen11.so
 90000000f7a9000 2adb17 /usr/openv/lib/libpioracle.so

这是在CRS关闭后，我运行了好几次slibclean之后，占用的lib文件.

 root@crmdb1rac1:/usr/sbin#genkld |grep oracle
 900000001160000 b2c7 /oracle/app/11.2.0/grid/jdk/jre/bin/libnio.a
 90000000113f000 203b1 /oracle/app/11.2.0/grid/jdk/jre/bin/libnet.a
 900000001127000 17840 /oracle/app/11.2.0/grid/jdk/jre/bin/libzip.a
 9000000010f9000 21700 /oracle/app/11.2.0/grid/jdk/jre/bin/libj9ute23.so
 9000000010e6000 12c7c /oracle/app/11.2.0/grid/jdk/jre/bin/libiverel23.so
 90000000107e000 6792f /oracle/app/11.2.0/grid/jdk/jre/bin/libjclscar_23.so
 900000001059000 2427d /oracle/app/11.2.0/grid/jdk/jre/bin/libj9vrb23.so
 900000001026000 32c3f /oracle/app/11.2.0/grid/jdk/jre/bin/libj9jvmti23.so
 900000000ff7000 2ed2c /oracle/app/11.2.0/grid/jdk/jre/bin/libj9dyn23.so
 900000000f5f000 97a31 /oracle/app/11.2.0/grid/jdk/jre/bin/libj9gc23.so
 900000000a97000 4af269 /oracle/app/11.2.0/grid/jdk/jre/bin/libj9jit23.so
 900000000a86000 10eef /oracle/app/11.2.0/grid/jdk/jre/bin/libj9trc23.so
 900000000a49000 2ac84 /oracle/app/11.2.0/grid/jdk/jre/bin/libj9dmp23.so
 900000000a74000 11bab /oracle/app/11.2.0/grid/jdk/jre/bin/libj9zlib23.so
 900000000a0f000 37b9e /oracle/app/11.2.0/grid/jdk/jre/bin/libjava.a
 900000000a47000 1d41 /oracle/app/11.2.0/grid/jdk/jre/bin/libwrappers.a
 90000000094e000 6f0fe /oracle/app/11.2.0/grid/jdk/jre/bin/libj9vm23.so
 9000000009be000 50434 /oracle/app/11.2.0/grid/jdk/jre/bin/libj9prt23.so
 900000000432000 1d2f /oracle/app/11.2.0/grid/jdk/jre/bin/libj9hookable23.so
 900000000926000 1b631 /oracle/app/11.2.0/grid/jdk/jre/bin/j9vm/libjvm.so
 900000000942000 bf6a /oracle/app/11.2.0/grid/jdk/jre/bin/libj9thr23.so
 900000000430000 1ff3 /oracle/app/11.2.0/grid/jdk/jre/bin/libjsig.so
 900000000916000 f6aa /oracle/app/11.2.0/grid/jdk/jre/bin/classic/libjvm.so
 90000000f7a9000 2adb17 /usr/openv/lib/libpioracle.so	

可以看到基本上全是和JDK相关的一些lib文件，没有我们需要copy的这些lib文件了。做完这些操作之后，PSU补丁才安装成功。不知道从何时开始起，非要折腾几下才能把事情做完做好。

原文地址：记一次打PSU遇到的Copy failed的问题, 感谢原作者分享。