One of my customers has Oracle 11.2.0.4 on windows
2008 R2 (x64). After reboot of this database server, listener did not come up
automatically and even manually starting the listener was returning following
error at the command prompt.
###################
Starting tnslsnr: please wait...
Failed to start service, error 0.
TNS-12560: TNS:protocol adapter error
Failed to start service, error 0.
TNS-12560: TNS:protocol adapter error
###################
Listener.log file was showing following messages
related to the failure of listener.
###################
###################
Tue Nov 25 19:29:04
2014
Message 279 not found; No message file for product=NETWORK, facility=NL
Message 509 not found; No message file for product=NETWORK, facility=NL
Message 510 not found; No message file for product=NETWORK, facility=NL
Tue Nov 25 19:31:22 2014
Message 279 not found; No message file for product=NETWORK, facility=NL
Message 509 not found; No message file for product=NETWORK, facility=NL
Message 510 not found; No message file for product=NETWORK, facility=NL
Message 279 not found; No message file for product=NETWORK, facility=NL
Message 509 not found; No message file for product=NETWORK, facility=NL
Message 510 not found; No message file for product=NETWORK, facility=NL
Tue Nov 25 19:31:22 2014
Message 279 not found; No message file for product=NETWORK, facility=NL
Message 509 not found; No message file for product=NETWORK, facility=NL
Message 510 not found; No message file for product=NETWORK, facility=NL
###################
Same server also had the application running on it
and this application was using an Oracle client installation in a different
home. While listener was down, TNSPING executed from the client home was
showing error "ORA-12541: TNS:no
listener " which was clearly showing that listener is not running.
Reason of Issue and Solution
Since production system was down, it was really a panicky situation for my customer and as a DBA, it was also a challenge to find out the root cause and resolve the solution. While "googling" it, I saw a lot of different discussions and also some MOS notes and almost all of them were related to UNIX based systems. On these systems, the reason of the problem were because of 1)Wrongly set permissions on “lsnrctl” executable. 2) Wrongly set ORACLE_HOME and PATH environment variables settings. 3) Wrongly set TNS_ADMIN environment variable.
Since production system was down, it was really a panicky situation for my customer and as a DBA, it was also a challenge to find out the root cause and resolve the solution. While "googling" it, I saw a lot of different discussions and also some MOS notes and almost all of them were related to UNIX based systems. On these systems, the reason of the problem were because of 1)Wrongly set permissions on “lsnrctl” executable. 2) Wrongly set ORACLE_HOME and PATH environment variables settings. 3) Wrongly set TNS_ADMIN environment variable.
I also made sure that
all of these environment variables are set properly and made sure that local
windows Administrator has all the permissions, but still my problem was not
fixed. While thinking of “permissions”, I just thought to check if any recent
windows updates were installed on the system and this revealed that system is
set to download and install Windows updates automatically and last time some
windows security updates were installed 7 days ago and this was first system
reboot after the installation of those security updates. So this thing clicked
me that this might be the cause of this issue.
Immediately I went to
uninstall these recently installed security updates and then rebooted the
server. After the reboot, I could just shout “Yahoooooooo” because listener
was up and running automatically.
Moral of the story
This is really dangerous to enable automatic updates download/install on Windows based systems, I always recommend to disable automatic download and installation of the Windows updates because no one knows how would it impact the system after the installation and a reboot. System administrators for Windows based Oracle servers should make sure that all updates are tested on a testing server before these could be manually downloaded and installed on the production database server.
This is really dangerous to enable automatic updates download/install on Windows based systems, I always recommend to disable automatic download and installation of the Windows updates because no one knows how would it impact the system after the installation and a reboot. System administrators for Windows based Oracle servers should make sure that all updates are tested on a testing server before these could be manually downloaded and installed on the production database server.
There should also be a
contingency plan so that these updates could be rolled back in case any issue
happens to the listener/database or any other Oracle executable after the
updates installation.
No comments:
Post a Comment