Troubleshooting Listener Startup or Connection Hang issues in 11g.

Since 11g Oracle resolves hostnames directly by querying DNS rather than from /etc/hosts, this can introduce delays in establishing a connection to the remote database server.

This could be observed in the strace for a SQL*Plus connection:-

open("/etc/nsswitch.conf", O_RDONLY|0x800, 0666) = 5
ioctl(5, TCGETA, 0x9fffffffffffa360) ERR#25 ENOTTY
read(5, "# \n# / e t c / n s s w i t c ".., 8192) = 92
read(5, 0x60000000001c9058, 8192) = 0
close(5) = 0
open("/usr/lib/hpux64/libnss_dns.so.1", O_RDONLY|0x800, 0) = 5
fstat(5, 0x9fffffffffff9de0) = 0
pread(5, "7fE L F 0202010101\0\0\0\0\0\0\0".., 1024, 0) = 1024
stat("/usr/lib/hpux64/dpd", 0x9fffffffffff9390) = 0
open("/usr/lib/hpux64/dpd/libnss_dns.so.1.bpd", O_RDONLY|0x800, 0) ERR#2 ENOENT
getuid() = 305 (305)
getgid() = 303 (303)
mmap(NULL, 49440, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_SHLIB, 5, 0) = 0xc00000000b054000
mmap(NULL, 800, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_SHLIB, 5, 65536) = 0x9fffffffbf68d000
close(5) = 0
getuid() = 305 (305)
getgid() = 303 (303)
getuid() = 305 (305)
getgid() = 303 (303)
open("/test/local/oracle/11.1.0.7/lib/libdl.so.1", O_RDONLY|0x800, 0) ERR#2 ENOENT
open("/test/local/oracle/11.1.0.7/lib32/libdl.so.1", O_RDONLY|0x800, 0) ERR#2 ENOENT
getuid() = 305 (305)
getgid() = 303 (303)
open("/usr/lib/hpux64/libdl.so.1", O_RDONLY|0x800, 0) = 5
fstat(5, 0x9fffffffffff9cf0) = 0
read(5, "7fE L F 0202010101\0\0\0\0\0\0\0".., 64) = 64
close(5) = 0
socket(AF_INET, SOCK_DGRAM, 0) = 5
ioctl(5, SIOCGIFNUM, 0x9fffffffffff9680) = 0
ioctl(5, SIOCGIFCONF, 0x9fffffffffff9690) = 0
socket(AF_INET6, SOCK_DGRAM, 0) = 6
ioctl(6, SIOCGLIFNUM, 0x9fffffffffff9684) = 0
ioctl(6, SIOCGLIFCONF, 0x9fffffffffff96a0) = 0
ioctl(5, SIOCGIFFLAGS, 0x9fffffffffff96b0) = 0
close(5) = 0
close(6) = 0
gettimeofday(0x9fffffffffff7dd0, NULL) = 0
getpid() = 22968 (22967)
open("/etc/resolv.conf", O_RDONLY|0x800, 0666) = 5
ioctl(5, TCGETA, 0x9fffffffffff7da0) ERR#25 ENOTTY
read(5, "d o m a i n t e s t . c o m \n".., 8192) = 453
read(5, 0x60000000001dddf8, 8192) = 0
close(5) = 0
...
...
...
socket(AF_INET, SOCK_STREAM, 0) = 5
connect(5, 0x60000000001eba50, 16) = 0
getsockname(5, 0x9fffffffffff9da0, 0x9fffffffffff94c0) = 0
getsockopt(5, SOL_SOCKET, SO_SNDBUF, 0x9fffffffffffa000, 0x9fffffffffffa004) = 0
getsockopt(5, SOL_SOCKET, SO_RCVBUF, 0x9fffffffffffa000, 0x9fffffffffffa004) = 0
setsockopt(5, 0x6, TCP_NODELAY, 0x9fffffffffffa00c, 4) = 0

As you can see from the above trace that after reading the nsswitch.conf,

open("/etc/nsswitch.conf", O_RDONLY|0x800, 0666) = 5

library “libnss_dns.so” is loaded,

open("/usr/lib/hpux64/libnss_dns.so.1", O_RDONLY|0x800, 0) = 5

and then /etc/resolv.conf is read and much later the socket for TCP/IP is opened.

open("/etc/resolv.conf", O_RDONLY|0x800, 0666) = 5

In 10g after nsswitch.conf and loading the library /etc/hosts file was read rather than /etc/resolv.conf. The reason is the change in system call, in 10g the system call was gethostbyname() whereas in 11g its getaddrinfo(). And these system calls require different configuration in nsswitch.conf, gethostbyname() require the use of keyword “hosts” while getaddrinfo() the use the keyword “ipnodes”

So after 11g if there is any delay with the DNS the same would be reflected in listener startup times and even with new connections. In order to resolve this we have to make some changes to nsswitch.conf.

Solution:-

Add below line in the /etc/nsswitch.conf:-

 ipnodes: files [NOTFOUND=continue] dns

Adding the above line would instruct calls made by getaddrinfo() to first search in /etc/hosts and then if not found, query the DNS server. Thus all the DNS related delays would be avoided.

Hope you like the articles in case of any doubts/queries please feel free to comment below.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s