impossible to resolve symbols in same binary loaded twice with dlmopen

impossible to resolve symbols in same binary loaded twice with dlmopen

[Mathieu Lacage]

[-- Attachment #1: Type: text/plain, Size: 1586 bytes --]

hi,

A couple of months ago, I reported on the gdb mailing-list this issue:
http://sourceware.org/ml/gdb/2010-02/msg00027.html (the original report
mentionned a crash which happened with a different program loader but
the root cause of the problem is the same as the one mentionned here)
and tom tromey gave hints on how it could be fixed. It took me a while
to eventually put together a patch and a reduced testcase for this issue
but here it is in the bugzilla database:
http://sourceware.org/bugzilla/show_bug.cgi?id=11766

For convenience, this email contains the reduced testcase and patch
attached. Here is the associated ChangeLog entry:

2010-XX-XX Mathieu Lacage <mathieu.lacage@inria.fr>

        Fix PR/gdb 11766: gdb does not resolve correctly symbols in
binaries loaded twice with dlmopen

        Because the name of an inferior binary present in the linkmap is
not sufficient for the test of uniqueness, of an entry in the list of
objfiles, we ensure uniqueness with an extra equality condition, the
addr_low member of the binary.

        * gdb/objfiles.h: add addr_low member variable
        * gdb/solib.c (solib_read_symbols): check for addr_low in
equality test for objfile, initialize addr_low if needed.


I really care about fixing this issue because it hits me and a lot of my
users and asking them to recompile gdb with a patch is not fun in the
long run. I am looking forward to fix this upstream so, please, let me
know what I can do to help with the resolution of this issue.

Mathieu
-- 
Mathieu Lacage <mathieu.lacage@sophia.inria.fr>
Tel: +33 4 9238 5056

[-- Attachment #2: test.tar.gz --]
[-- Type: application/x-compressed-tar, Size: 673 bytes --]

[-- Attachment #3: patch --]
[-- Type: text/plain, Size: 1774 bytes --]

? objdir
? patch
Index: gdb/objfiles.h
===================================================================
RCS file: /cvs/src/src/gdb/objfiles.h,v
retrieving revision 1.72
diff -c -p -r1.72 objfiles.h
*** gdb/objfiles.h	14 Apr 2010 17:26:11 -0000	1.72
--- gdb/objfiles.h	7 Jul 2010 18:35:38 -0000
*************** struct objfile
*** 193,198 ****
--- 193,200 ----
  
      char *name;
  
+     CORE_ADDR addr_low;
+ 
      /* Some flag bits for this objfile. */
  
      unsigned short flags;
Index: gdb/solib.c
===================================================================
RCS file: /cvs/src/src/gdb/solib.c,v
retrieving revision 1.140
diff -c -p -r1.140 solib.c
*** gdb/solib.c	16 May 2010 23:49:58 -0000	1.140
--- gdb/solib.c	7 Jul 2010 18:35:39 -0000
*************** solib_read_symbols (struct so_list *so, 
*** 634,644 ****
        TRY_CATCH (e, RETURN_MASK_ERROR)
  	{
  	  struct section_addr_info *sap;
- 
  	  /* Have we already loaded this shared object?  */
  	  ALL_OBJFILES (so->objfile)
  	    {
! 	      if (strcmp (so->objfile->name, so->so_name) == 0)
  		break;
  	    }
  	  if (so->objfile != NULL)
--- 634,644 ----
        TRY_CATCH (e, RETURN_MASK_ERROR)
  	{
  	  struct section_addr_info *sap;
  	  /* Have we already loaded this shared object?  */
  	  ALL_OBJFILES (so->objfile)
  	    {
! 	      if (strcmp (so->objfile->name, so->so_name) == 0
! 		  && so->objfile->addr_low == so->addr_low)
  		break;
  	    }
  	  if (so->objfile != NULL)
*************** solib_read_symbols (struct so_list *so, 
*** 648,653 ****
--- 648,654 ----
  							    so->sections_end);
  	  so->objfile = symbol_file_add_from_bfd (so->abfd,
  						  flags, sap, OBJF_SHARED);
+ 	  so->objfile->addr_low = so->addr_low;
  	  free_section_addr_info (sap);
  	}
  

Re: impossible to resolve symbols in same binary loaded twice with dlmopen

[Tom Tromey]

>>>>> "Mathieu" == Mathieu Lacage <mathieu.lacage@sophia.inria.fr> writes:

Mathieu> 2010-XX-XX Mathieu Lacage <mathieu.lacage@inria.fr>
Mathieu>         Fix PR/gdb 11766: gdb does not resolve correctly symbols in
Mathieu> binaries loaded twice with dlmopen

I think this is a reasonable idea.

First, I'm curious: does gdb detect dlmopen calls automatically?  I
thought there was some missing glibc feature here.  I haven't looked
into it too much since I wasn't aware of anybody really using dlmopen.
If gdb cannot do this, please file a bug report.

I am not completely sure that this patch is sufficient.

It doesn't seem like it would handle PIE properly.  I think you would
have to modify objfile_relocate as well.

I was wondering if there is a possible bug with a program dlmopen()ing
itself, but after thinking about it my guess is no.

Otherwise it seems reasonable to me.


Once you have this patch, does it really work?  It seems like it would
work ok for some things, like backtraces, but not other things.  E.g.,
does symbol lookup work properly in the dlmopen case?  I would imagine
that it does or does not depending on the ordering of objfiles in gdb's
internal list.

Tom

Re: impossible to resolve symbols in same binary loaded twice with dlmopen

[Mathieu Lacage]

On Thu, Jul 8, 2010 at 10:31 PM, Tom Tromey <tromey@redhat.com> wrote:

> First, I'm curious: does gdb detect dlmopen calls automatically?  I

Not with the libc implementation of dlmopen.

> thought there was some missing glibc feature here.  I haven't looked

The proper way to handle multiple namespaces in the dynamic loader
would be to use an implementation of the librtld_db.so solaris library
and modify gdb to use it. However, glibc does not provide such an
implementation to encapsulate the details of the loader data
structures.

> into it too much since I wasn't aware of anybody really using dlmopen.
> If gdb cannot do this, please file a bug report.

I do not believe that gdb can do this and I could file a bug but it's
probably going to be hard to fix (beyond my own abilities).

> I am not completely sure that this patch is sufficient.

It's sufficient for me because I use a dynamic loader that has a
different implementation of dlmopen:
  i. there is no compile-time bound on the number of namespaces which
can be created (i.e., the glibc has a 16 or 32 static upper bound)
  ii. there is extra API to be able to create a namespace
independently from loading a binary and extra API to specify some
redirections (say, map libc.so.6 to libc-ns3.so): it's an easy-to-use
API compared to using an audit library
  iii. there is extra API to shutdown a namespace without having to
invoke the destructors of the binaries loaded in it (important to
allow the main process to be shielded from the namespaces)
  iv. the linkmap exported to the debugger contains the binaries of
_all_ the binaries loaded in all namespaces. The loader is careful to
still resolve symbols correctly within each namespace but the linkmap
exported to the debugger contains all binaries to allow gdb to find
all binaries and put breakpoints in them without having to know about
the namespaces.

However, as a disclaimer, the patch I posted and the associated bug
report have been tested on a libc-based system.

> It doesn't seem like it would handle PIE properly.  I think you would
> have to modify objfile_relocate as well.

I am not sure I follow you. I have been using this code with PIEs
(with my other loader, arguably) and have been able to debug stuff.
What symptom do you have in mind ?

> I was wondering if there is a possible bug with a program dlmopen()ing
> itself, but after thinking about it my guess is no.

I don't see any problem with this but I did not test it. Will do so.

> Once you have this patch, does it really work?  It seems like it would
> work ok for some things, like backtraces, but not other things.  E.g.,
> does symbol lookup work properly in the dlmopen case?  I would imagine
> that it does or does not depending on the ordering of objfiles in gdb's
> internal list.

If you use the libc loader and run the test program I attached to the
bug report, you will see that it's indeed impossible to put a
breakpoint or print the address of a function located in a binary
loaded with dlmopen (LD_ID_NEWLM) because gdb is not notified of the
existence of the binaries loaded with dlmopen (and if you inspect the
inferior's linkmap through the _r_debug variable by hand as show
below, you can see that it really does not contain them). It works
fine with my loader though (for the reasons I outlined above).

I would be happy to do more work if needed to satisfy the needs of gdb
on top of the libc dlmopen (even though I am not going to use it): do
you have a specific idea of how you would like this to be done ?

Mathieu

[sample gdb session]
Breakpoint 1, main (argc=2, argv=0x7fffffffe158) at test.c:14
14	  bool doA = true;
(gdb) n
15	  if (strcmp (argv[1], "a") == 0)
(gdb)
17	      doA = true;
(gdb)
23	  void *a = dlmopen (LM_ID_NEWLM, "./libtest.so", RTLD_LAZY);
(gdb)
24	  void *b = dlmopen (LM_ID_NEWLM, "./libtest.so", RTLD_LAZY);
(gdb)
26	  if (doA)
(gdb) p _r_debug.r_map
$6 = (struct link_map_public *) 0x37fa61f0e8
(gdb) p *_r_debug.r_map
$7 = {l_addr = 0, l_name = 0x37fa419614 "", l_ld = 0x600948, l_next =
0x37fa61f678, l_prev = 0x0}
(gdb) p *_r_debug.r_map->l_next
$8 = {l_addr = 140737363566592, l_name = 0x37fa419614 "", l_ld =
0x7ffff7ffe580,
  l_next = 0x7ffff7ffd658, l_prev = 0x37fa61f0e8}
(gdb) p *_r_debug.r_map->l_next->l_next
$9 = {l_addr = 0, l_name = 0x7ffff7ffd640 "/lib64/libdl.so.2", l_ld =
0x37fb202da0,
  l_next = 0x7ffff7ffdb20, l_prev = 0x37fa61f678}
(gdb) p *_r_debug.r_map->l_next->l_next->l_next
$10 = {l_addr = 0, l_name = 0x7ffff7ffdb08 "/lib64/libc.so.6", l_ld =
0x37fab72b40,
  l_next = 0x37fa61e970, l_prev = 0x7ffff7ffd658}
(gdb) p *_r_debug.r_map->l_next->l_next->l_next->l_next
$11 = {l_addr = 0, l_name = 0x400200 "/lib64/ld-linux-x86-64.so.2",
l_ld = 0x37fa61ddf0, l_next = 0x0,
  l_prev = 0x7ffff7ffdb20}

-- 
Mathieu Lacage <mathieu.lacage@gmail.com>

Re: impossible to resolve symbols in same binary loaded twice with dlmopen

[Mathieu Lacage]

On Fri, Jul 9, 2010 at 1:36 PM, Mathieu Lacage <mathieu.lacage@gmail.com> wrote:

> It's sufficient for me because I use a dynamic loader that has a
> different implementation of dlmopen:

It occurred to me that I should have included a link to the relevant
code so that the above does not read as a reference to some sekret
proprietary piece of code: it's open source crap :)

http://code.nsnam.org/mathieu/elf-loader/
It's not fast or super portable but it runs on x86_64/ia32, various
linux distribution versions, and is binary compatible with the libc
loader.

Mathieu
-- 
Mathieu Lacage <mathieu.lacage@gmail.com>

Re: impossible to resolve symbols in same binary loaded twice with dlmopen

[Tom Tromey]

>>>>> "Mathieu" == Mathieu Lacage <mathieu.lacage@gmail.com> writes:

Tom> into it too much since I wasn't aware of anybody really using dlmopen.
Tom> If gdb cannot do this, please file a bug report.

Mathieu> I do not believe that gdb can do this and I could file a bug but it's
Mathieu> probably going to be hard to fix (beyond my own abilities).

Yeah, that is ok -- please file it anyway.

Tom> Once you have this patch, does it really work?  It seems like it would
Tom> work ok for some things, like backtraces, but not other things.  E.g.,
Tom> does symbol lookup work properly in the dlmopen case?  I would imagine
Tom> that it does or does not depending on the ordering of objfiles in gdb's
Tom> internal list.

Mathieu> If you use the libc loader and run the test program I attached to the
Mathieu> bug report, you will see that it's indeed impossible to put a
Mathieu> breakpoint or print the address of a function located in a binary
Mathieu> loaded with dlmopen [...]

Actually I am curious about failures even with your loader.

Can you set breakpoints on all instances of a function?  Does printing a
global variable defined in a dlmopen()d .so always work properly?  I
would guess that you could construct a case where it does not, something
like:

File x.c defines a function, file y.c defines a global.  Compile into a
shared library.  Have the main program dlmopen the library twice and
call the function in each one.  Then, step into each instance of the
function and print the address of the global.

The reason I think this will fail is that I am not sure that gdb will do
the symbol search properly.  I suspect it will always find the first
instance of the global, not the one in the current objfile.

Maybe I'm wrong about this, though :-)

Anyway, AFAICT, your patch won't break anything, and it is a step in the
right direction.  So, please check it in.

Tom

Re: impossible to resolve symbols in same binary loaded twice with dlmopen

[Mathieu Lacage]

On Tue, Jul 20, 2010 at 9:38 PM, Tom Tromey <tromey@redhat.com> wrote:

> Mathieu> I do not believe that gdb can do this and I could file a bug but it's
> Mathieu> probably going to be hard to fix (beyond my own abilities).
>
> Yeah, that is ok -- please file it anyway.

http://sourceware.org/bugzilla/show_bug.cgi?id=11839

> Actually I am curious about failures even with your loader.
>
> Can you set breakpoints on all instances of a function?  Does printing a
> global variable defined in a dlmopen()d .so always work properly?  I
> would guess that you could construct a case where it does not, something
> like:

ahhh. Good point. It works nicely for _static_ variables, but not for
global variables (which I never use).

> File x.c defines a function, file y.c defines a global.  Compile into a
> shared library.  Have the main program dlmopen the library twice and
> call the function in each one.  Then, step into each instance of the
> function and print the address of the global.

(gdb) b main
Breakpoint 1 at 0x400602: file test.c, line 14.
(gdb) r
Starting program: /home/mathieu/code/elf-loader/t
Breakpoint 1, main (argc=1, argv=0x7fffffffe128) at test.c:14
14	  void *a = dlmopen (LM_ID_NEWLM, "./libtest.so", RTLD_LAZY);
(gdb) n
15	  void *b = dlmopen (LM_ID_NEWLM, "./libtest.so", RTLD_LAZY);
(gdb)
17	  call (a, "foo");
(gdb) info sharedlibrary
From                To                  Syms Read   Shared Object Library
0x00007ffff7de8a50  0x00007ffff7df824f  Yes         ./ldso
0x00007ffff7be6ad0  0x00007ffff7be6cef  Yes         ./libvdl.so
0x00000037fa81e860  0x00000037fa9262fc  Yes (*)     /lib64/libc.so.6
0x00007ffff79e52e4  0x00007ffff79e52fa  Yes         ././libtest.so
0x00007ffff77e42e4  0x00007ffff77e42fa  Yes         ././libtest.so
(*): Shared library is missing debugging information.
(gdb) b foo
Breakpoint 2 at 0x7ffff79e52e8: file libtest.c, line 5. (2 locations)
(gdb) c
Continuing.
Breakpoint 2, foo () at libtest.c:5
5	  return 0;
(gdb) p &g_global
$4 = (int *) 0x7ffff7be543c
(gdb) p &g_static
$5 = (int *) 0x7ffff7be5438
(gdb) c
Continuing.
Breakpoint 2, foo () at libtest.c:5
5	  return 0;
(gdb) p &g_global
$6 = (int *) 0x7ffff7be543c
(gdb) p &g_static
$7 = (int *) 0x7ffff79e4438
(gdb)


> The reason I think this will fail is that I am not sure that gdb will do
> the symbol search properly.  I suspect it will always find the first
> instance of the global, not the one in the current objfile.
>
> Maybe I'm wrong about this, though :-)

Indeed. Shall I file a bug ? Maybe you can also give me a hint on
where I can fix it ? It might be a nice change from writing the
thesis.

> Anyway, AFAICT, your patch won't break anything, and it is a step in the
> right direction.  So, please check it in.

I don't have commit rights, and, really, it's better for you if I don't :)

Mathieu
-- 
Mathieu Lacage <mathieu.lacage@gmail.com>

Re: impossible to resolve symbols in same binary loaded twice with dlmopen

[Mathieu Lacage]

On Sun, Jul 25, 2010 at 14:24, Mathieu Lacage <mathieu.lacage@gmail.com> wrote:

>> Anyway, AFAICT, your patch won't break anything, and it is a step in the
>> right direction.  So, please check it in.
>
> I don't have commit rights, and, really, it's better for you if I don't :)

Would someone commit this patch for me ? It's been a couple of months now...

Mathieu
-- 
Mathieu Lacage <mathieu.lacage@gmail.com>

Re: impossible to resolve symbols in same binary loaded twice with dlmopen

[Thiago Jung Bauermann]

Hi,

On Thu, 2011-01-20 at 13:05 +0100, Mathieu Lacage wrote:
> On Sun, Jul 25, 2010 at 14:24, Mathieu Lacage <mathieu.lacage@gmail.com> wrote:
> 
> >> Anyway, AFAICT, your patch won't break anything, and it is a step in the
> >> right direction.  So, please check it in.
> >
> > I don't have commit rights, and, really, it's better for you if I don't :)
> 
> Would someone commit this patch for me ? It's been a couple of months now...

I'm not a maintainer, but given that the patch was already approved, I
committed the following.
-- 
[]'s
Thiago Jung Bauermann
IBM Linux Technology Center


2011-01-25  Mathieu Lacage  <mathieu.lacage@inria.fr>

	PR/symtab 11766:
	* gdb/objfiles.h (struct objfile) <addr_low>: New field.
	* gdb/solib.c (solib_read_symbols): Check for addr_low in
	equality test for objfile, initialize addr_low if needed.


diff --git a/gdb/objfiles.h b/gdb/objfiles.h
index c44517f..759c2f9 100644
--- a/gdb/objfiles.h
+++ b/gdb/objfiles.h
@@ -194,6 +194,8 @@ struct objfile
 
     char *name;
 
+    CORE_ADDR addr_low;
+
     /* Some flag bits for this objfile.  */
 
     unsigned short flags;
diff --git a/gdb/solib.c b/gdb/solib.c
index 909a23b..6748d87 100644
--- a/gdb/solib.c
+++ b/gdb/solib.c
@@ -638,7 +638,8 @@ solib_read_symbols (struct so_list *so, int flags)
 	  /* Have we already loaded this shared object?  */
 	  ALL_OBJFILES (so->objfile)
 	    {
-	      if (strcmp (so->objfile->name, so->so_name) == 0)
+	      if (strcmp (so->objfile->name, so->so_name) == 0
+		  && so->objfile->addr_low == so->addr_low)
 		break;
 	    }
 	  if (so->objfile != NULL)
@@ -648,6 +649,7 @@ solib_read_symbols (struct so_list *so, int flags)
 							    so->sections_end);
 	  so->objfile = symbol_file_add_from_bfd (so->abfd,
 						  flags, sap, OBJF_SHARED);
+	  so->objfile->addr_low = so->addr_low;
 	  free_section_addr_info (sap);
 	}
 

Re: impossible to resolve symbols in same binary loaded twice with dlmopen

[Tom Tromey]

>>>>> "Thiago" == Thiago Jung Bauermann <bauerman@br.ibm.com> writes:

Thiago> I'm not a maintainer, but given that the patch was already approved, I
Thiago> committed the following.

Thanks for doing this.  I originally forgot to apply it :-(
Also, thanks Mathieu for the reminders.

Tom

Re: impossible to resolve symbols in same binary loaded twice with dlmopen

[Thiago Jung Bauermann]

On Fri, 2011-01-28 at 03:42 -0700, Tom Tromey wrote:
> >>>>> "Thiago" == Thiago Jung Bauermann <bauerman@br.ibm.com> writes:
> 
> Thiago> I'm not a maintainer, but given that the patch was already approved, I
> Thiago> committed the following.
> 
> Thanks for doing this.  I originally forgot to apply it :-(
> Also, thanks Mathieu for the reminders.

BTW, even though I mentioned the PR number on the commit message,
nothing showed up on the bugzilla:

http://sourceware.org/ml/gdb-cvs/2011-01/msg00175.html
http://sourceware.org/bugzilla/show_bug.cgi?id=11766

Did I do something wrong?
-- 
[]'s
Thiago Jung Bauermann
IBM Linux Technology Center

Re: impossible to resolve symbols in same binary loaded twice with dlmopen

[Tom Tromey]

Thiago> BTW, even though I mentioned the PR number on the commit message,
Thiago> nothing showed up on the bugzilla:

Thiago> http://sourceware.org/ml/gdb-cvs/2011-01/msg00175.html
Thiago> http://sourceware.org/bugzilla/show_bug.cgi?id=11766

Thiago> Did I do something wrong?

Yeah, the parser is rather strict, and you had a syntax error.
You wrote:

	PR/symtab 11766:

but the actual syntax is:

	PR symtab/11766:

Tom

impossible to resolve symbols in same binary loaded twice with dlmopen

[Mathieu Lacage]

[-- Attachment #1: Type: text/plain, Size: 1559 bytes --]

hi,

A couple of months ago, I reported on the gdb mailing-list this issue:
http://sourceware.org/ml/gdb/2010-02/msg00027.html (the original
report mentionned a crash which happened with a different program
loader but the root cause of the problem is the same as the one
mentionned here) and tom tromey gave hints on how it could be fixed.
It took me a while to eventually put together a patch and a reduced
testcase for this issue but here it is in the bugzilla database:
http://sourceware.org/bugzilla/show_bug.cgi?id=11766

For convenience, this email contains the reduced testcase and patch
attached. Here is the associated ChangeLog entry:

2010-XX-XX Mathieu Lacage <mathieu.lacage@inria.fr>

        Fix PR/gdb 11766: gdb does not resolve correctly symbols in
binaries loaded twice with dlmopen

        Because the name of an inferior binary present in the linkmap
is not sufficient for the test of uniqueness, of an entry in the list
of objfiles, we ensure uniqueness with an extra equality condition,
the addr_low member of the binary.

        * gdb/objfiles.h: add addr_low member variable
        * gdb/solib.c (solib_read_symbols): check for addr_low in
equality test for objfile, initialize addr_low if needed.


I really care about fixing this issue because it hits me and a lot of
my users and asking them to recompile gdb with a patch is not fun in
the long run. I am looking forward to fix this upstream so, please,
let me know what I can do to help with the resolution of this issue.

Mathieu
-- 
Mathieu Lacage <mathieu.lacage@gmail.com>

[-- Attachment #2: test.tar.gz --]
[-- Type: application/x-gzip, Size: 673 bytes --]

[-- Attachment #3: gdb.patch --]
[-- Type: application/octet-stream, Size: 1774 bytes --]

? objdir
? patch
Index: gdb/objfiles.h
===================================================================
RCS file: /cvs/src/src/gdb/objfiles.h,v
retrieving revision 1.72
diff -c -p -r1.72 objfiles.h
*** gdb/objfiles.h	14 Apr 2010 17:26:11 -0000	1.72
--- gdb/objfiles.h	7 Jul 2010 18:35:38 -0000
*************** struct objfile
*** 193,198 ****
--- 193,200 ----
  
      char *name;
  
+     CORE_ADDR addr_low;
+ 
      /* Some flag bits for this objfile. */
  
      unsigned short flags;
Index: gdb/solib.c
===================================================================
RCS file: /cvs/src/src/gdb/solib.c,v
retrieving revision 1.140
diff -c -p -r1.140 solib.c
*** gdb/solib.c	16 May 2010 23:49:58 -0000	1.140
--- gdb/solib.c	7 Jul 2010 18:35:39 -0000
*************** solib_read_symbols (struct so_list *so, 
*** 634,644 ****
        TRY_CATCH (e, RETURN_MASK_ERROR)
  	{
  	  struct section_addr_info *sap;
- 
  	  /* Have we already loaded this shared object?  */
  	  ALL_OBJFILES (so->objfile)
  	    {
! 	      if (strcmp (so->objfile->name, so->so_name) == 0)
  		break;
  	    }
  	  if (so->objfile != NULL)
--- 634,644 ----
        TRY_CATCH (e, RETURN_MASK_ERROR)
  	{
  	  struct section_addr_info *sap;
  	  /* Have we already loaded this shared object?  */
  	  ALL_OBJFILES (so->objfile)
  	    {
! 	      if (strcmp (so->objfile->name, so->so_name) == 0
! 		  && so->objfile->addr_low == so->addr_low)
  		break;
  	    }
  	  if (so->objfile != NULL)
*************** solib_read_symbols (struct so_list *so, 
*** 648,653 ****
--- 648,654 ----
  							    so->sections_end);
  	  so->objfile = symbol_file_add_from_bfd (so->abfd,
  						  flags, sap, OBJF_SHARED);
+ 	  so->objfile->addr_low = so->addr_low;
  	  free_section_addr_info (sap);
  	}