public inbox for gcc-bugs@sourceware.org
help / color / mirror / Atom feed
* [Bug go/63731] New: Fallback to netgo does not work
@ 2014-11-04  8:08 yohei at jp dot ibm.com
  2014-11-14  4:15 ` [Bug go/63731] " yohei at jp dot ibm.com
                   ` (31 more replies)
  0 siblings, 32 replies; 33+ messages in thread
From: yohei at jp dot ibm.com @ 2014-11-04  8:08 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

            Bug ID: 63731
           Summary: Fallback to netgo does not work
           Product: gcc
           Version: 5.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: go
          Assignee: ian at airs dot com
          Reporter: yohei at jp dot ibm.com
                CC: cmang at google dot com

When the DNS resolver with CGO fails for statically linked binaries, it should
fall back to the pure-go DNS resolver, if I understand correctly.

This fallback mechanism does work at least on Linux for x86_64 and ppc64le.

The variable ok in lookupIP in go/net/lookup_unix.go seems always set to true,
so the fallback mechanism would never be called.

Here is an example code to demonstrate the problem.

$ gccgo -v
Using built-in specs.
COLLECT_GCC=gccgo
COLLECT_LTO_WRAPPER=/usr/local/gccgo-216834/libexec/gcc/x86_64-unknown-linux-gnu/5.0.0/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with: ../src/configure --enable-threads=posix --enable-shared
--enable-__cxa_atexit --enable-languages=c,c++,go --enable-secureplt
--enable-checking=yes --with-long-double-128 --enable-decimal-float
--disable-bootstrap --disable-alsa --disable-multilib
--prefix=/usr/local/gccgo-216834
Thread model: posix
gcc version 5.0.0 20141029 (experimental) (GCC) 

$ cat lookup.go 
package main

import (
    "fmt"
    "net"
)

func main() {
    addrs, err := net.LookupHost("gcc.gnu.org")
    if err != nil {
        fmt.Println(err)
    } else {
        for i := 0; i < len(addrs); i++ {
            fmt.Println(addrs[i])
        }
    }
}

$ gccgo lookup.go 
$ ./a.out 
209.132.180.131
$ gccgo -static lookup.go 
/usr/local/gccgo-216834/lib/gcc/x86_64-unknown-linux-gnu/5.0.0/../../../../lib64/libgo.a(net.o):
In function `net.cgoLookupPort':
/home/yohei/gccgo.216834/bld/x86_64-unknown-linux-gnu/libgo/../../../src/libgo/go/net/cgo_unix.go:83:
warning: Using 'getaddrinfo' in statically linked applications requires at
runtime the shared libraries from the glibc version used for linking
$ ./a.out 
lookup gcc.gnu.org: Name or service not known
$ LD_LIBRARY_PATH=/lib/x86_64-linux-gnu ./a.out 
209.132.180.131


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
@ 2014-11-14  4:15 ` yohei at jp dot ibm.com
  2014-11-14  4:49 ` yohei at jp dot ibm.com
                   ` (30 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: yohei at jp dot ibm.com @ 2014-11-14  4:15 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #1 from Yohei Ueda <yohei at jp dot ibm.com> ---
Created attachment 33966
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=33966&action=edit
Fix the DNS lookup problem for statically linked binaries

I created a patch file that fixes this problem.


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
  2014-11-14  4:15 ` [Bug go/63731] " yohei at jp dot ibm.com
@ 2014-11-14  4:49 ` yohei at jp dot ibm.com
  2014-11-14 15:54 ` ian at airs dot com
                   ` (29 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: yohei at jp dot ibm.com @ 2014-11-14  4:49 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #2 from Yohei Ueda <yohei at jp dot ibm.com> ---
I tested this issue with the latest GC trunk again, and noticed that GC always
compiles programs that contains DNS lookups with dynamic linking even if
-static is specified.

$ go version
go version devel +ae495517bd72 Fri Nov 14 11:43:01 2014 +1100 linux/amd64
$ go build -ldflags '-extldflags "-static"' lookup.go
$ ./lookup 
209.132.180.131
$ file lookup
lookup: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked
(uses shared libs), not stripped
$ ldd lookup
    linux-vdso.so.1 =>  (0x00007fff81550000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f50b943d000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f50b9078000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f50b9677000)

Any suggestions? Should I report this issue to GC?


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
  2014-11-14  4:15 ` [Bug go/63731] " yohei at jp dot ibm.com
  2014-11-14  4:49 ` yohei at jp dot ibm.com
@ 2014-11-14 15:54 ` ian at airs dot com
  2014-11-17  8:16 ` yohei at jp dot ibm.com
                   ` (28 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: ian at airs dot com @ 2014-11-14 15:54 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #3 from Ian Lance Taylor <ian at airs dot com> ---
The gc linker does not use the external linker merely because you link against
the net package.  You need to also pass -linkmode external in ldflags.


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (2 preceding siblings ...)
  2014-11-14 15:54 ` ian at airs dot com
@ 2014-11-17  8:16 ` yohei at jp dot ibm.com
  2014-11-17 15:44 ` ian at airs dot com
                   ` (27 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: yohei at jp dot ibm.com @ 2014-11-17  8:16 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #4 from Yohei Ueda <yohei at jp dot ibm.com> ---
Thank you for the correction.

I need to use the following instructions to enable netgo with GC.

$ go version
go version devel +ae495517bd72 Fri Nov 14 11:43:01 2014 +1100 linux/amd64
$ go build -ldflags '-linkmode external -extldflags -static' -a -tags netgo
lookup.go 
$ ./lookup 
209.132.180.131

Now, I have a question about how to do the same thing with GCCGO.

I think the -a option for the go command is essential to enable netgo, but the
-a option seems not working with the standard GO library of GCCGO.

$ go build -compiler gccgo -gccgoflags '-static' -a -tags netgo lookup.go 
# command-line-arguments
/usr/local/gccgo-216834/lib/gcc/x86_64-unknown-linux-gnu/5.0.0/../../../../lib64/libgo.a(net.o):
In function `net.cgoLookupPort':
/home/yohei/gccgo.216834/bld/x86_64-unknown-linux-gnu/libgo/../../../src/libgo/go/net/cgo_unix.go:83:
warning: Using 'getaddrinfo' in statically linked applications requires at
runtime the shared libraries from the glibc version used for linking
$ ./lookup 
lookup gcc.gnu.org: Name or service not known
$ LD_LIBRARY_PATH=/lib/x86_64-linux-gnu ./lookup
209.132.180.131

I confirmed that the source code exists under $GOROOT/src where $GOROOT is the
value reported by go env GOROOT. I also put libgo/go in the GCC source code
into $(PREFIX)/src/pkg.


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (3 preceding siblings ...)
  2014-11-17  8:16 ` yohei at jp dot ibm.com
@ 2014-11-17 15:44 ` ian at airs dot com
  2014-11-17 18:27 ` boger at us dot ibm.com
                   ` (26 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: ian at airs dot com @ 2014-11-17 15:44 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #5 from Ian Lance Taylor <ian at airs dot com> ---
You are correct.  The go command does not rebuild the standard library for
gccgo.  There is at present no reasonable way to use the netgo tag with gccgo.


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (4 preceding siblings ...)
  2014-11-17 15:44 ` ian at airs dot com
@ 2014-11-17 18:27 ` boger at us dot ibm.com
  2014-11-18  0:24 ` ian at airs dot com
                   ` (25 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: boger at us dot ibm.com @ 2014-11-17 18:27 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #6 from boger at us dot ibm.com ---
I understand why some functions like getaddrinfo don't work with static linking
unless the LD_LIBRARY_PATH is set to find libc.so, but then what should happen
if it isn't?


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (5 preceding siblings ...)
  2014-11-17 18:27 ` boger at us dot ibm.com
@ 2014-11-18  0:24 ` ian at airs dot com
  2014-11-18  6:15 ` yohei at jp dot ibm.com
                   ` (24 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: ian at airs dot com @ 2014-11-18  0:24 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #7 from Ian Lance Taylor <ian at airs dot com> ---
That's really a question for the glibc maintainers.  It's entirely a glibc
issue.

My understanding is that it's not actually libc.so that needs to be found. 
Glibc implements /etc/nsswitch.conf by loading a shared library for each entry
listed there ("db", "files", etc.).  That is a very flexible mechanism that
allows nsswitch.conf to be extended on a system-specific basis.  However, it
only works if those shared libraries are available.  It is those libraries (and
libdl.so) that must be found at runtime.  If the libraries can not be found,
/etc/nsswitch.conf can not be implemented, and name lookups of various sorts
fail by returning "not found."


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (6 preceding siblings ...)
  2014-11-18  0:24 ` ian at airs dot com
@ 2014-11-18  6:15 ` yohei at jp dot ibm.com
  2014-11-18 13:16 ` boger at us dot ibm.com
                   ` (23 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: yohei at jp dot ibm.com @ 2014-11-18  6:15 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #8 from Yohei Ueda <yohei at jp dot ibm.com> ---
If we can distinguish between the "not found" errors due to invalid host names
and no nss, we can fall back to netgo only when nss is not available.

I noticed that errno is set to 0 for invalid hostnames, and is set to ENOENT
when nss is not available. However, this behavior is not documented, so it is
probably implementation dependent.

I think a practical solution to this issue is to enable the -a option of the go
command for the standard library in GCCGO. I know Lynn is working on the go
command for GCCGO.
https://groups.google.com/forum/#!topic/gofrontend-dev/1Gm87ieI47Q

What do you think?


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (7 preceding siblings ...)
  2014-11-18  6:15 ` yohei at jp dot ibm.com
@ 2014-11-18 13:16 ` boger at us dot ibm.com
  2014-11-18 13:52 ` yohei at jp dot ibm.com
                   ` (22 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: boger at us dot ibm.com @ 2014-11-18 13:16 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #9 from boger at us dot ibm.com ---
My question was:  what is supposed to happen on fallback?  Sounds like some
code gets rebuilt and used instead?


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (8 preceding siblings ...)
  2014-11-18 13:16 ` boger at us dot ibm.com
@ 2014-11-18 13:52 ` yohei at jp dot ibm.com
  2014-11-19 20:32 ` boger at us dot ibm.com
                   ` (21 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: yohei at jp dot ibm.com @ 2014-11-18 13:52 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #10 from Yohei Ueda <yohei at jp dot ibm.com> ---
https://code.google.com/p/go/source/browse/src/net/lookup_unix.go#63

func lookupIP(host string) (addrs []IP, err error) {
        addrs, err, ok := cgoLookupIP(host)
        if !ok {
                addrs, err = goLookupIP(host)
        }
        return
}

This code shows how fallback works. If cgoLookup returns ok == false, then
pure-Go goLookupIP is called.

When the netgo tag is set, ok is always false.
https://code.google.com/p/go/source/browse/src/net/cgo_stub.go#5

When the netgo tag is not set, ok is always true.
https://code.google.com/p/go/source/browse/src/net/cgo_unix.go#147
https://code.google.com/p/go/source/browse/src/net/cgo_unix.go#84

If cgoLookupIP can also return ok == false when no nss is available, goLookupIP
will be called. This is my first idea.
https://code.google.com/p/go/source/browse/src/net/cgo_unix.go#116

However, I realized that we cannot easily distinguish the "not found" errors.
getaddrinfo returns "not found" in both cases of invalid hostname lookups and
static binaries.

If cgoLookup returns ok == false even when an invalid host name is looked up
with nss available, goLookupIP also looks it up again, so IP lookup occurs
twice, and both fails with "not found" error. I don't think this is desired
behavior.


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (9 preceding siblings ...)
  2014-11-18 13:52 ` yohei at jp dot ibm.com
@ 2014-11-19 20:32 ` boger at us dot ibm.com
  2014-11-20 13:25 ` boger at us dot ibm.com
                   ` (20 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: boger at us dot ibm.com @ 2014-11-19 20:32 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #11 from boger at us dot ibm.com ---
What I was asking is:  what does it mean to call the pure-Go goLookupIP?


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (10 preceding siblings ...)
  2014-11-19 20:32 ` boger at us dot ibm.com
@ 2014-11-20 13:25 ` boger at us dot ibm.com
  2014-11-20 14:49 ` yohei at jp dot ibm.com
                   ` (19 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: boger at us dot ibm.com @ 2014-11-20 13:25 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #13 from boger at us dot ibm.com ---
Then my question is: why isn't this "fallback" code always built for GO and
available to run instead of waiting until it hits this situation and then built
and run?  In the situation you are describing it sounds like it is related to
whether or not there is a static libnss available which could be determined at
GO build time.


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (11 preceding siblings ...)
  2014-11-20 13:25 ` boger at us dot ibm.com
@ 2014-11-20 14:49 ` yohei at jp dot ibm.com
  2014-11-20 20:02 ` boger at us dot ibm.com
                   ` (18 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: yohei at jp dot ibm.com @ 2014-11-20 14:49 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #14 from Yohei Ueda <yohei at jp dot ibm.com> ---
I am not the original author, so my description may be inaccurate.

> why isn't this "fallback" code always built for GO and available
> to run instead of waiting until it hits this situation and then 
> built and run?

The fallback code (goLookupIP) is always built, but is called at run time only
when the code is built with the netgo tag.

When the netgo tag is not set at build time, both cgoLookupIP and goLookupIP
are always built. However, only cgoLookupIP is called, and goLookupIP is never
called at run time. So, lookup fails when it is statically linked.

When the netgo tag is set at build time, cgoLookupIP is empty and goLookupIP is
built. So only goLookupIP is called at run time. 

lookupIP looks like it has a run time fallback mechanism, but it is not true.
The current code only selects cgoLookupIP or goLookIP at build time depending
on the netgo tag setting.

If we could distinguish "not found" errors of getaddrinfo as described earlier,
lookupIP could have a run time fallback mechanism from cgoLookIP to goLookupIP.

However, it is impossible to distinguish "not found" errors as far as I know,
so I guess the current code depends on the netgo tag to select cgoLookupIP or
goLookupIP at build time.

> In the situation you are describing it sounds like it is related to 
> whether or not there is a static libnss available which could be determined 
> at GO build time.

The existence of libnss at build time does not affect the behavior at run time.
The netgo tag affects which lookup function is called at runtime.


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (12 preceding siblings ...)
  2014-11-20 14:49 ` yohei at jp dot ibm.com
@ 2014-11-20 20:02 ` boger at us dot ibm.com
  2014-11-21  4:26 ` ian at airs dot com
                   ` (17 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: boger at us dot ibm.com @ 2014-11-20 20:02 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #15 from boger at us dot ibm.com ---
I think what Ian is saying is that mechanism to rebuild packages in this way
doesn't work with gccgo (and probably never should?)

Now I'm finally understanding this.  Originally with gc the net package is
built with netgo off, but the netgo tag says to rebuild the GO standard library
with the netgo tag set on and then build the program.

Yohei's original fix used the code that was built with netgo off but allowed
the go resolver to be called if the call to getaddrinfo failed.  There are
probably a small set of errnos that could be checked to determine if the go
resolver should be called after getaddrinfo failed, but is that important?  I
would expect the errnos are consistent across platforms but don't know for
sure.

To me it seems like Yohei's fix is probably OK for gccgo with some additional
checks for errno if needed, but would change existing behavior for gc and
because of that should not change there?


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (13 preceding siblings ...)
  2014-11-20 20:02 ` boger at us dot ibm.com
@ 2014-11-21  4:26 ` ian at airs dot com
  2014-11-21 15:12 ` boger at us dot ibm.com
                   ` (16 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: ian at airs dot com @ 2014-11-21  4:26 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

Ian Lance Taylor <ian at airs dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2014-11-21
     Ever confirmed|0                           |1

--- Comment #16 from Ian Lance Taylor <ian at airs dot com> ---
Gccgo and GCC act the same on glibc systems: if you choose static linking, DNS
lookups only work if the dynamic libraries are available.

The only difference between Go and C here is that the Go library happens to
have code that will work for some people some of the time.  I don't want the
library to automatically fall back to the Go code in all cases, because in some
cases it will turn a possibly-explicable failure into a completely inexplicable
failure, and in a few cases it will turn a correct failure into an incorrect
success.

It's OK with me to make the go tool's -a option work with gccgo.  It will only
work if people have the library sources available.  If we do that, this issue
will be fixed.  I don't know how hard that would be--it might be easy.

I don't know of another good way to fix this.


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (14 preceding siblings ...)
  2014-11-21  4:26 ` ian at airs dot com
@ 2014-11-21 15:12 ` boger at us dot ibm.com
  2014-11-21 17:10 ` ian at airs dot com
                   ` (15 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: boger at us dot ibm.com @ 2014-11-21 15:12 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #17 from boger at us dot ibm.com ---
Can you clarify how using -a -tags netgo actually works.  I know it requires
that the source be available, but it must mean that it rebuilds the package for
the current link only, throws it away after using it and next time it has to
rebuild it?

Couldn't some of your concerns about unexpected failures be resolved by
providing better error information when there were failures, or provide other
ways to indicate that the go resolver shouldn't be called (like an environment
variable)?  It just seems that rebuilding the package every time is a heavy
hammer to resolve this problem.

I think if someone wants to have their GO program use libnss if it is present
and then the go resolver if not, that should be an option and currently it is
not.


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (15 preceding siblings ...)
  2014-11-21 15:12 ` boger at us dot ibm.com
@ 2014-11-21 17:10 ` ian at airs dot com
  2014-11-25 15:35 ` boger at us dot ibm.com
                   ` (14 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: ian at airs dot com @ 2014-11-21 17:10 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #18 from Ian Lance Taylor <ian at airs dot com> ---
The -a option to "go build" means to rebuild all packages rather than using the
installed versions (see http://golang.org/cmd/go for documentation).  The
"-tags netgo" option means to build with the build tag netgo (see the build
constraints section in http://golang.org/pkg/go/build/).  So, yes, it rebuilds
the packages for the current link only.  This is more reasonable with the gc
compiler than with gccgo, since the gc compiler is so fast.

I'm OK in principle with coming up with some other approach to direct the Go
library to use the Go DNS lookup rather than calling getaddrinfo.  I don't
think that can be the default.  I don't think we want a program to
unpredictably sometimes use one and sometimes use the other.  I don't think an
environment variable would work well, since Yohei presumably wants the
statically linked binary to work this way by default.  Unfortunately all I can
think of would be adding another function to the net package directing lookups
to use pure DNS; this is unfortunate because the net package already has the
most complex and confusing API of all the standard Go packages.


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (16 preceding siblings ...)
  2014-11-21 17:10 ` ian at airs dot com
@ 2014-11-25 15:35 ` boger at us dot ibm.com
  2014-12-03 13:03 ` yohei at jp dot ibm.com
                   ` (13 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: boger at us dot ibm.com @ 2014-11-25 15:35 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #19 from boger at us dot ibm.com ---
(In reply to Ian Lance Taylor from comment #18)
> The -a option to "go build" means to rebuild all packages rather than using
> the installed versions (see http://golang.org/cmd/go for documentation). 
> The "-tags netgo" option means to build with the build tag netgo (see the
> build constraints section in http://golang.org/pkg/go/build/).  So, yes, it
> rebuilds the packages for the current link only.  This is more reasonable
> with the gc compiler than with gccgo, since the gc compiler is so fast.
> 

Most of the examples in the documentation show that the built packages are put
into the same directories as the source.  I assume that for an official release
with a binary distribution, that is not the way it works.  That's how it would
have to work with gccgo.  In that case everyone must share the same copy of the
source but then if build options are used that would cause packages to be
rebuilt, they must go somewhere that is only used for the curent build.  And I
don't understand what 'go install' would mean in that case.  The 'go install'
command documentation has very little information on where built packages are
stored or if there are cases when 'go install' can't be used.

> I'm OK in principle with coming up with some other approach to direct the Go
> library to use the Go DNS lookup rather than calling getaddrinfo.  I don't
> think that can be the default.  I don't think we want a program to
> unpredictably sometimes use one and sometimes use the other.  I don't think
> an environment variable would work well, since Yohei presumably wants the
> statically linked binary to work this way by default.  Unfortunately all I
> can think of would be adding another function to the net package directing
> lookups to use pure DNS; this is unfortunate because the net package already
> has the most complex and confusing API of all the standard Go packages.

I think providing another function that called the pure GO resolver would be
best.  Then the GO programmer can decide how to handle it if the first call
failed.


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (17 preceding siblings ...)
  2014-11-25 15:35 ` boger at us dot ibm.com
@ 2014-12-03 13:03 ` yohei at jp dot ibm.com
  2014-12-03 16:10 ` boger at us dot ibm.com
                   ` (12 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: yohei at jp dot ibm.com @ 2014-12-03 13:03 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #20 from Yohei Ueda <yohei at jp dot ibm.com> ---
I noticed a Docker issue saying GC 1.4 does not rebuild the standard library
with -a.

https://github.com/docker/docker/issues/9449 

I think the problem is now not limited to GCCGO.


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (18 preceding siblings ...)
  2014-12-03 13:03 ` yohei at jp dot ibm.com
@ 2014-12-03 16:10 ` boger at us dot ibm.com
  2014-12-03 18:39 ` boger at us dot ibm.com
                   ` (11 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: boger at us dot ibm.com @ 2014-12-03 16:10 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #21 from boger at us dot ibm.com ---
I'm confused by the description of -a in the go1.4 documentation.

I asked about this before and the answer was that each invocation of 'go build'
would create a copy of the built package which was then used for the current 
build but then thrown away.  But that must not be the way it works?


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (19 preceding siblings ...)
  2014-12-03 16:10 ` boger at us dot ibm.com
@ 2014-12-03 18:39 ` boger at us dot ibm.com
  2014-12-03 19:17 ` ian at airs dot com
                   ` (10 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: boger at us dot ibm.com @ 2014-12-03 18:39 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #23 from boger at us dot ibm.com ---
If I look at this documentation:  http://tip.golang.org/doc/go1.4#gocmd

It says this:

The behavior of the go build subcommand's -a flag has been changed for
non-development installations. For installations running a released
distribution, the -a flag will no longer rebuild the standard library and
commands, to avoid overwriting the installation's files. 

When I read this it sounds like the previous behavior with the -a option was to
rebuild the packages and put the newly built packages into the installed
directories, including the standard library.  If everyone who used 'go build'
with -a generated their own copy of the built packages and then threw them
away, how would the installation's files ever get overwritten?


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (20 preceding siblings ...)
  2014-12-03 18:39 ` boger at us dot ibm.com
@ 2014-12-03 19:17 ` ian at airs dot com
  2015-02-28 10:16 ` e29253 at jp dot ibm.com
                   ` (9 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: ian at airs dot com @ 2014-12-03 19:17 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #24 from Ian Lance Taylor <ian at airs dot com> ---
They would not have been overwritten, unless you used "go install -a".  That
line in the doc may be misleading.


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (21 preceding siblings ...)
  2014-12-03 19:17 ` ian at airs dot com
@ 2015-02-28 10:16 ` e29253 at jp dot ibm.com
  2015-02-28 21:01 ` ian at airs dot com
                   ` (8 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: e29253 at jp dot ibm.com @ 2015-02-28 10:16 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #25 from Tatsushi Inagaki <e29253 at jp dot ibm.com> ---
What is the most recommended way when we want to use the net package in a
statically linked binary? My impression is that a statically linked binary also
should call dlopen() (and thus we should export LD_LIBRARY_PATH), if the
corresponding dynamically linked binary do so to resolve DNS.
https://sourceware.org/glibc/wiki/FAQ#Even_statically_linked_programs_need_some_shared_libraries_which_is_not_acceptable_for_me.__What_can_I_do.3F

Or, can we expect that netgo can be enabled with 'go build -a' again in Go 1.5?
https://github.com/golang/go/issues/9369


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (22 preceding siblings ...)
  2015-02-28 10:16 ` e29253 at jp dot ibm.com
@ 2015-02-28 21:01 ` ian at airs dot com
  2015-03-01 23:40 ` ian at airs dot com
                   ` (7 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: ian at airs dot com @ 2015-02-28 21:01 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #26 from Ian Lance Taylor <ian at airs dot com> ---
Tatsushi: are you asking about gccgo, or about gc?


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (23 preceding siblings ...)
  2015-02-28 21:01 ` ian at airs dot com
@ 2015-03-01 23:40 ` ian at airs dot com
  2015-03-02 21:58 ` boger at us dot ibm.com
                   ` (6 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: ian at airs dot com @ 2015-03-01 23:40 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #28 from Ian Lance Taylor <ian at airs dot com> ---
Currently there is no reasonable way to use the Go DNS resolver when using
gccgo.  Any program that uses the net package will call glibc for DNS
resolution, meaning that you are limited to what glibc will do, which, as you
say, means calling dlopen.

go build -a does not work with gccgo.  The problem is that gccgo uses its own
copy of the Go library sources and they can not be built with go build -a.  It
would be nice to fix this but it is not at all a priority, since most people
will use the installed libgo.


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (24 preceding siblings ...)
  2015-03-01 23:40 ` ian at airs dot com
@ 2015-03-02 21:58 ` boger at us dot ibm.com
  2015-03-02 23:34 ` ian at airs dot com
                   ` (5 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: boger at us dot ibm.com @ 2015-03-02 21:58 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #29 from boger at us dot ibm.com ---
Yohei noted in comment 20 that this is also broken with gc in 1.4 when using
static linking.  That was a while ago -- is that no longer a problem?


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (25 preceding siblings ...)
  2015-03-02 21:58 ` boger at us dot ibm.com
@ 2015-03-02 23:34 ` ian at airs dot com
  2015-03-31 16:06 ` boger at us dot ibm.com
                   ` (4 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: ian at airs dot com @ 2015-03-02 23:34 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #30 from Ian Lance Taylor <ian at airs dot com> ---
The problem mentioned in comment #20 has nothing to do with gccgo.  To get
around that problem, use the -installsuffix option.  See
http://golang.org/issue/9344 .  Note that the docker issue mentioned in comment
#20 has been closed.


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (26 preceding siblings ...)
  2015-03-02 23:34 ` ian at airs dot com
@ 2015-03-31 16:06 ` boger at us dot ibm.com
  2015-04-01 14:11 ` boger at us dot ibm.com
                   ` (3 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: boger at us dot ibm.com @ 2015-03-31 16:06 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #32 from boger at us dot ibm.com ---
I have a prototype working for #2.  I am assuming #1 would not be accepted.

This fix consists of building a library called libnetgo.a which consists of the
net files that would be built if the netgo tag was used.  This new library was
installed into the same directory as libgo.a.

Once this library has been built and installed in the correct location, I was
able to get this to work by explicitly linking in this lib:

go build -gccgoflags '-static -lnetgo' lookup.go

I will attach a patch after some more testing.


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (27 preceding siblings ...)
  2015-03-31 16:06 ` boger at us dot ibm.com
@ 2015-04-01 14:11 ` boger at us dot ibm.com
  2015-04-07 18:10 ` ian at gcc dot gnu.org
                   ` (2 subsequent siblings)
  31 siblings, 0 replies; 33+ messages in thread
From: boger at us dot ibm.com @ 2015-04-01 14:11 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #34 from boger at us dot ibm.com ---
Created a codereview:  https://codereview.appspot.com/217620043


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (28 preceding siblings ...)
  2015-04-01 14:11 ` boger at us dot ibm.com
@ 2015-04-07 18:10 ` ian at gcc dot gnu.org
  2015-04-07 18:10 ` ian at airs dot com
  2015-04-08 14:41 ` boger at us dot ibm.com
  31 siblings, 0 replies; 33+ messages in thread
From: ian at gcc dot gnu.org @ 2015-04-07 18:10 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #35 from ian at gcc dot gnu.org <ian at gcc dot gnu.org> ---
Author: ian
Date: Tue Apr  7 18:09:28 2015
New Revision: 221906

URL: https://gcc.gnu.org/viewcvs?rev=221906&root=gcc&view=rev
Log:
    PR go/63731
libgo: Build and install libnetgo.a

libnetgo.a provides the net
package built with the netgo
tag enabled.  This provides the
netgo fallback solution for gccgo.
This lib must be explicitly linked
in using the -gccgoflags, so is
not included by default.

Modified:
    trunk/libgo/Makefile.am
    trunk/libgo/Makefile.in


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (29 preceding siblings ...)
  2015-04-07 18:10 ` ian at gcc dot gnu.org
@ 2015-04-07 18:10 ` ian at airs dot com
  2015-04-08 14:41 ` boger at us dot ibm.com
  31 siblings, 0 replies; 33+ messages in thread
From: ian at airs dot com @ 2015-04-07 18:10 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #36 from Ian Lance Taylor <ian at airs dot com> ---
Lynn added a new facility.  Some notes on docs:

As far as documentation, I tried to find some documentation on build tags in
general and netgo specifically because it seems like this should be documented
there, but did not find much.  Here are some ideas:
- add something to the 'go help build' output about the use of the netgo tag in
general and how it would be used to work around the static linking 
warning/problem against libnss and the workarounds for gc and gccgo
- if there is something somewhere else that describes the netgo tag and when to
use it, then add something about how to achieve the same effect in gccgo
- seems like it would be good to have documentation describing the differences
in using gccgo vs. gc and this would be included in that, however that is a
bigger work item and would require more thought on what to include in such a
document.


^ permalink raw reply	[flat|nested] 33+ messages in thread

* [Bug go/63731] Fallback to netgo does not work
  2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
                   ` (30 preceding siblings ...)
  2015-04-07 18:10 ` ian at airs dot com
@ 2015-04-08 14:41 ` boger at us dot ibm.com
  31 siblings, 0 replies; 33+ messages in thread
From: boger at us dot ibm.com @ 2015-04-08 14:41 UTC (permalink / raw)
  To: gcc-bugs

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63731

--- Comment #37 from boger at us dot ibm.com ---
Created attachment 35260
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=35260&action=edit
libgo/go/go/build/doc.go documentation update

Adding comments about the use of the netgo tag and the equivalent method for
use with gccgo.


^ permalink raw reply	[flat|nested] 33+ messages in thread

end of thread, other threads:[~2015-04-08 14:41 UTC | newest]

Thread overview: 33+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-11-04  8:08 [Bug go/63731] New: Fallback to netgo does not work yohei at jp dot ibm.com
2014-11-14  4:15 ` [Bug go/63731] " yohei at jp dot ibm.com
2014-11-14  4:49 ` yohei at jp dot ibm.com
2014-11-14 15:54 ` ian at airs dot com
2014-11-17  8:16 ` yohei at jp dot ibm.com
2014-11-17 15:44 ` ian at airs dot com
2014-11-17 18:27 ` boger at us dot ibm.com
2014-11-18  0:24 ` ian at airs dot com
2014-11-18  6:15 ` yohei at jp dot ibm.com
2014-11-18 13:16 ` boger at us dot ibm.com
2014-11-18 13:52 ` yohei at jp dot ibm.com
2014-11-19 20:32 ` boger at us dot ibm.com
2014-11-20 13:25 ` boger at us dot ibm.com
2014-11-20 14:49 ` yohei at jp dot ibm.com
2014-11-20 20:02 ` boger at us dot ibm.com
2014-11-21  4:26 ` ian at airs dot com
2014-11-21 15:12 ` boger at us dot ibm.com
2014-11-21 17:10 ` ian at airs dot com
2014-11-25 15:35 ` boger at us dot ibm.com
2014-12-03 13:03 ` yohei at jp dot ibm.com
2014-12-03 16:10 ` boger at us dot ibm.com
2014-12-03 18:39 ` boger at us dot ibm.com
2014-12-03 19:17 ` ian at airs dot com
2015-02-28 10:16 ` e29253 at jp dot ibm.com
2015-02-28 21:01 ` ian at airs dot com
2015-03-01 23:40 ` ian at airs dot com
2015-03-02 21:58 ` boger at us dot ibm.com
2015-03-02 23:34 ` ian at airs dot com
2015-03-31 16:06 ` boger at us dot ibm.com
2015-04-01 14:11 ` boger at us dot ibm.com
2015-04-07 18:10 ` ian at gcc dot gnu.org
2015-04-07 18:10 ` ian at airs dot com
2015-04-08 14:41 ` boger at us dot ibm.com

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).