[PATCH, committed] jit: on OS X, add -Wl,-undefined,dynamic_lookup to driver invocation

[PATCH, committed] jit: on OS X, add -Wl,-undefined,dynamic_lookup to driver invocation

[David Malcolm]

Reported as fixing the library on OS X Yosemite:
  https://gcc.gnu.org/ml/jit/2015-q3/msg00147.html

Committed to trunk as r226882.
Committed to gcc-5-branch as r226883.

gcc/jit/ChangeLog:
	* jit-playback.c (invoke_driver): On OS X, add
	"-Wl,-undefined,dynamic_lookup" to the driver arguments.
---
 gcc/jit/jit-playback.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/gcc/jit/jit-playback.c b/gcc/jit/jit-playback.c
index 925fa86..01cfd4b 100644
--- a/gcc/jit/jit-playback.c
+++ b/gcc/jit/jit-playback.c
@@ -2416,6 +2416,15 @@ invoke_driver (const char *ctxt_progname,
      time.  */
   ADD_ARG ("-fno-use-linker-plugin");
 
+#if defined (DARWIN_X86) || defined (DARWIN_PPC)
+  /* OS X's linker defaults to treating undefined symbols as errors.
+     If the context has any imported functions or globals they will be
+     undefined until the .so is dynamically-linked into the process.
+     Ensure that the driver passes in "-undefined dynamic_lookup" to the
+     linker.  */
+  ADD_ARG ("-Wl,-undefined,dynamic_lookup");
+#endif
+
   /* pex argv arrays are NULL-terminated.  */
   argvec.safe_push (NULL);
 
-- 
1.8.5.3

Re: libgccjit 5.2 on Mac OSX Yosemite

[Dibyendu Majumdar]

[-- Attachment #1: Type: text/plain, Size: 1014 bytes --]

Dave,

I was able to get the JIT compiler working on Mac OS X Yosemite using
your patch. Attached is the patched home-brew script I used.

When I reported the JIT compilation speed I reported earlier I forgot
to mention that I was running on SSD disks. With laptop hard disk
(5400 rpm) the JIT compiler is obviously even slower - the Lua Tests
that ran in 1 second in interpreted mode took 1648 seconds in JIT
mode. Timings below. I think that your work on getting rid of the
process invocations should be given highest priority as JIT
compilation times need to be improved significantly.

Interpreted mode:

test done on 13/08/2015, at 21:23:43
Lua 5.3
64-bit integers, 64-bit floats
cleaning all!!!!
......    ---- total memory: 60.0K, max memory: 2.7M ----
total time: 0.57s (wall time: 1s)


JIT mode:
testing date/time
test done on 13/08/2015, at 21:51:11
Lua 5.3
64-bit integers, 64-bit floats
cleaning all!!!!
......    ---- total memory: 59.9K, max memory: 2.7M ----
total time: 659.17s (wall time: 1648s)

[-- Attachment #2: gcc.rb --]
[-- Type: text/x-ruby-script, Size: 9454 bytes --]

class Gcc < Formula
  def arch
    if Hardware::CPU.type == :intel
      if MacOS.prefer_64_bit?
        "x86_64"
      else
        "i686"
      end
    elsif Hardware::CPU.type == :ppc
      if MacOS.prefer_64_bit?
        "powerpc64"
      else
        "powerpc"
      end
    end
  end

  def osmajor
    `uname -r`.chomp
  end

  desc "GNU compiler collection"
  homepage "https://gcc.gnu.org"
  url "http://ftpmirror.gnu.org/gcc/gcc-5.2.0/gcc-5.2.0.tar.bz2"
  mirror "https://ftp.gnu.org/gnu/gcc/gcc-5.2.0/gcc-5.2.0.tar.bz2"
  sha256 "5f835b04b5f7dd4f4d2dc96190ec1621b8d89f2dc6f638f9f8bc1b1014ba8cad"

  bottle do
    sha256 "b9a41591f99a25a5d57d818953def46c4c654eb5ec8738b5eb975b366068abc3" => :yosemite
    sha256 "c70b4f9c65930c0f127cb8a0b220d7b1494edea11218f5ba519b758c3eb7a2cd" => :mavericks
    sha256 "131566719249be8795422d86fff0c4b2ee12bb58ad5320dcbff0af2c40d3ece7" => :mountain_lion
  end

  option "with-java", "Build the gcj compiler"
  option "with-all-languages", "Enable all compilers and languages, except Ada"
  option "with-nls", "Build with native language support (localization)"
  option "with-jit", "Build the jit compiler"
  option "without-fortran", "Build without the gfortran compiler"
  # enabling multilib on a host that can't run 64-bit results in build failures
  option "without-multilib", "Build without multilib support" if MacOS.prefer_64_bit?

  depends_on "gmp"
  depends_on "libmpc"
  depends_on "mpfr"
  depends_on "isl"
  depends_on "ecj" if build.with?("java") || build.with?("all-languages")

  if MacOS.version < :leopard
    # The as that comes with Tiger isn't capable of dealing with the
    # PPC asm that comes in libitm
    depends_on "cctools" => :build
  end

  fails_with :gcc_4_0
  fails_with :llvm

  # GCC bootstraps itself, so it is OK to have an incompatible C++ stdlib
  cxxstdlib_check :skip

  # The bottles are built on systems with the CLT installed, and do not work
  # out of the box on Xcode-only systems due to an incorrect sysroot.
  def pour_bottle?
    MacOS::CLT.installed?
  end

  def version_suffix
    version.to_s.slice(/\d/)
  end

  # Fix for libgccjit.so linkage on Darwin
  # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64089
  patch :DATA

  def install
    # GCC will suffer build errors if forced to use a particular linker.
    ENV.delete "LD"

    if MacOS.version < :leopard
      ENV["AS"] = ENV["AS_FOR_TARGET"] = "#{Formula["cctools"].bin}/as"
    end

    if build.with? "all-languages"
      # Everything but Ada, which requires a pre-existing GCC Ada compiler
      # (gnat) to bootstrap. GCC 4.6.0 adds go as a language option, but it is
      # currently only compilable on Linux.
      languages = %w[c c++ objc obj-c++ fortran java jit]
    else
      # C, C++, ObjC compilers are always built
      languages = %w[c c++ objc obj-c++]

      languages << "fortran" if build.with? "fortran"
      languages << "java" if build.with? "java"
      languages << "jit" if build.with? "jit"
    end

    args = [
      "--build=#{arch}-apple-darwin#{osmajor}",
      "--prefix=#{prefix}",
      "--libdir=#{lib}/gcc/#{version_suffix}",
      "--enable-languages=#{languages.join(",")}",
      # Make most executables versioned to avoid conflicts.
      "--program-suffix=-#{version_suffix}",
      "--with-gmp=#{Formula["gmp"].opt_prefix}",
      "--with-mpfr=#{Formula["mpfr"].opt_prefix}",
      "--with-mpc=#{Formula["libmpc"].opt_prefix}",
      "--with-isl=#{Formula["isl"].opt_prefix}",
      "--with-system-zlib",
      "--enable-libstdcxx-time=yes",
      "--enable-stage1-checking",
      "--enable-checking=release",
      "--enable-lto",
      # Use 'bootstrap-debug' build configuration to force stripping of object
      # files prior to comparison during bootstrap (broken by Xcode 6.3).
      "--with-build-config=bootstrap-debug",
      "--disable-werror",
      "--with-pkgversion=Homebrew #{name} #{pkg_version} #{build.used_options*" "}".strip,
      "--with-bugurl=https://github.com/Homebrew/homebrew/issues"
    ]

    # "Building GCC with plugin support requires a host that supports
    # -fPIC, -shared, -ldl and -rdynamic."
    args << "--enable-plugin" if MacOS.version > :tiger

    # Otherwise make fails during comparison at stage 3
    # See: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=45248
    args << "--with-dwarf2" if MacOS.version < :leopard

    args << "--disable-nls" if build.without? "nls"

    if build.with?("java") || build.with?("all-languages")
      args << "--with-ecj-jar=#{Formula["ecj"].opt_share}/java/ecj.jar"
    end

    if build.without?("multilib") || !MacOS.prefer_64_bit?
      args << "--disable-multilib"
    else
      args << "--enable-multilib"
    end

    args << "--enable-host-shared" if build.with?("jit") || build.with?("all-languages")

    # Ensure correct install names when linking against libgcc_s;
    # see discussion in https://github.com/Homebrew/homebrew/pull/34303
    inreplace "libgcc/config/t-slibgcc-darwin", "@shlib_slibdir@", "#{HOMEBREW_PREFIX}/lib/gcc/#{version_suffix}"

    mkdir "build" do
      unless MacOS::CLT.installed?
        # For Xcode-only systems, we need to tell the sysroot path.
        # "native-system-headers" will be appended
        args << "--with-native-system-header-dir=/usr/include"
        args << "--with-sysroot=#{MacOS.sdk_path}"
      end

      system "../configure", *args
      system "make", "bootstrap"
      system "make", "install"

      if build.with?("fortran") || build.with?("all-languages")
        bin.install_symlink bin/"gfortran-#{version_suffix}" => "gfortran"
      end
    end

    # Handle conflicts between GCC formulae and avoid interfering
    # with system compilers.
    # Since GCC 4.8 libffi stuff are no longer shipped.
    # Rename man7.
    Dir.glob(man7/"*.7") { |file| add_suffix file, version_suffix }
    # Even when suffixes are appended, the info pages conflict when
    # install-info is run. TODO fix this.
    info.rmtree

    # Rename java properties
    if build.with?("java") || build.with?("all-languages")
      config_files = [
        "#{lib}/gcc/#{version_suffix}/logging.properties",
        "#{lib}/gcc/#{version_suffix}/security/classpath.security",
        "#{lib}/gcc/#{version_suffix}/i386/logging.properties",
        "#{lib}/gcc/#{version_suffix}/i386/security/classpath.security"
      ]
      config_files.each do |file|
        add_suffix file, version_suffix if File.exist? file
      end
    end
  end

  def add_suffix(file, suffix)
    dir = File.dirname(file)
    ext = File.extname(file)
    base = File.basename(file, ext)
    File.rename file, "#{dir}/#{base}-#{suffix}#{ext}"
  end

  def caveats
    if build.with?("multilib") then <<-EOS.undent
      GCC has been built with multilib support. Notably, OpenMP may not work:
        https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60670
      If you need OpenMP support you may want to
        brew reinstall gcc --without-multilib
      EOS
    end
  end

  test do
    (testpath/"hello-c.c").write <<-EOS.undent
      #include <stdio.h>
      int main()
      {
        puts("Hello, world!");
        return 0;
      }
    EOS
    system "#{bin}/gcc-#{version_suffix}", "-o", "hello-c", "hello-c.c"
    assert_equal "Hello, world!\n", `./hello-c`

    (testpath/"hello-cc.cc").write <<-EOS.undent
      #include <iostream>
      int main()
      {
        std::cout << "Hello, world!" << std::endl;
        return 0;
      }
    EOS
    system "#{bin}/g++-#{version_suffix}", "-o", "hello-cc", "hello-cc.cc"
    assert_equal "Hello, world!\n", `./hello-cc`

    if build.with?("fortran") || build.with?("all-languages")
      fixture = <<-EOS.undent
        integer,parameter::m=10000
        real::a(m), b(m)
        real::fact=0.5

        do concurrent (i=1:m)
          a(i) = a(i) + fact*b(i)
        end do
        print *, "done"
        end
      EOS
      (testpath/"in.f90").write(fixture)
      system "#{bin}/gfortran", "-c", "in.f90"
      system "#{bin}/gfortran", "-o", "test", "in.o"
      assert_equal "done", `./test`.strip
    end
  end
end
__END__
diff --git a/gcc/jit/Make-lang.in b/gcc/jit/Make-lang.in
index 44d0750..4df2a9c 100644
--- a/gcc/jit/Make-lang.in
+++ b/gcc/jit/Make-lang.in
@@ -85,8 +85,7 @@ $(LIBGCCJIT_FILENAME): $(jit_OBJS) \
	     $(jit_OBJS) libbackend.a libcommon-target.a libcommon.a \
	     $(CPPLIB) $(LIBDECNUMBER) $(LIBS) $(BACKENDLIBS) \
	     $(EXTRA_GCC_OBJS) \
-	     -Wl,--version-script=$(srcdir)/jit/libgccjit.map \
-	     -Wl,-soname,$(LIBGCCJIT_SONAME)
+	     -Wl,-install_name,$(LIBGCCJIT_SONAME)

 $(LIBGCCJIT_SONAME_SYMLINK): $(LIBGCCJIT_FILENAME)
	ln -sf $(LIBGCCJIT_FILENAME) $(LIBGCCJIT_SONAME_SYMLINK)
diff --git a/gcc/jit/jit-playback.c b/gcc/jit/jit-playback.c
index 925fa86..01cfd4b 100644
--- a/gcc/jit/jit-playback.c
+++ b/gcc/jit/jit-playback.c
@@ -2416,6 +2416,15 @@ invoke_driver (const char *ctxt_progname,
      time.  */
   ADD_ARG ("-fno-use-linker-plugin");

+#if defined (DARWIN_X86) || defined (DARWIN_PPC)
+  /* OS X's linker defaults to treating undefined symbols as errors.
+     If the context has any imported functions or globals they will be
+     undefined until the .so is dynamically-linked into the process.
+     Ensure that the driver passes in "-undefined dynamic_lookup" to the
+     linker.  */
+  ADD_ARG ("-Wl,-undefined,dynamic_lookup");
+#endif
+
   /* pex argv arrays are NULL-terminated.  */
   argvec.safe_push (NULL);

Re: libgccjit 5.2 on Mac OSX Yosemite

[Dibyendu Majumdar]

On 14 August 2015 at 02:38, David Malcolm <dmalcolm@redhat.com> wrote:
> Out of interest, how does this compare to LLVM.  IIRC you had said
> libgccjit was taking about 5x longer than LLVM; is that still the case?
>
>> Interpreted mode:
>>
>> test done on 13/08/2015, at 21:23:43
>> Lua 5.3
>> 64-bit integers, 64-bit floats
>> cleaning all!!!!
>> ......    ---- total memory: 60.0K, max memory: 2.7M ----
>> total time: 0.57s (wall time: 1s)
>>
>>
>> JIT mode:
>> testing date/time
>> test done on 13/08/2015, at 21:51:11
>> Lua 5.3
>> 64-bit integers, 64-bit floats
>> cleaning all!!!!
>> ......    ---- total memory: 59.9K, max memory: 2.7M ----
>> total time: 659.17s (wall time: 1648s)
>

LLVM 3.7 branch on same hardware and OS:

test done on 14/08/2015, at 21:49:47
Lua 5.3
64-bit integers, 64-bit floats
cleaning all!!!!
......    ---- total memory: 59.9K, max memory: 2.2M ----
total time: 149.18s (wall time: 151s)

Note that the LLVM version handles a few more byte codes so this would
be a disadvantage to LLVM as it will compile more functions.

With libgccjit you can literally watch the processes being invoked -
as, collect2, ld, etc. - as the OS X terminal shows any subprocesses
launched in the title bar.

On my other environment which is running Ubuntu as a VM under Windows,
the difference is less as that environment has SSD disks so that helps
libgccjit.

Regards
Dibyendu

Re: libgccjit 5.2 on Mac OSX Yosemite

[David Malcolm]

On Thu, 2015-08-13 at 22:04 +0100, Dibyendu Majumdar wrote:
> Dave,
> 
> I was able to get the JIT compiler working on Mac OS X Yosemite using
> your patch. Attached is the patched home-brew script I used.

Thanks; glad it's working.

As noted in the followup I posted, I've now committed the patch to both
branches.

I've also created this pull request:
  https://github.com/Homebrew/homebrew/pull/42924
requesting that they bring in your change to the homebrew gcc.rb recipe.



> When I reported the JIT compilation speed I reported earlier I forgot
> to mention that I was running on SSD disks. With laptop hard disk
> (5400 rpm) the JIT compiler is obviously even slower - the Lua Tests
> that ran in 1 second in interpreted mode took 1648 seconds in JIT
> mode. Timings below. I think that your work on getting rid of the
> process invocations should be given highest priority as JIT
> compilation times need to be improved significantly.

(nods)

If you get a chance to try gcc trunk, it might be interesting to try the
profiling hooks I added:
  https://gcc.gnu.org/ml/jit/2015-q3/msg00134.html

to see the libgccjit profile for Ravi.

In particular, I'm hoping that the Ravi profile is sufficiently similar
to that from test-benchmark.c so that my optimizations will benefit Ravi
(or maybe there are other unexpected things going on?)

Out of interest, how does this compare to LLVM.  IIRC you had said
libgccjit was taking about 5x longer than LLVM; is that still the case?

> Interpreted mode:
> 
> test done on 13/08/2015, at 21:23:43
> Lua 5.3
> 64-bit integers, 64-bit floats
> cleaning all!!!!
> ......    ---- total memory: 60.0K, max memory: 2.7M ----
> total time: 0.57s (wall time: 1s)
> 
> 
> JIT mode:
> testing date/time
> test done on 13/08/2015, at 21:51:11
> Lua 5.3
> 64-bit integers, 64-bit floats
> cleaning all!!!!
> ......    ---- total memory: 59.9K, max memory: 2.7M ----
> total time: 659.17s (wall time: 1648s)

RE: libgccjit 5.2 on Mac OSX Yosemite

[Dibyendu Majumdar]

Hi Dave,
You may want to submit the patch also to the homebrew folks once I have
confirmed it works as without this the official 5.2 distro isn't going
to work on Mac OSX.

Regards

From: David Malcolm
Sent: ‎11/‎08/‎2015 17:21
To: Dibyendu Majumdar
Cc: jit@gcc.gnu.org
Subject: Re: libgccjit 5.2 on Mac OSX Yosemite
On Mon, 2015-08-10 at 23:52 +0100, Dibyendu Majumdar wrote:

[CCing the list]

> On 10 August 2015 at 22:35, David Malcolm <dmalcolm@redhat.com> wrote:
> > Please can you try enabling logging using:
> > https://gcc.gnu.org/onlinedocs/jit/topics/contexts.html#gcc_jit_context_set_logfile
> >
> > to capture the driver invocation line (have a look for invoke_driver in
> > the log).
> >
> > You'll also need to set the bool option
> > GCC_JIT_BOOL_OPTION_KEEP_INTERMEDIATES to stop it deleting everything.
> >
>
> JIT:      entering: void
> gcc::jit::playback::context::invoke_driver(const char*, const char*,
> const char*, timevar_id_t, bool, bool)
>
> JIT:       argv[0]: x86_64-apple-darwin14.4.0-gcc-5.2.0
>
> JIT:       argv[1]: -shared
>
> JIT:       argv[2]:
> /var/folders/lm/ygx0frjs7h30b03r14k7jfhh0000gn/T//libgccjit-BmebwW/fake.s
>
> JIT:       argv[3]: -o
>
> JIT:       argv[4]:
> /var/folders/lm/ygx0frjs7h30b03r14k7jfhh0000gn/T//libgccjit-BmebwW/fake.so
>
> JIT:       argv[5]: -fno-use-linker-plugin
>
> JIT:       argv[6]: (null)
>
> Undefined symbols for architecture x86_64:
>
>   "_luaD_poscall", referenced from:
>
>       _ravif1 in ccouygd9.o
>
>   "_luaF_close", referenced from:
>
>       _ravif1 in ccouygd9.o
>
> ld: symbol(s) not found for architecture x86_64
>
>
>
> > With that, you can try to re-run the driver invocation directly from the
> > command-line, to see if you can reproduce the problem there, and if the
> > extra option above
> >   -Wl,-undefined,dynamic_lookup
> > fixes it.
> >
>
> Appears to fix the error.
>
> x86_64-apple-darwin14.4.0-gcc-5.2.0 -shared
> /var/folders/lm/ygx0frjs7h30b03r14k7jfhh0000gn/T//libgccjit-BmebwW/fake.s
> -o /var/folders/lm/ygx0frjs7h30b03r14k7jfhh0000gn/T//libgccjit-BmebwW/fake.so
> -fno-use-linker-plugin -Wl,-undefined,dynamic_lookup
>
> ls -l /var/folders/lm/ygx0frjs7h30b03r14k7jfhh0000gn/T//libgccjit-BmebwW/fake.so
>
> -rwxr-xr-x  1 dibyendumajumdar  staff  8504 10 Aug 23:51
> /var/folders/lm/ygx0frjs7h30b03r14k7jfhh0000gn/T//libgccjit-BmebwW/fake.so

Thanks.

Here's a patch which adds that argument to the driver invocation (it's
against trunk, but it ought to apply to gcc-5 also).

Does it fix things?


Dave

Re: libgccjit 5.2 on Mac OSX Yosemite

[David Malcolm]

[-- Attachment #1: Type: text/plain, Size: 2219 bytes --]

On Mon, 2015-08-10 at 23:52 +0100, Dibyendu Majumdar wrote:

[CCing the list]

> On 10 August 2015 at 22:35, David Malcolm <dmalcolm@redhat.com> wrote:
> > Please can you try enabling logging using:
> > https://gcc.gnu.org/onlinedocs/jit/topics/contexts.html#gcc_jit_context_set_logfile
> >
> > to capture the driver invocation line (have a look for invoke_driver in
> > the log).
> >
> > You'll also need to set the bool option
> > GCC_JIT_BOOL_OPTION_KEEP_INTERMEDIATES to stop it deleting everything.
> >
> 
> JIT:      entering: void
> gcc::jit::playback::context::invoke_driver(const char*, const char*,
> const char*, timevar_id_t, bool, bool)
> 
> JIT:       argv[0]: x86_64-apple-darwin14.4.0-gcc-5.2.0
> 
> JIT:       argv[1]: -shared
> 
> JIT:       argv[2]:
> /var/folders/lm/ygx0frjs7h30b03r14k7jfhh0000gn/T//libgccjit-BmebwW/fake.s
> 
> JIT:       argv[3]: -o
> 
> JIT:       argv[4]:
> /var/folders/lm/ygx0frjs7h30b03r14k7jfhh0000gn/T//libgccjit-BmebwW/fake.so
> 
> JIT:       argv[5]: -fno-use-linker-plugin
> 
> JIT:       argv[6]: (null)
> 
> Undefined symbols for architecture x86_64:
> 
>   "_luaD_poscall", referenced from:
> 
>       _ravif1 in ccouygd9.o
> 
>   "_luaF_close", referenced from:
> 
>       _ravif1 in ccouygd9.o
> 
> ld: symbol(s) not found for architecture x86_64
> 
> 
> 
> > With that, you can try to re-run the driver invocation directly from the
> > command-line, to see if you can reproduce the problem there, and if the
> > extra option above
> >   -Wl,-undefined,dynamic_lookup
> > fixes it.
> >
> 
> Appears to fix the error.
> 
> x86_64-apple-darwin14.4.0-gcc-5.2.0 -shared
> /var/folders/lm/ygx0frjs7h30b03r14k7jfhh0000gn/T//libgccjit-BmebwW/fake.s
> -o /var/folders/lm/ygx0frjs7h30b03r14k7jfhh0000gn/T//libgccjit-BmebwW/fake.so
> -fno-use-linker-plugin -Wl,-undefined,dynamic_lookup
> 
> ls -l /var/folders/lm/ygx0frjs7h30b03r14k7jfhh0000gn/T//libgccjit-BmebwW/fake.so
> 
> -rwxr-xr-x  1 dibyendumajumdar  staff  8504 10 Aug 23:51
> /var/folders/lm/ygx0frjs7h30b03r14k7jfhh0000gn/T//libgccjit-BmebwW/fake.so

Thanks.

Here's a patch which adds that argument to the driver invocation (it's
against trunk, but it ought to apply to gcc-5 also).

Does it fix things?


Dave

[-- Attachment #2: 0001-jit-on-OS-X-add-Wl-undefined-dynamic_lookup-to-drive.patch --]
[-- Type: text/x-patch, Size: 1211 bytes --]

From 9971d11fd5e151ae00d214492e14d8d3347bf471 Mon Sep 17 00:00:00 2001
From: David Malcolm <dmalcolm@redhat.com>
Date: Tue, 11 Aug 2015 11:48:46 -0400
Subject: [PATCH] jit: on OS X, add -Wl,-undefined,dynamic_lookup to driver
 invocation

gcc/jit/ChangeLog:
	* jit-playback.c (invoke_driver): On OS X, add
	"-Wl,-undefined,dynamic_lookup" to the driver arguments.
---
 gcc/jit/jit-playback.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/gcc/jit/jit-playback.c b/gcc/jit/jit-playback.c
index 925fa86..01cfd4b 100644
--- a/gcc/jit/jit-playback.c
+++ b/gcc/jit/jit-playback.c
@@ -2416,6 +2416,15 @@ invoke_driver (const char *ctxt_progname,
      time.  */
   ADD_ARG ("-fno-use-linker-plugin");
 
+#if defined (DARWIN_X86) || defined (DARWIN_PPC)
+  /* OS X's linker defaults to treating undefined symbols as errors.
+     If the context has any imported functions or globals they will be
+     undefined until the .so is dynamically-linked into the process.
+     Ensure that the driver passes in "-undefined dynamic_lookup" to the
+     linker.  */
+  ADD_ARG ("-Wl,-undefined,dynamic_lookup");
+#endif
+
   /* pex argv arrays are NULL-terminated.  */
   argvec.safe_push (NULL);
 
-- 
1.8.5.3

Re: libgccjit 5.2 on Mac OSX Yosemite

[David Malcolm]

On Mon, 2015-08-10 at 21:36 +0100, Dibyendu Majumdar wrote:
> Hi
> 
> I tried building gcc 5.2 on Mac OSX but could not get the linking to
> work. Then fortunately found that homebrew has a formula for deploy
> gcc 5.2 with jit enabled.


Aha.  Thanks.  This would seem to be:
 https://github.com/Homebrew/homebrew/blob/master/Library/Formula/gcc.rb

which I see slightly patches
  gcc/jit/Make-lang.in
to avoid an assumption that we're building using the GNU linker.



> All installed and then built Ravi successfully. But when I try to run
> Ravi any attempt to invoke the JIT fails with:
> 
> Lua 5.3.1  Copyright (C) 1994-2015 Lua.org, PUC-Rio
> 
> > function x() return end
> 
> > ravi.compile(x)
> 
> Undefined symbols for architecture x86_64:
> 
>   "_luaD_poscall", referenced from:
> 
>       _ravif1 in cczhNq92.o
> 
>   "_luaF_close", referenced from:
> 
>       _ravif1 in cczhNq92.o
> 
> ld: symbol(s) not found for architecture x86_64
> 
> collect2: error: ld returned 1 exit status
> 
> libgccjit.so: error: error invoking gcc driver: exit_status: 256 err: 0
> 
> libgccjit.so: error: whilst attempting to run a driver named:
> x86_64-apple-darwin14.4.0-gcc-5.2.0
> 
> libgccjit.so: error: PATH was:
> /Users/username/github/ravi/build:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/git/bin:/Library/TeX/texbin
> 
> aborting due to JIT error: error invoking gcc driver: exit_status: 256 err: 0
> 
> Abort trap: 6
> 
> Any clues?
> 
> I had set PATH and LD_LIBRARY_PATH.

Presumably "luaD_poscall" and "luaF_close" are symbols that you're
exporting from ravi for use by the JIT-generated code, via:
https://github.com/dibyendumajumdar/ravi/blob/master/src/ravi_gcctypes.c#L802

  t->luaD_poscallT = gcc_jit_context_new_function(
      ravi->context, NULL, GCC_JIT_FUNCTION_IMPORTED, t->C_intT,
"luaD_poscall",
      3, params, 0);

I'm not quite sure exactly where the leading underscore came from.

This appears to be a linker error from the static link that creates the
jit-generated .so file.  If I'm reading things right, it's complaining
that the code is using luaD_poscall, but it doesn't have a definition
for it.

My guess is that "ld" on OS X has slightly different behavior from the
GNU "ld" I've been developing against, and that it defaults to a
stricter linkage model where it wants to see those symbols upfront, when
we want to defer them until the later dynamic link phase (when we dlopen
the generated .so file).  This is a guess though.

What does "ld --version" print?

It may be easiest to try to work from a minimal reproducer for this.
To that end, what happens if you try building and running
  gcc/testsuite/jit.dg/test-calling-external-function.c
from the jit testsuite, using the homebrew build?


Thanks
Dave

Re: libgccjit 5.2 on Mac OSX Yosemite

[Dibyendu Majumdar]

On 11 August 2015 at 17:21, David Malcolm <dmalcolm@redhat.com> wrote:
> Here's a patch which adds that argument to the driver invocation (it's
> against trunk, but it ought to apply to gcc-5 also).
>
> Does it fix things?
>

Thanks Dave - unfortunately I have temporarily lost my Mac OSX
environment - I recently upgraded to Yosemite, but since it was slow I
thought I would do a clean install, and now am not stuck with the
installer not finishing :-(.

I will look at the patch once everything is back to normal.
I cannot build gcc myself so I will have to create a modified homebrew
formula I think.

Regards
Dibyendu

Re: libgccjit 5.2 on Mac OSX Yosemite

[David Malcolm]

On Mon, 2015-08-10 at 17:23 -0400, David Malcolm wrote:
> On Mon, 2015-08-10 at 21:36 +0100, Dibyendu Majumdar wrote:
> > Hi
> > 
> > I tried building gcc 5.2 on Mac OSX but could not get the linking to
> > work. Then fortunately found that homebrew has a formula for deploy
> > gcc 5.2 with jit enabled.
> 
> 
> Aha.  Thanks.  This would seem to be:
>  https://github.com/Homebrew/homebrew/blob/master/Library/Formula/gcc.rb
> 
> which I see slightly patches
>   gcc/jit/Make-lang.in
> to avoid an assumption that we're building using the GNU linker.
> 
> 
> 
> > All installed and then built Ravi successfully. But when I try to run
> > Ravi any attempt to invoke the JIT fails with:
> > 
> > Lua 5.3.1  Copyright (C) 1994-2015 Lua.org, PUC-Rio
> > 
> > > function x() return end
> > 
> > > ravi.compile(x)
> > 
> > Undefined symbols for architecture x86_64:
> > 
> >   "_luaD_poscall", referenced from:
> > 
> >       _ravif1 in cczhNq92.o
> > 
> >   "_luaF_close", referenced from:
> > 
> >       _ravif1 in cczhNq92.o
> > 
> > ld: symbol(s) not found for architecture x86_64
> > 
> > collect2: error: ld returned 1 exit status
> > 
> > libgccjit.so: error: error invoking gcc driver: exit_status: 256 err: 0
> > 
> > libgccjit.so: error: whilst attempting to run a driver named:
> > x86_64-apple-darwin14.4.0-gcc-5.2.0
> > 
> > libgccjit.so: error: PATH was:
> > /Users/username/github/ravi/build:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/git/bin:/Library/TeX/texbin
> > 
> > aborting due to JIT error: error invoking gcc driver: exit_status: 256 err: 0
> > 
> > Abort trap: 6
> > 
> > Any clues?
> > 
> > I had set PATH and LD_LIBRARY_PATH.
> 
> Presumably "luaD_poscall" and "luaF_close" are symbols that you're
> exporting from ravi for use by the JIT-generated code, via:
> https://github.com/dibyendumajumdar/ravi/blob/master/src/ravi_gcctypes.c#L802
> 
>   t->luaD_poscallT = gcc_jit_context_new_function(
>       ravi->context, NULL, GCC_JIT_FUNCTION_IMPORTED, t->C_intT,
> "luaD_poscall",
>       3, params, 0);
> 
> I'm not quite sure exactly where the leading underscore came from.
> 
> This appears to be a linker error from the static link that creates the
> jit-generated .so file.  If I'm reading things right, it's complaining
> that the code is using luaD_poscall, but it doesn't have a definition
> for it.
> 
> My guess is that "ld" on OS X has slightly different behavior from the
> GNU "ld" I've been developing against, and that it defaults to a
> stricter linkage model where it wants to see those symbols upfront, when
> we want to defer them until the later dynamic link phase (when we dlopen
> the generated .so file).  This is a guess though.

It looks like we might need to pass an "-undefined" option to the OS X
linker;

http://www.manpages.info/macosx/ld.1.html says:

(begin quote)

-undefined treatment
Specifies how undefined symbols are to be treated. treatment can be:
error, warning, or suppress. Which cause the treatment of undefined
symbols as either, errors, warnings, or suppresses the checking of
undefined symbols. The default is to treat undefined symbols as errors.
When the environment variable MACOSX_DEPLOYMENT_TARGET is set to 10.3
then -undefined dynamic_lookup can also be used to allow any undefined
symbols to be looked up dynamically at runtime. Use of a binary built
with this flag requires a system with a dynamic linker from Mac OS X
10.3 or later. The flag -undefined define_a_way can also be used to
cause the static linker to create a private definition for all undefined
symbols. This flag should only be used if it is known that the undefined
symbols are not referenced as any use of them may cause a crash.


(end of quote)

Note that it defaults to treating undefined symbols as errors, which
looks like what you're seeing.

Hence I *think* we need to pass:

  -undefined dynamic_lookup

to the linker.

Hence I think we need something like this when invoking the driver:

 -Wl,-undefined,dynamic_lookup

so that it passes the option to the linker.


> What does "ld --version" print?
> 
> It may be easiest to try to work from a minimal reproducer for this.
> To that end, what happens if you try building and running
>   gcc/testsuite/jit.dg/test-calling-external-function.c
> from the jit testsuite, using the homebrew build?


Please can you try enabling logging using:
https://gcc.gnu.org/onlinedocs/jit/topics/contexts.html#gcc_jit_context_set_logfile

to capture the driver invocation line (have a look for invoke_driver in
the log).

You'll also need to set the bool option
GCC_JIT_BOOL_OPTION_KEEP_INTERMEDIATES to stop it deleting everything.

With that, you can try to re-run the driver invocation directly from the
command-line, to see if you can reproduce the problem there, and if the
extra option above
  -Wl,-undefined,dynamic_lookup
fixes it.

Thanks
Dave

libgccjit 5.2 on Mac OSX Yosemite

[Dibyendu Majumdar]

Hi

I tried building gcc 5.2 on Mac OSX but could not get the linking to
work. Then fortunately found that homebrew has a formula for deploy
gcc 5.2 with jit enabled.

All installed and then built Ravi successfully. But when I try to run
Ravi any attempt to invoke the JIT fails with:

Lua 5.3.1  Copyright (C) 1994-2015 Lua.org, PUC-Rio

> function x() return end

> ravi.compile(x)

Undefined symbols for architecture x86_64:

  "_luaD_poscall", referenced from:

      _ravif1 in cczhNq92.o

  "_luaF_close", referenced from:

      _ravif1 in cczhNq92.o

ld: symbol(s) not found for architecture x86_64

collect2: error: ld returned 1 exit status

libgccjit.so: error: error invoking gcc driver: exit_status: 256 err: 0

libgccjit.so: error: whilst attempting to run a driver named:
x86_64-apple-darwin14.4.0-gcc-5.2.0

libgccjit.so: error: PATH was:
/Users/username/github/ravi/build:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/git/bin:/Library/TeX/texbin

aborting due to JIT error: error invoking gcc driver: exit_status: 256 err: 0

Abort trap: 6

Any clues?

I had set PATH and LD_LIBRARY_PATH.

Regards