[Bug debuginfod/25509] New: Break a cyclic dependency by core packages

[Bug debuginfod/25509] New: Break a cyclic dependency by core packages

[marxin.liska at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

            Bug ID: 25509
           Summary: Break a cyclic dependency by core packages
           Product: elfutils
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: debuginfod
          Assignee: unassigned at sourceware dot org
          Reporter: marxin.liska at gmail dot com
                CC: elfutils-devel at sourceware dot org
  Target Milestone: ---

In openSUSE, we do face a problem with cyclic dependencies. Many core packages
like gcc, glibc, elfutils or binutils depend on each other and create a cycle.
The cycle should contain a reasonable amount of packages.

When debuginfod was added to elfutils, we would have a huge bunch of
dependencies caused by libhttpmicro and libsqlite. These have very many
transitive dependencies. So that I was forced to split elfutils into 2
packages: elfutils and elfutils-debuginfod. The later contains all the new
packages and is not part of the boostrap cycle.

What's more problematic is that there are (and will be) tools that want to
utilize libdebuginfod such as Binutils. As mentioned, the tool is in the
bootstrap cycle and so that can't depend on elfutils-debuginfod.

So the question is how to unbreak all these dependencies for future core
packages?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/25509] Break a cyclic dependency by core packages

[marxin.liska at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

--- Comment #1 from Martin Liška <marxin.liska at gmail dot com> ---
I have basically 2 possible solutions:

- elfutils will provide a stub client library (libdebuginfod-stub.so) which
will then do dlopen for the real libdebuginfod.so during run-rime
- one can do execv of debuginfod-find which is a standalone tool that can
download a debuginfo

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/25509] Break a cyclic dependency by core packages

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

Mark Wielaard <mark at klomp dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mark at klomp dot org

--- Comment #2 from Mark Wielaard <mark at klomp dot org> ---
(In reply to Martin Liška from comment #0)
> In openSUSE, we do face a problem with cyclic dependencies. Many core
> packages like gcc, glibc, elfutils or binutils depend on each other and
> create a cycle. The cycle should contain a reasonable amount of packages.
> 
> When debuginfod was added to elfutils, we would have a huge bunch of
> dependencies caused by libhttpmicro and libsqlite. These have very many
> transitive dependencies.

debuginfod the tool has dependencies on libhttpmicro, libsqlite, libdw and
debuginfod-client.

debuginfod-client depends on libcurl (which pulls in a lot of the crypto stuff
for https support).

libdw depends on debuginfod-client, but only indirectly through ldopen.

It does make sense to put these in different (output/sub) packages.
Which is actually what the sample elfutils.spec file in config/ does.

That doesn't "split" the inputs/dependencies of the source build though. Which
is what you seem most concerned with.

> So that I was forced to split elfutils into 2
> packages: elfutils and elfutils-debuginfod. The later contains all the new
> packages and is not part of the boostrap cycle.
> 
> What's more problematic is that there are (and will be) tools that want to
> utilize libdebuginfod such as Binutils. As mentioned, the tool is in the
> bootstrap cycle and so that can't depend on elfutils-debuginfod.

Do you have a pointer to the spec files you are using now?

> So the question is how to unbreak all these dependencies for future core
> packages?

The GNU Guix people also had some concerns about all the new dependencies and
also suggested splitting the package:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38803

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/25509] Break a cyclic dependency by core packages

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fche at redhat dot com

--- Comment #3 from Frank Ch. Eigler <fche at redhat dot com> ---
Indeed, merely splitting the package up does not solve problematic incoming
dependencies.

IMHO we should encourage distros to have a bootstrap-oriented build of
elfutils, possibly prepared thusly:

- have the elfutils configury have a -debuginfod mode (--without-debuginfod?)
that, instead of just skipping building all that stuff, builds a no-op stub
instead

- expose that --without-debuginfod configure flag to another spec file (or a
--with-* flagged version of the main one) to generate a set of
elfutils-bootstrap-* subrpms

- have the normal non-bootstrap spec file variant Obsolete/Conflict with the
bootstrap siblings

- then these elfutils-bootstrap* rpms should be functional and satisfy all your
bootstrapping needs

- and after bootstrapping is done, nuke the elfutils-bootstrap rpms from your
repo, and build the real thing

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/25509] Break a cyclic dependency by core packages

[marxin.liska at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

--- Comment #4 from Martin Liška <marxin.liska at gmail dot com> ---
(In reply to Mark Wielaard from comment #2)
> (In reply to Martin Liška from comment #0)
> > In openSUSE, we do face a problem with cyclic dependencies. Many core
> > packages like gcc, glibc, elfutils or binutils depend on each other and
> > create a cycle. The cycle should contain a reasonable amount of packages.
> > 
> > When debuginfod was added to elfutils, we would have a huge bunch of
> > dependencies caused by libhttpmicro and libsqlite. These have very many
> > transitive dependencies.
> 
> debuginfod the tool has dependencies on libhttpmicro, libsqlite, libdw and
> debuginfod-client.
> 
> debuginfod-client depends on libcurl (which pulls in a lot of the crypto
> stuff for https support).
> 
> libdw depends on debuginfod-client, but only indirectly through ldopen.
> 
> It does make sense to put these in different (output/sub) packages.
> Which is actually what the sample elfutils.spec file in config/ does.
> 
> That doesn't "split" the inputs/dependencies of the source build though.
> Which is what you seem most concerned with.
> 
> > So that I was forced to split elfutils into 2
> > packages: elfutils and elfutils-debuginfod. The later contains all the new
> > packages and is not part of the boostrap cycle.
> > 
> > What's more problematic is that there are (and will be) tools that want to
> > utilize libdebuginfod such as Binutils. As mentioned, the tool is in the
> > bootstrap cycle and so that can't depend on elfutils-debuginfod.
> 
> Do you have a pointer to the spec files you are using now?

You can find the 2 spec files here:
https://build.opensuse.org/package/show/Base:System/elfutils

> 
> > So the question is how to unbreak all these dependencies for future core
> > packages?
> 
> The GNU Guix people also had some concerns about all the new dependencies
> and also suggested splitting the package:
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38803

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/25509] Break a cyclic dependency by core packages

[eschwartz at archlinux dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

Eli Schwartz <eschwartz at archlinux dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |eschwartz at archlinux dot org

--- Comment #5 from Eli Schwartz <eschwartz at archlinux dot org> ---
> # Look for libmicrohttpd, libcurl, libarchive, sqlite for debuginfo server
> # minimum versions as per rhel7.  Single --enable-* option arranges to build
> # both client and server.

Maybe this should not be the case? If libdebuginfod was enabled on its own as
long as libcurl is available, this would make things a lot simpler. At least
for Arch Linux, curl is part of the base system while libmicrohttpd is not.

So the tools in the bootstrap cycle would only need to depend on curl (which
itself can be bootstrapped without support for most things including SSL) and
that would be sufficient to build everything that links to libdebuginfod.so

This would be independently useful alongside making a stub loader.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/25509] Break a cyclic dependency by core packages

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

--- Comment #6 from Frank Ch. Eigler <fche at redhat dot com> ---
(In reply to Eli Schwartz from comment #5)
> > # Look for libmicrohttpd, libcurl, libarchive, sqlite for debuginfo server
> > # minimum versions as per rhel7.  Single --enable-* option arranges to build
> > # both client and server.
> 
> Maybe this should not be the case? If libdebuginfod was enabled on its own
> as long as libcurl is available, this would make things a lot simpler. At
> least for Arch Linux, curl is part of the base system while libmicrohttpd is
> not.

This part wouldn't affect the suse/fedora folks, because one would already
place the debuginfod server into a separate subpackage, so its prereqs don't
affect the buildroot.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/25509] Break a cyclic dependency by core packages

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

--- Comment #7 from Mark Wielaard <mark at klomp dot org> ---
So there are two ideas here:

- Split --disable-debuginfod which currently disables building
  debuginfod (the server), libdebuginfod (the library) and
  debuginfod-find (the helper binaries) in two:
  --disable-debuginfod to disable building of the server.
  --disable-libdebuginfod to disable building of the library and helper binary.

- Introduce a dummy-libdebuginfod which has the same interface as
  libdebuginfod, so applications (during bootstrapping) can link to
  it. But which would not link against libcurl and all functions would
  simply fail.

I assume that --disable-libdebuginfod would mean you cannot have
--enable-debuginfod (because debuginfod links against libdebuginfod). And that
--enable-dummy-libdebuginfod would imply --disable-libdebuginfod because they
provide the same library. I assume that debuginfod-find would be build if
either --enable-libdebuginfod or --enable-dummy-libdebuginfod is given (in the
last case it would simply not do anything useful, but would show the dummy
"works").

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/25509] Break a cyclic dependency by core packages

[mliska at suse dot cz]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

Martin Liska <mliska at suse dot cz> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mliska at suse dot cz

--- Comment #8 from Martin Liska <mliska at suse dot cz> ---
> 
> - Introduce a dummy-libdebuginfod which has the same interface as
>   libdebuginfod, so applications (during bootstrapping) can link to
>   it. But which would not link against libcurl and all functions would
>   simply fail.
> 

Yes, I support the creation of a stub library.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/25509] Break a cyclic dependency by core packages

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

Mark Wielaard <mark at klomp dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Last reconfirmed|                            |2020-06-19
           Assignee|unassigned at sourceware dot org   |mark at klomp dot org
     Ever confirmed|0                           |1
             Status|UNCONFIRMED                 |ASSIGNED

--- Comment #9 from Mark Wielaard <mark at klomp dot org> ---
Created attachment 12628
  --> https://sourceware.org/bugzilla/attachment.cgi?id=12628&action=edit
debuginfod: Add --disable-libdebuginfod and --enable-libdebuginfod=dummy.

How about the attached? It splits off disabling of the client by introducing
--disable-libdebuginfod and you can specificy --enable-libdebuginfod=dummy to
create a dummy libdebuginfod which simply fails all calls.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/25509] Break a cyclic dependency by core packages

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

--- Comment #10 from Frank Ch. Eigler <fche at redhat dot com> ---
Comment on attachment 12628
  --> https://sourceware.org/bugzilla/attachment.cgi?id=12628
debuginfod: Add --disable-libdebuginfod and --enable-libdebuginfod=dummy.

Looks workable.

I don't know if we need a new --enable-libdebuginfod=dummy
option, as opposed to it being simply the equivalent of
--disable-libdebuginfod, or even --disable-debuginfod
(to have a common configury option for both client & server,
as now).  But that's just a style nit.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/25509] Break a cyclic dependency by core packages

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

--- Comment #11 from Mark Wielaard <mark at klomp dot org> ---
(In reply to Frank Ch. Eigler from comment #10)
> Comment on attachment 12628 [details]
> debuginfod: Add --disable-libdebuginfod and --enable-libdebuginfod=dummy.
> 
> Looks workable.
> 
> I don't know if we need a new --enable-libdebuginfod=dummy
> option, as opposed to it being simply the equivalent of
> --disable-libdebuginfod, or even --disable-debuginfod
> (to have a common configury option for both client & server,
> as now).  But that's just a style nit.

Yeah, you are right that it is more complexity than is strictly necessary. But
it allowed me to easily test the dummy libdebuginfod by still building
debuginfod and debuginfod-find against it and see they still "work".

Martin, could you comment on the setup? Does this help your case? I will
probably not actually use it myself, so don't want to add it unless it really
makes someone happy to have it as an option.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/25509] Break a cyclic dependency by core packages

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

--- Comment #12 from Mark Wielaard <mark at klomp dot org> ---
(In reply to Mark Wielaard from comment #11)
> Martin, could you comment on the setup? Does this help your case? I will
> probably not actually use it myself, so don't want to add it unless it
> really makes someone happy to have it as an option.

Eli, same question for you. Is this setup useful for what you want to do in
Arch Linux?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/25509] Break a cyclic dependency by core packages

[eschwartz at archlinux dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

--- Comment #13 from Eli Schwartz <eschwartz at archlinux dot org> ---
I think we'd prefer to have the ability to build libdebuginfod without the
server. Ambivalent about a special option to force the libdebuginfod dummy to
be created, but...

If it's just for testing purposes, maybe you could always build the dummy, but
only install it if the real libdebuginfod was not enabled? The testsuite could
verify the dummy works by using LD_LIBRARY_PATH/LD_PRELOAD (depending on build
directory layout) to run some tests explicitly using the dummy.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/25509] Break a cyclic dependency by core packages

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

--- Comment #14 from Frank Ch. Eigler <fche at redhat dot com> ---
It's not just for testing purposes.  It's to aid bootstrapping new OS versions,
by reducing the transitive requirements of elfutils in the buildroot.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/25509] Break a cyclic dependency by core packages

[eschwartz at archlinux dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

--- Comment #15 from Eli Schwartz <eschwartz at archlinux dot org> ---
Wouldn't that be covered by "libdebuginfod disabled equals dummy enabled"?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/25509] Break a cyclic dependency by core packages

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

--- Comment #16 from Frank Ch. Eigler <fche at redhat dot com> ---
(yup, misinterpreted what the "this" was you meant, sorry!)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/25509] Break a cyclic dependency by core packages

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

--- Comment #17 from Mark Wielaard <mark at klomp dot org> ---
Just to be clear, the current setup is:

--enable-debuginfod or --enable-debuginfod=yes: builds all debuginfod
server/client artifacts (requires libcurl)
--disable-debuginfod or --enable-debuginfod=no: builds none of the debuginfod
server/client artifacts.

The proposed patch introduces:

--enable-libdebuginfod or --enable-libdebuginfod=yes: builds the debuginfod
client artifacts
--disable-libdebuginfod or --enable-libdebuginfod=no: don't build the
debuginfod client artifacts
--enable-libdebuginfod=dummy: builds all debuginfod clients artifacts, but the
libdebuginfod.so is just a stub (does not depend on libcurl).
--enable-debuginfod or --enable-debuginfod=yes: builds all debuginfod server
artifacts (requires either --enable-libdebuginfod=yes or
--enable-libdebuginfod=dummy and sqlite3, microhttpd, libarchive)
--disable-debuginfod or --enable-debuginfod=no: build none of the debuginfod
server artifacts.

I am hoping that helps both the Suse use case which would like a
bootstrap/dummy libdebuginfod.so (--enable-libdebuginfod=dummy
--disable-debuginfod) and the Arch use case which is to only have the client
library, but not the debuginfod server (--enable-libdebuginfod
--disable-debuginfod).

I admit that having the combination --enable-libdebuginfod=dummy and
--enable-libdebuginfod is somewhat redundant/non-sensical, but it helps with
(build time) testing. Other testing matrix would imho be as complicated (you'll
get extra install flags or need to setup compile time or runtime environment
variables).

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/25509] Break a cyclic dependency by core packages

[mliska at suse dot cz]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

--- Comment #18 from Martin Liska <mliska at suse dot cz> ---
> I am hoping that helps both the Suse use case which would like a
> bootstrap/dummy libdebuginfod.so (--enable-libdebuginfod=dummy
> --disable-debuginfod)

Yes, I can confirm the suggested scenario will work for us.
Thanks for working on that!

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/25509] Break a cyclic dependency by core packages

[eschwartz at archlinux dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

--- Comment #19 from Mark Wielaard <mark at klomp dot org> ---
(In reply to Eli Schwartz from comment #13)
> I think we'd prefer to have the ability to build libdebuginfod without the
> server.

I believe you get that with --enable-libdebuginfod --disable-debuginfod, but
maybe I don't understand your use case correctly. I tried to explain more fully
in comment #17. Maybe you can try out the patch and let me know if it helps
your use case or, if not, what would make things easier for you?

--- Comment #20 from Eli Schwartz <eschwartz at archlinux dot org> ---
That should be fine, I think.

> At least for Arch Linux, curl is part of the base system while libmicrohttpd is not.

Actually truth be told, by now we moved libmicrohttpd to core, which solves our
immediate problem there. But having --enable-libdebuginfod --disable-debuginfod
would let us remove it again, which might be beneficial. So it's still IMHO a
good-to-have. :)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/25509] Break a cyclic dependency by core packages

[eschwartz at archlinux dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

--- Comment #21 from Eli Schwartz <eschwartz at archlinux dot org> ---
(In reply to Mark Wielaard from comment #17)
> I admit that having the combination --enable-libdebuginfod=dummy and
> --enable-libdebuginfod is somewhat redundant/non-sensical, but it helps with
> (build time) testing. Other testing matrix would imho be as complicated
> (you'll get extra install flags or need to setup compile time or runtime
> environment variables).

Yeah, I just had a thought that you could avoid doing one configure/build/test
with --enable-libdebuginfod=dummy and another with --enable-libdebuginfod.

If the buildsystem logic to have test-time env vars testing both feels too
clumsy and you'd rather just build twice in your test matrix, then of course
you are the boss. :D

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug debuginfod/25509] Break a cyclic dependency by core packages

[mark at klomp dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=25509

Mark Wielaard <mark at klomp dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #22 from Mark Wielaard <mark at klomp dot org> ---
Thanks for all the comments. Pushed to master now.

commit f7f0cdc59a13780938ae3f578955737a75e60ea9
Author: Mark Wielaard <mark@klomp.org>
Date:   Fri Jun 19 19:41:08 2020 +0200

    debuginfod: Add --disable-libdebuginfod and --enable-libdebuginfod=dummy.

    Make it possible to build just the debuginfod client or to create a
    dummy libdebuginfod that doesn't link against libcurl. The dummy library
    can be used for bootstrapping. For testing purposes you can also build
    debuginfod against the dummy libdebuginfod but then the debuginfod
    server will not be able to do delegation.

    Signed-off-by: Mark Wielaard <mark@klomp.org>

-- 
You are receiving this mail because:
You are on the CC list for the bug.