[Bug lto/107078] New: LTO is causing that firebird build is core dumping

[Bug lto/107078] New: LTO is causing that firebird build is core dumping

[kloczko.tomasz at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

            Bug ID: 107078
           Summary: LTO is causing that firebird build is core dumping
           Product: gcc
           Version: unknown
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: lto
          Assignee: unassigned at gcc dot gnu.org
          Reporter: kloczko.tomasz at gmail dot com
                CC: marxin at gcc dot gnu.org
  Target Milestone: ---

firebird  4.0.2 and gcc-c++ 12.2.1-2 from fedora rawhide.
Build is crashing with sigsegv on execution of the linked binary

Quote from firebird maintainer comment:
"That's almost for sure (99%) pure virtual function call. Usual bug for
dtor/ctor - but I doubt half-constructed node was passed into node
copier. I try to lower compiler optimization level first of all in such
cases (taking into an account that this stuff builds and works OK for
some years)."

More details is in https://github.com/FirebirdSQL/firebird/issues/7308

I found that fedora spec file has disabled LTO as well with comment

# firebird is mis-compiled when LTO is enabled. A root
# cause analysis has not yet been completed. Reported upstream.
# Disable LTO for now

[Bug lto/107078] LTO is causing that firebird build is core dumping

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Version|unknown                     |12.2.1

--- Comment #1 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Try -O2 -fno-strict-aliasing with -flto.

[Bug lto/107078] LTO is causing that firebird build is core dumping

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

--- Comment #2 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
(In reply to Tomasz Kłoczko from comment #0)
> I found that fedora spec file has disabled LTO as well with comment
> 
> # firebird is mis-compiled when LTO is enabled. A root
> # cause analysis has not yet been completed. Reported upstream.
> # Disable LTO for now

https://src.fedoraproject.org/rpms/firebird/c/4c04d2ce892bcf0d2e7015dbeeb6069789ed4396

[Bug lto/107078] LTO is causing that firebird build is core dumping

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

--- Comment #3 from Richard Biener <rguenth at gcc dot gnu.org> ---
IIRC there are also unfixed devirt issues (and some not backported yet?)

[Bug lto/107078] LTO is causing that firebird build is core dumping

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
     Ever confirmed|0                           |1
             Status|UNCONFIRMED                 |WAITING
   Last reconfirmed|                            |2022-09-29

--- Comment #4 from Martin Liška <marxin at gcc dot gnu.org> ---
It may be related to -fno-lifetime-dse, can you please try this option if it
helps?

[Bug lto/107078] LTO is causing that firebird build is core dumping

[kloczko.tomasz at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

--- Comment #5 from Tomasz Kłoczko <kloczko.tomasz at gmail dot com> ---
FWD of the firebird developer from
https://github.com/FirebirdSQL/firebird/issues/7308#issuecomment-1262043660

"Firebird (that code left from interbase times) traditionally zeroes
memory when allocating a lot of internal data structures using function
like calloc(). When moving from C to C++ it was wrapped into operator
new of some base class in order to avoid type casts, be able to use
ctors and a lot of other c++ features. 20 years ago it was fine. Some
years ago an optimization removing any data initalization in new (data
returned by it is not initialized according to standard). By itself it
did not affect the code - our calloc() is placed into separate file,
it's not inline. But together with cross-file optimization... we get
what you've seen.

Certainly correct fix is to move memory initialization into ctor - but
that was not done yet.

May be there some more issues with LTO, I did not learn it deeper."

[Bug lto/107078] LTO is causing that firebird build is core dumping

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

--- Comment #6 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
(In reply to Tomasz Kłoczko from comment #5)
> FWD of the firebird developer from
> https://github.com/FirebirdSQL/firebird/issues/7308#issuecomment-1262043660
> 
> "Firebird (that code left from interbase times) traditionally zeroes
> memory when allocating a lot of internal data structures using function
> like calloc(). When moving from C to C++ it was wrapped into operator
> new of some base class in order to avoid type casts, be able to use
> ctors and a lot of other c++ features. 20 years ago it was fine. Some
> years ago an optimization removing any data initalization in new (data
> returned by it is not initialized according to standard). By itself it
> did not affect the code - our calloc() is placed into separate file,
> it's not inline. But together with cross-file optimization... we get
> what you've seen.
> 
> Certainly correct fix is to move memory initialization into ctor - but
> that was not done yet.
> 
> May be there some more issues with LTO, I did not learn it deeper."

Then almost certainly -fno-lifetime-dse will help.

[Bug lto/107078] LTO is causing that firebird build is core dumping

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

--- Comment #7 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Then as documented, -fno-lifetime-dse or -flifetime-dse=1 can be a temporary
workaround, but as has been said, such code is undefined behavior and should be
fixed in the application.
See
https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html#index-flifetime-dse

[Bug lto/107078] LTO is causing that firebird build is core dumping

[kloczko.tomasz at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

--- Comment #8 from Tomasz Kłoczko <kloczko.tomasz at gmail dot com> ---
(In reply to Andrew Pinski from comment #6)
[..]
> Then almost certainly -fno-lifetime-dse will help.

Tested -O2 + LTO + -fno-lifetime-dse and it crashes.

[Bug lto/107078] LTO is causing that firebird build is core dumping

[kloczko.tomasz at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

--- Comment #9 from Tomasz Kłoczko <kloczko.tomasz at gmail dot com> ---
(In reply to Jakub Jelinek from comment #7)
> Then as documented, -fno-lifetime-dse or -flifetime-dse=1 can be a temporary
> workaround, but as has been said, such code is undefined behavior and should
> be fixed in the application.
> See
> https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html#index-flifetime-dse

I just realised that +year ago working on my packages I've started removing all
hardcodes in source tree warnig and optimisation flags to be able to controll
that on rpm build layer and when I've removed in mesa -flifetime-dse=1 it
started crashing as well.

[Bug lto/107078] LTO is causing that firebird build is core dumping

[kloczko.tomasz at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

--- Comment #10 from Tomasz Kłoczko <kloczko.tomasz at gmail dot com> ---
Tested -O2 + LTO + -flifetime-dse=1 and it crashes.

[Bug lto/107078] LTO is causing that firebird build is core dumping

[kloczko.tomasz at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

--- Comment #11 from Tomasz Kłoczko <kloczko.tomasz at gmail dot com> ---
Tested -O2 + LTO + -flifetime-dse=1 and it crashes.

[Bug lto/107078] LTO is causing that firebird build is core dumping

[kloczko.tomasz at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

--- Comment #12 from Tomasz Kłoczko <kloczko.tomasz at gmail dot com> ---
Any update?🤔

[Bug lto/107078] LTO is causing that firebird build is core dumping

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|unassigned at gcc dot gnu.org      |marxin at gcc dot gnu.org
             Status|WAITING                     |ASSIGNED

--- Comment #13 from Martin Liška <marxin at gcc dot gnu.org> ---
I can take a look in this stage3 phase.

[Bug lto/107078] LTO is causing that firebird build is core dumping

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |WAITING

--- Comment #14 from Martin Liška <marxin at gcc dot gnu.org> ---
Can you please provide the exact steps on how to configure the project with the
corresponding options and run the crashing command? I would like to attach a
gdb to it.

[Bug lto/107078] LTO is causing that firebird build is core dumping

[kloczko.tomasz at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

--- Comment #15 from Tomasz Kłoczko <kloczko.tomasz at gmail dot com> ---
(In reply to Martin Liška from comment #14)
> Can you please provide the exact steps on how to configure the project with
> the corresponding options and run the crashing command? I would like to
> attach a gdb to it.

- take fedora firebird src.rpm and unpack it
- undomment ins spec file disable LTO
- build package

[Bug lto/107078] LTO is causing that firebird build is core dumping

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

--- Comment #16 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to Tomasz Kłoczko from comment #15)
> (In reply to Martin Liška from comment #14)
> > Can you please provide the exact steps on how to configure the project with
> > the corresponding options and run the crashing command? I would like to
> > attach a gdb to it.
> 
> - take fedora firebird src.rpm and unpack it
> - undomment ins spec file disable LTO
> - build package

I speak about direct use of the release tarball as I have almost no experience
with Fedora packaging system. I can configure it and build and I also see the
Segmenetation fault. However, I can't run the command manually:

/tmp/firebird-4.0.2/gen/Release/firebird/bin/isql -q -i
/tmp/firebird-4.0.2/src/dbs/metadata.sql
can't format message 17:0 -- message file /usr/local/firebird/firebird.msg not
found
Unable to complete network request to host "localhost".
-Failed to establish a connection.
can't format message 17:120 -- message file /usr/local/firebird/firebird.msg
not found

[Bug lto/107078] LTO is causing that firebird build is core dumping

[kloczko.tomasz at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

--- Comment #17 from Tomasz Kłoczko <kloczko.tomasz at gmail dot com> ---
> /tmp/firebird-4.0.2/gen/Release/firebird/bin/isql -q -i
> /tmp/firebird-4.0.2/src/dbs/metadata.sql
> can't format message 17:0 -- message file /usr/local/firebird/firebird.msg
> not found
> Unable to complete network request to host "localhost".
> -Failed to establish a connection.
> can't format message 17:120 -- message file /usr/local/firebird/firebird.msg
> not found

This issue has nothing to do with running firebird.
This issue is about compiler issue on linking firebird binary linked with LTO.

[Bug lto/107078] LTO is causing that firebird build is core dumping

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

--- Comment #18 from Martin Liška <marxin at gcc dot gnu.org> ---
> This issue is about compiler issue on linking firebird binary linked with
> LTO.

Good, so it's isql command that crashes:

make gpre
make[2]: Entering directory '/tmp/firebird-4.0.2/gen'
rm -f metadata.fdb
/tmp/firebird-4.0.2/gen/Release/firebird/bin/isql -q -i
/tmp/firebird-4.0.2/src/dbs/metadata.sql
make[2]: *** [Makefile:441: metadata.fdb] Segmentation fault (core dumped)
make[2]: *** Deleting file 'metadata.fdb'
make[2]: Leaving directory '/tmp/firebird-4.0.2/gen'
make[1]: *** [Makefile:292: master_process] Error 2

And my question is, how run it under gdb?

[Bug lto/107078] LTO is causing that firebird build is core dumping

[sam at gentoo dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

Sam James <sam at gentoo dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |sam at gentoo dot org

--- Comment #19 from Sam James <sam at gentoo dot org> ---
I get some ODR warnings in gpre:
```
/tmp/firebird/Firebird-4.0.2.2816-0/src/gpre/pat.cpp:60:21: warning: type
‘struct ops’ violates the C++ One Definition Rule [-Wodr]
   60 | static const struct ops
      |                     ^
/tmp/firebird/Firebird-4.0.2.2816-0/src/gpre/sqe.cpp:124:8: note: a different
type is defined in another translation unit
  124 | struct ops
      |        ^
/tmp/firebird/Firebird-4.0.2.2816-0/src/gpre/pat.cpp:62:15: note: the first
difference of corresponding definitions is field ‘ops_type’
   62 |         pat_t ops_type;
      |               ^
/tmp/firebird/Firebird-4.0.2.2816-0/src/gpre/sqe.cpp:126:15: note: a field with
different name is defined in another translation unit
  126 |         nod_t rel_op;
      |               ^
lto-wrapper: warning: using serial compilation of 11 LTRANS jobs
lto-wrapper: note: see the ‘-flto’ option documentation for more information
rm -f /tmp/firebird/Firebird-4.0.2.2816-0/gen/Release/firebird/bin/gpre_current
```

--

Anyway, as a hack, try this:
```
diff --git a/gen/Makefile b/gen/Makefile
index a121c31..8eabd44 100644
--- a/gen/Makefile
+++ b/gen/Makefile
@@ -438,7 +438,7 @@ yachts.lnk:         metadata.fdb

 metadata.fdb:  $(RUN_ISQL) $(SRC_ROOT)/dbs/metadata.sql
        -$(RM) $@
-       $(RUN_ISQL) -q -i $(SRC_ROOT)/dbs/metadata.sql
+       gdbserver 127.0.0.1:1234 -- $(RUN_ISQL) -q -i
$(SRC_ROOT)/dbs/metadata.sql
        $(RUN_GFIX) -mode read_only $@
        $(CHMOD) 0444 $@

@@ -467,9 +467,9 @@ security.fdb:       $(SRC_ROOT)/dbs/security.sql

 msg.timestamp: $(MSG_FILES)
        -$(RM) msg.fdb
-       echo create database \'msg.fdb\'\; | $(RUN_ISQL)
+       echo create database \'msg.fdb\'\; | gdbserver 127.0.0.1:1235 --
$(RUN_ISQL)
        $(RUN_GFIX) -write async msg.fdb
-       for sql in $(MSG_FILES); do (echo $$sql; $(RUN_ISQL) -i $$sql msg.fdb)
|| exit; done
+       for sql in $(MSG_FILES); do (echo $$sql; gdbserver 127.0.0.1:1236 --
$(RUN_ISQL) -i $$sql msg.fdb) || exit; done
        $(TOUCH) $@


```

[Bug lto/107078] LTO is causing that firebird build is core dumping

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |INVALID
             Status|WAITING                     |RESOLVED

--- Comment #20 from Martin Liška <marxin at gcc dot gnu.org> ---
Thanks, now I can reproduce and it:

Thread 1 "isql" received signal SIGSEGV, Segmentation fault.
0x00007ffff634da54 in Firebird::MemPool::releaseMemory (flagExtent=false,
object=<optimized out>) at
/home/marxin/Programming/firebird-4.0.2/src/common/classes/alloc.cpp:2402
2402                    pool->releaseBlock(block, !flagExtent);
(gdb) bt
#0  0x00007ffff634da54 in Firebird::MemPool::releaseMemory (flagExtent=false,
object=<optimized out>) at
/home/marxin/Programming/firebird-4.0.2/src/common/classes/alloc.cpp:2402
#1  Firebird::MemPool::deallocate (block=<optimized out>) at
/home/marxin/Programming/firebird-4.0.2/src/common/classes/alloc.cpp:2683
#2  Firebird::MemPool::globalFree (block=<optimized out>) at
/home/marxin/Programming/firebird-4.0.2/src/common/classes/alloc.cpp:2671
#3  Firebird::MemoryPool::globalFree (block=<optimized out>) at
/home/marxin/Programming/firebird-4.0.2/src/common/classes/alloc.cpp:2836

it crashes because releaseMemory is called with object == NULL:

void MemPool::releaseMemory(void* object, bool flagExtent) FB_NOTHROW
{
        if (object)
        {

LTO create a .part clone where it assumes object can't be null. That's true for
'this' pointer, which should never be null.

can be fixed with:
-O2 -flto=auto -flifetime-dse=1 -fno-delete-null-pointer-checks

please build the software with -fsanitize=undefined,address and investigate
where it violates that.

[Bug lto/107078] LTO is causing that firebird build is core dumping

[kloczko.tomasz at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

--- Comment #21 from Tomasz Kłoczko <kloczko.tomasz at gmail dot com> ---
On emore time.
You are commenting under GNU C Compilet issue during linking firebird binaries
linking.
*COMPILER* (not firebird) is core dumping.

Please discuss firebird issue opening ticket on
https://github.com/FirebirdSQL/firebird/issues/

[Bug lto/107078] LTO is causing that firebird build is core dumping

[sam at gentoo dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

--- Comment #22 from Sam James <sam at gentoo dot org> ---
(In reply to Tomasz Kłoczko from comment #21)
> On emore time.
> You are commenting under GNU C Compilet issue during linking firebird
> binaries linking.
> *COMPILER* (not firebird) is core dumping.
> 
> Please discuss firebird issue opening ticket on
> https://github.com/FirebirdSQL/firebird/issues/

It is definitely isql which is segfaulting. The Makefile uses a binary just
built to try do some bits and the binary-just-built crashes.

[Bug lto/107078] LTO is causing that firebird build is core dumping

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107078

--- Comment #23 from Martin Liška <marxin at gcc dot gnu.org> ---
(In reply to Tomasz Kłoczko from comment #21)
> On emore time.
> You are commenting under GNU C Compilet issue during linking firebird
> binaries linking.
> *COMPILER* (not firebird) is core dumping.

I see isql crashing, not GCC compiler! Show me compiler invocation that
crashes.

> 
> Please discuss firebird issue opening ticket on
> https://github.com/FirebirdSQL/firebird/issues/

I provided an explanation of why the built binary with LTO is crashing because
it has undefined behavior. Feel free to take any comments I provided and put
that to:
https://github.com/FirebirdSQL/firebird/issues/7308