missing symbols with bobcat C++ library on armel

missing symbols with bobcat C++ library on armel

[George Danchev]

Hi guys,

We've been experiencing missing symbols on armel architecture with a C++ 
library for quite some time, and application linking with that library fails 
to link as well.

The bug in question is:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=602817

stealth application build logs:
https://buildd.debian.org/status/package.php?p=stealth&suite=sid
bobcat library build logs:
https://buildd.debian.org/status/package.php?p=bobcat&suite=sid

(you can get the sources of these from Debian archive or from stealth.sf.net 
and bobcat.sf.net resp. )

The problematic part on armel is to be found in this class:
bobcat/multistreambuf/

Missing symbols are probably related to the following usage pattern (all valid 
code in my opinion) of members:
MultiStreambuf::overflow(int c)
MultiStreambuf::sync()
MultiStreambuf::xsputn(char const *buffer, std::streamsize n)

* all are private virtual;
* all implemented in separate translation unit (multistreambuf.ih);
* all inlined
* dtor for MultiStreambuf class is not explicitly defined.

Or any architecture (except armel) these symbols are there; 
on amd64 for instance:

$ objdump -T /usr/lib/libbobcat.so.2.15.02 | c++filt |  objdump -T 
/usr/lib/libbobcat.so.2.15.02 | c++filt | grep MultiStreambuf::sync
0000000000059b20  w   DF .text  0000000000000005  Base        
FBB::MultiStreambuf::sync()

On armel they are not, and the stealth program fails to link as well.
This is observed with g++-4.{5, 6} --std=c++0x.

You can grab the debs of the bobcat library and stealth app from Debian 
archive for any architecture and perform 'ar -x' on it, then unpack data.tar 
to decompose it:

http://ftp.de.debian.org/debian/pool/main/b/bobcat/libbobcat2_2.15.02-1_amd64.deb
http://ftp.de.debian.org/debian/pool/main/b/bobcat/libbobcat2_2.15.02-1_armel.deb

inspect with objdump as well.

The main question is: is this a bug in the toolchain (compiler, linker) or bug 
in the library code itself? Note, that we are not quite looking for 
workarounds (dropping inline, moving it to another translation unit, etc), but 
to understand the root cause behind that. Please, let us know whether we need 
to provide more information?

Also I wonder whether the missing symbols has anything to do with:
http://gcc.gnu.org/faq.html#vtables
but then again why is that only observed on armel?

Please CC as I'm not subscribed to this list. Thank you.

-- 
pub 4096R/0E4BD0AB <people.fccf.net/danchev/key pgp.mit.edu>

Re: missing symbols with bobcat C++ library on armel

[Ian Lance Taylor]

George Danchev <danchev@spnet.net> writes:

> Missing symbols are probably related to the following usage pattern (all valid 
> code in my opinion) of members:
> MultiStreambuf::overflow(int c)
> MultiStreambuf::sync()
> MultiStreambuf::xsputn(char const *buffer, std::streamsize n)
>
> * all are private virtual;
> * all implemented in separate translation unit (multistreambuf.ih);
> * all inlined
> * dtor for MultiStreambuf class is not explicitly defined.

You don't mention it, but the log shows undefined references to the
vtable for FBB:MultiStreambuf.  I have not looked at the code, but this
certainly sounds like a missing key method.  Can you show us just the
declaration of MultiStreambuf from the .h file?  That will let us tell
you what the key method is.  If that key method is not defined anywhere,
then these kinds of errors are possible.

> Also I wonder whether the missing symbols has anything to do with:
> http://gcc.gnu.org/faq.html#vtables
> but then again why is that only observed on armel?

Yes, I'm suggesting basically this.  It could be happening only on armel
because gcc could be making different inlinining decisions on armel.
The error can happen when gcc decides not to inline a function marked
inline, but to instead call the non-inlined instance; if there is no key
method, then gcc never creates a non-inlined instance.  I don't have an
explanation as to why the inlining decisions would be different; it's
possible that that is a compiler bug, or there could be some valid
reason.

Ian

Re: missing symbols with bobcat C++ library on armel

[Frank B. Brokken]

[-- Attachment #1: Type: text/plain, Size: 6266 bytes --]

Dear Ian Lance Taylor, you wrote:
> 
> George Danchev <danchev@spnet.net> writes:
> 
> > Missing symbols are probably related to the following usage pattern (all valid 
> > code in my opinion) of members:
> > MultiStreambuf::overflow(int c)
> > MultiStreambuf::sync()
> > MultiStreambuf::xsputn(char const *buffer, std::streamsize n)
> >
> > * all are private virtual;
> > * all implemented in separate translation unit (multistreambuf.ih);
> > * all inlined
> > * dtor for MultiStreambuf class is not explicitly defined.
> 
> You don't mention it, but the log shows undefined references to the
> vtable for FBB:MultiStreambuf.  I have not looked at the code, but this
> certainly sounds like a missing key method.  Can you show us just the
> declaration of MultiStreambuf from the .h file?  That will let us tell
> you what the key method is.  If that key method is not defined anywhere,
> then these kinds of errors are possible.


Thanks for your reaction to George's questions. 

If you want to have a look at MultiStreambuf's header file, here it is:

-------------------------------------------------------
#ifndef INCLUDED_BOBCAT_MULTISTREAMBUF_
#define INCLUDED_BOBCAT_MULTISTREAMBUF_

#include <cstdio>
#include <streambuf>
#include <vector>
#include <ostream>
#include <string>

namespace FBB
{

class MultiStreambuf: public std::streambuf
{
    public:
        enum Mode
        {
            OFF,                // stream not used
            ON,                 // stream always used
            ONCE,               // stream used until flushed
            RESET,              // stream once used. Set to ONCE to re-use
        };

        class stream            // holds a pointer to a stream and a indicator
        {                       // telling us whether or not to use the stream
            friend class MultiStreambuf;

            std::ostream *d_os;
            Mode          d_mode;

            public:
                stream(std::ostream &os, Mode mode = ON);
                void setMode(Mode mode);
                Mode mode() const;
                std::ostream &ostream();
            private:
                static void setOnce(stream &os);
        };

        typedef std::vector<stream>::iterator iterator;
        typedef std::vector<stream>::const_iterator const_iterator;

    private:
        std::string d_buffer;
        std::vector<stream> d_os;

    public:
        MultiStreambuf() = default;
        explicit MultiStreambuf(std::ostream &os, Mode mode = ON);
        explicit MultiStreambuf(std::vector<stream> const &osvector);

        void insert(std::ostream &os, Mode mode = ON);
        void insert(std::vector<stream> const &os);
        iterator begin();
        iterator end();
        const_iterator begin() const;
        const_iterator end() const;
        void setOnce();             // reset all `RESET' modes to `ONCE'
        
    protected:
        int pSync();

    private:
        virtual int overflow(int c);
        virtual std::streamsize xsputn(char const *buffer, std::streamsize n);
        virtual int sync();

        struct Insert
        {
            std::string &buffer;
            bool ok;
        };
        static void insertStruct(stream &os, Insert &insert); 
};

inline MultiStreambuf::stream::stream(std::ostream &os, Mode mode)
:
    d_os(&os),
    d_mode(mode)
{}

inline void MultiStreambuf::stream::setMode(Mode mode)
{
    d_mode = mode;
}

inline MultiStreambuf::Mode MultiStreambuf::stream::mode() const
{
    return d_mode;
}

inline std::ostream &MultiStreambuf::stream::ostream()
{
    return *d_os;
}

inline void MultiStreambuf::stream::setOnce(stream &os)
{
    if (os.d_mode == RESET)
        os.d_mode = ONCE;
}

inline MultiStreambuf::MultiStreambuf(std::ostream &os, Mode mode)
{
    insert(os, mode);
}

inline MultiStreambuf::MultiStreambuf(std::vector<stream> const &osvector)
{
    insert(osvector);
}

inline void MultiStreambuf::insert(std::ostream &os, Mode mode)
{
    d_os.push_back(stream(os, mode));
}

inline void MultiStreambuf::insert(std::vector<stream> const &os)
{
    d_os.insert(d_os.end(), os.begin(), os.end());
}

inline MultiStreambuf::iterator MultiStreambuf::begin()
{
    return d_os.begin();
}

inline MultiStreambuf::iterator MultiStreambuf::end()
{
    return d_os.end();
}

inline MultiStreambuf::const_iterator MultiStreambuf::begin() const
{
    return d_os.begin();
}

inline MultiStreambuf::const_iterator MultiStreambuf::end() const
{
    return d_os.end();
}

} // namespace FBB
        
#endif
-------------------------------------------------------


You won't see the implementations of the private virtual members here, as they
are provided in the following header file, which is included by
MultiStreambuf's sources:

-------------------------------------------------------
#include "multistreambuf"

#include <algorithm>
#include <bobcat/fnwrap>

using namespace FBB;

inline std::streamsize MultiStreambuf::xsputn(char const *buffer, 
                                              std::streamsize n)
{
    d_buffer.append(buffer, n);
    return n;
}

inline int MultiStreambuf::sync()
{
    return pSync();
}

inline int MultiStreambuf::overflow(int c)
{
    if (c == EOF)
        pSync();
    else 
        d_buffer += c;

    return c;
}
-------------------------------------------------------


Since it looks like armel's compiler skips the inline definitions of virtual
members when compiling for a shared library, we're in the process of removing
all inline definitions of virtual members, replacing them by non-inline
definitions in separate source files. From tests done so far by George Danchev
we get the impression that this might very well solve (or bypass?) the issue.

If the above is not what you wanted me to send you, please let me know.

Cheers,

-- 
    Frank B. Brokken
    Center for Information Technology, University of Groningen
    (+31) 50 363 9281 
    Public PGP key: http://pgp.surfnet.nl
    Key Fingerprint: DF32 13DE B156 7732 E65E  3B4D 7DB2 A8BE EAE4 D8AA

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 482 bytes --]

Re: missing symbols with bobcat C++ library on armel

[Ian Lance Taylor]

"Frank B. Brokken" <f.b.brokken@rug.nl> writes:

> class MultiStreambuf: public std::streambuf
> {
>     public:
>         enum Mode
>         {
>             OFF,                // stream not used
>             ON,                 // stream always used
>             ONCE,               // stream used until flushed
>             RESET,              // stream once used. Set to ONCE to re-use
>         };
>
>         class stream            // holds a pointer to a stream and a indicator
>         {                       // telling us whether or not to use the stream
>             friend class MultiStreambuf;
>
>             std::ostream *d_os;
>             Mode          d_mode;
>
>             public:
>                 stream(std::ostream &os, Mode mode = ON);
>                 void setMode(Mode mode);
>                 Mode mode() const;
>                 std::ostream &ostream();
>             private:
>                 static void setOnce(stream &os);
>         };
>
>         typedef std::vector<stream>::iterator iterator;
>         typedef std::vector<stream>::const_iterator const_iterator;
>
>     private:
>         std::string d_buffer;
>         std::vector<stream> d_os;
>
>     public:
>         MultiStreambuf() = default;
>         explicit MultiStreambuf(std::ostream &os, Mode mode = ON);
>         explicit MultiStreambuf(std::vector<stream> const &osvector);
>
>         void insert(std::ostream &os, Mode mode = ON);
>         void insert(std::vector<stream> const &os);
>         iterator begin();
>         iterator end();
>         const_iterator begin() const;
>         const_iterator end() const;
>         void setOnce();             // reset all `RESET' modes to `ONCE'
>         
>     protected:
>         int pSync();
>
>     private:
>         virtual int overflow(int c);
>         virtual std::streamsize xsputn(char const *buffer, std::streamsize n);
>         virtual int sync();
>
>         struct Insert
>         {
>             std::string &buffer;
>             bool ok;
>         };
>         static void insertStruct(stream &os, Insert &insert); 
> };


OK, so the key method should be overflow.



> You won't see the implementations of the private virtual members here, as they
> are provided in the following header file, which is included by
> MultiStreambuf's sources:
>
> -------------------------------------------------------
> #include "multistreambuf"
>
> #include <algorithm>
> #include <bobcat/fnwrap>
>
> using namespace FBB;
>
> inline std::streamsize MultiStreambuf::xsputn(char const *buffer, 
>                                               std::streamsize n)
> {
>     d_buffer.append(buffer, n);
>     return n;
> }
>
> inline int MultiStreambuf::sync()
> {
>     return pSync();
> }
>
> inline int MultiStreambuf::overflow(int c)
> {
>     if (c == EOF)
>         pSync();
>     else 
>         d_buffer += c;
>
>     return c;
> }
> -------------------------------------------------------


This approach will only work if you guarantee that every piece of code
that creates a new MultiStreambuf includes that header file.


> Since it looks like armel's compiler skips the inline definitions of virtual
> members when compiling for a shared library, we're in the process of removing
> all inline definitions of virtual members, replacing them by non-inline
> definitions in separate source files. From tests done so far by George Danchev
> we get the impression that this might very well solve (or bypass?) the issue.

Yes, that should work.

Ian

Re: missing symbols with bobcat C++ library on armel

[Frank B. Brokken]

[-- Attachment #1: Type: text/plain, Size: 2550 bytes --]

Dear Ian Lance Taylor, you wrote:
> 
> "Frank B. Brokken" <f.b.brokken@rug.nl> writes:
> 
> > class MultiStreambuf: public std::streambuf
> > ...
> 
> > You won't see the implementations of the private virtual members here, as they
> > are provided in the following header file, which is included by
> > MultiStreambuf's sources: 
> >
> > -------------------------------------------------------
> > #include "multistreambuf"
> >
> > #include <algorithm>
> > #include <bobcat/fnwrap>
> >
> > using namespace FBB;
> >
> > inline std::streamsize MultiStreambuf::xsputn(char const *buffer, 
> >                                               std::streamsize n)
> > ...
> > -------------------------------------------------------
> 
> 
> This approach will only work if you guarantee that every piece of code
> that creates a new MultiStreambuf includes that header file.

I'm not sure if I follow you here. All MultiStreambuf sources include the
above header, and no other (external to MultiStreambuf) functions need/have
access to MultiStreambuf's private members. 

So I guess you do not mean by 'creates a new MultiStreambuf' constructions
like:
    void fun()
    {
        MultiStreambuf ms;
        MultiStreambuf *msp = new MultiStreambuf;
    }

Btw, this approach, which uses an 'internal header' to minimize the
requirements that need to be specified in class's header files is extensively
documented in section 7.9 (near the end) of my C++ Annotations
(cf. http://cppannotations.sourceforge.net/cppannotations/html/cplusplus07.html).
If you think the approach outlined there is somehow flawed, I would appreciate
receiving your thoughts.



> > Since it looks like armel's compiler skips the inline definitions of virtual
> > members when compiling for a shared library, we're in the process of removing
> > all inline definitions of virtual members, replacing them by non-inline
> > definitions in separate source files. From tests done so far by George Danchev
> > we get the impression that this might very well solve (or bypass?) the issue.
> 
> Yes, that should work.


The new library is already available, and I think George will test it out on
armel RSN. We'll see what he'll find.

But in any case: thanks for your help, so far!

Cheers,

-- 
    Frank B. Brokken
    Center for Information Technology, University of Groningen
    (+31) 50 363 9281 
    Public PGP key: http://pgp.surfnet.nl
    Key Fingerprint: DF32 13DE B156 7732 E65E  3B4D 7DB2 A8BE EAE4 D8AA

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 482 bytes --]

Re: missing symbols with bobcat C++ library on armel

[Ian Lance Taylor]

"Frank B. Brokken" <f.b.brokken@rug.nl> writes:

> Dear Ian Lance Taylor, you wrote:
>
>> This approach will only work if you guarantee that every piece of code
>> that creates a new MultiStreambuf includes that header file.
>
> I'm not sure if I follow you here. All MultiStreambuf sources include the
> above header, and no other (external to MultiStreambuf) functions need/have
> access to MultiStreambuf's private members. 
>
> So I guess you do not mean by 'creates a new MultiStreambuf' constructions
> like:
>     void fun()
>     {
>         MultiStreambuf ms;
>         MultiStreambuf *msp = new MultiStreambuf;
>     }

That is pretty much exactly what I do mean, actually.  There is some
more information at

http://gcc.gnu.org/onlinedocs/gcc-4.6.0/gcc/Vague-Linkage.html

Note specifically: "Make sure that any inline virtuals are declared
inline in the class body, even if they are not defined there."  You
aren't currently doing that.  It's possible that that by itself might
fix this problem; I'm not sure.


> Btw, this approach, which uses an 'internal header' to minimize the
> requirements that need to be specified in class's header files is extensively
> documented in section 7.9 (near the end) of my C++ Annotations
> (cf. http://cppannotations.sourceforge.net/cppannotations/html/cplusplus07.html).
> If you think the approach outlined there is somehow flawed, I would appreciate
> receiving your thoughts.

The notes there don't seem to cover this case: the case in which all
virtual methods are inline.

Ian

Re: missing symbols with bobcat C++ library on armel

[Axel Freyn]

On Wed, May 25, 2011 at 11:12:02AM -0700, Ian Lance Taylor wrote:
> "Frank B. Brokken" <f.b.brokken@rug.nl> writes:
> 
> > Dear Ian Lance Taylor, you wrote:
> >
> >> This approach will only work if you guarantee that every piece of code
> >> that creates a new MultiStreambuf includes that header file.
> >
> > I'm not sure if I follow you here. All MultiStreambuf sources include the
> > above header, and no other (external to MultiStreambuf) functions need/have
> > access to MultiStreambuf's private members. 
> >
> > So I guess you do not mean by 'creates a new MultiStreambuf' constructions
> > like:
> >     void fun()
> >     {
> >         MultiStreambuf ms;
> >         MultiStreambuf *msp = new MultiStreambuf;
> >     }
> 
> That is pretty much exactly what I do mean, actually.  There is some
> more information at
> 
> http://gcc.gnu.org/onlinedocs/gcc-4.6.0/gcc/Vague-Linkage.html
> 
> Note specifically: "Make sure that any inline virtuals are declared
> inline in the class body, even if they are not defined there."  You
> aren't currently doing that.  It's possible that that by itself might
> fix this problem; I'm not sure.

Nevertheless, I'm also wondering also about another situation, which
might occur (or is it the same ?):

the function "overflow" is defined already as protected in the
base-class std::streambuf, from which MultiStreambuf is derived
(public).

Now, what would happen if (on armel):
 - your MultiStreambuf-class and all derived classes ONLY call
   "this->overflow" in situations where g++ can guarantee that "this" is
   a MultiStreambuf, and thus inline the code
   ==> it is possible to generate the library without exporting
   "MultiStreambuf::overflow"
   (i'm not sure: would that be OK from G++, or a violation of the
   standard?)
 - but when the library is used, an object of type MultiStreambuf is
   used through the std::streambuf - interface, such that a call to
   overflow is generated -- which can't be satisfied due to the missing
   exported function.

In fact, I tried a minimal example of this situation, where I also get
an error on amd64 (Debian, g++-4.4.5) with the following code:

base.h:
class Base{ // base class defining an inline virtual function as public
  public:
    inline virtual int operator()()const{ return 2; }
    int get2()const{ return this->operator()(); }
    virtual ~Base(){};
};

derived.h:
#include "base.h" // derived class, defining the function as private &
	// using it only once
class Derived : public Base{
  public:
    ~Derived(){}
    int get()const;
  private:
    inline virtual int operator()()const;
};

derived.cc: 
#include "derived.h" // separate compile unit, where the type of "this"
		    // is well-defined ==> operator() can be inlined
int Derived::get()const{  	 return this->operator()();}
int Derived::operator()()const{  return 4;	}

test.cc:
#include "derived.h"	// access the virtual function of Derived
			// through "get2"
#include <iostream>
int main(){
  Base *b = new Derived();
  std::cout << b->get2() << std::endl;
}

Compiling with "g++ test.cc derived.cc" gives a warning:
warning: inline function ‘virtual int Derived::operator()() const’ used but never defined
and a linker error:
/tmp/ccU781xY.o:(.rodata._ZTV7Derived[vtable for Derived]+0x10): undefined reference to `Derived::operator()() const'


Maybe in the MultiStreambuf-case something similar is happen?

Axel

Re: missing symbols with bobcat C++ library on armel

[Frank B. Brokken]

[-- Attachment #1: Type: text/plain, Size: 1573 bytes --]

Dear Ian Lance Taylor, you wrote:

> > So I guess you do not mean by 'creates a new MultiStreambuf' constructions
> > like:
> >     void fun()
> >     {
> >         MultiStreambuf ms;
> >         MultiStreambuf *msp = new MultiStreambuf;
> >     }
> 
> That is pretty much exactly what I do mean, actually.  There is some
> more information at
> 
> http://gcc.gnu.org/onlinedocs/gcc-4.6.0/gcc/Vague-Linkage.html

Interesting. I like the term 'vague linkage'. Apparently a vtable isn't by
default/always/on different architectures filled with the addresses of virtual
functions if these functions are defined inline and not explicitly called from
other members. But the repair seems to be simple: as virtual functions need
addresses simply avoid using inline definitions for them. Remarkable that the
problem only shows up on the armel architecture. But better now than never,
isn't it?

> > If you think the approach outlined there is somehow flawed, I would appreciate
> > receiving your thoughts.
> 
> The notes there don't seem to cover this case: the case in which all
> virtual methods are inline.

That's a nice subtlety that neatly blends in with the practice of avoiding
inline virtual members. Something to add to the Annotations, I guess.

Thanks for your quick replies and supportive comments!


-- 
    Frank B. Brokken
    Center for Information Technology, University of Groningen
    (+31) 50 363 9281 
    Public PGP key: http://pgp.surfnet.nl
    Key Fingerprint: DF32 13DE B156 7732 E65E  3B4D 7DB2 A8BE EAE4 D8AA

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 482 bytes --]

Re: missing symbols with bobcat C++ library on armel

[George Danchev]

On Wednesday 25 May 2011 17:47:04 Ian Lance Taylor wrote:

Hi,

> > Since it looks like armel's compiler skips the inline definitions of
> > virtual members when compiling for a shared library, we're in the
> > process of removing all inline definitions of virtual members, replacing
> > them by non-inline definitions in separate source files. From tests done
> > so far by George Danchev we get the impression that this might very well
> > solve (or bypass?) the issue.
> 
> Yes, that should work.

Dropping inline from virtual methods worked indeed:

build daemons logs:
https://buildd.debian.org/status/fetch.php?pkg=bobcat&arch=armel&ver=2.16.00-1&stamp=1306411374
https://buildd.debian.org/status/fetch.php?pkg=stealth&arch=armel&ver=2.01.00-3&stamp=1306426051

However, it is still unclear why the compiler would behave differently on 
armel, and not simply ignoring the inline request as it does on other 
architectures. This is a subtle behavior and smells like a bug in the compiler 
itself. Should we prepare a sample code manifesting the issue and file a bug 
against gcc?

Thanks for the help so far.

-- 
pub 4096R/0E4BD0AB <people.fccf.net/danchev/key pgp.mit.edu>

Re: missing symbols with bobcat C++ library on armel

[Ian Lance Taylor]

George Danchev <danchev@spnet.net> writes:

> However, it is still unclear why the compiler would behave differently on 
> armel, and not simply ignoring the inline request as it does on other 
> architectures. This is a subtle behavior and smells like a bug in the compiler 
> itself. Should we prepare a sample code manifesting the issue and file a bug 
> against gcc?

It's normal for gcc to make different inlining decisions on different
targets.  It is odd that this only happens on the one target, and it
couldn't hurt to set up a bug report, but it's reasonably likely that it
will turn out to simply be some ordinary difference in code generation.
It's also possible that there is a bug in the inliner; I don't know.

Ian