[Bug libc/23960] [2.28 Regression]: New getdents{64} implementation breaks qemu-user

["danny.milo at gmail dot com" <sourceware-bugzilla@sourceware.org>]

https://sourceware.org/bugzilla/show_bug.cgi?id=23960

Danny Milosavljevic <danny.milo at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |danny.milo at gmail dot com

--- Comment #58 from Danny Milosavljevic <danny.milo at gmail dot com> ---
The title should be changed to be more general because this does not only break
qemu-user.  In fact, mentioning qemu would make it seem that it can be fixed in
qemu-user--which it can't.

The same happens on aarch64 if running armhf executables (no qemu anywhere).

More details on
https://lists.gnu.org/archive/html/guix-patches/2020-10/msg00059.html .

In order to test, patch fuse-2.9.9 like this:

diff -ru orig/fuse-2.9.9/lib/fuse_lowlevel.c fuse-2.9.9/lib/fuse_lowlevel.c
--- orig/fuse-2.9.9/lib/fuse_lowlevel.c 1970-01-01 01:00:01.000000000 +0100
+++ fuse-2.9.9/lib/fuse_lowlevel.c      2020-09-25 17:09:26.744879224 +0200
@@ -257,7 +257,7 @@
        struct fuse_dirent *dirent = (struct fuse_dirent *) buf;

        dirent->ino = stbuf->st_ino;
-       dirent->off = off;
+       dirent->off = off | 0x1234567890123;
        dirent->namelen = namelen;
        dirent->type = (stbuf->st_mode & 0170000) >> 12;
        strncpy(dirent->name, name, namelen);

Then run make.

mkdir -p /tmp/foo
Then invoke:
   examples/hello_ll /tmp/foo

Then run this test program:

#include <stdio.h>
#include <errno.h>
#include <assert.h>
#include <dirent.h>
#if defined( __ILP32__)
#warning ILP32
#endif

int main() {
        DIR* d;
        struct dirent* ent;
        d = opendir("/tmp/foo");
        if (d == NULL) {
                perror("opendir");
                return 1;
        }
        errno = 0;
        assert(sizeof(ent->d_off) == sizeof(off_t));
        while ((ent = readdir(d)) != NULL) {
                printf("%llX %s\n", (unsigned long long) ent->d_off,
ent->d_name);
                if (ent->d_off > 0xffffffff)
                        printf("BIG\n");
        }
        if (errno)
                perror("readdir");
        return sizeof(off_t);
}

Compile once with -D_FILE_OFFSET_BITS=64, once with -D_FILE_OFFSET_BITS=32 and
once with no -D_FILE_OFFSET_BITS.

You get this result:

system   _FILE_OFFSET_BITS off_t   d_off-sizeof   d_off-values
---------------------------------------------------------------
x86_64   -                 8 Byte  8 Byte         8 Byte
i686     -                 4 Byte  4 Byte         4 Byte
i686     64                8 Byte  8 Byte         FAIL*
i686     32                4 Byte  4 Byte         FAIL*
armhf    -                 4 Byte  4 Byte         FAIL*
armhf    64                8 Byte  8 Byte         8 Byte
armhf    32                4 Byte  4 Byte         FAIL*
a64armhf -                 4 Byte  4 Byte         FAIL*
a64armhf 64                8 Byte  8 Byte         8 Byte
a64armhf 32                4 Byte  4 Byte         FAIL*
aarch64  -                 8 Byte  8 Byte         8 Byte

*: Using FUSE filesystem with big d_off value.

None of those tests were done with qemu.  They were all native.

"i686" means "i686 on x86_64".

I argue that the only safe way to fix that once and for all is to use
_FILE_OFFSET_BITS=64 on 32 bits.

I would implore glibc maintainers to mandate choosing a _FILE_OFFSET_BITS and
fail compilation otherwise.  15 years of migration to LFS is more than enough.

Patch to do that:  At the end of dirent.h, add:

#ifndef _LIBC
#if __SIZEOF_LONG__ < 8
#ifndef __USE_FILE_OFFSET64
#if defined(_FILE_OFFSET_BITS) && _FILE_OFFSET_BITS == 32
#warning \"Using -D_FILE_OFFSET_BITS=32 and using readdir is a bad idea, see
<https://bugzilla.kernel.org/show_bug.cgi?id=205957>\"
#else
#undef readdir
#define readdir @READDIR_WITHOUT_FILE_OFFSET64_IS_A_REALLY_BAD_IDEA@
#endif
#endif
#endif
#endif

And then in posix/glob.c at the beginning:

  #undef readdir

This makes it much easier for distributions to find the problem.

Otherwise the problem would be hidden in the sense that a lot of programs
COMPILE just fine without _FILE_OFFSET_BITS--but they fail at runtime in
unexpected ways.

Unexpected because they usually don't fail right away at runtime but only on
the first strange readdir result.  A strange readdir result is one where d_off
> 2**32 but it doesn't fit into the d_off slot.

Worse, a lot of clients do not check errno and just leave all the other files
of that directory out once that happens.  When the strange readdir result
appears depends on filesystem internals.

-- 
You are receiving this mail because:
You are on the CC list for the bug.