[Bug dynamic-link/18402] New: FPU use in strlen() before main()

[Bug dynamic-link/18402] New: FPU use in strlen() before main()

[dave at sr71 dot net]

https://sourceware.org/bugzilla/show_bug.cgi?id=18402

            Bug ID: 18402
           Summary: FPU use in strlen() before main()
           Product: glibc
           Version: 2.21
            Status: NEW
          Severity: normal
          Priority: P2
         Component: dynamic-link
          Assignee: unassigned at sourceware dot org
          Reporter: dave at sr71 dot net
  Target Milestone: ---

I was working on some kernel FPU bugs and noticed that virtually all of the
processes in the system were using the FPU at some point in their execution. 
That means that the kernel's "lazy" FPU mode was not being put to good use.

I took a hello world app and compiled it both statically and dynamically.  I
have some kernel instrumentation to see the instructions which first use the
FPU.  In both the static and dynamic cases, it was the first instruction in
strlen().  I ran under gdb, set a breakpoint at strlen ("set break strlen") and
started the app.

Neither the dynamic or static versions reached main() before hitting the
breakpoint.  The gdb output from the dynamically linked one is below.

I think it would be advantageous to avoid using the FPU before the user code
even starts in an application so that the FPU can remain disabled.

(gdb) break strlen
Function "strlen" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (strlen) pending.
(gdb) c
The program is not being run.
(gdb) run
Starting program: /root/hellodave 

Breakpoint 1, strlen () at ../sysdeps/x86_64/multiarch/../rtld-strlen.S:26
26      ../sysdeps/x86_64/multiarch/../rtld-strlen.S: No such file or
directory.
(gdb) bt
#0  strlen () at ../sysdeps/x86_64/multiarch/../rtld-strlen.S:26
#1  0x00007ffff7df1635 in _dl_sysdep_start
(start_argptr=start_argptr@entry=0x7fffffffe590,
dl_main=dl_main@entry=0x7ffff7ddb910 <dl_main>) at ../elf/dl-sysdep.c:232
#2  0x00007ffff7ddecf8 in _dl_start_final (arg=0x7fffffffe590) at rtld.c:332
#3  _dl_start (arg=0x7fffffffe590) at rtld.c:558
#4  0x00007ffff7ddb2d8 in _start () from /lib64/ld-linux-x86-64.so.2
#5  0x0000000000000001 in ?? ()
#6  0x00007fffffffe7ff in ?? ()
#7  0x0000000000000000 in ?? ()
(gdb) q

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/18402] FPU use in strlen() before main()

[dave at sr71 dot net]

https://sourceware.org/bugzilla/show_bug.cgi?id=18402

Dave Hansen <dave at sr71 dot net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |hongjiu.lu at intel dot com

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/18402] FPU use in strlen() before main()

[schwab@linux-m68k.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=18402

Andreas Schwab <schwab@linux-m68k.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |WAITING

--- Comment #1 from Andreas Schwab <schwab@linux-m68k.org> ---
AFAICT there is no use of FPU insns in sysdeps/x86_64/rtld-strlen.S.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/18402] FPU use in application before main()

[dave at sr71 dot net]

https://sourceware.org/bugzilla/show_bug.cgi?id=18402

Dave Hansen <dave at sr71 dot net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|FPU use in strlen() before  |FPU use in application
                   |main()                      |before main()

--- Comment #2 from Dave Hansen <dave at sr71 dot net> ---
You're right.  For the dynamically linked app, it looks like the FPU use was in
strchr.  Let me go back and double-check the statically linked version.

The kernel told me the address from a tracepoint in the fpu code:

       hellodave-1707  [003] d..1 36047.630641: fpu_not_available:
ip=0x7ffff7a9f7c0 error_code=0x0 fpu_counter: 0


(gdb) disassemble 0x7ffff7a9f7c0 
Dump of assembler code for function strrchr:
   0x00007ffff7a9f7c0 <+0>:     movd   %esi,%xmm1


For the record, my test app is this:

int main() {
    int i;
    char msg[] = "hello dave\n";

    write(1, msg, sizeof(msg));
    sleep(9999);
    return 0;

}

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/18402] FPU use in application before main()

[hjl.tools at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18402

H.J. Lu <hjl.tools at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|hongjiu.lu at intel dot com        |hjl.tools at gmail dot com

--- Comment #3 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Dave Hansen from comment #2)
> You're right.  For the dynamically linked app, it looks like the FPU use was
> in strchr.  Let me go back and double-check the statically linked version.
> 

Are you using glibc 2.21?  We fixed some bugs before 2.21.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/18402] FPU use in application before main()

[dave at sr71 dot net]

https://sourceware.org/bugzilla/show_bug.cgi?id=18402

--- Comment #4 from Dave Hansen <dave at sr71 dot net> ---
For a statically-linked app, things are a bit different.  I set a breakpoint
*at* strlen and then waited to see the FPU get used.  When strlen() was called,
the FPU was not in use, but I single-stepped and saw it get used on the next
instruction.  So perhaps there are two separate issues here?

(gdb) r
Starting program: /root/hellodave 

Breakpoint 2, 0x00000000004191f0 in strlen ()
# Note, no FPU use here
(gdb) stepi
0x00000000004191f5 in strlen ()
# FPU has been used
(gdb) bt
#0  0x00000000004191f5 in strlen ()
#1  0x00000000004073a6 in getenv ()
#2  0x000000000043793e in _dl_non_dynamic_init ()
#3  0x00000000004386f8 in __libc_init_first ()
#4  0x0000000000401222 in __libc_start_main ()
#5  0x0000000000400f77 in _start ()

(gdb) disassemble strlen
Dump of assembler code for function strlen:
=> 0x00000000004191f0 <+0>:     pxor   %xmm8,%xmm8
   0x00000000004191f5 <+5>:     pxor   %xmm9,%xmm9
   0x00000000004191fa <+10>:    pxor   %xmm10,%xmm10
   0x00000000004191ff <+15>:    pxor   %xmm11,%xmm11
   0x0000000000419204 <+20>:    mov    %rdi,%rax

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/18402] FPU use in application before main()

[hjl.tools at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18402

--- Comment #5 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Dave Hansen from comment #4)
> For a statically-linked app, things are a bit different.  I set a breakpoint
> *at* strlen and then waited to see the FPU get used.  When strlen() was
> called, the FPU was not in use, but I single-stepped and saw it get used on
> the next instruction.  So perhaps there are two separate issues here?

This is a separate issue. Please either close this bug and open a new bug,
or reclassify this bug.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/18402] FPU use in application before main()

[dave at sr71 dot net]

https://sourceware.org/bugzilla/show_bug.cgi?id=18402

--- Comment #6 from Dave Hansen <dave at sr71 dot net> ---
(In reply to H.J. Lu from comment #3)
> (In reply to Dave Hansen from comment #2)
> > You're right.  For the dynamically linked app, it looks like the FPU use was
> > in strchr.  Let me go back and double-check the statically linked version.
> > 
> 
> Are you using glibc 2.21?  We fixed some bugs before 2.21.

I originally found this on 2.19, but I upgraded to 2.21 and retested.  The
behavior is the same.  The backtrace to strrchr looks like this with 2.21:

#0  strrchr () at ../sysdeps/x86_64/strrchr.S:25
#1  0x00007ffff7b15d83 in __init_misc (argc=-6145, argc@entry=1, argv=0x2f,
argv@entry=0x7fffffffe598, envp=envp@entry=0x7fffffffe5a8) at init-misc.c:33
#2  0x00007ffff7a2f8c4 in _init (argc=1, argv=0x7fffffffe598,
envp=0x7fffffffe5a8) at ../csu/init-first.c:86
#3  0x00007ffff7de957d in call_init (l=0x7ffff7ff74c0, argc=argc@entry=1,
argv=argv@entry=0x7fffffffe598, env=env@entry=0x7fffffffe5a8) at dl-init.c:58
#4  0x00007ffff7de96cb in call_init (env=<optimized out>, argv=<optimized out>,
argc=<optimized out>, l=<optimized out>) at dl-init.c:30
#5  _dl_init (main_map=0x7ffff7ffe188, argc=1, argv=0x7fffffffe598,
env=0x7fffffffe5a8) at dl-init.c:120
#6  0x00007ffff7dd9d0a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#7  0x0000000000000001 in ?? ()
#8  0x00007fffffffe7ff in ?? ()
#9  0x0000000000000000 in ?? ()


root@ubuntu-amd64:~# ldd hellodave
        linux-vdso.so.1 =>  (0x00007ffff7ffd000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffff7c24000)
        /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
root@ubuntu-amd64:~# ls -l /lib/x86_64-linux-gnu/libc.so.6
lrwxrwxrwx 1 root root 12 Mar 26 00:02 /lib/x86_64-linux-gnu/libc.so.6 ->
libc-2.21.so

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/18402] FPU use in application before main()

[hjl.tools at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18402

H.J. Lu <hjl.tools at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |drepper.fsp at gmail dot com
          Component|dynamic-link                |libc

--- Comment #7 from H.J. Lu <hjl.tools at gmail dot com> ---
It is not ld.so which calls strrchr.  It is __init_misc in libc.so.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/18402] FPU use in application before main()

[hjl.tools at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18402

--- Comment #8 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to H.J. Lu from comment #7)
> It is not ld.so which calls strrchr.  It is __init_misc in libc.so.

If we want to get rid of SSE optimized strrchr before main, we
need a different strrchr.  However, if program or compiler call
any string/memory functions, SSE registers may be used to optimize
those string/memory functions.  The question is how much SSE usage
impacts performance on real applications.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/18402] FPU use in application before main()

[dave at sr71 dot net]

https://sourceware.org/bugzilla/show_bug.cgi?id=18402

--- Comment #9 from Dave Hansen <dave at sr71 dot net> ---
(In reply to H.J. Lu from comment #8)
> (In reply to H.J. Lu from comment #7)
> > It is not ld.so which calls strrchr.  It is __init_misc in libc.so.
> 
> If we want to get rid of SSE optimized strrchr before main, we
> need a different strrchr.  However, if program or compiler call
> any string/memory functions, SSE registers may be used to optimize
> those string/memory functions.  The question is how much SSE usage
> impacts performance on real applications.

There's also the matter that the kernel tries to be lazy about allocating and
saving/restoring FPU state.  If the task is virtually guaranteed to have FPU
state, then we should reconsider the lazy mode being the default.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/18402] FPU use in application before main()

[hjl.tools at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18402

--- Comment #10 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Dave Hansen from comment #9)
> 
> There's also the matter that the kernel tries to be lazy about allocating
> and saving/restoring FPU state.  If the task is virtually guaranteed to have
> FPU state, then we should reconsider the lazy mode being the default.

Since most, if not all, applications will call string/memory functions,
which are optimized with SSE, kernel needs to prepare for that.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

Re: [Bug libc/18402] FPU use in application before main()

[Ondřej Bílka]

On Tue, May 12, 2015 at 04:46:31PM +0000, hjl.tools at gmail dot com wrote:
> https://sourceware.org/bugzilla/show_bug.cgi?id=18402
> 
> --- Comment #10 from H.J. Lu <hjl.tools at gmail dot com> ---
> (In reply to Dave Hansen from comment #9)
> > 
> > There's also the matter that the kernel tries to be lazy about allocating
> > and saving/restoring FPU state.  If the task is virtually guaranteed to have
> > FPU state, then we should reconsider the lazy mode being the default.
> 
> Since most, if not all, applications will call string/memory functions,
> which are optimized with SSE, kernel needs to prepare for that.
> 
Yes, lazy FPU is relict from 486 days. Now it only decreases performance
as every application uses sse. Also gcc vectorizer will likely introduce sse
instruction to improve performance of vectorizable loop.

I would mark this as invalid as I don't see why its problem.

[Bug libc/18402] FPU use in application before main()

[neleai at seznam dot cz]

https://sourceware.org/bugzilla/show_bug.cgi?id=18402

--- Comment #11 from Ondrej Bilka <neleai at seznam dot cz> ---
On Tue, May 12, 2015 at 04:46:31PM +0000, hjl.tools at gmail dot com wrote:
> https://sourceware.org/bugzilla/show_bug.cgi?id=18402
> 
> --- Comment #10 from H.J. Lu <hjl.tools at gmail dot com> ---
> (In reply to Dave Hansen from comment #9)
> > 
> > There's also the matter that the kernel tries to be lazy about allocating
> > and saving/restoring FPU state.  If the task is virtually guaranteed to have
> > FPU state, then we should reconsider the lazy mode being the default.
> 
> Since most, if not all, applications will call string/memory functions,
> which are optimized with SSE, kernel needs to prepare for that.
> 
Yes, lazy FPU is relict from 486 days. Now it only decreases performance
as every application uses sse. Also gcc vectorizer will likely introduce sse
instruction to improve performance of vectorizable loop.

I would mark this as invalid as I don't see why its problem.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/18402] FPU use in application before main()

[dave at sr71 dot net]

https://sourceware.org/bugzilla/show_bug.cgi?id=18402

--- Comment #12 from Dave Hansen <dave at sr71 dot net> ---
No objections from me if you want to close it.  You asked me to open it. :)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug libc/18402] FPU use in application before main()

[hjl.tools at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18402

H.J. Lu <hjl.tools at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |RESOLVED
         Resolution|---                         |WONTFIX

--- Comment #13 from H.J. Lu <hjl.tools at gmail dot com> ---
(In reply to Dave Hansen from comment #12)
> No objections from me if you want to close it.  You asked me to open it. :)

SSE register usage in ld.so is a bug, not because of lazy FPU, since it
may overwrite parameters passed in SSE registers.  Close it since SSE
registers aren't used in ld.so.

-- 
You are receiving this mail because:
You are on the CC list for the bug.