[Bug ada/35284] New: Branch to 0x0 from Ada run-time

[Bug ada/35284] New: Branch to 0x0 from Ada run-time

[joel at gcc dot gnu dot org]

As reported in http://gcc.gnu.org/ml/gcc-testresults/2008-02/msg00747.html, I
am seeing a few hundred failures for ACATS on i386-rtems4.9.  The failure set
appears to be about the same for 5.3.4, 4.3-branch, and SVN trunk.  I am trying
to debug
this on the SVN trunk.  I have finally captured the failure and hope someone
can 
help me run down what is broken. This report focuses on a99006a.

I am running on qemu and by turning on its debugging I have  managed find that
the Ada run-time does an indirect jump to 0x0.  I will attach a symbol table
along with the compressed qemu.log.

The last few instructions before the jump to 0x0 involve a call to
system__soft_links__set_jmpbuf_address_soft so I set a breakpoint there.
On the 8th time it hits it, the address passed in is 0x0 and the backtrace
is this:

 Breakpoint 1, system.soft_links.set_jmpbuf_address_soft (addr=(system.address)
0x0) at s-soflin.adb:270
270        procedure Set_Jmpbuf_Address_Soft (Addr : Address) is
(gdb) bt
#0  system.soft_links.set_jmpbuf_address_soft (addr=(system.address) 0x0) at
s-soflin.adb:270
#1  0x00117c52 in system.file_io.chain_file (file=0x14ffa0) at s-fileio.adb:166
#2  0x001119d8 in <ada__text_io___elabb> () at a-textio.adb:2272
#3  0x001002ea in adainit () at
../../../../../../../../../423/rtems/c/src/lib/libbsp/i386/pc386/start/start.S:192
#4  0x0010036d in gnat_main () at
../../../../../../../../../423/rtems/c/src/lib/libbsp/i386/pc386/start/start.S:192
#5  0x0010039c in start_gnat_main ()
    at
../../../../../../../../../423/rtems/c/src/lib/libbsp/i386/pc386/start/start.S:192
#6  0x00137107 in _Thread_Handler ()
    at
../../../../../../../423/rtems/c/src/../../cpukit/score/src/threadhandler.c:151

It runs and does some more calls to that same routine but eventually we 
end up doing the indirect call to 0x0 from the middle of 
system__tasking__stages__task_wrapper

I have checked and memory is initialized to zero on this BSP in case this was
an issue and none of the tasks in this test are blowing their stacks.

Advice definitely appreciated on this one.


-- 
           Summary: Branch to 0x0 from Ada run-time
           Product: gcc
           Version: 4.4.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: ada
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: joel at gcc dot gnu dot org
GCC target triplet: i386-rtems4.9


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel at gcc dot gnu dot org]

------- Comment #1 from joel at gcc dot gnu dot org  2008-02-21 22:10 -------
Created an attachment (id=15200)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=15200&action=view)
QEMU trace up to branch to 0x0

Search for "^0x00000000" and then scroll backwards to see the indirect call in
question.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel at gcc dot gnu dot org]

------- Comment #2 from joel at gcc dot gnu dot org  2008-02-21 22:11 -------
Created an attachment (id=15201)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=15201&action=view)
numerically sorted symbol table matching qemu.log

This is the numerically sorted symbol table matching the attached log.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[laurent at guerby dot net]

------- Comment #3 from laurent at guerby dot net  2008-02-21 22:32 -------
The great majority of FAIL are tests with an Ada task so my guess is that
tasking is broken.

I'm not a specialist but GNAT soft links are pointers to function which are set
by the elaboration code to point to tasking or non-tasking version of several
runtime primitives, these are in s-soflin.ads. 

May be what you see is that elaboration for s-soflin.ads is not done at the
right time during elaboration so all pointers end up null hence the jump to
null. Might be a binder problem.

Could you try the following small tasking program:

-- begin t.adb
with GNAT.IO; use GNAT.IO;
procedure T is
   task A;
   task body A is
   begin
      Put_Line ("A");
   end A;
begin
   Put_Line ("T");
end T;
-- end t.adb

When I compile it on i386 native with gnatmake -g t.adb I see in the binder
generated b~t.adb :

     System.Restrictions'Elab_Spec;
      System.Restrictions'Elab_Body;
      E076 := True;
      Ada.Exceptions'Elab_Spec;
      System.Soft_Links'Elab_Body;
      E011 := True;
      System.Secondary_Stack'Elab_Body;
      E013 := True;
      System.Exception_Table'Elab_Body;
      E022 := True;
      Ada.Io_Exceptions'Elab_Spec;
      E082 := True;
      Ada.Tags'Elab_Spec;
      Ada.Tags'Elab_Body;
      E074 := True;
      Ada.Streams'Elab_Spec;
      E077 := True;
      Interfaces.C'Elab_Spec;
      E041 := True;
      System.Finalization_Root'Elab_Spec;
      E079 := True;
      System.Tasking.Initialization'Elab_Body;
      E084 := True;
      System.Tasking.Protected_Objects'Elab_Body;
      E090 := True;
      E018 := True;
      System.Finalization_Implementation'Elab_Spec;
      System.Finalization_Implementation'Elab_Body;
      E072 := True;
      Ada.Finalization'Elab_Spec;
      E096 := True;
      Ada.Finalization.List_Controller'Elab_Spec;
      E098 := True;
      System.Tasking.Protected_Objects.Entries'Elab_Spec;
      E094 := True;
      System.Tasking.Queuing'Elab_Body;
      E088 := True;
      System.Tasking.Stages'Elab_Body;
      E068 := True;


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel at gcc dot gnu dot org]

------- Comment #4 from joel at gcc dot gnu dot org  2008-02-21 22:52 -------
Runs OK on powerpc/psim and sparc/sis built from the same source tree for GCC
and RTEMS.

Fails like a99006a on i386. 

Are there only ~300 tests which rely on tasking?  I posted this URL:
http://gcc.gnu.org/ml/gcc-testresults/2008-02/msg00747.html
to point out what was failing and passing in case there was a common
thread.ju

I am attaching the b~t.adb.  I don't know enough to comfortably judge whether
it is right or not.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel at gcc dot gnu dot org]

------- Comment #5 from joel at gcc dot gnu dot org  2008-02-21 22:53 -------
Created an attachment (id=15202)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=15202&action=view)
generated from Laurent's t.adb


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[laurent at guerby dot net]

------- Comment #6 from laurent at guerby dot net  2008-02-21 22:59 -------
Could you post the exact backtrace on my testcase? It purposely doesn't use the
complex Text_IO so things should be clearer. Also if you could use gdb to get
an understanding of what is done during adainit execution before the failure
point.

PS: continuing after sleep for me :)


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel dot sherrill at oarcorp dot com]

------- Comment #7 from joel dot sherrill at oarcorp dot com  2008-02-21 23:55 -------
Subject: Re:  Branch to 0x0 from Ada run-time

laurent at guerby dot net wrote:
> ------- Comment #6 from laurent at guerby dot net  2008-02-21 22:59 -------
> Could you post the exact backtrace on my testcase? 

Breakpoint 1, 0x00000000 in ?? ()
(gdb) bt
#0  0x00000000 in ?? ()
#1  0x00106079 in system.tasking.stages.task_wrapper (self_id=0x1de234)
    at s-tassta.adb:1140
#2  0x0012c72f in _Thread_Handler ()
    at 
../../../../../../../423/rtems/c/src/../../cpukit/score/src/threadhandler.c:151
#3  0x00000071 in ?? ()
#4  0x0000037f in ?? ()
#5  0x00000000 in ?? ()

I don't know if this helps but

(gdb) p/x _Thread_Executing->Object.id
$1 = 0xb010003

Which indicates this is the third POSIX thread created in
this test.

(1) RTEMS POSIX_Init
(2) main Ada task
(3) your task A

> It purposely doesn't use the
> complex Text_IO so things should be clearer. Also if you could use gdb to get
> an understanding of what is done during adainit execution before the failure
> point.
>   
I also disabled clock tick interrupts since this test doesn't
appear to have any need for time to be passing.

I didn't attach a log yet because I don't think this is in
initialization.  I think adainit is doing
the right thing.  I think this is a finalization problem.  When
I break at system__standard_library__adafinal (aka (Do_Finalize),
T has been  printed but not the A.  When the main thread blocks
in "Timed_Sleep" from something near here:

system.tasking.stages.finalize_global_tasks () at s-tassta.adb:796

I traced it through the context switch that happens via Timed_Sleep
and I was surprised to see that your Task T has NEVER run.  When
we context switch to your Task T, we go directly to the RTEMS
_Thread_Handler wrapper for all thread entries which calls the
Ada thread wrapper entry.  So while we are finalizing, it is
getting context switched to for the first time.  That seems
like a solid lead.

I will trace further into what is happening in the morning. 
At this point, it seems like finalizing a thread that hasn't
started yet is tripping something funny.

If someone gets a good idea before I start again in the
morning, just post it.

> PS: continuing after sleep for me :)
>
>   
I am on my way to register my youngest
son for high school for the fall.  Wow!

You should be sleeping.  It is late in Paris.

Thanks.

--joel
> --
>
>
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284
>
> ------- You are receiving this mail because: -------
> You reported the bug, or are watching the reporter.
>   


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[charlet at gcc dot gnu dot org]

------- Comment #8 from charlet at gcc dot gnu dot org  2008-02-22 08:02 -------
Sounds like something specific to RTEMS, since I've never seen something like
that on any other platform.


-- 

charlet at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|charlet at adacore dot com  |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[laurent at guerby dot net]

------- Comment #9 from laurent at guerby dot net  2008-02-22 08:29 -------
Arnaud, do you have any idea on what to look at (interesting breakpoint and
data structures)? RTEMS tasking works on other platforms. May be some Ada/C
thread interface data structure has the wrong size in the case of x86 and
things get corrupted (s-oisinte-rtems.ads).

Joel could you break into all RTEMS calls done by "(3) your task A" (and main
creation of task A) and look at struct and parameters to see if something looks
wrong? May be you could do a parrallel gdb of i386 and sparc to see where
things diverge.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

Re: [Bug ada/35284] Branch to 0x0 from Ada run-time

[Arnaud Charlet]

> Arnaud, do you have any idea on what to look at (interesting breakpoint and
> data structures)? RTEMS tasking works on other platforms. May be some Ada/C
> thread interface data structure has the wrong size in the case of x86 and
> things get corrupted (s-oisinte-rtems.ads).

I'd suggest putting a watchpoint on the softlink that gets cleared to 0x0 (or
never set ?), you should see pretty quickly what's happening.

Note that I removed myself from cc: because I already receive all bugzilla
messages related to Ada PRs, so being in cc: means I receive the
messages twice, which is annoying given the amount of email I have
to deal with. It does not mean I'm ignoring this PR :-)

Arno

[Bug ada/35284] Branch to 0x0 from Ada run-time

[charlet at adacore dot com]

------- Comment #10 from charlet at adacore dot com  2008-02-22 08:34 -------
Subject: Re:  Branch to 0x0 from Ada run-time

> Arnaud, do you have any idea on what to look at (interesting breakpoint and
> data structures)? RTEMS tasking works on other platforms. May be some Ada/C
> thread interface data structure has the wrong size in the case of x86 and
> things get corrupted (s-oisinte-rtems.ads).

I'd suggest putting a watchpoint on the softlink that gets cleared to 0x0 (or
never set ?), you should see pretty quickly what's happening.

Note that I removed myself from cc: because I already receive all bugzilla
messages related to Ada PRs, so being in cc: means I receive the
messages twice, which is annoying given the amount of email I have
to deal with. It does not mean I'm ignoring this PR :-)

Arno


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel at gcc dot gnu dot org]

------- Comment #11 from joel at gcc dot gnu dot org  2008-02-22 15:34 -------
Good Morning!  By setting a breakpoint on _CPU_Context_switch in RTEMS, I can
see where things went wrong.  On both SPARC and PowerPC, we dispatched out of
this backtrace:

(gdb) bt
#0  pthread_setschedparam (thread=184614915, policy=0, param=0x175000)
    at ../../cpukit/../../../pc386/lib/include/rtems/score/thread.inl:209
#1  0x0010118f in system.task_primitives.operations.set_priority (t=0x1de1f4, 
    prio=0, loss_of_inheritance=0) at s-taprop.adb:777
#2  0x0010121d in system.task_primitives.operations.create_task (t=0x1de1f4, 
    wrapper=1072820, stack_size=0, priority=0) at s-taprop.adb:1016
#3  0x00105a99 in system.tasking.stages.activate_tasks (chain_access=0x1752c8)
    at s-tassta.adb:338
#4  0x001005ea in t () at t.adb:2

At this point, the Ada run-time is setting the priority of the POSIX
thread it just created.

On the PowerPC (and I assume SPARC since it works there also), the 
contents of the scheduling parameter structure is:

(gdb) p *param
$1 = {sched_priority = 122, ss_low_priority = 7462340, ss_replenish_period = {
    tv_sec = 0, tv_nsec = 0}, ss_initial_budget = {tv_sec = 1, 
    tv_nsec = 32724}}

On the i386, this same structure at the same point in RTEMS has this:

$1 = {sched_priority = 0, ss_low_priority = 2, ss_replenish_period = {
    tv_sec = 0, tv_nsec = 0}, ss_initial_budget = {tv_sec = 0, tv_nsec = 0}}

The sched_priority field is NOT a valid priority and RTEMS returns EINVAL.
There is a pragma assert after this call in the Ada
system.task_primitives.operations.set_priority so I assume assertions
are disabled.  FWIW sys/sched.h is the same file with no CPU ifdefs
on all RTEMS targets.

Since this is the first field in the structure, we are not dealing with
an alignment problem.  All of the Ada run-time target specific files 
for RTEMS are shared across the architectures so I doubt that is a problem.

I noticed that in the good backtrace, I see this:

#0  pthread_setschedparam (thread=184614915, policy=0, param=0x744c30)
    at
../../../../../../../423/rtems/c/src/../../cpukit/posix/src/pthreadsetschedparam.c:47
#1  0x00001954 in system.task_primitives.operations.set_priority (t=0x5e868, 
    prio=122, loss_of_inheritance=<value optimized out>) at s-taprop.adb:777
#2  0x00001a38 in system.task_primitives.operations.create_task (t=0x5e868, 
    wrapper=32724, stack_size=32768, priority=122) at s-taprop.adb:1016
#3  0x00007bb4 in system.tasking.stages.activate_tasks (chain_access=0x744d34)
    at s-tassta.adb:338
#4  0x000007ec in t () at t.adb:2

In the bad backtrace, I see this:

#0  0x0010118f in system.task_primitives.operations.set_priority (t=0x1de1f4, 
    prio=0, loss_of_inheritance=0) at s-taprop.adb:777
#1  0x0010121d in system.task_primitives.operations.create_task (t=0x1de1f4, 
    wrapper=1072820, stack_size=0, priority=0) at s-taprop.adb:1016
#2  0x00105a99 in system.tasking.stages.activate_tasks (chain_access=0x1752c8)
    at s-tassta.adb:338
#3  0x001005ea in t () at t.adb:2

Notice that create.task is passed a 0 for priority on the i386 and a 122 on the 
good backtrace.  

What to look at next?  That bad priority value has to come from somewhere. 


-- 

joel at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
     Ever Confirmed|0                           |1
   Last reconfirmed|0000-00-00 00:00:00         |2008-02-22 15:34:43
               date|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel at gcc dot gnu dot org]

------- Comment #12 from joel at gcc dot gnu dot org  2008-02-22 16:43 -------
I just noticed that the stack size passed to
system.task_primitives.operations.create_task is also 0 in the back trace on
the i386. It is 32724 on the PowerPC.

So two parameters are bad.  The 0 just indicates that the pthread code should
use the default stack size so it happens to work.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel at gcc dot gnu dot org]

------- Comment #13 from joel at gcc dot gnu dot org  2008-02-22 20:53 -------
Laurent's t.adb works on arm/edb7312.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[laurent at guerby dot net]

------- Comment #14 from laurent at guerby dot net  2008-02-22 21:12 -------
Default values come from ada/s-parame-rtems.adb, for the stack it ends up in
RTEMS ada_pthread_minimum_stack_size:

package body System.Parameters is

   function ada_pthread_minimum_stack_size return Interfaces.C.size_t;
   pragma Import (C, ada_pthread_minimum_stack_size,
     "_ada_pthread_minimum_stack_size");

   ------------------------
   -- Default_Stack_Size --
   ------------------------

   function Default_Stack_Size return Size_Type is
   begin
      return Size_Type (ada_pthread_minimum_stack_size);
   end Default_Stack_Size;
...

In ada/s-tassta.adb above line 333 you should be able to trace where Priority
and Stack_Size come from, compare between the two targets:

      --  Activate all the tasks in the chain. Creation of the thread of
      --  control was deferred until activation. So create it now.

      C := Chain_Access.T_ID;
      while C /= null loop
         if C.Common.State /= Terminated then
            pragma Assert (C.Common.State = Unactivated);

            P := C.Common.Parent;
            Write_Lock (P);
            Write_Lock (C);

            if C.Common.Base_Priority < Get_Priority (Self_ID) then
               Activate_Prio := Get_Priority (Self_ID);
            else
               Activate_Prio := C.Common.Base_Priority;
            end if;

            System.Task_Primitives.Operations.Create_Task
              (C, Task_Wrapper'Address,
               Parameters.Size_Type
                 (C.Common.Compiler_Data.Pri_Stack_Info.Size),
               Activate_Prio, Success);
...


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel at gcc dot gnu dot org]

------- Comment #15 from joel at gcc dot gnu dot org  2008-02-22 21:56 -------
(In reply to comment #14)
> Default values come from ada/s-parame-rtems.adb, for the stack it ends up in
> RTEMS ada_pthread_minimum_stack_size:
> 
> package body System.Parameters is
> 
>    function ada_pthread_minimum_stack_size return Interfaces.C.size_t;
>    pragma Import (C, ada_pthread_minimum_stack_size,
>      "_ada_pthread_minimum_stack_size");

This is called twice on powerpc and i386 and both appear to be from the 
same backtrace.  The first is where the rtems_init.c creates the ada_main
thread.  The second back traces to here:

#0  _ada_pthread_minimum_stack_size ()
    at ../../../../../../../423/rtems/c/src/../../cpukit/libgnat/adasupp.c:24
#1  0x001040a7 in system.tasking.initialize_atcb (self_id=0x1dda08, 
    task_entry_point=0x1752ce, task_arg=1528516, parent=0x1dda08, 
    elaborated=0x1752cd, base_priority=0, task_info=2, stack_size=-2147483648, 
    t=0x1de1f4) at s-taskin.adb:136
#2  0x0010577e in system.tasking.stages.create_task (priority=-1, 
    size=-2147483648, task_info=2, num_entries=0, master=4, state=0x1752ce, 
    discriminants=1528516, elaborated=0x1752cd, chain=@0x1752c8, task_image=
      {P_ARRAY = 0x138948, P_BOUNDS = 0x138954}, created_task=0x0)
    at s-tassta.adb:605
#3  0x001005d7 in t () at t.adb:3

> 
>    ------------------------
>    -- Default_Stack_Size --
>    ------------------------
> 
>    function Default_Stack_Size return Size_Type is
>    begin
>       return Size_Type (ada_pthread_minimum_stack_size);
>    end Default_Stack_Size;
> ...
> 
> In ada/s-tassta.adb above line 333 you should be able to trace where Priority
> and Stack_Size come from, compare between the two targets:
> 
>       --  Activate all the tasks in the chain. Creation of the thread of
>       --  control was deferred until activation. So create it now.
> 
>       C := Chain_Access.T_ID;
>       while C /= null loop
>          if C.Common.State /= Terminated then
>             pragma Assert (C.Common.State = Unactivated);
> 
>             P := C.Common.Parent;
>             Write_Lock (P);
>             Write_Lock (C);
> 
>             if C.Common.Base_Priority < Get_Priority (Self_ID) then
>                Activate_Prio := Get_Priority (Self_ID);
>             else
>                Activate_Prio := C.Common.Base_Priority;
>             end if;

Both powerpc and i386 take the else path.

Breakpoint 1, system.tasking.stages.activate_tasks (chain_access=0x1752c8)
    at s-tassta.adb:332
332                 if C.Common.Base_Priority < Get_Priority (Self_ID) then
(gdb) n
335                    Activate_Prio := C.Common.Base_Priority;
(gdb) 
338                 System.Task_Primitives.Operations.Create_Task


>             System.Task_Primitives.Operations.Create_Task
>               (C, Task_Wrapper'Address,
>                Parameters.Size_Type
>                  (C.Common.Compiler_Data.Pri_Stack_Info.Size),
>                Activate_Prio, Success);
> ...
> 

I haven't had much luck dereferencing the C.Common stuff or figuring out where
it is. 

FWIW this is pretty easy to debug with qemu, a 1.44 MB floppy file, and the
executable.  I double checked and you can even use the native Fedora 8 gdb.
qemu speaks gdb remote protocol.  

Sorry for not having more information.  I checked and at the end of the
run, each RTEMS thread control block shows an RTEMS native priority of
253 (255 being lowest natively).  On the PowerPC the threads have a native
priority of 133 which is in the middle of the 1-254 range.  I have no idea
why it is pulling a 0 out of the air. :(


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel at gcc dot gnu dot org]

------- Comment #16 from joel at gcc dot gnu dot org  2008-02-22 22:01 -------
I can add mips/jmr3904 to the list that runs t.adb correctly.  So we have
this as a summary:

sparc/sis, powerpc/psim, mips/jmr3904, arm/edb7312 - t.adb runs OK
i386/pc386 fails
bfin/exkit533 - GNAT bug bo

I doublechecked the RTEMS target specific files used on each configuration from
the build log.  They are the same.  Is there anything specific to the x86 the 
target entry in the Makefile.in might be missing?


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[laurent at guerby dot net]

------- Comment #17 from laurent at guerby dot net  2008-02-22 22:02 -------
If you have it on CFARM let me know where and what command to launch to gdb
your testcase.

What does RTEMS ada_pthread_minimum_stack_size returns in both case (under
gdb)?


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel at gcc dot gnu dot org]

------- Comment #18 from joel at gcc dot gnu dot org  2008-02-22 22:35 -------
(In reply to comment #17)
> If you have it on CFARM let me know where and what command to launch to gdb
> your testcase.

I have been building and running it on my laptop but there is now enough on
CFARM to run it.

 qemu -M isapc -m 8 -boot a -fda /home/joel/qemu/pc386_fda \
    -hda fat:/home/joel/qemu/hd  -serial stdio

will run the program without a debugger.  It pops up an xterm-ish thing
and shows a srub menu before the output switches to the serial port which
is redirected to stdio.  Just copy /home/joel/qemu from gcc12 and change
paths as appropriate.

Add "-s" on the end of the qemu command line and it has a gdb server.
Start qemu and quickly in another window do this:

gdb --command g t.exe

g has a "target remote" command in it which attaches to the qemu.

qemu has some debug trace which is interesting and how I got started.
Add "-d in_asm,cpu" and will dump a lot of info on assembly instructions
and cpu state into /tmp/qemu.log.

For your convenience, there is a pr35284-a/ subdirectory under ~/qemu 
which is a copy of my current build directory.

> What does RTEMS ada_pthread_minimum_stack_size returns in both case (under
> gdb)?
> 

16K.  Just like a good one line routine should. :-D

I will email you from my gmail account so if you need to ping me this weekend
you can.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel at gcc dot gnu dot org]

------- Comment #19 from joel at gcc dot gnu dot org  2008-02-25 20:45 -------
How early can I look at the task priority?  Is it stored in some data structure
that I can see with objdump before the program is run?  

I have yet to see anything in the debugger except 0 getting passed to task
create for priority and stack size.  They have to come from somewhere.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel at gcc dot gnu dot org]

------- Comment #20 from joel at gcc dot gnu dot org  2008-03-14 21:46 -------
I don't want this bug to die from inattention. Returning to my analysis of
Laurent's program, the task does not get to run at all before the system is
finalized.  I suspect this indicates the initialization of the task depends
upon something destroyed at finalization time or vice-versa.  I don't know
though.

I am sure by breaking on context switch points that the task has not run
even it outer most RTEMS entry point, much less the Ada task wrapper when
finalization is hit.  This would seem to point to the problem.

I would appreciate suggestions on what could be wrong, what to look at,
and what to try. 

Thanks.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[laurent at guerby dot net]

------- Comment #21 from laurent at guerby dot net  2008-03-14 22:29 -------
Best think would be to trace rtems calls on a platform where it works vs on the
simulator. On a platform where it works, look at the backtrace of the task
creation call and try to find out why it doesn't get called on i386 qemu.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel at gcc dot gnu dot org]

------- Comment #22 from joel at gcc dot gnu dot org  2008-03-31 20:16 -------
(In reply to comment #21)
> Best think would be to trace rtems calls on a platform where it works vs on the
> simulator. On a platform where it works, look at the backtrace of the task
> creation call and try to find out why it doesn't get called on i386 qemu.
> 

I am back to this.  Which pthread calls are of interest?  I am starting
with pthread create, mutex init, lock and unlock, context switches and a few
condition variable calls.  What else is of interest?


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[laurent at guerby dot net]

------- Comment #23 from laurent at guerby dot net  2008-03-31 20:21 -------
pthread create and context switch should be the first to look at, in comparison
with a platform where it works.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel at gcc dot gnu dot org]

------- Comment #24 from joel at gcc dot gnu dot org  2008-04-01 18:02 -------
With Laurent's test program, I have traces of good (powerpc/psim) and bad
(qemu) runs.  The traced include only entry and exit status info for the
following calls are:


_CPU_Context_switch
pthread_cond_broadcast
pthread_cond_destroy
pthread_cond_init
pthread_cond_signal
pthread_cond_timedwait
pthread_cond_wait
pthread_create
pthread_exit
pthread_getschedparam
pthread_mutex_destroy
pthread_mutex_init
pthread_mutex_lock
pthread_mutex_timedlock
pthread_mutex_unlock
pthread_setschedparam
rtems_stack_checker_is_blown
rtems_task_create

There are two anomalies in the i386 run:

The first call to pthread_setschedparam() returns EINVAL because it is passed a
sched_param with priority == 0.  This field is 122 on the valid psim run. 
Backtrace indicates we are still in initialization.  I do not know how to tell
where the 0 came from before here. 

#2  0x0010153f in system.task_primitives.operations.set_priority (t=0x1dbb38, 
    prio=0, loss_of_inheritance=0) at s-taprop.adb:764
#3  0x0010451d in system.tasking.initialize () at s-taskin.adb:188
#4  0x00103fc8 in system.tasking.initialization.init_rts () at s-tasini.adb:322
#5  0x0010032b in adainit ()


Later there is a pthread_mutex_unlock() call which passes in a bad mutex id. 
This does not occur on the psim run.

The bad pthread_setschedparam call is not THAT far into the execution and the
traced calls up until this point are the same.

ENTER pthread_create
EXIT pthread_create (0)
ENTER _CPU_Context_switch
ENTER pthread_create
EXIT pthread_create (0)
ENTER pthread_exit
ENTER _CPU_Context_switch
ENTER pthread_mutex_init
EXIT pthread_mutex_init (0)
ENTER pthread_cond_init
EXIT pthread_cond_init (0)
ENTER pthread_mutex_init
EXIT pthread_mutex_init (0)
ENTER pthread_mutex_lock
EXIT pthread_mutex_lock (0)
ENTER pthread_mutex_unlock
EXIT pthread_mutex_unlock (0)
ENTER pthread_setschedparam


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[laurent at guerby dot net]

------- Comment #25 from laurent at guerby dot net  2008-04-01 19:40 -------
The binder will generate a call to Set_Globals
      pragma Import (C, Set_Globals, "__gnat_set_globals");

      Set_Globals
        (Main_Priority            => -1,
         Time_Slice_Value         => -1,
         WC_Encoding              => 'b',
         Locking_Policy           => ' ',
         Queuing_Policy           => ' ',
         Task_Dispatching_Policy  => ' ',
         Restrictions             => Restrictions'Address,
         Interrupt_States         => Interrupt_States'Address,
         Num_Interrupt_States     => 0,
         Unreserve_All_Interrupts => 0,
         Exception_Tracebacks     => 0,
         Zero_Cost_Exceptions     => 1,
         Detect_Blocking          => 0);


That should set the following global in s-taskin.adb:

   Main_Priority : Integer;
   pragma Import (C, Main_Priority, "__gl_main_priority");
   --  Priority for main task. Note that this is of type Integer, not
   --  Priority, because we use the value -1 to indicate the default
   --  main priority, and that is of course not in Priority'range.

Which is checked in System.tasking.initialize

      --  Initialize Environment Task

      if Main_Priority = Unspecified_Priority then
         Base_Priority := Default_Priority;
      else
         Base_Priority := Priority (Main_Priority);
      end if;

Default_Priority is defined in system-rtems.ads :

   Default_Priority : constant Priority := 122;

so my advice is breakpoint on the if above and see what happens. double check
that you compile using the right system.ads.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel at gcc dot gnu dot org]

------- Comment #26 from joel at gcc dot gnu dot org  2008-04-01 22:40 -------
(In reply to comment #25)
> The binder will generate a call to Set_Globals

The code is different for the head but the intent is clear. Thanks
for the explanation.

A diff of the generated b~ file for powerpc and i386 only turned up
target path name differences.  So no explanation from this file.

>       Set_Globals
>         (Main_Priority            => -1,
>          Time_Slice_Value         => -1,
>          WC_Encoding              => 'b',
>          Locking_Policy           => ' ',
>          Queuing_Policy           => ' ',
>          Task_Dispatching_Policy  => ' ',

These are the same values for RTEMS.

>          Restrictions             => Restrictions'Address,
>          Interrupt_States         => Interrupt_States'Address,

These are too different to tell.

>          Num_Interrupt_States     => 0,
>          Unreserve_All_Interrupts => 0,
>          Exception_Tracebacks     => 0,
>          Zero_Cost_Exceptions     => 1,
>          Detect_Blocking          => 0);

We have: 

      Num_Interrupt_States := 0;
      Unreserve_All_Interrupts := 0;
      Zero_Cost_Exceptions := 0;
      Detect_Blocking := 0;
      Default_Stack_Size := -1;
      Leap_Seconds_Support := 0;

Which I think is OK.

> That should set the following global in s-taskin.adb:
> 
>    Main_Priority : Integer;
>    pragma Import (C, Main_Priority, "__gl_main_priority");
>    --  Priority for main task. Note that this is of type Integer, not
>    --  Priority, because we use the value -1 to indicate the default
>    --  main priority, and that is of course not in Priority'range.
> 
> Which is checked in System.tasking.initialize
> 
>       --  Initialize Environment Task
> 
>       if Main_Priority = Unspecified_Priority then
>          Base_Priority := Default_Priority;
>       else
>          Base_Priority := Priority (Main_Priority);
>       end if;
> 
> Default_Priority is defined in system-rtems.ads :
> 
>    Default_Priority : constant Priority := 122;

It looks OK.  

What bothers me is this:

#1  0x001007fe in __wrap_pthread_setschedparam (pthread=184614914, policy=0, 
    param=0x177654) at wrap_pthread_setschedparam.c:29
#2  0x0010154b in system.task_primitives.operations.set_priority (t=0x1dbb18, 
    prio=0, loss_of_inheritance=0) at s-taprop.adb:764
#3  0x00104529 in system.tasking.initialize () at s-taskin.adb:188

Notice that the call #2 has priority of 0.  That maps to this call from
s-tasin.adb:188:

      STPO.Set_Priority (T, T.Common.Base_Priority);

Earlier in Initialize_ATCB, the field was set to 122.  I confirmed
it in the subprogram.  But I can't print the T.Common.Base_Priority from gdb 
right before the call to Set_Priority.  So I don't really know if is it
still right.  It 

At this point, I think the global variables are right but somehow the
T.Common.Base_Priority is really not getting set in a way that survives
the return from Initialize_ATCB or it is overwritten in the few lines of 
s-taskin.adb that are between or it is dereferenced wrong.

Any ideas or hints on peeking at the data in gdb would be appreciated.  
Otherwise my next step is probably using the qemu assembly log and see
what I can figure out from that.  Ugly and slow.


> so my advice is breakpoint on the if above and see what happens. double check
> that you compile using the right system.ads.
> 


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[laurent at guerby dot net]

------- Comment #27 from laurent at guerby dot net  2008-04-02 07:36 -------
My guess is that Initialize_TCB you wrote for RTEMS (s-taprop-rtems) does not
set success to True for some reason:

   procedure Initialize_ATCB
...
      STPO.Initialize_TCB (T, Success); --- here call into RTEMS specific code

      if not Success then
         return;
      end if;

      T.Common.Parent := Parent;
      T.Common.Base_Priority := Base_Priority;
      T.Common.Current_Priority := 0;   ---- <==== here comes the zero

To look for variables under gdb try "set lang c", and "info local", "info arg",
you'll get strange names but you should be able to print.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[laurent at guerby dot net]

------- Comment #28 from laurent at guerby dot net  2008-04-02 11:44 -------
I did not notice that s-taprop for rtems was the posix version

   procedure Initialize_TCB (Self_ID : Task_Id; Succeeded : out Boolean) is
      Mutex_Attr : aliased pthread_mutexattr_t;
      Result     : Interfaces.C.int;
      Cond_Attr  : aliased pthread_condattr_t;

   begin
...

         if Result /= 0 then
            Succeeded := False;
            return;
         end if;
...
      if Result = 0 then
         Succeeded := True;
      else
         if not Single_Lock then
            Result := pthread_mutex_destroy (Self_ID.Common.LL.L'Access);
            pragma Assert (Result = 0);
         end if;

         Succeeded := False;
      end if;
...

So it's now just a matter of finding which posix call in there fails.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel at gcc dot gnu dot org]

------- Comment #29 from joel at gcc dot gnu dot org  2008-04-02 15:08 -------
I have spent the morning debugging at the assembly level and I am nearly 100%
positive %ebx is getting corrupted.  It is correct before the call to       

STPO.Initialize_TCB (T, Success);

at s-taskin.adb and 0x0 upon return. There are ~2000 lines in the qemu.log 
between the two points so I have some reduction. 

I grep'ed the RTEMS source for references to ebx and I didn't see any which
were not in ISR or context switch code.  I did a run with no IO or interrupts
and got the same result.  At this point, I am looking for some subroutine that
isn't preserving ebx properly.


(In reply to comment #28)
> I did not notice that s-taprop for rtems was the posix version
> 
>    procedure Initialize_TCB (Self_ID : Task_Id; Succeeded : out Boolean) is
>       Mutex_Attr : aliased pthread_mutexattr_t;
>       Result     : Interfaces.C.int;
>       Cond_Attr  : aliased pthread_condattr_t;
> 
>    begin
> ...
> 
>          if Result /= 0 then
>             Succeeded := False;
>             return;
>          end if;
> ...
>       if Result = 0 then
>          Succeeded := True;
>       else
>          if not Single_Lock then
>             Result := pthread_mutex_destroy (Self_ID.Common.LL.L'Access);
>             pragma Assert (Result = 0);
>          end if;
> 
>          Succeeded := False;
>       end if;
> ...
> 
> So it's now just a matter of finding which posix call in there fails.
> 


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[laurent at guerby dot net]

------- Comment #30 from laurent at guerby dot net  2008-04-02 16:44 -------
Did you look at what happens in Initialize_TCB to the variable Success ?


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel at gcc dot gnu dot org]

------- Comment #31 from joel at gcc dot gnu dot org  2008-04-03 00:10 -------

(In reply to comment #30)
> Did you look at what happens in Initialize_TCB to the variable Success ?

After waiting and waiting at the hospital today, I had plenty of time
to think.  My guess was that the saved copy of %ebx on the stack was
getting corrupted.  About an hour of playing on my laptop, I found the
culprit!!  

The pthread_condattr_t structure in s-opsinte-rtems.ads was one 4 byte 
integer field too short!!! 

I also noticed a typo in the pthread_attr_t structure.  I have no
idea how this worked on powerpc, sparc, and mips so well and only
failed on i386.  Must just be lucky on the alignment or something
and missed stuff on the stack that mattered.  

Here is the much improved ACATS with this change on i386.  Many appear to
be floating point issues. The last run I sent to the mailing list had
307 failures.

                === acats Summary ===
# of expected passes            2286
# of unexpected failures        26
# of unsupported tests          3
*** FAILURES: c34014h c52103q c52104k c64005c c64005d c953002 cd1009w cd5011e
cxg2002 cxg2003 cxg2004 cxg2006 cxg2007 cxg2010 cxg2011 cxg2012 cxg2013 cxg2014
cxg2015 cxg2016 cxg2017 cxg2018 cxg2019 cxg2020 cxg2021 la14012

I will run overnight and file more test results on the mailing list. 
Patch will be attached in a minute.  


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel at gcc dot gnu dot org]

------- Comment #32 from joel at gcc dot gnu dot org  2008-04-03 00:14 -------
Created an attachment (id=15417)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=15417&action=view)
Correct structure bindings

pthread_condattr_t was missing a field.

sched_param had an incorrect type.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[laurent at guerby dot net]

------- Comment #33 from laurent at guerby dot net  2008-04-03 03:46 -------
Great news!


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[laurent at guerby dot net]

------- Comment #34 from laurent at guerby dot net  2008-04-03 03:47 -------
May be other platform aligned the record 8 byte hence hiding the issue, and x86
only 4 bytes hence showing it.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel dot sherrill at oarcorp dot com]

------- Comment #35 from joel dot sherrill at oarcorp dot com  2008-04-03 14:30 -------
Subject: Re:  Branch to 0x0 from Ada run-time

laurent at guerby dot net wrote:
> ------- Comment #34 from laurent at guerby dot net  2008-04-03 03:47 -------
> May be other platform aligned the record 8 byte hence hiding the issue, and x86
> only 4 bytes hence showing it.
>
>   
That's got to be it but how so many tests would pass is baffling.
This seems like something that should have resulted in more
failures.

FYI PR35795 kept me from getting test results on the sparc.  It
was fixed overnight and I want to make sure sparc is OK before
submitting test results.

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35795


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel at gcc dot gnu dot org]

------- Comment #36 from joel at gcc dot gnu dot org  2008-04-04 14:30 -------
Test results running with osinte.diff have been posted to gcc-testresults for
i386, mips, sparc, and powerpc.  They look very good.  Most failures are
cross-target or FPU exceptions now.

This needs to be applied to the 4.2 branch as well. 

2008-04-04  Joel Sherrill <joel.sherrill@oarcorp.com>

        * s-osinte-rtems.ads: Correct type of ss_low_priority field in struct 
        sched_param. Add missing process_shared field to pthread_condattr_t.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[joel at gcc dot gnu dot org]

------- Comment #37 from joel at gcc dot gnu dot org  2008-04-04 15:11 -------
Forgot we are up to 4.4... it applies to 4.2 and 4.3 branches.  


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[laurent at guerby dot net]

------- Comment #38 from laurent at guerby dot net  2008-04-05 09:51 -------
It's usually best to develop a patch against trunk and then port it if interest
for other branches.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284

[Bug ada/35284] Branch to 0x0 from Ada run-time

[charlet at gcc dot gnu dot org]

------- Comment #39 from charlet at gcc dot gnu dot org  2008-05-12 22:43 -------
Fixed on trunk.
Patch pre-approved on active branches if deemed necessary.


-- 

charlet at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED
   Target Milestone|---                         |4.4.0


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35284