Running programs on aarch64 simulator

Running programs on aarch64 simulator

[Joel Sherrill]

Hi

Can someone please provide an example of how to compile a program to run on
the aarch64 simulator in gdb? The simple "aarch64-elf main.c -o main" does
not link and appears to be missing libgloss type symbols.

I asked on newlib but got no response. I'm hoping someone here knows.

Thanks.

--joel
RTEMS

Re: Running programs on aarch64 simulator

[Simon Marchi]

On 2020-05-08 9:44 a.m., Joel Sherrill wrote:
> Hi
> 
> Can someone please provide an example of how to compile a program to run on
> the aarch64 simulator in gdb? The simple "aarch64-elf main.c -o main" does
> not link and appears to be missing libgloss type symbols.
> 
> I asked on newlib but got no response. I'm hoping someone here knows.
> 
> Thanks.
> 
> --joel
> RTEMS
> 

Hi Joel,

If I just compile a "main" function with the bare-metal aarch64 or arm compiler, it does indeed
complain about a missing _exit function:

<snip>/gcc/aarch64-none-elf/9.2.1/../../../../aarch64-none-elf/lib/libg.a(lib_a-exit.o): in function `exit':
<snip>/build/src/newlib-cygwin/newlib/libc/stdlib/exit.c:64: undefined reference to `_exit'

I think this is expected, as the runtime requires you to provide some implementation for _exit,
in other words what you want to do when the program has finished.  Maybe shut down the board,
reset it, shut down the simulator, that's use case-specific.  libgloss indeed provides an
implementation, so I guess you could include that in your program:

  https://sourceware.org/git/?p=newlib-cygwin.git;a=blob;f=libgloss/aarch64/_exit.c;h=16564bbdaaaa3fa20c68198903432dfbf7f768a7;hb=HEAD

But for the GDB simulator, you can just provide a simple implementation of _exit.  In my case,
I just defined it as an empty loop, but maybe there's a way to signal the simulator to shut
down, I don't know.

---
void _exit(int status)
{
  for (;;);
}

int x = 0;

int main(void)
{
  for (int i = 0; i < 1000; i++) {
      x++;
  }
  return 0;
}
---

Compiled with:

$ aarch64-none-elf-gcc test.c -g3 -O0 -o test

or

$ arm-none-eabi-gcc test.c -g3 -O0 -o test

With my arm binary, here's how I can run it with the simulator:

$ ./gdb -nx -q --data-directory=data-directory ./test -ex "target sim" -ex "load" -ex "b main" -ex "run"
Reading symbols from ./test...
Connected to the simulator.
Loading section .init, size 0x18 lma 0x8000
Loading section .text, size 0x630 lma 0x8018
Loading section .fini, size 0x18 lma 0x8648
Loading section .rodata, size 0x4 lma 0x8660
Loading section .ARM.exidx, size 0x8 lma 0x8664
Loading section .eh_frame, size 0x4 lma 0x866c
Loading section .init_array, size 0x4 lma 0x18670
Loading section .fini_array, size 0x4 lma 0x18674
Loading section .data, size 0x430 lma 0x18678
Start address 0x8128
Transfer rate: 21824 bits in <1 sec.
Breakpoint 1 at 0x824c: file test.c, line 10.
Starting program: /home/smarchi/build/binutils-gdb-target-arm/gdb/test

Breakpoint 1, main () at test.c:10
10        for (int i = 0; i < 1000; i++) {
(gdb) n
11            x++;
(gdb)
10        for (int i = 0; i < 1000; i++) {

I tried to build a GDB for the AArch64 target, but it did not include the simulator.  I then
did the following change to enable it:

diff --git a/gdb/configure.tgt b/gdb/configure.tgt
index b3f31af763c..1dcd153f52f 100644
--- a/gdb/configure.tgt
+++ b/gdb/configure.tgt
@@ -114,6 +114,7 @@ case "${targ}" in
 aarch64*-*-elf | aarch64*-*-rtems*)
        # Target: AArch64 embedded system
        gdb_target_obs="aarch64-newlib-tdep.o"
+       gdb_sim=../sim/aarch64/libsim.a
        ;;

This got me a GDB for the AArch64 target with simulator support.  However, it does not
run correctly:

$ ./gdb -nx -q --data-directory=data-directory ./test -ex "target sim" -ex "load" -ex "b main" -ex "run"
Reading symbols from ./test...
Connected to the simulator.
Loading section .init, size 0x34 lma 0x400000
Loading section .text, size 0x67c lma 0x400040
Loading section .fini, size 0x34 lma 0x4006bc
Loading section .rodata, size 0x58 lma 0x4006f0
Loading section .eh_frame, size 0x4 lma 0x400748
Loading section .init_array, size 0x8 lma 0x410750
Loading section .fini_array, size 0x8 lma 0x410758
Loading section .data, size 0x758 lma 0x410760
Start address 0x400168
Transfer rate: 30016 bits in <1 sec.
Breakpoint 1 at 0x4001e8: file test.c, line 10.
Starting program: /home/smarchi/build/binutils-gdb-target-aarch64/gdb/test
core: 8 byte write to unmapped address 0xfffffff0 at 0x0

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()


I did not push the investigation further.

Simon

Re: Running programs on aarch64 simulator

[Luis Machado]

On 5/8/20 10:44 AM, Joel Sherrill wrote:
> Hi
> 
> Can someone please provide an example of how to compile a program to run on
> the aarch64 simulator in gdb? The simple "aarch64-elf main.c -o main" does
> not link and appears to be missing libgloss type symbols.
> 
> I asked on newlib but got no response. I'm hoping someone here knows.
> 
> Thanks.
> 
> --joel
> RTEMS
> 

Would QEMU be a better option to exercise some bare-metal programs? The 
GDB simulator has not been receiving a lot of maintenance.

Re: Running programs on aarch64 simulator

[Joel Sherrill]

On Fri, May 8, 2020 at 11:16 AM Luis Machado <luis.machado@linaro.org>
wrote:

> On 5/8/20 10:44 AM, Joel Sherrill wrote:
> > Hi
> >
> > Can someone please provide an example of how to compile a program to run
> on
> > the aarch64 simulator in gdb? The simple "aarch64-elf main.c -o main"
> does
> > not link and appears to be missing libgloss type symbols.
> >
> > I asked on newlib but got no response. I'm hoping someone here knows.
> >
> > Thanks.
> >
> > --joel
> > RTEMS
> >
>
> Would QEMU be a better option to exercise some bare-metal programs? The
> GDB simulator has not been receiving a lot of maintenance.
>

Long term it is a better option but the gdb simulators have proven more
than
sufficient to work through the basic issues of an RTEMS port and they are
quite simple to use. You can get initialization, context switch, etc.
Definitely
have to move to Qemu to get interrupts worked out and real device drivers.

--joel

Re: Running programs on aarch64 simulator

[Joel Sherrill]

On Fri, May 8, 2020 at 11:09 AM Simon Marchi <simark@simark.ca> wrote:

> On 2020-05-08 9:44 a.m., Joel Sherrill wrote:
> > Hi
> >
> > Can someone please provide an example of how to compile a program to run
> on
> > the aarch64 simulator in gdb? The simple "aarch64-elf main.c -o main"
> does
> > not link and appears to be missing libgloss type symbols.
> >
> > I asked on newlib but got no response. I'm hoping someone here knows.
> >
> > Thanks.
> >
> > --joel
> > RTEMS
> >
>
> Hi Joel,
>
> If I just compile a "main" function with the bare-metal aarch64 or arm
> compiler, it does indeed
> complain about a missing _exit function:
>
> <snip>/gcc/aarch64-none-elf/9.2.1/../../../../aarch64-none-elf/lib/libg.a(lib_a-exit.o):
> in function `exit':
> <snip>/build/src/newlib-cygwin/newlib/libc/stdlib/exit.c:64: undefined
> reference to `_exit'
>
> I think this is expected, as the runtime requires you to provide some
> implementation for _exit,
> in other words what you want to do when the program has finished.  Maybe
> shut down the board,
> reset it, shut down the simulator, that's use case-specific.  libgloss
> indeed provides an
> implementation, so I guess you could include that in your program:
>
>
> https://sourceware.org/git/?p=newlib-cygwin.git;a=blob;f=libgloss/aarch64/_exit.c;h=16564bbdaaaa3fa20c68198903432dfbf7f768a7;hb=HEAD
>
> But for the GDB simulator, you can just provide a simple implementation of
> _exit.  In my case,
> I just defined it as an empty loop, but maybe there's a way to signal the
> simulator to shut
> down, I don't know.
>

On all the elf/eabi targets I have tried over the years, CPU-elf-gcc
produced an a.out that was linked with the installed libgloss. I randomly
had an sh-elf toolchain laying around and it worked like "sh-elf-gcc m.c"
and then both sh-elf-gdb and sh-elf-run could run it on the gdb simulator.

The aarch64-elf toolchain installs 4 .specs files from libgloss but adding
-specs XXX and trying each with the gdb simulator (via aarch64-elf-run) all
fail.

home/joel/test-gcc/install-master/aarch64-elf/lib/aem-validation.specs
/home/joel/test-gcc/install-master/aarch64-elf/lib/aem-ve.specs
/home/joel/test-gcc/install-master/aarch64-elf/lib/nosys.specs
/home/joel/test-gcc/install-master/aarch64-elf/lib/rdimon.specs

This seems to be an odd case where there is a simulator in the source tree
and there is no clear way to use it. Assuming it works.

Looks like Nick Clifton added it back in 2015. Surely he had a way to run
programs on it. :)




> ---
> void _exit(int status)
> {
>   for (;;);
> }
>
> int x = 0;
>
> int main(void)
> {
>   for (int i = 0; i < 1000; i++) {
>       x++;
>   }
>   return 0;
> }
> ---
>
> Compiled with:
>
> $ aarch64-none-elf-gcc test.c -g3 -O0 -o test
>
> or
>
> $ arm-none-eabi-gcc test.c -g3 -O0 -o test
>
> With my arm binary, here's how I can run it with the simulator:
>
> $ ./gdb -nx -q --data-directory=data-directory ./test -ex "target sim" -ex
> "load" -ex "b main" -ex "run"
> Reading symbols from ./test...
> Connected to the simulator.
> Loading section .init, size 0x18 lma 0x8000
> Loading section .text, size 0x630 lma 0x8018
> Loading section .fini, size 0x18 lma 0x8648
> Loading section .rodata, size 0x4 lma 0x8660
> Loading section .ARM.exidx, size 0x8 lma 0x8664
> Loading section .eh_frame, size 0x4 lma 0x866c
> Loading section .init_array, size 0x4 lma 0x18670
> Loading section .fini_array, size 0x4 lma 0x18674
> Loading section .data, size 0x430 lma 0x18678
> Start address 0x8128
> Transfer rate: 21824 bits in <1 sec.
> Breakpoint 1 at 0x824c: file test.c, line 10.
> Starting program: /home/smarchi/build/binutils-gdb-target-arm/gdb/test
>
> Breakpoint 1, main () at test.c:10
> 10        for (int i = 0; i < 1000; i++) {
> (gdb) n
> 11            x++;
> (gdb)
> 10        for (int i = 0; i < 1000; i++) {
>
> I tried to build a GDB for the AArch64 target, but it did not include the
> simulator.  I then
> did the following change to enable it:
>
> diff --git a/gdb/configure.tgt b/gdb/configure.tgt
> index b3f31af763c..1dcd153f52f 100644
> --- a/gdb/configure.tgt
> +++ b/gdb/configure.tgt
> @@ -114,6 +114,7 @@ case "${targ}" in
>  aarch64*-*-elf | aarch64*-*-rtems*)
>         # Target: AArch64 embedded system
>         gdb_target_obs="aarch64-newlib-tdep.o"
> +       gdb_sim=../sim/aarch64/libsim.a
>         ;;
>
> This got me a GDB for the AArch64 target with simulator support.  However,
> it does not
> run correctly:
>
> $ ./gdb -nx -q --data-directory=data-directory ./test -ex "target sim" -ex
> "load" -ex "b main" -ex "run"
> Reading symbols from ./test...
> Connected to the simulator.
> Loading section .init, size 0x34 lma 0x400000
> Loading section .text, size 0x67c lma 0x400040
> Loading section .fini, size 0x34 lma 0x4006bc
> Loading section .rodata, size 0x58 lma 0x4006f0
> Loading section .eh_frame, size 0x4 lma 0x400748
> Loading section .init_array, size 0x8 lma 0x410750
> Loading section .fini_array, size 0x8 lma 0x410758
> Loading section .data, size 0x758 lma 0x410760
> Start address 0x400168
> Transfer rate: 30016 bits in <1 sec.
> Breakpoint 1 at 0x4001e8: file test.c, line 10.
> Starting program: /home/smarchi/build/binutils-gdb-target-aarch64/gdb/test
> core: 8 byte write to unmapped address 0xfffffff0 at 0x0
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x0000000000000000 in ?? ()
>

That matches what luck I had on master. I suspect that is a mismatch between
the address map of the simulator and whatever the default linker script
does.


>
> I did not push the investigation further.
>

I think Nick may be the key to getting an answer here.

Thanks.

--joel

>
> Simon
>

Re: Running programs on aarch64 simulator

[Simon Marchi]

On 2020-05-08 12:57 p.m., Joel Sherrill wrote:
> On all the elf/eabi targets I have tried over the years, CPU-elf-gcc produced an a.out that was linked with the installed libgloss. I randomly had an sh-elf toolchain laying around and it worked like "sh-elf-gcc m.c" and then both sh-elf-gdb and sh-elf-run could run it on the gdb simulator.
> 
> The aarch64-elf toolchain installs 4 .specs files from libgloss but adding -specs XXX and trying each with the gdb simulator (via aarch64-elf-run) all fail.
> 
> home/joel/test-gcc/install-master/aarch64-elf/lib/aem-validation.specs
> /home/joel/test-gcc/install-master/aarch64-elf/lib/aem-ve.specs
> /home/joel/test-gcc/install-master/aarch64-elf/lib/nosys.specs
> /home/joel/test-gcc/install-master/aarch64-elf/lib/rdimon.specs

Ok, I am not familiar with that.

Well, it did work for me to build with:

$ aarch64-none-elf-gcc test.c -g3 -O0 -o test -specs=nosys.specs

I didn't have to provide my own _exit.  According to the DWARF info, the _exit now
included in my program comes from:

  /tmp/dgboter/bbs/rhev-vm1--rhe6x86_64/buildbot/rhe6x86_64--aarch64-none-elf/build/src/newlib-cygwin/libgloss/libnosys/_exit.c

> 
> This seems to be an odd case where there is a simulator in the source tree and there is no clear way to use it. Assuming it works.

It's also possible to run it directly like this:

$ ./sim/aarch64/run gdb/test
core: 8 byte write to unmapped address 0xfffffff0 at 0x0
program stopped with signal 11 (Segmentation fault).

The result is the same as when I ran it through GDB.  I have no idea if it's the sim that
is faulty, or the binary needs to be compiled differently.

> 
>     Program received signal SIGSEGV, Segmentation fault.
>     0x0000000000000000 in ?? ()
> 
> 
> That matches what luck I had on master. I suspect that is a mismatch between
> the address map of the simulator and whatever the default linker script does.

Perhaps.  With the ARM simulator, when I do "starti" in GDB, I see that it starts
executing at the ELF file's entry point.  With the AArch64 simulator, it starts
at 0 (the entry point of the ELF is not 0).  So I also suspect that the initial
PC is not right.

> 
>     I did not push the investigation further.
> 
> 
> I think Nick may be the key to getting an answer here. 

Added him in CC, we'll see :).

Simon

Re: Running programs on aarch64 simulator

[Simon Marchi]

On 2020-05-08 1:17 p.m., Simon Marchi wrote:
> It's also possible to run it directly like this:
> 
> $ ./sim/aarch64/run gdb/test
> core: 8 byte write to unmapped address 0xfffffff0 at 0x0
> program stopped with signal 11 (Segmentation fault).
> 
> The result is the same as when I ran it through GDB.  I have no idea if it's the sim that
> is faulty, or the binary needs to be compiled differently.
> 
>>
>>     Program received signal SIGSEGV, Segmentation fault.
>>     0x0000000000000000 in ?? ()
>>
>>
>> That matches what luck I had on master. I suspect that is a mismatch between
>> the address map of the simulator and whatever the default linker script does.
> 
> Perhaps.  With the ARM simulator, when I do "starti" in GDB, I see that it starts
> executing at the ELF file's entry point.  With the AArch64 simulator, it starts
> at 0 (the entry point of the ELF is not 0).  So I also suspect that the initial
> PC is not right.

I was wrong here, the PC is fine.  I found how to enable traces when running the simulator
(not that it's complicated).  It crashes after a few instructions:

$ ./sim/aarch64/run --trace=on --trace-disasm=on ./sim/aarch64/a.out
memory:   ERROR: executable is too big: ffffffffffffffff
insn:      pc = 400168 instr = 58000281
disasm:   ldr   x1, 0x00000000004001b8
memory:   read of 0 (8 bytes) from 4001b8
insn:      pc = 40016c instr = 927cec20
disasm:   and   x0, x1, #0xfffffffffffffff0
insn:      pc = 400170 instr = 9100001f
disasm:   mov   sp, x0
insn:      pc = 400174 instr = d280001d
disasm:   mov   x29, #0x0                       // #0
insn:      pc = 400178 instr = a9bf77fd
disasm:   stp   x29, x29, [sp, #-16]!
memory:   write of 0 (8 bytes) to fffffffffffffff0
core: 8 byte write to unmapped address 0xfffffff0 at 0x0
program stopped with signal 11 (Segmentation fault).

What seems to happen is that the code tries to set up the stack pointer, but its initial value
read from 0x4001b8 is 0.  So it crashes when we try to write to the stack.

I have no idea why, I posted this question here:

  https://sourceware.org/pipermail/newlib/2020/017635.html

Simon

Re: Running programs on aarch64 simulator

[Nick Clifton]

Hi Guys, 
>> I think Nick may be the key to getting an answer here. 
> Added him in CC, we'll see :).

Gosh - it has been a long time since I played with that simulator.

It should work.  Or at least it used to work.  But maybe things have changed.

Would one of you mind doing me a favour though and send me a binary that
needs to be simulated ?  I no longer have the infrastructure set up for
building aarch64 binaries, but in theory that should be no obstacle to 
simulating them...

Cheers
  Nick

Re: Running programs on aarch64 simulator

[Joel Sherrill]

On Mon, May 11, 2020 at 9:26 AM Nick Clifton <nickc@redhat.com> wrote:

> Hi Guys,
> >> I think Nick may be the key to getting an answer here.
> > Added him in CC, we'll see :).
>
> Gosh - it has been a long time since I played with that simulator.
>
> It should work.  Or at least it used to work.  But maybe things have
> changed.
>
> Would one of you mind doing me a favour though and send me a binary that
> needs to be simulated ?  I no longer have the infrastructure set up for
> building aarch64 binaries, but in theory that should be no obstacle to
> simulating them...
>

I'm trying to build one that will run on the simulator. I haven't managed to
do that yet. :(

Sorry.

--joel

>
> Cheers
>   Nick
>
>
>

Re: Running programs on aarch64 simulator

[Jim Wilson]

On Mon, May 11, 2020 at 7:41 AM Joel Sherrill <joel@rtems.org> wrote:
> I'm trying to build one that will run on the simulator. I haven't managed to
> do that yet. :(

You can find the answer in dejagnu.  See baseboards/aarch64-sim.exp
which uses -specs=rdimon.specs.

The aarch64 gdb sim is incomplete by the way.  ARM won't let ARM
engineers contribute to it because of IP concerns.  I did some work on
it when I was at Linaro, but there are still a number of instructions
that are missing or implemented wrong.  I got the gcc testsuite
failures down to about 1200 before I left Linaro.  I don't test it
anymore, so I don't know the current state.  I would suggest using
qemu instead unless you want to fix simulator bugs.

Jim

Re: Running programs on aarch64 simulator

[Joel Sherrill]

On Mon, May 11, 2020 at 12:59 PM Jim Wilson <jimw@sifive.com> wrote:

> On Mon, May 11, 2020 at 7:41 AM Joel Sherrill <joel@rtems.org> wrote:
> > I'm trying to build one that will run on the simulator. I haven't
> managed to
> > do that yet. :(
>
> You can find the answer in dejagnu.  See baseboards/aarch64-sim.exp
> which uses -specs=rdimon.specs.
>
> The aarch64 gdb sim is incomplete by the way.  ARM won't let ARM
> engineers contribute to it because of IP concerns.  I did some work on
> it when I was at Linaro, but there are still a number of instructions
> that are missing or implemented wrong.  I got the gcc testsuite
> failures down to about 1200 before I left Linaro.  I don't test it
> anymore, so I don't know the current state.  I would suggest using
> qemu instead unless you want to fix simulator bugs.
>

Thanks Jim.  That is very helpful.  I managed to get an executable
to run with aarch64-elf-run and traced it failing quite early at
newlib-cygwin/libgloss/aarch64/syscalls.c:287 doing something with
the semi-hosting extensions.  Have you seen this simulator work
well enough recently to run the gcc testsuite?

It sounds like for my purposes of where to start development of an
aarch64 RTEMS port, this is one time we need to skip the gdb simulator
step. The gdb simulators are just so nice to work through basic issues
like context switching, stack setup, etc. Since we would only need C
with basic integer operations and enough FPU support to context
switch the FPU, 1200 gcc failures may not have been a stopper.

I know that it has limited use since there are so many alternatives
and embedded folks are just going to use CMSIS or some OS support.
I was just hoping to use it to add to the free RTOS list. Guess we will
start with Qemu unless someone has some brilliant idea that makes
this work.

Thanks again Jim.

--joel

>
> Jim
>

Re: Running programs on aarch64 simulator

[Jim Wilson]

On Mon, May 11, 2020 at 11:30 AM Joel Sherrill <joel@rtems.org> wrote:
> Thanks Jim.  That is very helpful.  I managed to get an executable
> to run with aarch64-elf-run and traced it failing quite early at
> newlib-cygwin/libgloss/aarch64/syscalls.c:287 doing something with
> the semi-hosting extensions.  Have you seen this simulator work
> well enough recently to run the gcc testsuite?

It has been maybe 2.5 years since I last tried.  That is about when I
left Linaro.

> It sounds like for my purposes of where to start development of an
> aarch64 RTEMS port, this is one time we need to skip the gdb simulator
> step. The gdb simulators are just so nice to work through basic issues
> like context switching, stack setup, etc. Since we would only need C
> with basic integer operations and enough FPU support to context
> switch the FPU, 1200 gcc failures may not have been a stopper.

I don't know if any of the system instructions work.  A gcc testsuite
run will only test user instructions.  I like the gdb simulators too.
That is why I did some work with it.

Jim