[QUERY]: Adding support for a platform

[QUERY]: Adding support for a platform

[Lad, Prabhakar]

Hi All,

If this is not the right place to ask this question please let me know.

So we have a RISCV platform (Renesas RZ/Five) for which we need to
adjust the TEXT_START_ADDR. So below are my queries:
* What is the procedure for upstreaming?
* Are patches accepted for individual platforms (for adjusting TEXT_START_ADDR)?

Cheers,
Prabhakar

Re: [QUERY]: Adding support for a platform

[Palmer Dabbelt]

On Tue, 29 Nov 2022 11:16:29 PST (-0800), binutils@sourceware.org wrote:
> Hi All,
>
> If this is not the right place to ask this question please let me know.
>
> So we have a RISCV platform (Renesas RZ/Five) for which we need to
> adjust the TEXT_START_ADDR. So below are my queries:
> * What is the procedure for upstreaming?
> * Are patches accepted for individual platforms (for adjusting TEXT_START_ADDR)?

Is this still related to that bug with the static binaries you'd 
mentioned before?  I think we'd really need to figure out the root cause 
before we can make a reasoned decision here, my guess would be that 
changing TEXT_START_ADDR just works around the real bug.

> Cheers,
> Prabhakar

Re: [QUERY]: Adding support for a platform

[Lad, Prabhakar]

Hi Palmer,

Oops I should have included linux-riscv to the CC list here (doing it now)

On Tue, Nov 29, 2022 at 7:20 PM Palmer Dabbelt <palmer@dabbelt.com> wrote:
>
> On Tue, 29 Nov 2022 11:16:29 PST (-0800), binutils@sourceware.org wrote:
> > Hi All,
> >
> > If this is not the right place to ask this question please let me know.
> >
> > So we have a RISCV platform (Renesas RZ/Five) for which we need to
> > adjust the TEXT_START_ADDR. So below are my queries:
> > * What is the procedure for upstreaming?
> > * Are patches accepted for individual platforms (for adjusting TEXT_START_ADDR)?
>
> Is this still related to that bug with the static binaries you'd
> mentioned before?  I think we'd really need to figure out the root cause
> before we can make a reasoned decision here, my guess would be that
> changing TEXT_START_ADDR just works around the real bug.
>
Below is the root cause and the solution:

TEXT_START_ADDR is the start of the text segment of an application.
This is being set to 0x10000 for RISCV platforms.

So when an application is compiled with the static flag the load would
start from 0x10000 - xyz (depending on size of the application)

Entry point 0x101c0
There are 5 program headers, starting at offset 64Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000000 0x0000000000010000 0x0000000000010000
                 0x0000000000059b48 0x0000000000059b48  R E    0x1000
  LOAD           0x0000000000059b60 0x000000000006ab60 0x000000000006ab60
                 0x0000000000001f68 0x0000000000003528  RW     0x1000

So for the above application which is compiled statically we can see
the entry point is 0x101c0 and load 0x0000000000010000.

Andes AX45MP cores have local memory ILM and DLM that are mapped in
the region H’0_0003_0000 - H’0_0004_FFFF on the RZ/Five SoC. When the
virtual address falls in this range the MMU doesn't trigger a page
fault and assumes the virtual address as physical address and hence
the application fails to run (panics somewhere).

So to avoid this issue we set the TEXT_START_ADDR to 0x50000 so that
the virtual address of any statically compiled application does not
fall in the range of H’0_0003_0000 - H’0_0004_FFFF.

Elf file type is EXEC (Executable file)
Entry point 0x504e4
There are 5 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000000 0x0000000000050000 0x0000000000050000
                 0x0000000000057dc8 0x0000000000057dc8  R E    0x1000
  LOAD           0x00000000000585b8 0x00000000000a95b8 0x00000000000a95b8
                 0x0000000000004ee0 0x00000000000064b0  RW     0x1000
  NOTE           0x0000000000000158 0x0000000000050158 0x0000000000050158
                 0x0000000000000044 0x0000000000000044  R      0x4

So now with the fix for statically compiled applications we can see
its offsetted and entry point is 0x504e4 and load is at
0x0000000000050000. So with this we are for sure the MMU will always
trigger a page fault.

Cheers,
Prabhakar

Re: [QUERY]: Adding support for a platform

[Palmer Dabbelt]

On Tue, 29 Nov 2022 11:28:11 PST (-0800), prabhakar.csengg@gmail.com wrote:
> Hi Palmer,
>
> Oops I should have included linux-riscv to the CC list here (doing it now)
>
> On Tue, Nov 29, 2022 at 7:20 PM Palmer Dabbelt <palmer@dabbelt.com> wrote:
>>
>> On Tue, 29 Nov 2022 11:16:29 PST (-0800), binutils@sourceware.org wrote:
>> > Hi All,
>> >
>> > If this is not the right place to ask this question please let me know.
>> >
>> > So we have a RISCV platform (Renesas RZ/Five) for which we need to
>> > adjust the TEXT_START_ADDR. So below are my queries:
>> > * What is the procedure for upstreaming?
>> > * Are patches accepted for individual platforms (for adjusting TEXT_START_ADDR)?
>>
>> Is this still related to that bug with the static binaries you'd
>> mentioned before?  I think we'd really need to figure out the root cause
>> before we can make a reasoned decision here, my guess would be that
>> changing TEXT_START_ADDR just works around the real bug.
>>
> Below is the root cause and the solution:
>
> TEXT_START_ADDR is the start of the text segment of an application.
> This is being set to 0x10000 for RISCV platforms.
>
> So when an application is compiled with the static flag the load would
> start from 0x10000 - xyz (depending on size of the application)
>
> Entry point 0x101c0
> There are 5 program headers, starting at offset 64Program Headers:
>   Type           Offset             VirtAddr           PhysAddr
>                  FileSiz            MemSiz              Flags  Align
>   LOAD           0x0000000000000000 0x0000000000010000 0x0000000000010000
>                  0x0000000000059b48 0x0000000000059b48  R E    0x1000
>   LOAD           0x0000000000059b60 0x000000000006ab60 0x000000000006ab60
>                  0x0000000000001f68 0x0000000000003528  RW     0x1000
>
> So for the above application which is compiled statically we can see
> the entry point is 0x101c0 and load 0x0000000000010000.
>
> Andes AX45MP cores have local memory ILM and DLM that are mapped in
> the region H’0_0003_0000 - H’0_0004_FFFF on the RZ/Five SoC. When the
> virtual address falls in this range the MMU doesn't trigger a page
> fault and assumes the virtual address as physical address and hence
> the application fails to run (panics somewhere).
>
> So to avoid this issue we set the TEXT_START_ADDR to 0x50000 so that
> the virtual address of any statically compiled application does not
> fall in the range of H’0_0003_0000 - H’0_0004_FFFF.
>
> Elf file type is EXEC (Executable file)
> Entry point 0x504e4
> There are 5 program headers, starting at offset 64
>
> Program Headers:
>   Type           Offset             VirtAddr           PhysAddr
>                  FileSiz            MemSiz              Flags  Align
>   LOAD           0x0000000000000000 0x0000000000050000 0x0000000000050000
>                  0x0000000000057dc8 0x0000000000057dc8  R E    0x1000
>   LOAD           0x00000000000585b8 0x00000000000a95b8 0x00000000000a95b8
>                  0x0000000000004ee0 0x00000000000064b0  RW     0x1000
>   NOTE           0x0000000000000158 0x0000000000050158 0x0000000000050158
>                  0x0000000000000044 0x0000000000000044  R      0x4
>
> So now with the fix for statically compiled applications we can see
> its offsetted and entry point is 0x504e4 and load is at
> 0x0000000000050000. So with this we are for sure the MMU will always
> trigger a page fault.

OK, cool, that explains it.

So I think we need to teach Linux about this platform-specific behavior 
so we can prevent any addresses in the offending VA range from being 
allocated.  Then it'll be up to userspace to deal with the fallout, if 
that means changing TEXT_START_ADDRESS it seems fine to me.  I'm not 
sure if there's already some sort of autoconf-type thing to change the 
default in binutils, if not then adding one seems like the way to go.

Then it's up to distros that want to run on the impacted hardware to 
make sure they rebuild all the impacted binaries.  I don't think there's 
any way around that for userspace-visible errata, and it this case the 
check for impacted binaries seems pretty manageable to write down (some 
sort of `find | objdump | grep` thing would probably do it).

> Cheers,
> Prabhakar

Re: [QUERY]: Adding support for a platform

[Jessica Clarke]

On 29 Nov 2022, at 19:28, Lad, Prabhakar <prabhakar.csengg@gmail.com> wrote:
> 
> Hi Palmer,
> 
> Oops I should have included linux-riscv to the CC list here (doing it now)
> 
> On Tue, Nov 29, 2022 at 7:20 PM Palmer Dabbelt <palmer@dabbelt.com> wrote:
>> 
>> On Tue, 29 Nov 2022 11:16:29 PST (-0800), binutils@sourceware.org wrote:
>>> Hi All,
>>> 
>>> If this is not the right place to ask this question please let me know.
>>> 
>>> So we have a RISCV platform (Renesas RZ/Five) for which we need to
>>> adjust the TEXT_START_ADDR. So below are my queries:
>>> * What is the procedure for upstreaming?
>>> * Are patches accepted for individual platforms (for adjusting TEXT_START_ADDR)?
>> 
>> Is this still related to that bug with the static binaries you'd
>> mentioned before?  I think we'd really need to figure out the root cause
>> before we can make a reasoned decision here, my guess would be that
>> changing TEXT_START_ADDR just works around the real bug.
>> 
> Below is the root cause and the solution:
> 
> TEXT_START_ADDR is the start of the text segment of an application.
> This is being set to 0x10000 for RISCV platforms.
> 
> So when an application is compiled with the static flag the load would
> start from 0x10000 - xyz (depending on size of the application)
> 
> Entry point 0x101c0
> There are 5 program headers, starting at offset 64Program Headers:
>  Type           Offset             VirtAddr           PhysAddr
>                 FileSiz            MemSiz              Flags  Align
>  LOAD           0x0000000000000000 0x0000000000010000 0x0000000000010000
>                 0x0000000000059b48 0x0000000000059b48  R E    0x1000
>  LOAD           0x0000000000059b60 0x000000000006ab60 0x000000000006ab60
>                 0x0000000000001f68 0x0000000000003528  RW     0x1000
> 
> So for the above application which is compiled statically we can see
> the entry point is 0x101c0 and load 0x0000000000010000.
> 
> Andes AX45MP cores have local memory ILM and DLM that are mapped in
> the region H’0_0003_0000 - H’0_0004_FFFF on the RZ/Five SoC. When the
> virtual address falls in this range the MMU doesn't trigger a page
> fault and assumes the virtual address as physical address and hence
> the application fails to run (panics somewhere).
> 
> So to avoid this issue we set the TEXT_START_ADDR to 0x50000 so that
> the virtual address of any statically compiled application does not
> fall in the range of H’0_0003_0000 - H’0_0004_FFFF.
> 
> Elf file type is EXEC (Executable file)
> Entry point 0x504e4
> There are 5 program headers, starting at offset 64
> 
> Program Headers:
>  Type           Offset             VirtAddr           PhysAddr
>                 FileSiz            MemSiz              Flags  Align
>  LOAD           0x0000000000000000 0x0000000000050000 0x0000000000050000
>                 0x0000000000057dc8 0x0000000000057dc8  R E    0x1000
>  LOAD           0x00000000000585b8 0x00000000000a95b8 0x00000000000a95b8
>                 0x0000000000004ee0 0x00000000000064b0  RW     0x1000
>  NOTE           0x0000000000000158 0x0000000000050158 0x0000000000050158
>                 0x0000000000000044 0x0000000000000044  R      0x4
> 
> So now with the fix for statically compiled applications we can see
> its offsetted and entry point is 0x504e4 and load is at
> 0x0000000000050000. So with this we are for sure the MMU will always
> trigger a page fault.

Well that’s just a blatant violation of the spec. What happens if a
malicious (or buggy) userspace binary goes and accesses unmapped memory
in that range? Does it actually get access to this ILM/DLM? If so
that’s a gaping side channel.

Jess

Re: [QUERY]: Adding support for a platform

[Andrew Waterman]

On Tue, Nov 29, 2022 at 12:12 PM Jessica Clarke <jrtc27@jrtc27.com> wrote:
>
> On 29 Nov 2022, at 19:28, Lad, Prabhakar <prabhakar.csengg@gmail.com> wrote:
> >
> > Hi Palmer,
> >
> > Oops I should have included linux-riscv to the CC list here (doing it now)
> >
> > On Tue, Nov 29, 2022 at 7:20 PM Palmer Dabbelt <palmer@dabbelt.com> wrote:
> >>
> >> On Tue, 29 Nov 2022 11:16:29 PST (-0800), binutils@sourceware.org wrote:
> >>> Hi All,
> >>>
> >>> If this is not the right place to ask this question please let me know.
> >>>
> >>> So we have a RISCV platform (Renesas RZ/Five) for which we need to
> >>> adjust the TEXT_START_ADDR. So below are my queries:
> >>> * What is the procedure for upstreaming?
> >>> * Are patches accepted for individual platforms (for adjusting TEXT_START_ADDR)?
> >>
> >> Is this still related to that bug with the static binaries you'd
> >> mentioned before?  I think we'd really need to figure out the root cause
> >> before we can make a reasoned decision here, my guess would be that
> >> changing TEXT_START_ADDR just works around the real bug.
> >>
> > Below is the root cause and the solution:
> >
> > TEXT_START_ADDR is the start of the text segment of an application.
> > This is being set to 0x10000 for RISCV platforms.
> >
> > So when an application is compiled with the static flag the load would
> > start from 0x10000 - xyz (depending on size of the application)
> >
> > Entry point 0x101c0
> > There are 5 program headers, starting at offset 64Program Headers:
> >  Type           Offset             VirtAddr           PhysAddr
> >                 FileSiz            MemSiz              Flags  Align
> >  LOAD           0x0000000000000000 0x0000000000010000 0x0000000000010000
> >                 0x0000000000059b48 0x0000000000059b48  R E    0x1000
> >  LOAD           0x0000000000059b60 0x000000000006ab60 0x000000000006ab60
> >                 0x0000000000001f68 0x0000000000003528  RW     0x1000
> >
> > So for the above application which is compiled statically we can see
> > the entry point is 0x101c0 and load 0x0000000000010000.
> >
> > Andes AX45MP cores have local memory ILM and DLM that are mapped in
> > the region H’0_0003_0000 - H’0_0004_FFFF on the RZ/Five SoC. When the
> > virtual address falls in this range the MMU doesn't trigger a page
> > fault and assumes the virtual address as physical address and hence
> > the application fails to run (panics somewhere).
> >
> > So to avoid this issue we set the TEXT_START_ADDR to 0x50000 so that
> > the virtual address of any statically compiled application does not
> > fall in the range of H’0_0003_0000 - H’0_0004_FFFF.
> >
> > Elf file type is EXEC (Executable file)
> > Entry point 0x504e4
> > There are 5 program headers, starting at offset 64
> >
> > Program Headers:
> >  Type           Offset             VirtAddr           PhysAddr
> >                 FileSiz            MemSiz              Flags  Align
> >  LOAD           0x0000000000000000 0x0000000000050000 0x0000000000050000
> >                 0x0000000000057dc8 0x0000000000057dc8  R E    0x1000
> >  LOAD           0x00000000000585b8 0x00000000000a95b8 0x00000000000a95b8
> >                 0x0000000000004ee0 0x00000000000064b0  RW     0x1000
> >  NOTE           0x0000000000000158 0x0000000000050158 0x0000000000050158
> >                 0x0000000000000044 0x0000000000000044  R      0x4
> >
> > So now with the fix for statically compiled applications we can see
> > its offsetted and entry point is 0x504e4 and load is at
> > 0x0000000000050000. So with this we are for sure the MMU will always
> > trigger a page fault.
>
> Well that’s just a blatant violation of the spec.

Right - this isn't a platform quirk; it's a violation of the ISA.
Unless this behavior is engaged by a custom mode bit, that is, in
which case, it should just be disabled by default and none of this
would be a problem.  If the behavior is unconditionally enabled, this
should be treated as an erratum.

> What happens if a
> malicious (or buggy) userspace binary goes and accesses unmapped memory
> in that range? Does it actually get access to this ILM/DLM? If so
> that’s a gaping side channel.
>
> Jess
>

Re: [QUERY]: Adding support for a platform

[Palmer Dabbelt]

On Tue, 29 Nov 2022 12:21:31 PST (-0800), Andrew Waterman wrote:
> On Tue, Nov 29, 2022 at 12:12 PM Jessica Clarke <jrtc27@jrtc27.com> wrote:
>>
>> On 29 Nov 2022, at 19:28, Lad, Prabhakar <prabhakar.csengg@gmail.com> wrote:
>> >
>> > Hi Palmer,
>> >
>> > Oops I should have included linux-riscv to the CC list here (doing it now)
>> >
>> > On Tue, Nov 29, 2022 at 7:20 PM Palmer Dabbelt <palmer@dabbelt.com> wrote:
>> >>
>> >> On Tue, 29 Nov 2022 11:16:29 PST (-0800), binutils@sourceware.org wrote:
>> >>> Hi All,
>> >>>
>> >>> If this is not the right place to ask this question please let me know.
>> >>>
>> >>> So we have a RISCV platform (Renesas RZ/Five) for which we need to
>> >>> adjust the TEXT_START_ADDR. So below are my queries:
>> >>> * What is the procedure for upstreaming?
>> >>> * Are patches accepted for individual platforms (for adjusting TEXT_START_ADDR)?
>> >>
>> >> Is this still related to that bug with the static binaries you'd
>> >> mentioned before?  I think we'd really need to figure out the root cause
>> >> before we can make a reasoned decision here, my guess would be that
>> >> changing TEXT_START_ADDR just works around the real bug.
>> >>
>> > Below is the root cause and the solution:
>> >
>> > TEXT_START_ADDR is the start of the text segment of an application.
>> > This is being set to 0x10000 for RISCV platforms.
>> >
>> > So when an application is compiled with the static flag the load would
>> > start from 0x10000 - xyz (depending on size of the application)
>> >
>> > Entry point 0x101c0
>> > There are 5 program headers, starting at offset 64Program Headers:
>> >  Type           Offset             VirtAddr           PhysAddr
>> >                 FileSiz            MemSiz              Flags  Align
>> >  LOAD           0x0000000000000000 0x0000000000010000 0x0000000000010000
>> >                 0x0000000000059b48 0x0000000000059b48  R E    0x1000
>> >  LOAD           0x0000000000059b60 0x000000000006ab60 0x000000000006ab60
>> >                 0x0000000000001f68 0x0000000000003528  RW     0x1000
>> >
>> > So for the above application which is compiled statically we can see
>> > the entry point is 0x101c0 and load 0x0000000000010000.
>> >
>> > Andes AX45MP cores have local memory ILM and DLM that are mapped in
>> > the region H’0_0003_0000 - H’0_0004_FFFF on the RZ/Five SoC. When the
>> > virtual address falls in this range the MMU doesn't trigger a page
>> > fault and assumes the virtual address as physical address and hence
>> > the application fails to run (panics somewhere).
>> >
>> > So to avoid this issue we set the TEXT_START_ADDR to 0x50000 so that
>> > the virtual address of any statically compiled application does not
>> > fall in the range of H’0_0003_0000 - H’0_0004_FFFF.
>> >
>> > Elf file type is EXEC (Executable file)
>> > Entry point 0x504e4
>> > There are 5 program headers, starting at offset 64
>> >
>> > Program Headers:
>> >  Type           Offset             VirtAddr           PhysAddr
>> >                 FileSiz            MemSiz              Flags  Align
>> >  LOAD           0x0000000000000000 0x0000000000050000 0x0000000000050000
>> >                 0x0000000000057dc8 0x0000000000057dc8  R E    0x1000
>> >  LOAD           0x00000000000585b8 0x00000000000a95b8 0x00000000000a95b8
>> >                 0x0000000000004ee0 0x00000000000064b0  RW     0x1000
>> >  NOTE           0x0000000000000158 0x0000000000050158 0x0000000000050158
>> >                 0x0000000000000044 0x0000000000000044  R      0x4
>> >
>> > So now with the fix for statically compiled applications we can see
>> > its offsetted and entry point is 0x504e4 and load is at
>> > 0x0000000000050000. So with this we are for sure the MMU will always
>> > trigger a page fault.
>>
>> Well that’s just a blatant violation of the spec.
>
> Right - this isn't a platform quirk; it's a violation of the ISA.
> Unless this behavior is engaged by a custom mode bit, that is, in
> which case, it should just be disabled by default and none of this
> would be a problem.  If the behavior is unconditionally enabled, this
> should be treated as an erratum.

I don't see this as any different than the T-Head page table attributes, 
where it was made abundantly clear that vendors self-certify their 
implementations and thus anything goes.  It doesn't really matter if we 
called it an errata, bug, surprise feature, vendor extension, whatever; 
it's RISC-V so we've got to live with it.

>> What happens if a
>> malicious (or buggy) userspace binary goes and accesses unmapped memory
>> in that range? Does it actually get access to this ILM/DLM? If so
>> that’s a gaping side channel.

IIUC that's the case, but that's just from reading this post.

There's nothing we can do to prevent vendors from building hardware with 
side channels.  In this case I suppose we could swap out the memory 
region on context switches, from a quick look at the datasheet these are 
single-core so that'd be safe.  My guess is it'd be too slow for users 
to want that on by default, but I don't have any use case for this HW so 
I'm not sure.

>>
>> Jess
>>

Re: [QUERY]: Adding support for a platform

[Lad, Prabhakar]

Hi Palmer,

On Tue, Nov 29, 2022 at 7:35 PM Palmer Dabbelt <palmer@dabbelt.com> wrote:
>
> On Tue, 29 Nov 2022 11:28:11 PST (-0800), prabhakar.csengg@gmail.com wrote:
> > Hi Palmer,
> >
> > Oops I should have included linux-riscv to the CC list here (doing it now)
> >
> > On Tue, Nov 29, 2022 at 7:20 PM Palmer Dabbelt <palmer@dabbelt.com> wrote:
> >>
> >> On Tue, 29 Nov 2022 11:16:29 PST (-0800), binutils@sourceware.org wrote:
> >> > Hi All,
> >> >
> >> > If this is not the right place to ask this question please let me know.
> >> >
> >> > So we have a RISCV platform (Renesas RZ/Five) for which we need to
> >> > adjust the TEXT_START_ADDR. So below are my queries:
> >> > * What is the procedure for upstreaming?
> >> > * Are patches accepted for individual platforms (for adjusting TEXT_START_ADDR)?
> >>
> >> Is this still related to that bug with the static binaries you'd
> >> mentioned before?  I think we'd really need to figure out the root cause
> >> before we can make a reasoned decision here, my guess would be that
> >> changing TEXT_START_ADDR just works around the real bug.
> >>
> > Below is the root cause and the solution:
> >
> > TEXT_START_ADDR is the start of the text segment of an application.
> > This is being set to 0x10000 for RISCV platforms.
> >
> > So when an application is compiled with the static flag the load would
> > start from 0x10000 - xyz (depending on size of the application)
> >
> > Entry point 0x101c0
> > There are 5 program headers, starting at offset 64Program Headers:
> >   Type           Offset             VirtAddr           PhysAddr
> >                  FileSiz            MemSiz              Flags  Align
> >   LOAD           0x0000000000000000 0x0000000000010000 0x0000000000010000
> >                  0x0000000000059b48 0x0000000000059b48  R E    0x1000
> >   LOAD           0x0000000000059b60 0x000000000006ab60 0x000000000006ab60
> >                  0x0000000000001f68 0x0000000000003528  RW     0x1000
> >
> > So for the above application which is compiled statically we can see
> > the entry point is 0x101c0 and load 0x0000000000010000.
> >
> > Andes AX45MP cores have local memory ILM and DLM that are mapped in
> > the region H’0_0003_0000 - H’0_0004_FFFF on the RZ/Five SoC. When the
> > virtual address falls in this range the MMU doesn't trigger a page
> > fault and assumes the virtual address as physical address and hence
> > the application fails to run (panics somewhere).
> >
> > So to avoid this issue we set the TEXT_START_ADDR to 0x50000 so that
> > the virtual address of any statically compiled application does not
> > fall in the range of H’0_0003_0000 - H’0_0004_FFFF.
> >
> > Elf file type is EXEC (Executable file)
> > Entry point 0x504e4
> > There are 5 program headers, starting at offset 64
> >
> > Program Headers:
> >   Type           Offset             VirtAddr           PhysAddr
> >                  FileSiz            MemSiz              Flags  Align
> >   LOAD           0x0000000000000000 0x0000000000050000 0x0000000000050000
> >                  0x0000000000057dc8 0x0000000000057dc8  R E    0x1000
> >   LOAD           0x00000000000585b8 0x00000000000a95b8 0x00000000000a95b8
> >                  0x0000000000004ee0 0x00000000000064b0  RW     0x1000
> >   NOTE           0x0000000000000158 0x0000000000050158 0x0000000000050158
> >                  0x0000000000000044 0x0000000000000044  R      0x4
> >
> > So now with the fix for statically compiled applications we can see
> > its offsetted and entry point is 0x504e4 and load is at
> > 0x0000000000050000. So with this we are for sure the MMU will always
> > trigger a page fault.
>
> OK, cool, that explains it.
>
> So I think we need to teach Linux about this platform-specific behavior
> so we can prevent any addresses in the offending VA range from being
> allocated.
Agreed, any pointers on doing this?

> Then it'll be up to userspace to deal with the fallout, if
> that means changing TEXT_START_ADDRESS it seems fine to me.  I'm not
> sure if there's already some sort of autoconf-type thing to change the
> default in binutils, if not then adding one seems like the way to go.
>
TBH I haven't explored the code yet if it was configurable using
autoconf. I'll have to do some digging.

> Then it's up to distros that want to run on the impacted hardware to
> make sure they rebuild all the impacted binaries.  I don't think there's
> any way around that for userspace-visible errata, and it this case the
> check for impacted binaries seems pretty manageable to write down (some
> sort of `find | objdump | grep` thing would probably do it).
>
Agreed.

Cheers,
Prabhakar

Re: [QUERY]: Adding support for a platform

[Lad, Prabhakar]

Hi Palmer,

On Tue, Nov 29, 2022 at 8:57 PM Palmer Dabbelt <palmer@dabbelt.com> wrote:
>
> On Tue, 29 Nov 2022 12:21:31 PST (-0800), Andrew Waterman wrote:
> > On Tue, Nov 29, 2022 at 12:12 PM Jessica Clarke <jrtc27@jrtc27.com> wrote:
> >>
> >> On 29 Nov 2022, at 19:28, Lad, Prabhakar <prabhakar.csengg@gmail.com> wrote:
> >> >
> >> > Hi Palmer,
> >> >
> >> > Oops I should have included linux-riscv to the CC list here (doing it now)
> >> >
> >> > On Tue, Nov 29, 2022 at 7:20 PM Palmer Dabbelt <palmer@dabbelt.com> wrote:
> >> >>
> >> >> On Tue, 29 Nov 2022 11:16:29 PST (-0800), binutils@sourceware.org wrote:
> >> >>> Hi All,
> >> >>>
> >> >>> If this is not the right place to ask this question please let me know.
> >> >>>
> >> >>> So we have a RISCV platform (Renesas RZ/Five) for which we need to
> >> >>> adjust the TEXT_START_ADDR. So below are my queries:
> >> >>> * What is the procedure for upstreaming?
> >> >>> * Are patches accepted for individual platforms (for adjusting TEXT_START_ADDR)?
> >> >>
> >> >> Is this still related to that bug with the static binaries you'd
> >> >> mentioned before?  I think we'd really need to figure out the root cause
> >> >> before we can make a reasoned decision here, my guess would be that
> >> >> changing TEXT_START_ADDR just works around the real bug.
> >> >>
> >> > Below is the root cause and the solution:
> >> >
> >> > TEXT_START_ADDR is the start of the text segment of an application.
> >> > This is being set to 0x10000 for RISCV platforms.
> >> >
> >> > So when an application is compiled with the static flag the load would
> >> > start from 0x10000 - xyz (depending on size of the application)
> >> >
> >> > Entry point 0x101c0
> >> > There are 5 program headers, starting at offset 64Program Headers:
> >> >  Type           Offset             VirtAddr           PhysAddr
> >> >                 FileSiz            MemSiz              Flags  Align
> >> >  LOAD           0x0000000000000000 0x0000000000010000 0x0000000000010000
> >> >                 0x0000000000059b48 0x0000000000059b48  R E    0x1000
> >> >  LOAD           0x0000000000059b60 0x000000000006ab60 0x000000000006ab60
> >> >                 0x0000000000001f68 0x0000000000003528  RW     0x1000
> >> >
> >> > So for the above application which is compiled statically we can see
> >> > the entry point is 0x101c0 and load 0x0000000000010000.
> >> >
> >> > Andes AX45MP cores have local memory ILM and DLM that are mapped in
> >> > the region H’0_0003_0000 - H’0_0004_FFFF on the RZ/Five SoC. When the
> >> > virtual address falls in this range the MMU doesn't trigger a page
> >> > fault and assumes the virtual address as physical address and hence
> >> > the application fails to run (panics somewhere).
> >> >
> >> > So to avoid this issue we set the TEXT_START_ADDR to 0x50000 so that
> >> > the virtual address of any statically compiled application does not
> >> > fall in the range of H’0_0003_0000 - H’0_0004_FFFF.
> >> >
> >> > Elf file type is EXEC (Executable file)
> >> > Entry point 0x504e4
> >> > There are 5 program headers, starting at offset 64
> >> >
> >> > Program Headers:
> >> >  Type           Offset             VirtAddr           PhysAddr
> >> >                 FileSiz            MemSiz              Flags  Align
> >> >  LOAD           0x0000000000000000 0x0000000000050000 0x0000000000050000
> >> >                 0x0000000000057dc8 0x0000000000057dc8  R E    0x1000
> >> >  LOAD           0x00000000000585b8 0x00000000000a95b8 0x00000000000a95b8
> >> >                 0x0000000000004ee0 0x00000000000064b0  RW     0x1000
> >> >  NOTE           0x0000000000000158 0x0000000000050158 0x0000000000050158
> >> >                 0x0000000000000044 0x0000000000000044  R      0x4
> >> >
> >> > So now with the fix for statically compiled applications we can see
> >> > its offsetted and entry point is 0x504e4 and load is at
> >> > 0x0000000000050000. So with this we are for sure the MMU will always
> >> > trigger a page fault.
> >>
> >> Well that’s just a blatant violation of the spec.
> >
> > Right - this isn't a platform quirk; it's a violation of the ISA.
> > Unless this behavior is engaged by a custom mode bit, that is, in
> > which case, it should just be disabled by default and none of this
> > would be a problem.  If the behavior is unconditionally enabled, this
> > should be treated as an erratum.
>
> I don't see this as any different than the T-Head page table attributes,
> where it was made abundantly clear that vendors self-certify their
> implementations and thus anything goes.  It doesn't really matter if we
> called it an errata, bug, surprise feature, vendor extension, whatever;
> it's RISC-V so we've got to live with it.
>
phew that's a relief.

> >> What happens if a
> >> malicious (or buggy) userspace binary goes and accesses unmapped memory
> >> in that range? Does it actually get access to this ILM/DLM? If so
> >> that’s a gaping side channel.
>
> IIUC that's the case, but that's just from reading this post.
>
> There's nothing we can do to prevent vendors from building hardware with
> side channels.  In this case I suppose we could swap out the memory
> region on context switches, from a quick look at the datasheet these are
> single-core so that'd be safe.  My guess is it'd be too slow for users
> to want that on by default, but I don't have any use case for this HW so
> I'm not sure.
>
Enabling the local memory (ILM/DLM) on the core is a specification
option and is enabled on RZ/Five SoC. I'm checking with Andes if this
can be forcefully disabled.
The only use case would be to speed up things for some slower block
(but said that its user application specific)

Cheers,
Prabhakar

Re: [QUERY]: Adding support for a platform

[Palmer Dabbelt]

On Tue, 29 Nov 2022 13:40:53 PST (-0800), prabhakar.csengg@gmail.com wrote:
> Hi Palmer,
>
> On Tue, Nov 29, 2022 at 8:57 PM Palmer Dabbelt <palmer@dabbelt.com> wrote:
>>
>> On Tue, 29 Nov 2022 12:21:31 PST (-0800), Andrew Waterman wrote:
>> > On Tue, Nov 29, 2022 at 12:12 PM Jessica Clarke <jrtc27@jrtc27.com> wrote:
>> >>
>> >> On 29 Nov 2022, at 19:28, Lad, Prabhakar <prabhakar.csengg@gmail.com> wrote:
>> >> >
>> >> > Hi Palmer,
>> >> >
>> >> > Oops I should have included linux-riscv to the CC list here (doing it now)
>> >> >
>> >> > On Tue, Nov 29, 2022 at 7:20 PM Palmer Dabbelt <palmer@dabbelt.com> wrote:
>> >> >>
>> >> >> On Tue, 29 Nov 2022 11:16:29 PST (-0800), binutils@sourceware.org wrote:
>> >> >>> Hi All,
>> >> >>>
>> >> >>> If this is not the right place to ask this question please let me know.
>> >> >>>
>> >> >>> So we have a RISCV platform (Renesas RZ/Five) for which we need to
>> >> >>> adjust the TEXT_START_ADDR. So below are my queries:
>> >> >>> * What is the procedure for upstreaming?
>> >> >>> * Are patches accepted for individual platforms (for adjusting TEXT_START_ADDR)?
>> >> >>
>> >> >> Is this still related to that bug with the static binaries you'd
>> >> >> mentioned before?  I think we'd really need to figure out the root cause
>> >> >> before we can make a reasoned decision here, my guess would be that
>> >> >> changing TEXT_START_ADDR just works around the real bug.
>> >> >>
>> >> > Below is the root cause and the solution:
>> >> >
>> >> > TEXT_START_ADDR is the start of the text segment of an application.
>> >> > This is being set to 0x10000 for RISCV platforms.
>> >> >
>> >> > So when an application is compiled with the static flag the load would
>> >> > start from 0x10000 - xyz (depending on size of the application)
>> >> >
>> >> > Entry point 0x101c0
>> >> > There are 5 program headers, starting at offset 64Program Headers:
>> >> >  Type           Offset             VirtAddr           PhysAddr
>> >> >                 FileSiz            MemSiz              Flags  Align
>> >> >  LOAD           0x0000000000000000 0x0000000000010000 0x0000000000010000
>> >> >                 0x0000000000059b48 0x0000000000059b48  R E    0x1000
>> >> >  LOAD           0x0000000000059b60 0x000000000006ab60 0x000000000006ab60
>> >> >                 0x0000000000001f68 0x0000000000003528  RW     0x1000
>> >> >
>> >> > So for the above application which is compiled statically we can see
>> >> > the entry point is 0x101c0 and load 0x0000000000010000.
>> >> >
>> >> > Andes AX45MP cores have local memory ILM and DLM that are mapped in
>> >> > the region H’0_0003_0000 - H’0_0004_FFFF on the RZ/Five SoC. When the
>> >> > virtual address falls in this range the MMU doesn't trigger a page
>> >> > fault and assumes the virtual address as physical address and hence
>> >> > the application fails to run (panics somewhere).
>> >> >
>> >> > So to avoid this issue we set the TEXT_START_ADDR to 0x50000 so that
>> >> > the virtual address of any statically compiled application does not
>> >> > fall in the range of H’0_0003_0000 - H’0_0004_FFFF.
>> >> >
>> >> > Elf file type is EXEC (Executable file)
>> >> > Entry point 0x504e4
>> >> > There are 5 program headers, starting at offset 64
>> >> >
>> >> > Program Headers:
>> >> >  Type           Offset             VirtAddr           PhysAddr
>> >> >                 FileSiz            MemSiz              Flags  Align
>> >> >  LOAD           0x0000000000000000 0x0000000000050000 0x0000000000050000
>> >> >                 0x0000000000057dc8 0x0000000000057dc8  R E    0x1000
>> >> >  LOAD           0x00000000000585b8 0x00000000000a95b8 0x00000000000a95b8
>> >> >                 0x0000000000004ee0 0x00000000000064b0  RW     0x1000
>> >> >  NOTE           0x0000000000000158 0x0000000000050158 0x0000000000050158
>> >> >                 0x0000000000000044 0x0000000000000044  R      0x4
>> >> >
>> >> > So now with the fix for statically compiled applications we can see
>> >> > its offsetted and entry point is 0x504e4 and load is at
>> >> > 0x0000000000050000. So with this we are for sure the MMU will always
>> >> > trigger a page fault.
>> >>
>> >> Well that’s just a blatant violation of the spec.
>> >
>> > Right - this isn't a platform quirk; it's a violation of the ISA.
>> > Unless this behavior is engaged by a custom mode bit, that is, in
>> > which case, it should just be disabled by default and none of this
>> > would be a problem.  If the behavior is unconditionally enabled, this
>> > should be treated as an erratum.
>>
>> I don't see this as any different than the T-Head page table attributes,
>> where it was made abundantly clear that vendors self-certify their
>> implementations and thus anything goes.  It doesn't really matter if we
>> called it an errata, bug, surprise feature, vendor extension, whatever;
>> it's RISC-V so we've got to live with it.
>>
> phew that's a relief.
>
>> >> What happens if a
>> >> malicious (or buggy) userspace binary goes and accesses unmapped memory
>> >> in that range? Does it actually get access to this ILM/DLM? If so
>> >> that’s a gaping side channel.
>>
>> IIUC that's the case, but that's just from reading this post.
>>
>> There's nothing we can do to prevent vendors from building hardware with
>> side channels.  In this case I suppose we could swap out the memory
>> region on context switches, from a quick look at the datasheet these are
>> single-core so that'd be safe.  My guess is it'd be too slow for users
>> to want that on by default, but I don't have any use case for this HW so
>> I'm not sure.
>>
> Enabling the local memory (ILM/DLM) on the core is a specification
> option and is enabled on RZ/Five SoC. I'm checking with Andes if this
> can be forcefully disabled.
> The only use case would be to speed up things for some slower block
> (but said that its user application specific)

If it's possible to disable this somehow that'd be great, but after 
writing up the "maybe we can context switch this" bit I think we could 
probably get away with some sort of compatibility mode here.  
Essentially: Don't allocate out of this region by default, but if 
userspace explicitly maps this region (as happens with static binaries) 
then allow it, but track those allocations and begin swapping it on 
context switches.

That might be a lot of work and would definitely be slow, but as long as 
other VAs can map to these PAs then I think we'd be pretty much safe -- 
we'd lose read-only and execute-only permission tracking, but existing 
binaries would still run.  That would allow this TEXT_START_ADDR change 
to be just a performance thing, and we might want that as a tunable for 
distros anyway (for huge page alignment, for example).

A lot of that would depend on exactly how the hardware treats these 
special VAs, though.  Probably best if you just dig through the docs, 
see how this all works, and then propose some rough patches?

Re: [QUERY]: Adding support for a platform

[Lad, Prabhakar]

Hi Palmer,

Sorry for the late reply.

On Tue, Nov 29, 2022 at 9:54 PM Palmer Dabbelt <palmer@dabbelt.com> wrote:
>
> On Tue, 29 Nov 2022 13:40:53 PST (-0800), prabhakar.csengg@gmail.com wrote:
> > Hi Palmer,
> >
> > On Tue, Nov 29, 2022 at 8:57 PM Palmer Dabbelt <palmer@dabbelt.com> wrote:
> >>
> >> On Tue, 29 Nov 2022 12:21:31 PST (-0800), Andrew Waterman wrote:
> >> > On Tue, Nov 29, 2022 at 12:12 PM Jessica Clarke <jrtc27@jrtc27.com> wrote:
> >> >>
> >> >> On 29 Nov 2022, at 19:28, Lad, Prabhakar <prabhakar.csengg@gmail.com> wrote:
> >> >> >
> >> >> > Hi Palmer,
> >> >> >
> >> >> > Oops I should have included linux-riscv to the CC list here (doing it now)
> >> >> >
> >> >> > On Tue, Nov 29, 2022 at 7:20 PM Palmer Dabbelt <palmer@dabbelt.com> wrote:
> >> >> >>
> >> >> >> On Tue, 29 Nov 2022 11:16:29 PST (-0800), binutils@sourceware.org wrote:
> >> >> >>> Hi All,
> >> >> >>>
> >> >> >>> If this is not the right place to ask this question please let me know.
> >> >> >>>
> >> >> >>> So we have a RISCV platform (Renesas RZ/Five) for which we need to
> >> >> >>> adjust the TEXT_START_ADDR. So below are my queries:
> >> >> >>> * What is the procedure for upstreaming?
> >> >> >>> * Are patches accepted for individual platforms (for adjusting TEXT_START_ADDR)?
<snip>
> > Enabling the local memory (ILM/DLM) on the core is a specification
> > option and is enabled on RZ/Five SoC. I'm checking with Andes if this
> > can be forcefully disabled.
> > The only use case would be to speed up things for some slower block
> > (but said that its user application specific)
>
> If it's possible to disable this somehow that'd be great, but after
> writing up the "maybe we can context switch this" bit I think we could
> probably get away with some sort of compatibility mode here.
> Essentially: Don't allocate out of this region by default, but if
> userspace explicitly maps this region (as happens with static binaries)
> then allow it, but track those allocations and begin swapping it on
> context switches.
>
Do you think even with TEXT_START_ADDR change user space can map into LM?

> That might be a lot of work and would definitely be slow, but as long as
> other VAs can map to these PAs then I think we'd be pretty much safe --
> we'd lose read-only and execute-only permission tracking, but existing
> binaries would still run.  That would allow this TEXT_START_ADDR change
> to be just a performance thing, and we might want that as a tunable for
> distros anyway (for huge page alignment, for example).
>
Okay, I'll create a patch for the TEXT_START_ADDR change.

> A lot of that would depend on exactly how the hardware treats these
> special VAs, though.  Probably best if you just dig through the docs,
> see how this all works, and then propose some rough patches?

I checked with Andes and it looks like we cannot disable the local
memory by registers. As per Andes [0] (page 30) if the VA is Local
memory (LM) it straight aways treats it as PA and it never searches
through TLB (note there is a arrow missing in the diagram VA->PA if
LM)

[0] http://www.andestech.com/wp-content/uploads/AX45MP-1C-Rev.-5.0.0-Datasheet.pdf

Cheers,
Prabhakar