Help porting newlib to a new CPU architecture (sorta)

Help porting newlib to a new CPU architecture (sorta)

[ElijaxApps]

Hi, thank you a lot four your great responses, them really got me on some
direction.

>even DOS (iiuc) requires at least a 16-bit CPU.  not sure FreeDOS supports
>16-bit though, or if it too requires 32-bit.
>-mike

Well, Libre 8 has:
4 GP 8 Bit registers for math operations. 16 Bit integers are not a big
problem,  if:
  - I could use 2 cycles for batching the operation,
  - I could use registers 2 on 2 (and update ALU width).
Or even I could rewrite the microcode of my CPU to make it work
independently, as in
  - Sum lower 8 bits, push to stack,
  - Sum higher 8 bits, push to stack,
  - On return (int) pop twice.

In memory terms it has 24 bit address RAM, all of them addressable and
directionable in 3 cycle jumps. No segmentation, no paragraphs, just plain
16MB of RAM in a single component. That's why I am confident about using a
DOS-like OS.

>Consider the AVR toolchain, used in Arduino.  Isn't that an 8-bit
machine?  It certainly is small.
>paul

Thanks for the tip! I have checked, I think this is the source code:

https://github.com/esp8266/Arduino/tree/master/cores/esp8266

Could you point out what files I need to rewrite? Thanks in advance.
Correct me if that is not the code, and in case I finally decide this
approach I'll move to its mailing list.

>The full toolchain includes libc, libbfd, binutils, gcc, which would
>have to be bootstrapped by cross-compiling from a host system.

>There is an AVR target which may support 8b and avr-libc which should
>support the toolchain.

>There is also an Arduino toolchain based off the AVR toolchain which
>supports developing and loading sketches which run on pretty bare 8b MCUs.

>You may be better off targeting the latter approach for simplicity.
>A lot may depend on how close a match you can find between your target
>architecture and some existing architecture.

Yeah, I've been checking GCC repository and there is already an AVR target,
that I could use as a base to make Libre 8 target. Another candidate is z80
you mentioned, with the only caveat of having to implement 16b integer
operations on my CPU firmware (not hard at all).

I know I will have to cross-compile at least once libc for using it, then
crosscompiling gcc itself, and it should be done then the compilation of
the full toolchain in the Libre 8 CPU itself. But the latter part is for
the moment out of scope, as I have very limited resources on the CPU and no
disk nor filesystem, just plain RAM (I was thinking on defining programs
and files as memory offsets, and made them reside in memory, in a future,
or . . . designing a component in logisim suitable as a hard disk with its
filesystem and such...)

Or, compiling every monolithic program I want to run as STATIC, so it will
include dependant libraries binary code on the executable.

For this, I need the libc library to be embedded, and that is why I reached
newlib.

@Orlando: You're pointing the very right direction there, I'll be quiet on
the quotes.

- I got already an assembler, written in Java. Is more like a parser, it
just transcribes a bunch of implemented opcodes into its binary format for
Libre 8. But I want to use any preexisting toolchain, as gcc ones. In other
case I would end rewriting the full toolchain. However, your indication
about porting GAS is actually a thing I really considered before.

So as far as I understand now, it is not only porting newlib, *I have to
port also to port GAS (GNU Assembler) and GCC (GNU compiler)? Take in mind
I just want to CROSS-COMPILE binaries.*

- How its an ABI formally defined on code? How is defined AVR - ABI? Could
you point some sources? I googled without finding much about ABI
definitions - "as is".

- Architecture is Von-Neumann (Instructions and Data ar mixed in a single
memory compartiment). Decoder will find instructions and data in the same
input.

- I found this project (
https://docs.platformio.org/en/latest/platforms/creating_platform.html )
which allows you to define toolchain components and right, I would need
only to change gas to libre 8 compiler. BUT, I would need some kind of
workaround to make C compiler to output mine pseudo assy, which I no doubt
it is possible but would be crappy. - OR, I just have to port GAS to Libre
8, and use "Standard ASSY".

- Thanks a lot for suggesting other 8 bit DOS OS's.



*So, what would be the first step? - Defining ABI (bring examples of
existing 8 bit ABI's) for entire toolchain. LINK NEEDED - Porting GAS, and
use a custom GAS-libre8 in platformio project. Could anyone point out out
which files need to be edited exactly?*

Thank you all.
Elijax Apps.

Re: Help porting newlib to a new CPU architecture (sorta)

[Joel Sherrill]

On Tue, Jul 6, 2021, 4:10 PM ElijaxApps <elijaxapps@gmail.com> wrote:

> Hi, thank you a lot four your great responses, them really got me on some
> direction.
>
> >even DOS (iiuc) requires at least a 16-bit CPU.  not sure FreeDOS supports
> >16-bit though, or if it too requires 32-bit.
> >-mike
>
> Well, Libre 8 has:
> 4 GP 8 Bit registers for math operations. 16 Bit integers are not a big
> problem,  if:
>   - I could use 2 cycles for batching the operation,
>   - I could use registers 2 on 2 (and update ALU width).
> Or even I could rewrite the microcode of my CPU to make it work
> independently, as in
>   - Sum lower 8 bits, push to stack,
>   - Sum higher 8 bits, push to stack,
>   - On return (int) pop twice.
>
> In memory terms it has 24 bit address RAM, all of them addressable and
> directionable in 3 cycle jumps. No segmentation, no paragraphs, just plain
> 16MB of RAM in a single component. That's why I am confident about using a
> DOS-like OS.
>

I think you might get some ideas from the old m6809 port of gcc

http://vectrexc.malban.de/documentation/gcc-6809-documentation

That was an 8 bit CPU with only 64k memory space if I remember everything
correctly.

I think given an add with carry, gcc can be taught to use 8 bit operations
in sequence on larger types. I recall seeing this on the h8 or avr.


> >Consider the AVR toolchain, used in Arduino.  Isn't that an 8-bit
> machine?  It certainly is small.
> >paul
>
> Thanks for the tip! I have checked, I think this is the source code:
>
> https://github.com/esp8266/Arduino/tree/master/cores/esp8266
>
> Could you point out what files I need to rewrite? Thanks in advance.
> Correct me if that is not the code, and in case I finally decide this
> approach I'll move to its mailing list.
>

The gcc source would be in a directory like gcc/config/avr


> >The full toolchain includes libc, libbfd, binutils, gcc, which would
> >have to be bootstrapped by cross-compiling from a host system.
>
> >There is an AVR target which may support 8b and avr-libc which should
> >support the toolchain.
>
> >There is also an Arduino toolchain based off the AVR toolchain which
> >supports developing and loading sketches which run on pretty bare 8b MCUs.
>
> >You may be better off targeting the latter approach for simplicity.
> >A lot may depend on how close a match you can find between your target
> >architecture and some existing architecture.
>
> Yeah, I've been checking GCC repository and there is already an AVR target,
> that I could use as a base to make Libre 8 target. Another candidate is z80
> you mentioned, with the only caveat of having to implement 16b integer
> operations on my CPU firmware (not hard at all).
>
> I know I will have to cross-compile at least once libc for using it, then
> crosscompiling gcc itself, and it should be done then the compilation of
> the full toolchain in the Libre 8 CPU itself. But the latter part is for
> the moment out of scope, as I have very limited resources on the CPU and no
> disk nor filesystem, just plain RAM (I was thinking on defining programs
> and files as memory offsets, and made them reside in memory, in a future,
> or . . . designing a component in logisim suitable as a hard disk with its
> filesystem and such...)
>
> Or, compiling every monolithic program I want to run as STATIC, so it will
> include dependant libraries binary code on the executable.
>
> For this, I need the libc library to be embedded, and that is why I reached
> newlib.
>
> @Orlando: You're pointing the very right direction there, I'll be quiet on
> the quotes.
>
> - I got already an assembler, written in Java. Is more like a parser, it
> just transcribes a bunch of implemented opcodes into its binary format for
> Libre 8. But I want to use any preexisting toolchain, as gcc ones. In other
> case I would end rewriting the full toolchain. However, your indication
> about porting GAS is actually a thing I really considered before.
>
> So as far as I understand now, it is not only porting newlib, *I have to
> port also to port GAS (GNU Assembler) and GCC (GNU compiler)? Take in mind
> I just want to CROSS-COMPILE binaries.*
>
> - How its an ABI formally defined on code? How is defined AVR - ABI? Could
> you point some sources? I googled without finding much about ABI
> definitions - "as is".
>
> - Architecture is Von-Neumann (Instructions and Data ar mixed in a single
> memory compartiment). Decoder will find instructions and data in the same
> input.
>
> - I found this project (
> https://docs.platformio.org/en/latest/platforms/creating_platform.html )
> which allows you to define toolchain components and right, I would need
> only to change gas to libre 8 compiler. BUT, I would need some kind of
> workaround to make C compiler to output mine pseudo assy, which I no doubt
> it is possible but would be crappy. - OR, I just have to port GAS to Libre
> 8, and use "Standard ASSY".
>
> - Thanks a lot for suggesting other 8 bit DOS OS's.
>
>
>
> *So, what would be the first step? - Defining ABI (bring examples of
> existing 8 bit ABI's) for entire toolchain. LINK NEEDED - Porting GAS, and
> use a custom GAS-libre8 in platformio project. Could anyone point out out
> which files need to be edited exactly?*
>
> Thank you all.
> Elijax Apps.
>

Re: Help porting newlib to a new CPU architecture (sorta)

[Paul Koning]

> On Jul 6, 2021, at 6:00 PM, Joel Sherrill <joel@rtems.org> wrote:
> 
> I think you might get some ideas from the old m6809 port of gcc
> 
> http://vectrexc.malban.de/documentation/gcc-6809-documentation
> 
> That was an 8 bit CPU with only 64k memory space if I remember everything
> correctly.

pdp11 also has 64k of address space, and the current gcc still supports it.

> I think given an add with carry, gcc can be taught to use 8 bit operations
> in sequence on larger types. I recall seeing this on the h8 or avr.

pdp11 also, for 32 or 64 bit integers given 16 bit arithmetic.  You don't actually need add with carry, but having one makes the code more compact.  I think gcc core will do that flavor if the target code doesn't teach it the more efficient way (which is easy to do).

	paul

Re: Help porting newlib to a new CPU architecture (sorta)

[ElijaxApps]

Hi

*@Paul*,  I am looking at the code you maintain. I downloaded Eclipse for C
/ C ++ and I am currently cloning gcc repository.

My intention is to read carefully your code and rewrite the less possible.
Also I have looked up that internals gcc manual mentioned above, and it is
really all I need to read now. I am very thankful to you,
It is really recomfortating to find out the very right person to ask for
help. Really appreciate it.

I have not had time yet to check yet avr, h8, m6809 codes, but seems also
candidate alternatives, so,

I will note below the characteristics of my architecture, in depth, to help
us decide which one is closer to mine.


*4x8b GP Registers*- A, Accumulator, Argc
- B, Summand, Auxiliar in indirections for higher 8 bit addressing
- C, Auxiliar, Auxiliar used in indirections for mid 8 bit addressing,
String register (not connected to ALU but it is connected to other REGS via
bus)
- D, Auxiliar, Auxiliar used in indirections for lower 8 bit addressing.


*Memory Address Registers*Those point to RAM for in/redirections and look
ups into RAM.
- M0 lower 8 bit
- M1 mid 8 bit
- M2 high 8 bit
* M2 + M1 + M0 conforms a full 24 bit address.


*Stack registers:*- SR0  stores M0 on push while calling a new branch. Pops
to M0 on return. Allows REG B to be pushed.-
- SR1  stores M1 on push while calling a new branch. Pops to M1 on return.
Allows REG C to be pushed.
- SR2 stores M1 on push while calling a new branch. Pops to M2 on return.
Allows REG D to be pushed.


*Output register*- Stores temporarily one char to be output. It will be
replaced by a text based GPU with 64K of memory to allow cursor
operations on the lcd and also speed up printing, with a separate
independant clock.

*Flags Register*
- Carry flag
- Zero flag
- Parity flag (not yet shown not used)
- Borrow flag (''  ''  '' '' '')

*So It's basically an automat, with stack, with 8 bit ALU, with 4 GP
registers, with 24bit addressable memory, and 24bit max length push/pop
item.*

1) Let's decide the appropiate basis first.
2) As a side part, I am able now I read the code, to better understand how
gcc relates to gas, and, now I am able to understand also how would it be
to just, "make gcc to spit my pseudoassy" and just use my assembler.
Knowing this is possible, I will apart the idea for the moment as plab - B
to the purpose of compiling existing C / C ++ 8 bit software into Libre 8,
with at least GCC. (This is good at least, and very very much easier :)) ).
Now I am interested into use the full chain, including gas, with the
"internals" manual provided. If everything fails or I got overwhelmed, I
got plan B.

Thank you all,
Elijax Apps.

El mié, 7 jul 2021 a las 1:50, Paul Koning (<paulkoning@comcast.net>)
escribió:

>
>
> > On Jul 6, 2021, at 6:00 PM, Joel Sherrill <joel@rtems.org> wrote:
> >
> > I think you might get some ideas from the old m6809 port of gcc
> >
> > http://vectrexc.malban.de/documentation/gcc-6809-documentation
> >
> > That was an 8 bit CPU with only 64k memory space if I remember everything
> > correctly.
>
> pdp11 also has 64k of address space, and the current gcc still supports it.
>
> > I think given an add with carry, gcc can be taught to use 8 bit
> operations
> > in sequence on larger types. I recall seeing this on the h8 or avr.
>
> pdp11 also, for 32 or 64 bit integers given 16 bit arithmetic.  You don't
> actually need add with carry, but having one makes the code more compact.
> I think gcc core will do that flavor if the target code doesn't teach it
> the more efficient way (which is easy to do).
>
>         paul
>
>

Re: Help porting newlib to a new CPU architecture (sorta)

[Grant Edwards]

On 2021-07-06, Joel Sherrill <joel@rtems.org> wrote:

> I think you might get some ideas from the old m6809 port of gcc
>
> http://vectrexc.malban.de/documentation/gcc-6809-documentation
>
> That was an 8 bit CPU with only 64k memory space if I remember everything
> correctly.

The 6809 el al could treat pairs of registers as 16-bit values for
some operations (similar to 8080/Z80). All of the C compilers I've
used for 68xx architecture relied heavily on those 16-bit
register-pair operations.

--
Grant

Help porting newlib to a new CPU architecture (sorta)

[ElijaxApps]

Hi all,

I designed a Logisim schematic of a full system, able to run programs in
the simulation, as shown in this video:

https://www.youtube.com/watch?v=UP6tO8x5I5A

Is based on a SAP-1 (Simplest as Possible) basis, containing 4 GP registers
(8 bit), stack for function operations (up to 256 depth levels of
recursivity), 24bit plain RAM component, and a simple ALU able to echo
strings and perform not floating point math.

AFAIK, newlib is suitable for embedded devices, and I want to create the
full toolchain for C/C++ language at least.

At first I tried porting it by my own using this guide:

https://www.embecosm.com/appnotes/ean9/ean9-howto-newlib-1.0.html#sec_crt0

And I am stuck here:

   1.

   Implement the Board Support Package(s) for the target (Chapter 5
   <https://www.embecosm.com/appnotes/ean9/ean9-howto-newlib-1.0.html#chap_libgloss>
   ).
   -

      Implement the C Runtime start up, crt0.o for each BSP (Section 5.2
      <https://www.embecosm.com/appnotes/ean9/ean9-howto-newlib-1.0.html#sec_crt0>
      ).
      -

      Implement the environment global variable and 18 system call
      functions for each BSP following the convention namespace and
      reentrancy conventions specified in newlib/configure.host (Section 5.3
      <https://www.embecosm.com/appnotes/ean9/ean9-howto-newlib-1.0.html#sec_syscalls>
       and Section 5.4
      <https://www.embecosm.com/appnotes/ean9/ean9-howto-newlib-1.0.html#sec_reentrant_syscalls>
      ).
      -

      Create libgloss/*target*/Makefile.in and libgloss/*target*/
      configure.ac, based on the versions in the libgloss/libnosys directory
      and run aclocal and autoconf in libgloss/*target* (Section 5.5
      <https://www.embecosm.com/appnotes/ean9/ean9-howto-newlib-1.0.html#sec_bsp_config_make>
      ).

I have some questions regarding those points:

1) ATM my system is programmable, using a kind of own pseudo-assembler code
I created for that purpose, alongside with a Java compiler which translates
it to text hex file, then I copy its contents to the simulation's RAM. But
it's not "normal", "standard" ASSY, just an approximation made by not an
assy guy... (never coded any in actual ASSY).

So:
 - How do I write CRT code? In standard ASSY? In my own assy, and somewhere
I tell which binary format represents each mnemonic? In case of the first,
how will  the toolchain know the opcodes and binary format of the
instructions and data? In case of the second, could you please clarify the
paths of those files in the docs? :)

The full source to Libre 8 CPU is here. Some recent changes are not yet
pushed but the commited and de documentation provided in github should be
enough to make you an idea of what I tell by pseudo-assy.

https://github.com/ElijaxApps/Libre-8

I want to push it further, and perhaps port some existing software to it as
FreeDOS ... , as far as I can get. All ideas are welcome.

If you have to make questions about the architecture don't hesitate and do
it. If I have to modify os made changes to fit any requirement it won't be
a problem, even now I've reached this far.

I hope this is possible. Thank you,
Elijax Apps.

Re: Help porting newlib to a new CPU architecture (sorta)

[Mike Frysinger]

On 06 Jul 2021 02:49, ElijaxApps wrote:
> I designed a Logisim schematic of a full system, able to run programs in
> the simulation, as shown in this video:
> 
> https://www.youtube.com/watch?v=UP6tO8x5I5A
> 
> Is based on a SAP-1 (Simplest as Possible) basis, containing 4 GP registers
> (8 bit), stack for function operations (up to 256 depth levels of
> recursivity), 24bit plain RAM component, and a simple ALU able to echo
> strings and perform not floating point math.
> 
> AFAIK, newlib is suitable for embedded devices, and I want to create the
> full toolchain for C/C++ language at least.

i don't know that these would work that well (or at all) on an 8-bit CPU.
you'd really want a 32-bit CPU nowadays as a minimum if you want to support
modern software.

even DOS (iiuc) requires at least a 16-bit CPU.  not sure FreeDOS supports
16-bit though, or if it too requires 32-bit.
-mike

Re: Help porting newlib to a new CPU architecture (sorta)

[Paul Koning]

> On Jul 6, 2021, at 12:35 AM, Mike Frysinger <vapier@gentoo.org> wrote:
> 
> On 06 Jul 2021 02:49, ElijaxApps wrote:
>> I designed a Logisim schematic of a full system, able to run programs in
>> the simulation, as shown in this video:
>> 
>> https://www.youtube.com/watch?v=UP6tO8x5I5A
>> 
>> Is based on a SAP-1 (Simplest as Possible) basis, containing 4 GP registers
>> (8 bit), stack for function operations (up to 256 depth levels of
>> recursivity), 24bit plain RAM component, and a simple ALU able to echo
>> strings and perform not floating point math.
>> 
>> AFAIK, newlib is suitable for embedded devices, and I want to create the
>> full toolchain for C/C++ language at least.
> 
> i don't know that these would work that well (or at all) on an 8-bit CPU.
> you'd really want a 32-bit CPU nowadays as a minimum if you want to support
> modern software.

Consider the AVR toolchain, used in Arduino.  Isn't that an 8-bit machine?  It certainly is small.  

	paul

Re: Help porting newlib to a new CPU architecture (sorta)

[ElijaxApps]

Hi all,

I started reading GCC Internals Book, chapter 16 & 17.

I am deciding how exactly define insn statements at this very right moment.
I am using pdp11 as a template. However, there are lots of things I don't
really need, in this config spec.

As far as I have understood, I can follow two paths:

a) Define instructions as my pseudo assy, and use my own "assembler".
b) Define instructions as Linux ASM and use GAS, GNU Assembler.

At first I decided to apart A and use it as a Fallback if B fails. But I
really think I am interested in both approaches. You see, I haven't coded
any in real ASM and I just mimic'ed what it looks like.
But I am a bit lost here, I feel like A is easier but it is not the
"correct" way, but it can be done perhaps in a week or two... And let B for
the long term, as I got practice and training with A.


See the full instruction set for the moment (at the end of the document).
What I would need for A is help writing the insn definition (for A this
moment)
I'll try to define the instructions BNF so you can help me to write insn
definitions:

*Offset*: [ 000000-ffffff ] [h] | <variable_name>
*1) LDx *(Load REG from the BUS):   [ LDA | LDB | LDC | LDD ]
offset_to_load
*2) STx *(Store REG in RAM):   [ STA | STB | STC | STD ]
offset_where_to_store
*3) ADD,SUB*: [ ADD | SUB ] offset (to whatever already there is in A REG)
*4) Jumps*: Always redirect to another direction in RAM, so they always are
followed by an offset.
*5) Branches*: Always redirect to another direction in RAM, this time a
function beginning offset. They return a 24 bit address, and continue
execution below where Branch was called.
*6)* *DEC *(Read char and store in A REG) *Is all hardwired, is just the
mnemonic - DEC - (cannot use another REG)
*7) OUTx *(Output REG on LCD Screen)   [ OUTA | OUTB | OUTC | OUTD ]
*8) NOP, HLT *are just the mnemonic.

MOV is not fully supported yet...
"Spawning" a value into any register is not supported yet. You have to
previously store the value you'll load into RAM. I have to encode ASCII
table in the firmware, yet I not had time.

So I need basically 2 TWO isns:


*[ LDx | STx | Bx | Jx | ADD | SUB ] <offset> (24 bit hexadecimal, or
variable name)[ DEC | OUTx | HLT | NOP ] - as is, just the mnemonic...*


So that way I could continue the reading and go on.

Thank You,
Elijax Apps.

Code tokenMeaningExamples
;; This is a comment. line will be ignored ;;This is a comment
LDA Basic load A register. Always followed by a variable name or an memory
address (up to 24 bit) LDA one
LDA ff0010h
LDB Same as LDA with B register
LDC Same as LDA with C register
LDD Same as LDA with D register
STA Store contents of register A into the following address memory (also up
to 24 bit) STA one
STA ff0010h
STB Same as STA with B register
STC Same as STA with C register
STD Same as STA with D register
OUTA Echoes the value of register A to LCD Screen LDA one
OUTA (prints value of variable "one")
OUTB Same as OUTA with B register
OUTC Same as OUTA with C register
OUTD Same as OUTA with D register
DEC Reads a byte from keyboard and stores in A register DEC
STA readChar
ADD Adds the value of the following address or variable to the contents of
register A DEC
ADD one
SUB Substracts the value of the following address or variable to the
contents of register A DEC
ADD one
JZ Jumps to the specified address or to specified address contained within
a variable if Zero Flag == 0 JZ addressToJumpTo, JZ 0000ffh
JNZ Jumps to the specified address or to specified address contained within
a variable if Zero Flag != 0 JNZ addressToJumpTo, JNZ 0000ffh
JC Jumps to the specified address or to specified address contained within
a variable if Carry Flag == 0 JC addressToJumpTo, JC 0000ffh
JNC Jumps to the specified address or to specified address contained within
a variable if Carry Flag != 0 JNC addressToJumpTo, JNC 0000ffh
J Jumps always J addressToJumpTo
BZ Creates a branch (calls a function) if Zero Flag == 0 BZ offsetOfFunction
BNZ Creates a branch (calls a function) if Zero Flag != 0 BNZ
offsetOfFunction
BC Creates a branch (calls a function) Carry Flag == 0 BC offsetOfFunction
BNC Creates a branch (calls a function) if Carry Flag != 0 BNC
offsetOfFunction
B Creates a branch (calls a function) always it is found B offsetOfFunction
BX Returns a value contained in the following address BX returnValueAddress
NOP No operation. Just reads the next operation NOP
HLT Halts the computer HLT
MOV Limited, not documented support for some MOV operations. SEE THE
CODE!!! MOV
A, C

El mar, 6 jul 2021 a las 15:05, Paul Koning (<paulkoning@comcast.net>)
escribió:

>
>
> > On Jul 6, 2021, at 12:35 AM, Mike Frysinger <vapier@gentoo.org> wrote:
> >
> > On 06 Jul 2021 02:49, ElijaxApps wrote:
> >> I designed a Logisim schematic of a full system, able to run programs in
> >> the simulation, as shown in this video:
> >>
> >> https://www.youtube.com/watch?v=UP6tO8x5I5A
> >>
> >> Is based on a SAP-1 (Simplest as Possible) basis, containing 4 GP
> registers
> >> (8 bit), stack for function operations (up to 256 depth levels of
> >> recursivity), 24bit plain RAM component, and a simple ALU able to echo
> >> strings and perform not floating point math.
> >>
> >> AFAIK, newlib is suitable for embedded devices, and I want to create the
> >> full toolchain for C/C++ language at least.
> >
> > i don't know that these would work that well (or at all) on an 8-bit CPU.
> > you'd really want a 32-bit CPU nowadays as a minimum if you want to
> support
> > modern software.
>
> Consider the AVR toolchain, used in Arduino.  Isn't that an 8-bit
> machine?  It certainly is small.
>
>         paul
>
>
>

Re: Help porting newlib to a new CPU architecture (sorta)

[Orlando Arias]

[-- Attachment #1.1: Type: text/plain, Size: 2287 bytes --]

Greetings,

On 7/7/21 4:32 PM, ElijaxApps wrote:
> Hi all,
> 
> I started reading GCC Internals Book, chapter 16 & 17.
> 
> I am deciding how exactly define insn statements at this very right moment.
> I am using pdp11 as a template. However, there are lots of things I don't
> really need, in this config spec.
> 
> As far as I have understood, I can follow two paths:
> 
> a) Define instructions as my pseudo assy, and use my own "assembler".
> b) Define instructions as Linux ASM and use GAS, GNU Assembler.
> 

Please allow me to contribute to the discussion at hand rather than
derail it any further.

Nikolai Kim has a very nice series of videos in YouTube describing how
he added a back-end to gcc for his CPU [1]. Notice that the instructions
he is adding are actually pseudoassembly (as far as I can tell anyway).
From the perspective of gcc [technically cc1 and cc1plus], the code that
gets emitted by the back-end of the compiler does not matter. The
compiler's job stops there. It's the assembler's job to actually
generate object files. If you are in Linux (or any other platform that
allows for it), you can

strace -f -o trace_gcc_driver gcc -o test.elf my_code.c

and see how the driver program calls different portions of the toolchain
to actually build an executable. You will see calls to the fork() and
execve() family of functions calling things like the compiler proper,
assembler, and linker.

Anyway, Nikolai Kim also has videos on adding a backend to binutils,
which is also worth checking out if you are interested. I believe his
code is online somewhere. Keep in mind that his code works for the
versions of gcc [and binutils] that he is modifying. These projects are
moving targets and internals are bound to change across releases.

As for your question about ABIs, you can find a portion of the ABI for
avr in [2]. It is missing things like how to deal with unwind sections,
debug infos, and the likes. Since this is your own CPU, you define the
ABI as you wish it to be. The one for Arm is found in [3].

Cheers,
Orlando.

[1] https://www.youtube.com/playlist?list=PLqKduIIK_NvI_BPpsMKEQ7ZsURitDZShr

[2] https://gcc.gnu.org/wiki/avr-gcc

[3] https://developer.arm.com/documentation/ihi0036/latest/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Help porting newlib to a new CPU architecture (sorta)

[Brian Inglis]

On 2021-07-05 22:35, Mike Frysinger wrote:
> On 06 Jul 2021 02:49, ElijaxApps wrote:
>> I designed a Logisim schematic of a full system, able to run programs in
>> the simulation, as shown in this video:
>>
>> https://www.youtube.com/watch?v=UP6tO8x5I5A
>>
>> Is based on a SAP-1 (Simplest as Possible) basis, containing 4 GP registers
>> (8 bit), stack for function operations (up to 256 depth levels of
>> recursivity), 24bit plain RAM component, and a simple ALU able to echo
>> strings and perform not floating point math.
>>
>> AFAIK, newlib is suitable for embedded devices, and I want to create the
>> full toolchain for C/C++ language at least.

The full toolchain includes libc, libbfd, binutils, gcc, which would 
have to be bootstrapped by cross-compiling from a host system.

There is an AVR target which may support 8b and avr-libc which should 
support the toolchain.

There is also an Arduino toolchain based off the AVR toolchain which 
supports developing and loading sketches which run on pretty bare 8b MCUs.

You may be better off targeting the latter approach for simplicity.
A lot may depend on how close a match you can find between your target 
architecture and some existing architecture.

> i don't know that these would work that well (or at all) on an 8-bit CPU.
> you'd really want a 32-bit CPU nowadays as a minimum if you want to support
> modern software.
> 
> even DOS (iiuc) requires at least a 16-bit CPU.  not sure FreeDOS supports
> 16-bit though, or if it too requires 32-bit.

The only current release of DOS is FreeDOS, which supports the 1MB 16b 
8088 address space as 64K segments of 16B paragraphs or 16 segments of 
64KB, but requires 16b segment CS DS ES SS and general AX BX CX DX SI DI 
BP SP registers. FreeDOS supports DOS extenders and toolchains such as 
DJGPP to develop and run 32b programs on 32b CPUs.

Even the Small-C compiler in early Dr. Dobb's Magazine issues required 
an 8080/Z80 supporting 16b ints.

All the original compilers targeted 12-18b CPUs and required at least 
4KW (~8KB), and C used a 16b CPU and supported up to 64KB memory.
Many early compilers for small machines generated threaded code (q.v.) 
which saves space when each elementary operation requires multiple 
instructions.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: Help porting newlib to a new CPU architecture (sorta)

[Orlando Arias]

[-- Attachment #1.1: Type: text/plain, Size: 5051 bytes --]

Greetings,

Appending to this discussion:

> The full toolchain includes libc, libbfd, binutils, gcc, which would
> have to be bootstrapped by cross-compiling from a host system.
> 
> There is an AVR target which may support 8b and avr-libc which should
> support the toolchain.
> 
> There is also an Arduino toolchain based off the AVR toolchain which
> supports developing and loading sketches which run on pretty bare 8b MCUs.
> 
> You may be better off targeting the latter approach for simplicity.
> A lot may depend on how close a match you can find between your target
> architecture and some existing architecture.
> 

The Arduino toolchain is the same as the AVR toolchain. It is bundled
with the Wiring environment, which provides a `Board Support Package'
and hardware abstraction layer to the underlying microcontroller. The
main() function, for example, is provided by the runtime.

Now, specific to your case:

> I designed a Logisim schematic of a full system, able to run programs in
> the simulation, as shown in this video:
> 
> https://www.youtube.com/watch?v=UP6tO8x5I5A
> 
> Is based on a SAP-1 (Simplest as Possible) basis, containing 4 GP registers
> (8 bit), stack for function operations (up to 256 depth levels of
> recursivity), 24bit plain RAM component, and a simple ALU able to echo
> strings and perform not floating point math.
> 
> AFAIK, newlib is suitable for embedded devices, and I want to create the
> full toolchain for C/C++ language at least.
> 

You are getting ahead of yourself here. As others have mentioned, before
you go into the C library, you need to have a working assembler [and
possibly a bootstrapping compiler, depending on how far you want to go].
A linker is not required at this point if you're just looking to
bootstrap your toolchain. Usually, the GNU Assembler (gas) from binutils
is used for this scenario. Porting gas to a new architecture requires
adding your new target to libopcodes (assembly and disassembly) and to
libbfd (object file generation).

However, I would argue that before even going there you need to define
an application binary interface (ABI). Your CPU has 8 bit registers. The
C language requires the int type to be at least 16 bits wide. You need
to define how you plan to handle this scenario. AVR, for example, uses
register pairs to handle integers. You also need to define your calling
conventions. That is, how are arguments being passed to functions, how
returns are handled, any stack alignment constraints... As examples, in
IA-32, arguments are normally passed in the stack; in AMD64 the first 6
integer arguments are passed in registers; RISC-V uses the a* registers
to pass arguments to functions. This kind of thing is specially
important when you are mixing languages such as C and assembly.

Keep in mind that C assumes you are working on a von Neumann
architecture. This means that the language expects both program and data
to be in the same address space. If your CPU [which I can not really
check since I do not work on Logisim, I use Verilog/SystemVerilog/VHDL
myself] uses a Harvard or modified Harvard architecture, you need to
figure out a way for your C runtime to emulate a von Neumann machine.
AVR does this by ensuring that things that would go in .data, .bss, and
.rodata are copied over to the microcontroller's RAM before user code is
called [the toolchain provides ways to override this, but that is not
important at the moment]. They also extend the C language to allow for
data to be loaded from program memory under certain conditions [which is
also not important for the discussion at hand].

This brings me to the next point:
>  - How do I write CRT code? In standard ASSY? In my own assy, and somewhere
> I tell which binary format represents each mnemonic? In case of the first,
> how will  the toolchain know the opcodes and binary format of the
> instructions and data? In case of the second, could you please clarify the
> paths of those files in the docs? :)

You probably want to write your code in standard assembly. I looked over
at your Java `assembler' and you do seem to have instruction encodings
in there. The encodings seem to be rather irregular, but we are not here
to evaluate computer architecture stuff, so I digress. If you can get
your Java assembler to behave the way gas behaves, you may be able to
use it in place of gas. I do not know how scalable that would be though.

> I want to push it further, and perhaps port some existing software to it as
> FreeDOS ... , as far as I can get. All ideas are welcome.

I think FreeDOS may be too much at the moment. I would suggest looking
at FreeRTOS or RTEMS instead. These operating systems have been designed
with embedded CPUs in mind, and FreeRTOS has support for AVR, which is
an 8 bit CPU. Keep in mind that porting an OS is quite a daunting task,
and will require you to have at the very least a working assembler,
compiler, and linker.

Cheers,
Orlando.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Help porting newlib to a new CPU architecture (sorta)

[Brian Inglis]

On 2021-07-06 08:35, Orlando Arias wrote:
> Keep in mind that C assumes you are working on a von Neumann
> architecture. This means that the language expects both program and data
> to be in the same address space. If your CPU [which I can not really
> check since I do not work on Logisim, I use Verilog/SystemVerilog/VHDL
> myself] uses a Harvard or modified Harvard architecture, you need to
> figure out a way for your C runtime to emulate a von Neumann machine.
C has no assumptions or requirements for outdated von Neumann 
architectures, but it also may not assume it is not running on such 
architectures.
Early C and Unix took advantage of the Harvard architecture of larger 
PDP-11s which supported split I/D space with mapping registers to 
support double the program size (64KB each 16b I and D) as well as 
splitting kernel/supervisor/runtime/user spaces [and similar features on 
other systems], and shared text (the original meaning of the 't' "sticky 
text" perm on executables), which is why there is wording in the 
standards about unmodifiable constant values.

One of the big issues with systems around the time C was developed was 
the unreliability resulting from systems, runtimes, and programs using 
self-modifying code "tricks", and programs ability to change constant 
data values used in programs (PI did not always stay π).
That was one of the benefits touted for higher level system 
implementation languages such as Bliss, C, PL/S.
In the face of failures frequently caused by arbitrary code and data 
modifications, branches and jumps, worries about trivial issues such as 
delimited strings, unsized APIs, dynamic memory management were decades 
in the future!

Various implementations, ABIs, and systems require modifiable code and 
constant data, but it is *NOT* a C language assumption or requirement; 
in fact I would consider it to be a C anti-pattern, that has long 
limited the abilities of compilers and optimizers to do better jobs.
Copying sections tends to be due to limitations of CPUs, systems, 
runtimes, or compilers to DTRT to make programs more reliable. ;^>

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: Help porting newlib to a new CPU architecture (sorta)

[Orlando Arias]

[-- Attachment #1.1: Type: text/plain, Size: 6687 bytes --]

Greetings,

On 7/6/21 2:08 PM, Brian Inglis wrote:
> C has no assumptions or requirements for outdated von Neumann
> architectures, but it also may not assume it is not running on such
> architectures.
> Early C and Unix took advantage of the Harvard architecture of larger
> PDP-11s which supported split I/D space with mapping registers to
> support double the program size (64KB each 16b I and D) as well as
> splitting kernel/supervisor/runtime/user spaces [and similar features on
> other systems], and shared text (the original meaning of the 't' "sticky
> text" perm on executables), which is why there is wording in the
> standards about unmodifiable constant values.
> 

Right, went back and looked at the standard. There is no description of
what the abstract machine for the execution environment should be. I
guess my confusion came from the second paragraph in [1]. Harvard
architectures still have the thing that you have to define whether a
pointer refers to something in program space or data space, and standard
C has no way of signaling this. In AVR, for example, requires the use of
the LPM instruction to perform loads from program memory, and the use of
SPM to store data into program memory (there are limitations to the
latter). In contrast, accesses to data memory are performed using
LDS/STS and family. The compiler will normally issue LD*/ST*
instructions when dereferencing pointers. If program memory is desired,
then the PROGMEM macro needs to be used for variables that need to be in
program memory, and a series of macros/intrinsics need to be used to
actually load/store the data in that region [forcing the use of the
LPM/SPM instructions].

I/O space is also something that can be hard to handle in C. IA-32
requires the use of the IN/OUT instructions to talk to devices in the
I/O space. The compiler will not emit this stuff unless you use some
form of macro/intrinsic. AVR has an I/O space that is also memory
mapped, so the compiler can optimize assembly output and use the I/O
space when talking to some addresses, avoiding the use of load/store
instructions (which are larger and take longer to execute). The
optimization is only possible though if the address is known at compile
time, or that it can be replaced with a symbol address at link time. For
the latter, the linker needs to check that the instruction can be
properly `patched' with the address being accessed.

This is what I meant by the von Neumann requirement: all pointers
dereference to the same address space. In a hosted system, the OS may be
able to perform some tricks to mask away different address spaces
[trapping on a memory access to a specific address range and emulating
the access for the program, for example]. In a freestanding system, your
[statically linked] C runtime is responsible for initializing the data
section of the program somehow. Usually, this involves copying data from
a read-only location [storage medium, such as memory-mapped flash] to a
read/write location [such as SRAM]. If you need to cross different
address spaces when performing this copy operation, then standard C will
not help you. You need an extension to the language of some sort [or to
use assembly]. Yes, most modern machines have split I/D caches, and the
processor's implementation is effectively that of a Harvard machine.
Some CPUs have multiple ports which can be used to service different
memory access requests to a unified memory system. Cortex-M processors,
for example, have multiple AMBA AXI ports [PPB, D-bus, I-bus, and
S-bus], but the memory system is still unified into a single address
space making the overall machine follow the von Neumann paradigm. This
is not the case with AVR, where address 0 means different things when
talking about data space and program spaces.

One thing though, I was under the impression that the PDP-11 was a von
Neumann machine with its unified memory model and all. Unix on the
machine made use of memory management facilities provided by the
hardware to provide a virtual memory environment for executing processes
[this being more of a function of the OS rather than C itself]?

> One of the big issues with systems around the time C was developed was
> the unreliability resulting from systems, runtimes, and programs using
> self-modifying code "tricks", and programs ability to change constant
> data values used in programs (PI did not always stay π).
> That was one of the benefits touted for higher level system
> implementation languages such as Bliss, C, PL/S.
> In the face of failures frequently caused by arbitrary code and data
> modifications, branches and jumps, worries about trivial issues such as
> delimited strings, unsized APIs, dynamic memory management were decades
> in the future!
>> Various implementations, ABIs, and systems require modifiable code and
> constant data, but it is *NOT* a C language assumption or requirement;
> in fact I would consider it to be a C anti-pattern, that has long
> limited the abilities of compilers and optimizers to do better jobs.
> Copying sections tends to be due to limitations of CPUs, systems,
> runtimes, or compilers to DTRT to make programs more reliable. ;^>
> 

This latter part is really dependent on the system, and I would consider
it outside the scope of the C language proper and more towards the C
runtime environment, program loader, and dynamic linker [which the
standard makes no assumption of as you state]. For example, on systems
that support ASLR, binaries may need to be `patched' when loaded to fix
relocation information. I believe DLLs in Windows have some weird thing
like they have a preferred base address to be loaded, otherwise the DLL
needs to be `rewritten' in memory as it is loaded using information from
one of the PE sections. Not too sure how that works or if they still do
that. I have not looked at Windows internals in a long time. IA-32 code
has some limitations on that too, given that %eip is not
addressable/accessible without using some hacks [which require the use
of assembly: call a stub, load the value of %eip from the stack into a
general purpose register such as %eax from the stack, then return. The
value in %eax is now the same as the address of the instruction you
returned to]. AMD64 allows for %rip to be indexed, which allows for
position independent code to be generated a lot easier,  although some
relocation information may still be needed [again, not too familiar with
this aspects of the architecture/ABI].

Cheers,
Orlando.

[1] https://www.nongnu.org/avr-libc/user-manual/pgmspace.html


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Help porting newlib to a new CPU architecture (sorta)

[Hans-Bernhard Bröker]

Am 06.07.2021 um 21:04 schrieb Orlando Arias:

> Right, went back and looked at the standard. There is no description of
> what the abstract machine for the execution environment should be. I
> guess my confusion came from the second paragraph in [1]. Harvard
> architectures still have the thing that you have to define whether a
> pointer refers to something in program space or data space, and standard
> C has no way of signaling this. 

You're mixing thing up there.  Standard C has a perfectly fine 
distinction between program space and data space, including pointers 
thereto.  Function pointers and data pointers _are_ distinct.

What Standard C does lack is a standardized distinction between pointers 
into ROM data and RAM data.  const-qualified pointers may seem like they 
offer that, but ultimately they don't.

> This is what I meant by the von Neumann requirement: all pointers
> dereference to the same address space. 

That's stated broadly enough to be wrong.  The C virtual machine is, in 
fact, a Harvard architecture.  It assumes that const and non-const data 
live in the same address space, but that doesn't make it von-Neumann.

Re: Help porting newlib to a new CPU architecture (sorta)

[Orlando Arias]

[-- Attachment #1.1: Type: text/plain, Size: 3629 bytes --]

Greetings,

On 7/6/21 4:01 PM, Hans-Bernhard Bröker wrote:
> Am 06.07.2021 um 21:04 schrieb Orlando Arias:
> 
>> Right, went back and looked at the standard. There is no description of
>> what the abstract machine for the execution environment should be. I
>> guess my confusion came from the second paragraph in [1]. Harvard
>> architectures still have the thing that you have to define whether a
>> pointer refers to something in program space or data space, and standard
>> C has no way of signaling this. 
> 
> You're mixing thing up there.  Standard C has a perfectly fine
> distinction between program space and data space, including pointers
> thereto.  Function pointers and data pointers _are_ distinct.
> 
> What Standard C does lack is a standardized distinction between pointers
> into ROM data and RAM data.  const-qualified pointers may seem like they
> offer that, but ultimately they don't.

Possibly I am explaining myself incorrectly here, and likely I am mixing
terminology, yes. There is also a good likelyhood I am conflating things
as well. If that is the case, my apologies and please feel free to
correct my understanding/terminology. What I mean to say is the
following [through an example].

Consider the AVR architecture, where program and data spaces have
distinct address spaces. We have a pointer to a string literal that
resides in program memory. We wish to compare it to a string that
resides in data memory. We could use a [naive] comparison method, such
as strcpy().

const char* str PROGMEM = "hello";

const char* a = str;
const char* b = data_memory_location;

while(*a != '\0' && *a == *b) {
	a++; b++;
}
return *a - *b;

The problem with this code is that we are treating a as a pointer in
data memory. Declaring a to be PROGMEM does not help. We actually need
to rewrite the code to force the compiler to use the proper instruction:

char t;
while((t = pgm_read_byte(a)) != '\0' && t == *b)
	a++; b++;
}

return t - *b;

We use the pgm_read_byte() macro to issue the LPM instruction, instead
of a regular load instruction. In fact, avr-libc provides a collection
of functions [which can be identified by their suffix _P] for the
particular event where data resides in program memory. For example, we
have strcmp_P(), where the second argument refers to a pointer to data
in the program memory address space.

> 
>> This is what I meant by the von Neumann requirement: all pointers
>> dereference to the same address space. 
> 
> That's stated broadly enough to be wrong.  The C virtual machine is, in
> fact, a Harvard architecture.  It assumes that const and non-const data
> live in the same address space, but that doesn't make it von-Neumann.

Right, so herein lies a problem. A Harvard machine implies that program
and data are in different address spaces. Unless my understanding is
wrong, this means that there is one address bus for data, and one
address bus for instructions. Dereferencing a function pointer and
dereferencing a data pointer would result in dereferencing to different
address spaces. Now, I believe that doing something like (char*)fn_ptr
in C is either undefined behavior or implementation-defined behavior.
However, the implementations I have seen would treat this pointer as
something in data memory, rather than something in program memory.
Actually modifying what fn_ptr points to would require the use of an
extension to the language [which would be implied if the behavior was
indeed UB or implementation defined]. Please correct me on this one.

Cheers,
Orlando.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Help porting newlib to a new CPU architecture (sorta)

[Brian Inglis]

On 2021-07-06 14:46, Orlando Arias wrote:
> Greetings,
> 
> On 7/6/21 4:01 PM, Hans-Bernhard Bröker wrote:
>> Am 06.07.2021 um 21:04 schrieb Orlando Arias:
>>
>>> Right, went back and looked at the standard. There is no description of
>>> what the abstract machine for the execution environment should be. I
>>> guess my confusion came from the second paragraph in [1]. Harvard
>>> architectures still have the thing that you have to define whether a
>>> pointer refers to something in program space or data space, and standard
>>> C has no way of signaling this.
>>
>> You're mixing thing up there.  Standard C has a perfectly fine
>> distinction between program space and data space, including pointers
>> thereto.  Function pointers and data pointers _are_ distinct.
>>
>> What Standard C does lack is a standardized distinction between pointers
>> into ROM data and RAM data.  const-qualified pointers may seem like they
>> offer that, but ultimately they don't.
> 
> Possibly I am explaining myself incorrectly here, and likely I am mixing
> terminology, yes. There is also a good likelyhood I am conflating things
> as well. If that is the case, my apologies and please feel free to
> correct my understanding/terminology. What I mean to say is the
> following [through an example].
> 
> Consider the AVR architecture, where program and data spaces have
> distinct address spaces. We have a pointer to a string literal that
> resides in program memory. We wish to compare it to a string that
> resides in data memory. We could use a [naive] comparison method, such
> as strcpy().
> 
> const char* str PROGMEM = "hello";
> 
> const char* a = str;
> const char* b = data_memory_location;
> 
> while(*a != '\0' && *a == *b) {
> 	a++; b++;
> }
> return *a - *b;
> 
> The problem with this code is that we are treating a as a pointer in
> data memory. Declaring a to be PROGMEM does not help. We actually need
> to rewrite the code to force the compiler to use the proper instruction:
> 
> char t;
> while((t = pgm_read_byte(a)) != '\0' && t == *b)
> 	a++; b++;
> }
> 
> return t - *b;
> 
> We use the pgm_read_byte() macro to issue the LPM instruction, instead
> of a regular load instruction. In fact, avr-libc provides a collection
> of functions [which can be identified by their suffix _P] for the
> particular event where data resides in program memory. For example, we
> have strcmp_P(), where the second argument refers to a pointer to data
> in the program memory address space.
> 
>>
>>> This is what I meant by the von Neumann requirement: all pointers
>>> dereference to the same address space.
>>
>> That's stated broadly enough to be wrong.  The C virtual machine is, in
>> fact, a Harvard architecture.  It assumes that const and non-const data
>> live in the same address space, but that doesn't make it von-Neumann.
> 
> Right, so herein lies a problem. A Harvard machine implies that program
> and data are in different address spaces. Unless my understanding is
> wrong, this means that there is one address bus for data, and one
> address bus for instructions. Dereferencing a function pointer and
> dereferencing a data pointer would result in dereferencing to different
> address spaces. Now, I believe that doing something like (char*)fn_ptr
> in C is either undefined behavior or implementation-defined behavior.

Function and object pointer inter-conversions are UB - daemons fly out 
your nose! For example, when code and data model pointer sizes are 
different, pointers are physically incompatible.

> However, the implementations I have seen would treat this pointer as
> something in data memory, rather than something in program memory.
> Actually modifying what fn_ptr points to would require the use of an
> extension to the language [which would be implied if the behavior was
> indeed UB or implementation defined]. Please correct me on this one.

Works on von Neumann architecture implementations, or where mapping 
registers map the same address ranges for code and data, perhaps with 
different access modes.
Modifying what a function pointer object points to is fairly common in 
C, as long as when they are used, they are (cast to) a pointer to a 
function of the correct type; c.f. qsort pointer to comparison function 
in its last argument, and Unix system driver interfaces which are 
effectively arrays of function pointers.

If you look at e.g the PDP11 architecture, somewhat similar 6800 series 
models, or the like, it had a number of mainly orthogonal general 
register addressing modes, including PC relative, indirect PC relative, 
and either with autoinc-/decrement, so it could use many registers to 
access "program" memory, absolute "program" addresses, and move through 
that space like an IP for threaded code, or as a subroutine stack, or 
access RO data in the instruction space directly.
For example, to copy RO instruction space data to RAM, the move source 
register uses autoincrement PC relative addressing and the destination 
register uses autoincrement relative addressing from a RAM base address.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: Help porting newlib to a new CPU architecture (sorta)

[Orlando Arias]

[-- Attachment #1.1: Type: text/plain, Size: 2794 bytes --]

Greetings,

On 7/7/21 1:45 AM, Brian Inglis wrote:
> Function and object pointer inter-conversions are UB - daemons fly out
> your nose! For example, when code and data model pointer sizes are
> different, pointers are physically incompatible.

Yes, such is with the case of AVR. Attempting to cast a function pointer
to a data pointer and dereferencing writes to that new pointer is a nice
way to bring about disaster.

>> However, the implementations I have seen would treat this pointer as
>> something in data memory, rather than something in program memory.
>> Actually modifying what fn_ptr points to would require the use of an
>> extension to the language [which would be implied if the behavior was
>> indeed UB or implementation defined]. Please correct me on this one.
> 
> Works on von Neumann architecture implementations, or where mapping
> registers map the same address ranges for code and data, perhaps with
> different access modes.
> Modifying what a function pointer object points to is fairly common in
> C, as long as when they are used, they are (cast to) a pointer to a
> function of the correct type; c.f. qsort pointer to comparison function
> in its last argument, and Unix system driver interfaces which are
> effectively arrays of function pointers.
> 

I have done similar to this before: mapping a page as RWX, writing to
it, then jumping to it to execute code. It was done as a way to
demonstrate shellcode injection in AMD64. I also had a few students do
something similar on an MSP430 for a homework [gave them a binary with a
vulnerability, had them reverse engineer it, then exploit it to trigger
certain behaviors to get points in the assignment].

> If you look at e.g the PDP11 architecture, somewhat similar 6800 series
> models, or the like, it had a number of mainly orthogonal general
> register addressing modes, including PC relative, indirect PC relative,
> and either with autoinc-/decrement, so it could use many registers to
> access "program" memory, absolute "program" addresses, and move through
> that space like an IP for threaded code, or as a subroutine stack, or
> access RO data in the instruction space directly.
> For example, to copy RO instruction space data to RAM, the move source
> register uses autoincrement PC relative addressing and the destination
> register uses autoincrement relative addressing from a RAM base address.

Speaking of, this sounds a lot like the MSP430 ISA. I do recall reading
somewhere that the MSP430 ISA was rather similar to the PDP11's. I can
not recall where I found that bit though. I have never used a PDP11
before, acquiring [and possibly restoring] one is in my to do list.

Anyway, thank you for your time and insights.

Cheers,
Orlando.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Help porting newlib to a new CPU architecture (sorta)

[Dave Nadler]

Lots of architectures have *really* different addressing.
AVR needs 20 or more bits for (pointers to) flash over 64kb.
16-bit x6 needs 20 bits for far pointers.
GCC mostly thinks the world is flat (and AVR __flash extension is only 
C, aarrggg...).
Some of us still support applications on those architectures!
This might give you some ideas:
http://www.nadler.com/backups/20210703_PROGMEM_notes.md 
<http://www.nadler.com/backups/20210703_PROGMEM_notes.md>
Have fun,
Best Regards, Dave

On 7/7/2021 9:58 AM, Orlando Arias wrote:
> Greetings,
>
> On 7/7/21 1:45 AM, Brian Inglis wrote:
>> Function and object pointer inter-conversions are UB - daemons fly out
>> your nose! For example, when code and data model pointer sizes are
>> different, pointers are physically incompatible.
> Yes, such is with the case of AVR. Attempting to cast a function pointer
> to a data pointer and dereferencing writes to that new pointer is a nice
> way to bring about disaster.
>
>>> However, the implementations I have seen would treat this pointer as
>>> something in data memory, rather than something in program memory.
>>> Actually modifying what fn_ptr points to would require the use of an
>>> extension to the language [which would be implied if the behavior was
>>> indeed UB or implementation defined]. Please correct me on this one.
>> Works on von Neumann architecture implementations, or where mapping
>> registers map the same address ranges for code and data, perhaps with
>> different access modes.
>> Modifying what a function pointer object points to is fairly common in
>> C, as long as when they are used, they are (cast to) a pointer to a
>> function of the correct type; c.f. qsort pointer to comparison function
>> in its last argument, and Unix system driver interfaces which are
>> effectively arrays of function pointers.
>>
> I have done similar to this before: mapping a page as RWX, writing to
> it, then jumping to it to execute code. It was done as a way to
> demonstrate shellcode injection in AMD64. I also had a few students do
> something similar on an MSP430 for a homework [gave them a binary with a
> vulnerability, had them reverse engineer it, then exploit it to trigger
> certain behaviors to get points in the assignment].
>
>> If you look at e.g the PDP11 architecture, somewhat similar 6800 series
>> models, or the like, it had a number of mainly orthogonal general
>> register addressing modes, including PC relative, indirect PC relative,
>> and either with autoinc-/decrement, so it could use many registers to
>> access "program" memory, absolute "program" addresses, and move through
>> that space like an IP for threaded code, or as a subroutine stack, or
>> access RO data in the instruction space directly.
>> For example, to copy RO instruction space data to RAM, the move source
>> register uses autoincrement PC relative addressing and the destination
>> register uses autoincrement relative addressing from a RAM base address.
> Speaking of, this sounds a lot like the MSP430 ISA. I do recall reading
> somewhere that the MSP430 ISA was rather similar to the PDP11's. I can
> not recall where I found that bit though. I have never used a PDP11
> before, acquiring [and possibly restoring] one is in my to do list.
>
> Anyway, thank you for your time and insights.
>
> Cheers,
> Orlando.


-- 
Dave Nadler, USA East Coast voice (978) 263-0097, drn@nadler.com, Skype
  Dave.Nadler1

Re: Help porting newlib to a new CPU architecture (sorta)

[Hans-Bernhard Bröker]

Am 06.07.2021 um 22:46 schrieb Orlando Arias:

 > Consider the AVR architecture, where program and data spaces have
 > distinct address spaces. We have a pointer to a string literal that
 > resides in program memory.

You're already mixing stuff up again.  The memory C string literals are 
in is, by definition, _not_ program memory.  It's read-only data memory. 
  That distinction is crucial.

Small-ish embedded CPUs do not usually implement the strict Harvard 
architecture principle, precisely because that does not support constant 
data.  A strict Harvard would have all data, including the 
const-qualified parts, in RAM, and initialize it all by running a very 
boring piece of program that just writes it all using immediate 
operands.  const data would thus consume normal RAM, without any 
write-protection by the hardware at all.

Micro controller designers have pulled different kinds of tricks to get 
around the need to have constants directly in ROM, ranging from the 
simple loop-hole instruction that does read from program memory anyway 
(like the 8051's MOVC), to various kinds of mirroring schemes that just 
map ROM into data space, essentially breaking the Harvard architecture 
rather fundamentally.

But that's ultimately a problem for the implementer of the C compiler 
and run-time library to address, if they decide to try doing that on 
such small architectures.

 > The problem with this code is that we are treating a as a pointer in
 > data memory. Declaring a to be PROGMEM does not help. We actually need
 > to rewrite the code to force the compiler to use the proper instruction:
That's what you get for throwing Standard C compatibility out the window 
by declaring that string constant using a compiler extension like PROGMEM.

Generally the compiler would be required by the Standard to implement 
"generic pointers" that can reach _all_ kinds of data defined without 
use of non-standard means.  If it doesn't do that, it is by definition 
not a C compiler.  Which can be fine, e.g. if the architecture just 
cannot have a correct C implementation otherwise, or only a horribly 
inefficient one.

But porting a generic standard C library like newlib or glibc onto a 
platform that needs non-standard compiler extensions just to emulate 
strcmp() may quickly turn into a lost cause.

 > Now, I believe that doing something like (char*)fn_ptr
 > in C is either undefined behavior

It quite explicitly is.

Re: Help porting newlib to a new CPU architecture (sorta)

[Orlando Arias]

[-- Attachment #1.1: Type: text/plain, Size: 7929 bytes --]

Greetings,

On 7/7/21 2:43 PM, Hans-Bernhard Bröker wrote:
> Am 06.07.2021 um 22:46 schrieb Orlando Arias:
> 
>> Consider the AVR architecture, where program and data spaces have
>> distinct address spaces. We have a pointer to a string literal that
>> resides in program memory.
> 
> You're already mixing stuff up again.  The memory C string literals are
> in is, by definition, _not_ program memory.  It's read-only data memory.
>  That distinction is crucial.
> 
> Small-ish embedded CPUs do not usually implement the strict Harvard
> architecture principle, precisely because that does not support constant
> data.  A strict Harvard would have all data, including the
> const-qualified parts, in RAM, and initialize it all by running a very
> boring piece of program that just writes it all using immediate
> operands.  const data would thus consume normal RAM, without any
> write-protection by the hardware at all.

At the risk of further derailing the initial conversation, I feel like
there is some misunderstanding here on the AVR architecture. Address 0
in program memory contains the code that executes as part of the reset
vector. Address 0 in data memory is a mirror of r0. There are two
physically different address spaces in that architecture. This is very
explicitly stated in the datasheet for any megaAVR or tinyAVR
microcontroller. The C compiler (gcc) treats (void*)0 as address 0 in
data memory. To initialize the .data section, the C runtime has to copy
data from one address space to a different address space. This is where
the lpm instruction comes into play: it allows you to load data across
physical address spaces. There is [unfortunately, we can debate] no
remapping/mirroring that takes place.

Now, in AVR, as you mention C string literals are expected to be in data
memory, so they need to be copied over. Because of limited SRAM,
however, the compiler provides an extension to the C language to keep
string literals in the program memory address space. String literals
stored this way are not copied over to SRAM by the runtime. Declaring
the literal as:

const char* m = "hello, world!\n";

is not enough to keep them in program memory. You have to utilize the
PROGMEM macro:

const char* m PROGMEM = "hello, world!\n";

which actually expands to __attribute__((section(".progmem"))) or some
such. To access them, you need to utilize very specific macros/functions
since the load has to be done with the lpm instruction. It may be
confusing looking at a flat dump of the binary, since gcc still treats
the end result as a "flat single address space" but in reality, that is
not how the hardware operates. There are two physically distinct address
spaces, and addresses between them share nothing in common.

This is in contrast to something like a Cortex-M based core, where
address 0 contains the initial value for the main stack pointer. The C
runtime still has to initialize .data using information from a read only
memory [usually flash]. However, this read only memory shares the same
address space as RAM. Yes, the Cortex-M core has multiple AMBA AXI ports
to connect into a bus matrix, but the memory system is still unified.
Both program memory [flash/ROM/FeRAM...] and data memory [SRAM...] are
in the same address space.

You can declare something like:

const char*m = "hello, world!\n";

and the compiler is smart enough to keep that data in the read only
portions of memory [namely flash/ROM/FeRAM...]. They will not be copied
over to SRAM by the C runtime. Accesses and references will be performed
[using the ldr* family of instructions]. In fact, the C compiler will
embed large integer literals in program code, and load them directly
from read only memory into registers. This is because there is a limit
as to how large of an integer literal can be encoded in a mov
instruction. This is also how things like jump tables are implemented by
gcc on Arm architectures [both A and M profiles, can not say for R
profiles since I have not used them, but I imagine it is the same].

> Micro controller designers have pulled different kinds of tricks to get
> around the need to have constants directly in ROM, ranging from the
> simple loop-hole instruction that does read from program memory anyway
> (like the 8051's MOVC), to various kinds of mirroring schemes that just
> map ROM into data space, essentially breaking the Harvard architecture
> rather fundamentally.

I have seen the the mirroring scheme at work before. The STM32F4
microcontrollers [Cortex-M4F cores], for example, map internal flash to
both at address 0 and address 0x08000000. SRAM begins at 0x20000000 as
per the Armv7-M standard mandates, followed by a bit banding region for
SRAM. From the perspective of the Cortex core, this is all in a single,
unified address space. Yes, both flash and SRAM are different memory
types, with different characteristics and power, clock, and access
requirements, but they all lie in the same address space. This is unlike
AVR, where no such schemes are available.

This also has the side effect that you can not really do code injection
in AVR. You can copy as much shellcode as you want to SRAM, but you will
not be able to execute it, unless the currently executing code is in the
bootloader section of program memory, the bootloader copies the code to
program memory, then proceeds to execute it [and this requires a rather
convoluted process]. In Arm Cortex-M cores, however, you can have code
execute from SRAM as if it was executing from a read only memory [MPU
permissions notwithstanding]. OpenOCD does this a lot, actually, when
dealing with Arm-based microcontrollers. In order to load a program into
flash, they inject code into SRAM which configures the [memory mapped]
flash controller for the core you are working with to allow for writes,
then proceed to have the flash controller store the program. Because how
the address map in an Armv7-M [and Armv8-M for that matter] core is
structured, the end result is the program code available starting at
address 0 [with the initial vector table at that location].

> But that's ultimately a problem for the implementer of the C compiler
> and run-time library to address, if they decide to try doing that on
> such small architectures.
> 
>> The problem with this code is that we are treating a as a pointer in
>> data memory. Declaring a to be PROGMEM does not help. We actually need
>> to rewrite the code to force the compiler to use the proper instruction:
> That's what you get for throwing Standard C compatibility out the window
> by declaring that string constant using a compiler extension like PROGMEM.
> 
> Generally the compiler would be required by the Standard to implement
> "generic pointers" that can reach _all_ kinds of data defined without
> use of non-standard means.  If it doesn't do that, it is by definition
> not a C compiler.  Which can be fine, e.g. if the architecture just
> cannot have a correct C implementation otherwise, or only a horribly
> inefficient one.
> 
> But porting a generic standard C library like newlib or glibc onto a
> platform that needs non-standard compiler extensions just to emulate
> strcmp() may quickly turn into a lost cause.
> 

Except that you need to do this, because it is how the architecture
works. If you do not care about conserving SRAM in AVR, you can declare
your literals as const. The compiler will do its thing and assume
constness for optimization purposes, but the runtime will happily copy
them over to SRAM at startup and you can use your standard C library
functions. Now, if you want to be more conscious about your SRAM usage,
you need to use the non-standard means I mentioned. The fact that there
are two physically distinct address spaces requires that.

Cheers,
Orlando.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]