Rutger's NAND flash now has a synth package

Rutger's NAND flash now has a synth package

[Rutger Hofman]

Good afternoon list,

in the usual work storm in my office, there was a lull last week. I took 
the opportunity to make a synthetic target for my NAND flash package. 
Simon Kallweit had already started work on this (but put it on hold when 
eCosCentric's NAND package came up); the level of emulation in his 
implementation was the NAND chip with its set of wires.

I discussed this with Simon, and we agreed that the fastest way to a 
synth package would be to make an integrated NFC (NAND flash controller) 
and chip(s) package. This turned out to be straightforward. The NAND 
chip(s) are emulated in the same way as in the NOR flash synth targets: 
by memory-mapping a file to represent a chip.

The synth target I built thus has the following properties/limitations:
  - it generically supports 'regular' large-page chips; there is no 
support for small-page chips; there is also no real support (yet) for 
ONFI chips, which differ from 'regular' chips in the interrogation. This 
didn't seem to me to be a harmful limitation because there seem to be no 
ONFI chips on the market (yet);
  - it supports x8 and x16 chips (8 and 16 being the chip's bus width);
  - it supports multiple chips connected to one NFC;
  - NAND chip size is limited to max file size and to max mappable file 
size. This limit could be extended by spreading the chip over multiple 
files, if the need arises;
  - like the NOR flash synth target, the NAND synth target is completely 
linked into the target executable. There is no host auxiliary component;
  - it lives under packages/devs/nand/nfc_synth/.

I tested this synth target with an (emulated) x8 and an x16 chip, and 
also with a configuration with both an x8 and an x16 chip. YAFFS 
requires one chip, but my Abstract NAND Chip layer hides the presence of 
multiple chips so I could run YAFFS tests that use a filesystem that 
spreads over both chips. Caveat: the page/spare/block size of both 
emulated chips was identical. I didn't (yet) test with chips that differ 
in those respects.

A few small bugs came out, related to x16 (I erroneously divided by bus 
width somewhere), and related to ANC-to-physical address calculations 
for multiple chips.

Reflecting discussion on the eCos lists, I also changed the package 
names from flash_nand/ to nand/, and moved NAND flash device drivers 
from under devs/flash to devs/nand/.

The code is published in the same place as before:
http://www.cs.vu.nl/~rutger/software/ecos/nand-flash/

I appreciate comments and usage.

An aside: I run Ubuntu. At first, I couldn't run synth at all. 
Applications would crash, and gdb would crash on the application too! 
After some list searching, I found out that this probably is 
Ubuntu-specific. We need to include -fno-stack-protector in the 
GLOBAL_CFLAGS configure flag. Request: cannot this be automated for 
synth building? My guess is that it will not harm on systems other than 
Ubuntu, and it will save Ubuntu users effort.

Rutger Hofman
VU Amsterdam

Re: Rutger's NAND flash now has a synth package

[Simon Kallweit]

Rutger Hofman wrote:
> Good afternoon list,
> 
> in the usual work storm in my office, there was a lull last week. I took 
> the opportunity to make a synthetic target for my NAND flash package. 
> Simon Kallweit had already started work on this (but put it on hold when 
> eCosCentric's NAND package came up); the level of emulation in his 
> implementation was the NAND chip with its set of wires.
> 
> I discussed this with Simon, and we agreed that the fastest way to a 
> synth package would be to make an integrated NFC (NAND flash controller) 
> and chip(s) package. This turned out to be straightforward. The NAND 
> chip(s) are emulated in the same way as in the NOR flash synth targets: 
> by memory-mapping a file to represent a chip.
> 
> The synth target I built thus has the following properties/limitations:
>  - it generically supports 'regular' large-page chips; there is no 
> support for small-page chips; there is also no real support (yet) for 
> ONFI chips, which differ from 'regular' chips in the interrogation. This 
> didn't seem to me to be a harmful limitation because there seem to be no 
> ONFI chips on the market (yet);
>  - it supports x8 and x16 chips (8 and 16 being the chip's bus width);
>  - it supports multiple chips connected to one NFC;
>  - NAND chip size is limited to max file size and to max mappable file 
> size. This limit could be extended by spreading the chip over multiple 
> files, if the need arises;
>  - like the NOR flash synth target, the NAND synth target is completely 
> linked into the target executable. There is no host auxiliary component;
>  - it lives under packages/devs/nand/nfc_synth/.
> 
> I tested this synth target with an (emulated) x8 and an x16 chip, and 
> also with a configuration with both an x8 and an x16 chip. YAFFS 
> requires one chip, but my Abstract NAND Chip layer hides the presence of 
> multiple chips so I could run YAFFS tests that use a filesystem that 
> spreads over both chips. Caveat: the page/spare/block size of both 
> emulated chips was identical. I didn't (yet) test with chips that differ 
> in those respects.
> 
> A few small bugs came out, related to x16 (I erroneously divided by bus 
> width somewhere), and related to ANC-to-physical address calculations 
> for multiple chips.
> 
> Reflecting discussion on the eCos lists, I also changed the package 
> names from flash_nand/ to nand/, and moved NAND flash device drivers 
> from under devs/flash to devs/nand/.
> 
> The code is published in the same place as before:
> http://www.cs.vu.nl/~rutger/software/ecos/nand-flash/
> 
> I appreciate comments and usage.

That sounds like good news. Soon I'll have to do a little studies on 
flash filesystems for our platform, so I think I'll port the UFFS 
filesystem to eCos and write the necessary device drivers for the STM32 
family. I think your framework just got a little more attractive with 
the recent changes.

> An aside: I run Ubuntu. At first, I couldn't run synth at all. 
> Applications would crash, and gdb would crash on the application too! 
> After some list searching, I found out that this probably is 
> Ubuntu-specific. We need to include -fno-stack-protector in the 
> GLOBAL_CFLAGS configure flag. Request: cannot this be automated for 
> synth building? My guess is that it will not harm on systems other than 
> Ubuntu, and it will save Ubuntu users effort.

I observed the same bug. But I started using eCosCentrics i386 toolchain 
for my synth builds which works fine with standard configuration. It 
will also make sure that I don't run into GCC specifics from time to 
time, as  the distro's toolchain is updated quite often.

Simon

GCC stack protector with linux synthetic target

[John Dallaway]

Hi Rutger

Rutger Hofman wrote:

> An aside: I run Ubuntu. At first, I couldn't run synth at all.
> Applications would crash, and gdb would crash on the application too!
> After some list searching, I found out that this probably is
> Ubuntu-specific. We need to include -fno-stack-protector in the
> GLOBAL_CFLAGS configure flag. Request: cannot this be automated for
> synth building? My guess is that it will not harm on systems other than
> Ubuntu, and it will save Ubuntu users effort.

Do you understand _why_ eCos is crashing with the GCC stack protector
enabled? Is it due to an assumption made in eCos regarding the stack
frame which could be corrected? We should ensure we understand the
nature of the failure before disabling this feature.

John Dallaway

Re: GCC stack protector with linux synthetic target

[Andrew Lunn]

> Do you understand _why_ eCos is crashing with the GCC stack protector
> enabled? Is it due to an assumption made in eCos regarding the stack
> frame which could be corrected? We should ensure we understand the
> nature of the failure before disabling this feature.

Bart looked at this a while back. Ubuntu is leading the way here, but
other distros are scheduled to follow. So this issue is going to get
more important with time.

Bart said it is messy and not easy to make eCos do the right
thing. Either we take the short term solution of turning it off
globally for synth, or somebody invests some time to properly fix
the synth HAL.

        Andrew

Re: Rutger's NAND flash now has a synth package

[Sergei Gavrikov]

On Mon, Jun 29, 2009 at 08:42:07AM +0200, Simon Kallweit wrote:
> Rutger Hofman wrote:

> >An aside: I run Ubuntu. At first, I couldn't run synth at all. 
> >Applications would crash, and gdb would crash on the application too! 
> >After some list searching, I found out that this probably is 
> >Ubuntu-specific. We need to include -fno-stack-protector in the 
> >GLOBAL_CFLAGS configure flag. Request: cannot this be automated for 
> >synth building? My guess is that it will not harm on systems other than 
> >Ubuntu, and it will save Ubuntu users effort.
> 
> I observed the same bug. But I started using eCosCentrics i386 toolchain 
> for my synth builds which works fine with standard configuration. It 
> will also make sure that I don't run into GCC specifics from time to 
> time, as  the distro's toolchain is updated quite often.

Hi

My 2 cents. That i386-elf GCC build has not `libgcc_eh.a' library

find /opt/ecos/gnutools/i386-elf/ -name libgcc_eh.a
<< no output >>

but the library is "used" for synth

hal/synth/arch/current/src/synth.ld:

#if (__GNUC__ >= 3)
GROUP(libtarget.a libgcc.a libsupc++.a libgcc_eh.a)
#else
GROUP(libtarget.a libgcc.a)
#endif

Can we manage the check the above with CDL?

Because 2 odd? steps to make synth tests are necessary

1) Import this option

cdl_option CYGBLD_GLOBAL_COMMAND_PREFIX {
    user_value i386-elf
};

2) Workaround for the absent libgcc_eh.a in ld GROUP

sed -i 's/ libgcc_eh.a//' install/lib/target.ld


Sergei

Re: GCC stack protector with linux synthetic target

[Bart Veer]

>>>>> "John" == John Dallaway <john@dallaway.org.uk> writes:

    John> Hi Rutger
    John> Rutger Hofman wrote:

    >> An aside: I run Ubuntu. At first, I couldn't run synth at all.
    >> Applications would crash, and gdb would crash on the
    >> application too! After some list searching, I found out that
    >> this probably is Ubuntu-specific. We need to include
    >> -fno-stack-protector in the GLOBAL_CFLAGS configure flag.
    >> Request: cannot this be automated for synth building? My guess
    >> is that it will not harm on systems other than Ubuntu, and it
    >> will save Ubuntu users effort.

    John> Do you understand _why_ eCos is crashing with the GCC stack
    John> protector enabled? Is it due to an assumption made in eCos
    John> regarding the stack frame which could be corrected? We
    John> should ensure we understand the nature of the failure before
    John> disabling this feature.

This was investigated some time ago - look at the archives.

Basically -fstack-protector depends on some extra work done by the
glibc startup code. Of course the synthetic target does not use glibc
so that extra bit of initialization does not happen. The extra init is
not straightforward. It involves manipulating the x86 segment
registers via a Linux system call. I could find no documentation for
exactly what has to be done, and I could not immediately figure it out
from the glibc sources. Straightforward copying from glibc is not
acceptable either because of licensing issues.

Stack protection support was added to the compiler a few years back,
offhand I don't know exactly when. However the default setting varies
between distributions. Fedora and its ilk default to
-fno-stack-protector, so everything just works. Debian and its ilk
default to -fstack-protector, so things go wrong.

Possible solutions are:

1) add -fno-stack-protector to the default flags for synth. This
should work fine with all current distros. However it will break the
world for anybody using an older distro where the gcc being used did
not yet know about this compiler flag, or for anybody deliberately
using an older gcc e.g. to use the same compiler version for synth and
for real embedded targets. This was not really acceptable when we last
looked at this. It may be more acceptable now, but is still not ideal.

2) try to do some run-time detection to figure out whether or not
-fno-stack-protector should be added. There are various complications,
e.g. if people change the COMMAND_PREFIX setting.

3) fix the problem properly by doing the segment register
initialization. This should solve the problem irrespective of the
distro or the version of gcc being used. It would also mean that the
synthetic targets gains whatever benefits -fstack-protector offers.


Since I run Fedora on most of my systems I am not affected by any of
this, so sorting it out is a low priority for me. If you want to look
at option (3), that would be great. I would much prefer that to option
(1), since if you go for that then there will never be any incentive
to do the job properly.

Bart

-- 
Bart Veer                                   eCos Configuration Architect
eCosCentric Limited    The eCos experts      http://www.ecoscentric.com/
Barnwell House, Barnwell Drive, Cambridge, UK.      Tel: +44 1223 245571
Registered in England and Wales: Reg No 4422071.

Re: Rutger's NAND flash now has a synth package

[Rutger Hofman]

Simon Kallweit wrote:
> Rutger Hofman wrote:
>> Good afternoon list,
>>
>> in the usual work storm in my office, there was a lull last week. I 
>> took the opportunity to make a synthetic target for my NAND flash 
>> package.
>
> That sounds like good news. Soon I'll have to do a little studies on 
> flash filesystems for our platform, so I think I'll port the UFFS 
> filesystem to eCos and write the necessary device drivers for the STM32 
> family.
...

I am very curious about the usability and stability of UFFS!

As to the STM32 driver: if you decide to use my NAND framework, I will 
be happy to assist or discuss.

Rutger

Re: GCC stack protector with linux synthetic target

[John Dallaway]

Hi Bart

Bart Veer wrote:

> Basically -fstack-protector depends on some extra work done by the
> glibc startup code. Of course the synthetic target does not use glibc
> so that extra bit of initialization does not happen. The extra init is
> not straightforward. It involves manipulating the x86 segment
> registers via a Linux system call. I could find no documentation for
> exactly what has to be done, and I could not immediately figure it out
> from the glibc sources. Straightforward copying from glibc is not
> acceptable either because of licensing issues.
> 
> Stack protection support was added to the compiler a few years back,
> offhand I don't know exactly when. However the default setting varies
> between distributions. Fedora and its ilk default to
> -fno-stack-protector, so everything just works. Debian and its ilk
> default to -fstack-protector, so things go wrong.
> 
> Possible solutions are:
> 
> 1) add -fno-stack-protector to the default flags for synth. This
> should work fine with all current distros. However it will break the
> world for anybody using an older distro where the gcc being used did
> not yet know about this compiler flag, or for anybody deliberately
> using an older gcc e.g. to use the same compiler version for synth and
> for real embedded targets. This was not really acceptable when we last
> looked at this. It may be more acceptable now, but is still not ideal.
> 
> 2) try to do some run-time detection to figure out whether or not
> -fno-stack-protector should be added. There are various complications,
> e.g. if people change the COMMAND_PREFIX setting.
> 
> 3) fix the problem properly by doing the segment register
> initialization. This should solve the problem irrespective of the
> distro or the version of gcc being used. It would also mean that the
> synthetic targets gains whatever benefits -fstack-protector offers.
> 
> Since I run Fedora on most of my systems I am not affected by any of
> this, so sorting it out is a low priority for me. If you want to look
> at option (3), that would be great. I would much prefer that to option
> (1), since if you go for that then there will never be any incentive
> to do the job properly.

Agreed. Thanks for the analysis. I have raised a bug report so this
doesn't get forgotten about:

  http://bugs.ecos.sourceware.org/show_bug.cgi?id=1000801

There is also an issue with very old GCC:

  http://bugs.ecos.sourceware.org/show_bug.cgi?id=1000723

John Dallaway