[ECOS] FLASH API v.2 and interrupts

[ECOS] FLASH API v.2 and interrupts

[Iris Lindner]

Hi everyone,

(Please excuse if I have overlooked a solution to our problem in mailing lists 
for example.)

we have the following problem with our system (using redboot romram mode):
We can't properly harmonize flash operations and interrupt handling. There are 
regularly frames coming in on CAN bus (lifesign signals of the nodes) which 
have to be treated. 

We use Version 1 eCos FLASH API (chip: Spansion S29GL256P) and get terrible 
crashes (including completely empty flash afterwards) when interrupts are NOT 
disabled and flash is programmed for example when meanwhile CAN bus traffic 
is increased.

If we disable interrupts on device driver level (HAL_[DISABLE|
RESTORE]_INTERRUPTS()) flash operations all work fine 
but CAN frames get lost and the task goes offline (-> whole machine no longer 
able to work).

Could Version 2 of eCos FLASH API solve the problem? (It is interrupt safe 
now, isn't it?)
Do you need more information about our system/configuration (I'm not sure 
about what details are useful to know)? 

Thank you very much in advance for your help, I'm looking forward to any 
comment!
Kind regards,
Iris Lindner

-- 
Iris Lindner
Software Development

Industrial Print and Apply Labelling

www.Logopak.com

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[ECOS] IDE failed to identify unit 0 - wrote: a0, read: 7f

[Jim Bradleigh]

Hi,


I`ve trying to get Redboot to "see" my compact flash (16Mb) IDE drive.

I`m using GRUB to boot Redboot, which then reports the error in the subject (IDE failed to identify unit 0 - wrote: a0, read: 7f) before you see the "RedBoot (tm) bootstrap and debug.." message.

Any reason why redboot would be having problems with seeing the drive? 

For the record, GRUB can use it fine and I can mount and view a normal IDE hard disk from redboot without issues.

What debug should I enable? turning on "Enable debugging output" in "Disk Sevice Drivers" (CYGPKG_IO_DISK) package causes redboot to crash on start-up (I beleive its a GDB debug string printed to serial port)

Thanks,

Jim


--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[ECOS] Debugging Redboot..

[Jim Bradleigh]

Hi,

Is there a guide to how to debug "into" redboot?

I`ve managed to get redboot loaded via GDB (using redboot itself) and I can get the application to run, and certain breakpoints to function during easrly startup. 

However when the redboot prompt appears I cannot Ctrl+C and any breakpoints don`t seem to trigger (i.e. when debugging redboot commands)

Are there special options required to allow me to gain full access to debug redboot?

Jim


--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Debugging Redboot..

[Bart Veer]

>>>>> "Jim" == Jim Bradleigh <jim.bradleigh1@btinternet.com> writes:

    Jim> Hi,
    Jim> Is there a guide to how to debug "into" redboot?

    Jim> I`ve managed to get redboot loaded via GDB (using redboot
    Jim> itself) and I can get the application to run, and certain
    Jim> breakpoints to function during easrly startup.

    Jim> However when the redboot prompt appears I cannot Ctrl+C and
    Jim> any breakpoints don`t seem to trigger (i.e. when debugging
    Jim> redboot commands)

    Jim> Are there special options required to allow me to gain full
    Jim> access to debug redboot?

In general you cannot use a ROM RedBoot to debug a RAM RedBoot, or
anything along those lines. The problem is that RedBoot always takes
over certain bits of hardware, including the h/w exception vector(s)
used for debugging and the uart for communicating with the host. Hence
shortly after the RAM RedBoot start running the ROM RedBoot will
become disconnected from the rest of the world, and can no longer
interact with gdb. Changing this behaviour is non-trivial.

Hence debugging RedBoot itself generally involves jtag or similar h/w
debug support. If that is not available then you may have to resort to
debugging via diag_printf()'s.

Bart

-- 
Bart Veer                                   eCos Configuration Architect
eCosCentric Limited    The eCos experts      http://www.ecoscentric.com/
Barnwell House, Barnwell Drive, Cambridge, UK.      Tel: +44 1223 245571
Registered in England and Wales: Reg No 4422071.
       >>>> Visit us at ESC-UK  http://www.embedded.co.uk <<<<
       >>>> Oct 7-8 on Stand 433 at FIVE ISC, Farnborough <<<<

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] FLASH API v.2 and interrupts

[Stanislav Meduna]

Iris Lindner wrote:

> We use Version 1 eCos FLASH API (chip: Spansion S29GL256P) and get terrible 
> crashes (including completely empty flash afterwards) when interrupts are NOT 
> disabled and flash is programmed for example when meanwhile CAN bus traffic 
> is increased.

I have no problems writing in a jffs2 filesystem on
a Dataflash device while processing interrupts from
timer, network card, CAN controller, so I suppose
the V2 API is pretty safe.

I did have issues with locking the flash device
on the thread level, but I am on hardware that
is not really well thought out and shares a slow bus
for things with contradicting requirements.

> Do you need more information about our system/configuration (I'm not sure 
> about what details are useful to know)? 

I'd check the following first:

- I don't know the flash chip - what driver are you using?
  Are there time-critical tasks that can't be interrupted?
  What driver model - nearly everything in ISR, or does DSR
  take care of the transfers? Interrupt-driven or polled?

  Normally you disable the interrupts for shifting data
  to the chip, then leave it enabled and only check
  when the flash operation ends - periodically
  or interrupt-driven, again with interrupts disabled
  while the check operation itself.

- Check the math - time to transfer data to flash or to check
  whether the flash completed writing versus shortest CAN frame,
  CAN bus speed and buffer depth of your controller. Shortest CAN
  frame at 1 Mbit takes tens of microseconds, how much time
  does transfer of a data page to your flash take (here
  the interrupts normally have to be disabled)?

  If the worst-case margin is too narrow, slow the CAN bus speed.

- Are you accessing the flash from several threads?
  Especially triggered by reception of a CAN frame?
  The link to crashes are not necessarily the enabled
  interrupts per se, maybe they are just triggering something
  else.

- Stack sizes. If processing of the CAN interrupt takes
  non-trivial amount of stack, this can have various
  effects.


Regards
-- 
                                     Stano

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] FLASH API v.2 and interrupts

[Bart Veer]

>>>>> "Iris" == Iris Lindner <ilindner@logopak.de> writes:

    Iris> Hi everyone,
    Iris> (Please excuse if I have overlooked a solution to our
    Iris> problem in mailing lists for example.)

    Iris> we have the following problem with our system (using redboot
    Iris> romram mode): We can't properly harmonize flash operations
    Iris> and interrupt handling. There are regularly frames coming in
    Iris> on CAN bus (lifesign signals of the nodes) which have to be
    Iris> treated.

    Iris> We use Version 1 eCos FLASH API (chip: Spansion S29GL256P)
    Iris> and get terrible crashes (including completely empty flash
    Iris> afterwards) when interrupts are NOT disabled and flash is
    Iris> programmed for example when meanwhile CAN bus traffic is
    Iris> increased.

    Iris> If we disable interrupts on device driver level
    Iris> (HAL_[DISABLE| RESTORE]_INTERRUPTS()) flash operations all
    Iris> work fine but CAN frames get lost and the task goes offline
    Iris> (-> whole machine no longer able to work).

    Iris> Could Version 2 of eCos FLASH API solve the problem? (It is
    Iris> interrupt safe now, isn't it?) Do you need more information
    Iris> about our system/configuration (I'm not sure about what
    Iris> details are useful to know)?

    Iris> Thank you very much in advance for your help, I'm looking
    Iris> forward to any comment!

As a general rule, interrupts and updating NOR flash do not mix. While
a flash erase or program operation is taking place the flash is
unusable for anything else (there are complications with some flash
chips supporting multiple banks, but I don't want to get into all the
unpleasant details). If an interrupt occurs during this time and the
interrupt handling code needs to access the flash, you are in trouble.
If an interrupt results in a context switch to another thread and that
thread needs to access the flash, you are also in trouble. If the
application is being debugged via a ROM RedBoot executing from flash
then hitting a breakpoint or calling printf() or diag_printf() will
call into that ROM RedBoot, which means code trying to execute from
flash.

By default the anoncvs AMD V2 driver (which I think is what you would
want for that Spansion chip) avoids crashes by completely disabling
interrupts around the program and erase operations. Hence it is safe,
but interrupts can remain disabled for a long time.

On some hardware there will be an active configuration option
CYGIMP_DEVS_FLASH_AMD_AM29XXXXX_V2_LEAVE_INTERRUPTS_ENABLED, which is
off by default. Toggling this option means that the flash driver will
no longer disable interrupts. However, you are responsible for making
sure that nothing else will access the flash while an erase or program
operation is happening. That means appropriate locking within your
application, not debugging via a ROM RedBoot, and quite possibly other
complications depending on your hardware. It is a lot easier for
something like dataflash rather than NOR flash: the dataflash is not
directly accessible to the cpu so you will never have RedBoot
executing out of it.

On other hardware the flash driver has to disable the cache before it
can talk directly to the flash chip. You do not want interrupts or
context switches while the cache is disabled, the resulting system
performance would be terrible. The LEAVE_INTERRUPTS_ENABLED option is
not available on hardware like this.

So, switching to the V2 API would prevent the crashes. It would no
longer be necessary to disable interrupts in your own code, instead
that happens in the driver. However, a switch would not address the
performance problems unless the hardware supports
LEAVE_INTERRUPTS_ENABLED and you can guarantee that its requirements
are satisfied.

There is a more advanced version of the AMD V2 driver which disables
interrupts for shorter periods of time. Writes happen in burst with
interrupts briefly reenabled between bursts. Similarly erases happen
in bursts, using the erase suspend/resume facilities available in most
NOR chips. Interrupt latency is much improved but still not great - if
the erase bursts are too short then the chip may never make any
progress. The details vary between flash chips. This more advanced
driver is currently only available from eCosCentric.

Bart

-- 
Bart Veer                                   eCos Configuration Architect
eCosCentric Limited    The eCos experts      http://www.ecoscentric.com/
Barnwell House, Barnwell Drive, Cambridge, UK.      Tel: +44 1223 245571
Registered in England and Wales: Reg No 4422071.
       >>>> Visit us at ESC-UK  http://www.embedded.co.uk <<<<
       >>>> Oct 7-8 on Stand 433 at FIVE ISC, Farnborough <<<<

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

RE: [ECOS] FLASH API v.2 and interrupts

[Paul D. DeRocco]

> From: Bart Veer
>
> On other hardware the flash driver has to disable the cache before it
> can talk directly to the flash chip. You do not want interrupts or
> context switches while the cache is disabled, the resulting system
> performance would be terrible. The LEAVE_INTERRUPTS_ENABLED option is
> not available on hardware like this.

That seems an odd assumption to make. In lots of systems, max CPU horsepower
is only needed intermittently, and keeping the system responsive, say, to
slow user input would be preferable to shutting everything down during flash
writes.

--

Ciao,               Paul D. DeRocco
Paul                mailto:pderocco@ix.netcom.com


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] FLASH API v.2 and interrupts

[Bart Veer]

>>>>> "Paul" == Paul D DeRocco <pderocco@ix.netcom.com> writes:

    >> From: Bart Veer
    >> 
    >> On other hardware the flash driver has to disable the cache
    >> before it can talk directly to the flash chip. You do not want
    >> interrupts or context switches while the cache is disabled, the
    >> resulting system performance would be terrible. The
    >> LEAVE_INTERRUPTS_ENABLED option is not available on hardware
    >> like this.

    Paul> That seems an odd assumption to make. In lots of systems,
    Paul> max CPU horsepower is only needed intermittently, and
    Paul> keeping the system responsive, say, to slow user input would
    Paul> be preferable to shutting everything down during flash
    Paul> writes.

Running with the cache disabled is likely to mean a performance
penalty of between one and two orders of magnitude. Now, if the system
is basically idle at the time of the flash operation then you might
get away with running for a while with the cache disabled. This would
require that the interrupt be fully serviced (ISR, possibly DSR,
possibly some thread-level work) before the next interrupt. The cpu
gets to context switch back to the thread performing the flash
operation, and progress would be made.

However, if the system is at all busy then chances are it won't manage
to service the interrupt before the next one comes in. If this happens
control may not return to the right thread for a long time, if ever,
so the flash operation does not complete in a reasonable time and the
cache remains disabled. To make matters worse, debugging this kind of
problem is going to be tricky.

In my opinion you would need an awful lot of confidence in your
predictions of the expected I/O loads and in your performance
profiling data to take the risk of running for an extended period of
time with no cache. Or else you would need a cpu which is spec'd
significantly higher than what the application really needs, adding to
the product costs.

A further complication is that the cache state is a simple toggle, not
a stack. Hence you could have code running after the interrupts which
disables the cache (even though it was still disabled by the flash
operation), then reenables it later. If/when the flash thread gets to
run again the driver will find the cache unexpectedly enabled, and
at that point you can expect things to go badly wrong.

So I believe it is sensible for the AMD V2 driver to disallow
LEAVE_INTERRUPTS_ENABLED on hardware where there are cache issues. The
alternative is just too dangerous. If people are really unhappy about
it then they have the source and can change it for their needs.

Bart
    
-- 
Bart Veer                                   eCos Configuration Architect
eCosCentric Limited    The eCos experts      http://www.ecoscentric.com/
Barnwell House, Barnwell Drive, Cambridge, UK.      Tel: +44 1223 245571
Registered in England and Wales: Reg No 4422071.
       >>>> Visit us at ESC-UK  http://www.embedded.co.uk <<<<
       >>>> Oct 7-8 on Stand 433 at FIVE ISC, Farnborough <<<<

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] FLASH API v.2 and interrupts

[Iris Lindner]

Dear Stanislav, dear Bart, dear Paul,
thank you very much for your detailed and helpful response!

> I have no problems writing in a jffs2 filesystem on
> a Dataflash device while processing interrupts from
> timer, network card, CAN controller, so I suppose
> the V2 API is pretty safe.

That's good to hear. =) Currently I try to update our existing configuration. 
I removed the old API ("Flash device drivers" in config tool) and the device 
dirver itself ("AMD AM29XXXXX FLASH memory support"). After inserting the new 
packages and updating ecos.db I could change our template and so insert the 
new device driver and API into the configuration. Unfortunately I can't 
compile yet, first error is:
../cyg/io/am29xxxxx_dev.h:81: Fehler: Fehler beim Parsen vor "cyg_flashaddr_t"
(error parsing before "cyg_flashaddr_t")
Still investigating...

> I'd check the following first:
>
> - I don't know the flash chip - what driver are you using?
>   Are there time-critical tasks that can't be interrupted?
>   What driver model - nearly everything in ISR, or does DSR
>   take care of the transfers? Interrupt-driven or polled?
We used the old V.1 API and driver ("AMD AM29XXXXX FLASH memory support") eCos 
packages. Except from the flash update operations (see also Bart's response) 
no more non-interruptable operations as far as I can see (using interrupts 
and their ISRs).

> - Check the math - time to transfer data to flash or to check
>   whether the flash completed writing versus shortest CAN frame,
>   CAN bus speed and buffer depth of your controller. Shortest CAN
>   frame at 1 Mbit takes tens of microseconds, how much time
>   does transfer of a data page to your flash take (here
>   the interrupts normally have to be disabled)?
Our shortest CAN frame has 58 bits, the baudrate is 125 kBit. 
The device is only able to store one frame - so at least every 464 us the 
CAN-interrupt must be able to be executed.
Sometimes it is necessary to transfer bigger files (up to 3 MB) to flash which 
takes several seconds. (But I haven't measured time yet for transferring a 
data page...*to do*)

> - Are you accessing the flash from several threads?
>   Especially triggered by reception of a CAN frame?
>   The link to crashes are not necessarily the enabled
>   interrupts per se, maybe they are just triggering something
>   else.
No, normally everything _should_ be fine... Every task could access flash but 
only via the same interface where we have locking/semaphores. A CAN frame 
cannot trigger anything which leads to flash access (in theory).

Am Mittwoch, 7. Oktober 2009 18:55:40 schrieben Sie:
>
> As a general rule, interrupts and updating NOR flash do not mix. While
> a flash erase or program operation is taking place the flash is
> unusable for anything else (there are complications with some flash
> chips supporting multiple banks, but I don't want to get into all the
> unpleasant details). If an interrupt occurs during this time and the
> interrupt handling code needs to access the flash, you are in trouble.
> If an interrupt results in a context switch to another thread and that
> thread needs to access the flash, you are also in trouble. If the
> application is being debugged via a ROM RedBoot executing from flash
> then hitting a breakpoint or calling printf() or diag_printf() will
> call into that ROM RedBoot, which means code trying to execute from
> flash.
Ah, that's valuable to know! I think all these three cases do not match at our 
system but I'll dig deeper into it to become sure.

> By default the anoncvs AMD V2 driver (which I think is what you would
> want for that Spansion chip) avoids crashes by completely disabling
> interrupts around the program and erase operations. Hence it is safe,
> but interrupts can remain disabled for a long time.
Ok, so that would probably be the same behaviour as with our old driver where 
disabling interrupts has manually been inserted. As result we get stable 
flash behaviour but that frame loss phenomenon which results in a disabled 
can bus...

> On other hardware the flash driver has to disable the cache before it
> can talk directly to the flash chip. You do not want interrupts or
> context switches while the cache is disabled, the resulting system
> performance would be terrible. The LEAVE_INTERRUPTS_ENABLED option is
> not available on hardware like this.
I couldn't find disabling of cache in sources or chip documentation.

> So, switching to the V2 API would prevent the crashes. It would no
> longer be necessary to disable interrupts in your own code, instead
> that happens in the driver. However, a switch would not address the
> performance problems unless the hardware supports
> LEAVE_INTERRUPTS_ENABLED and you can guarantee that its requirements
> are satisfied.
>
> There is a more advanced version of the AMD V2 driver which disables
> interrupts for shorter periods of time. Writes happen in burst with
> interrupts briefly reenabled between bursts. Similarly erases happen
> in bursts, using the erase suspend/resume facilities available in most
> NOR chips. Interrupt latency is much improved but still not great - if
> the erase bursts are too short then the chip may never make any
> progress. The details vary between flash chips. This more advanced
> driver is currently only available from eCosCentric.
So I think I could try to use the driver with interrupts left enabled and 
making sure that flash is not accessed from somewhere else, right? But when 
the machine is running it is absolutely necessary to keep the can bus 
working. Anything which has to do with flash (transferring a new file e.g.) 
is of lower priority, so your last proposition of the advanced AMD V2 driver 
sounds like a good solution, especially with using suspend/resume 
functionality. (Does one have to purchase it from eCosCentric?)

Thank you very much again!
Kind Regards,
Iris

-- 
Iris Lindner
Software Development

Industrial Print and Apply Labelling

www.Logopak.com

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] FLASH API v.2 and interrupts

[Bart Veer]

>>>>> "Iris" == Iris Lindner <ilindner@logopak.de> writes:

    Iris> Dear Stanislav, dear Bart, dear Paul,
    Iris> thank you very much for your detailed and helpful response!

    <snip>

    >> On other hardware the flash driver has to disable the cache
    >> before it can talk directly to the flash chip. You do not want
    >> interrupts or context switches while the cache is disabled, the
    >> resulting system performance would be terrible. The
    >> LEAVE_INTERRUPTS_ENABLED option is not available on hardware
    >> like this.
    
    Iris> I couldn't find disabling of cache in sources or chip
    Iris> documentation.

http://ecos.sourceware.org/docs-latest/ref/am29xxxxx-instance.html
Then scroll down to the bottom, the last section describes Cache
Management. In the current sources look at am29xxxxx.c, especially
line 260 onwards.

    >> So, switching to the V2 API would prevent the crashes. It would
    >> no longer be necessary to disable interrupts in your own code,
    >> instead that happens in the driver. However, a switch would not
    >> address the performance problems unless the hardware supports
    >> LEAVE_INTERRUPTS_ENABLED and you can guarantee that its
    >> requirements are satisfied.
    >> 
    >> There is a more advanced version of the AMD V2 driver which
    >> disables interrupts for shorter periods of time. Writes happen
    >> in burst with interrupts briefly reenabled between bursts.
    >> Similarly erases happen in bursts, using the erase
    >> suspend/resume facilities available in most NOR chips.
    >> Interrupt latency is much improved but still not great - if the
    >> erase bursts are too short then the chip may never make any
    >> progress. The details vary between flash chips. This more
    >> advanced driver is currently only available from eCosCentric.
    
    Iris> So I think I could try to use the driver with interrupts
    Iris> left enabled and making sure that flash is not accessed from
    Iris> somewhere else, right? But when the machine is running it is
    Iris> absolutely necessary to keep the can bus working. Anything
    Iris> which has to do with flash (transferring a new file e.g.) is
    Iris> of lower priority, so your last proposition of the advanced
    Iris> AMD V2 driver sounds like a good solution, especially with
    Iris> using suspend/resume functionality. (Does one have to
    Iris> purchase it from eCosCentric?)

Yes. http://www.ecoscentric.com/cgi/info.cgi, or email
info@ecoscentric.com 

Bart

-- 
Bart Veer                                   eCos Configuration Architect
eCosCentric Limited    The eCos experts      http://www.ecoscentric.com/
Barnwell House, Barnwell Drive, Cambridge, UK.      Tel: +44 1223 245571
Registered in England and Wales: Reg No 4422071.
       >>>> Visit us at ESC-UK  http://www.embedded.co.uk <<<<
       >>>> Oct 7-8 on Stand 433 at FIVE ISC, Farnborough <<<<

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss