RE: [ECOS] JFFS2 Access and interrupts

RE: [ECOS] JFFS2 Access and interrupts

[Zimman, Chris]

Most likely, it's not really a problem with JFFS2 itself, but rather
with the underlying flash driver as they're not interrupt safe.  eCosPro
has newer versions of the flash drivers that are interrupt safe.
 
--Chris

-----Original Message-----
From: ecos-discuss-owner@ecos.sourceware.org
[mailto:ecos-discuss-owner@ecos.sourceware.org] On Behalf Of Gary Thomas
Sent: Wednesday, September 06, 2006 8:32 PM
To: Gernot Zankl
Cc: ecos-discuss@sources.redhat.com
Subject: Re: [ECOS] JFFS2 Access and interrupts

Gernot Zankl wrote:
> Hi all,
> 
> while writing a file in a jffs2 filesystem (by means of open, write,
> close)
> my application seems to hang in function write(). 
> After guarding the write access with
> HA_DISABLE_INTERRUPTS/HAL_RESTORES_INTERRUPTS 
> the write accesses worked properly.
> 
> After a short inspection of flash.c (in packages/io/flash/current/src)
> it seems that in my port these guards are already present,
> placed around the "__anonymizer"-ed flash access functions.
> 
> Is there a general rule, how to handle mutual IRQs and flash access ?
> Do I miss something ?
> Is there a configuration option, I've forgotten to enable ?

There aren't any configuration options for this - it's assumed to work.

Can you please explain in more detail exactly what you've done to get
things to work?  Given that the generic functions (e.g. flash_program)
already disable interrupts, I'm a bit confused.

> 
> Kind regards,
> Gernot Zankl
> 
> Btw: PowerPC MPC5200B and external flash Spansion S29GL064M

Given that the eCos port for the MPC5200 is not public (you got it
from Analogue & Micro), it makes sense to ask these questions of
A&M, not the general list.


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] JFFS2 Access and interrupts

[Bart Veer]

>>>>> "Gernot" == Gernot Zankl <zankl@decomsys.com> writes:

    >> > FYI I finally found the/our bug, causing the application to
    >> > corrupt the jffs2. We just used the wrong block size in
    >> > the configuration of the underlying flash driver, causing to
    >> > erase too much flash memory (and destroying still active inodes).
    >> 
    >> FAOD - it had *nothing* to do with interrupts during FLASH operations?

    Gernot> At least not in "our" version of the flash driver (v1)
    Gernot> which has been provided and modified by Analogue&Micro,
    Gernot> where all calls to the critical functions are guarded by
    Gernot> the HAL_DISABLE_INTERRUPTS and HAL_RESTORES_INTERRUPTS
    Gernot> macros.

Of course that approach gives you an interrupt latency of somewhere
from several 100 milliseconds up to 5 seconds, depending on the exact
flash hardware and the environmental conditions. That is the time
needed to erase a whole flash sector. So any time you perform jffs2
file I/O your application may become completely unresponsive to all
external events for a number of seconds. It depends on the application
whether or not such behaviour is acceptable.

Bart

-- 
Bart Veer                       eCos Configuration Architect
http://www.ecoscentric.com/     The eCos and RedBoot experts

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] JFFS2 Access and interrupts

[Gary Thomas]

Gernot Zankl wrote:
>  
> Gary Thomas wrote:
>> Gernot Zankl wrote:
>>> Hi all !
>>>
>>> Thanks for your answers.
>>> FYI I finally found the/our bug, causing the application to 
>>> corrupt the jffs2. We just used the wrong block size in
>>> the configuration of the underlying flash driver, causing to
>>> erase too much flash memory (and destroying still active inodes).
>> FAOD - it had *nothing* to do with interrupts during FLASH operations?
> 
> At least not in "our" version of the flash driver (v1) which has been 
> provided and modified by Analogue&Micro, where all calls to the 
> critical functions are guarded by the HAL_DISABLE_INTERRUPTS and
> HAL_RESTORES_INTERRUPTS macros.

Good, thanks.

> 
> Btw. what does "FAOD" mean, "Fatty acid oxidation disorder" ?
> 

For Avoidance Of Doubt (it's a common British saying which I also
had to have explained to me many years ago...)


-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

RE: [ECOS] JFFS2 Access and interrupts

[Gernot Zankl]

 
Gary Thomas wrote:
> Gernot Zankl wrote:
> > Hi all !
> > 
> > Thanks for your answers.
> > FYI I finally found the/our bug, causing the application to 
> > corrupt the jffs2. We just used the wrong block size in
> > the configuration of the underlying flash driver, causing to
> > erase too much flash memory (and destroying still active inodes).
> 
> FAOD - it had *nothing* to do with interrupts during FLASH operations?

At least not in "our" version of the flash driver (v1) which has been 
provided and modified by Analogue&Micro, where all calls to the 
critical functions are guarded by the HAL_DISABLE_INTERRUPTS and
HAL_RESTORES_INTERRUPTS macros.

Regards,
Gernot Zankl

Btw. what does "FAOD" mean, "Fatty acid oxidation disorder" ?



--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] JFFS2 Access and interrupts

[Gary Thomas]

Gernot Zankl wrote:
> Hi all !
> 
> Thanks for your answers.
> FYI I finally found the/our bug, causing the application to 
> corrupt the jffs2. We just used the wrong block size in
> the configuration of the underlying flash driver, causing to
> erase too much flash memory (and destroying still active inodes).

FAOD - it had *nothing* to do with interrupts during FLASH operations?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

RE: [ECOS] JFFS2 Access and interrupts

[Gernot Zankl]

Hi all !

Thanks for your answers.
FYI I finally found the/our bug, causing the application to 
corrupt the jffs2. We just used the wrong block size in
the configuration of the underlying flash driver, causing to
erase too much flash memory (and destroying still active inodes).

Regards,
Gernot Zankl


--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] JFFS2 Access and interrupts

[Andrew Lunn]

On Fri, Sep 08, 2006 at 11:59:48AM +0200, Gernot Zankl wrote:
> Hi all !
> 
> > A basic rule of working with flash is that if any code is erasing or
> > programming part of the flash then no other code can access it. Some
> > flash chips relax that restriction somewhat by providing several
> > banks, but at best that is only a partial solution. Flash accesses can
> > come from a variety of sources.
> 
> I'm using ROMRAM versin of redboot and the applications runs in RAM,
> too.
> Are there any calls to the ROM version of the redboot during the
> execution
> of a "normal" application, which would cause reading access to the flash
> ?

There shoudn't be. But it is worth checking the interrupt vectors and
VV are pointing to RAM addresses not ROM addresses. It could be the
ROMRAM startup code is not quite right and its still using ROM
addresses. Such a system might run, but experiance problems with
interrupts during flash handling.

> I want to give the v2 version a try. As far as i've seen, there exists a
> flas_v2 branch in the CVS.
> Are the modules devs/flash, io/flash and fs/jffs2 the only ones, I've to
> upgrade ?

If your problem is in Redboot, you might also need the redboot
directory from the branch.

> I built a test application (heavily based on one of the jffs2 tests),
> where
> cyclically files (with size f) are allocated, filed with rand()-values, 
> crc-checked and finally erased. 
> Starting with a fresh partition (all filled with 0xff) of size S, 
> it seems that after a number of iterations i, jffs2 produces a lot of
> warnings
> (see below), where always approx. i * f = S.
> The test continues, but most of the time hangs some iterations later.

It might be interesting to run this on the synthetic target. It has no
interrupts as such, so you can test if this is a jffs2 problem, not a
flash problem.

           Andrew            

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

RE: [ECOS] JFFS2 Access and interrupts

[Gernot Zankl]

Hi all !

> A basic rule of working with flash is that if any code is erasing or
> programming part of the flash then no other code can access it. Some
> flash chips relax that restriction somewhat by providing several
> banks, but at best that is only a partial solution. Flash accesses can
> come from a variety of sources.

I'm using ROMRAM versin of redboot and the applications runs in RAM,
too.
Are there any calls to the ROM version of the redboot during the
execution
of a "normal" application, which would cause reading access to the flash
?

>   3) some other thread may access the flash for some other reason,
>      either via the flash API or directly. Especially if the
>      application itself resides in flash.

The application is (currently) loaded from the tftp server and is
executed in
RAM. There exists just one jffs2 partition and it's the only one
mounted.

> Anoncvs also has a V2 flash development branch which addresses some of
> the problems. In particular it adds mutex locking for accessing flash
> devices, so if the only other flash accesses happen via the V2 flash
> API then you'll be ok. Again a ROMRAM RedBoot would be a very good
> idea.

I want to give the v2 version a try. As far as i've seen, there exists a
flas_v2 branch in the CVS.
Are the modules devs/flash, io/flash and fs/jffs2 the only ones, I've to
upgrade ?

I built a test application (heavily based on one of the jffs2 tests),
where
cyclically files (with size f) are allocated, filed with rand()-values, 
crc-checked and finally erased. 
Starting with a fresh partition (all filled with 0xff) of size S, 
it seems that after a number of iterations i, jffs2 produces a lot of
warnings
(see below), where always approx. i * f = S.
The test continues, but most of the time hangs some iterations later.

><INFO> Iteration 0000058: passed=0000058, failed=0000000
><5>JFFS2 notice:  read_unknown: node header CRC failed at %#08x. But it
must have been OK earlier.
><4>Node totlen on flash (0xffffffff) != totlen from node ref
(0x00001034)
><4>JFFS2 warning:  jffs2_do_read_inode_internal: no data nodes found
for ino #60
>jffs2_read_inode() failed
>jffs2_iget() failed for ino #60
><ERROR>: open() returned -1 I/O error
><ERROR>: read() returned -1 Bad file handle
>File <test0000058> is corrupt<ERROR>: read() returned -1 Bad file
handle
><4>JFFS2 warning:  jffs2_get_inode_nodes: no valid nodes for ino #60
><4>JFFS2 warning:  jffs2_do_read_inode_internal: no data nodes found
for ino #60
>jffs2_read_inode() failed
>jffs2_iget() failed for ino #60
><ERROR>: unlink() returned -1 I/O error
><4>Node CRC ffffffff != calculated CRC f09e7845 for node at 0003f610
><ERROR>: read() returned 3
>File <test0000059> is corrupt<INFO> Iteration 0000059: passed=0000058,
failed=0000007
><5>JFFS2 notice:  read_unknown: node header CRC failed at %#08x. But it
must have been OK earlier.
><4>Node totlen on flash (0xffffffff) != totlen from node ref
(0x000009f0)
>File <test0000059> is corrupt<4>Node totlen on flash (0xffffffff) !=
totlen from node ref (0x00000034)
><INFO> Iteration 0000060: passed=0000058, failed=0000008

Inbetween the aplication also produces:

><INFO> Iteration 0000207: passed=0000201, failed=0000024
><4>Node totlen on flash (0xffffffff) != totlen from node ref
(0x00000034)
><4>Node totlen on flash (0xffffffff) != totlen from node ref
(0x00000034)
><4>Node totlen on flash (0xffffffff) != totlen from node ref
(0x00000034)
><4>Node totlen on flash (0xffffffff) != totlen from node ref
(0x00000034)
><4>Node totlen on flash (0xffffffff) != totlen from node ref
(0x00000034)
><4>Node totlen on flash (0xffffffff) != totlen from node ref
(0x00000034)
><4>Node totlen on flash (0xffffffff) != totlen from node ref
(0x00000034)
><4>Node totlen on flash (0xffffffff) != totlen from node ref
(0x00000034)
><INFO> Iteration 0000208: passed=0000202, failed=0000024

Regards,
Gernot Zankl


--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] JFFS2 Access and interrupts

[Andrew Lunn]

>   1) I do not know too much about jffs2 internals (I leave that to
>      others) but I suspect that there is enough locking within jffs2
>      to prevent two threads from accessing the flash h/w concurrently
>      while performing file I/O.

I don't know JFFS2 too well either, but im sure with a single
filesystem, it will serialise access to the flash. When there are
multiple filesystems mounted from the same flash i've no idea if
simultanious access to different filesystems are serialised. It could
be than in the Linux world the MTD layer sorts this out?

   Andrew

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] JFFS2 Access and interrupts

[Bart Veer]

>>>>> "Gernot" == Gernot Zankl <zankl@decomsys.com> writes:

    Gernot> while writing a file in a jffs2 filesystem (by means of
    Gernot> open, write, close) my application seems to hang in
    Gernot> function write(). After guarding the write access with
    Gernot> HA_DISABLE_INTERRUPTS/HAL_RESTORES_INTERRUPTS the write
    Gernot> accesses worked properly.

    Gernot> After a short inspection of flash.c (in
    Gernot> packages/io/flash/current/src) it seems that in my port
    Gernot> these guards are already present, placed around the
    Gernot> "__anonymizer"-ed flash access functions.

    Gernot> Is there a general rule, how to handle mutual IRQs and
    Gernot> flash access ? Do I miss something ? Is there a
    Gernot> configuration option, I've forgotten to enable ?

A basic rule of working with flash is that if any code is erasing or
programming part of the flash then no other code can access it. Some
flash chips relax that restriction somewhat by providing several
banks, but at best that is only a partial solution. Flash accesses can
come from a variety of sources.

  1) I do not know too much about jffs2 internals (I leave that to
     others) but I suspect that there is enough locking within jffs2
     to prevent two threads from accessing the flash h/w concurrently
     while performing file I/O.

  2) If your board uses a RedBoot residing in flash, as opposed to a
     ROMRAM version, then the application may switch to RedBoot and
     hence execute from flash for a variety of reasons. The thread
     performing file I/O may get preempted and some other thread then
     calls printf(), diag_printf(), or anything similar which involves
     I/O via the virtual vector mechanism. Or the other thread may hit
     a breakpoint, activating the gdb stubs. Or there may be a virtual
     vector access to the fis or fconfig data. Or the code may throw
     an exception.

  3) some other thread may access the flash for some other reason,
     either via the flash API or directly. Especially if the
     application itself resides in flash.

The anoncvs trunk still uses the V1 API. Basically this does not
address any locking issues at all. Note that the default versions of
the HAL_FLASH_CACHES_...() macros only affect the cache, not
interrupts, so there is nothing to stop the current thread from being
interrupted or preempted. (Actually, manipulating the caches without
considering interrupts is itself rather dodgy). Possibly the platform
HAL for your port defines more advanced macros. There are some
scenarios where you may just be able to get away with this - a ROMRAM
RedBoot, absolutely nothing else using the flash - but it is not a
robust solution.

Anoncvs also has a V2 flash development branch which addresses some of
the problems. In particular it adds mutex locking for accessing flash
devices, so if the only other flash accesses happen via the V2 flash
API then you'll be ok. Again a ROMRAM RedBoot would be a very good
idea.

eCosCentric has done further work on the V2 flash drivers which has
not yet gone into anoncvs. This adds optional interrupt
disabling/reenabling where it is needed, right inside the drivers
themselves, together with support for program burst and for erase
suspend mode to limit the effect on real-time behaviour. There are
various configuration options so you can fine tune the drivers'
behaviour, basically trading off performance for interrupt response
times. With these drivers the application no longer needs to worry
about IRQs during flash accesses on non-SMP systems, it is all done
automatically.

Bart

-- 
Bart Veer                       eCos Configuration Architect
http://www.ecoscentric.com/     The eCos and RedBoot experts

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] JFFS2 Access and interrupts

[Gary Thomas]

Gernot Zankl wrote:
> Hi all,
> 
> while writing a file in a jffs2 filesystem (by means of open, write,
> close)
> my application seems to hang in function write(). 
> After guarding the write access with
> HA_DISABLE_INTERRUPTS/HAL_RESTORES_INTERRUPTS 
> the write accesses worked properly.
> 
> After a short inspection of flash.c (in packages/io/flash/current/src)
> it seems that in my port these guards are already present,
> placed around the "__anonymizer"-ed flash access functions.
> 
> Is there a general rule, how to handle mutual IRQs and flash access ?
> Do I miss something ?
> Is there a configuration option, I've forgotten to enable ?

There aren't any configuration options for this - it's assumed to work.

Can you please explain in more detail exactly what you've done to get
things to work?  Given that the generic functions (e.g. flash_program)
already disable interrupts, I'm a bit confused.

> 
> Kind regards,
> Gernot Zankl
> 
> Btw: PowerPC MPC5200B and external flash Spansion S29GL064M

Given that the eCos port for the MPC5200 is not public (you got it
from Analogue & Micro), it makes sense to ask these questions of
A&M, not the general list.


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[ECOS] JFFS2 Access and interrupts

[Gernot Zankl]

Hi all,

while writing a file in a jffs2 filesystem (by means of open, write,
close)
my application seems to hang in function write(). 
After guarding the write access with
HA_DISABLE_INTERRUPTS/HAL_RESTORES_INTERRUPTS 
the write accesses worked properly.

After a short inspection of flash.c (in packages/io/flash/current/src)
it seems that in my port these guards are already present,
placed around the "__anonymizer"-ed flash access functions.

Is there a general rule, how to handle mutual IRQs and flash access ?
Do I miss something ?
Is there a configuration option, I've forgotten to enable ?

Kind regards,
Gernot Zankl

Btw: PowerPC MPC5200B and external flash Spansion S29GL064M


--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss