[ECOS] Re: FTP runs out of JFFS2 nodes and trashes the file system

[ECOS] Re: FTP runs out of JFFS2 nodes and trashes the file system

[Jürgen Lambrecht]

Hello,

I have the problem that I cannot FTP more than 200 files at once to my 
board. It always fails with this message:

  arena=301672, uordblks=30728, fordblks=270924, maxfree=261196.
  arena=301672, uordblks=41160, fordblks=260492, maxfree=260188.
  arena=301672, uordblks=44808, fordblks=256844, maxfree=252148.
  arena=301672, uordblks=46000, fordblks=255652, maxfree=252148.
  arena=301672, uordblks=49176, fordblks=252476, maxfree=240948.
  Unable to allocate raw_node_ref
    count=16000
  Unable to allocate raw_node_ref
    count=16000
  Unable to allocate raw_node_ref
    count=16000
  arena=301672, uordblks=47552, fordblks=254100, maxfree=245340.

- My raw node data structures (jffs2_raw_node_ref) are of course 
statically allocated - CYGNUM_FS_JFFS2_RAW_NODE_REF_CACHE_POOL_SIZE is 
set to 16000. (This needs 250kB; the user can use 55MB, so with 4kB 
nodes, I have 1920 free nodes for smaller nodes at the end of a file.)
  Compression is disabled (as I mainly store mp3 files). I put the 
CYGNUM_IO_FILEIO_MAX_INODE_CACHE_DEAD = 0.

- In jffs2/../malloc-ecos.c I added a counter as you can see in the log 
above.
  In this test, after FTP'ing 189 files with in total 3706kB, the file 
system crashes, but in a "valid" way: indeed, as the counter proofs, the 
raw node pool is empty, *but how is this possible?*
Therefore my question about FTP in 
http://ecos.sourceware.org/ml/ecos-discuss/2008-04/msg00110.html.
  I have to unmount/mount or reboot to be able to delete files again. If 
I then don't delete but add files instead, it fails with the same error 
after a few files. When I repeat this cycle of reboot-add files a few 
times (depending on previous state of the file system) *jffs2 
"crashes"*: I cannot anymore delete or add any file - listing 
directories still works, also the application still runs.
I have to format the flash to solve the problem.

- When I do the same test with TFTP, I can put over 1000 files in 1 
directory (and then TFTP times out because jffs2 becomes too slow with 
so many files in the directory).
Mark: the test is reproducible, independent of the SW and HW.

- I use now the flash-v2 driver to avoid problems with the main cvs tree 
non-interrupt save driver.
  But I read this in the documentation: "However the library may not be 
interrupt safe. An interrupt must not cause execution of code that is 
resident in FLASH." If I understand it well, *this means the library is 
thread safe on the condition that all code is always executed in RAM?*
This is ok in my case.

- Our ftp server code is based on the Minimal FTP server from Matthias 
Wandel. After a TCP connect, we just have a loop that calls: 
recv(socket,buf,...); fwrite(buf,...);.
I use FTP client in Total Commander.

- My board has a 32MHz AT91M55800A ARM7TDMI processor, with external 1MB 
SRAM and 64MB flash of which 61MB is in a jffs2 partition. Please don't 
remind me that my amount of SRAM is not enough ;-). But as you can see 
in the log, I still have enough heap.

- Maybe this is a *timing issue*, as jffs2 needs more time the more full 
the file system gets?

Kind regards,
Juergen


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Re: FTP runs out of JFFS2 nodes and trashes the file system

[Andrew Lunn]

>  In this test, after FTP'ing 189 files with in total 3706kB, the file  
> system crashes, but in a "valid" way: indeed, as the counter proofs, the  
> raw node pool is empty, *but how is this possible?*

Jffs2 keeps an in memory copy of the FS meta information. Each write
to the FS results in the creating of a node. Smaller writes results in
more nodes/meta information and so greater RAM usage for the meta
information. Hence you can get into the state of being out of RAM for
meta information but still have space on the filesystem. So i would
suggest writing bigger blocks, use fwrite, not write and use setbuf to
set the buffering to 4K or similar. 

> Therefore my question about FTP in  
> http://ecos.sourceware.org/ml/ecos-discuss/2008-04/msg00110.html.
>  I have to unmount/mount or reboot to be able to delete files again. If  
> I then don't delete but add files instead, it fails with the same error  
> after a few files. When I repeat this cycle of reboot-add files a few  
> times (depending on previous state of the file system) *jffs2  
> "crashes"*: I cannot anymore delete or add any file - listing  
> directories still works, also the application still runs.
> I have to format the flash to solve the problem.

There are a couple of possibilities here. 

umount/mount causes it to rebuild its in RAM meta information which
allows it to remove old nodes which are no longer needed.

It has chance to do a garbage collect of the filesystem so freeing up
some meta information and blocks on the FS.

Note that JFFS2 has unusual behaviour when full. It needs 4 to 5 free
erase blocks in order to do garbage collection. So things like df will
say there is free space, but the filesystem may refuse to write
because garbage collection does not have enough blocks to be able to
run.

>  But I read this in the documentation: "However the library may not be  
> interrupt safe. An interrupt must not cause execution of code that is  
> resident in FLASH." If I understand it well, *this means the library is  
> thread safe on the condition that all code is always executed in RAM?*
> This is ok in my case.

Just watch out for redboot and the virtual vectors. If you are using a
ROM redboot, diag_printf can jump into the ROM redboot. Similar an
exception, or using the debugger could cause the gdb stub in the ROM
redboot to get called. 

        Andrew

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Re: FTP runs out of JFFS2 nodes and trashes the file system

[Jürgen Lambrecht]

Andrew Lunn wrote:

>> In this test, after FTP'ing 189 files with in total 3706kB, the file  
>>system crashes, but in a "valid" way: indeed, as the counter proofs, the  
>>raw node pool is empty, *but how is this possible?*
>>    
>>
>
>Jffs2 keeps an in memory copy of the FS meta information. Each write
>to the FS results in the creating of a node. Smaller writes results in
>more nodes/meta information and so greater RAM usage for the meta
>information. Hence you can get into the state of being out of RAM for
>meta information but still have space on the filesystem. So i would
>suggest writing bigger blocks, use fwrite, not write and use setbuf to
>set the buffering to 4K or similar. 
>  
>
I already use fwrite, but I'll try 'setbuf'.
But Gary says 
(http://ecos.sourceware.org/ml/ecos-discuss/2008-04/msg00102.html) that 
jffs2 automatically buffers data to fill a node, and I read yesterday 
something about that on infradead.org ???
But I must say: we adapted the eCos TFTP server to buffer 8 512B packets 
to have 4kB per write and with TFTP there is no problem with running out 
of raw nodes.

But there is something more wrong: I have 61MB in flash and 16000 
(statically allocated) raw nodes available.
3706kB/16000 = 237.184 Bytes per node. (ok, a bit more, because I have 
have directories /ERR, /AUDIO/PRM, /AUDIO/JINGLE; and /ERR always 
contains a log file of a few kB).
Is this really possible: only approx. 300B per node? I specially 
replaced my flash chip with a new one on my test board, created the 
directories in the jffs2 partition and then started FTP.
With Ethereal I see that a transfer always has the same behavior, the 
data is transferred with packets of this size in Bytes: 512, 1448, 88, 
and then max. sized 1448 B packets until the complete file is 
transferred. There is 1 strange thing: all data packets have this 
comment in Ethereal: "TCP segment of a reassembled PDU". I don't 
understand why because there is no IP fragmentation (don't fragment bit 
is on, and there are no fragments).
Of course, with TCP, there is no concept of packets anymore: I just do 
recv(socket,buf..);fwrite(buf..);

I will now add debugging to my code to see the actual size of fwrite().
I will also check the mounting - set jffs2 debug to 2 - to see the 
scanning of all nodes.

>>Therefore my question about FTP in  
>>http://ecos.sourceware.org/ml/ecos-discuss/2008-04/msg00110.html.
>> I have to unmount/mount or reboot to be able to delete files again. If  
>>I then don't delete but add files instead, it fails with the same error  
>>after a few files. When I repeat this cycle of reboot-add files a few  
>>times (depending on previous state of the file system) *jffs2  
>>"crashes"*: I cannot anymore delete or add any file - listing  
>>directories still works, also the application still runs.
>>I have to format the flash to solve the problem.
>>    
>>
>
>There are a couple of possibilities here. 
>
>umount/mount causes it to rebuild its in RAM meta information which
>allows it to remove old nodes which are no longer needed.
>
>It has chance to do a garbage collect of the filesystem so freeing up
>some meta information and blocks on the FS.
>  
>
indeed that's the case

>Note that JFFS2 has unusual behaviour when full. It needs 4 to 5 free
>erase blocks in order to do garbage collection. So things like df will
>say there is free space, but the filesystem may refuse to write
>because garbage collection does not have enough blocks to be able to
>run.
>
>  
>
that should not be the case at all: I still have 57 MB free! But I'll 
check it.

>> But I read this in the documentation: "However the library may not be  
>>interrupt safe. An interrupt must not cause execution of code that is  
>>resident in FLASH." If I understand it well, *this means the library is  
>>thread safe on the condition that all code is always executed in RAM?*
>>This is ok in my case.
>>    
>>
>
>Just watch out for redboot and the virtual vectors. If you are using a
>ROM redboot, diag_printf can jump into the ROM redboot. Similar an
>exception, or using the debugger could cause the gdb stub in the ROM
>redboot to get called. 
>
>        Andrew
>
>  
>
As far as I know that is OK: my ROM redboot jumps to the application 
ROMRAM binary, and ROMRAM ecos sets 
CYGSEM_HAL_VIRTUAL_VECTOR_INIT_WHOLE_TABLE to 1 (I checked). So the 
diag_printft in the VVT is overwritten by the RAM version I guess?
And I don't use gdb stubs, they are even disabled to save code size.

Kind regards,
Jürgen

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Re: FTP runs out of JFFS2 nodes and trashes the file system

[Andrew Lunn]

> But Gary says  
> (http://ecos.sourceware.org/ml/ecos-discuss/2008-04/msg00102.html) that  
> jffs2 automatically buffers data to fill a node, and I read yesterday  
> something about that on infradead.org ???

I don't think that is correct for eCos. On Linux the VFS maybe
blocking together writes, but for eCos there is no VFS. Writes go
directly to the filesystem. Enabling debugging may prove this.

         Andrew

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Re: FTP runs out of JFFS2 nodes and trashes the file system

[Jürgen Lambrecht]

Andrew Lunn wrote:

>>But Gary says  
>>(http://ecos.sourceware.org/ml/ecos-discuss/2008-04/msg00102.html) that  
>>jffs2 automatically buffers data to fill a node, and I read yesterday  
>>something about that on infradead.org ???
>>    
>>
>
>I don't think that is correct for eCos. On Linux the VFS maybe
>blocking together writes, but for eCos there is no VFS. Writes go
>directly to the filesystem. Enabling debugging may prove this.
>
>         Andrew
>
>  
>
Here are my debug results:

- When looking to the jffs2 debug output, I see there are nodes with a 
huge number of versions: 326, 628... . Each time of size 0x100 or 256B.
- I have problems with printf... tomorrow hopefully more...

Kind regards,
Jürgen

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Re: FTP runs out of JFFS2 nodes and trashes the file system

[Jürgen Lambrecht]

Cause of error found! See below.

Andrew Lunn wrote:

>>But Gary says  
>>(http://ecos.sourceware.org/ml/ecos-discuss/2008-04/msg00102.html) that  
>>jffs2 automatically buffers data to fill a node, and I read yesterday  
>>something about that on infradead.org ???
>>    
>>
>
>I don't think that is correct for eCos. On Linux the VFS maybe
>blocking together writes, but for eCos there is no VFS. Writes go
>directly to the filesystem. Enabling debugging may prove this.
>
>         Andrew
>
>  
>
Here are my debug results:

- *jffs2 mount* debug output:
   I see there are nodes with a huge number of versions: 326, 628... . 
Each time of size 0x100 or 256B. The smallest versions are 100!
- *FTP* debug output:
   I see the buffer size is 2kB, and fwrite() is called per 2kB (untill 
the file is finished)
   Solution: I need to set the buffer size to 4kB to fit the jffs2 node 
size. I  guess this is a TCP/IP stack buffer size I have to set with ioctl()?
- *libc fwrite*:
   I see the data is written per 256B!! OK: the default value of 
CYGNUM_LIBC_STDIO_BUFSIZE is 256.
   Solution: set buffer size with setbuf() to 4kB
- *jffs2_fo_write*
   Gets a request to allocate 256B, request a minimum of 196B, and gets 
much more.
   Writes a dnode of 256B.
   So indeed, jffs2 does not buffer to have the optimal node size.
   So this way, my 16000 raw nodes (costing 250kB RAM) can only store 4MB 
instead of 62.5MB!! This explains my problem with FTP.

Conclusion:
To use a flash with jffs2 efficiently
- users should be aware they must call write with the jffs2 buffer size
- users should be aware they must set the stream libc buffer size to the 
jffs2 buffer size before using fwrite!
   -> even better: the default value of CYGNUM_LIBC_STDIO_BUFSIZE should 
be set to the jffs2 buffer size if jffs2 is used. OK, if you use 2 
different file systems this is not valid anymore...

Anyhow, ecos misses a good jffs2 usage manual.
If I want to write one, how should I do that? I don't know sgml.. OK, my 
xemacs knows it ;-).

Thanks for your help!,
Jürgen


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss