* [ECOS] Re: FTP runs out of JFFS2 nodes and trashes the file system
[not found] <47FA32AC.4070705@televic.com>
@ 2008-04-07 23:22 ` Jürgen Lambrecht
2008-04-08 6:48 ` Andrew Lunn
0 siblings, 1 reply; 6+ messages in thread
From: Jürgen Lambrecht @ 2008-04-07 23:22 UTC (permalink / raw)
To: ecos-discuss
Hello,
I have the problem that I cannot FTP more than 200 files at once to my
board. It always fails with this message:
arena=301672, uordblks=30728, fordblks=270924, maxfree=261196.
arena=301672, uordblks=41160, fordblks=260492, maxfree=260188.
arena=301672, uordblks=44808, fordblks=256844, maxfree=252148.
arena=301672, uordblks=46000, fordblks=255652, maxfree=252148.
arena=301672, uordblks=49176, fordblks=252476, maxfree=240948.
Unable to allocate raw_node_ref
count=16000
Unable to allocate raw_node_ref
count=16000
Unable to allocate raw_node_ref
count=16000
arena=301672, uordblks=47552, fordblks=254100, maxfree=245340.
- My raw node data structures (jffs2_raw_node_ref) are of course
statically allocated - CYGNUM_FS_JFFS2_RAW_NODE_REF_CACHE_POOL_SIZE is
set to 16000. (This needs 250kB; the user can use 55MB, so with 4kB
nodes, I have 1920 free nodes for smaller nodes at the end of a file.)
Compression is disabled (as I mainly store mp3 files). I put the
CYGNUM_IO_FILEIO_MAX_INODE_CACHE_DEAD = 0.
- In jffs2/../malloc-ecos.c I added a counter as you can see in the log
above.
In this test, after FTP'ing 189 files with in total 3706kB, the file
system crashes, but in a "valid" way: indeed, as the counter proofs, the
raw node pool is empty, *but how is this possible?*
Therefore my question about FTP in
http://ecos.sourceware.org/ml/ecos-discuss/2008-04/msg00110.html.
I have to unmount/mount or reboot to be able to delete files again. If
I then don't delete but add files instead, it fails with the same error
after a few files. When I repeat this cycle of reboot-add files a few
times (depending on previous state of the file system) *jffs2
"crashes"*: I cannot anymore delete or add any file - listing
directories still works, also the application still runs.
I have to format the flash to solve the problem.
- When I do the same test with TFTP, I can put over 1000 files in 1
directory (and then TFTP times out because jffs2 becomes too slow with
so many files in the directory).
Mark: the test is reproducible, independent of the SW and HW.
- I use now the flash-v2 driver to avoid problems with the main cvs tree
non-interrupt save driver.
But I read this in the documentation: "However the library may not be
interrupt safe. An interrupt must not cause execution of code that is
resident in FLASH." If I understand it well, *this means the library is
thread safe on the condition that all code is always executed in RAM?*
This is ok in my case.
- Our ftp server code is based on the Minimal FTP server from Matthias
Wandel. After a TCP connect, we just have a loop that calls:
recv(socket,buf,...); fwrite(buf,...);.
I use FTP client in Total Commander.
- My board has a 32MHz AT91M55800A ARM7TDMI processor, with external 1MB
SRAM and 64MB flash of which 61MB is in a jffs2 partition. Please don't
remind me that my amount of SRAM is not enough ;-). But as you can see
in the log, I still have enough heap.
- Maybe this is a *timing issue*, as jffs2 needs more time the more full
the file system gets?
Kind regards,
Juergen
--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [ECOS] Re: FTP runs out of JFFS2 nodes and trashes the file system
2008-04-07 23:22 ` [ECOS] Re: FTP runs out of JFFS2 nodes and trashes the file system Jürgen Lambrecht
@ 2008-04-08 6:48 ` Andrew Lunn
2008-04-08 8:41 ` Jürgen Lambrecht
0 siblings, 1 reply; 6+ messages in thread
From: Andrew Lunn @ 2008-04-08 6:48 UTC (permalink / raw)
To: j.lambrecht; +Cc: ecos-discuss
> In this test, after FTP'ing 189 files with in total 3706kB, the file
> system crashes, but in a "valid" way: indeed, as the counter proofs, the
> raw node pool is empty, *but how is this possible?*
Jffs2 keeps an in memory copy of the FS meta information. Each write
to the FS results in the creating of a node. Smaller writes results in
more nodes/meta information and so greater RAM usage for the meta
information. Hence you can get into the state of being out of RAM for
meta information but still have space on the filesystem. So i would
suggest writing bigger blocks, use fwrite, not write and use setbuf to
set the buffering to 4K or similar.
> Therefore my question about FTP in
> http://ecos.sourceware.org/ml/ecos-discuss/2008-04/msg00110.html.
> I have to unmount/mount or reboot to be able to delete files again. If
> I then don't delete but add files instead, it fails with the same error
> after a few files. When I repeat this cycle of reboot-add files a few
> times (depending on previous state of the file system) *jffs2
> "crashes"*: I cannot anymore delete or add any file - listing
> directories still works, also the application still runs.
> I have to format the flash to solve the problem.
There are a couple of possibilities here.
umount/mount causes it to rebuild its in RAM meta information which
allows it to remove old nodes which are no longer needed.
It has chance to do a garbage collect of the filesystem so freeing up
some meta information and blocks on the FS.
Note that JFFS2 has unusual behaviour when full. It needs 4 to 5 free
erase blocks in order to do garbage collection. So things like df will
say there is free space, but the filesystem may refuse to write
because garbage collection does not have enough blocks to be able to
run.
> But I read this in the documentation: "However the library may not be
> interrupt safe. An interrupt must not cause execution of code that is
> resident in FLASH." If I understand it well, *this means the library is
> thread safe on the condition that all code is always executed in RAM?*
> This is ok in my case.
Just watch out for redboot and the virtual vectors. If you are using a
ROM redboot, diag_printf can jump into the ROM redboot. Similar an
exception, or using the debugger could cause the gdb stub in the ROM
redboot to get called.
Andrew
--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [ECOS] Re: FTP runs out of JFFS2 nodes and trashes the file system
2008-04-08 6:48 ` Andrew Lunn
@ 2008-04-08 8:41 ` Jürgen Lambrecht
2008-04-08 10:32 ` Andrew Lunn
0 siblings, 1 reply; 6+ messages in thread
From: Jürgen Lambrecht @ 2008-04-08 8:41 UTC (permalink / raw)
To: Andrew Lunn, Gary Thomas; +Cc: ecos-discuss
Andrew Lunn wrote:
>> In this test, after FTP'ing 189 files with in total 3706kB, the file
>>system crashes, but in a "valid" way: indeed, as the counter proofs, the
>>raw node pool is empty, *but how is this possible?*
>>
>>
>
>Jffs2 keeps an in memory copy of the FS meta information. Each write
>to the FS results in the creating of a node. Smaller writes results in
>more nodes/meta information and so greater RAM usage for the meta
>information. Hence you can get into the state of being out of RAM for
>meta information but still have space on the filesystem. So i would
>suggest writing bigger blocks, use fwrite, not write and use setbuf to
>set the buffering to 4K or similar.
>
>
I already use fwrite, but I'll try 'setbuf'.
But Gary says
(http://ecos.sourceware.org/ml/ecos-discuss/2008-04/msg00102.html) that
jffs2 automatically buffers data to fill a node, and I read yesterday
something about that on infradead.org ???
But I must say: we adapted the eCos TFTP server to buffer 8 512B packets
to have 4kB per write and with TFTP there is no problem with running out
of raw nodes.
But there is something more wrong: I have 61MB in flash and 16000
(statically allocated) raw nodes available.
3706kB/16000 = 237.184 Bytes per node. (ok, a bit more, because I have
have directories /ERR, /AUDIO/PRM, /AUDIO/JINGLE; and /ERR always
contains a log file of a few kB).
Is this really possible: only approx. 300B per node? I specially
replaced my flash chip with a new one on my test board, created the
directories in the jffs2 partition and then started FTP.
With Ethereal I see that a transfer always has the same behavior, the
data is transferred with packets of this size in Bytes: 512, 1448, 88,
and then max. sized 1448 B packets until the complete file is
transferred. There is 1 strange thing: all data packets have this
comment in Ethereal: "TCP segment of a reassembled PDU". I don't
understand why because there is no IP fragmentation (don't fragment bit
is on, and there are no fragments).
Of course, with TCP, there is no concept of packets anymore: I just do
recv(socket,buf..);fwrite(buf..);
I will now add debugging to my code to see the actual size of fwrite().
I will also check the mounting - set jffs2 debug to 2 - to see the
scanning of all nodes.
>>Therefore my question about FTP in
>>http://ecos.sourceware.org/ml/ecos-discuss/2008-04/msg00110.html.
>> I have to unmount/mount or reboot to be able to delete files again. If
>>I then don't delete but add files instead, it fails with the same error
>>after a few files. When I repeat this cycle of reboot-add files a few
>>times (depending on previous state of the file system) *jffs2
>>"crashes"*: I cannot anymore delete or add any file - listing
>>directories still works, also the application still runs.
>>I have to format the flash to solve the problem.
>>
>>
>
>There are a couple of possibilities here.
>
>umount/mount causes it to rebuild its in RAM meta information which
>allows it to remove old nodes which are no longer needed.
>
>It has chance to do a garbage collect of the filesystem so freeing up
>some meta information and blocks on the FS.
>
>
indeed that's the case
>Note that JFFS2 has unusual behaviour when full. It needs 4 to 5 free
>erase blocks in order to do garbage collection. So things like df will
>say there is free space, but the filesystem may refuse to write
>because garbage collection does not have enough blocks to be able to
>run.
>
>
>
that should not be the case at all: I still have 57 MB free! But I'll
check it.
>> But I read this in the documentation: "However the library may not be
>>interrupt safe. An interrupt must not cause execution of code that is
>>resident in FLASH." If I understand it well, *this means the library is
>>thread safe on the condition that all code is always executed in RAM?*
>>This is ok in my case.
>>
>>
>
>Just watch out for redboot and the virtual vectors. If you are using a
>ROM redboot, diag_printf can jump into the ROM redboot. Similar an
>exception, or using the debugger could cause the gdb stub in the ROM
>redboot to get called.
>
> Andrew
>
>
>
As far as I know that is OK: my ROM redboot jumps to the application
ROMRAM binary, and ROMRAM ecos sets
CYGSEM_HAL_VIRTUAL_VECTOR_INIT_WHOLE_TABLE to 1 (I checked). So the
diag_printft in the VVT is overwritten by the RAM version I guess?
And I don't use gdb stubs, they are even disabled to save code size.
Kind regards,
Jürgen
--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [ECOS] Re: FTP runs out of JFFS2 nodes and trashes the file system
2008-04-08 8:41 ` Jürgen Lambrecht
@ 2008-04-08 10:32 ` Andrew Lunn
2008-04-08 15:58 ` Jürgen Lambrecht
2008-04-09 11:42 ` Jürgen Lambrecht
0 siblings, 2 replies; 6+ messages in thread
From: Andrew Lunn @ 2008-04-08 10:32 UTC (permalink / raw)
To: J?rgen Lambrecht; +Cc: Andrew Lunn, Gary Thomas, ecos-discuss
> But Gary says
> (http://ecos.sourceware.org/ml/ecos-discuss/2008-04/msg00102.html) that
> jffs2 automatically buffers data to fill a node, and I read yesterday
> something about that on infradead.org ???
I don't think that is correct for eCos. On Linux the VFS maybe
blocking together writes, but for eCos there is no VFS. Writes go
directly to the filesystem. Enabling debugging may prove this.
Andrew
--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [ECOS] Re: FTP runs out of JFFS2 nodes and trashes the file system
2008-04-08 10:32 ` Andrew Lunn
@ 2008-04-08 15:58 ` Jürgen Lambrecht
2008-04-09 11:42 ` Jürgen Lambrecht
1 sibling, 0 replies; 6+ messages in thread
From: Jürgen Lambrecht @ 2008-04-08 15:58 UTC (permalink / raw)
To: Andrew Lunn; +Cc: Gary Thomas, ecos-discuss
Andrew Lunn wrote:
>>But Gary says
>>(http://ecos.sourceware.org/ml/ecos-discuss/2008-04/msg00102.html) that
>>jffs2 automatically buffers data to fill a node, and I read yesterday
>>something about that on infradead.org ???
>>
>>
>
>I don't think that is correct for eCos. On Linux the VFS maybe
>blocking together writes, but for eCos there is no VFS. Writes go
>directly to the filesystem. Enabling debugging may prove this.
>
> Andrew
>
>
>
Here are my debug results:
- When looking to the jffs2 debug output, I see there are nodes with a
huge number of versions: 326, 628... . Each time of size 0x100 or 256B.
- I have problems with printf... tomorrow hopefully more...
Kind regards,
Jürgen
--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [ECOS] Re: FTP runs out of JFFS2 nodes and trashes the file system
2008-04-08 10:32 ` Andrew Lunn
2008-04-08 15:58 ` Jürgen Lambrecht
@ 2008-04-09 11:42 ` Jürgen Lambrecht
1 sibling, 0 replies; 6+ messages in thread
From: Jürgen Lambrecht @ 2008-04-09 11:42 UTC (permalink / raw)
To: Andrew Lunn, Gary Thomas, ecos-discuss
Cause of error found! See below.
Andrew Lunn wrote:
>>But Gary says
>>(http://ecos.sourceware.org/ml/ecos-discuss/2008-04/msg00102.html) that
>>jffs2 automatically buffers data to fill a node, and I read yesterday
>>something about that on infradead.org ???
>>
>>
>
>I don't think that is correct for eCos. On Linux the VFS maybe
>blocking together writes, but for eCos there is no VFS. Writes go
>directly to the filesystem. Enabling debugging may prove this.
>
> Andrew
>
>
>
Here are my debug results:
- *jffs2 mount* debug output:
I see there are nodes with a huge number of versions: 326, 628... .
Each time of size 0x100 or 256B. The smallest versions are 100!
- *FTP* debug output:
I see the buffer size is 2kB, and fwrite() is called per 2kB (untill
the file is finished)
Solution: I need to set the buffer size to 4kB to fit the jffs2 node
size. I guess this is a TCP/IP stack buffer size I have to set with ioctl()?
- *libc fwrite*:
I see the data is written per 256B!! OK: the default value of
CYGNUM_LIBC_STDIO_BUFSIZE is 256.
Solution: set buffer size with setbuf() to 4kB
- *jffs2_fo_write*
Gets a request to allocate 256B, request a minimum of 196B, and gets
much more.
Writes a dnode of 256B.
So indeed, jffs2 does not buffer to have the optimal node size.
So this way, my 16000 raw nodes (costing 250kB RAM) can only store 4MB
instead of 62.5MB!! This explains my problem with FTP.
Conclusion:
To use a flash with jffs2 efficiently
- users should be aware they must call write with the jffs2 buffer size
- users should be aware they must set the stream libc buffer size to the
jffs2 buffer size before using fwrite!
-> even better: the default value of CYGNUM_LIBC_STDIO_BUFSIZE should
be set to the jffs2 buffer size if jffs2 is used. OK, if you use 2
different file systems this is not valid anymore...
Anyhow, ecos misses a good jffs2 usage manual.
If I want to write one, how should I do that? I don't know sgml.. OK, my
xemacs knows it ;-).
Thanks for your help!,
Jürgen
--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2008-04-09 9:46 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <47FA32AC.4070705@televic.com>
2008-04-07 23:22 ` [ECOS] Re: FTP runs out of JFFS2 nodes and trashes the file system Jürgen Lambrecht
2008-04-08 6:48 ` Andrew Lunn
2008-04-08 8:41 ` Jürgen Lambrecht
2008-04-08 10:32 ` Andrew Lunn
2008-04-08 15:58 ` Jürgen Lambrecht
2008-04-09 11:42 ` Jürgen Lambrecht
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).