[ECOS] Re: JFFS2 needs approx 72B of RAM per file

[ECOS] Re: JFFS2 needs approx 72B of RAM per file

[Jürgen Lambrecht]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=windows-1252; format=flowed, Size: 3506 bytes --]

Hello David,

I have missed your mail in some way (I moved to using a new laptop - had 
to use Outlook for a while...), sorry.
(I don't think my mail is a valid reply now because I copied it from the 
web - not following your mail rules ;-)

 > On Tue, 2008-04-08 at 01:14 +0200, JÃrgen Lambrecht wrote:
 > > Hello to ecos-discuss and especially linux-mtd,
 > >
 > > Tests have revealed to me that jffs2 does not only need raw nodes 
in RAM
 > > (I use CYGNUM_FS_JFFS2_RAW_NODE_REF_CACHE_POOL_SIZE to have this
 > > statically), but also "eats RAM" per file in flash.
 > > I want to remove this allocation, but I don't know if this is 
possible,
 > > what are the disadvantages....?
 > > (my basic problem is that I have not enough RAM on my board, but it is
 > > too late to change HW for the running projects)
 > >
 > > After reading some jffs2 documentation, I think that my problem is the
 > > inode cache?
 >
 > I don't think so -- I think the raw node references are likely to be the
 > major contributing factor. It should be easy enough to check though,
 > surely?
indeed
I used the "evict" function to clear the inode cache. Did not help.
 >
 > The problem, fundamentally, is that JFFS2 has no structure on the
 > medium. We have to _remember_ where every node (log entry) is.
Indeed, I understand now.
 >
 > I recently did some work to reduce the amount of memory per node from 16
 > to 8 bytes -- dropping a ->next pointer by using arrays of the things,
 > and dropping the 'length' field by observing that it's actually
 > redundant with 'offset' and '->next->offset' 99% of the time, and
 > finding tricks to make it redundant _100%_ of the time. If you're not
 > already using the latest JFFS2 code, it's worth doing that.
 >
 > I say "recently" -- it's been over the last two years or so, but I don't
 > think eCos is using the results yet. It'd be good if someone would take
 > responsibility for merging newer JFFS2 code into eCos (or just tell me
 > it's not going to happen, and I'll stop trying to keep it feasible).
 >
 > I extract the JFFS2 core files for use in eCos automatically, into a git
 > tree at git.infradead.org/users/dwmw2/jffs2-ecos-core.git -- this tracks
 > Linus' git tree and shuffles the files into the arrangement that they
 > have in the eCos source tree.
 >
 > The intention was that someone working on eCos would periodically pull
 > from that tree into something like the 'jffs2-ecos.git' tree next to it,
 > which also has the eCos-specific files, and then build and test on eCos.
 > It doesn't seem to have happened though.
I would like to do that.
But my boss must give me time for that...
Anyhow, I would like to do it in my free time, but that can take a 
while. I'll do my best.

 >
 > Another way to reduce the amount of memory taken would be to reduce the
 > number of nodes, which might be achieved by increasing the 'page size'.
 > JFFS2 on Linux never writes a data node which crosses from one MMU page
 > to another, but on eCos you could relax that quite easily and write
 > larger data nodes -- which depending on the files you store may well
 > mean you have fewer of them.
Indeed. That's what I have done. I have increased the node size to 32kB. 
And adapted the eCos TFTP server to use the linux page size in case it 
is defined. (I should commit it back..)

Thanks for your reply,
Juergen


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[ECOS] Re: JFFS2 needs approx 72B of RAM per file

[David Woodhouse]

On Tue, 2008-04-08 at 01:14 +0200, Jürgen Lambrecht wrote:
> Hello to ecos-discuss and especially linux-mtd,
> 
> Tests have revealed to me that jffs2 does not only need raw nodes in RAM 
> (I use CYGNUM_FS_JFFS2_RAW_NODE_REF_CACHE_POOL_SIZE to have this 
> statically), but also "eats RAM" per file in flash.
> I want to remove this allocation, but I don't know if this is possible, 
> what are the disadvantages....?
> (my basic problem is that I have not enough RAM on my board, but it is 
> too late to change HW for the running projects)
> 
> After reading some jffs2 documentation, I think that my problem is the 
> inode cache?

I don't think so -- I think the raw node references are likely to be the
major contributing factor. It should be easy enough to check though,
surely? 

The problem, fundamentally, is that JFFS2 has no structure on the
medium. We have to _remember_ where every node (log entry) is.

I recently did some work to reduce the amount of memory per node from 16
to 8 bytes -- dropping a ->next pointer by using arrays of the things,
and dropping the 'length' field by observing that it's actually
redundant with 'offset' and '->next->offset' 99% of the time, and
finding tricks to make it redundant _100%_ of the time. If you're not
already using the latest JFFS2 code, it's worth doing that.

I say "recently" -- it's been over the last two years or so, but I don't
think eCos is using the results yet. It'd be good if someone would take
responsibility for merging newer JFFS2 code into eCos (or just tell me
it's not going to happen, and I'll stop trying to keep it feasible).

I extract the JFFS2 core files for use in eCos automatically, into a git
tree at git.infradead.org/users/dwmw2/jffs2-ecos-core.git -- this tracks
Linus' git tree and shuffles the files into the arrangement that they
have in the eCos source tree.

The intention was that someone working on eCos would periodically pull
from that tree into something like the 'jffs2-ecos.git' tree next to it,
which also has the eCos-specific files, and then build and test on eCos.
It doesn't seem to have happened though.

Another way to reduce the amount of memory taken would be to reduce the
number of nodes, which might be achieved by increasing the 'page size'. 
JFFS2 on Linux never writes a data node which crosses from one MMU page
to another, but on eCos you could relax that quite easily and write
larger data nodes -- which depending on the files you store may well
mean you have fewer of them.

-- 
dwmw2


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss