[ECOS] JFFS2 eats memory

[ECOS] JFFS2 eats memory

[Jürgen Lambrecht]

Hello Oyvind and David,

in 2004, you talked about "JFFS2 eats memory". But the thread was 
stopped when David went to "OLS".
I have the same problem now, see my last mail about: JFFS2 "eats RAM" 
per file in flash.
Did you continue on this?
Or should I look to jffs3 or ubifs?

Kind regards,

-- 
Jürgen Lambrecht
R&D Engineer
Televic Transport Systems
http://www.televic.com
Televic NV / SA (main office)  	
Leo Bekaertlaan 1
B-8870 Izegem
Tel: +32 (0)51 303045
Fax: +32 (0)51 310670


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] JFFS2 eats memory

[Øyvind Harboe]

[-- Attachment #1: Type: text/plain, Size: 308 bytes --]

After your walkthrough if the approach on IRC, I came up with this
patch, which seems to take care of the problem.

If there is a consensus that this is something that would be generally
useful for eCos/JFFS2 users, I'll be happy to clean it up and submit a
patch.


-- 
Øyvind Harboe
http://www.zylin.com


[-- Attachment #2: memleakfix.txt --]
[-- Type: text/x-patch, Size: 2041 bytes --]

Index: nodemgmt.c
===================================================================
RCS file: /cvs/ecos/ecos/packages/fs/jffs2/current/src/nodemgmt.c,v
retrieving revision 1.6
diff -w -u -r1.6 nodemgmt.c
--- nodemgmt.c	11 Dec 2003 23:33:54 -0000	1.6
+++ nodemgmt.c	13 Jul 2004 13:35:44 -0000
@@ -549,6 +549,60 @@
 		printk(KERN_WARNING "Short write in obliterating obsoleted node at 0x%08x: %zd\n", ref_offset(ref), retlen);
 		return;
 	}
+
+	// Merge obsolete nodes into its previous node, i.e. always leave
+	// one node behind so as not to screw up ref_totlen()
+	if (ref->next_in_ino!=NULL)
+	  {	
+	    bool moreToDo;
+	    do {
+	      moreToDo=false;	      
+	      struct jffs2_inode_cache *ic;
+	      ic = jffs2_raw_ref_to_ic(ref);
+
+	      // unlink the node and 
+	      struct jffs2_raw_node_ref *raw;
+	      struct jffs2_raw_node_ref *prev=NULL;
+	      raw = ic->nodes;
+	      for (raw = ic->nodes; raw != (void *)ic; raw = raw->next_in_ino) {
+		// if this node *and* the previous *physical node* are obsolete, combine them.
+		if ((prev!=NULL)&&ref_obsolete(raw)) {
+		  // now take take it off the physcial list, unless it is the 
+		  // first node.
+		  struct jffs2_raw_node_ref *t;
+		  struct jffs2_raw_node_ref *phys_prev=NULL;
+		  t=jeb->first_node;
+		  while (t!=NULL) {
+		    if ((phys_prev!=NULL)&&(t==raw)&&ref_obsolete(prev)) {
+		      // take it off the inode list.
+		      prev->next_in_ino=t->next_in_ino;
+		      
+		      // take it off the physical list
+		      phys_prev->next_phys=t->next_phys;
+		      // update last physical entry pointer...
+		      if (jeb->last_node==t) {
+			jeb->last_node=t->next_phys;
+		      }
+		      
+		      // update physical __totlen field.
+		      phys_prev->__totlen+=t->__totlen;
+	      
+	  
+		      jffs2_free_raw_node_ref(raw);
+		      moreToDo=true;
+
+		      break;
+		    }
+		    phys_prev=t;
+		    t=t->next_phys;
+		  }
+		  break;
+		}
+		prev=raw;
+	      }
+	    } while (moreToDo);
+	      }
+	
 }
 
 #if CONFIG_JFFS2_FS_DEBUG > 0


[-- Attachment #3: Type: text/plain, Size: 148 bytes --]

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] JFFS2 eats memory

[Øyvind Harboe]

On Tue, 2004-07-13 at 12:05, David Woodhouse wrote:
> On Tue, 2004-07-13 at 11:49 +0200, Øyvind Harboe wrote:
> > Changing the size of jffs2_raw_node_ref does not help much, since the
> > problem is that my system runs out of memory since it continously
> > overwrites existing files, thus filling up the flash with obsoleted
> > nodes.
> 
> It'll help in a lot of cases. You have a jffs2_raw_node_ref for _all_
> data nodes which are physically present on the flash, whether the inodes
> to which they belong are opened or not. There can be thousands of these.
> 
> In Linux we also use a different allocator, allocating these from
> fixed-size slabs so there isn't an 8-byte overhead for each one. 

When I say that reducing size does not "help", I mean that the problem
with memory usage constantly increasing persits. It just takes a bit
longer, e.g. my system falls over in 4 days instead of 1, which in my
case makes no difference.

> 
> > I'm pretty sure the problem is in the file fragment list...
> 
> What's the problem?

I'd like to take this oportunity to say that I'm trying to address this
problem because I need to, not because my eCos/JFFS2 knowledge is up to
the task. 

>  You say the problem goes away when you prune the
> icache...

My problem *almost* goes away with the icache_evict() change.

>  that implies that there's nothing being _lost_, but you're
> just objecting to the fact that your inode cache is doing any caching.
> 
> It's keeping the inode around for you in case you open it again
> immediately, because it _knows_ the garbage collector tends to exhibit
> that kind of behaviour. You made it stop doing that, and your problem
> went away, right?

No, it almost went away. 24 bytes is still lost every time I
overwrite/delete a file

> Can you join #mtd on irc.freenode.net?

I took it for a quick spin, but got some error messages. Perhaps some
firewall issues, etc. I'm not familiar with IRC.

> > Thats a lot more than I want to attempt with my current knowledge about
> > JFFS2. :-)
> 
> It's not about JFFS2. It's about eCos. The complex parts of the JFFS2
> bits are in the core JFFS2 code which you're not expected to touch. IT's
> the eCos wrapper around that core code which needs a rewrite :)

I see.

-- 
Øyvind Harboe
http://www.zylin.com



-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] JFFS2 eats memory

[Thomas Koeller]

On Tuesday 13 July 2004 11:49, Øyvind Harboe wrote:

> Changing the size of jffs2_raw_node_ref does not help much, since the
> problem is that my system runs out of memory since it continously
> overwrites existing files, thus filling up the flash with obsoleted
> nodes.

If this is so, then you are actually using only a fraction of the flash
memory you have available. In this case you could reduce the amount of
flash allocated to your JFFS2 file system, so that the garbage collection
would be triggered earlier, and you'd get fewer obsoleted nodes in RAM.

-- 
Thomas Koeller
thomas@koeller.dyndns.org

--
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] JFFS2 eats memory

[David Woodhouse]

On Tue, 2004-07-13 at 11:49 +0200, Øyvind Harboe wrote:
> Changing the size of jffs2_raw_node_ref does not help much, since the
> problem is that my system runs out of memory since it continously
> overwrites existing files, thus filling up the flash with obsoleted
> nodes.

It'll help in a lot of cases. You have a jffs2_raw_node_ref for _all_
data nodes which are physically present on the flash, whether the inodes
to which they belong are opened or not. There can be thousands of these.

In Linux we also use a different allocator, allocating these from
fixed-size slabs so there isn't an 8-byte overhead for each one. 

> I'm pretty sure the problem is in the file fragment list...

What's the problem? You say the problem goes away when you prune the
icache... that implies that there's nothing being _lost_, but you're
just objecting to the fact that your inode cache is doing any caching.

It's keeping the inode around for you in case you open it again
immediately, because it _knows_ the garbage collector tends to exhibit
that kind of behaviour. You made it stop doing that, and your problem
went away, right?

Can you join #mtd on irc.freenode.net?

> Thats a lot more than I want to attempt with my current knowledge about
> JFFS2. :-)

It's not about JFFS2. It's about eCos. The complex parts of the JFFS2
bits are in the core JFFS2 code which you're not expected to touch. IT's
the eCos wrapper around that core code which needs a rewrite :)

-- 
dwmw2


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] JFFS2 eats memory

[Øyvind Harboe]

On Tue, 2004-07-13 at 11:30, David Woodhouse wrote:
> On Mon, 2004-07-12 at 16:42 +0200, Øyvind Harboe wrote:
> > This issue has been discussed before, and although I have a workaround,
> > I'd dearly like to have it put to bed since it is starting to cause
> > problems elsewhere in my application:
> > 
> > - My code opens a file for writing with O_TRUNC set, performs
> > a single write call, closes the file.
> > - After closing the file, JFFS2 has eaten memory.
> > - With the attached modifcations to JFFS2, it "only" eats 24 bytes.
> 
> That would be a single struct jffs2_raw_node_ref. That's 16 bytes and I
> assume the other 8 is allocator overhead -- unless you're on a 64-bit
> platform where it is actually 24 bytes.

Yes.

> It wouldn't be hard to cut the jffs2_raw_node_ref down to 12 bytes
> instead of 16, btw. There are _many_ of these. See the discussion about
> making the commented ref_totlen() macro in nodelist.h actually work in
> 100% of cases, rather than only 95% as it would at the moment, and then
> we could drop the totlen field.

Changing the size of jffs2_raw_node_ref does not help much, since the
problem is that my system runs out of memory since it continously
overwrites existing files, thus filling up the flash with obsoleted
nodes.

> > Presumably when the raw nodes in the file fragement list are marked as
> > obsolete, they are no longer required, but are not freed.
> 
> The jffs2_raw_node_ref structures _are_ required. You need those in
> order to be able to find all the parts of a given inode if/when it gets
> opened again.
> 
> But the file fragment list, which is held in memory for as long as the
> eCos-specific "struct _inode" exists, is not necessarily required.

I'm pretty sure the problem is in the file fragment list...

> You've made the eCos code prune its inode cache more aggressively. Be
> careful that you don't pessimise the garbage-collection code. It will
> often spend a fair bit of time opening an inode (with
> jffs2_gc_fetch_inode()), doing some garbage collection, and then
> releasing it again (with jffs2_gc_release_inode()).

Once/if I get the file fragment memory loss fixed, I'll try to remove it
and run some stress testing for steady state memory usage to see if I
can take out those changes.

> 
> Now that we have those methods, actually, you could mark only _those_
> inodes so that they remain in cache for a little while longer, while
> instantly releasing inodes from your icache if they were opened by the
> user.
> 
> > Q: Is this fundamentally impossible or a "bad idea" to fix?
> 
> Not at all. Everything under fs/ecos/ is yours to do with as you see
> fit. I'm happy to advise but I'd _really_ like to see someone rewrite
> the eCos parts altogether; they still look _far_ too much like a
> hacked-up Linux compatibility layer, and they no longer need to.

Thats a lot more than I want to attempt with my current knowledge about
JFFS2. :-)



-- 
Øyvind Harboe
http://www.zylin.com



-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] JFFS2 eats memory

[David Woodhouse]

On Mon, 2004-07-12 at 16:42 +0200, Øyvind Harboe wrote:
> This issue has been discussed before, and although I have a workaround,
> I'd dearly like to have it put to bed since it is starting to cause
> problems elsewhere in my application:
> 
> - My code opens a file for writing with O_TRUNC set, performs
> a single write call, closes the file.
> - After closing the file, JFFS2 has eaten memory.
> - With the attached modifcations to JFFS2, it "only" eats 24 bytes.

That would be a single struct jffs2_raw_node_ref. That's 16 bytes and I
assume the other 8 is allocator overhead -- unless you're on a 64-bit
platform where it is actually 24 bytes.

It wouldn't be hard to cut the jffs2_raw_node_ref down to 12 bytes
instead of 16, btw. There are _many_ of these. See the discussion about
making the commented ref_totlen() macro in nodelist.h actually work in
100% of cases, rather than only 95% as it would at the moment, and then
we could drop the totlen field.

> Presumably when the raw nodes in the file fragement list are marked as
> obsolete, they are no longer required, but are not freed.

The jffs2_raw_node_ref structures _are_ required. You need those in
order to be able to find all the parts of a given inode if/when it gets
opened again.

But the file fragment list, which is held in memory for as long as the
eCos-specific "struct _inode" exists, is not necessarily required.

You've made the eCos code prune its inode cache more aggressively. Be
careful that you don't pessimise the garbage-collection code. It will
often spend a fair bit of time opening an inode (with
jffs2_gc_fetch_inode()), doing some garbage collection, and then
releasing it again (with jffs2_gc_release_inode()).

Now that we have those methods, actually, you could mark only _those_
inodes so that they remain in cache for a little while longer, while
instantly releasing inodes from your icache if they were opened by the
user.

> Q: Is this fundamentally impossible or a "bad idea" to fix?

Not at all. Everything under fs/ecos/ is yours to do with as you see
fit. I'm happy to advise but I'd _really_ like to see someone rewrite
the eCos parts altogether; they still look _far_ too much like a
hacked-up Linux compatibility layer, and they no longer need to.

-- 
dwmw2


-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] JFFS2 eats memory

[Øyvind Harboe]

This mail explains a lot. I'll try to see if I can't follow his
suggestions.

http://sources.redhat.com/ml/ecos-discuss/2003-10/msg00248.html


-- 
Øyvind Harboe
http://www.zylin.com



-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] JFFS2 eats memory

[Øyvind Harboe]

On Tue, 2004-07-13 at 09:53, Andrew Lunn wrote:
> > > Presumably when the raw nodes in the file fragement list are marked as
> > > obsolete, they are no longer required, but are not freed.
> 
> One other thing. You might consider try gettting the garbage collect
> code runnig. This might help solve your problem. 

I just took it for a spin: no change.

This is not surprising. 

There is nothing to garbage collect, i.e. there is plenty of free space
on the JFFS2 drive and very few deleted raw nodes.

> I guess as a proof of concept just call
> jffs2_garbage_collect_pass(JFFS2_SB_INFO(jffs2_sb));
> 
> This is not strictly correct since you need to add locking to one of
> the lists to make this thread safe. I forget which and i've forgotten
> where i read about this, but im sure google will find it for you.
> 
>         Andrew
-- 
Øyvind Harboe
http://www.zylin.com



-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] JFFS2 eats memory

[Øyvind Harboe]

On Tue, 2004-07-13 at 09:40, Andrew Lunn wrote:
> On Mon, Jul 12, 2004 at 04:42:11PM +0200, ?yvind Harboe wrote:
> > This issue has been discussed before, and although I have a workaround,
> > I'd dearly like to have it put to bed since it is starting to cause
> > problems elsewhere in my application:
> > 
> > - My code opens a file for writing with O_TRUNC set, performs
> > a single write call, closes the file.
> > - After closing the file, JFFS2 has eaten memory.
> > - With the attached modifcations to JFFS2, it "only" eats 24 bytes.
> > - If I unmount and remount JFFS2, no memory is "lost" and JFFS2 works
> > fine.
> > 
> > Presumably when the raw nodes in the file fragement list are marked as
> > obsolete, they are no longer required, but are not freed.
> > 
> > Q: Is this fundamentally impossible or a "bad idea" to fix?
> 
> How much memory are we talking about here in this example?

24 bytes* number of write() operations for each time a file is
truncated.

Additionally, memory is lost for other operations(e.g. deleting files),
but I don't have any numbers(24 bytes for every time a file is
deleted?).

I'd really like to see that JFFS2 memory usage was constant if the
number of files(and sections within those files) is constant. That way I
could just run my system and measure the maximum usage.

> The cache is there for a reason, so i would not want to
> unconditionally disable it. At least you need some CDL to control
> between fat & fast and slow & slim.

Is JFFS2 faster because it caches raw_node_ref's to deleted nodes?

Sounds strange.

The mount code simply ignores these nodes, which insinuates that caching
these nodes does not help performance.

> I would also suggest you consider extending the
> CYGNUM_FS_JFFS2_RAW_NODE_REF_CACHE_POOL_SIZE concept to the inode
> cache.  You can then control the size of the cache. 

AFAICT, this setting is no help in this case, because all it would do is
to cause JFFS2 operations to fail when running out of pool space instead
of when running out of overall system memory.

Further it would lock a piece of the overall system memory to JFFS2,
making the global memory situation worse.

> 
>         Andrew
-- 
Øyvind Harboe
http://www.zylin.com



-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] JFFS2 eats memory

[Andrew Lunn]

> > Presumably when the raw nodes in the file fragement list are marked as
> > obsolete, they are no longer required, but are not freed.

One other thing. You might consider try gettting the garbage collect
code runnig. This might help solve your problem. 

I guess as a proof of concept just call
jffs2_garbage_collect_pass(JFFS2_SB_INFO(jffs2_sb));

This is not strictly correct since you need to add locking to one of
the lists to make this thread safe. I forget which and i've forgotten
where i read about this, but im sure google will find it for you.

        Andrew

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] JFFS2 eats memory

[Andrew Lunn]

On Mon, Jul 12, 2004 at 04:42:11PM +0200, ?yvind Harboe wrote:
> This issue has been discussed before, and although I have a workaround,
> I'd dearly like to have it put to bed since it is starting to cause
> problems elsewhere in my application:
> 
> - My code opens a file for writing with O_TRUNC set, performs
> a single write call, closes the file.
> - After closing the file, JFFS2 has eaten memory.
> - With the attached modifcations to JFFS2, it "only" eats 24 bytes.
> - If I unmount and remount JFFS2, no memory is "lost" and JFFS2 works
> fine.
> 
> Presumably when the raw nodes in the file fragement list are marked as
> obsolete, they are no longer required, but are not freed.
> 
> Q: Is this fundamentally impossible or a "bad idea" to fix?

How much memory are we talking about here in this example?

The cache is there for a reason, so i would not want to
unconditionally disable it. At least you need some CDL to control
between fat & fast and slow & slim.

I would also suggest you consider extending the
CYGNUM_FS_JFFS2_RAW_NODE_REF_CACHE_POOL_SIZE concept to the inode
cache.  You can then control the size of the cache. 

        Andrew

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

[ECOS] JFFS2 eats memory

[Øyvind Harboe]

[-- Attachment #1: Type: text/plain, Size: 707 bytes --]

This issue has been discussed before, and although I have a workaround,
I'd dearly like to have it put to bed since it is starting to cause
problems elsewhere in my application:

- My code opens a file for writing with O_TRUNC set, performs
a single write call, closes the file.
- After closing the file, JFFS2 has eaten memory.
- With the attached modifcations to JFFS2, it "only" eats 24 bytes.
- If I unmount and remount JFFS2, no memory is "lost" and JFFS2 works
fine.

Presumably when the raw nodes in the file fragement list are marked as
obsolete, they are no longer required, but are not freed.

Q: Is this fundamentally impossible or a "bad idea" to fix?


-- 
Øyvind Harboe
http://www.zylin.com


[-- Attachment #2: jffsmemfix.txt --]
[-- Type: text/x-patch, Size: 1184 bytes --]

Index: src/fs-ecos.c
===================================================================
RCS file: /cvs/ecos/ecos/packages/fs/jffs2/current/src/fs-ecos.c,v
retrieving revision 1.27
diff -u -w -r1.27 fs-ecos.c
--- src/fs-ecos.c	21 Apr 2004 18:51:21 -0000	1.27
+++ src/fs-ecos.c	12 Jul 2004 14:25:41 -0000
@@ -199,7 +199,9 @@
 	// held where needed for dotdot filepaths)
 	while (this) {
 		next = this->i_cache_next;
-		if (this != i && this->i_count == 0) {
+		if (
+		    //this != i && 
+		    this->i_count == 0) {
 			struct _inode *parent = this->i_parent;
 			if (this->i_cache_next)
 				this->i_cache_next->i_cache_prev = this->i_cache_prev;
@@ -1466,12 +1468,22 @@
 {
 	struct _inode *node = (struct _inode *) fp->f_data;
 
+	// cache values before we destroy the node
+	struct jffs2_sb_info *c = JFFS2_SB_INFO(node->i_sb);
+	struct _inode *root = node->i_sb->s_root;
+
 	D2(printf("jffs2_fo_close\n"));
 
 	jffs2_iput(node);
 
 	fp->f_data = 0;		// zero data pointer
 
+
+	// free as much of cached structures as possible to make sure
+	// that memory usage stays stable and that idle memory usage
+	// is at a minimum.
+	icache_evict(root, node);	
+
 	return ENOERR;
 }
 


[-- Attachment #3: Type: text/plain, Size: 148 bytes --]

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss