* [ECOS] Simple flash filesystem? @ 2001-02-05 14:10 Grant Edwards 2001-02-05 14:29 ` Lewin A.R.W. Edwards 0 siblings, 1 reply; 11+ messages in thread From: Grant Edwards @ 2001-02-05 14:10 UTC (permalink / raw) To: ecos-discuss I've been looking for info on flash filesystems, and have found pretty much nothing. What I have found are flash device-drivers that emulate 512-byte block devices so that you can use them in place of disk drives underneath file systems designed for disk drives. These filesystems are designed with (and optimized for) the following assumptions: 1) The physical media has 512 blocks that are independently erasable/writable. 2) Read and write operations are both an order of magnitude slower than RAM. 3) Read and write times for a sector depend on the sector number (i.e. there is a seek-time). 4) There is zero cost associated with an erase (either elapsed time or media lifetime). For flash, none of these assumptions are true. Blocks can only be erased in large chunks (typically 64k). Read operations are the same speed (roughly) as RAM. Write operations are an order of magnitude slower. Erase operations are _another_ order of magnitude slower and have an associated media life cost. Read/write speeds are independant of address. Though it may be the most expedient thing to do in the short-term, It seems like a bad idea to take a complex filesystem designed for disk and use it for a flash. For example, it seems like a waste of effort to impliment 512 byte blocks when the filesystem uses them in 4K clusters. Rather than try to make a flash act like a disk drive (and using a disk-drive filesystem), has anybody seen any information on a simple filesystem designed for flash? -- Grant Edwards grante@visi.com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [ECOS] Simple flash filesystem? 2001-02-05 14:10 [ECOS] Simple flash filesystem? Grant Edwards @ 2001-02-05 14:29 ` Lewin A.R.W. Edwards 2001-02-06 8:05 ` [ECOS] " Grant Edwards 0 siblings, 1 reply; 11+ messages in thread From: Lewin A.R.W. Edwards @ 2001-02-05 14:29 UTC (permalink / raw) To: Grant Edwards, ecos-discuss Hi Grant, >I've been looking for info on flash filesystems, and have found pretty >much nothing. Random notes: I suspect that the reason you haven't found many references is that these sorts of applications (in my experience so far) fall into these categories: 1. Cases where you _want_ to emulate a DOS-type filesystem, especially cases where you have an underlying flash controller that does the level-wearing and error correction for you, and 2. Cases where you're only going to write data once in a blue moon, and it's most efficient to solve the problem on an ad hoc basis. It really doesn't matter what logical sector size you use as long as you have a reasonably efficient scatter/gather system. I have a very very similar problem working on SmartMedia - the filesystem is organized into 512-byte sectors but the card is only erasable in blocks (which by default are cluster-sized). In order to keep the upper layers of the filesystem as generic as possible, I let my DOS filesystem work with the 512-byte sectors it knows and loves. In older versions of my SSFDC code, the DOS layer decomposed each R/W op into a series of single sector R/W ops. In the case of a random sector W op, the flash interface layer would then read in a block, update, erase and verify. This was obviously very inefficient (though it works fine on CompactFlash, which has an intelligent R/W controller), so I later changed the breakdown. In my current code, the DOS filesystem layer decomposes R/W ops into a series of variable-length ops, each of which is no more than one cluster long. This implicitly allows the underlying flash driver to make the R/W/erase loop much more efficient. By judiciously adding a few extra K of RAM, it is also possible to increase the write caching capability to cope with any erase block size. The main performance hit I encountered in emulating a DOS filesystem over dumb flash is updating the FAT. If you can keep the whole FAT, or at least all the sectors for open-for-writing chains, in RAM (and only update sections when open-for-write files are closed), you'll have a huge performance increase. In the eval board for the processor in our old products, the vendor uses a very simple, dumb, intended-for-read-only filesystem. It's easiest to illustrate by example: If your flash free space starts at 0x20000: 0x20000 00 12 00 00 = length of file (x00001200), 0x00000000 for deleted file, or 0xFFFFFFFF if no more files 0x20004 FREDXXXX.XXX\0 = filename (ASCIIZ) 0x2000x file data 0x2120x next file header To find a particular file you start at the beginning of filespace and get the first word (pointer to next file). You then check the filename immediately after the word. If it's the file you want, then read it out. If it's not the desired file, then use the pointer to skip to the next file until you find the one you want or reach the end of the chain. Each file is guaranteed (by manipulating the length field) to start on a write-page boundary (not an erase-block boundary though). If you do something like this, you will eventually get so fragmented that you'll need to gather up all the files again (which you could do with an "optimize" option in your UI if you wanted to). But as a quick and dirty solution, it has some merit. Maybe you might want to consider NAND flash (SmartMedia in a chip package, essentially) for your file storage. It does have the benefit of having a block size that is exactly the same as the best DOS cluster size for the media capacity. And it's fairly easy to work with. === Lewin A.R.W. Edwards (Embedded Engineer) Work: http://www.digi-frame.com/ Personal: http://www.zws.com/ and http://www.larwe.com/ "Und setzet ihr nicht das Leben ein, Nie wird euch das Leben gewonnen sein." ^ permalink raw reply [flat|nested] 11+ messages in thread
* [ECOS] Re: Simple flash filesystem? 2001-02-05 14:29 ` Lewin A.R.W. Edwards @ 2001-02-06 8:05 ` Grant Edwards 2001-02-06 8:21 ` Lewin A.R.W. Edwards 2001-02-06 9:20 ` Wilson Kwan 0 siblings, 2 replies; 11+ messages in thread From: Grant Edwards @ 2001-02-06 8:05 UTC (permalink / raw) To: Lewin A.R.W. Edwards; +Cc: ecos-discuss Lewin A.R.W. Edwards writes: > >I've been looking for info on flash filesystems, and have found pretty > >much nothing. > > Random notes: > > I suspect that the reason you haven't found many references is that > these sorts of applications (in my experience so far) fall into these > categories: > > 1. Cases where you _want_ to emulate a DOS-type filesystem, especially > cases where you have an underlying flash controller that does the > level-wearing and error correction for you, and In my case, I've got no reason to want to emulate a DOS filesystem. > 2. Cases where you're only going to write data once in a blue moon, and > it's most efficient to solve the problem on an ad hoc basis. That's pretty much my situation. The filesystem will be use to hold read-only stuff that's only changed infrequently, and the system doesn't have to operate "normally" during the update process. The filesystem will basically contain web pages and Java applets. Once the user has got the web pages working they way he wants, the filesystem will be strictly read-only for normal operation. I may just forget about writing individual files and force the user to download an entire ROM filesystem everytime anything changes. That would be way simpler, but a little slower. > The main performance hit I encountered in emulating a DOS filesystem over > dumb flash is updating the FAT. If you can keep the whole FAT, or at least > all the sectors for open-for-writing chains, in RAM (and only update > sections when open-for-write files are closed), you'll have a huge > performance increase. Since writes are going to be rare, and the system is allowed to be "down" while they happen, even a DOS filesystem is probably overkill. > To find a particular file you start at the beginning of filespace and get > the first word (pointer to next file). You then check the filename > immediately after the word. If it's the file you want, then read it out. If > it's not the desired file, then use the pointer to skip to the next file > until you find the one you want or reach the end of the chain. That's similar to the ROM filesystem currently used by the GoAhead web server. If I stick with that, I'll just download the entire filesystem every time. What I'm trying to figure out is what the simplest solution would be that would allow downloading individual files. I was surprised that I couldn't find much using google/deja. > Maybe you might want to consider NAND flash (SmartMedia in a chip package, > essentially) for your file storage. It does have the benefit of having a > block size that is exactly the same as the best DOS cluster size for the > media capacity. And it's fairly easy to work with. I'm pretty much stuck with "regular" flash (64k sectors). -- Grant Edwards grante@visi.com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [ECOS] Re: Simple flash filesystem? 2001-02-06 8:05 ` [ECOS] " Grant Edwards @ 2001-02-06 8:21 ` Lewin A.R.W. Edwards 2001-02-06 8:43 ` Grant Edwards 2001-02-06 9:20 ` Wilson Kwan 1 sibling, 1 reply; 11+ messages in thread From: Lewin A.R.W. Edwards @ 2001-02-06 8:21 UTC (permalink / raw) To: Grant Edwards; +Cc: ecos-discuss Hi Grant, >The filesystem will basically contain web pages and Java applets. I figured this. So you're dealing with numerous small filelets which will mostly be smaller than the block size, and it is not unreasonable to reserve at least one block for directory information. How about this for a solution: Reserve one block (64K) as a directory block. Keep that whole block in RAM at all times. Inside it, have a number of structures like this: 32bits pointer to file data or NULL if no file exists 32bits file size in bytes asciiz variable length filename (\0 for non-file) Keep the directory sorted by file pointer. When you want to create a new file, you can follow this algorithm: * if there is a free block (64K), allocate part of it for the file, erase that block and write the file to the start of it, and update the directory. * if there is no completely free block, search for a block that has free space in its tail big enough to accommodate your file. Read that block into RAM, append your new file into the tail, and erase/writeback the block. The same algorithm can be used to replace an existing file - simply change the existing file's pointer in the directory to indicate that it's no longer in use, then follow the steps above. Depending on how power-failure-tolerant you need this to be, you can get a big performance improvement on writes because your directory is all in RAM and you only need to commit it back to flash periodically. This does not perform any explicit wear leveling, obviously, and it also doesn't make the most efficient use of space (note the lack of support for noncontiguous allocation!). But I think this system might do well for your application where you're mostly dealing with tiny files that are not going to be rewritten frequently. >I may just forget about writing individual files and force the user >to download an entire ROM filesystem everytime anything changes. That >would be way simpler, but a little slower. I wanted to do this for one of our products, but I was shouted down (legitimately, I think) on the grounds that end-users want to be able to read/write randomly. So the project is sitting on the back-burner right now ;) === Lewin A.R.W. Edwards (Embedded Engineer) Work: http://www.digi-frame.com/ Personal: http://www.zws.com/ and http://www.larwe.com/ "Und setzet ihr nicht das Leben ein, Nie wird euch das Leben gewonnen sein." ^ permalink raw reply [flat|nested] 11+ messages in thread
* [ECOS] Re: Simple flash filesystem? 2001-02-06 8:21 ` Lewin A.R.W. Edwards @ 2001-02-06 8:43 ` Grant Edwards 0 siblings, 0 replies; 11+ messages in thread From: Grant Edwards @ 2001-02-06 8:43 UTC (permalink / raw) To: Lewin A.R.W. Edwards; +Cc: ecos-discuss Lewin A.R.W. Edwards writes: > >The filesystem will basically contain web pages and Java applets. > > I figured this. So you're dealing with numerous small filelets which will > mostly be smaller than the block size, and it is not unreasonable to > reserve at least one block for directory information. > > How about this for a solution: Reserve one block (64K) as a directory > block. Keep that whole block in RAM at all times. Inside it, have a number > of structures like this: > > 32bits pointer to file data or NULL if no file exists > 32bits file size in bytes > asciiz variable length filename (\0 for non-file) > > Keep the directory sorted by file pointer. > > When you want to create a new file, you can follow this algorithm: > > * if there is a free block (64K), allocate part of it for the file, erase > that block and write the file to the start of it, and update the directory. > * if there is no completely free block, search for a block that has free > space in its tail big enough to accommodate your file. Read that block into > RAM, append your new file into the tail, and erase/writeback the block. And the erase/writeback might not be needed. If the space is unused, it may still be all 1's from the last time it was erased. > The same algorithm can be used to replace an existing file - simply change > the existing file's pointer in the directory to indicate that it's no > longer in use, then follow the steps above. > > Depending on how power-failure-tolerant you need this to be, Not very. If the power fails while somebody is downloading (either a new file or overwriting a file), It's fine if that entire file is lost as long as the filesystem doesn't leak memory or loose other files. The latter can be avoided by designing the erase/writeback operation so that the writeback goes to a different block before the original block is deallocated. It's possible to end up with duplicate files, but that's easy enough to repair during start-up. > you can get a > big performance improvement on writes because your directory is all in RAM > and you only need to commit it back to flash periodically. Write performance isn't much of a concern. It would be nice if I could write on the fly during a TFTP transfer, but not required. > This does not perform any explicit wear leveling, obviously, and it also > doesn't make the most efficient use of space (note the lack of support for > noncontiguous allocation!). But I think this system might do well for your > application where you're mostly dealing with tiny files that are not going > to be rewritten frequently. Right -- I'm not too worried about wear levelling. The device will probably get written to a couple dozen times while the customer tests web pages, then it will sit for years. > >I may just forget about writing individual files and force the user > >to download an entire ROM filesystem everytime anything changes. That > >would be way simpler, but a little slower. > > I wanted to do this for one of our products, but I was shouted down > (legitimately, I think) on the grounds that end-users want to be able to > read/write randomly. So the project is sitting on the back-burner right now ;) The secret is to start with something so horrific that your fallback position sounds good. I initially told them that the web pages and java applets will be linked in with the executable at build time, so the customer will have to have a complete toolset and object files. Which is they way things work in the prototype. By comparison, running a host-end application to build a ROM filesystem from a directory tree and then download it will seem brilliant. ;) I'd be happy to do a nice little filesystem, but I don't think there's room in the schedule. -- Grant Edwards grante@visi.com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [ECOS] Re: Simple flash filesystem? 2001-02-06 8:05 ` [ECOS] " Grant Edwards 2001-02-06 8:21 ` Lewin A.R.W. Edwards @ 2001-02-06 9:20 ` Wilson Kwan 2001-02-06 9:28 ` Grant Edwards 1 sibling, 1 reply; 11+ messages in thread From: Wilson Kwan @ 2001-02-06 9:20 UTC (permalink / raw) To: Grant Edwards, Lewin A.R.W. Edwards; +Cc: ecos-discuss > Lewin A.R.W. Edwards writes: > > > >I've been looking for info on flash filesystems, and have found pretty > > >much nothing. You might want to have a look at ROMFS for eCos. I haven't tried it yet but came across it the other day. It's available at www.3glab.org/developer/ecos/. Wilson ^ permalink raw reply [flat|nested] 11+ messages in thread
* [ECOS] Re: Simple flash filesystem? 2001-02-06 9:20 ` Wilson Kwan @ 2001-02-06 9:28 ` Grant Edwards 0 siblings, 0 replies; 11+ messages in thread From: Grant Edwards @ 2001-02-06 9:28 UTC (permalink / raw) To: Wilson Kwan; +Cc: Lewin A.R.W. Edwards, ecos-discuss Wilson Kwan writes: > > Lewin A.R.W. Edwards writes: > > > > > >I've been looking for info on flash filesystems, and have found pretty > > > >much nothing. > > You might want to have a look at ROMFS for eCos. I haven't tried it yet but > came across it the other day. It's available at > www.3glab.org/developer/ecos/. Thanks, I came across that yesterday, and it looks nice and simple. Right now the only thing that needs a filesystem is the web server, and it comes with it's own rom filesystem. So unless I decide to go with a writable filesystem, I'll probably stick with the one I'm using now. It isn't a eCos "filesystem" in that it hooks to the eCos system calls, but as long as no other application code needs to access it, I don't really have to worry about having a real filesystem. -- Grant Edwards grante@visi.com ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <981465384.28425.ezmlm@sources.redhat.com>]
* [ECOS] Re: Simple flash filesystem? [not found] <981465384.28425.ezmlm@sources.redhat.com> @ 2001-02-06 12:30 ` Kristian Otnes 2001-02-06 12:50 ` Grant Edwards 0 siblings, 1 reply; 11+ messages in thread From: Kristian Otnes @ 2001-02-06 12:30 UTC (permalink / raw) To: ecos-discuss Hi Grant, the reason for emulating a disk in a flash based filesystem is probably twofold: - It fits in with the normal disk approach usage - It breaks the larger blocks (typically 64KB or 128KB) into virtual smaller blocks, so that other software is not bothered by the problem of handling the large flash blocks efficiently. In other words, it is a relatively simple way of managing some of the harder parts of flash usage. As Lewin indicated, there are easy ways out depending on the needs. But if you - need some sort of efficient use of the flash - intend to use the disk to some extent for repeated write operations - want to be sure that the flash survives power failures etc. without corrupting the logics of the data within it then I believe you need some sort of controller. I have seen one implementer doing it with variable block sizes for increased efficiency, others with fixed 512 bytes blocks and some with user defined sizes. From my experience, all the suggested solutions that seem usable for a broader audience, boils down to variations of the block controller/disk emulator concept. I think the main problem with emulating a disk, is that memory is needed for a mapping table between logical and physical blocks. One could maybe avoid this by some rather slow search mechanism. I guess it depends a bit on the needs, but again the emulating disk concept seems as a good allround solution. Ok, so you got a flash disk controller. Then, the next problem is the file system. It would be nice to have something - that doesn't do any unnecessary updates to the disk (this will waste virtual blocks, since changing just one bit on a disk will invalidate a virtual block). E.g. use the flash remembering that: 1. writing is slow, 2. erasing is sometimes even slower - that is somewhat secure so that it can doesn't leave you with an unreadable flash disk after a power failure or uncontrolled restart. - that is not too big (big being a relative term though...) Thus the following "stack" seems a practical approach: ------------ Application ------------ File System ------------ Flash Disk Controller ------------ Flash Driver ------------ The flash disk controller can be reused with different flash file systems, since the file functionality holds the more or less advanced file system features required. One could start off with something very simple, and then grow... All in all, it is probably not made in a day, but once mastered, it will be easier to use for new products, especially as one gets used to having a flash disk... Alas, it is hard to find any free and good system on the net... I couldn't when I needed it. The Journalling flash file system ( http://www.developer.axis.com/software/jffs/ ) is possibly one candidate, but it is for Linux (maybe portable). I don't know the situation of its overall performance and recovery mechanisms... Kris. ****************************************************************** Tevero AS Kristian Otnes P.O.Box 96 Tel: +47 67 56 29 95 N-1317 BAERUMS VERK Mobile: +47 93 05 48 58 NORWAY E-mail: kristian.otnes@tevero.no Web: www.tevero.no/products ****************************************************************** > Subject: Simple flash filesystem? > Date: Mon, 05 Feb 2001 22:10:12 GMT > From: "Grant Edwards" <grante@visi.com> > To: ecos-discuss@sources.redhat.com > > I've been looking for info on flash filesystems, and have found pretty > much nothing. > > What I have found are flash device-drivers that emulate 512-byte > block devices so that you can use them in place of disk drives > underneath file systems designed for disk drives. These filesystems > are designed with (and optimized for) the following assumptions: > > 1) The physical media has 512 blocks that are independently > erasable/writable. > > 2) Read and write operations are both an order of magnitude slower > than RAM. > > 3) Read and write times for a sector depend on the sector number (i.e. > there is a seek-time). > > 4) There is zero cost associated with an erase (either elapsed time > or media lifetime). > > For flash, none of these assumptions are true. Blocks can only be erased > in large chunks (typically 64k). Read operations are the same speed > (roughly) as RAM. Write operations are an order of magnitude slower. > Erase operations are _another_ order of magnitude slower and have an > associated media life cost. Read/write speeds are independant of address. > > Though it may be the most expedient thing to do in the short-term, > It seems like a bad idea to take a complex filesystem designed for disk > and use it for a flash. For example, it seems like a waste of effort > to impliment 512 byte blocks when the filesystem uses them in 4K clusters. > > Rather than try to make a flash act like a disk drive (and using a > disk-drive filesystem), has anybody seen any information on a simple > filesystem designed for flash? > > -- > Grant Edwards > grante@visi.com > > -- ****************************************************************** Tevero AS Kristian Otnes P.O.Box 96 Tel: +47 67 56 29 95 N-1317 BAERUMS VERK Mobile: +47 93 05 48 58 NORWAY E-mail: kristian.otnes@tevero.no ****************************************************************** ^ permalink raw reply [flat|nested] 11+ messages in thread
* [ECOS] Re: Simple flash filesystem? 2001-02-06 12:30 ` Kristian Otnes @ 2001-02-06 12:50 ` Grant Edwards 2001-02-06 12:54 ` Lewin A.R.W. Edwards 0 siblings, 1 reply; 11+ messages in thread From: Grant Edwards @ 2001-02-06 12:50 UTC (permalink / raw) To: Kristian Otnes; +Cc: ecos-discuss Kristian Otnes writes: [excellent analsys of problem] > Alas, it is hard to find any free and good system on the > net... I couldn't when I needed it. The Journalling flash file system > ( http://www.developer.axis.com/software/jffs/ ) is possibly one > candidate, but it is for Linux (maybe portable). I don't know > the situation of its overall performance and recovery mechanisms... I found jffs and a couple other journalling approaches to implimenting a block device underneath a Unix or DOS filesystem, but they all seemed way too big/complex when my filesystem isn't going to be more than a few hundred KBytes and will be rarely written to. For now, I'm going to try to use a ROM filesystem and have users download the whole directory tree when a file changes. \x18It could be worse, I could ask them to link the files as data with the application code and eCos. Maybe in phase II (AKA: the list of things that will never get done) I'll come up with a simple flash filesystem for my situation: writes are rare and are allowed to block reads for a long time (several seconds). -- Grant Edwards grante@visi.com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [ECOS] Re: Simple flash filesystem? 2001-02-06 12:50 ` Grant Edwards @ 2001-02-06 12:54 ` Lewin A.R.W. Edwards 2001-02-06 13:03 ` Grant Edwards 0 siblings, 1 reply; 11+ messages in thread From: Lewin A.R.W. Edwards @ 2001-02-06 12:54 UTC (permalink / raw) To: Grant Edwards; +Cc: ecos-discuss >way too big/complex when my filesystem isn't going to be more than a few >hundred KBytes and will be rarely written to. For now, I'm going to try Aha!! Do you have a "few hundred KBytes" of free RAM? You could get the best of all possible worlds by loading the FS into RAM, and allowing random R/W as convenient, then committing the entire thing back to flash when your device is idle. === Lewin A.R.W. Edwards (Embedded Engineer) Work: http://www.digi-frame.com/ Personal: http://www.zws.com/ and http://www.larwe.com/ "Und setzet ihr nicht das Leben ein, Nie wird euch das Leben gewonnen sein." ^ permalink raw reply [flat|nested] 11+ messages in thread
* [ECOS] Re: Simple flash filesystem? 2001-02-06 12:54 ` Lewin A.R.W. Edwards @ 2001-02-06 13:03 ` Grant Edwards 0 siblings, 0 replies; 11+ messages in thread From: Grant Edwards @ 2001-02-06 13:03 UTC (permalink / raw) To: Lewin A.R.W. Edwards; +Cc: ecos-discuss Lewin A.R.W. Edwards writes: > >way too big/complex when my filesystem isn't going to be more than a few > >hundred KBytes and will be rarely written to. For now, I'm going to try > > Aha!! Do you have a "few hundred KBytes" of free RAM? I should. > You could get the best of all possible worlds by loading the FS into RAM, > and allowing random R/W as convenient, then committing the entire thing > back to flash when your device is idle. Yup. That's probably the simplest solution for a small filesystem. Burns can happen in the background without interrupting read access to the RAM version of the files. Of course once marketing gets hold of the web pages, I may end up with several megabytes of real-audio and dancing GIFs. Then half of my assumptions go out the window. :) -- Grant Edwards grante@visi.com ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2001-02-06 13:03 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2001-02-05 14:10 [ECOS] Simple flash filesystem? Grant Edwards 2001-02-05 14:29 ` Lewin A.R.W. Edwards 2001-02-06 8:05 ` [ECOS] " Grant Edwards 2001-02-06 8:21 ` Lewin A.R.W. Edwards 2001-02-06 8:43 ` Grant Edwards 2001-02-06 9:20 ` Wilson Kwan 2001-02-06 9:28 ` Grant Edwards [not found] <981465384.28425.ezmlm@sources.redhat.com> 2001-02-06 12:30 ` Kristian Otnes 2001-02-06 12:50 ` Grant Edwards 2001-02-06 12:54 ` Lewin A.R.W. Edwards 2001-02-06 13:03 ` Grant Edwards
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).