[ECOS] A question about function readdir and stat

[ECOS] A question about function readdir and stat

[eCos]

Dear all：
We have to do a total sd card scanning and check for some certain type of 
files in our application, some application code is as below:

-----------------------------------------------
DIR *d;
struct dirent *de;
struct stat s;
d = opendir("/");
while((de = readdir(d)) != NULL)
{
        stat(de->d_name,&s);
        ...
}
-----------------------------------------------
In order to accomplish the purpose, we have to call stat function to 
distinguish file and directories, i.e. stat(de->d_name, &s), however, this 
does makes speep slow.

Suppose that we have nearly 10000 files on our sd card root directory, so 
root directory has entries:
..
.
f1
f2
f3
......
f10000
......
Then for each file, take the file f3 for example, funciton stat should 
compare de->d_name(i.e. f3) with f1,  then with f2, then with f3 and entry 
found, function stat sets file f3's total info into the stat buffer i.e. 
s, we could check s.st_mode to see f3 is a file or a directory, so f3 
needs 3 times comparing. However, as far as f10000 is concerned, it needs 
10000 times comparing, and all this done requires almost several minutes 
and it is too slow, the more files the worse. Actually, for file fx(x is a 
numeric), application program should do x+1 times comparing(In our 
application, we have to check some certain type files, after function 
stat's comparing, we could pass the directories, buf we still have to 
compare its name to decide whether it is the correct type), we can't bear 
the slow speed.

On linux, readdir's mannual 3, we found that the struct dirent has another 
field named d_type, which is not in eCos(eCos has only one field d_name). 
So on linux, when readdir returns a struct dirent, we could check the 
d_type field to pass the direcoties,  without function stat's comparing 
names with all the entries. Of course, using of d_type will harm the 
portability of our program, but we could accept this fact.

Hope that we express our problem clearly, and we want to know how could we 
solve the problem in eCos. Could you please give us some advice or could 
we add the d_type field to struct dirent in eCos? We are glad to offer a 
patch. Thank you.

Best wishes
Sincerely Yours


***********************************************
Beijing Sunnorth eCos Maintainer Group

Maintainers:
liqin@sunnorth.com.cn
wanghui@sunnorth.com.cn
taiyun@sunnorth.com.cn
yxinghua@sunnorth.com.cn

Bejing Sunnorth Electronic Technology Co.,LTD
***********************************************

Re: [ECOS] A question about function readdir and stat

[Andrew Lunn]

On Fri, Mar 28, 2008 at 05:04:04PM +0800, eCos@sunnorth.com.cn wrote:
> Dear all??
> We have to do a total sd card scanning and check for some certain type of 
> files in our application, some application code is as below:
> 
> -----------------------------------------------
> DIR *d;
> struct dirent *de;
> struct stat s;
> d = opendir("/");
> while((de = readdir(d)) != NULL)
> {
>         stat(de->d_name,&s);
>         ...
> }
> -----------------------------------------------
> In order to accomplish the purpose, we have to call stat function to 
> distinguish file and directories, i.e. stat(de->d_name, &s), however, this 
> does makes speep slow.
> 
> Suppose that we have nearly 10000 files on our sd card root directory, so 
> root directory has entries:
> ..
> .
> f1
> f2
> f3
> ......
> f10000
> ......

Why not impose a directory structure. eg my camera has a directory for
every 100 images. That would mean you only need a maximum of 100
comparisons for the last file in the directory and for 10000 files you
need 100 directories.

Another option which would give a speed up is not to use numbers, but
a more dense naming scheme. eg fAA, fAB, fAC, ..., fAZ, fAa, ... fAz,
You still end up doing 10000 comparisons, but the strings you are
comparing are shorted so there is less work to do.

> On linux, readdir's mannual 3, we found that the struct dirent has another 
> field named d_type, which is not in eCos(eCos has only one field d_name). 
> So on linux, when readdir returns a struct dirent, we could check the 
> d_type field to pass the direcoties,  without function stat's comparing 
> names with all the entries. Of course, using of d_type will harm the 
> portability of our program, but we could accept this fact.

I assume you are using fatfs? Looking at fatfs_fo_dirread, this would
be easy to implement. You have the information you need in dentry. So
just add the new member d_type and fill it in.

     Andrew

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss