[ECOS] Re: fatfs help.

[ECOS] Re: fatfs help.

[Savin Zlobec]

David Brennan wrote:

> I am using fatfs on i386 pc ide device and am having a problem. I was 
> wondering if you could point me in the right direction for a solution.
>
> First, I am not a FS genius, nor a FAT expert. So please bear with me.
>
> If I unlink a file in eCos, a "dir" from within eCos shows the file 
> missing.
> When I reboot the system into FreeDos (0.9rc5, if that matters) the 
> file is still in the directory listing.
> Rebooting back into the eCos app still shows the file missing (it's 
> not a cached fat table issue).
>
> Is the problem most likely eCos fatfs, pc-ide driver, FreeDos?
>
> Any tips on how to narrow the problem down would be greatly 
> appreciated. I know I could turn on some of the tracing in the fatfs, 
> but I still wouldn't be able to read/understand that. But if someone 
> else can decode it, I can provide the data.
>
> The fat file system was formatted originally with MS-DOS 6.22. The 
> drive has been converted to FreeDos 0.9rc5 kernel, but never 
> reformatted. It is a 192MB CF card.


Did you try to read the CF in anodher system (linux or windows) ?

Does this error also occour with an CF formatted from linux or windows ?

Do you use the fatfs from CVS or some newer version (I posted some 
patches on ecos-patches list) ?

Can you provide a test case in form of a small program wich creates and 
deletes a file ?

Alternatively if you send me the trace output I'll check it.

savin

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Re: fatfs help.

[David Brennan]

I honestly have not done much (any) troubleshooting on this yet. And I 
am on vacation this week, so I do not have access to the hardware. 
However, here are the answers to you questions.
1. No, I have not tried reading the CF in windows/Linux.
2. I have only played with the one CF so far and it was formatted MS-DOS 
6.22. (Or may have been pre-formatted, I actually don't remember now. I 
know it used to boot to MS-DOS 6.22.)
3. The code is from CVS. I saw your patches, but I figured I'd wait 
until someone decides to commit them before trying them out. Unless you 
think this may be the problem.
4. I don't have the code at home. I can provide a code sample next week.

Thanks for your advice. I was definitely planning on getting around to 
testing the more when I get back next week.

Thanks
David Brennan

Savin Zlobec wrote:

> David Brennan wrote:
>
>> I am using fatfs on i386 pc ide device and am having a problem. I was 
>> wondering if you could point me in the right direction for a solution.
>>
>> First, I am not a FS genius, nor a FAT expert. So please bear with me.
>>
>> If I unlink a file in eCos, a "dir" from within eCos shows the file 
>> missing.
>> When I reboot the system into FreeDos (0.9rc5, if that matters) the 
>> file is still in the directory listing.
>> Rebooting back into the eCos app still shows the file missing (it's 
>> not a cached fat table issue).
>>
>> Is the problem most likely eCos fatfs, pc-ide driver, FreeDos?
>>
>> Any tips on how to narrow the problem down would be greatly 
>> appreciated. I know I could turn on some of the tracing in the fatfs, 
>> but I still wouldn't be able to read/understand that. But if someone 
>> else can decode it, I can provide the data.
>>
>> The fat file system was formatted originally with MS-DOS 6.22. The 
>> drive has been converted to FreeDos 0.9rc5 kernel, but never 
>> reformatted. It is a 192MB CF card.
>
>
>
> Did you try to read the CF in anodher system (linux or windows) ?
>
> Does this error also occour with an CF formatted from linux or windows ?
>
> Do you use the fatfs from CVS or some newer version (I posted some 
> patches on ecos-patches list) ?
>
> Can you provide a test case in form of a small program wich creates 
> and deletes a file ?
>
> Alternatively if you send me the trace output I'll check it.
>
> savin
>

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Re: fatfs help.

[David Brennan]

Ok, I feel pretty stupid now. I actually only have a partial problem. My 
original problem was actually that I was mounting the VME hard drive 
instead of the CF under eCos. (They had similar original file structures 
so it was not immediately noticed.)
At least now everyone's file listing matches. But I still have a problem 
unlinking files from the hard drive under eCos.
The fileio1 test was running fine (up until it got to max file, which on 
a 20 Gig HD is not a good test to run). But when I deleted file.max from 
the HD with my application, it showed the file deleted. But it was still 
there under FreeDOS and eCos after a reboot. The other files 
created/deleted under fileio1 worked just fine. One difference is 
file.max was never actually correctly closed. Its file size is listed as 
0 bytes.

I will re-run fileio1 (without file.max) on the CF tomorrow if time permits.


David Brennan wrote:

> I honestly have not done much (any) troubleshooting on this yet. And I 
> am on vacation this week, so I do not have access to the hardware. 
> However, here are the answers to you questions.
> 1. No, I have not tried reading the CF in windows/Linux.
> 2. I have only played with the one CF so far and it was formatted 
> MS-DOS 6.22. (Or may have been pre-formatted, I actually don't 
> remember now. I know it used to boot to MS-DOS 6.22.)
> 3. The code is from CVS. I saw your patches, but I figured I'd wait 
> until someone decides to commit them before trying them out. Unless 
> you think this may be the problem.
> 4. I don't have the code at home. I can provide a code sample next week.
>
> Thanks for your advice. I was definitely planning on getting around to 
> testing the more when I get back next week.
>
> Thanks
> David Brennan
>
> Savin Zlobec wrote:
>
>> David Brennan wrote:
>>
>>> I am using fatfs on i386 pc ide device and am having a problem. I 
>>> was wondering if you could point me in the right direction for a 
>>> solution.
>>>
>>> First, I am not a FS genius, nor a FAT expert. So please bear with me.
>>>
>>> If I unlink a file in eCos, a "dir" from within eCos shows the file 
>>> missing.
>>> When I reboot the system into FreeDos (0.9rc5, if that matters) the 
>>> file is still in the directory listing.
>>> Rebooting back into the eCos app still shows the file missing (it's 
>>> not a cached fat table issue).
>>>
>>> Is the problem most likely eCos fatfs, pc-ide driver, FreeDos?
>>>
>>> Any tips on how to narrow the problem down would be greatly 
>>> appreciated. I know I could turn on some of the tracing in the 
>>> fatfs, but I still wouldn't be able to read/understand that. But if 
>>> someone else can decode it, I can provide the data.
>>>
>>> The fat file system was formatted originally with MS-DOS 6.22. The 
>>> drive has been converted to FreeDos 0.9rc5 kernel, but never 
>>> reformatted. It is a 192MB CF card.
>>
>>
>>
>>
>> Did you try to read the CF in anodher system (linux or windows) ?
>>
>> Does this error also occour with an CF formatted from linux or windows ?
>>
>> Do you use the fatfs from CVS or some newer version (I posted some 
>> patches on ecos-patches list) ?
>>
>> Can you provide a test case in form of a small program wich creates 
>> and deletes a file ?
>>
>> Alternatively if you send me the trace output I'll check it.
>>
>> savin
>>
>

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Re: fatfs help.

[David Brennan]

More info:
fileio1.c works great on the CF (minus the file.max test).

However, I still cannot delete files permanently from within my 
application. I believe the difference is that the end of fileio1.c code 
umounts root. I do not have that luxury. I have seen posts referencing 
needing to do an fsync(). But how do I fsync the directory after 
deleting a file?

Thanks
David


David Brennan wrote:

> Ok, I feel pretty stupid now. I actually only have a partial problem. 
> My original problem was actually that I was mounting the VME hard 
> drive instead of the CF under eCos. (They had similar original file 
> structures so it was not immediately noticed.)
> At least now everyone's file listing matches. But I still have a 
> problem unlinking files from the hard drive under eCos.
> The fileio1 test was running fine (up until it got to max file, which 
> on a 20 Gig HD is not a good test to run). But when I deleted file.max 
> from the HD with my application, it showed the file deleted. But it 
> was still there under FreeDOS and eCos after a reboot. The other files 
> created/deleted under fileio1 worked just fine. One difference is 
> file.max was never actually correctly closed. Its file size is listed 
> as 0 bytes.
>
> I will re-run fileio1 (without file.max) on the CF tomorrow if time 
> permits.
>
>
> David Brennan wrote:
>
>> I honestly have not done much (any) troubleshooting on this yet. And 
>> I am on vacation this week, so I do not have access to the hardware. 
>> However, here are the answers to you questions.
>> 1. No, I have not tried reading the CF in windows/Linux.
>> 2. I have only played with the one CF so far and it was formatted 
>> MS-DOS 6.22. (Or may have been pre-formatted, I actually don't 
>> remember now. I know it used to boot to MS-DOS 6.22.)
>> 3. The code is from CVS. I saw your patches, but I figured I'd wait 
>> until someone decides to commit them before trying them out. Unless 
>> you think this may be the problem.
>> 4. I don't have the code at home. I can provide a code sample next week.
>>
>> Thanks for your advice. I was definitely planning on getting around 
>> to testing the more when I get back next week.
>>
>> Thanks
>> David Brennan
>>
>> Savin Zlobec wrote:
>>
>>> David Brennan wrote:
>>>
>>>> I am using fatfs on i386 pc ide device and am having a problem. I 
>>>> was wondering if you could point me in the right direction for a 
>>>> solution.
>>>>
>>>> First, I am not a FS genius, nor a FAT expert. So please bear with me.
>>>>
>>>> If I unlink a file in eCos, a "dir" from within eCos shows the file 
>>>> missing.
>>>> When I reboot the system into FreeDos (0.9rc5, if that matters) the 
>>>> file is still in the directory listing.
>>>> Rebooting back into the eCos app still shows the file missing (it's 
>>>> not a cached fat table issue).
>>>>
>>>> Is the problem most likely eCos fatfs, pc-ide driver, FreeDos?
>>>>
>>>> Any tips on how to narrow the problem down would be greatly 
>>>> appreciated. I know I could turn on some of the tracing in the 
>>>> fatfs, but I still wouldn't be able to read/understand that. But if 
>>>> someone else can decode it, I can provide the data.
>>>>
>>>> The fat file system was formatted originally with MS-DOS 6.22. The 
>>>> drive has been converted to FreeDos 0.9rc5 kernel, but never 
>>>> reformatted. It is a 192MB CF card.
>>>
>>>
>>>
>>>
>>>
>>> Did you try to read the CF in anodher system (linux or windows) ?
>>>
>>> Does this error also occour with an CF formatted from linux or 
>>> windows ?
>>>
>>> Do you use the fatfs from CVS or some newer version (I posted some 
>>> patches on ecos-patches list) ?
>>>
>>> Can you provide a test case in form of a small program wich creates 
>>> and deletes a file ?
>>>
>>> Alternatively if you send me the trace output I'll check it.
>>>
>>> savin
>>>
>>
>

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Re: fatfs help.

[Andrew Lunn]

On Thu, Sep 16, 2004 at 08:31:49PM -0700, David Brennan wrote:
> More info:
> fileio1.c works great on the CF (minus the file.max test).
> 
> However, I still cannot delete files permanently from within my 
> application. I believe the difference is that the end of fileio1.c code 
> umounts root. I do not have that luxury. I have seen posts referencing 
> needing to do an fsync(). But how do I fsync the directory after 
> deleting a file?

Savin is the expert in this area, but he is on Holiday at the moment.

Basically if anybody can pull the disk at any time, you have to expect
some level of corruption. You can however minimise this. You basically
need the filesystem to use write through caching. At the moment you
cannot control this. You need to look at all the operations in fatfs.c
and decide which ones can change what should be on the disk. To these
add code something like:

#ifdef CYGOPT_FS_FAT_WRTIETHROUGH
        cyg_blib_sync(&disk->blib);
#endif
 
This will flush the block layer cache. 

This is a step forward, but their is still room for improvement. The
block cache does not keep track of order information. You first remove
the file from the directory inode and then free the blocks back into
the pool. Doing it this way it does not matter too much if the disk is
pulled part way through the operation. The file has gone from the
directory so if half the blocks are back in the pool and half are
still allocated is not so importent. There is no user visible
corruption. But the current code does not maintain ordering
information, so it code free the blocks and never get around to
removing the file from the directory inode.

You have to decide how much effort you want to spend on this vs how
well you can train you users to not pull the disk until they are told
it is safe to do so.

        Andrew

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Re: fatfs help.

[David Brennan]

Andrew Lunn wrote:

>On Thu, Sep 16, 2004 at 08:31:49PM -0700, David Brennan wrote:
>  
>
>>More info:
>>fileio1.c works great on the CF (minus the file.max test).
>>
>>However, I still cannot delete files permanently from within my 
>>application. I believe the difference is that the end of fileio1.c code 
>>umounts root. I do not have that luxury. I have seen posts referencing 
>>needing to do an fsync(). But how do I fsync the directory after 
>>deleting a file?
>>    
>>
>
>Savin is the expert in this area, but he is on Holiday at the moment.
>
>Basically if anybody can pull the disk at any time, you have to expect
>some level of corruption. You can however minimise this. You basically
>need the filesystem to use write through caching. At the moment you
>cannot control this. You need to look at all the operations in fatfs.c
>and decide which ones can change what should be on the disk. To these
>add code something like:
>
>#ifdef CYGOPT_FS_FAT_WRTIETHROUGH
>        cyg_blib_sync(&disk->blib);
>#endif
> 
>This will flush the block layer cache. 
>
>This is a step forward, but their is still room for improvement. The
>block cache does not keep track of order information. You first remove
>the file from the directory inode and then free the blocks back into
>the pool. Doing it this way it does not matter too much if the disk is
>pulled part way through the operation. The file has gone from the
>directory so if half the blocks are back in the pool and half are
>still allocated is not so importent. There is no user visible
>corruption. But the current code does not maintain ordering
>information, so it code free the blocks and never get around to
>removing the file from the directory inode.
>
>You have to decide how much effort you want to spend on this vs how
>well you can train you users to not pull the disk until they are told
>it is safe to do so.
>
>        Andrew
>
>
>  
>
Thanks for the advice, I will look into this. Our problem is not the 
disk being removed so much as the processor being rebooted at some 
future time. If I knew the delete would eventually be written to disk 
without an unmount, that would be fine. Or if there was a primitive that 
I could call to manually flush the write, that would also be acceptable. 
(I don't delete files very often, and they can only be deleted from one 
thread. However, there is another thread that could conceivably be 
writing to disk. So, if I wanted to unmount/remount the file system, I 
could with only one semaphore. But that sure seems like an in efficient 
way of doing it.)

Thanks
David

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss

Re: [ECOS] Re: fatfs help.

[Savin Zlobec]

David Brennan wrote:

>
>
> Andrew Lunn wrote:
>
>> On Thu, Sep 16, 2004 at 08:31:49PM -0700, David Brennan wrote:
>>  
>>
>>> More info:
>>> fileio1.c works great on the CF (minus the file.max test).
>>>
>>> However, I still cannot delete files permanently from within my 
>>> application. I believe the difference is that the end of fileio1.c 
>>> code umounts root. I do not have that luxury. I have seen posts 
>>> referencing needing to do an fsync(). But how do I fsync the 
>>> directory after deleting a file?
>>>   
>>
>>
>> Savin is the expert in this area, but he is on Holiday at the moment.
>>
>> Basically if anybody can pull the disk at any time, you have to expect
>> some level of corruption. You can however minimise this. You basically
>> need the filesystem to use write through caching. At the moment you
>> cannot control this. You need to look at all the operations in fatfs.c
>> and decide which ones can change what should be on the disk. To these
>> add code something like:
>>
>> #ifdef CYGOPT_FS_FAT_WRTIETHROUGH
>>        cyg_blib_sync(&disk->blib);
>> #endif
>>
>> This will flush the block layer cache.
>> This is a step forward, but their is still room for improvement. The
>> block cache does not keep track of order information. You first remove
>> the file from the directory inode and then free the blocks back into
>> the pool. Doing it this way it does not matter too much if the disk is
>> pulled part way through the operation. The file has gone from the
>> directory so if half the blocks are back in the pool and half are
>> still allocated is not so importent. There is no user visible
>> corruption. But the current code does not maintain ordering
>> information, so it code free the blocks and never get around to
>> removing the file from the directory inode.
>>
>> You have to decide how much effort you want to spend on this vs how
>> well you can train you users to not pull the disk until they are told
>> it is safe to do so.
>>
>>        Andrew
>>
>>
>>  
>>
> Thanks for the advice, I will look into this. Our problem is not the 
> disk being removed so much as the processor being rebooted at some 
> future time. If I knew the delete would eventually be written to disk 
> without an unmount, that would be fine. Or if there was a primitive 
> that I could call to manually flush the write, that would also be 
> acceptable. (I don't delete files very often, and they can only be 
> deleted from one thread. However, there is another thread that could 
> conceivably be writing to disk. So, if I wanted to unmount/remount the 
> file system, I could with only one semaphore. But that sure seems like 
> an in efficient way of doing it.)

Syncing the fs can be done through cyg_fs_setinfo, by defining  a new 
key (something like FILE_INFO_SYNC)
and extending the fatfs.c:fatfs_setinfo() with

case FILE_INFO_SYNC:
    err = cyg_blib_sync(&disk->blib);
    if (ENOERR != err)
       return err;
    break;

This (and the fs usage) should already be implemented, but we just 
couldn't decide how
and where to place the generic and fs specific keys with accompanying 
structures (if any) :-( .

savin

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss