zp_man-db-update-index.dash creates C:\cygwin64\%SystemDrive%

zp_man-db-update-index.dash creates C:\cygwin64\%SystemDrive%

[Миронов Леонид Владимирович]

Recently when cygwin setup runs windows 10 started creating "C:\cygwin64\%SystemDrive%\ProgramData\Microsoft\Windows\Caches" folder - exactly like that, with %SystemDrive% not expanded, containing the following files



cversions.2.db

{104937D1-3D41-4EB0-BE48-1725A96CC3A1}.2.ver0x0000000000000001.db

{42542F41-6950-48FC-BDFB-3961A7C4A653}.2.ver0x0000000000000001.db

{6AF0698E-D558-4F6E-9B3C-3716689AF493}.2.ver0x0000000000000001.db

{DDF571F2-BE98-426D-8288-1A9A39C3FDA2}.2.ver0x0000000000000001.db

{FAC7F33B-D66E-413B-9358-52C759023A9D}.2.ver0x0000000000000001.db



and logs a number of similar messages to application windows log from Property System


Omitted duplicate property.
Keeping: 'Microsoft.OneNote.TaggedNotes' ({641064BA-9329-47E6-8F36-5FA81AA461A0} 3)  Publisher: 'Microsoft'  Product: 'Windows'  URL: 'windowspropertydescriptions'
Omitting: 'Microsoft.OneNote.TaggedNotes' ({641064BA-9329-47E6-8F36-5FA81AA461A0} 3)  Publisher: 'Microsoft'  Product: 'OneNote'  URL: 'custom.propdesc'



Turned out that the culprit is the /etc/postinstall/zp_man-db-update-index.dash script which runs at the end of every setup session, although I couldn't figure why: it just runs mandb with some fancy redirection and nothing untoward happens when it is run manually and with admin privileges, but the folder in question was never created until I ran mandb to create man index which for some reason was not created automatically during installation, and when /var/cache/man/index.db is removed which effectively disables this script this folder is not created. I am baffled.

Re: zp_man-db-update-index.dash creates C:\cygwin64\%SystemDrive%

[Brian Inglis]

On 2021-11-24 02:06, Миронов Леонид Владимирович via Cygwin wrote:
> Recently when cygwin setup runs windows 10 started creating "C:\cygwin64\%SystemDrive%\ProgramData\Microsoft\Windows\Caches" folder - exactly like that, with %SystemDrive% not expanded, containing the following files
> cversions.2.db
> {104937D1-3D41-4EB0-BE48-1725A96CC3A1}.2.ver0x0000000000000001.db
> {42542F41-6950-48FC-BDFB-3961A7C4A653}.2.ver0x0000000000000001.db
> {6AF0698E-D558-4F6E-9B3C-3716689AF493}.2.ver0x0000000000000001.db
> {DDF571F2-BE98-426D-8288-1A9A39C3FDA2}.2.ver0x0000000000000001.db
> {FAC7F33B-D66E-413B-9358-52C759023A9D}.2.ver0x0000000000000001.db
> and logs a number of similar messages to application windows log from Property System
> Omitted duplicate property.
> Keeping: 'Microsoft.OneNote.TaggedNotes' ({641064BA-9329-47E6-8F36-5FA81AA461A0} 3)  Publisher: 'Microsoft'  Product: 'Windows'  URL: 'windowspropertydescriptions'
> Omitting: 'Microsoft.OneNote.TaggedNotes' ({641064BA-9329-47E6-8F36-5FA81AA461A0} 3)  Publisher: 'Microsoft'  Product: 'OneNote'  URL: 'custom.propdesc'
> Turned out that the culprit is the /etc/postinstall/zp_man-db-update-index.dash script which runs at the end of every setup session, although I couldn't figure why: it just runs mandb with some fancy redirection and nothing untoward happens when it is run manually and with admin privileges, but the folder in question was never created until I ran mandb to create man index which for some reason was not created automatically during installation, and when /var/cache/man/index.db is removed which effectively disables this script this folder is not created. I am baffled.

I noticed that too but could never trace it back to a script. Good work.

That may happen because mandb does some undocumented processing, 
including scanning the system for "cat#..." directories and "handles" 
them, which may include moving them to / root so they will be noticed if 
they are not part of a man hierarchy, or removes empty "cat#..." 
directories.

Windows uses a number of cat... folders for downloaded components which 
man can mess around with.
I submitted a mandb bug report requesting that undocumented behaviour be 
documented, and mandb limit its searches, or suppress cleanup (e.g. -s, 
--no-straycats Do not spend time looking for or adding information to 
the databases regarding stray cats [whatever they are, also 
undocumented, possibly relevant]) or suppressed under Windows, as it 
could affect Windows components or the system:

https://www.majorgeeks.com/content/page/catroot_and_catroot2_folder_explained.html

but I don't remember if it was even addressed.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: zp_man-db-update-index.dash creates C:\cygwin64\%SystemDrive%

[Achim Gratz]

Миронов Леонид Владимирович via Cygwin writes:
> Recently when cygwin setup runs windows 10 started creating
> "C:\cygwin64\%SystemDrive%\ProgramData\Microsoft\Windows\Caches"
> folder - exactly like that, with %SystemDrive% not expanded,
> containing the following files
[…]
> Turned out that the culprit is the
> /etc/postinstall/zp_man-db-update-index.dash script which runs at the
> end of every setup session, although I couldn't figure why: it just
> runs mandb with some fancy redirection and nothing untoward happens
> when it is run manually and with admin privileges, but the folder in
> question was never created until I ran mandb to create man index which
> for some reason was not created automatically during installation, and
> when /var/cache/man/index.db is removed which effectively disables
> this script this folder is not created. I am baffled.

Me too.  Apparently this is some bug deep in the bowels of Windows that
triggers when neither SystemDrive nor ProgramData are defined in the
environment and a new console session gets created.  It seems I can work
around getting the extra directories created by defining SystemDrive in
a certain nonsensical way, but it really is an ugly hack; I'll have to
see if there's a better fix.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: zp_man-db-update-index.dash creates C:\cygwin64\%SystemDrive%

[Achim Gratz]

Brian Inglis writes:
> I submitted a mandb bug report requesting that undocumented behaviour
> be documented, and mandb limit its searches, or suppress cleanup

What made you think this is related to mandb itself?


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: zp_man-db-update-index.dash creates C:\cygwin64\%SystemDrive%

[Brian Inglis]

On 2021-11-24 11:19, Achim Gratz wrote:
> Leonid Vladimirovič Mironov via Cygwin writes:
>> Recently when cygwin setup runs windows 10 started creating
>> "C:\cygwin64\%SystemDrive%\ProgramData\Microsoft\Windows\Caches"
>> folder - exactly like that, with %SystemDrive% not expanded,
>> containing the following files
> […]
>> Turned out that the culprit is the
>> /etc/postinstall/zp_man-db-update-index.dash script which runs at the
>> end of every setup session, although I couldn't figure why: it just
>> runs mandb with some fancy redirection and nothing untoward happens
>> when it is run manually and with admin privileges, but the folder in
>> question was never created until I ran mandb to create man index which
>> for some reason was not created automatically during installation, and
>> when /var/cache/man/index.db is removed which effectively disables
>> this script this folder is not created. I am baffled.
> 
> Me too.  Apparently this is some bug deep in the bowels of Windows that
> triggers when neither SystemDrive nor ProgramData are defined in the
> environment and a new console session gets created.  It seems I can work
> around getting the extra directories created by defining SystemDrive in
> a certain nonsensical way, but it really is an ugly hack; I'll have to
> see if there's a better fix.

Problem mentioned by me some time ago, may be related to this, from 
undocumented mandb handling of Windows localization catalog folders:

https://cygwin.com/pipermail/cygwin/2018-January/235776.html

which may include moving "stray cats" (also undocumented) to / [Cygwin] 
root so they get noticed and handled?

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: zp_man-db-update-index.dash creates C:\cygwin64\%SystemDrive%

[Brian Inglis]

On 2021-11-24 11:23, Achim Gratz wrote:
> Brian Inglis writes:
>> I submitted a mandb bug report requesting that undocumented behaviour
>> be documented, and mandb limit its searches, or suppress cleanup
> 
> What made you think this is related to mandb itself?

Problem mentioned by me some time ago, may be related to this, from 
undocumented mandb handling of Windows localization catalog folders:

https://cygwin.com/pipermail/cygwin/2018-January/235776.html

which may include moving "stray cats" (also undocumented) to / [Cygwin] 
root so they get noticed and handled?

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: zp_man-db-update-index.dash creates C:\cygwin64\%SystemDrive%

[Achim Gratz]

Brian Inglis writes:
> Problem mentioned by me some time ago, may be related to this, from
> undocumented mandb handling of Windows localization catalog folders:

It isn't, which could have easily been tested.

> https://cygwin.com/pipermail/cygwin/2018-January/235776.html
>
> which may include moving "stray cats" (also undocumented) to /
> [Cygwin] root so they get noticed and handled?

That's most likely caused by configuring the C:\ drive as the root of
Cygwin and/or additional configurations not mentioned by the OP.  You'd
have to run man-db in debugging / dry-run mode to see how it gets to the
conclusion that it should scan all those sub-directories and
sub-sub-directories in the root.  It doesn't do that by default and I am
unable to reproduce it.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: zp_man-db-update-index.dash creates C:\cygwin64\%SystemDrive%

[Brian Inglis]

On 2021-11-24 12:54, Achim Gratz wrote:
> Brian Inglis writes:
>> Problem mentioned by me some time ago, may be related to this, from
>> undocumented mandb handling of Windows localization catalog folders:

> It isn't, which could have easily been tested.

Are you saying that the <CygwinRoot>/%SystemDrive% issue is unrelated to 
mandb execution, contrary to the information from the OP?

How can it be tested, as I would like to do so, as I get those symptoms.

I found that when setup is executed, as the PATH sanitized for scripts 
includes cygpath -S, mandb appears to search directories in the Windows 
PATH, for cat... directories, may remove the contents, even when they 
are not manual pages, and may perform other poorly or undocumented 
operations.

>> https://cygwin.com/pipermail/cygwin/2018-January/235776.html
>>
>> which may include moving "stray cats" (also undocumented) to /
>> [Cygwin] root so they get noticed and handled?

> That's most likely caused by configuring the C:\ drive as the root of
> Cygwin and/or additional configurations not mentioned by the OP. You'd
> have to run man-db in debugging / dry-run mode to see how it gets to the
> conclusion that it should scan all those sub-directories and
> sub-sub-directories in the root. It doesn't do that by default and I am
> unable to reproduce it.

That is not the case in either the OP C:\cygwin64\ or my own setup.

It appears that mandb climbs the tree trying to find related bin and man 
dirs.

It may be having shared man and "stray" cat dirs for other Windows 
package commands, cmd, Windows utilities, and systems (-m) under 
/proc/cygdrive/c/usr/local/.
I don't know any other way I could keep the system level shared contents 
accessible while isolating it.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: zp_man-db-update-index.dash creates C:\cygwin64\%SystemDrive%

[Achim Gratz]

Brian Inglis writes:
> On 2021-11-24 12:54, Achim Gratz wrote:
>> Brian Inglis writes:
>>> Problem mentioned by me some time ago, may be related to this, from
>>> undocumented mandb handling of Windows localization catalog folders:
>
>> It isn't, which could have easily been tested.
>
> Are you saying that the <CygwinRoot>/%SystemDrive% issue is unrelated
> to mandb execution, contrary to the information from the OP?

Yes, all the time, and now again.

> How can it be tested, as I would like to do so, as I get those symptoms.

mkdir -p /tmp/test/{s,p}d2 /tmp/test/{s,p}d3/sub
cd /tmp/test
env -i                          /usr/bin/cygstart --hide /usr/bin/dash -c echo
env -i SystemDrive=sd1          /usr/bin/cygstart --hide /usr/bin/dash -c echo
env -i ProgramData=pd1          /usr/bin/cygstart --hide /usr/bin/dash -c echo
env -i SystemDrive=sd2          /usr/bin/cygstart --hide /usr/bin/dash -c echo
env -i ProgramData=pd2          /usr/bin/cygstart --hide /usr/bin/dash -c echo
env -i SystemDrive=sd3/sub      /usr/bin/cygstart --hide /usr/bin/dash -c echo
env -i ProgramData=pd3/sub      /usr/bin/cygstart --hide /usr/bin/dash -c echo
env -i SystemDrive=sd3/nil      /usr/bin/cygstart --hide /usr/bin/dash -c echo
env -i ProgramData=pd3/nil      /usr/bin/cygstart --hide /usr/bin/dash -c echo
env -i SystemDrive=sd3/void/nil /usr/bin/cygstart --hide /usr/bin/dash -c echo
env -i ProgramData=pd3/void/nil /usr/bin/cygstart --hide /usr/bin/dash -c echo
ls -R
# cd ; rm -fr /tmp/test

The behaviour with absolute paths is left as an exercise for the reader.

> I found that when setup is executed, as the PATH sanitized for scripts
> includes cygpath -S, mandb appears to search directories in the
> Windows PATH, for cat... directories, may remove the contents, even
> when they are not manual pages, and may perform other poorly or
> undocumented operations.

Again, in no variant of trying to reproduce this was I ever successful
in seeing that happen.  So either this problem is no longer occuring or
there is some extra condition for it to trigger.

> It appears that mandb climbs the tree trying to find related bin and
> man dirs.

There's no code in mandb that obviously does this, so you'll have to
provide some evidence of when and where it happens.  And it should
already show up in the output of 'manpath -dc' I think, since I've only
ever seen mandb removing something that is in catdirs.  So for instance
if I drop bogus directories cat9 and de_AT into /var/cache/man (which
don't have a corresponding manpath obviously), I'll see:

--8<---------------cut here---------------start------------->8---
Removing obsolete cat directory /var/cache/man/cat9...
Seen mandir for /var/cache/man/ca; not deleting
Seen mandir for /var/cache/man/cs; not deleting
Seen mandir for /var/cache/man/da; not deleting
Seen mandir for /var/cache/man/da.ISO8859-1; not deleting
Seen mandir for /var/cache/man/da.UTF-8; not deleting
Seen mandir for /var/cache/man/de; not deleting
Seen mandir for /var/cache/man/de.ISO8859-1; not deleting
Seen mandir for /var/cache/man/de.UTF-8; not deleting
Removing obsolete cat directory /var/cache/man/de_AT...
--8<---------------cut here---------------end--------------->8---

Which is correct since /var/cache/man is the only catdir on my system.
If I drop these anywhere else in any of the directory in PATH mandb will
not touch them (in fact it never bothers to look there).

> It may be having shared man and "stray" cat dirs for other Windows
> package commands, cmd, Windows utilities, and systems (-m) under 
> /proc/cygdrive/c/usr/local/.
> I don't know any other way I could keep the system level shared
> contents accessible while isolating it.

Again, if that's the problem I have not been able to reproduce it.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: zp_man-db-update-index.dash creates C:\cygwin64\%SystemDrive%

[Brian Inglis]

On 2021-11-25 11:51, Achim Gratz wrote:
> Brian Inglis writes:
>> On 2021-11-24 12:54, Achim Gratz wrote:
>>> Brian Inglis writes:
>>>> Problem mentioned by me some time ago, may be related to this, from
>>>> undocumented mandb handling of Windows localization catalog folders:
>>
>>> It isn't, which could have easily been tested.
>>
>> Are you saying that the <CygwinRoot>/%SystemDrive% issue is unrelated
>> to mandb execution, contrary to the information from the OP?
> 
> Yes, all the time, and now again.
> 
>> How can it be tested, as I would like to do so, as I get those symptoms.
> 
> mkdir -p /tmp/test/{s,p}d2 /tmp/test/{s,p}d3/sub
> cd /tmp/test
> env -i                          /usr/bin/cygstart --hide /usr/bin/dash -c echo
> env -i SystemDrive=sd1          /usr/bin/cygstart --hide /usr/bin/dash -c echo
> env -i ProgramData=pd1          /usr/bin/cygstart --hide /usr/bin/dash -c echo
> env -i SystemDrive=sd2          /usr/bin/cygstart --hide /usr/bin/dash -c echo
> env -i ProgramData=pd2          /usr/bin/cygstart --hide /usr/bin/dash -c echo
> env -i SystemDrive=sd3/sub      /usr/bin/cygstart --hide /usr/bin/dash -c echo
> env -i ProgramData=pd3/sub      /usr/bin/cygstart --hide /usr/bin/dash -c echo
> env -i SystemDrive=sd3/nil      /usr/bin/cygstart --hide /usr/bin/dash -c echo
> env -i ProgramData=pd3/nil      /usr/bin/cygstart --hide /usr/bin/dash -c echo
> env -i SystemDrive=sd3/void/nil /usr/bin/cygstart --hide /usr/bin/dash -c echo
> env -i ProgramData=pd3/void/nil /usr/bin/cygstart --hide /usr/bin/dash -c echo
> ls -R
> # cd ; rm -fr /tmp/test
> 
> The behaviour with absolute paths is left as an exercise for the reader.

What a mess! No sign of any obvious reports, except a decade old report 
of the same bug from C#:

http://jeremymurray.org/doku.php/blog:20110804_directories_popping_up_named_systemdrive

blaming the use of ShellExecute, which may have changed in cygutils 
cygstart, as the bug did not appear to occur until recently.
Patching setup to pass those env vars through to scripts would be one 
approach.
Patching cygstart to set those env vars if unset, using 
CSIDLs/KNOWNFOLDERIDs like cygpath -F is another.

>> I found that when setup is executed, as the PATH sanitized for scripts
>> includes cygpath -S, mandb appears to search directories in the
>> Windows PATH, for cat... directories, may remove the contents, even
>> when they are not manual pages, and may perform other poorly or
>> undocumented operations.
> 
> Again, in no variant of trying to reproduce this was I ever successful
> in seeing that happen.  So either this problem is no longer occuring or
> there is some extra condition for it to trigger.
> 
>> It appears that mandb climbs the tree trying to find related bin and
>> man dirs.
> 
> There's no code in mandb that obviously does this, so you'll have to
> provide some evidence of when and where it happens.  And it should
> already show up in the output of 'manpath -dc' I think, since I've only
> ever seen mandb removing something that is in catdirs.  So for instance
> if I drop bogus directories cat9 and de_AT into /var/cache/man (which
> don't have a corresponding manpath obviously), I'll see:
> 
> --8<---------------cut here---------------start------------->8---
> Removing obsolete cat directory /var/cache/man/cat9...
> Seen mandir for /var/cache/man/ca; not deleting
> Seen mandir for /var/cache/man/cs; not deleting
> Seen mandir for /var/cache/man/da; not deleting
> Seen mandir for /var/cache/man/da.ISO8859-1; not deleting
> Seen mandir for /var/cache/man/da.UTF-8; not deleting
> Seen mandir for /var/cache/man/de; not deleting
> Seen mandir for /var/cache/man/de.ISO8859-1; not deleting
> Seen mandir for /var/cache/man/de.UTF-8; not deleting
> Removing obsolete cat directory /var/cache/man/de_AT...
> --8<---------------cut here---------------end--------------->8---
> 
> Which is correct since /var/cache/man is the only catdir on my system.
> If I drop these anywhere else in any of the directory in PATH mandb will
> not touch them (in fact it never bothers to look there).
> 
>> It may be having shared man and "stray" cat dirs for other Windows
>> package commands, cmd, Windows utilities, and systems (-m) under
>> /proc/cygdrive/c/usr/local/.
>> I don't know any other way I could keep the system level shared
>> contents accessible while isolating it.
> 
> Again, if that's the problem I have not been able to reproduce it.

I'll create a new 0p_man-db-patch.dash script to add -d after mandb 
postinstall to produce diagnostic output, and get back.

I previously had a patch script to add around mandb in postinstall:

	/usr/bin/nohup ... /usr/bin/mandb ... &

which did not create any artifacts I believe but can not be sure of 
timing, and that could be an option for your script instead of cygstart?

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: zp_man-db-update-index.dash creates C:\cygwin64\%SystemDrive%

[ASSI]

Brian Inglis writes:
> I'll create a new 0p_man-db-patch.dash script to add -d after mandb
> postinstall to produce diagnostic output, and get back.

Just add that flag to the already provided postinstall script.

> I previously had a patch script to add around mandb in postinstall:
>
> 	/usr/bin/nohup ... /usr/bin/mandb ... &

That doesn't work when run from setup.  The process will abort when
setup exits.

> which did not create any artifacts I believe but can not be sure of
> timing, and that could be an option for your script instead of
> cygstart?

I'm not sure why you're harping on cygstart, but if you want to avoid it
just install the package that makes the mandb update synchronous.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds