64 bit Cygwin 1.7.18-12

64 bit Cygwin 1.7.18-12

[Corinna Vinschen]

Hi guys,


I just uploaded a new 64 bit Cygwin DLL.  This version fixes a few
problems, namely:

- Since Vista and the introduction of native symlinks, the OS has
  multiple ways to suppress symlink usage.  By default, remote symlinks
  are disallowed, or better, they are not evaluated and the OS returns
  an error instead.  This can be changed with the on-board fsutil
  utility.  Cygwin didn't yet handle the case that symlinks couldn't be
  read.  That's fixed now.  Cygwin returns ELOOP for unreadable
  symlinks.  ENOENT wouldn't work in this scenario.

- The wrong defines were set for the available build environment.  So
  far, _POSIX_V6_ILP32_OFFBIG was still 1, the others -1, which was only
  correct for the 32 bit environment.  Now on x86_64,
  _POSIX_V6_LP64_OFF64 and _POSIX_V6_LPBIG_OFFBIG are 1 instead.
  I changed confstr accordingly.

- getservbyname and getservbyport usually crashed on 64 bit.  The reason
  was that the servent structure on 64 bit Windows has reordered two
  members, one the port number, the other a pointer.  I have not the
  faintest idea what that was good for.  The Cygwin code duplicating the
  content to make it available across fork didn't take that into
  account, so it crashed instead.  Should be fixed now.

I appreciate testing and bug reports and... PATCHES!


Have fun,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

native symlink

[James Gregurich]

Why don't you add an API call and utility to actually convert an existing cygwin symlink into a native symlink.  I'll give you code that does the work. Cygwin already reads and uses the native symlinks. you might as well provide a way to create them.




On Mar 27, 2013, at 8:16 AM, Corinna Vinschen <corinna-cygwin@cygwin.com> wrote:

> - Since Vista and the introduction of native symlinks, the OS has
> multiple ways to suppress symlink usage.  By default, remote symlinks
> are disallowed, or better, they are not evaluated and the OS returns
> an error instead.  This can be changed with the on-board fsutil
> utility.  Cygwin didn't yet handle the case that symlinks couldn't be
> read.  That's fixed now.  Cygwin returns ELOOP for unreadable
> symlinks.  ENOENT wouldn't work in this scenario.
> 

Re: native symlink

[Larry Hall (Cygwin Developers)]

On 3/27/2013 5:53 PM, James Gregurich wrote:
> Why don't you add an API call and utility to actually convert an existing  > cygwin symlink into a native symlink. I'll give you code that does the work.
> Cygwin already reads and uses the native symlinks. you might as well provide
> a way to create them.

The main list is really the right place to make Cygwin feature requests.
Patches to the DLL can go to cygwin-patches.  Other code can go to the
main list as well.

As for why this hasn't been done before, some of it is just no one has
done it.  But perhaps more importantly is the question of the benefit.
There are (non-Cygwin) tools already that support this.  Also, since
native symlinks only work on NTFS, providing a utility to create them
opens us up to bug reports and questions about these limitations.  In
addition, there will be questions about why yet another "symlink"
utility exists and when should it be used.  But if you want to argue
for such a tool, please review the previous discussions in the email
archives (to make sure you're not covering the same ground again) and
bring up your proposal on the main list.  Be prepared for a fight
(ah, "robust discussion" ;-) ) though.

-- 
Larry

Re: native symlink

[Jeffrey Altman]

On 3/27/2013 6:41 PM, Larry Hall (Cygwin Developers) wrote:
> On 3/27/2013 5:53 PM, James Gregurich wrote:
>> Why don't you add an API call and utility to actually convert an
>> existing  > cygwin symlink into a native symlink. I'll give you code
>> that does the work.
>> Cygwin already reads and uses the native symlinks. you might as well
>> provide
>> a way to create them.
> 
> The main list is really the right place to make Cygwin feature requests.
> Patches to the DLL can go to cygwin-patches.  Other code can go to the
> main list as well.
> 
> As for why this hasn't been done before, some of it is just no one has
> done it.  But perhaps more importantly is the question of the benefit.
> There are (non-Cygwin) tools already that support this.  Also, since
> native symlinks only work on NTFS, providing a utility to create them
> opens us up to bug reports and questions about these limitations.  In
> addition, there will be questions about why yet another "symlink"
> utility exists and when should it be used.  But if you want to argue
> for such a tool, please review the previous discussions in the email
> archives (to make sure you're not covering the same ground again) and
> bring up your proposal on the main list.  Be prepared for a fight
> (ah, "robust discussion" ;-) ) though.

James,

In addition to searching the cygwin mailing list archive for discussions
on "reparse point" and "symlink" I would also recommend reading a recent
blog post I wrote on the subject of symlinks on Windows

 http://blog.secure-endpoints.com/2013/03/symbolic-links-on-windows.html

which summarizes what I learned when implementing support for creating,
using and deleting AFS symlinks within a legacy file system driver for
Microsoft Windows XP and above.

Larry's statement that native symlinks only work on NTFS is no longer
true.  They also work on Microsoft's ReFS as well as in OpenAFS 1.7.23
and above.  I will note that whereas native symlinks can be stored in
AFS, Cygwin symlinks cannot be due to the inability of AFS to store the
DOS SYSTEM attribute.

I look forward to participating in a discussion on the cygwin mailing list.

Jeffrey Altman

Re: native symlink

[Larry Hall]

On 3/31/2013 11:55 AM, Jeffrey Altman wrote:

<snip>

> Larry's statement that native symlinks only work on NTFS is no longer
> true.  They also work on Microsoft's ReFS as well as in OpenAFS 1.7.23
> and above.

True.  My original statement was a bit of an overstatement. ;-)  What is
also true is that Cygwin-native symlinks work with Cygwin tools on both NTFS
and FAT filesystems.  This was an important consideration when they were
originally implemented and continues to be important today.  Someday,
though, the benefits of this transparent handling of symlinks on FAT 
filesystems fade.   But let me stop there as I fear I'm getting dangerously
close to rehashing old topics. :-)

Thanks Jeffrey for the pointer to your blog post.

-- 
Larry

Re: native symlink

[James Gregurich]

On Mar 27, 2013, at 3:41 PM, Larry Hall (Cygwin Developers) <lhall@cygwin.com> wrote:

> On 3/27/2013 5:53 PM, James Gregurich wrote:
>> Why don't you add an API call and utility to actually convert an existing  > cygwin symlink into a native symlink. I'll give you code that does the work.
>> Cygwin already reads and uses the native symlinks. you might as well provide
>> a way to create them.
> 
> The main list is really the right place to make Cygwin feature requests.

I'm using this list because this is where I found the original debate on the subject. I wanted to add my voice.


> Patches to the DLL can go to cygwin-patches.  Other code can go to the
> main list as well.
> 
> As for why this hasn't been done before, some of it is just no one has
> done it.  

I have. I am actively using a version of cygwin with a modified DLL that implements use of native symlinks. I've previously described the work I've done. I've offered up my code. I never really got rational, technical reasons for why such support was a bad idea.



> But perhaps more importantly is the question of the benefit.

Here is a benefit…

My company maintains  cross-platform source code and and binary  library git repositories. These repositories heavily use symlinks to organize the files so that redundancies are minimized and libraries are presented in representations that allow clean integration with both Xcode and Visual Studio across multiple projects and developers' workflows. Since my version of cygwin properly sets up native symlinks, it all works cleanly with Visual Studio, Windows Explorer and every other Windows development tool I've worked with. The standard version of Cygwin does not produce repositories that work with Windows developer tools if those repositories contain symlinks.

I have given up on the possibly of Cygwin's developers supporting the modifying of the posix symlink() function (as I did) to create native symlinks when possible. However, my usage case is still accomplishable with just the ability to convert Cygwin symlinks to native symlinks after the fact through a separate utility and corresponding posix-level API call. So, now I shall lobby for that instead.




> There are (non-Cygwin) tools already that support this.  Also, since
> native symlinks only work on NTFS, providing a utility to create them
> opens us up to bug reports and questions about these limitations.  In
> addition, there will be questions about why yet another "symlink"
> utility exists and when should it be used.  But if you want to argue
> for such a tool, please review the previous discussions in the email
> archives (to make sure you're not covering the same ground again) and
> bring up your proposal on the main list.  Be prepared for a fight
> (ah, "robust discussion" ;-) ) though.


Frankly, I've never understood the arguments presented. As I said in the past, you ALREADY read native symlinks. Much if not all of the technical consternation I've seen discussed on the subject already exists in what you have today. Since you already READ the symlinks, surely it makes sense to offer more complete interoperability. Is not the purpose of Cygwin to allow unix software to interoperate with a Windows environment? Is not its purpose to allow useful work to be done on Windows using unix software? 

If you are going to limit yourself to features that won't cause reactions and objections, then exactly how to you plan to make progress?  ANY change disrupts someone. ANY change is going to cause tech support tickets. I'm sure these 64 bit changes that are being described will inconvenience someone out there, yet it needs to be done to keep up with the times.


That said…..

What I am lobbying for is not to rework the core symlink mechanism as I have suggested in the past. All I am lobbying for is a new symlink API call and a corresponding command line utility to convert an existing cygwin symlink to a native symlink if it is possible. I don't see how adding that ability would disrupt anyone or cause tech support headaches. People like me who need native symlinks can use this new functionality. Since we will already understand the limitations of native symlinks, we won't be likely to file tech support tickets frivolously. People who don't need native symlinks can safely ignore the new utility and API call.

Even the details of the code is worked out already. The code I have would just need to be massaged into a little different form. 


-James

Re: native symlink

[Christopher Faylor]

On Mon, Apr 01, 2013 at 12:24:51PM -0700, James Gregurich wrote:
>On Mar 27, 2013, at 3:41 PM, Larry Hall wrote:
>>On 3/27/2013 5:53 PM, James Gregurich wrote:
>>>Why don't you add an API call and utility to actually convert an
>>>existing > cygwin symlink into a native symlink.  I'll give you code
>>>that does the work.  Cygwin already reads and uses the native symlinks.
>>>you might as well provide a way to create them.
>>
>>The main list is really the right place to make Cygwin feature
>>requests.
>
>I'm using this list because this is where I found the original debate
>on the subject.  I wanted to add my voice.
>
>
>>Patches to the DLL can go to cygwin-patches.  Other code can go to the
>>main list as well.
>>
>>As for why this hasn't been done before, some of it is just no one has
>>done it.
>
>I have.  I am actively using a version of cygwin with a modified DLL
>that implements use of native symlinks.  I've previously described the
>work I've done.  I've offered up my code.  I never really got rational,
>technical reasons for why such support was a bad idea.

If you're referring to this:
http://sourceware.org/ml/cygwin-developers/2012-12/msg00000.html

Then I did respond later in the thread.  So, please don't claim that
I'm irrational or nontechnical.

To summarize my objection:  It doesn't sound like the native symlink
can be made to completely emulate a Linux symlink.  That has always
been the problem with Windows symlinks.

>What I am lobbying for...

As Larry indicated, this is not the mailing list for "lobbying" however,
to save you the trouble of moving to the cygwin mailing list:  As I
(and Corinna) have said before, I'd rather not complicate the
labyrinthian path handling code by introducing a new API.  I don't
really see why one would be needed.

However, if you think it's a great idea to have a utility which does
stuff to and with native symlinks then that's something that you could
write yourself and propose to be included in the Cygwin distribution.
That would be something you could discuss in the cygwin mailing list
and, ultimately, the cygwin-apps mailing list.

cgf

Re: native symlink

[James Gregurich]

On Apr 1, 2013, at 12:52 PM, Christopher Faylor <cgf-use-the-mailinglist-please@cygwin.com> wrote:

> 
> If you're referring to this:
> http://sourceware.org/ml/cygwin-developers/2012-12/msg00000.html
> 
> Then I did respond later in the thread.  So, please don't claim that
> I'm irrational or nontechnical.
> 
> To summarize my objection:  It doesn't sound like the native symlink
> can be made to completely emulate a Linux symlink.  That has always
> been the problem with Windows symlinks.


As I said before (and this is discussion of technical detail for programmers), it doesn't need to fully emulate a unix symlink. The design I proposed would attempt a native symlink and then fall back to normal symlink if the operation failed. The posix layer doesn't care which physical form the symlink takes. it works the same either way. The mechanism works nicely...I have prototyped it. 

However, as I said… I'm not going to ask for that model any further as I doubt I could get any traction with and I could live with a lesser capability that is less invasive.




> 
>> What I am lobbying for...
> 
> As Larry indicated, this is not the mailing list for "lobbying" however,
> to save you the trouble of moving to the cygwin mailing list:  As I
> (and Corinna) have said before, I'd rather not complicate the
> labyrinthian path handling code by introducing a new API.  I don't
> really see why one would be needed.

This is why I already throw around words like "irrational."   I have said multiple times that you already read native symlinks. Your "labyrinthian" code already codes does it. Nothing needs to be added to your labyrinthian code for handling paths. But, it seems like this sentence always gets piped right to /dev/null for some reason. That point never gets acknowledged. The same argument just gets repeated over and over despite the fact that it has been answered. To me, that is not rational thinking. sorry.


In fact, let me document for you exactly what I changed in your labyrinthian path handling code…

I added two data members to symlink_info:

size_t mReparsePointType;  //jmg   0 = false. 1 = file reparse point type  2 = directory reparse point type
NTSTATUS mDefaultMethodOpenStatus; //jmg


I then added the following accessor functions:

    bool isReparsePoint(); //jmg
    bool isDirReparsePoint(); //jmg
    NTSTATUS defaultMethodOpenStatus(); //jmg


I modified...
symlink_info::check_reparse_point (HANDLE h, bool remote)

at the end of the function, I added:


    //jmg
    if( (fileattr & FILE_ATTRIBUTE_DIRECTORY) == FILE_ATTRIBUTE_DIRECTORY )
    {
        this->mReparsePointType = 2;
    }
    else
    {
        this->mReparsePointType = 1;
    }


I modified…

int
symlink_info::check (char *path, const suffix_info *suffixes, fs_info &fs,
                     path_conv_handle &conv_hdl)


toward the end, I added..


            status = NtOpenFile (&sym_h, SYNCHRONIZE | GENERIC_READ, &attr, &io,
                                 FILE_SHARE_VALID_FLAGS,
                                 FILE_OPEN_FOR_BACKUP_INTENT
                                 | FILE_SYNCHRONOUS_IO_NONALERT);
            
            if (!NT_SUCCESS (status))
            {
                this->mDefaultMethodOpenStatus = status; //jmg



Basically, the only  modification I did to the existing labyrinthian path handling code was to store some extra data I needed make decisions in my new code higher up in the code stack. That's it!   Now, I hardly think these changes constitute a significant increase in the complexity of the existing path and symlink reading code. 


So what exactly did I change to implement creation of native symlinks (the actual new functionality?  …

 
I modified... 

int
symlink_worker (const char *oldpath, const char *newpath, bool use_winsym,
                unsigned long use_nativesym, bool isdevice) //bool isdevice)



I added a new parameter as you can see. As well as a line to set a local variable:

    bool tmpUseReparsePoints = use_nativesym != 0;
    bool mk_winsym = use_winsym && !tmpUseReparsePoints;


then, I added this section…


    debug_printf ("tmpUseReparsePoints 2 (%d)", tmpUseReparsePoints);
    
    
    if (!isdevice && tmpUseReparsePoints)
    {
        assert(use_nativesym <= 2);  //
        
        bool tmpLinkIsOfDirectoryType = false;
        NTSTATUS tmpNTErrorId = 0;
        int tmpPathConvErrorId = 0;
        size_t tmpResult = attemptReparsePointSymlink(*newpath, *oldpath, tmpNTErrorId, tmpPathConvErrorId, NULL, tmpLinkIsOfDirectoryType); //win32_newpath
        
        debug_printf("result=%d, tmpNTErrorId=%x, tmpPathConvErrorId=%d, tmpLinkIsOfDirectoryType=%d.", tmpResult, tmpNTErrorId, tmpPathConvErrorId, tmpLinkIsOfDirectoryType);
        
        
        if(tmpResult == kSymlinkSuccess)
        {
            res = 0;
            goto done;
        }
        
        if(use_nativesym == 1)
        {
            //weak. if native fails, fall back to default
            //
            //fall through and let the default action happen
            tmpUseReparsePoints = false;
        }
        else  //== 2
        {
            //strong. if reparse point cannot be set, fail.
            //However, if the target doesn't exist, use the default method 
            //come back later and update the link to a reparse point using 'lnmakenative'
            
            if(tmpResult != kSymlinkPathDoesNotExist)
            {
                if (!NT_SUCCESS (tmpNTErrorId))
                {
                    __seterrno_from_nt_status (tmpNTErrorId);
                    goto done;
                }            
                else if(tmpPathConvErrorId != 0)
                {
                    set_errno (tmpPathConvErrorId);
                    goto done;
                }
                else if(tmpResult == kSymlinkPathsTooLong)
                {
                    set_errno (ENAMETOOLONG);
                }
                
                
                //if the error is not an NT function error, and not path conversion error, then it is a win32 error. 
                //fall back to using GetLastError()
                __seterrno();
                goto done;
            }
        }
    }
    
    
    if (mk_winsym)




The rest of the code changes are to add a new environment variable and expose a new function... 

extern "C" int cygwin_update_symlink_to_reparse_point (const char *theLinkPathPtr)


which will update the symlink if it is a default form or leave it alone if it is already native.  Also, there is the actual implementation of attemptReparsePointSymlink() which I can provide if that is useful.



What's my point?   The new functionality I've added…namely…creating native symlinks is entirely encapsulated in one small area of code.  It does not add anything significant to the labyrinth of which you are concerned. The only thing I added to the labyrinth was a couple of extra member variables to persist data I needed later in the process. Your existing labyrinth is pretty much sufficient as it is!


All of that said, I'm no longer asking for the changes to symlink_worker. A person who has my needs can function with just cygwin_update_symlink_to_reparse_point() and the lnmakenative utility I wrote. Certainly a new function..

extern "C" int
symlink_native (const char *oldpath, const char *newpath)

would be nice,  but I won't push my luck if I can get the bare minimum.





> However, if you think it's a great idea to have a utility which does
> stuff to and with native symlinks then that's something that you could
> write yourself and propose to be included in the Cygwin distribution.
> That would be something you could discuss in the cygwin mailing list
> and, ultimately, the cygwin-apps mailing list.


I did write the utility myself….as I have said repeatedly. It is in production use at my company.  But, more is desirable than just an app. Your system is designed so that so that all the logic dealing with paths and symlinks is in path.cc. That is where the internal structure of the cygwin symlink is encapsulated. It is most appropriate for the core logic for such a feature to go there as that fits into the existing design model.  Furthermore, I already have the code written for you. You may wish to repackage it a bit to suit your tastes and make sure there are no flaws in my design (I make no claim to be an expert on the internals of cygwin), but the core logic is done and it works.


I'm certainly willing to join this other mailing list and make a formal proposal. But, I don't want to waste my time if the programmers don't want to be bothered. So the question is… have I made my case sufficiently so that the programmers will be willing to work with me?


-James

Re: native symlink

[Christopher Faylor]

On Mon, Apr 01, 2013 at 03:02:50PM -0700, James Gregurich wrote:
>
>On Apr 1, 2013, at 12:52 PM, Christopher Faylor <cgf-use-the-mailinglist-please@cygwin.com> wrote:
>
>> 
>> If you're referring to this:
>> http://sourceware.org/ml/cygwin-developers/2012-12/msg00000.html
>> 
>> Then I did respond later in the thread.  So, please don't claim that
>> I'm irrational or nontechnical.
>> 
>> To summarize my objection:  It doesn't sound like the native symlink
>> can be made to completely emulate a Linux symlink.  That has always
>> been the problem with Windows symlinks.
>
>
>As I said before (and this is discussion of technical detail for
>programmers), it doesn't need to fully emulate a unix symlink.  The
>design I proposed would attempt a native symlink and then fall back to
>normal symlink if the operation failed.  The posix layer doesn't care
>which physical form the symlink takes.  it works the same either way.
>The mechanism works nicely...I have prototyped it.

In the model that you originally proposed, a fallback wouldn't be
sufficient.  As described, you were creating symlinks with absolute
paths and those won't fly.

>>> What I am lobbying for...
>> 
>>As Larry indicated, this is not the mailing list for "lobbying"
>>however, to save you the trouble of moving to the cygwin mailing list:
>>As I (and Corinna) have said before, I'd rather not complicate the
>>labyrinthian path handling code by introducing a new API.  I don't
>>really see why one would be needed.
>
>This is why I already throw around words like "irrational." I have said
>multiple times that you already read native symlinks.  Your
>"labyrinthian" code already codes does it.  Nothing needs to be added
>to your labyrinthian code for handling paths.  But, it seems like this
>sentence always gets piped right to /dev/null for some reason.  That
>point never gets acknowledged.  The same argument just gets repeated
>over and over despite the fact that it has been answered.  To me, that
>is not rational thinking.  sorry.

As far as I can see, you posted two messages to this mailing list.  I
responded to one of them.  You didn't, as far as I could see, respond to
my message which offered (despite your words to the contrary) technical
objections.

>In fact, let me document for you exactly what I changed in your
>labyrinthian path handling code?

I count 76 lines of code there, total.  I don't know if this code
addressed my technical concerns but I suspect that it doesn't.

>...
>Basically, the only modification I did to the existing labyrinthian
>path handling code was to store some extra data I needed make decisions
>in my new code higher up in the code stack.  That's it! Now, I hardly
>think these changes constitute a significant increase in the complexity
>of the existing path and symlink reading code.

I probably didn't make my point clear.  Adding any amount of code to the
path handling is something that we try to avoid but I do think that your
additions add more complexity than I'd be comfortable with, especially
given the limitations of Windows symlinks.

>So what exactly did I change to implement creation of native symlinks
>(the actual new functionality?  ?

So that's another 53 lines to the path handling code.  That brings the
total to about 129 lines added/changed.

>>However, if you think it's a great idea to have a utility which does
>>stuff to and with native symlinks then that's something that you could
>>write yourself and propose to be included in the Cygwin distribution.
>>That would be something you could discuss in the cygwin mailing list
>>and, ultimately, the cygwin-apps mailing list.
>
>
>I did write the utility myself?.as I have said repeatedly.

You mentioned in your original mail that you'd written some utilities
but those appeared to be using the changes that you'd made to Cygwin.  I
was suggesting that you could just write a utility which didn't use a
Cygwin API to create a symlink.  Since you said:

'I have said multiple times that you already read native symlinks.  Your
"labyrinthian" code already codes does it.  Nothing needs to be added to
your labyrinthian code for handling paths.'

I was trying to tell you that you could just propose packaging a new
Cygwin utility to create the symlinks that you claim Cygwin already
reads.

>I'm certainly willing to join this other mailing list and make a formal
>proposal.  But, I don't want to waste my time if the programmers don't
>want to be bothered.  So the question is?  have I made my case
>sufficiently so that the programmers will be willing to work with me?

I'm not interested but, if Corinna thought this was must-have
functionality, I would certainly not block any code additions to Cygwin.

I suspect that what Dan Colascione said back in 2012-12-08 is still
true, though:

"Cygwin symbolic links work well enough, and I don't think trying to
overcome these difficulties is a high priority."

However, A utility which made no changes to the DLL but just got
packaged up with the rest of the Cygwin distro wouldn't require any
programmers besides yourself.  You'd just have to take it through the
Cygwin package adoption process and get the requisite number of votes.

cgf

Re: native symlink

[James Gregurich]

[resent from sourceware spam trap by cgf]
On Apr 1, 2013, at 5:06 PM, Christopher Faylor wrote:

> 
> In the model that you originally proposed, a fallback wouldn't be
> sufficient.  As described, you were creating symlinks with absolute
> paths and those won't fly.

That isn't a problem. This only happens in the case where the path exceed PATH_MAX. In that case, instead of invoking the logic to contrive the extended path name, just abort the effort and fall back to the default symlink.



> 
>>>> What I am lobbying for...
>>> 
>>> As Larry indicated, this is not the mailing list for "lobbying"
>>> however, to save you the trouble of moving to the cygwin mailing list:
>>> As I (and Corinna) have said before, I'd rather not complicate the
>>> labyrinthian path handling code by introducing a new API.  I don't
>>> really see why one would be needed.
>> 
>> This is why I already throw around words like "irrational." I have said
>> multiple times that you already read native symlinks.  Your
>> "labyrinthian" code already codes does it.  Nothing needs to be added
>> to your labyrinthian code for handling paths.  But, it seems like this
>> sentence always gets piped right to /dev/null for some reason.  That
>> point never gets acknowledged.  The same argument just gets repeated
>> over and over despite the fact that it has been answered.  To me, that
>> is not rational thinking.  sorry.
> 
> As far as I can see, you posted two messages to this mailing list.  I
> responded to one of them.  You didn't, as far as I could see, respond to
> my message which offered (despite your words to the contrary) technical
> objections.

Yes. I did answer your objections. You never responded. 




> 
>> In fact, let me document for you exactly what I changed in your
>> labyrinthian path handling code?
> 
> I count 76 lines of code there, total.  I don't know if this code
> addressed my technical concerns but I suspect that it doesn't.

The purpose of supplying those snippets was to establish that my suggestion would not deepen the labyrinth. If you would like to see the entirety of what I did to path.cc, I can certainly supply that by whatever communication channel you prefer. Just let me know.




> 
>> ...
>> Basically, the only modification I did to the existing labyrinthian
>> path handling code was to store some extra data I needed make decisions
>> in my new code higher up in the code stack.  That's it! Now, I hardly
>> think these changes constitute a significant increase in the complexity
>> of the existing path and symlink reading code.
> 
> I probably didn't make my point clear.  Adding any amount of code to the
> path handling is something that we try to avoid but I do think that your
> additions add more complexity than I'd be comfortable with, especially
> given the limitations of Windows symlinks.

Why is adding a couple extra data members to your existing classes considered a significant amount of complexity increase?


> 
>> So what exactly did I change to implement creation of native symlinks
>> (the actual new functionality?  ?
> 
> So that's another 53 lines to the path handling code.  That brings the
> total to about 129 lines added/changed.

yes.  That constitutes the limits of my changes to the existing path.cc.   That is why I keep telling you that my suggestion is not a significant increase in complexity. If you take out my changes to symlink_worker() and just supply an additional API call to convert the symlink after the fact, then the changes to existing code are negligible. One would just need to append a few functions that don't do anything more with the existing path logic other than access the new data members.

I don't see reason for consternation here.


> 
> I was trying to tell you that you could just propose packaging a new
> Cygwin utility to create the symlinks that you claim Cygwin already
> reads.

path.cc already contains all of the logic necessary to make all the decisions necessary to convert a cygwin symlink into native symlink. Why would one want to reimplement all of that in a client application? It makes perfect sense for that logic to exist in path.cc.

Secondly, exposing an API call would enable people to programmatically use the functionality from any application without an additional library to worry about.





> 
>> I'm certainly willing to join this other mailing list and make a formal
>> proposal.  But, I don't want to waste my time if the programmers don't
>> want to be bothered.  So the question is?  have I made my case
>> sufficiently so that the programmers will be willing to work with me?
> 
> I'm not interested but, if Corinna thought this was must-have
> functionality, I would certainly not block any code additions to Cygwin.
> 
> I suspect that what Dan Colascione said back in 2012-12-08 is still
> true, though:
> 
> "Cygwin symbolic links work well enough, and I don't think trying to
> overcome these difficulties is a high priority."
> 
> However, A utility which made no changes to the DLL but just got
> packaged up with the rest of the Cygwin distro wouldn't require any
> programmers besides yourself.  You'd just have to take it through the
> Cygwin package adoption process and get the requisite number of votes.


"work well enough" is in the eye of the beholder.  Cygwin is worthless to my purpose without proper support for native symlinks. Native symlinks are critical to the design of our git repositories and well cygwin  shell scripts we write as part of our Visual Studio workflow.

There are two reasons why I went with cygwin over MSYS-git for our purpose…

1) Cygwin already supported extended path names. 
2) The source code and design of Cygwin were sufficiently straight-forward that I could modify it to add the symlink support I needed.


So, if some developer of Cygwin had decided that 260 character path names was "good enough," then I wouldn't have had #1 and perhaps I would have sought out some other solution. Perhaps one should not be so short-sighted as to what constitutes "work well enough" if he wants to attract additional users to his platform.

Did not Cygwin also "work well enough" as a 32 bit process? Why bother with 64 bit support if this is the attitude?  I bet the modifications to support 64 bit usage cost far more in time, complexity and support tickets than my suggestion would. Why dismiss my suggestion out of hand just because you don't personally need it? 

I certainly have not suggested someone do the work for me. I've done the work for you and am offering it up at no cost to you other than to cooperate with integrating it into the system. I've even acknowledged that I might have to spend some time modifying the work to meet requirements I'm unaware of.


If I have the support of the programmers, then I'll go through the necessary bureaucratic rigamarole to get the changes adopted. 

Re: native symlink

[Christopher Faylor]

On Tue, Apr 02, 2013 at 03:25:52PM -0700, James Gregurich wrote:
>[resent from sourceware spam trap by cgf]
>On Apr 1, 2013, at 5:06 PM, Christopher Faylor wrote:
>>In the model that you originally proposed, a fallback wouldn't be
>>sufficient.  As described, you were creating symlinks with absolute
>>paths and those won't fly.
>
>That isn't a problem.  This only happens in the case where the path
>exceed PATH_MAX.  In that case, instead of invoking the logic to
>contrive the extended path name, just abort the effort and fall back to
>the default symlink.

So, sometimes people would find that their windows programs would fall
over when accessing a symlink.  That's not a tech support email I'm
interested in handling.

>>>>> What I am lobbying for...
>>>> 
>>>> As Larry indicated, this is not the mailing list for "lobbying"
>>>> however, to save you the trouble of moving to the cygwin mailing list:
>>>> As I (and Corinna) have said before, I'd rather not complicate the
>>>> labyrinthian path handling code by introducing a new API.  I don't
>>>> really see why one would be needed.
>>> 
>>> This is why I already throw around words like "irrational." I have said
>>> multiple times that you already read native symlinks.  Your
>>> "labyrinthian" code already codes does it.  Nothing needs to be added
>>> to your labyrinthian code for handling paths.  But, it seems like this
>>> sentence always gets piped right to /dev/null for some reason.  That
>>> point never gets acknowledged.  The same argument just gets repeated
>>> over and over despite the fact that it has been answered.  To me, that
>>> is not rational thinking.  sorry.
>> 
>> As far as I can see, you posted two messages to this mailing list.  I
>> responded to one of them.  You didn't, as far as I could see, respond to
>> my message which offered (despite your words to the contrary) technical
>> objections.
>
>Yes. I did answer your objections. You never responded. 

I posted, in a previous message, the link to the discussion from back in
December.  My response was the last one in the archives.

>>> In fact, let me document for you exactly what I changed in your
>>> labyrinthian path handling code?
>> 
>> I count 76 lines of code there, total.  I don't know if this code
>> addressed my technical concerns but I suspect that it doesn't.
>
>The purpose of supplying those snippets was to establish that my
>suggestion would not deepen the labyrinth.  If you would like to see
>the entirety of what I did to path.cc, I can certainly supply that by
>whatever communication channel you prefer.  Just let me know.
>
>>> ...
>>> Basically, the only modification I did to the existing labyrinthian
>>> path handling code was to store some extra data I needed make decisions
>>> in my new code higher up in the code stack.  That's it! Now, I hardly
>>> think these changes constitute a significant increase in the complexity
>>> of the existing path and symlink reading code.
>> 
>> I probably didn't make my point clear.  Adding any amount of code to the
>> path handling is something that we try to avoid but I do think that your
>> additions add more complexity than I'd be comfortable with, especially
>> given the limitations of Windows symlinks.
>
>Why is adding a couple extra data members to your existing classes
>considered a significant amount of complexity increase?

You can be sure that since I didn't use the term "data members" I was
talking about all of the lines of code that you posted.

>>> So what exactly did I change to implement creation of native symlinks
>>> (the actual new functionality?  ?
>> 
>> So that's another 53 lines to the path handling code.  That brings the
>> total to about 129 lines added/changed.
>
>yes.  That constitutes the limits of my changes to the existing
>path.cc.
>
>... my suggestion is not a significant increase in complexity.

Sorry, but I think that, given the cost/benefit of this change, 129
lines of code in a sensitive data path + extra data members in a
structure is significant.  If this change added full handling of Windows
symlinks with no compromises, that would be a different story.

>If you take out my changes to symlink_worker() and just supply an
>additional API call to convert the symlink after the fact, then the
>changes to existing code are negligible.  One would just need to append
>a few functions that don't do anything more with the existing path
>logic other than access the new data members.
>
>> I was trying to tell you that you could just propose packaging a new
>> Cygwin utility to create the symlinks that you claim Cygwin already
>> reads.
>
>path.cc already contains all of the logic necessary to make all the
>decisions necessary to convert a cygwin symlink into native symlink.
>Why would one want to reimplement all of that in a client application?
>It makes perfect sense for that logic to exist in path.cc.

We try hard not to add non-Linux interfaces to Cygwin.  IIRC, we haven't
done anything like that in years.  And, neither Corinna nor I want to do
that for this case.

>Secondly, exposing an API call would enable people to programmatically
>use the functionality from any application without an additional
>library to worry about.

Every API doesn't have to come from Cygwin.  You can always add a library
to a Cygwin package and instruct people to use that if they want to roll
their own.

>>>I'm certainly willing to join this other mailing list and make a formal
>>>proposal.  But, I don't want to waste my time if the programmers don't
>>>want to be bothered.  So the question is?  have I made my case
>>>sufficiently so that the programmers will be willing to work with me?
>>
>>I'm not interested but, if Corinna thought this was must-have
>>functionality, I would certainly not block any code additions to
>>Cygwin.
>>
>>I suspect that what Dan Colascione said back in 2012-12-08 is still
>>true, though:
>>
>>"Cygwin symbolic links work well enough, and I don't think trying to
>>overcome these difficulties is a high priority."
>>
>>However, A utility which made no changes to the DLL but just got
>>packaged up with the rest of the Cygwin distro wouldn't require any
>>programmers besides yourself.  You'd just have to take it through the
>>Cygwin package adoption process and get the requisite number of votes.
>
>"work well enough" is in the eye of the beholder.  Cygwin is worthless
>to my purpose without proper support for native symlinks.  Native
>symlinks are critical to the design of our git repositories and well
>cygwin shell scripts we write as part of our Visual Studio workflow.

I only cut part of Daniel's message but what he was saying was that
the Cygwin developers think that symbolic links work well enough.  The
"Cygwin developers" part is the important part.

>There are two reasons why I went with cygwin over MSYS-git for our
>purpose
>
>1) Cygwin already supported extended path names. 
>2) The source code and design of Cygwin were sufficiently
>straight-forward that I could modify it to add the symlink support I
>needed.
>
>So, if some developer of Cygwin had decided that 260 character path
>names was "good enough," then I wouldn't have had #1 and perhaps I
>would have sought out some other solution.  Perhaps one should not be
>so short-sighted as to what constitutes "work well enough" if he wants
>to attract additional users to his platform.

FWIW, the reason that we didn't think that 260 characters was "good
enough" is because Linux doesn't have such a limit and it was, for the
most part, possible to make longer pathnames work without compromise.
Your approach, on the other hand, can't be a complete replacement for
Cygwin symlinks.  So, it's not really an apples-to-apples comparison.

>Did not Cygwin also "work well enough" as a 32 bit process?  Why bother
>with 64 bit support if this is the attitude?  I bet the modifications
>to support 64 bit usage cost far more in time, complexity and support
>tickets than my suggestion would.  Why dismiss my suggestion out of
>hand just because you don't personally need it?

You're really pulling out all of the stops.

Since you asked, the major motivation behind 64-bit Cygwin is that
someone at Red Hat thought it would make financial sense to finally port
Cygwin to 64-bit.  And, now all of the package maintainers are willing
to do the work to get onto a new platform.  Everyone will benefit from a
64-bit Cygwin.  It's a clear win.

>I certainly have not suggested someone do the work for me.

No one said that you had.

>I've done the work for you and am offering it up at no cost to you
>other than to cooperate with integrating it into the system.  I've even
>acknowledged that I might have to spend some time modifying the work to
>meet requirements I'm unaware of.

There *is* a cost to me or Corinna here.  We'd have to carefully review
your 129 lines of code and be involved in the back and forth that
entails.  And, then we'd eventually have to support the proposed code,
respond to bug reports on it, and make sure that it was properly
documented.  And, since neither of us is keen on this approach that's
more than we're willing to do.

>If I have the support of the programmers, then I'll go through the
>necessary bureaucratic rigamarole to get the changes adopted.

Thanks for the offer of a patch but this is not something that we are
willing to add to the code.  And, we are not interested in adding a new
non-POSIX/linux API to Cygwin for manipulating native windows symlinks.

cgf

Re: native symlink

[James Gregurich]

On Apr 2, 2013, at 6:40 PM, Christopher Faylor wrote:

> So, sometimes people would find that their windows programs would fall
> over when accessing a symlink.  That's not a tech support email I'm
> interested in handling.

(keep in mind that on this point we are discussing a hypothetical since I am no longer asking for symlink() to be modified.)

How many support emails do you think you would get for a failure to create a relative path symlink where the path exceeds 254 (if memory serves me correctly) chars? Its a pretty rare circumstance.

Secondly, the support for native symlinks is off by default and would only be enabled by someone who actually wants that functionality.  That person would have been provided documentation for the use of the feature that he would have had to reference to understand how to enable the feature in the first place. So, these things could be documented.

Why are you so worried about theoretical tech support tickets that may never happen in the real world? 


My team supports a pretty good size matrix of commercial products that people use for real-world, for-profit, production work. We certainly don't avoid taking risks on useful features. We are careful and thoughtful about it and make every effort to make sure we don't break existing functionality, but evolving our software is our reason to exist. If we get support emails coming in…we deal with them! that is our job. I really don't get this fear of getting support tickets. 



>> Yes. I did answer your objections. You never responded. 
> 
> I posted, in a previous message, the link to the discussion from back in
> December.  My response was the last one in the archives.

I never saw a response. Perhaps I missed it in my email box for whatever reason. Its not worth going back through the logs to find out for sure. I'll just take your word for it and drop the point.


>> 
>> Why is adding a couple extra data members to your existing classes
>> considered a significant amount of complexity increase?
> 
> You can be sure that since I didn't use the term "data members" I was
> talking about all of the lines of code that you posted.

With the exception of the data members, all of my new code is factored off by itself.  That is the only part that is intermingled with existing code. Thus, the labyrinth is almost entirely unaffected. 


I took on your code in all its complexity knowing nothing about it or its design. I'm not even primarily a Windows developer. I'm a mac developer. Yet, i worked my way through it all, figured out everything I needed to figure out and made it all work. While the code certainly could be better designed and would be benefitted by some modern C++ practices, it certainly wasn't impossible to understand or modify it. I'll admit it is fairly hairy, but I've seen much worse in my life. 

Why am I unafraid to dig into your code and extend it, but you (who should know it better than I do) are so afraid to extend it? I invested a couple of months in these modifications given I had a ton of research and experimentation to do. That is tens of thousands of dollars of engineering time my company risked to make a useful tool. I'm really quite puzzled by this fear to modify the code.




>>>> So what exactly did I change to implement creation of native symlinks
>>>> (the actual new functionality?  ?
>>> 
>>> So that's another 53 lines to the path handling code.  That brings the
>>> total to about 129 lines added/changed.
>> 
>> yes.  That constitutes the limits of my changes to the existing
>> path.cc.
>> 
>> ... my suggestion is not a significant increase in complexity.
> 
> Sorry, but I think that, given the cost/benefit of this change, 129
> lines of code in a sensitive data path + extra data members in a
> structure is significant.  If this change added full handling of Windows
> symlinks with no compromises, that would be a different story.


There are not 129 lines of code in a "sensitive data path."  There is a couple of data members and associated accessor functions. That's it. Why do you insist on imagining that there is more to it than there is?



Secondly, maybe there is no "benefit" to YOU.  But, to those of us who need unix repositories to work on Windows and Visual Studio, there is a HUGE benefit. I suspect I'm not the only guy out there who needs the functionality. I've gotten a couple of private emails from people who want the functionality. I know there is an ongoing effort on MSYS-GIT to make it work with native symlinks on Windows. There are people who use this. 


thirdly, what is the "cost" to you? I've done the heavy lifting. We use the code in production.  It works. I haven't had to go back and fix any bugs in it since I deployed it. Granted, we are a small operation. But we do hit it with developer tools and various unix programs. It has never failed or exhibited weird behavior that could be attributed to my changes.

Is your "cost" some number of support tickets that may never happen? If not, please explicitly list the "costs" that you envision besides support tickets.





>> 
>> path.cc already contains all of the logic necessary to make all the
>> decisions necessary to convert a cygwin symlink into native symlink.
>> Why would one want to reimplement all of that in a client application?
>> It makes perfect sense for that logic to exist in path.cc.
> 
> We try hard not to add non-Linux interfaces to Cygwin.  IIRC, we haven't
> done anything like that in years.  And, neither Corinna nor I want to do
> that for this case.

Is cygwin_conv_to_win32_path() a "Linux interface"? You already provide functions in the posix layer..in fact, in path.cc itself... to help integrate the unix environment into Windows better.


The purpose of cygwin is to integrate unix software with a Windows environment. If a person just wants to run unix software on a pure Linux interface, he can just run Linux. He doesn't need Cygwin. There is minimal purpose to Cygwin if all it is intended to be is just Linux on top the NT kernel.


What is the practical purpose to this rule you have made to not add "non-Linux interfaces" to Cygwin? That seems silly given the purpose of the project is to make Windows more useful.





>> Secondly, exposing an API call would enable people to programmatically
>> use the functionality from any application without an additional
>> library to worry about.
> 
> Every API doesn't have to come from Cygwin.  You can always add a library
> to a Cygwin package and instruct people to use that if they want to roll
> their own.


Should I write my own code to do all the path parsing and processing that is already done in path.cc? Particularly since I've already coded al the work to the classes in path.cc? That makes no technical sense. All of this work is already done in path.cc. 



> I only cut part of Daniel's message but what he was saying was that
> the Cygwin developers think that symbolic links work well enough.  The
> "Cygwin developers" part is the important part.


If the purpose is just to please this narrow of a demographic of potential users, why bother making it open source…or even making it public at all.

Perhaps Cygwin's developers should widen their scope and perspective if they want to attract more users.






> FWIW, the reason that we didn't think that 260 characters was "good
> enough" is because Linux doesn't have such a limit and it was, for the
> most part, possible to make longer pathnames work without compromise.
> Your approach, on the other hand, can't be a complete replacement for
> Cygwin symlinks.  So, it's not really an apples-to-apples comparison.

And yet that mechanism is imperfect in that it cannot handle relative paths. But, you implemented it anyway.



> 
> You're really pulling out all of the stops.

Persistence and doggedness are useful virtues. :)

One needs these traits to get things done…PARTICULARLY when dealing with engineers.



> Since you asked, the major motivation behind 64-bit Cygwin is that
> someone at Red Hat thought it would make financial sense to finally port
> Cygwin to 64-bit.  And, now all of the package maintainers are willing
> to do the work to get onto a new platform.  Everyone will benefit from a
> 64-bit Cygwin.  It's a clear win.


I don't benefit from it. 32 bit Cygwin works fine for me. I'd be willing to bet that the people who actually need a 64 bit cygwin are a very small percentage of the user base. The only reason you need a 64 bit process is for perhaps a little better performance for some problems due to extra registers on x86_64 and access to more than 2 GB of memory per process. Frankly, anybody who has serious needs for those two things is unlikely to be running such processes on Cygwin when he can just run it on *nix where he gets a much more reliable and predictable runtime. 

But, I can certainly appreciate that you did it for money. Also, it should have been done to keep up with the times if nothing else. I'm just making the point that a useful feature should not be denied just because you personally don't need it. You need to think of your customers.




>> I've done the work for you and am offering it up at no cost to you
>> other than to cooperate with integrating it into the system.  I've even
>> acknowledged that I might have to spend some time modifying the work to
>> meet requirements I'm unaware of.
> 
> There *is* a cost to me or Corinna here.  We'd have to carefully review
> your 129 lines of code and be involved in the back and forth that
> entails.  And, then we'd eventually have to support the proposed code,
> respond to bug reports on it, and make sure that it was properly
> documented.  And, since neither of us is keen on this approach that's
> more than we're willing to do.

But then you would also gain access to an additional development resource who will have demonstrated that he can competently operate in path.cc to get useful work done. I certainly would not petition to add the feature and then not make myself available to help out in supporting the project.

Perhaps you didn't count that as a "benefit."



> 
>> If I have the support of the programmers, then I'll go through the
>> necessary bureaucratic rigamarole to get the changes adopted.
> 
> Thanks for the offer of a patch but this is not something that we are
> willing to add to the code.  And, we are not interested in adding a new
> non-POSIX/linux API to Cygwin for manipulating native windows symlinks.

I've think I've demonstrated that this is a statement of dogma and personal preference…not technical substance.

You already have "non-posix/non-Linux" API calls in Cygwin.dll and path.cc. There is absolutely no technical reason to have a rule not to add more as long it is done thoughtfully and carefully. You've already shown that you will add features (64 bit support) that few people actually need. Finally, you have shown with extended path names that you will implement features that are imperfect solutions if they are useful. I think those three answers shoot down your technical objections.

The only non-technical objection you've made is that it will theoretically create support tickets…But as I said…you are likely to get way more support tickets on things you broke adding 64 bit support than you would from this feature. If you are really concerned about support tickets, you would have not make the 64 bit changes.

Re: native symlink

[Jeffrey Altman]

Perhaps we would all be better off taking this discussion to main cygiwn
mailing list.  I've been reluctant to respond here because of the
request to take the discussion elsewhere but I believe that have a
unique perspective as the implementer of cross-platform (but primarily
UNIX focused) file system on Microsoft Windows.

I completely understand Christopher's inclination to treat Cygwin as a
Linux compatible black box that happens to run on Windows and not as a
layer that glues the two worlds together.  There are many contributors
to OpenAFS that take the position that the file system semantics of AFS
on Windows must be the UNIX semantics.

The downside of that approach is that it does address the needs of
organizations which have a diverse mixture of applications all of which
need to view the same file system name space because data is never
constrained to just one of the environment or framework.

For AFS this meant simulating a broad range of Windows file system
semantics which have no equivalents in AFS.  One of the areas in which
OpenAFS has been forced to adapt is in the exporting of AFS mount points
and symlinks into the Windows environment.  Ensuring compatibility with
Cygwin applications is important because just about every organization
that uses a cross-platform file system such as AFS also uses Cygwin for
its ability to provide cross-platform application support.

In fact, many of the AFS end user organizations provide Cygwin
applications to their end users by installing Cygwin once in the AFS
name space and then executing it locally on each endpoint.

I tried representing AFS Symlinks using a Microsoft assigned Reparse
Point Tag.  The downside of following that approach was that Cygwin does
not properly handle Reparse Point Tags that it does not recognize.  By
discarding the RP attribute and preserving the other reparse point stat
information (timestamps, attributes, size, etc) it introduces data
corrupting behaviors into Cygwin applications.

As Christopher knows, I started down the path of adding OpenAFS Reparse
Point support to Cygwin but unlike James I do not have the months to
dive into the path.cc labyrinth.  I also discovered that Cygwin wasn't
the only environment/framework that poorly handled non-Microsoft Reparse
Point Tags.  In fact, Explorer Shell, .NET, Pre-1.7 Java, and many home
grown applications completely botch it.  In the end I concluded the
safest thing to so was to piggyback on the Microsoft Symlink Tag.

From the Cygwin perspective the nice thing about the change that went
into OpenAFS 1.7.22 is that Cygwin can read an AFS symlink and make use
of it with no code changes.   However, Cygwin "ln" cannot be used to
create a symlink in AFS because Cygwin doesn't write Microsoft Symlinks
and AFS cannot store the DOS SYSTEM attribute.  OpenAFS has for years
provided its own "symlink.exe" command that can be used to create
symlinks but the fact is that if Cygwin exposes AFS symlinks as Cygwin
symlinks end users are going to expect the standard UNIX processes to
create them as well.

Speaking as a representative of the OpenAFS user community, many of whom
are very large RedHat customers, adding support for creating Windows
native symlinks when it is safe to do so would have significant
benefits.  Especially for those that value a single seamless name space
accessible to all application.

At the moment Cygwin symlinks are not exposed outside the Cygwin
environment.  As such they don't generate help desk requests from
non-Cygwin applications.  As I recently discussed in my blog post, the
biggest issue with Microsoft Symlinks is the lack of support across all
applications and all file system interfaces.  Exporting Cygwin symlinks
as Microsoft Symlinks can have downsides for applications that are not
prepared to handle them.  I respect the fact that the Cygwin developers
might be reluctant to accept responsibility for furthering their use.
However, there is a very clear test that can be applied to determine
when Microsoft Symlinks should be generated in preference to Cygwin
symlinks:

  Does the Cygwin application have the
  SeCreateSymbolicLinkPrivilege privilege?

Processes will only have this privilege if the organization has granted
it to the user/group the process is running as OR if the process is
running as an elevated Administrator.

Another test that can be performed at run time is:

  Does the FileSystem Characteristics include
  FILE_SUPPORTS_REPARSE_POINTS?

Of course there are external to Cygwin methods that could be implemented
to expose Cygwin symlinks to applications.

 1. James proposes an external tool that reads a Cygwin's
    symlinks with Microsoft Symlinks.

 2. A file system filter driver could be installed on XP and
    above to capture the creation of Cygwin symlinks and replace
    them with Microsoft Symlink.

 3. A file system filter driver could be installed which
    recognizes Cygwin's (and for that matter Interix's) symlink
    files and performs the necessary file system path rewriting
    and reparsing.

There are probably additional approaches but none of them are clean and
transparent.  The second two involved significantly more complexity then
maintaining support within Cygwin's path.cc and could potentially
introduce incompatibilities with future Cygwin path.cc changes.

As I see it, as flawed as Microsoft Symlinks are they are the common
interface that enables mixed applications to communicate with one
another.  As such, where they can be used, they should be used.  What is
the point of cross-platform support if mixed platform applications
cannot transparently share the data?

Jeffrey Altman
OpenAFS Gatekeeper and Elder Emeritis

Re: native symlink

[Corinna Vinschen]

On Apr  3 00:33, Jeffrey Altman wrote:
> I tried representing AFS Symlinks using a Microsoft assigned Reparse
> Point Tag.  The downside of following that approach was that Cygwin does
> not properly handle Reparse Point Tags that it does not recognize.  By
> discarding the RP attribute and preserving the other reparse point stat
> information (timestamps, attributes, size, etc) it introduces data
> corrupting behaviors into Cygwin applications.

You never explained why this happens and at which point in the code.  So
far it was the right thing to do, and I'm pretty sure you know why.  I
don't change that, unless you can show me where and when this leads to
wrong behaviour.  I asked for these details but you didn't offer an
explanation besides the fact itself so far.  And it would have been
no problem to add a special handling for AFS in the cases where it went
wrong.  I guess this is kind of a moot point, now that you converted
to native symlinks, but this had to be said.

> However, there is a very clear test that can be applied to determine
> when Microsoft Symlinks should be generated in preference to Cygwin
> symlinks:
> [...]
> There are probably additional approaches but none of them are clean and
> transparent.  The second two involved significantly more complexity then
> maintaining support within Cygwin's path.cc and could potentially
> introduce incompatibilities with future Cygwin path.cc changes.

No test could be sufficient to switch on native symlinks automatically.

We were all very excited when it became clear that Microsoft introduced
native symlinks on NTFS with Vista, and I was early on playing around
with them and to try integrating them into Cygwin.  My local testcase
uses DeviceIoControl to workaround any restrictions imposed by
CreateSymbolicLink.  And I'm still playing around with them every now
and then, thinking that we could use them, but the restrictions are
disappointing me each time anew.

There are some downsides to native symlinks which make them hard to
justify, if not downright useless in a POSIX environment.

- The inability of normal users to create symlinks by default.

    This can be worked around by changing the policy, but it's still a
    PITA.  Normal users don't know about the policy, some of them don't
    even have the "Local Security Policy" MMC snap in.  Even in a
    corporate environment it requires to change the policy settings and
    we all know how admins don't like to *soften* a policy.  But let's
    say we can help along with a FAQ entry.

- Native symlinks are marked as file or directory.

    This has been added clearly for the benefit of Windows Explorer.
    But it's a PITA as well because it destroys interoperability.  It's
    common that POSIX symlinks are created before the target exists.
    How on earth should the symlink(2) function know if the target is
    supposed to be a dir or a file.  But Explorer as well as CMD will do
    the wrong thing if the symlink is using a non-matching dir/file
    marker.

- Only Windows paths are stored.

    In a POSIX env a symlink created by POSIX tools should point to a
    POSIX path.  For instance, mount points change the fact where a
    symlink actually points to and the symlink should not still
    magically work afterwards.

    But, hey, native symlinks store the path twice, the SubstituteName
    and the PrintName.  Shouldn't it be possible to store the Windows
    path in one of them and the POSIX path in the other?  Yes and no.
    It's possible to write into these members whatever you like, but for
    some weird reason, both members have to be Windows paths to work for
    native Windows tools.

    But we could store the POSIX path with backslashes, thus working
    around the issue, no?  No.  An absolute path starting with a
    backslash is possible, but the Windows tools will evaluate it as
    root-relative to the current drive.  cd to another drive in cmd,
    and interop is broken again.

- Remote and local symlinks may behave different in different environments.

    Apart from the security policy, symlinks are also affected by an
    fsutil setting.  The admins can decide if symlinks work at all, or
    if symlinks don't work depending on their own location and the
    location of the target they are pointing to (local->local, local->remote,
    remote->local, remote->remote)

    So it's possible that local->local symlinks can be resolved while
    opening local->remote symlinks simply fail with ugly status codes.
    How on earth do you integrate that reliably into an environment in
    which a symlink is a plain and simple thing, readable and writable
    by everyone, whereever located, just apart from parent dir permissions.

> As I see it, as flawed as Microsoft Symlinks are they are the common
> interface that enables mixed applications to communicate with one
> another.  As such, where they can be used, they should be used.  What is
> the point of cross-platform support if mixed platform applications
> cannot transparently share the data?

Cygwin is a POSIX environment in the first place.  Interop is fine,
but if it collides with POSIX, we're clearly favoring POSIX.

Having said that.

Chris and I had a private discussion (not the first one on the subject!)
and we're willing to revisit the use of native symlinks in Cygwin but
it will be a while before that happens.  A change to the path handling
code like this is not something that we'd consider for 1.7.18 which is
long overdue anyway.

What I will do is to add a new CYGWIN environment variable option, along
the lines of the winsymlinks option(*), or, which is very likely the
more elgant solution, a mount option, which will result in trying to
create native symlinks first, and a Cygwin symlink only if creating
a native symlink failed.  That should help you along.


Corinna


(*) In your blog you were musing why Cygwin supports lnk files but
not native symlinks.  Here's the answer:  lnk files support using 
POSIX paths.


-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: native symlink

[Larry Hall]

On 4/3/2013 11:29 AM, Corinna Vinschen wrote:
> (*) In your blog you were musing why Cygwin supports lnk files but
> not native symlinks.  Here's the answer:  lnk files support using
> POSIX paths.

And while supporting lnk files does have some benefits for
interoperability, I think the history of this option is significant.
At a point in the past, Cygwin symlinks were lnk files by default.
However, they fell out of favor when support for UTF filenames was added
to Cygwin.  Given this background, one could certainly posit that lnk file
support is not in its ascendency and could even be viewed as exactly the
opposite.  To me, that makes it a weak basis for building an argument to
add support of new but similar functionality.

Despite the fact that I'm replying directly to Corinna's statement, I'm
addressing the above comment to Jeffrey.  Somehow I get the feeling that
Corinna is aware of the history of this option. ;-)

-- 
Larry

Re: native symlink

[Jeffrey Altman]

On 4/3/2013 12:32 PM, Larry Hall wrote:
> On 4/3/2013 11:29 AM, Corinna Vinschen wrote:
>> (*) In your blog you were musing why Cygwin supports lnk files but
>> not native symlinks.  Here's the answer:  lnk files support using
>> POSIX paths.
> 
> And while supporting lnk files does have some benefits for
> interoperability, I think the history of this option is significant.
> At a point in the past, Cygwin symlinks were lnk files by default.
> However, they fell out of favor when support for UTF filenames was added
> to Cygwin.  Given this background, one could certainly posit that lnk file
> support is not in its ascendency and could even be viewed as exactly the
> opposite.  To me, that makes it a weak basis for building an argument to
> add support of new but similar functionality.
> 
> Despite the fact that I'm replying directly to Corinna's statement, I'm
> addressing the above comment to Jeffrey.  Somehow I get the feeling that
> Corinna is aware of the history of this option. ;-)

Larry, Thanks for the additional history,

I wasn't making the argument in favor of native symlinks based upon the
existence of LNK files.  In my blog post I was simply documenting
various approaches that had been used.  LNK files clearly pre-date NTFS
and Reparse Points in general let alone the Microsoft Symlink Reparse
Point.  It was very unclear from the Cygwin sources what the motivations
were.

Jeffrey Altman

Re: native symlink

[Jeffrey Altman]

On 4/3/2013 11:29 AM, Corinna Vinschen wrote:
> On Apr  3 00:33, Jeffrey Altman wrote:
>> I tried representing AFS Symlinks using a Microsoft assigned Reparse
>> Point Tag.  The downside of following that approach was that Cygwin does
>> not properly handle Reparse Point Tags that it does not recognize.  By
>> discarding the RP attribute and preserving the other reparse point stat
>> information (timestamps, attributes, size, etc) it introduces data
>> corrupting behaviors into Cygwin applications.
> 
> You never explained why this happens and at which point in the code.  So
> far it was the right thing to do, and I'm pretty sure you know why.  I
> don't change that, unless you can show me where and when this leads to
> wrong behaviour.  I asked for these details but you didn't offer an
> explanation besides the fact itself so far.  And it would have been
> no problem to add a special handling for AFS in the cases where it went
> wrong.  I guess this is kind of a moot point, now that you converted
> to native symlinks, but this had to be said.

I highlighted the bad section of code in the patch Christopher commented
on.  The code in question is:

path.cc symlink_info::check_reparse_point() final else block.

/* Maybe it's a reparse point, but it's certainly not one we recognize.
   Drop REPARSE attribute so we don't try to use the flag accidentally.
   It's just some arbitrary file or directory for us. */

   fileattr &= ~FILE_ATTRIBUTE_REPARSE_POINT;

As a result of this change, the timestamps and size of the reparse point
are reported to the application instead of the reparse point target's
stat information.

There are two options that I believe could be implemented here in place
of discarding the reparse point attribute:

1. Open the reparse point target and read its stat information.  Replace
the stat information of the reparse point with that of the target.

2. Open the reparse point target.  Perform a FileNameInformation query
to obtain the actual path of the target.  Replace the reparse point with
a virtual symlink using the FileNameInformation response as the target path.

I believe the 2nd option is the better of the two because it is possible
for a file system driver to implement CreateFile followed by
FileNameInformation queries without requiring that the target be
accessed unless the target is required to determine authorization.

In either case, appropriate checks for reparse points as targets of
reparse points and recursion must be implemented.

>> However, there is a very clear test that can be applied to determine
>> when Microsoft Symlinks should be generated in preference to Cygwin
>> symlinks:
>> [...]
>> There are probably additional approaches but none of them are clean and
>> transparent.  The second two involved significantly more complexity then
>> maintaining support within Cygwin's path.cc and could potentially
>> introduce incompatibilities with future Cygwin path.cc changes.
> 
> No test could be sufficient to switch on native symlinks automatically.
> 
> We were all very excited when it became clear that Microsoft introduced
> native symlinks on NTFS with Vista, and I was early on playing around
> with them and to try integrating them into Cygwin.  My local testcase
> uses DeviceIoControl to workaround any restrictions imposed by
> CreateSymbolicLink.  And I'm still playing around with them every now
> and then, thinking that we could use them, but the restrictions are
> disappointing me each time anew.
> 
> There are some downsides to native symlinks which make them hard to
> justify, if not downright useless in a POSIX environment.
> 
> - The inability of normal users to create symlinks by default.
> 
>     This can be worked around by changing the policy, but it's still a
>     PITA.  Normal users don't know about the policy, some of them don't
>     even have the "Local Security Policy" MMC snap in.  Even in a
>     corporate environment it requires to change the policy settings and
>     we all know how admins don't like to *soften* a policy.  But let's
>     say we can help along with a FAQ entry.

Working around the policy by issuing DeviceIoControl() operations is
possible but will open another can of worms.  I do not believe that
Cygwin should provide a backdoor.

> - Native symlinks are marked as file or directory.
> 
>     This has been added clearly for the benefit of Windows Explorer.
>     But it's a PITA as well because it destroys interoperability.  It's
>     common that POSIX symlinks are created before the target exists.
>     How on earth should the symlink(2) function know if the target is
>     supposed to be a dir or a file.  But Explorer as well as CMD will do
>     the wrong thing if the symlink is using a non-matching dir/file
>     marker.

The target of the symlink must be resolved and the
FILE_ATTRIBUTE_DIRECTORY flag set appropriately for all
GetFileAttribute[Ex] and Find*File[Ex] operation responses.  It is the
inclusion of stat information in the directory enumeration output which
mandates this behavior.

Given the inclusion of stat information and the fact that reparse points
can refer to objects that have a very high latency to access, it is a
reasonable design choice to require the reparse point expose the
FILE_ATTRIBUTE_DIRECTORY bit that the target will have.

I have come to the conclusion that given the need to provide stat
information in the directory enumeration, the implementation of reparse
points is sane.  The implementation permits directory enumeration to be
fast by not requiring the target objects be opened.   For example, a
reparse point to an object stored in a HSM may take hours to load.
Another is a reparse point to a backup snapshot which may require
extended time to restore before it can be accessed.

> - Only Windows paths are stored.
> 
>     In a POSIX env a symlink created by POSIX tools should point to a
>     POSIX path.  For instance, mount points change the fact where a
>     symlink actually points to and the symlink should not still
>     magically work afterwards.
> 
>     But, hey, native symlinks store the path twice, the SubstituteName
>     and the PrintName.  Shouldn't it be possible to store the Windows
>     path in one of them and the POSIX path in the other?  Yes and no.
>     It's possible to write into these members whatever you like, but for
>     some weird reason, both members have to be Windows paths to work for
>     native Windows tools.
> 
>     But we could store the POSIX path with backslashes, thus working
>     around the issue, no?  No.  An absolute path starting with a
>     backslash is possible, but the Windows tools will evaluate it as
>     root-relative to the current drive.  cd to another drive in cmd,
>     and interop is broken again.

It took me a long time to understand how these fields are used.  The
field names were poorly chosen.

The SubstituteName is a path that is used as-is by the Multiple UNC
Provider to redirect a request to the correct file system for
processing.  This is always an absolute path.  In other words, this is
the kernel version of the path.

The PrintName is a user-land UNC path or relative path which is not only
intended for user readability but also for user-land tools such as
robocopy to use when moving a symlink from one location to another.

When storing absolute paths, you must store them as absolute paths from
the device namespace not from the drive namespace.  For example, here is
a symlink stored in AFS which refers to C:\.

[\\afs\yfs\user\jaltman]junction local_disk

\\afs\yfs\user\jaltman\local_disk: SYMBOLIC LINK
   Print Name     : c:\
   Substitute Name: \??\c:\

And here is a symlink stored in c:\ which refers to the root of AFS.

C:\afs: SYMBOLIC LINK
   Print Name     : \\afs\all
   Substitute Name: \??\UNC\afs\all

The output is from the SysInternal's tool, junction v1.06.

Note the inclusion of \??\UNC prior to UNC references and \??\<drive>:\
for DOS device name references.   The DOS device maps to a volume name
and you could provide a link to the volume instead of the DOS device if
that was desirable.

Does this help?

> - Remote and local symlinks may behave different in different environments.
> 
>     Apart from the security policy, symlinks are also affected by an
>     fsutil setting.  The admins can decide if symlinks work at all, or
>     if symlinks don't work depending on their own location and the
>     location of the target they are pointing to (local->local, local->remote,
>     remote->local, remote->remote)
> 
>     So it's possible that local->local symlinks can be resolved while
>     opening local->remote symlinks simply fail with ugly status codes.
>     How on earth do you integrate that reliably into an environment in
>     which a symlink is a plain and simple thing, readable and writable
>     by everyone, whereever located, just apart from parent dir permissions.
> 
>> As I see it, as flawed as Microsoft Symlinks are they are the common
>> interface that enables mixed applications to communicate with one
>> another.  As such, where they can be used, they should be used.  What is
>> the point of cross-platform support if mixed platform applications
>> cannot transparently share the data?
> 
> Cygwin is a POSIX environment in the first place.  Interop is fine,
> but if it collides with POSIX, we're clearly favoring POSIX.

Understood.  Which is why I haven't suggested that cygwin symlinks be
replaced by microsoft symlinks in cases where they cannot be used safely.

> Having said that.
> 
> Chris and I had a private discussion (not the first one on the subject!)
> and we're willing to revisit the use of native symlinks in Cygwin but
> it will be a while before that happens.  A change to the path handling
> code like this is not something that we'd consider for 1.7.18 which is
> long overdue anyway.

Understood.

> What I will do is to add a new CYGWIN environment variable option, along
> the lines of the winsymlinks option(*), or, which is very likely the
> more elgant solution, a mount option, which will result in trying to
> create native symlinks first, and a Cygwin symlink only if creating
> a native symlink failed.  That should help you along.

An environment variable should address James' use case.  For creating
Symlinks in AFS a test for File System name "AFSRDRFsd" in the volume
information can be used as an indicator that DOS SYSTEM attribute is not
supported.

> 
> Corinna
> 
> 
> (*) In your blog you were musing why Cygwin supports lnk files but
> not native symlinks.  Here's the answer:  lnk files support using 
> POSIX paths.

Whereby POSIX paths you mean specifying a path with forward slashes and
without also indicating the type of the object.

Thank you.

Jeffrey Altman

Re: native symlink

[Corinna Vinschen]

On Apr  3 12:52, Jeffrey Altman wrote:
> On 4/3/2013 11:29 AM, Corinna Vinschen wrote:
> > You never explained why this happens and at which point in the code.  So
> > far it was the right thing to do, and I'm pretty sure you know why.  I
> > don't change that, unless you can show me where and when this leads to
> > wrong behaviour.  I asked for these details but you didn't offer an
> > explanation besides the fact itself so far.  And it would have been
> > no problem to add a special handling for AFS in the cases where it went
> > wrong.  I guess this is kind of a moot point, now that you converted
> > to native symlinks, but this had to be said.
> 
> I highlighted the bad section of code in the patch Christopher commented
> on.  The code in question is:
> 
> path.cc symlink_info::check_reparse_point() final else block.
> 
> /* Maybe it's a reparse point, but it's certainly not one we recognize.
>    Drop REPARSE attribute so we don't try to use the flag accidentally.
>    It's just some arbitrary file or directory for us. */
> 
>    fileattr &= ~FILE_ATTRIBUTE_REPARSE_POINT;
>
> As a result of this change, the timestamps and size of the reparse point
> are reported to the application instead of the reparse point target's
> stat information.

No, no, no.  Removing FILE_ATTRIBUTE_REPARSE_POINT is done for
non-symlink reparse points only.  Removing this flag makes sense,
because it avoids the usage of FILE_OPEN_REPARSE_POINT in later
NtCreateFile/NtOpenFile calls.

This is definitely not the culprit why stat information for the reparse
point target is wrong, since that's exactly what's supposed to happen.
Something else is amiss in this situation, but right now, debugging
emacs (*shudder*), I don't see what it is.

> > - The inability of normal users to create symlinks by default.
> > 
> >     This can be worked around by changing the policy, but it's still a
> >     PITA.  Normal users don't know about the policy, some of them don't
> >     even have the "Local Security Policy" MMC snap in.  Even in a
> >     corporate environment it requires to change the policy settings and
> >     we all know how admins don't like to *soften* a policy.  But let's
> >     say we can help along with a FAQ entry.
> 
> Working around the policy by issuing DeviceIoControl() operations is
> possible but will open another can of worms.  I do not believe that
> Cygwin should provide a backdoor.

DeviceIoControl is no backdoor.  It still imposes kernel restrictions.
And it's how CreateSymbolicLink is implemented under the hood anyway
(or rather: NtFsControlFile).  We're not using W32api functions for
file access.

> > - Native symlinks are marked as file or directory.
> > 
> >     This has been added clearly for the benefit of Windows Explorer.
> >     But it's a PITA as well because it destroys interoperability.  It's
> >     common that POSIX symlinks are created before the target exists.
> >     How on earth should the symlink(2) function know if the target is
> >     supposed to be a dir or a file.  But Explorer as well as CMD will do
> >     the wrong thing if the symlink is using a non-matching dir/file
> >     marker.
> 
> The target of the symlink must be resolved and the
> FILE_ATTRIBUTE_DIRECTORY flag set appropriately for all
> GetFileAttribute[Ex] and Find*File[Ex] operation responses.  It is the
> inclusion of stat information in the directory enumeration output which
> mandates this behavior.
> 
> Given the inclusion of stat information and the fact that reparse points
> can refer to objects that have a very high latency to access, it is a
> reasonable design choice to require the reparse point expose the
> FILE_ATTRIBUTE_DIRECTORY bit that the target will have.

It's non-POSIXy.  That's my point of view in the first place and
that's what we're discussing here.

> >     But we could store the POSIX path with backslashes, thus working
> >     around the issue, no?  No.  An absolute path starting with a
> >     backslash is possible, but the Windows tools will evaluate it as
> >     root-relative to the current drive.  cd to another drive in cmd,
> >     and interop is broken again.
> 
> It took me a long time to understand how these fields are used.  The
> field names were poorly chosen.
> [...]
> Does this help?

Well, not really since it's nothing new.  I guess we should stop trying
to educate each other.  We all had our share of debugging how native
symlinks work, apparently.

> An environment variable should address James' use case.  For creating
> Symlinks in AFS a test for File System name "AFSRDRFsd" in the volume
> information can be used as an indicator that DOS SYSTEM attribute is not
> supported.

http://cygwin.com/acronyms/#PTC

An addition to the fs_info class, the fs_names array, and the
fs_info::update function would be fine (post-1.7.18).


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: native symlink

[Corinna Vinschen]

On Apr  3 19:29, Corinna Vinschen wrote:
> On Apr  3 12:52, Jeffrey Altman wrote:
> > On 4/3/2013 11:29 AM, Corinna Vinschen wrote:
> > > You never explained why this happens and at which point in the code.  So
> > > far it was the right thing to do, and I'm pretty sure you know why.  I
> > > don't change that, unless you can show me where and when this leads to
> > > wrong behaviour.  I asked for these details but you didn't offer an
> > > explanation besides the fact itself so far.  And it would have been
> > > no problem to add a special handling for AFS in the cases where it went
> > > wrong.  I guess this is kind of a moot point, now that you converted
> > > to native symlinks, but this had to be said.
> > 
> > I highlighted the bad section of code in the patch Christopher commented
> > on.  The code in question is:
> > 
> > path.cc symlink_info::check_reparse_point() final else block.
> > 
> > /* Maybe it's a reparse point, but it's certainly not one we recognize.
> >    Drop REPARSE attribute so we don't try to use the flag accidentally.
> >    It's just some arbitrary file or directory for us. */
> > 
> >    fileattr &= ~FILE_ATTRIBUTE_REPARSE_POINT;
> >
> > As a result of this change, the timestamps and size of the reparse point
> > are reported to the application instead of the reparse point target's
> > stat information.
> 
> No, no, no.  Removing FILE_ATTRIBUTE_REPARSE_POINT is done for
> non-symlink reparse points only.  Removing this flag makes sense,
> because it avoids the usage of FILE_OPEN_REPARSE_POINT in later
> NtCreateFile/NtOpenFile calls.
> 
> This is definitely not the culprit why stat information for the reparse
> point target is wrong, since that's exactly what's supposed to happen.
> Something else is amiss in this situation, but right now, debugging
> emacs (*shudder*), I don't see what it is.

I think I know what the problem is here.  Even though Cygwin didn't
recognize the reparse point type and even though it lets the OS handle
the reparse point in subsequent calls, the PC_KEEP_HANDLE flag is not
removed.  The effect is that stat uses the metadata collected in the
symlink_info::check call, which is the reparse point metadata.  So, I
think the right thing to do here is to remove the PC_KEEP_HANDLE flag so
stat reopens the reparse point, but this time without the
FILE_OPEN_REPARSE_POINT flag. 

You're set up to build Cygwin yourself, right?  Can you please try the
below patch?


Thanks,
Corinna


Index: path.cc
===================================================================
RCS file: /cvs/src/src/winsup/cygwin/path.cc,v
retrieving revision 1.672
diff -u -p -r1.672 path.cc
--- path.cc	3 Apr 2013 11:20:36 -0000	1.672
+++ path.cc	3 Apr 2013 20:46:08 -0000
@@ -2668,7 +2668,14 @@ restart:
 		 to the volumes root dir. */
 	      pflags &= ~PC_KEEP_HANDLE;
 	    }
-	  else if (res)
+	  else if (!res)
+	    {
+	      /* Make sure the open handle is not used in later stat calls.
+		 We didn't recognize the reparse point type, so let the
+		 OS handle this the default way. */
+	      pflags &= ~PC_KEEP_HANDLE;
+	    }
+	  else
 	    {
 	      /* A symlink is never a directory. */
 	      conv_hdl.fnoi ()->FileAttributes &= ~FILE_ATTRIBUTE_DIRECTORY;

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: native symlink

[James Gregurich]

On Apr 3, 2013, at 8:29 AM, Corinna Vinschen wrote:

> 
> 
> There are some downsides to native symlinks which make them hard to
> justify, if not downright useless in a POSIX environment.
> 

We all understand the limitations and we grant you that. That is why I have proposed an add-on feature rather than an augmentation of the symlink() function. Add an additional API call in the spirit of existing functions such as cygwin_conv_path_list() that will allow an existing cygwin symlink to be replaced by a native symlink. Those who need native symlinks can use that function call. As an add-on, optional API call, he function can accept flags that allow the user to choose how to handle non-optimal situations such as relative paths that exceed PATH_MAX, and it can return error codes that indicate exactly why a native symlink failed to be produced.  This way, there is no ambiguity that can generate support tickets.





>> As I see it, as flawed as Microsoft Symlinks are they are the common
>> interface that enables mixed applications to communicate with one
>> another.  As such, where they can be used, they should be used.  What is
>> the point of cross-platform support if mixed platform applications
>> cannot transparently share the data?
> 
> Cygwin is a POSIX environment in the first place.  Interop is fine,
> but if it collides with POSIX, we're clearly favoring POSIX.

I'm suggesting an approach that does not conflict with posix. However, I don't know if that concept works for Jeffery or not. 


> Having said that.
> 
> Chris and I had a private discussion (not the first one on the subject!)
> and we're willing to revisit the use of native symlinks in Cygwin but
> it will be a while before that happens.  A change to the path handling
> code like this is not something that we'd consider for 1.7.18 which is
> long overdue anyway.

fair enough. If its on the radar, I'm happy. What I have should be functional for a long time. Like I said, I can contribute my code that you can use as a guide.





> 
> What I will do is to add a new CYGWIN environment variable option, along
> the lines of the winsymlinks option(*), or, which is very likely the
> more elgant solution, a mount option, which will result in trying to
> create native symlinks first, and a Cygwin symlink only if creating
> a native symlink failed.  That should help you along.

If you want to go all the way, that works for me too.

Re: native symlink

[Jeffrey Altman]

I will add testing this to my to-do list.
It may be a few days.


On 4/3/2013 4:46 PM, Corinna Vinschen wrote:

> I think I know what the problem is here.  Even though Cygwin didn't
> recognize the reparse point type and even though it lets the OS handle
> the reparse point in subsequent calls, the PC_KEEP_HANDLE flag is not
> removed.  The effect is that stat uses the metadata collected in the
> symlink_info::check call, which is the reparse point metadata.  So, I
> think the right thing to do here is to remove the PC_KEEP_HANDLE flag so
> stat reopens the reparse point, but this time without the
> FILE_OPEN_REPARSE_POINT flag. 
> 
> You're set up to build Cygwin yourself, right?  Can you please try the
> below patch?
> 
> 
> Thanks,
> Corinna
> 
> 
> Index: path.cc
> ===================================================================
> RCS file: /cvs/src/src/winsup/cygwin/path.cc,v
> retrieving revision 1.672
> diff -u -p -r1.672 path.cc
> --- path.cc	3 Apr 2013 11:20:36 -0000	1.672
> +++ path.cc	3 Apr 2013 20:46:08 -0000
> @@ -2668,7 +2668,14 @@ restart:
>  		 to the volumes root dir. */
>  	      pflags &= ~PC_KEEP_HANDLE;
>  	    }
> -	  else if (res)
> +	  else if (!res)
> +	    {
> +	      /* Make sure the open handle is not used in later stat calls.
> +		 We didn't recognize the reparse point type, so let the
> +		 OS handle this the default way. */
> +	      pflags &= ~PC_KEEP_HANDLE;
> +	    }
> +	  else
>  	    {
>  	      /* A symlink is never a directory. */
>  	      conv_hdl.fnoi ()->FileAttributes &= ~FILE_ATTRIBUTE_DIRECTORY;
> 

Re: native symlink

[Corinna Vinschen]

Hi Jeffrey,

On Apr  3 17:35, Jeffrey Altman wrote:
> I will add testing this to my to-do list.
> It may be a few days.

any chance to test this?


Thanks,
Corinna

> 
> 
> On 4/3/2013 4:46 PM, Corinna Vinschen wrote:
> 
> > I think I know what the problem is here.  Even though Cygwin didn't
> > recognize the reparse point type and even though it lets the OS handle
> > the reparse point in subsequent calls, the PC_KEEP_HANDLE flag is not
> > removed.  The effect is that stat uses the metadata collected in the
> > symlink_info::check call, which is the reparse point metadata.  So, I
> > think the right thing to do here is to remove the PC_KEEP_HANDLE flag so
> > stat reopens the reparse point, but this time without the
> > FILE_OPEN_REPARSE_POINT flag. 
> > 
> > You're set up to build Cygwin yourself, right?  Can you please try the
> > below patch?
> > 
> > 
> > Thanks,
> > Corinna
> > 
> > 
> > Index: path.cc
> > ===================================================================
> > RCS file: /cvs/src/src/winsup/cygwin/path.cc,v
> > retrieving revision 1.672
> > diff -u -p -r1.672 path.cc
> > --- path.cc	3 Apr 2013 11:20:36 -0000	1.672
> > +++ path.cc	3 Apr 2013 20:46:08 -0000
> > @@ -2668,7 +2668,14 @@ restart:
> >  		 to the volumes root dir. */
> >  	      pflags &= ~PC_KEEP_HANDLE;
> >  	    }
> > -	  else if (res)
> > +	  else if (!res)
> > +	    {
> > +	      /* Make sure the open handle is not used in later stat calls.
> > +		 We didn't recognize the reparse point type, so let the
> > +		 OS handle this the default way. */
> > +	      pflags &= ~PC_KEEP_HANDLE;
> > +	    }
> > +	  else
> >  	    {
> >  	      /* A symlink is never a directory. */
> >  	      conv_hdl.fnoi ()->FileAttributes &= ~FILE_ATTRIBUTE_DIRECTORY;
> > 

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: native symlink

[Jeffrey Altman]

Corinna,

Sorry for the delay in testing but I can now confirm that this patch
corrects the behavior in which the wrong status information was being
used.

Jeffrey Altman

On 4/3/2013 4:46 PM, Corinna Vinschen wrote:
> You're set up to build Cygwin yourself, right?  Can you please try the
> below patch?
> 
> 
> Thanks,
> Corinna
> 
> 
> Index: path.cc
> ===================================================================
> RCS file: /cvs/src/src/winsup/cygwin/path.cc,v
> retrieving revision 1.672
> diff -u -p -r1.672 path.cc
> --- path.cc	3 Apr 2013 11:20:36 -0000	1.672
> +++ path.cc	3 Apr 2013 20:46:08 -0000
> @@ -2668,7 +2668,14 @@ restart:
>  		 to the volumes root dir. */
>  	      pflags &= ~PC_KEEP_HANDLE;
>  	    }
> -	  else if (res)
> +	  else if (!res)
> +	    {
> +	      /* Make sure the open handle is not used in later stat calls.
> +		 We didn't recognize the reparse point type, so let the
> +		 OS handle this the default way. */
> +	      pflags &= ~PC_KEEP_HANDLE;
> +	    }
> +	  else
>  	    {
>  	      /* A symlink is never a directory. */
>  	      conv_hdl.fnoi ()->FileAttributes &= ~FILE_ATTRIBUTE_DIRECTORY;
> 

Re: native symlink

[Corinna Vinschen]

On Apr 11 17:55, Jeffrey Altman wrote:
> Corinna,
> 
> Sorry for the delay in testing

No worries.

> but I can now confirm that this patch
> corrects the behavior in which the wrong status information was being
> used.

Thanks for testing.  I checked in a slightly reworked version of the
patch.  Can you give it another try, just to be sure?


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: native symlink

[Jeffrey Altman]

On 4/12/2013 4:33 AM, Corinna Vinschen wrote:
> On Apr 11 17:55, Jeffrey Altman wrote:
>> Corinna,
>>
>> Sorry for the delay in testing
> 
> No worries.
> 
>> but I can now confirm that this patch
>> corrects the behavior in which the wrong status information was being
>> used.
> 
> Thanks for testing.  I checked in a slightly reworked version of the
> patch.  Can you give it another try, just to be sure?

Reading the code I believe the revised patch is fine.
However, I'm suddenly unable to build anything.  gcc 4.5.3 produces:

$ gcc hello.c
gcc: Internal error: Segmentation fault (program cc1)

on even the most basic C programs.  I haven't had time to dive into what
might be happening.

Re: native symlink

[Corinna Vinschen]

On Apr 13 09:08, Jeffrey Altman wrote:
> On 4/12/2013 4:33 AM, Corinna Vinschen wrote:
> > On Apr 11 17:55, Jeffrey Altman wrote:
> >> Corinna,
> >>
> >> Sorry for the delay in testing
> > 
> > No worries.
> > 
> >> but I can now confirm that this patch
> >> corrects the behavior in which the wrong status information was being
> >> used.
> > 
> > Thanks for testing.  I checked in a slightly reworked version of the
> > patch.  Can you give it another try, just to be sure?
> 
> Reading the code I believe the revised patch is fine.
> However, I'm suddenly unable to build anything.  gcc 4.5.3 produces:
> 
> $ gcc hello.c
> gcc: Internal error: Segmentation fault (program cc1)
> 
> on even the most basic C programs.  I haven't had time to dive into what
> might be happening.

Weird.  I just built Cygwin CVS HEAD from scratch and it works fine
for me.  I tried tcsh, ls, and building a simple executable with gcc.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: native symlink

[Corinna Vinschen]

On Apr  3 17:29, Corinna Vinschen wrote:
> Cygwin is a POSIX environment in the first place.  Interop is fine,
> but if it collides with POSIX, we're clearly favoring POSIX.
> 
> Having said that.
> 
> Chris and I had a private discussion (not the first one on the subject!)
> and we're willing to revisit the use of native symlinks in Cygwin but
> it will be a while before that happens.  A change to the path handling
> code like this is not something that we'd consider for 1.7.18 which is
> long overdue anyway.
> 
> What I will do is to add a new CYGWIN environment variable option, along
> the lines of the winsymlinks option(*), or, which is very likely the
> more elgant solution, a mount option, which will result in trying to
> create native symlinks first, and a Cygwin symlink only if creating
> a native symlink failed.  That should help you along.

I just applied a patch to CVS which adds AFS support as well as native
symlink support.  On AFS, native symlinks are used exclusively, on any
other filesystem supporting native symlinks Cygwin will try to create
them if you specify "winsymlinks:native" in the $CYGWIN environment
variable.

After mulling over this problem I found that using an environment
solution is better than the mount point solution, because this allows
on-the-fly creating of native symlinks in certain scenarios, while the
default can be kept at using Cygwin sysfile symlinks, which are still
better suited for a POSIX environment.

For completeness, you can also specify "winsymlinks:lnk" or just 
"winsymlinks".  This will result in trying to generate shortcut
type symlinks, as before.

Jeffrey, please give especially AFS at try here.


HTH,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: native symlink

[Jeffrey Altman]

On 4/24/2013 6:34 AM, Corinna Vinschen wrote:
> On Apr  3 17:29, Corinna Vinschen wrote:
>> Cygwin is a POSIX environment in the first place.  Interop is fine,
>> but if it collides with POSIX, we're clearly favoring POSIX.
>>
>> Having said that.
>>
>> Chris and I had a private discussion (not the first one on the subject!)
>> and we're willing to revisit the use of native symlinks in Cygwin but
>> it will be a while before that happens.  A change to the path handling
>> code like this is not something that we'd consider for 1.7.18 which is
>> long overdue anyway.
>>
>> What I will do is to add a new CYGWIN environment variable option, along
>> the lines of the winsymlinks option(*), or, which is very likely the
>> more elegant solution, a mount option, which will result in trying to
>> create native symlinks first, and a Cygwin symlink only if creating
>> a native symlink failed.  That should help you along.
> 
> I just applied a patch to CVS which adds AFS support as well as native
> symlink support.  On AFS, native symlinks are used exclusively, on any
> other filesystem supporting native symlinks Cygwin will try to create
> them if you specify "winsymlinks:native" in the $CYGWIN environment
> variable.
> 
> After mulling over this problem I found that using an environment
> solution is better than the mount point solution, because this allows
> on-the-fly creating of native symlinks in certain scenarios, while the
> default can be kept at using Cygwin sysfile symlinks, which are still
> better suited for a POSIX environment.
> 
> For completeness, you can also specify "winsymlinks:lnk" or just 
> "winsymlinks".  This will result in trying to generate shortcut
> type symlinks, as before.
> 
> Jeffrey, please give especially AFS at try here.
> 
> 
> HTH,
> Corinna

Hi Corinna,

I've confirmed that the unrecognized reparse point fix in 1.7.18-1
does work.  Unfortunately, I'm still unable to get gcc (actually cc1)
to build even a simple hello world c program.   Is there is a nightly
binary distribution I can use to test from?

Jeffrey Altman

Re: native symlink

[Corinna Vinschen]

On Apr 24 08:06, Jeffrey Altman wrote:
> On 4/24/2013 6:34 AM, Corinna Vinschen wrote:
> > I just applied a patch to CVS which adds AFS support as well as native
> > symlink support.  On AFS, native symlinks are used exclusively, on any
> > other filesystem supporting native symlinks Cygwin will try to create
> > them if you specify "winsymlinks:native" in the $CYGWIN environment
> > variable.
> > 
> > After mulling over this problem I found that using an environment
> > solution is better than the mount point solution, because this allows
> > on-the-fly creating of native symlinks in certain scenarios, while the
> > default can be kept at using Cygwin sysfile symlinks, which are still
> > better suited for a POSIX environment.
> > 
> > For completeness, you can also specify "winsymlinks:lnk" or just 
> > "winsymlinks".  This will result in trying to generate shortcut
> > type symlinks, as before.
> > 
> > Jeffrey, please give especially AFS at try here.
> 
> I've confirmed that the unrecognized reparse point fix in 1.7.18-1
> does work.  Unfortunately, I'm still unable to get gcc (actually cc1)
> to build even a simple hello world c program.   Is there is a nightly
> binary distribution I can use to test from?

i686:

  http://cygwin.com/snapshots/

x86_64:

  ftp://cygwin.com/pub/cygwin/64bit/setup64.exe

The latest snapshot from today, as well as the latest 64 bit Cygwin
package (1.7.19-2) both contain these patches.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: native symlink

[James Gregurich]

Thanks. I will test the symlinks with my repositories.

-James 

On Apr 24, 2013, at 5:50 AM, Corinna Vinschen wrote:

> On Apr 24 08:06, Jeffrey Altman wrote:
>> On 4/24/2013 6:34 AM, Corinna Vinschen wrote:
>>> I just applied a patch to CVS which adds AFS support as well as native
>>> symlink support.  On AFS, native symlinks are used exclusively, on any
>>> other filesystem supporting native symlinks Cygwin will try to create
>>> them if you specify "winsymlinks:native" in the $CYGWIN environment
>>> variable.
>>> 
>>> After mulling over this problem I found that using an environment
>>> solution is better than the mount point solution, because this allows
>>> on-the-fly creating of native symlinks in certain scenarios, while the
>>> default can be kept at using Cygwin sysfile symlinks, which are still
>>> better suited for a POSIX environment.
>>> 
>>> For completeness, you can also specify "winsymlinks:lnk" or just 
>>> "winsymlinks".  This will result in trying to generate shortcut
>>> type symlinks, as before.
>>> 
>>> Jeffrey, please give especially AFS at try here.
>> 
>> I've confirmed that the unrecognized reparse point fix in 1.7.18-1
>> does work.  Unfortunately, I'm still unable to get gcc (actually cc1)
>> to build even a simple hello world c program.   Is there is a nightly
>> binary distribution I can use to test from?
> 
> i686:
> 
>  http://cygwin.com/snapshots/
> 
> x86_64:
> 
>  ftp://cygwin.com/pub/cygwin/64bit/setup64.exe
> 
> The latest snapshot from today, as well as the latest 64 bit Cygwin
> package (1.7.19-2) both contain these patches.
> 
> 
> Corinna
> 
> -- 
> Corinna Vinschen                  Please, send mails regarding Cygwin to
> Cygwin Maintainer                 cygwin AT cygwin DOT com
> Red Hat

Re: native symlink

[Jeffrey Altman]

On 4/24/2013 8:50 AM, Corinna Vinschen wrote:
> On Apr 24 08:06, Jeffrey Altman wrote:
>> On 4/24/2013 6:34 AM, Corinna Vinschen wrote:
>>> I just applied a patch to CVS which adds AFS support as well as native
>>> symlink support.  On AFS, native symlinks are used exclusively, on any
>>> other filesystem supporting native symlinks Cygwin will try to create
>>> them if you specify "winsymlinks:native" in the $CYGWIN environment
>>> variable.
>>>
>>> After mulling over this problem I found that using an environment
>>> solution is better than the mount point solution, because this allows
>>> on-the-fly creating of native symlinks in certain scenarios, while the
>>> default can be kept at using Cygwin sysfile symlinks, which are still
>>> better suited for a POSIX environment.
>>>
>>> For completeness, you can also specify "winsymlinks:lnk" or just 
>>> "winsymlinks".  This will result in trying to generate shortcut
>>> type symlinks, as before.
>>>
>>> Jeffrey, please give especially AFS at try here.
>>
>> I've confirmed that the unrecognized reparse point fix in 1.7.18-1
>> does work.  Unfortunately, I'm still unable to get gcc (actually cc1)
>> to build even a simple hello world c program.   Is there is a nightly
>> binary distribution I can use to test from?
> 
> i686:
> 
>   http://cygwin.com/snapshots/

I have confirmed that the change works on AFS without the environment
setting.

Thanks.

Jeffrey Altman

Re: native symlink

[Corinna Vinschen]

On Apr 24 13:55, Jeffrey Altman wrote:
> On 4/24/2013 8:50 AM, Corinna Vinschen wrote:
> > On Apr 24 08:06, Jeffrey Altman wrote:
> >> On 4/24/2013 6:34 AM, Corinna Vinschen wrote:
> >>> I just applied a patch to CVS which adds AFS support as well as native
> >>> symlink support.  On AFS, native symlinks are used exclusively, on any
> >>> other filesystem supporting native symlinks Cygwin will try to create
> >>> them if you specify "winsymlinks:native" in the $CYGWIN environment
> >>> variable.
> >>>
> >>> After mulling over this problem I found that using an environment
> >>> solution is better than the mount point solution, because this allows
> >>> on-the-fly creating of native symlinks in certain scenarios, while the
> >>> default can be kept at using Cygwin sysfile symlinks, which are still
> >>> better suited for a POSIX environment.
> >>>
> >>> For completeness, you can also specify "winsymlinks:lnk" or just 
> >>> "winsymlinks".  This will result in trying to generate shortcut
> >>> type symlinks, as before.
> >>>
> >>> Jeffrey, please give especially AFS at try here.
> >>
> >> I've confirmed that the unrecognized reparse point fix in 1.7.18-1
> >> does work.  Unfortunately, I'm still unable to get gcc (actually cc1)
> >> to build even a simple hello world c program.   Is there is a nightly
> >> binary distribution I can use to test from?
> > 
> > i686:
> > 
> >   http://cygwin.com/snapshots/
> 
> I have confirmed that the change works on AFS without the environment
> setting.

Cool.  Thanks for the feedback.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: native symlink

[Jeffrey Altman]

On 4/24/2013 2:14 PM, Corinna Vinschen wrote:
> 
> Cool.  Thanks for the feedback.
> 

Thanks again for the patch.

Jeffrey Altman

Re: native symlink

[James Gregurich]

I had one of my guys test the work this morning. The creation of symlinks works fine. Did you implement a utility that can convert an existing cygwin symlink to native form?






On Apr 24, 2013, at 11:14 AM, Corinna Vinschen  wrote:

> On Apr 24 13:55, Jeffrey Altman wrote:
>> On 4/24/2013 8:50 AM, Corinna Vinschen wrote:
>>> On Apr 24 08:06, Jeffrey Altman wrote:
>>>> On 4/24/2013 6:34 AM, Corinna Vinschen wrote:
>>>>> I just applied a patch to CVS which adds AFS support as well as native
>>>>> symlink support.  On AFS, native symlinks are used exclusively, on any
>>>>> other filesystem supporting native symlinks Cygwin will try to create
>>>>> them if you specify "winsymlinks:native" in the $CYGWIN environment
>>>>> variable.
>>>>> 
>>>>> After mulling over this problem I found that using an environment
>>>>> solution is better than the mount point solution, because this allows
>>>>> on-the-fly creating of native symlinks in certain scenarios, while the
>>>>> default can be kept at using Cygwin sysfile symlinks, which are still
>>>>> better suited for a POSIX environment.
>>>>> 
>>>>> For completeness, you can also specify "winsymlinks:lnk" or just 
>>>>> "winsymlinks".  This will result in trying to generate shortcut
>>>>> type symlinks, as before.
>>>>> 
>>>>> Jeffrey, please give especially AFS at try here.
>>>> 
>>>> I've confirmed that the unrecognized reparse point fix in 1.7.18-1
>>>> does work.  Unfortunately, I'm still unable to get gcc (actually cc1)
>>>> to build even a simple hello world c program.   Is there is a nightly
>>>> binary distribution I can use to test from?
>>> 
>>> i686:
>>> 
>>>  http://cygwin.com/snapshots/
>> 
>> I have confirmed that the change works on AFS without the environment
>> setting.
> 
> Cool.  Thanks for the feedback.
> 
> 
> Corinna
> 
> -- 
> Corinna Vinschen                  Please, send mails regarding Cygwin to
> Cygwin Maintainer                 cygwin AT cygwin DOT com
> Red Hat

utility to update existing cygwin symlinks to native format? (was Re: native symlink)

[James Gregurich]

I have not gotten a response to this question yet.  I'd like to get into a position to roll this version of cygwin out so that we can use it daily and look for glitches, but I do need the ability to upgrade existing symlinks before that is feasible. Is there an answer to my question?


-James


On Apr 26, 2013, at 4:39 PM, James Gregurich  wrote:

> I had one of my guys test the work this morning. The creation of symlinks works fine. Did you implement a utility that can convert an existing cygwin symlink to native form?
> 
> 
> 

Re: utility to update existing cygwin symlinks to native format? (was Re: native symlink)

[Larry Hall (Cygwin Developers)]

On 4/29/2013 6:04 PM, James Gregurich wrote:
> I have not gotten a response to this question yet. I'd like to get into
> a  position to roll this version of cygwin out so that we can use it daily and
> look for glitches, but I do need the ability to upgrade existing symlinks
> before that is feasible. Is there an answer to my question?

You'll have to wait until Corinna returns from vacation at least for the
answer to whether or not this has been done already.  I can't say if
you've inspired Corinna (or someone else) to implement this for you.
If not, you may have to create your own utility for this part.

-- 
Larry

Re: utility to update existing cygwin symlinks to native format? (was Re: native symlink)

[James Gregurich]

On Apr 29, 2013, at 4:45 PM, "Larry Hall (Cygwin Developers)"  wrote:

> On 4/29/2013 6:04 PM, James Gregurich wrote:
>> I have not gotten a response to this question yet. I'd like to get into
>> a  position to roll this version of cygwin out so that we can use it daily and
>> look for glitches, but I do need the ability to upgrade existing symlinks
>> before that is feasible. Is there an answer to my question?
> 
> You'll have to wait until Corinna returns from vacation at least for the
> answer to whether or not this has been done already.  I can't say if
> you've inspired Corinna (or someone else) to implement this for you.
> If not, you may have to create your own utility for this part.
> 
> -- 
> Larry

ok. thanks for the info

The utility that I wrote for myself depends upon an exposed API function in path.cc since all the logic to handle the internal format of the cygwin symlink file is contained in path.cc. I could easily write one for myself if such a function were exposed as a public API. I'd just need the specs to use the call.

-James

Re: utility to update existing cygwin symlinks to native format? (was Re: native symlink)

[James Gregurich]

On Apr 29, 2013, at 4:49 PM, James Gregurich <bayoubengal@mac.com> wrote:

> 
> 
> On Apr 29, 2013, at 4:45 PM, "Larry Hall (Cygwin Developers)"  wrote:
> 
>> On 4/29/2013 6:04 PM, James Gregurich wrote:
>>> I have not gotten a response to this question yet. I'd like to get into
>>> a  position to roll this version of cygwin out so that we can use it daily and
>>> look for glitches, but I do need the ability to upgrade existing symlinks
>>> before that is feasible. Is there an answer to my question?
>> 
>> You'll have to wait until Corinna returns from vacation at least for the
>> answer to whether or not this has been done already.  I can't say if
>> you've inspired Corinna (or someone else) to implement this for you.
>> If not, you may have to create your own utility for this part.
>> 
>> -- 
>> Larry
> 
> ok. thanks for the info
> 
> The utility that I wrote for myself depends upon an exposed API function in path.cc since all the logic to handle the internal format of the cygwin symlink file is contained in path.cc. I could easily write one for myself if such a function were exposed as a public API. I'd just need the specs to use the call.
> 
> -James
> 


BTW: Since one cannot rely on the native link being created, such a utility is necessary for practical use of the feature.

-James

Re: utility to update existing cygwin symlinks to native format? (was Re: native symlink)

[Christopher Faylor]

On Mon, Apr 29, 2013 at 04:49:33PM -0700, James Gregurich wrote:
>
>
>On Apr 29, 2013, at 4:45 PM, "Larry Hall (Cygwin Developers)"  wrote:
>
>> On 4/29/2013 6:04 PM, James Gregurich wrote:
>>> I have not gotten a response to this question yet. I'd like to get into
>>> a  position to roll this version of cygwin out so that we can use it daily and
>>> look for glitches, but I do need the ability to upgrade existing symlinks
>>> before that is feasible. Is there an answer to my question?
>> 
>> You'll have to wait until Corinna returns from vacation at least for the
>> answer to whether or not this has been done already.  I can't say if
>> you've inspired Corinna (or someone else) to implement this for you.
>> If not, you may have to create your own utility for this part.
>
>ok. thanks for the info
>
>The utility that I wrote for myself depends upon an exposed API
>function in path.cc since all the logic to handle the internal format
>of the cygwin symlink file is contained in path.cc.  I could easily
>write one for myself if such a function were exposed as a public API.
>I'd just need the specs to use the call.

If you're asking "Did Corinna write a utility to do this?" the answer
is extremely likely to be "No" since she would have mentioned it if
she had.

If you're asking "Will Corinna write a utility to do this?" I suspect
that the answer is likely also "No".

cgf

Re: utility to update existing cygwin symlinks to native format? (was Re: native symlink)

[James Gregurich]

On Apr 29, 2013, at 5:25 PM, Christopher Faylor  wrote:

> 
> If you're asking "Did Corinna write a utility to do this?" the answer
> is extremely likely to be "No" since she would have mentioned it if
> she had.
> 
> If you're asking "Will Corinna write a utility to do this?" I suspect
> that the answer is likely also "No".
> 
> cgf

Unfortunately, the utility is a necessity for practical use of the native symlink feature.  I'm not opposed to writing it if I can get the necessary public function exposed in path.cc.  I suppose the discussion can be tabled until she gets back. At a bare minimum, I'd need a function that returns the format of the symlink in question. With such a function, I could just delete the link and recreate it if the format is not native.

-James

Re: utility to update existing cygwin symlinks to native format? (was Re: native symlink)

[Charles Wilson]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=windows-1252; format=flowed, Size: 2893 bytes --]

On 4/29/2013 8:34 PM, James Gregurich wrote:
> On Apr 29, 2013, at 5:25 PM, Christopher Faylor  wrote:
>> If you're asking "Did Corinna write a utility to do this?" the answer
>> is extremely likely to be "No" since she would have mentioned it if
>> she had.
>>
>> If you're asking "Will Corinna write a utility to do this?" I suspect
>> that the answer is likely also "No".
>
> Unfortunately, the utility is a necessity for practical use of the native symlink feature.  I'm not opposed to writing it if I can get the necessary public function exposed in path.cc.  I suppose the discussion can be tabled until she gets back. At a bare minimum, I'd need a function that returns the format of the symlink in question. With such a function, I could just delete the link and recreate it if the format is not native.

The winln program is part of the most recent cygutils package (1.4.12). 
It was written by Daniel Colascione...


NAME
        winln - create a Windows symbolic link

SYNOPSIS
        winln [-svfdFA] TARGET LINKNAME
        winln [-svfdFA] TARGET
        winln [-svfdFA] TARGET... DIRECTORY
        winln [-svfdFA] -t DIRECTORY TARGET...

DESCRIPTION
        winln  is  a  drop-in  replacement for ln(1), the difference 
being that
        winln creates Windows symbolic links instead of Cygwin ones. 
Note that
        Windows	has  two  kinds	 of  symbolic  links: file links and 
directory
        links.  winln automatically chooses the correct type of link 
when  the
        target  exists,	but  if	 the  target  does  not exist, you may 
want to
        explicitly specify the kind of link to create.  Links  to 
non-existent
        targets default to file links.

OPTIONS
        -s, --symbolic
	      Create  a symbolic (as opposed to hard) link.  For compatibility
	      with ln(1), creating hard links is the default.


        -v, --verbose
	      Print to standard output the names of links created.


        -f, --force
	      Replace existing destination file.


        -d, --directory
	      Create a directory symbolic link.


        -F, --file
	      Create a file symbolic link.


        -A, --auto
	      Automatically determine the type of symbolic link to create.  If
	      the  target  exists  and is a directory, create a directory sym‐
	      bolic link.  Otherwise, create a file symbolic link.


        -t, --target-directory
	      Treat DIRECTORY as the directory in which to create links	 based
	      on the remaining arguments.


        -T, --no-target-directory
	      Treat LINKNAME as a normal file always.


        -h, --help
	      Show brief help message.


        --version
	      Display version information


NOT IMPLEMENTED
        winln  does not implement the following command line arguments 
from GNU
        ln(1):
        -b, --backup, -i, --interactive, -L, --logical,	-n, 
--no-dereference,
        -P, --physical

Re: utility to update existing cygwin symlinks to native format? (was Re: native symlink)

[James Gregurich]

On Apr 29, 2013, at 5:44 PM, Charles Wilson wrote:

> On 4/29/2013 8:34 PM, James Gregurich wrote:
>> On Apr 29, 2013, at 5:25 PM, Christopher Faylor  wrote:
>>> If you're asking "Did Corinna write a utility to do this?" the answer
>>> is extremely likely to be "No" since she would have mentioned it if
>>> she had.
>>> 
>>> If you're asking "Will Corinna write a utility to do this?" I suspect
>>> that the answer is likely also "No".
>> 
>> Unfortunately, the utility is a necessity for practical use of the native symlink feature.  I'm not opposed to writing it if I can get the necessary public function exposed in path.cc.  I suppose the discussion can be tabled until she gets back. At a bare minimum, I'd need a function that returns the format of the symlink in question. With such a function, I could just delete the link and recreate it if the format is not native.
> 
> The winln program is part of the most recent cygutils package (1.4.12). It was written by Daniel Colascione...
> 


Reading the man-page you supplied, it doesn't sound like this utility is intended to replace an existing default symlink with a native one. 

The issue I see with this utility in terms of using it as a replacement for 'ln -s' is that it defaults to a file symlink type if the target does not exist. The desired behavior would be to fall back to the default implementation so that the directory tree is never invalid.  Also, if you are using software that has embedded calls to 'ln -s' or uses the symlinks() API call, this wouldn't help.


However, (with a little inspiration from Jeffery), I did think of a way to do it without further changing cygwin.  One could use the Win32 APIs to verify that a target file is a reparse point symlink. if it isn't, then one could delete it and recreate it through cygwin.  I will attempt that approach and report back in the near future.


-James

native symlink support should fallback to default format if target missing

[James Gregurich]

The guy I have testing the native symlink support in the new cygwin is reporting to me that if the target of the link does not exist, the mechanism is creating a file reparse point. This is not desirable behavior. When the target comes into existence, if it is a folder, then the native symlink is invalid.  What the mechanism should do is fall back to the native symlink format if the target doesn't exist. That way, the link is never invalid. Since it is a default format symlink, then my test for the need to replace the link by checking if it is not a reparse point will work. Otherwise, I would have to take into consideration that the reparse point may exist but be invalid.

-James

Re: native symlink

[Corinna Vinschen]

Please don't http://cygwin.com/acronyms/#TOFU

Thank you.


On Apr 26 16:39, James Gregurich wrote:
> I had one of my guys test the work this morning. The creation of
> symlinks works fine. Did you implement a utility that can convert an
> existing cygwin symlink to native form?

This can easily be accomplished by a script, utilizing readlink(1) and
ln(1).


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: native symlink support should fallback to default format if target missing

[Corinna Vinschen]

On May  3 14:53, James Gregurich wrote:
> The guy I have testing the native symlink support in the new cygwin is
> reporting to me that if the target of the link does not exist, the
> mechanism is creating a file reparse point. This is not desirable
> behavior. When the target comes into existence, if it is a folder,
> then the native symlink is invalid.  What the mechanism should do is
> fall back to the native symlink format if the target doesn't exist.
> That way, the link is never invalid. Since it is a default format
> symlink, then my test for the need to replace the link by checking if
> it is not a reparse point will work. Otherwise, I would have to take
> into consideration that the reparse point may exist but be invalid.

Makes sense.  I'll fix that shortly.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: native symlink

[Charles Wilson]

On 5/13/2013 11:00 AM, Corinna Vinschen wrote:
> On Apr 26 16:39, James Gregurich wrote:
>> I had one of my guys test the work this morning. The creation of
>> symlinks works fine. Did you implement a utility that can convert an
>> existing cygwin symlink to native form?
>
> This can easily be accomplished by a script, utilizing readlink(1) and
> ln(1).

Well, if the he wants to convert to a *native* link, then he'd use 
winln(1) from cygutils rather than ln(1). But the larger point remains: 
this conversion can be done from a script.

--
Chuck

Re: native symlink support should fallback to default format if target missing

[Jeffrey Altman]

On 5/13/2013 11:00 AM, Corinna Vinschen wrote:
> On May  3 14:53, James Gregurich wrote:
>> The guy I have testing the native symlink support in the new cygwin is
>> reporting to me that if the target of the link does not exist, the
>> mechanism is creating a file reparse point. This is not desirable
>> behavior. When the target comes into existence, if it is a folder,
>> then the native symlink is invalid.  What the mechanism should do is
>> fall back to the native symlink format if the target doesn't exist.
>> That way, the link is never invalid. Since it is a default format
>> symlink, then my test for the need to replace the link by checking if
>> it is not a reparse point will work. Otherwise, I would have to take
>> into consideration that the reparse point may exist but be invalid.
> 
> Makes sense.  I'll fix that shortly.

Corinna,

Don't worry about falling back for AFS.  The correct thing will happen
there because AFS does not save the target type information as part of
the backend link information.

Thanks.

Jeffrey Altman

Re: native symlink

[Corinna Vinschen]

On May 13 11:12, Charles Wilson wrote:
> On 5/13/2013 11:00 AM, Corinna Vinschen wrote:
> >On Apr 26 16:39, James Gregurich wrote:
> >>I had one of my guys test the work this morning. The creation of
> >>symlinks works fine. Did you implement a utility that can convert an
> >>existing cygwin symlink to native form?
> >
> >This can easily be accomplished by a script, utilizing readlink(1) and
> >ln(1).
> 
> Well, if the he wants to convert to a *native* link, then he'd use
> winln(1) from cygutils rather than ln(1).

Why?  What speaks against

  #!/bin/bash
  tgt=$(readlink "$1")
  rm "$1" && CYGWIN=winsymlinks:native ln -s "${tgt}" "$1"

?

> But the larger point
> remains: this conversion can be done from a script.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: native symlink support should fallback to default format if target missing

[Corinna Vinschen]

On May 13 11:25, Jeffrey Altman wrote:
> On 5/13/2013 11:00 AM, Corinna Vinschen wrote:
> > On May  3 14:53, James Gregurich wrote:
> >> The guy I have testing the native symlink support in the new cygwin is
> >> reporting to me that if the target of the link does not exist, the
> >> mechanism is creating a file reparse point. This is not desirable
> >> behavior. When the target comes into existence, if it is a folder,
> >> then the native symlink is invalid.  What the mechanism should do is
> >> fall back to the native symlink format if the target doesn't exist.
> >> That way, the link is never invalid. Since it is a default format
> >> symlink, then my test for the need to replace the link by checking if
> >> it is not a reparse point will work. Otherwise, I would have to take
> >> into consideration that the reparse point may exist but be invalid.
> > 
> > Makes sense.  I'll fix that shortly.
> 
> Corinna,
> 
> Don't worry about falling back for AFS.  The correct thing will happen
> there because AFS does not save the target type information as part of
> the backend link information.

Thanks for the reminder.  I'll keep that in mind for the patch.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: native symlink support should fallback to default format if target missing

[Christopher Faylor]

On Mon, May 13, 2013 at 05:40:07PM +0200, Corinna Vinschen wrote:
>On May 13 11:25, Jeffrey Altman wrote:
>> On 5/13/2013 11:00 AM, Corinna Vinschen wrote:
>> > On May  3 14:53, James Gregurich wrote:
>> >> The guy I have testing the native symlink support in the new cygwin is
>> >> reporting to me that if the target of the link does not exist, the
>> >> mechanism is creating a file reparse point. This is not desirable
>> >> behavior. When the target comes into existence, if it is a folder,
>> >> then the native symlink is invalid.  What the mechanism should do is
>> >> fall back to the native symlink format if the target doesn't exist.
>> >> That way, the link is never invalid. Since it is a default format
>> >> symlink, then my test for the need to replace the link by checking if
>> >> it is not a reparse point will work. Otherwise, I would have to take
>> >> into consideration that the reparse point may exist but be invalid.
>> > 
>> > Makes sense.  I'll fix that shortly.
>> 
>> Corinna,
>> 
>> Don't worry about falling back for AFS.  The correct thing will happen
>> there because AFS does not save the target type information as part of
>> the backend link information.
>
>Thanks for the reminder.  I'll keep that in mind for the patch.

I've had a private discussion with Corinna about this and she asked me
to make my concerns known.

It seems to me that if you tell Cygwin to create native windows symlinks
then, if it is unable to do so, it should not be falling back to using
its own symlinks.  I would think that would be surprising to someone who
set the CYGWIN environment variable to force that behavior.

If we fall back then a user will create a symlink, assuming that their
native windows app will be able to use it but, will get a strange error
when they attempt to run the program.  With the fallback there will be
no way to test, within cygwin at least, what format the symlink is and
so they won't even be able to verify that the symlink is what they want
it to be.

I don't feel strongly about this but I thought that the fallback behavior
could be confusing to the end user.

cgf

Re: native symlink support should fallback to default format if target missing

[Earnie Boyd]

On Mon, May 13, 2013 at 2:59 PM, Christopher Faylor wrote:
> On Mon, May 13, 2013 at 05:40:07PM +0200, Corinna Vinschen wrote:
>>On May 13 11:25, Jeffrey Altman wrote:
>>> On 5/13/2013 11:00 AM, Corinna Vinschen wrote:
>>> > On May  3 14:53, James Gregurich wrote:
>>> >> The guy I have testing the native symlink support in the new cygwin is
>>> >> reporting to me that if the target of the link does not exist, the
>>> >> mechanism is creating a file reparse point. This is not desirable
>>> >> behavior. When the target comes into existence, if it is a folder,
>>> >> then the native symlink is invalid.  What the mechanism should do is
>>> >> fall back to the native symlink format if the target doesn't exist.
>>> >> That way, the link is never invalid. Since it is a default format
>>> >> symlink, then my test for the need to replace the link by checking if
>>> >> it is not a reparse point will work. Otherwise, I would have to take
>>> >> into consideration that the reparse point may exist but be invalid.
>>> >
>>> > Makes sense.  I'll fix that shortly.
>>>
>>> Corinna,
>>>
>>> Don't worry about falling back for AFS.  The correct thing will happen
>>> there because AFS does not save the target type information as part of
>>> the backend link information.
>>
>>Thanks for the reminder.  I'll keep that in mind for the patch.
>
> I've had a private discussion with Corinna about this and she asked me
> to make my concerns known.
>
> It seems to me that if you tell Cygwin to create native windows symlinks
> then, if it is unable to do so, it should not be falling back to using
> its own symlinks.  I would think that would be surprising to someone who
> set the CYGWIN environment variable to force that behavior.
>
> If we fall back then a user will create a symlink, assuming that their
> native windows app will be able to use it but, will get a strange error
> when they attempt to run the program.  With the fallback there will be
> no way to test, within cygwin at least, what format the symlink is and
> so they won't even be able to verify that the symlink is what they want
> it to be.
>
> I don't feel strongly about this but I thought that the fallback behavior
> could be confusing to the end user.

I agree with Chris on this.  Instead of a fallback behavior it should
be an EACCES, EINVAL or EXDEV error.

-- 
Earnie
-- https://sites.google.com/site/earnieboyd

Re: native symlink

[Charles Wilson]

On 5/13/2013 11:39 AM, Corinna Vinschen wrote:
> On May 13 11:12, Charles Wilson wrote:
>> Well, if the he wants to convert to a *native* link, then he'd use
>> winln(1) from cygutils rather than ln(1).
>
> Why?  What speaks against
>
>    #!/bin/bash
>    tgt=$(readlink "$1")
>    rm "$1" && CYGWIN=winsymlinks:native ln -s "${tgt}" "$1"

Oh, I forgot about the CYGWIN setting; I thought that the cygwin 
"support" for native windows symlinks was just that it could now read 
them and understand what they were (instead of being fooled 
transparently by the underlying filesystem) -- and that Daniel's winln 
program, contributed to cygutils, was the way to create them from user 
space.

Now that I engaged the brain, that really doesn't make much sense, does it?

Right, 'CYGWIN=winsymlinks:native ln -s' would work fine. Sorry for the 
noise.

--
Chuck

Re: native symlink support should fallback to default format if target missing

[James Gregurich]

>> 
> 
> I agree with Chris on this.  Instead of a fallback behavior it should
> be an EACCES, EINVAL or EXDEV error.
> 
> -- 
> Earnie
> -- https://sites.google.com/site/earnieboyd

This may sound rational from a theoretical perspective, but for practical use of the system, it is not good. Consider the case when one clones  a git repository. The link will often be created before the target file exists. If the system fails with an error, then you won't be able to use your git working tree as the checkout won't be successful.

Output a warning if you like, but the call to make the link needs to succeed so that unix software works correctly. 

Outputting a broken link would be more confusing for a user than outputting a default form symlink. An experienced user of windows and Cygwin will typically have the sophistication to know how to recognize the cygwin symlink file in the file system. After all. The system has already been doing this behavior for years and people have seen it. Also, the system is not on by default. People who enable native symlink support can be expected to have the sophistication to understand the details of using it.

-James

Re: native symlink support should fallback to default format if target missing

[Corinna Vinschen]

On May 14 08:52, James Gregurich wrote:
> 
> >> 
> > 
> > I agree with Chris on this.  Instead of a fallback behavior it should
> > be an EACCES, EINVAL or EXDEV error.
> > 
> > -- 
> > Earnie
> > -- https://sites.google.com/site/earnieboyd
> 
> This may sound rational from a theoretical perspective, but for
> practical use of the system, it is not good. Consider the case when
> one clones  a git repository. The link will often be created before
> the target file exists. If the system fails with an error, then you
> won't be able to use your git working tree as the checkout won't be
> successful.

But here's the deal:

If the user *deliberately* added CYGWIN=winsymlinks:native to the
call, then the user *deliberately* wants to generate native symlinks.
The *only* good reason to do that is to support the usage of the
symlinks from native (==non-Cygwin) tools.

Now consider:  If you silently create cygwin symlinks, the symlink
will be unusable for native tools and your use case will be broken.

OTOH, if it doesn't matter if native tools work with that symlink, then
why did you set CYGWIN=winsymlinks:native at all?

So, cgf has a good point here, IMHO.  Either you don't care for
interoperability of the symlink, then don't set
CYGWIN=winsymlinks:native.  Or, if interoperability is important, set
CYGWIN=winsymlinks:native, but then creating a non-native symlink
doesn't make sense.

This is a valid argument which has to be seriously considered.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: native symlink support should fallback to default format if target missing

[Christopher Faylor]

On Tue, May 14, 2013 at 08:52:09AM -0600, James Gregurich wrote:
>
>>> 
>> 
>> I agree with Chris on this.  Instead of a fallback behavior it should
>> be an EACCES, EINVAL or EXDEV error.
>> 
>> -- 
>> Earnie
>> -- https://sites.google.com/site/earnieboyd
>
>This may sound rational from a theoretical perspective, but for
>practical use of the system, it is not good.  Consider the case when
>one clones a git repository.  The link will often be created before the
>target file exists.  If the system fails with an error, then you won't
>be able to use your git working tree as the checkout won't be
>successful.
>
>Output a warning if you like, but the call to make the link needs to
>succeed so that unix software works correctly.

Having Cygwin issue warnings when it can't perform a low-level operation
is not a path we want to take.

>Outputting a broken link would be more confusing for a user than
>outputting a default form symlink.  An experienced user of windows and
>Cygwin will typically have the sophistication to know how to recognize
>the cygwin symlink file in the file system.  After all.  The system has
>already been doing this behavior for years and people have seen it.
>Also, the system is not on by default.  People who enable native
>symlink support can be expected to have the sophistication to
>understand the details of using it.

You really can't argue "sophistication".  That is supposition on your
part.  This kind of argument is easily counterable by my strongly
asserting that Cygwin users are not sophisticated and don't really
understand how things work under the hood.  I could assert that someone
could set the environment variable because they misread the
documentation and then could become confused when it doesn't do what it
apparently says it should: "I told Cygwin to use windows symlinks and it
didn't! WTH!!!!!"

You might disagree with me but each point is valid.

Also, having a git repository which is part cygwin symlinks and part
windows symlinks and which fails strangely when a native windows git is
used doesn't sound like a great idea to me.

If you want to argue that users are sophisticated then why aren't they
sophisticated enough to understand the issue and deal with the
consequences?

Re: native symlink support should fallback to default format if target missing

[Jeffrey Altman]

On 5/14/2013 11:04 AM, Corinna Vinschen wrote:
> But here's the deal:
> 
> If the user *deliberately* added CYGWIN=winsymlinks:native to the
> call, then the user *deliberately* wants to generate native symlinks.
> The *only* good reason to do that is to support the usage of the
> symlinks from native (==non-Cygwin) tools.
> 
> Now consider:  If you silently create cygwin symlinks, the symlink
> will be unusable for native tools and your use case will be broken.
> 
> OTOH, if it doesn't matter if native tools work with that symlink, then
> why did you set CYGWIN=winsymlinks:native at all?
> 
> So, cgf has a good point here, IMHO.  Either you don't care for
> interoperability of the symlink, then don't set
> CYGWIN=winsymlinks:native.  Or, if interoperability is important, set
> CYGWIN=winsymlinks:native, but then creating a non-native symlink
> doesn't make sense.
> 
> This is a valid argument which has to be seriously considered.
> 
> 
> Corinna

I definitely see Christopher's point and I am very sympathetic to it as
someone that has to support a large anonymous user community.  Provide
rope but not enough to permit the user to hang themselves from the
perspective of losing data through a lack of understanding of a how a
feature works.

I suspect the behavior that James really wants is the following:

 1. if the target of a symlink exists and the underlying file system
    supports symlink reparse points, create a symlink reparse point.

 2. if underlying file system doesn't support symlink reparse points,
    generate an error.

 3. if the target of a symlink doesn't exist, create a cygwin
    symlink as a place holder.

 4. if a cygwin symlink is accessed and the target exists and the
    file system supports symlink reparse points, replace the cygwin
    symlink with the symlink reparse point.

I believe the above behavior is a different mode than
"winsymlinks:native".  Perhaps "winsymlinks:native-preferred".

Jeffrey Altman

Re: native symlink support should fallback to default format if target missing

[Corinna Vinschen]

On May 14 11:54, Jeffrey Altman wrote:
> On 5/14/2013 11:04 AM, Corinna Vinschen wrote:
> > But here's the deal:
> > 
> > If the user *deliberately* added CYGWIN=winsymlinks:native to the
> > call, then the user *deliberately* wants to generate native symlinks.
> > The *only* good reason to do that is to support the usage of the
> > symlinks from native (==non-Cygwin) tools.
> > 
> > Now consider:  If you silently create cygwin symlinks, the symlink
> > will be unusable for native tools and your use case will be broken.
> > 
> > OTOH, if it doesn't matter if native tools work with that symlink, then
> > why did you set CYGWIN=winsymlinks:native at all?
> > 
> > So, cgf has a good point here, IMHO.  Either you don't care for
> > interoperability of the symlink, then don't set
> > CYGWIN=winsymlinks:native.  Or, if interoperability is important, set
> > CYGWIN=winsymlinks:native, but then creating a non-native symlink
> > doesn't make sense.
> > 
> > This is a valid argument which has to be seriously considered.
> > 
> > 
> > Corinna
> 
> I definitely see Christopher's point and I am very sympathetic to it as
> someone that has to support a large anonymous user community.  Provide
> rope but not enough to permit the user to hang themselves from the
> perspective of losing data through a lack of understanding of a how a
> feature works.
> 
> I suspect the behavior that James really wants is the following:
> 
>  1. if the target of a symlink exists and the underlying file system
>     supports symlink reparse points, create a symlink reparse point.
> 
>  2. if underlying file system doesn't support symlink reparse points,
>     generate an error.

We're falling back to Cygwin symlinks here.  The check for the FS
capabilities is always done since, for instance, MVFS doesn't support
the SYSTEM bit so only .lnk files work as symlinks, and NFS supports
real symlinks so it gets real symlinks.

>  3. if the target of a symlink doesn't exist, create a cygwin
>     symlink as a place holder.
> 
>  4. if a cygwin symlink is accessed and the target exists and the
>     file system supports symlink reparse points, replace the cygwin
>     symlink with the symlink reparse point.

No, that won't happen.  When reading a symlink it will be used, not
overwritten with another one under some weird circumstances.
Consider:

  $ export CYGWIN=winsymlinks:native
  $ ln -s foo bar
  $ [... doing some other unrelated stuff, forgetting to unset CYGWIN...]
  $ find . -name baz

This will accidentally read all symlinks under CWD and convert them to
native symlinks.  Nope, not good.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: native symlink support should fallback to default format if target missing

[Christopher Faylor]

On Tue, May 14, 2013 at 11:54:50AM -0400, Jeffrey Altman wrote:
>On 5/14/2013 11:04 AM, Corinna Vinschen wrote:
>> But here's the deal:
>> 
>> If the user *deliberately* added CYGWIN=winsymlinks:native to the
>> call, then the user *deliberately* wants to generate native symlinks.
>> The *only* good reason to do that is to support the usage of the
>> symlinks from native (==non-Cygwin) tools.
>> 
>> Now consider:  If you silently create cygwin symlinks, the symlink
>> will be unusable for native tools and your use case will be broken.
>> 
>> OTOH, if it doesn't matter if native tools work with that symlink, then
>> why did you set CYGWIN=winsymlinks:native at all?
>> 
>> So, cgf has a good point here, IMHO.  Either you don't care for
>> interoperability of the symlink, then don't set
>> CYGWIN=winsymlinks:native.  Or, if interoperability is important, set
>> CYGWIN=winsymlinks:native, but then creating a non-native symlink
>> doesn't make sense.
>> 
>> This is a valid argument which has to be seriously considered.
>> 
>> 
>> Corinna
>
>I definitely see Christopher's point and I am very sympathetic to it as
>someone that has to support a large anonymous user community.  Provide
>rope but not enough to permit the user to hang themselves from the
>perspective of losing data through a lack of understanding of a how a
>feature works.
>
>I suspect the behavior that James really wants is the following:
>
> 1. if the target of a symlink exists and the underlying file system
>    supports symlink reparse points, create a symlink reparse point.
>
> 2. if underlying file system doesn't support symlink reparse points,
>    generate an error.
>
> 3. if the target of a symlink doesn't exist, create a cygwin
>    symlink as a place holder.

I see 2. and 3. as basically the same thing.  By the logic advanced
here it seems like, for consistency, 2. should silently create a
cygwin symlink.  I don't think that's a good idea but I do think it
would at least be consistent.

> 4. if a cygwin symlink is accessed and the target exists and the
>    file system supports symlink reparse points, replace the cygwin
>    symlink with the symlink reparse point.

Ugh.  Please no.  Silently modifying the filesystem is rather surprising
behavior.  It gives the user no control.

cgf

Re: native symlink support should fallback to default format if target missing

[James Gregurich]

>> 
>> 
>> 4. if a cygwin symlink is accessed and the target exists and the
>>   file system supports symlink reparse points, replace the cygwin
>>   symlink with the symlink reparse point.
> 
> No, that won't happen.  When reading a symlink it will be used, not
> overwritten with another one under some weird circumstances.
> Consider:
> 
> $ export CYGWIN=winsymlinks:native
> $ ln -s foo bar
> $ [... doing some other unrelated stuff, forgetting to unset CYGWIN...]
> $ find . -name baz
> 
> This will accidentally read all symlinks under CWD and convert them to
> native symlinks.  Nope, not good.

You don't need to do that. Just leave it as a native default form symlink. One can use a command line tool to test the symlink and retry the creation of the native symlink at a later time. That works in practical use.

Re: native symlink support should fallback to default format if target missing

[James Gregurich]

> 
> I definitely see Christopher's point and I am very sympathetic to it as
> someone that has to support a large anonymous user community.  Provide
> rope but not enough to permit the user to hang themselves from the
> perspective of losing data through a lack of understanding of a how a
> feature works.
> 
> I suspect the behavior that James really wants is the following:
> 
> 1. if the target of a symlink exists and the underlying file system
>   supports symlink reparse points, create a symlink reparse point.
> 
> 2. if underlying file system doesn't support symlink reparse points,
>   generate an error.
> 
> 3. if the target of a symlink doesn't exist, create a cygwin
>   symlink as a place holder.
> 
> 4. if a cygwin symlink is accessed and the target exists and the
>   file system supports symlink reparse points, replace the cygwin
>   symlink with the symlink reparse point.
> 
> I believe the above behavior is a different mode than
> "winsymlinks:native".  Perhaps "winsymlinks:native-preferred".
> 
> Jeffrey Altman

Works for me.  The key is that the system must work as unix software expects in terms of the creation of symlinks. You don't want the filesystem in an invalid state that cannot be recovered from or is much harder to recover from. If you have that, you can post-process to update default format symlinks to native symlinks once the targets are known.

Re: native symlink support should fallback to default format if target missing

[James Gregurich]

> Ugh.  Please no.  Silently modifying the filesystem is rather surprising
> behavior.  It gives the user no control.
> 
> cgf

I'll repeat for emphasis... You don't need to do that. Just set the default form symlink and reply on a post-processing tool to update the symlink once the conditions are met.

Re: native symlink support should fallback to default format if target missing

[Christopher Faylor]

On Tue, May 14, 2013 at 03:11:14PM -0600, James Gregurich wrote:
>
>> Ugh.  Please no.  Silently modifying the filesystem is rather surprising
>> behavior.  It gives the user no control.
>
>I'll repeat for emphasis...  You don't need to do that.  Just set the
>default form symlink and reply on a post-processing tool to update the
>symlink once the conditions are met.

Repetition is unnecessary.  Your points are simple and don't bear repeating.

Re: native symlink support should fallback to default format if target missing

[James Gregurich]

Sent from my iPad

On May 14, 2013, at 8:28 PM, Christopher Faylor  wrote:

> On Tue, May 14, 2013 at 03:11:14PM -0600, James Gregurich wrote:
>> 
>>> Ugh.  Please no.  Silently modifying the filesystem is rather surprising
>>> behavior.  It gives the user no control.
>> 
>> I'll repeat for emphasis...  You don't need to do that.  Just set the
>> default form symlink and reply on a post-processing tool to update the
>> symlink once the conditions are met.
> 
> Repetition is unnecessary.  Your points are simple and don't bear repeating.

I hope so. I'm awfully tired of repeating them.