Bash silently truncates the Command Line when called programatically via CreateProcess as MAXPATHLEN was reduced to 8192 from 16384

Bash silently truncates the Command Line when called programatically via CreateProcess as MAXPATHLEN was reduced to 8192 from 16384

[Abhijit Bhattacharjee]

On a typical configuration, we use gmake on windows as the build
system. GMAKE calls the Cygwin bash in order to invoke the MSVS
compiler in-order to satisfy the dependency if a rule to call the
compiler is present.
Now, for certain modules, the command line was seemingly getting
truncated. On investigating further it revealed the reason for the
truncation was because in the glob.cc, MAXPATHLEN was now defined as
8192 so, when bash was compiled with the changed crt code, the problem
started to emerge.

Now, it also seems that there was a discussion
(https://cygwin.com/ml/cygwin-patches/2012-q1/msg00037.html) where
Corinna mentioned about decreasing the size to 4096 and later
increasing it to 8192. Unfortunately, for our case, 8192 is low a
limit and ideally the previous limit of 16384 was working perfectly.

The Argument that Corinna placed was because the default stack size is
2 Megs  and sizeof(Char)==8, a local array of 8* 16384  would be too
strenuous for the stack.

It was still not clear to me as to why?

1. The Temporary buffer was not allocated on heap?
2. Why the entire argument (both quoted and non-quoted string) was
passed to the glob module. If instead, the argument string could had
been tokeinized  as interleaving quoted and non-quoted string, the
restriction imposed on the argument length would be alleviated. What I
am envisaging is, currently in the globify module, you are escaping
all the quoted characters to demarcate from the non-quoted characters
before calling glob once, so that the glob can only expand non-quoted
sub-string. Instead, if we could call glob multiple times whenever we
encounter a sub-string of non-quoted string, wouldn't it be cleaner?


In any case, as of now, for the time being I have increased the size
of MAXPATHLEN back to 16384  and recompiled the cygwin followed by
bash but am looking forward for a solution from your end, as this is
dirty, difficult to maintain, and likely to break in the future as our
cygwin build is different from what we have in the repository.



Rgrds,
Abhijit

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Bash silently truncates the Command Line when called programatically via CreateProcess as MAXPATHLEN was reduced to 8192 from 16384

[Christopher Faylor]

On Mon, May 26, 2014 at 08:59:56PM +0530, Abhijit Bhattacharjee wrote:
>On a typical configuration, we use gmake on windows as the build
>system. GMAKE calls the Cygwin bash in order to invoke the MSVS
>compiler in-order to satisfy the dependency if a rule to call the
>compiler is present.

If you're using Cygwin's make it will not truncate paths if it is
running cygwin programs.  Not sure what MSVS is.

>Now, for certain modules, the command line was seemingly getting
>truncated. On investigating further it revealed the reason for the
>truncation was because in the glob.cc, MAXPATHLEN was now defined as
>8192 so, when bash was compiled with the changed crt code, the problem
>started to emerge.
>
>Now, it also seems that there was a discussion
>(https://cygwin.com/ml/cygwin-patches/2012-q1/msg00037.html) where
>Corinna mentioned about decreasing the size to 4096 and later
>increasing it to 8192. Unfortunately, for our case, 8192 is low a
>limit and ideally the previous limit of 16384 was working perfectly.
>
>The Argument that Corinna placed was because the default stack size is
>2 Megs  and sizeof(Char)==8, a local array of 8* 16384  would be too
>strenuous for the stack.
>
>It was still not clear to me as to why?
>
>1. The Temporary buffer was not allocated on heap?

The source code comes from FreeBSD.   Managing this on the heap would
mean a rewrite and would have its own problems.  Since this isn't really
an issue when Cygwin calls Cygwin programs it likely isn't really high
on anyone's priority list.

>2. Why the entire argument (both quoted and non-quoted string) was
>passed to the glob module. If instead, the argument string could had
>been tokeinized  as interleaving quoted and non-quoted string, the
>restriction imposed on the argument length would be alleviated. What I
>am envisaging is, currently in the globify module, you are escaping
>all the quoted characters to demarcate from the non-quoted characters
>before calling glob once, so that the glob can only expand non-quoted
>sub-string. Instead, if we could call glob multiple times whenever we
>encounter a sub-string of non-quoted string, wouldn't it be cleaner?

I don't really know what you're talking about but patches are always
better than word descriptions.

It sounds like you're using a non-Cygwin make to call a Cygwin process.
If that is the case then it seems like the trivial fix is "Don't Do
That" (tm)

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Bash silently truncates the Command Line when called programatically via CreateProcess as MAXPATHLEN was reduced to 8192 from 16384

[René Berber]

On 5/26/2014 3:08 PM, Christopher Faylor wrote:

> If you're using Cygwin's make it will not truncate paths if it is
> running cygwin programs.  Not sure what MSVS is.

Microsoft Visual Studio (or Something)?
-- 
René Berber


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Bash silently truncates the Command Line when called programatically via CreateProcess as MAXPATHLEN was reduced to 8192 from 16384

[Christopher Faylor]

On Mon, May 26, 2014 at 03:38:13PM -0500, René Berber wrote:
>On 5/26/2014 3:08 PM, Christopher Faylor wrote:
>
>> If you're using Cygwin's make it will not truncate paths if it is
>> running cygwin programs.  Not sure what MSVS is.
>
>Microsoft Visual Studio (or Something)?

Probably.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Bash silently truncates the Command Line when called programatically via CreateProcess as MAXPATHLEN was reduced to 8192 from 16384

[Abhijit Bhattacharjee]

Yes, MSVS is Microsoft Visual Studio.

and I am using the make for Windows
(http://gnuwin32.sourceforge.net/packages/make.htm).

Please note, when I am saying the path is getting truncated, its the
bash that is truncating it. Also, its immaterial how and what is
getting called. With the latest Cygwin, any program that is compiled
and linked with the CRT would exhibit this behavior. The truncation
issue in the CRT is best demonstrated by the following Code couplets

Caller.cpp
--------------
#include <iostream>
#include <windows.h>
#include <string.h>
#define _countof(_Array) (sizeof(_Array) / sizeof(_Array[0]))
int main(int argc, char *argv[]) {
    if (argc <= 1) return -1;
    char arg[10000] = {0};
    strncpy(arg, argv[argc-1], _countof(arg));
    strncat(arg," ", _countof(arg));
    memset(arg + strlen(arg), 'X', _countof(arg) - strlen(arg) - 2);
    strncat(arg, "\"", _countof(arg));
    STARTUPINFO startInfo;
    PROCESS_INFORMATION procInfo;
    memset( &startInfo, 0, sizeof(startInfo) );
    memset( &procInfo, 0, sizeof(procInfo) );
    if (CreateProcess(NULL,
        arg,
        NULL,
        0,
        TRUE,
        0,
        NULL,
        NULL,
        &startInfo,
        &procInfo) == FALSE) {
            std::cout<<"Error Code "<<GetLastError()<<std::endl;
    } else {
        WaitForSingleObject(procInfo.hProcess, INFINITE );
        CloseHandle( procInfo.hProcess );
        CloseHandle( procInfo.hThread );
        std::cout<<"Send Length "<<strlen(arg)<<std::endl;
    }
    return 0;
};

Callee.cpp
-------------

#include <stdio.h>
#include <assert.h>
#include <string.h>
#include <windows.h>
int main (int argc, char **argv) {
    printf("%s\n",argv[argc-1]);
    printf("%d\n",argc);
    printf("Received Length %d\n",strlen(argv[argc-1]));
    return 0;

}


Compile and Run The Programs as
-----------------------------------------

$g++ -g -o Caller.exe Caller.cpp
$g++ -g -o Callee.exe Callee.cpp
$./Caller.exe Callee.exe




Also Note, if you would like to review the code that is responsible
for the truncation, I would suggest you to refer the function

Source: winsup\cygwin\glob.cc
int glob(const char *__restrict pattern, int flags, int
(*errfunc)(const char *, int), glob_t *__restrict pglob)

and notice pathbuf is defined as

Char *bufnext, *bufend, patbuf[MAXPATHLEN], prot;

where

#undef MAXPATHLEN
#define MAXPATHLEN 8192

and to complete here is the call stack

glob(const char *__restrict pattern, int flags, int (*errfunc)(const
char *, int), glob_t *__restrict pglob)
globify (char *word, char **&argv, int &argc, int &argvlen)  - Line
265 of winsup\cygwin\dcrt0.cc
build_argv (char *cmd, char **&argv, int &argc, int winshell) - Line
353 of winsup\cygwin\dcrt0.cc
dll_crt0_1 (void *) - Line 953 of winsup\cygwin\dcrt0.cc
_dll_crt0 () - Line 1098 of winsup\cygwin\dcrt0.cc
dll_crt0 - Line 1110 of winsup\cygwin\dcrt0.cc

Rgrds,
Abhijit

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Bash silently truncates the Command Line when called programatically via CreateProcess as MAXPATHLEN was reduced to 8192 from 16384

[Christopher Faylor]

On Tue, May 27, 2014 at 06:49:43PM +0530, Abhijit Bhattacharjee wrote:
>Yes, MSVS is Microsoft Visual Studio.
>
>and I am using the make for Windows
>(http://gnuwin32.sourceforge.net/packages/make.htm).
>
>Please note, when I am saying the path is getting truncated, its the
>bash that is truncating it. Also, its immaterial how and what is
>getting called. With the latest Cygwin, any program that is compiled
>and linked with the CRT would exhibit this behavior. The truncation
>issue in the CRT is best demonstrated by the following Code couplets

A Cygwin program which calls another Cygwin program does not have this
problem so it isn't completely immaterial.

>Also Note, if you would like to review the code that is responsible
>for the truncation, I would suggest you to refer the function

Not sure who you are talking to but, on the off chance that you are
hoping to engage a Cygwin developer, we do know where the glob code
lives.  As I indicated, it originally comes from FreeBSD.

Quoting myself:

On Mon, May 26, 2014 at 04:08:57PM -0400, Christopher Faylor wrote:
>Since this isn't really an issue when Cygwin calls Cygwin programs it
>likely isn't really high on anyone's priority list.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Bash silently truncates the Command Line when called programatically via CreateProcess as MAXPATHLEN was reduced to 8192 from 16384

[David Stacey]

On 27/05/14 14:19, Abhijit Bhattacharjee wrote:
> Yes, MSVS is Microsoft Visual Studio.
>
> and I am using the make for Windows

Using the Windows version of gmake to invoke Cygwin bash to call the 
Microsoft compiler seems a little complicated to me. Instead of gmake 
and bash, are you able to use 'nmake' (that comes with MSVC)?

Obviously, that is a non-Cygwin suggestion, and this is a Cygwin mailing 
list, so we wouldn't be able to help you with that here. However, by 
simplifying your call chain, the problems that you're experiencing might 
go away.

Dave.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Bash silently truncates the Command Line when called programatically via CreateProcess as MAXPATHLEN was reduced to 8192 from 16384

[Warren Young]

On 5/28/2014 02:55, David Stacey wrote:
> are you able to use 'nmake' (that comes with MSVC)?

nmake is awfully primitive in itself compared to GNU make, and it is 
doubly crippled by being restricted to cmd.exe syntax for shell commands 
in the targets.

Instead, Abhijit, I suggest that you switch to CMake or SCons, either of 
which will let you build software in a platform-independent way.

And to keep a tenuous hold on topicality, both are packaged for Cygwin.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Bash silently truncates the Command Line when called programatically via CreateProcess as MAXPATHLEN was reduced to 8192 from 16384

[Abhijit Bhattacharjee]

Thank you everyone for your time and reply. I believe from your reply
I am able to get the problem sorted out. I would also try to reply to
each of your queries

@Chris J. Breisch
@Warren Young
Qs/Stmt: Why are you using that make and not the Cygwin make?
Qs/Stmt:  I suggest that you switch to CMake or SCons, either of which
will let you build software in a platform-independent way.
Ans: We had some hickups in the initial days while running builds on
multiple threads and the alternative worked for us. As it had been
working for ages we would rather want to stick with it. Its just that
with the recent Cygwin adoption, things started to fail.

@Christopher Faylor
Qs/Stmt: A Cygwin program which calls another Cygwin program does not
have this problem so it isn't completely immaterial.
Ans: I am not sure what you mean by Cygwin program. If you mean those
that are in your repository, I cannot comment. Instead, if you mean,
that's linked with cygwin1.dll, I believe, I have already shared with
you sample programs that exhibits the truncation behavior.

Qs/Stmt: Not sure who you are talking to but, on the off chance that
you are hoping to engage a Cygwin developer, we do know where the glob
code lives.  As I indicated, it originally comes from FreeBSD.
Ans: I believe not everyone out here are Cygwin developers. Even so,
with my experience, I know, even in my product, I do not know where
all the code comes from. If someone has already done the search, it is
really helpful.

Qs/Stmt: Possibly setting the CYGWIN environment variable to "noglob"
might cause things to work as desired.
Ans: I am yet to test this, but I trust your answer. I am yet to
figure out as to how I can set the environment variables CYGWIN with
multiple values i.e. I need to set it with nodosfilewarning and
noglob. I was trying to read though your code environ.cc and seems to
me I can simply separate it with a delimiter. Your documentation is
silent about it. If you know it off hand, please let me know, that
will reduce some effort for me to read and debug though your code :-)

@David Stacey
Qs/Stmt:  Instead of gmake and bash, are you able to use 'nmake' (that
comes with MSVC)?
Ans: nmake is awful and not even portable.




Rgrds,
Abhijit



On Wed, May 28, 2014 at 11:29 PM, Warren Young <warren@etr-usa.com> wrote:
> On 5/28/2014 02:55, David Stacey wrote:
>>
>> are you able to use 'nmake' (that comes with MSVC)?
>
>
> nmake is awfully primitive in itself compared to GNU make, and it is doubly
> crippled by being restricted to cmd.exe syntax for shell commands in the
> targets.
>
> Instead, Abhijit, I suggest that you switch to CMake or SCons, either of
> which will let you build software in a platform-independent way.
>
> And to keep a tenuous hold on topicality, both are packaged for Cygwin.
>
>
> --
> Problem reports:       http://cygwin.com/problems.html
> FAQ:                   http://cygwin.com/faq/
> Documentation:         http://cygwin.com/docs.html
> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Bash silently truncates the Command Line when called programatically via CreateProcess as MAXPATHLEN was reduced to 8192 from 16384

[Cliff Hones]

On 31/05/2014 14:28, Abhijit Bhattacharjee wrote:
> ...
> Qs/Stmt: Possibly setting the CYGWIN environment variable to "noglob"
> might cause things to work as desired.
> Ans: I am yet to test this, but I trust your answer. I am yet to
> figure out as to how I can set the environment variables CYGWIN with
> multiple values i.e. I need to set it with nodosfilewarning and
> noglob. I was trying to read though your code environ.cc and seems to
> me I can simply separate it with a delimiter. Your documentation is
> silent about it. If you know it off hand, please let me know, that
> will reduce some effort for me to read and debug though your code :-)

Hmm - you didn't look very hard.  Why not read the Cygwin User manual - in
particular this section:

  https://cygwin.com/cygwin-ug-net/using-cygwinenv.html

where it clearly says the items are space-separated.

-- Cliff


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Bash silently truncates the Command Line when called programatically via CreateProcess as MAXPATHLEN was reduced to 8192 from 16384

[Christopher Faylor]

On Sat, May 31, 2014 at 06:58:59PM +0530, Abhijit Bhattacharjee wrote:
>Thank you everyone for your time and reply. I believe from your reply
>I am able to get the problem sorted out. I would also try to reply to
>each of your queries
>
>@Christopher Faylor
>Qs/Stmt: A Cygwin program which calls another Cygwin program does not
>have this problem so it isn't completely immaterial.
>Ans: I am not sure what you mean by Cygwin program. If you mean those
>that are in your repository, I cannot comment. Instead, if you mean,
>that's linked with cygwin1.dll, I believe, I have already shared with
>you sample programs that exhibits the truncation behavior.

I should have been clearer.  What I mean by Cygwin programs are programs
linked with the Cygwin DLL, using only Linux/POSIX functions.  If
program A exec's program B using one of Cygwin's exec/spawn functions it
will not suffer from command line truncation.  But, if you choose to use
Windows APIs then you are on your own.  There is no guarantee that
Cygwin functionality like process handling, signals, pipes, etc.  will
work reliably.

>Qs/Stmt: Possibly setting the CYGWIN environment variable to "noglob"
>might cause things to work as desired.
>Ans: I am yet to test this, but I trust your answer. I am yet to
>figure out as to how I can set the environment variables CYGWIN with
>multiple values i.e. I need to set it with nodosfilewarning and
>noglob. I was trying to read though your code environ.cc and seems to
>me I can simply separate it with a delimiter. Your documentation is
>silent about it. If you know it off hand, please let me know, that
>will reduce some effort for me to read and debug though your code :-)

The documentation is most definitely not silent.

https://cygwin.com/cygwin-ug-net/using-cygwinenv.html

makes it very clear in the second sentence of the above section.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Bash silently truncates the Command Line when called programatically via CreateProcess as MAXPATHLEN was reduced to 8192 from 16384

[Christopher Faylor]

On Wed, May 28, 2014 at 05:41:19PM +0200, Houder wrote:
>As far as I can tell, it is not bash, but the Cygwin supervisor, that
>does the truncation.

The code in Cygwin has already been repeatedly identified.

Possibly setting the CYGWIN environment variable to "noglob" might
cause things to work as desired.

See: https://cygwin.com/cygwin-ug-net/using-cygwinenv.html

Also remember that this environment variable has to be set before any
Cygwin program is run, i.e., from the CMD prompt or by setting the
environment variable globally via, e.g.,

Control Panel->System->Advanced system settings->Environment Variables

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Bash silently truncates the Command Line when called programatically via CreateProcess as MAXPATHLEN was reduced to 8192 from 16384

[Houder]

Hi Abhijit,

> Please note, when I am saying the path is getting truncated, its the
> bash that is truncating it. Also, its immaterial how and what is
> getting called. With the latest Cygwin, any program that is compiled
> and linked with the CRT would exhibit this behavior. The truncation
> issue in the CRT is best demonstrated by the following Code couplets
>
> Caller.cpp
> --------------
> [snip]
>
> Callee.cpp
> -------------
> [snip]

Using your code couplets (and modifying them a bit), I did some experimentation, and observed:

 - the first parameter to callee, if quoted, is truncated to 8186 bytes in case the length of
   the argument without quotes, exceeds the aforementioned length.
 - no truncation occurs in case the parameter is not quoted.

As far as I can tell, it is not bash, but the Cygwin supervisor, that does the truncation.

Henri


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple