Odd behaviour of __cygwin_environ?

Odd behaviour of __cygwin_environ?

[Dave Korn]

[  Cygwin-1.5.23-2, in case it matters; I haven't yet checked if anything has
changed in this area of the code in 1.5.24/1.5.25/cvs head, but it's a fairly
long-standing area of the code so probably relatively stable.  ]

  I'm trying to track down a bug where a child process doesn't inherit all the
environment variables set in its parent, so I figured I'd add a bunch of
strace debugging.  I wrote this routine to dump an envp[]-style pointer-array
environment block:

void dump_env (const char * const *envp, const char *banner)
{
#if 01
  myfault efault;
  if (efault.faulted ())
  {
    dk_printf ("oops, fault - stop dumping!");
    return;
  }

  int count = 0;
  dk_printf("%sdump environment: %p", banner, envp);
  dk_printf ("attempt");
  dk_printf ("try to de-ref envp %p", envp);
  dk_printf ("de-ref envp %p", *envp);
  while (*envp)
  {
    dk_printf ("try to de-ref envp %p", envp);
    dk_printf ("de-ref envp %p", *envp);
    dk_printf ("#%p#%p#%s", envp, *envp, *envp);
    ++envp;
    ++count;
  }
  dk_printf ("total: %d entries", count);
#endif
}

  ... where dk_printf() is just like system_printf() only using a spare strace
flag mask bit, and I call it from cur_environ() like so:

extern "C" char ** __stdcall
cur_environ ()
{
  if (*main_environ != __cygwin_environ)
    {
    dk_printf ("switch main environ %p for cygwin environ %p", *main_environ,
__cygwin_environ);
  dump_env (*main_environ , "main environ: ");
  dump_env (__cygwin_environ, "cygwin environ: ");
      __cygwin_environ = *main_environ;
      update_envptrs ();
    }

  return __cygwin_environ;
}

  The declaration of __cygwin_environ in environ.h says:

extern char **__cygwin_environ, ***main_environ;

and most of the time this works fine; I see output such as:


  110  676992 [main] bash 4884 dump_env: cygwin environ: dump environment:
0x6D0090
  111  677103 [main] bash 4884 dump_env: attempt
  108  677211 [main] bash 4884 dump_env: try to de-ref envp 0x6D0090
-7469933 -6792722 [main] bash 4884 dump_env: de-ref envp 0x6D0238
  172 -6792550 [main] bash 4884 dump_env: try to de-ref envp 0x6D0090
7484835  692285 [main] bash 4884 dump_env: de-ref envp 0x6D0238
  123  692408 [main] bash 4884 dump_env: #0x6D0090#0x6D0238#!C:=C:\cygwin\bin
  111  692519 [main] bash 4884 dump_env: try to de-ref envp 0x6D0094
  110  692629 [main] bash 4884 dump_env: de-ref envp 0x6D0250
-7469734 -6777105 [main] bash 4884 dump_env:
#0x6D0094#0x6D0250#!EXITCODE=00000000
7484815  707710 [main] bash 4884 dump_env: try to de-ref envp 0x6D0098
  114  707824 [main] bash 4884 dump_env: de-ref envp 0x6D0268
  113  707937 [main] bash 4884 dump_env:
#0x6D0098#0x6D0268#ALLUSERSPROFILE=C:\Documents and Settings\All Users
  167  708104 [main] bash 4884 dump_env: try to de-ref envp 0x6D009C
  113  708217 [main] bash 4884 dump_env: de-ref envp 0x6D02A0
  110  708327 [main] bash 4884 dump_env:
#0x6D009C#0x6D02A0#APPDATA=C:\Documents and Settings\dk\Application Data
14904  723231 [main] bash 4884 dump_env: try to de-ref envp 0x6D00A0
  115  723346 [main] bash 4884 dump_env: de-ref envp 0x6D02E0

    [ ... lots of snippage here ... ]

-7470630 -6448921 [main] bash 4884 dump_env: try to de-ref envp 0x6D0140
7484822 1035901 [main] bash 4884 dump_env: de-ref envp 0x6D1190
  121 1036022 [main] bash 4884 dump_env: #0x6D0140#0x6D1190#WINDIR=C:\WINDOWS
  122 1036144 [main] bash 4884 dump_env: try to de-ref envp 0x6D0144
  116 1036260 [main] bash 4884 dump_env: de-ref envp 0x6D11A8
  118 1036378 [main] bash 4884 dump_env:
#0x6D0144#0x6D11A8#_NT_SYMBOL_PATH=SRV*C:\symbols*http://msdl.microsoft.com/do
wnload/symbols
  183 1036561 [main] bash 4884 dump_env: try to de-ref envp 0x6D0148
-7469846 -6433285 [main] bash 4884 dump_env: de-ref envp 0x6D11F8
7484822 1051537 [main] bash 4884 dump_env:
#0x6D0148#0x6D11F8#__COMPAT_LAYER=EnableNXShowUI 
  182 1051719 [main] bash 4884 dump_env: try to de-ref envp 0x6D014C
  118 1051837 [main] bash 4884 dump_env: de-ref envp 0x6D1220
  118 1051955 [main] bash 4884 dump_env: #0x6D014C#0x6D1220#TERM=cygwin
  119 1052074 [main] bash 4884 dump_env: try to de-ref envp 0x6D0150
-7469720 -6417646 [main] bash 4884 dump_env: de-ref envp 0x6D0160
7484832 1067186 [main] bash 4884 dump_env: #0x6D0150#0x6D0160#HOME=/home/dk
  132 1067318 [main] bash 4884 dump_env: total: 49 entries

  But sometimes it seems as if __cygwin_environ is in fact pointing to a
win32-style environment block made of contiguous concatenated strings with a
double-NUL terminator, leading to exceptions:

  144 2468502 [main] bash 4644 dump_env: cygwin environ: dump environment:
0x6DFC88
  150 2468652 [main] bash 4644 dump_env: attempt
  150 2468802 [main] bash 4644 dump_env: try to de-ref envp 0x6DFC88
  151 2468953 [main] bash 4644 dump_env: de-ref envp 0x48544150
  126 2469079 [main] bash 4644 dump_env: try to de-ref envp 0x6DFC88
  141 2469220 [main] bash 4644 dump_env: de-ref envp 0x48544150
--- Process 4644, exception C0000005 at 610A5EE2
-7484282 -5015062 [main] bash 4644 dump_env: oops, fault - stop dumping!

  In that example, 0x48544150 is the ascii for "PATH".  At other times, it
seems to be partly there and partly not:


  106 5778781 [main] bash 4664 dump_env: cygwin environ: dump environment:
0x6E1BF8
  107 5778888 [main] bash 4664 dump_env: attempt
  105 5778993 [main] bash 4664 dump_env: try to de-ref envp 0x6E1BF8
  105 5779098 [main] bash 4664 dump_env: de-ref envp 0x6E1C08
  107 5779205 [main] bash 4664 dump_env: try to de-ref envp 0x6E1BF8
  107 5779312 [main] bash 4664 dump_env: de-ref envp 0x6E1C08
  105 5779417 [main] bash 4664 dump_env: #0x6E1BF8#0x6E1C08#fort
  107 5779524 [main] bash 4664 dump_env: try to de-ref envp 0x6E1BFC
  107 5779631 [main] bash 4664 dump_env: de-ref envp 0x6E5320
  106 5779737 [main] bash 4664 dump_env: #0x6E1BFC#0x6E5320#for x in monitor
chiptest factorytest ; do ( echo Building "${x} ..."; time remake
ARTIMI_CFLAGS="--save-temps -fverbose-asm -O2" ASIC_BUILD=1 USE_GC=1
ARTIMI_SYSTEM=${x} 2>&1 all ) ; done | tee buildb2b.log
  259 5779996 [main] bash 4664 dump_env: try to de-ref envp 0x6E1C00
  107 5780103 [main] bash 4664 dump_env: de-ref envp 0x20017800
--- Process 4664, exception C0000005 at 610A5EE2
  295 5780398 [main] bash 4664 dump_env: oops, fault - stop dumping!

  Don't know exactly /what/ kind of error that is; the two strings it manages
to print out are the matching variable name and definition parts of a
"VAR=DEFN" for a bash alias, the first two entries look like valid pointers,
but then the third entry looks like it could be ascii again (modulo the
dubious 0x01 byte).  I'm suspicious of the fact that it goes wrong exactly at
the moment when the envp pointer crosses a page boundary, but that could just
be a meaningless coincidence.



  Am I giving myself a race here?  I thought it would be OK to dump
__cygwin_environ from inside cur_environ because we're in the "if
(*main_environ != __cygwin_environ)" clause, which IIUIC is switching over
from the earlystartup minimal win32 environment to the full posix environment,
so __cygwin_environ ought to be valid and initialised here.  But perhaps I
have a wrong assumption: for instance, does the __cygwin_environ get switched
in before it is complete because the startup code knows it isn't going to be
accessed for a while?  Does __cygwin_environ deliberately sometimes point to
one kind of data rather than another?  My understanding of the cygwin
process-startup dance is a bit sketchy here, can anyone offer a bit of
insight?


    cheers,
      DaveK
-- 
Can't think of a witty .sigline today....


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Odd behaviour of __cygwin_environ?

[Corinna Vinschen]

On Dec  4 18:17, Dave Korn wrote:
>   I'm trying to track down a bug where a child process doesn't inherit all the
> environment variables set in its parent, so I figured I'd add a bunch of
> strace debugging.  I wrote this routine to dump an envp[]-style pointer-array
> environment block:
> [...]
>   Am I giving myself a race here?  I thought it would be OK to dump
> __cygwin_environ from inside cur_environ because we're in the "if
> (*main_environ != __cygwin_environ)" clause, which IIUIC is switching over
> from the earlystartup minimal win32 environment to the full posix environment,

Did you read the comment right in front of the cur_environ() function?
This happens every time the application code has changed the
environment, not using the setenv/putenv functions, but rather by
replacing the global environ pointer with it's self-built environment
block.  Usually this also means that the former environment block has
been free'd by the application.  This means that at this point you see
__cygwin_environ with some non-0 probability referencing garbage.

If you want to see the actual environment block, print __cygwin_environ
after the if-clause.  If you encounter an incomplete environment block
at this point, it might also be of some interest to see from where
cur_environ() has been called.

Do you have a very big environment?  Keep in mind that the size of the
native environment block is restricted to 32K.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

RE: Odd behaviour of __cygwin_environ?

[Dave Korn]

On 05 December 2007 12:31, Corinna Vinschen wrote:

> On Dec  4 18:17, Dave Korn wrote:
>>   I'm trying to track down a bug where a child process doesn't inherit all
>> the environment variables set in its parent, so I figured I'd add a bunch
>> of strace debugging.  I wrote this routine to dump an envp[]-style
>> pointer-array environment block: [...]
>>   Am I giving myself a race here?  I thought it would be OK to dump
>> __cygwin_environ from inside cur_environ because we're in the "if
>> (*main_environ != __cygwin_environ)" clause, which IIUIC is switching over
>> from the earlystartup minimal win32 environment to the full posix
>> environment, 
> 
> Did you read the comment right in front of the cur_environ() function?

  Yep, but I didn't get the implication!

> This happens every time the application code has changed the
> environment, not using the setenv/putenv functions, but rather by
> replacing the global environ pointer with it's self-built environment
> block.  Usually this also means that the former environment block has
> been free'd by the application.  This means that at this point you see
> __cygwin_environ with some non-0 probability referencing garbage.

  Ah, that's reassuring.

> If you want to see the actual environment block, print __cygwin_environ
> after the if-clause.  If you encounter an incomplete environment block
> at this point, it might also be of some interest to see from where
> cur_environ() has been called.
> 
> Do you have a very big environment? 

  Yep, huge!

> Keep in mind that the size of the
> native environment block is restricted to 32K.

  I thought when cygexec was in effect the posix environment and commandline
were passed around-behind-the-back-of CreateProcess, bypassing that limit?  Or
does it only handle the commandline but not environment?

    cheers,
      DaveK
-- 
Can't think of a witty .sigline today....


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Odd behaviour of __cygwin_environ?

[Corinna Vinschen]

On Dec  5 15:00, Dave Korn wrote:
> On 05 December 2007 12:31, Corinna Vinschen wrote:
> > Keep in mind that the size of the
> > native environment block is restricted to 32K.
> 
>   I thought when cygexec was in effect the posix environment and commandline
> were passed around-behind-the-back-of CreateProcess, bypassing that limit?  Or
> does it only handle the commandline but not environment?

It also handles the environment block.  Is there a native process in the
call chain, maybe?  If not, I don't see any other chance than to debug
this further.  spawn_guts() and the build_env() functions might be good
starting points for that.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

RE: Odd behaviour of __cygwin_environ?

[Dave Korn]

On 05 December 2007 15:32, Corinna Vinschen wrote:

> On Dec  5 15:00, Dave Korn wrote:
>> On 05 December 2007 12:31, Corinna Vinschen wrote:
>>> Keep in mind that the size of the
>>> native environment block is restricted to 32K.
>> 
>>   I thought when cygexec was in effect the posix environment and
>> commandline were passed around-behind-the-back-of CreateProcess, bypassing
>> that limit?  Or does it only handle the commandline but not environment?
> 
> It also handles the environment block.  Is there a native process in the
> call chain, maybe? 

  Nope, it's all cygwin through and through.  Big project using
make/gcc/binutils, nothing non-posix.

> If not, I don't see any other chance than to debug
> this further.  spawn_guts() and the build_env() functions might be good
> starting points for that.

  Thanks, that's what I'm trying right now.

  Disappointingly it disappears under strace.  That might have been down to
the strace-is-a-win32-process-launching-a-cygwin-process anomaly, which is why
I'm hacking about the dll.  Early results suggest that, even though I'm not
launching under strace, but instead enabling it (by calling strace.hello()
during initial_env() if I detect an environment variable in the win32
environment), the problem still disappears, which makes me suspicious of it
still being a race somehow.

    cheers,
      DaveK
-- 
Can't think of a witty .sigline today....


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Odd behaviour of __cygwin_environ?

[Christopher Faylor]

On Wed, Dec 05, 2007 at 03:50:31PM -0000, Dave Korn wrote:
>On 05 December 2007 15:32, Corinna Vinschen wrote:
>
>> On Dec  5 15:00, Dave Korn wrote:
>>> On 05 December 2007 12:31, Corinna Vinschen wrote:
>>>> Keep in mind that the size of the
>>>> native environment block is restricted to 32K.
>>> 
>>>   I thought when cygexec was in effect the posix environment and
>>> commandline were passed around-behind-the-back-of CreateProcess, bypassing
>>> that limit?  Or does it only handle the commandline but not environment?
>> 
>> It also handles the environment block.  Is there a native process in the
>> call chain, maybe? 
>
>  Nope, it's all cygwin through and through.  Big project using
>make/gcc/binutils, nothing non-posix.
>
>> If not, I don't see any other chance than to debug
>> this further.  spawn_guts() and the build_env() functions might be good
>> starting points for that.
>
>  Thanks, that's what I'm trying right now.
>
>  Disappointingly it disappears under strace.  That might have been down to
>the strace-is-a-win32-process-launching-a-cygwin-process anomaly, which is why
>I'm hacking about the dll.  Early results suggest that, even though I'm not
>launching under strace, but instead enabling it (by calling strace.hello()
>during initial_env() if I detect an environment variable in the win32
>environment), the problem still disappears, which makes me suspicious of it
>still being a race somehow.

I have a vague feeling that I'll regret mentioning this but, if you've
built the DLL with --enable-debugging, you can set the CYGWIN_SLEEP
environment variable, to some millisecond value that you want the cygwin
process to sleep for and then attach to the process using 'strace -p'.

You can also set the CYGWIN_DEBUG variable to the name of the program
that you want to debug to have a debugger popped up when that program
executes.

cgf

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

RE: Odd behaviour of __cygwin_environ?

[Dave Korn]

On 05 December 2007 17:25, Christopher Faylor wrote:

> I have a vague feeling that I'll regret mentioning this 

  Oh, I hope not.

> but, if you've
> built the DLL with --enable-debugging, you can set the CYGWIN_SLEEP
> environment variable, to some millisecond value that you want the cygwin
> process to sleep for and then attach to the process using 'strace -p'.
> 
> You can also set the CYGWIN_DEBUG variable to the name of the program
> that you want to debug to have a debugger popped up when that program
> executes.

  Thanks, I've been looking in that area...

static void
initial_env ()
{
  char buf[CYG_MAX_PATH];
  if (GetEnvironmentVariable ("CYGWIN_TESTING", buf, sizeof (buf) - 1))
    _cygwin_testing = 1;

  if (GetEnvironmentVariable ("CYGWIN_STRACE", buf, sizeof (buf) - 1))
    {
      strace.hello ();
      dk_printf ("dk: found CYGWIN STRACE, hello'd.");
    }
#ifdef DEBUGGING
  DWORD len;


  ... along with that patch, I also hacked about strace.hello to auto-activate
itself when it spots that the program name contains "cc1plus.exe", which is
where I'm getting the problem.  I'm currently trying to test the theory that
the environment might not be correct inbetween cygwin1 DLL_PROCESS_ATTACH time
and application main() invocation when it's >32k.

  Just in case anyone's curious, I'm finding that an mno-cygwin invocation of
g++ turns into an mcygwin invocation of cc1plus, and it occurs to me to wonder
if

static void
set_mingw (void)
{
  char *env = getenv ("GCC_CYGWIN_MINGW");

isn't guaranteed to work when the environment is >32k, owing to set_mingw
being called:

static void set_mingw (void) __attribute__ ((constructor));

prior to entering main() (and hence without a clear ordering WRT. crt0 process
init sequence).


    cheers,
      DaveK
-- 
Can't think of a witty .sigline today....


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

question with xargs and jar- seems to drop files.

[Mike Marchywka]

Hi,
I  seem to have an odd problem with xargs and jar. I'm not sure if I can isolate a simple test
case but essentially it only seems to be taking some of the input files.

This seems to work suggesting the file list is acceptable to jar:

$ more latest65day | jar -@cvf wtf2.zip>jarlog2.txt

this, however, only includes a some files, apparently lopping off certain directories:

$ more latest65day | xargs jar -cvf wtf2.zip>jarlog.txt


$ xargs --version
GNU xargs version 4.3.8
Built using GNU gnulib version 2007-05-26

Administrator@TESTBED01 /cygdrive/e/new/temp
$ jar -V
jar (fastjar) 0.92-gcc

Copyright 1999, 2000, 2001  Bryan Burns
Copyright 2002 Free Software Foundation

Thoughts on diagnostic process of known problems? At first, I thought it had to do with
errors encountered during the jarring but I think I've eliminated all of those ( missing files
mostly). Also, as I indicated before, my configuration is suspect but I wouldn't expect it
to cause problems like this.


Thanks.

Mike Marchywka
586 Saint James Walk
Marietta GA 30067-7165
404-788-1216 (C)<- leave message
989-348-4796 (P)<- emergency only
marchywka@hotmail.com
Note: Hotmail is blocking my mom's entire
ISP claiming it is to reduce spam but probably
to force users to use hotmail. Please DON'T
assume I am ignoring you and try
me on marchywka@yahoo.com if no reply
here. Thanks.

_________________________________________________________________
Share life as it happens with the new Windows Live.Download today it's FREE!
http://www.windowslive.com/share.html?ocid=TXT_TAGLM_Wave2_sharelife_112007

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

RE: question with xargs and jar- seems to drop files.

[Dave Korn]

On 05 December 2007 18:04, Mike Marchywka wrote:

> This seems to work suggesting the file list is acceptable to jar:
> 
> $ more latest65day | jar -@cvf wtf2.zip>jarlog2.txt
> 
> this, however, only includes a some files, apparently lopping off certain
> directories: 
> 
> $ more latest65day | xargs jar -cvf wtf2.zip>jarlog.txt

> Thoughts on diagnostic process of known problems? At first, I thought it
> had to do with errors encountered during the jarring but I think I've
> eliminated all of those ( missing files mostly). Also, as I indicated
> before, my configuration is suspect but I wouldn't expect it 
> to cause problems like this.

  I've had problems before with xargs, where it ends up generating a
command-line that is too long to spawn!

  There must be some weakness in the way xargs calculates the amount of space
available for a commandline under cygwin; I found that an effective workaround
was to give xargs the --max-chars flag with a size of around 16000 to 18000,
but depending on the size of your environment variables.

  Your mileage *will* vary, so you might need to experiment a bit.

    cheers,
      DaveK
-- 
Can't think of a witty .sigline today....


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: question with xargs and jar- seems to drop files.

[Richard Copley]

On 05/12/2007, Mike Marchywka <marchywka@hotmail.com> wrote:
> I  seem to have an odd problem with xargs and jar. I'm not sure if I can isolate a simple test
> case but essentially it only seems to be taking some of the input files.
>
> This seems to work suggesting the file list is acceptable to jar:
>
> $ more latest65day | jar -@cvf wtf2.zip>jarlog2.txt
>
> this, however, only includes a some files, apparently lopping off certain directories:
>
> $ more latest65day | xargs jar -cvf wtf2.zip>jarlog.txt
>
>
> $ xargs --version
> GNU xargs version 4.3.8
> Built using GNU gnulib version 2007-05-26
>
> Administrator@TESTBED01 /cygdrive/e/new/temp
> $ jar -V
> jar (fastjar) 0.92-gcc
>
> Copyright 1999, 2000, 2001  Bryan Burns
> Copyright 2002 Free Software Foundation
>
> Thoughts on diagnostic process of known problems? At first, I thought it had to do with
> errors encountered during the jarring but I think I've eliminated all of those ( missing files
> mostly). Also, as I indicated before, my configuration is suspect but I wouldn't expect it
> to cause problems like this.

I expect you've got this covered, but last week when I had a similar
problem with xargs and tar, it turned out that my problem was that I
hadn't read the xargs manpage (not recently anyway). Its second
sentence is "xargs ... executes the  command ... one or more times
...". The solution was to use tar's --files-from option. For jar: -@,
see above (except that you should redirect jar's standard input to
your file-of-files with "<latest65day", rather than using "more" and a
pipe. http://groups.google.co.uk/group/comp.unix.shell/search?q=useless+use+of+cat
;).

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/