Re: pthread problem with latest cygwin dll snapshot

Re: pthread problem with latest cygwin dll snapshot

[Arash Partow]

Greetings to Thomas and all others involved in cygwin pthreads
implementation,


I made the changes that you advised, however the ThreadTest still crashes,
I also put some text to standard out when the result of pthread_create is
not equal to 0, the text does not show which leads to believe me the
problem does not occur there.

I also made modification to the garbage collector not to add the dead
thread to the thread list, rather just delete it immediately and also to
bark an error to the standard out which never gets displayed, hence further
proving that the problem is somewhere else.

There is no deterministic way to make it crash, just that it will
eventually crash in a very short period of time, if some other things are
done on the system. I wonder if you have tried downloading the shareware
version of the TaskInfo program I use, during its trial period everything
is enabled so you will be able to do the Free RAM (fast/slow) and also
maybe you can do something like play an mpeg in the background, and maybe
use the winsock by doing something on the net. One very unusual yet
interesting thing I noticed whilst running the ThreadTest, was that a split
second before it crashes, TaskInfo reports a huge amount of memory being
acquired by ThreadTest, by huge I mean around 100mb+ the amount is never
the same, but I've never seen it go below 100Mb, I'm sure there is
something wrong happening for something like that to be occur. the only
problem is I can't figure out whether the memory is being requested by the
cygwin layer on behalf of ThreadTest or by the cygwin layer for its own
purpose. I can be sure that no where in the ThreadTest do I acquire 100MB+
of memory from the OS.

Another thing i've noticed about the test crashing, is that even though its
a console app, windows barks an error, via an error window with the title
"ThreadTest.exe - Application Error", in the body of the window it says:
The instruction at "some mem address" refrenced memory at "some mem 
address".
The memory could not be "read".

But the strange thing is that even after this error window appears the
ThreadTest seems to run for some time still, you can see threads being
created, but after a certain amount of time that too stops. When you try to
close the error window down, another pops up under it, I think there is an
error window for each thread that stuffed up. But as before, GDB still says
the last call before the crash was to the cygwin1.dll. I've done my testing
with both debug on and off and different levels of optimization, in all the
cases they produce the same outcome.

All you need to do is be doing something and it will crash the more intense
the processing of the other tasks the more quickly the ThreadTest will
crash.

I've tried some other combinations for running the ThreadTest, in one
instance I ran 30 instances of the test and saw that "sometimes" when 1 of
them crashed other instances would also crash, this lead me to believe that
only 1 instance of cygwin dll is ever loaded into memory and that apps
using it look for one in the memory first before deciding whether or not
they should be loading the cygwin1.dll, One theory I came up with was that
the other instances that were crashing were in someway sequentially related
to the first instance that crashed meaning that either the instance that
crashed and instances that were executed prior to it crashed, or that the
instance and instances executed after it crashed, this theory turned out to
be false, when I copied the ThreadTest app 30 times each time appending a
number to its file name, then executing them sequentially based on their
number, I found that the apps died in a random fashion, no pattern could be
found.


I've tried running the ThreadTest in bash ( I know it wont change much) but
it crashes in there too, one deterministic way of making it crash is if
you pipe the stdout of ThreadTest to tee, ie:
ThreadTest | tee out.dat

This I can guarantee will crash within the 1st 1000000 threads being 
created.


Other than that I have no other ideas of ways to make it crash more
regurarly. I hope this will be enough information for you to continue your
debugging.

I think the goal should be 5million threads being created, with other tasks
also running in the background. If the ThreadTest can achive that level of
stability, I wont bug this list anymore :)

Regards


Arash Partow



PS: The updated version of ThreadTest can be downloaded from:
    http://www.partow.net/downloads/ThreadTest-Prototype.zip



__________________________________________________
Be one who knows what they don't know,
Instead of being one who knows not what they don't know,
Thinking they know everything about all things.
http://www.partow.net






>
>this time I am not able to reproduce your problem.
>
>Before I continue debugging I suggest that you check the return codes of 
>some pthread API functions first and call the pthread_functions only with a 
>valid thread id.
>
>For example change
>
>void Thread::start()
>{
>     pthread_create(&threadID,NULL,&(Thread::threadFunction),this);
>     pthread_detach(threadID);
>}
>
>to
>
>void Thread::start()
>{
>    int res;
>    res = pthread_create(&threadID,NULL,&(Thread::threadFunction),this);
>    if (res)
>      setThreadState(THREAD_DEAD);
>    else
>      pthread_detach(threadID);
>}
>
>Please contact me if this still doesn't help.
>
>Thomas
>
>















.

_________________________________________________________________
Hot chart ringtones and polyphonics. Go to  
http://ninemsn.com.au/mobilemania/default.asp


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: pthread problem with latest cygwin dll snapshot

[Thomas Pfaff]

Hi Arash,

i downloaded TaskInfo, installed it on a 2,5 GHz P4 W2000 box, tried 
free RAM (slow and fast) and your Test did not crash. I rebuilded the 
cygwin DLL while your test was running, still no crash.

If you are interested to dig a little deeper download the snapshot 
source and build your own dll.

The steps are:

1. Download cygwin-src-20031028.tar.bz2
2. Create a directory /usr/src/cygwin. Go to that directory.
3. Untar the source tarball (tar -xpjvf cygwin-src-20031028.tar.bz2)
4. Create a directory named BUILD.
5. Goto BUILD
6. set CFLAGS, CXXFLAGS and LDFLAGS to '-g' (export CFLAGS='-g'; export 
CXXFLAGS='-g'; export LDFLAGS='-g')
7. configure (../cygwin-snapshot-20031028-1/configure --prefix=/usr)
8. make

There is no need for a make install, just copy new-cygwin1.dll from 
i686-pc-cygwin/winsup/cygwin into your /cygwin/bin directory as 
cygwin1.dll after you have terminated all cygwin programs.

If you see a crash in the cygwin dll you can use addr2line to get the 
source line (addr2line -e /bin/cygwin1.dll "address", for example 
addr2line -e /bin/cygwin1.dll 0x61061756).

If you want to debug the dll you might want to set the CFLAGS and 
CXXFLAGS to '-O0 -g' before you configure. To set breakpoints in the 
cygwin DLL start 'gdb TestThread', set a breakpoint at main and start 
your app. After it stops at main you can set breakpoints at functions in 
the cygwin dll (b pthread_create for example).

Good luck.

Thomas

Arash Partow wrote:
> Greetings to Thomas and all others involved in cygwin pthreads
> implementation,
> 
> 
> I made the changes that you advised, however the ThreadTest still crashes,
> I also put some text to standard out when the result of pthread_create is
> not equal to 0, the text does not show which leads to believe me the
> problem does not occur there.
> 
> I also made modification to the garbage collector not to add the dead
> thread to the thread list, rather just delete it immediately and also to
> bark an error to the standard out which never gets displayed, hence further
> proving that the problem is somewhere else.
> 
> There is no deterministic way to make it crash, just that it will
> eventually crash in a very short period of time, if some other things are
> done on the system. I wonder if you have tried downloading the shareware
> version of the TaskInfo program I use, during its trial period everything
> is enabled so you will be able to do the Free RAM (fast/slow) and also
> maybe you can do something like play an mpeg in the background, and maybe
> use the winsock by doing something on the net. One very unusual yet
> interesting thing I noticed whilst running the ThreadTest, was that a split
> second before it crashes, TaskInfo reports a huge amount of memory being
> acquired by ThreadTest, by huge I mean around 100mb+ the amount is never
> the same, but I've never seen it go below 100Mb, I'm sure there is
> something wrong happening for something like that to be occur. the only
> problem is I can't figure out whether the memory is being requested by the
> cygwin layer on behalf of ThreadTest or by the cygwin layer for its own
> purpose. I can be sure that no where in the ThreadTest do I acquire 100MB+
> of memory from the OS.
> 
> Another thing i've noticed about the test crashing, is that even though its
> a console app, windows barks an error, via an error window with the title
> "ThreadTest.exe - Application Error", in the body of the window it says:
> The instruction at "some mem address" refrenced memory at "some mem 
> address".
> The memory could not be "read".
> 
> But the strange thing is that even after this error window appears the
> ThreadTest seems to run for some time still, you can see threads being
> created, but after a certain amount of time that too stops. When you try to
> close the error window down, another pops up under it, I think there is an
> error window for each thread that stuffed up. But as before, GDB still says
> the last call before the crash was to the cygwin1.dll. I've done my testing
> with both debug on and off and different levels of optimization, in all the
> cases they produce the same outcome.
> 
> All you need to do is be doing something and it will crash the more intense
> the processing of the other tasks the more quickly the ThreadTest will
> crash.
> 
> I've tried some other combinations for running the ThreadTest, in one
> instance I ran 30 instances of the test and saw that "sometimes" when 1 of
> them crashed other instances would also crash, this lead me to believe that
> only 1 instance of cygwin dll is ever loaded into memory and that apps
> using it look for one in the memory first before deciding whether or not
> they should be loading the cygwin1.dll, One theory I came up with was that
> the other instances that were crashing were in someway sequentially related
> to the first instance that crashed meaning that either the instance that
> crashed and instances that were executed prior to it crashed, or that the
> instance and instances executed after it crashed, this theory turned out to
> be false, when I copied the ThreadTest app 30 times each time appending a
> number to its file name, then executing them sequentially based on their
> number, I found that the apps died in a random fashion, no pattern could be
> found.
> 
> 
> I've tried running the ThreadTest in bash ( I know it wont change much) but
> it crashes in there too, one deterministic way of making it crash is if
> you pipe the stdout of ThreadTest to tee, ie:
> ThreadTest | tee out.dat
> 
> This I can guarantee will crash within the 1st 1000000 threads being 
> created.
> 
> 
> Other than that I have no other ideas of ways to make it crash more
> regurarly. I hope this will be enough information for you to continue your
> debugging.
> 
> I think the goal should be 5million threads being created, with other tasks
> also running in the background. If the ThreadTest can achive that level of
> stability, I wont bug this list anymore :)
> 


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: pthread problem with latest cygwin dll snapshot

[Arash Partow]

Hi Thomas,

I'll try doing as you have suggested, I'll rebuild the dll and see how
things turn out from there.

One thing I forgot to mention in my previous e- mail, was that when you run
multiple instances of ThreadTest, when they start crashing, the Error
Windows that pop-up all have the same "referenced memory" location. I just
thought this is rather strange.

1 thing though, did you try piping the stdout to tee? I found that doing
that was a sure way of crashing it.

Anywayz thanx heaps for the help so-far I'll get back to you after I get
the building done.



Regards


Arash Partow



__________________________________________________
Be one who knows what they don't know,
Instead of being one who knows not what they don't know,
Thinking they know everything about all things.
http://www.partow.net




>
>Hi Arash,
>
>i downloaded TaskInfo, installed it on a 2,5 GHz P4 W2000 box, tried free 
>RAM (slow and fast) and your Test did not crash. I rebuilded the cygwin DLL 
>while your test was running, still no crash.
>
>If you are interested to dig a little deeper download the snapshot source 
>and build your own dll.
>
>The steps are:
>
>1. Download cygwin-src-20031028.tar.bz2
>2. Create a directory /usr/src/cygwin. Go to that directory.
>3. Untar the source tarball (tar -xpjvf cygwin-src-20031028.tar.bz2)
>4. Create a directory named BUILD.
>5. Goto BUILD
>6. set CFLAGS, CXXFLAGS and LDFLAGS to '-g' (export CFLAGS='-g'; export 
>CXXFLAGS='-g'; export LDFLAGS='-g')
>7. configure (../cygwin-snapshot-20031028-1/configure --prefix=/usr)
>8. make
>
>There is no need for a make install, just copy new-cygwin1.dll from 
>i686-pc-cygwin/winsup/cygwin into your /cygwin/bin directory as cygwin1.dll 
>after you have terminated all cygwin programs.
>
>If you see a crash in the cygwin dll you can use addr2line to get the 
>source line (addr2line -e /bin/cygwin1.dll "address", for example addr2line 
>-e /bin/cygwin1.dll 0x61061756).
>
>If you want to debug the dll you might want to set the CFLAGS and CXXFLAGS 
>to '-O0 -g' before you configure. To set breakpoints in the cygwin DLL 
>start 'gdb TestThread', set a breakpoint at main and start your app. After 
>it stops at main you can set breakpoints at functions in the cygwin dll (b 
>pthread_create for example).
>
>Good luck.
>
>Thomas
>
>Arash Partow wrote:
>>Greetings to Thomas and all others involved in cygwin pthreads
>>implementation,
>>
>>
>>I made the changes that you advised, however the ThreadTest still crashes,
>>I also put some text to standard out when the result of pthread_create is
>>not equal to 0, the text does not show which leads to believe me the
>>problem does not occur there.
>>
>>I also made modification to the garbage collector not to add the dead
>>thread to the thread list, rather just delete it immediately and also to
>>bark an error to the standard out which never gets displayed, hence 
>>further
>>proving that the problem is somewhere else.
>>
>>There is no deterministic way to make it crash, just that it will
>>eventually crash in a very short period of time, if some other things are
>>done on the system. I wonder if you have tried downloading the shareware
>>version of the TaskInfo program I use, during its trial period everything
>>is enabled so you will be able to do the Free RAM (fast/slow) and also
>>maybe you can do something like play an mpeg in the background, and maybe
>>use the winsock by doing something on the net. One very unusual yet
>>interesting thing I noticed whilst running the ThreadTest, was that a 
>>split
>>second before it crashes, TaskInfo reports a huge amount of memory being
>>acquired by ThreadTest, by huge I mean around 100mb+ the amount is never
>>the same, but I've never seen it go below 100Mb, I'm sure there is
>>something wrong happening for something like that to be occur. the only
>>problem is I can't figure out whether the memory is being requested by the
>>cygwin layer on behalf of ThreadTest or by the cygwin layer for its own
>>purpose. I can be sure that no where in the ThreadTest do I acquire 100MB+
>>of memory from the OS.
>>
>>Another thing i've noticed about the test crashing, is that even though 
>>its
>>a console app, windows barks an error, via an error window with the title
>>"ThreadTest.exe - Application Error", in the body of the window it says:
>>The instruction at "some mem address" refrenced memory at "some mem 
>>address".
>>The memory could not be "read".
>>
>>But the strange thing is that even after this error window appears the
>>ThreadTest seems to run for some time still, you can see threads being
>>created, but after a certain amount of time that too stops. When you try 
>>to
>>close the error window down, another pops up under it, I think there is an
>>error window for each thread that stuffed up. But as before, GDB still 
>>says
>>the last call before the crash was to the cygwin1.dll. I've done my 
>>testing
>>with both debug on and off and different levels of optimization, in all 
>>the
>>cases they produce the same outcome.
>>
>>All you need to do is be doing something and it will crash the more 
>>intense
>>the processing of the other tasks the more quickly the ThreadTest will
>>crash.
>>
>>I've tried some other combinations for running the ThreadTest, in one
>>instance I ran 30 instances of the test and saw that "sometimes" when 1 of
>>them crashed other instances would also crash, this lead me to believe 
>>that
>>only 1 instance of cygwin dll is ever loaded into memory and that apps
>>using it look for one in the memory first before deciding whether or not
>>they should be loading the cygwin1.dll, One theory I came up with was that
>>the other instances that were crashing were in someway sequentially 
>>related
>>to the first instance that crashed meaning that either the instance that
>>crashed and instances that were executed prior to it crashed, or that the
>>instance and instances executed after it crashed, this theory turned out 
>>to
>>be false, when I copied the ThreadTest app 30 times each time appending a
>>number to its file name, then executing them sequentially based on their
>>number, I found that the apps died in a random fashion, no pattern could 
>>be
>>found.
>>
>>
>>I've tried running the ThreadTest in bash ( I know it wont change much) 
>>but
>>it crashes in there too, one deterministic way of making it crash is if
>>you pipe the stdout of ThreadTest to tee, ie:
>>ThreadTest | tee out.dat
>>
>>This I can guarantee will crash within the 1st 1000000 threads being 
>>created.
>>
>>
>>Other than that I have no other ideas of ways to make it crash more
>>regurarly. I hope this will be enough information for you to continue your
>>debugging.
>>
>>I think the goal should be 5million threads being created, with other 
>>tasks
>>also running in the background. If the ThreadTest can achive that level of
>>stability, I wont bug this list anymore :)
>>
>
>










.

_________________________________________________________________
Hot chart ringtones and polyphonics. Go to  
http://ninemsn.com.au/mobilemania/default.asp


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: pthread problem with latest cygwin dll snapshot

[Thomas Pfaff]

Arash Partow wrote:
> Greetings to Thomas and all others involved in cygwin pthreads
> implementation,
> 
> I've downloaded the snapshots of cygwin1.dll (cygwin1-20031025.dll.bz2 and
> cygwin1-20031027.dll.bz2), I think the prior is the one where Thomas made
> changes and in the latter Corinna made changes to semaphores which I don't
> how that might effect pthreads. In both cases the stress test can run
> successfully under certain circumstances (which for normal application
> execution are not acceptable) in-one instance reaching 9million+ thread
> creations, however if one tries to do other things like edit a text file or
> play an mp3 or some other task, the Thread-Test will crash. In GDB the call
> stack shows the last call before the crash was to the cygwin1.dll, also
> sometimes if you try to restart the Thread-Test right after it has crashed,
> the new instance crashes almost immediately only ever getting up to about
> ~5000 thread creations.

Hi Arash,

this time i am not able to reproduce your problem.

Before i continue debugging i suggest that you check the return codes of 
some pthread API functions first and call the pthread_functions only 
with a valid thread id.

For example change

void Thread::start()
{
    pthread_create(&threadID,NULL,&(Thread::threadFunction),this);
    pthread_detach(threadID);
}

to

void Thread::start()
{
   int res;
   res = pthread_create(&threadID,NULL,&(Thread::threadFunction),this);
   if (res)
     setThreadState(THREAD_DEAD);
   else
     pthread_detach(threadID);
}

Please contact me if this still doesn't help.

Thomas


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

pthread problem with latest cygwin dll snapshot

[Arash Partow]

Greetings to Thomas and all others involved in cygwin pthreads
implementation,

I've downloaded the snapshots of cygwin1.dll (cygwin1-20031025.dll.bz2 and
cygwin1-20031027.dll.bz2), I think the prior is the one where Thomas made
changes and in the latter Corinna made changes to semaphores which I don't
how that might effect pthreads. In both cases the stress test can run
successfully under certain circumstances (which for normal application
execution are not acceptable) in-one instance reaching 9million+ thread
creations, however if one tries to do other things like edit a text file or
play an mp3 or some other task, the Thread-Test will crash. In GDB the call
stack shows the last call before the crash was to the cygwin1.dll, also
sometimes if you try to restart the Thread-Test right after it has crashed,
the new instance crashes almost immediately only ever getting up to about
~5000 thread creations.

There is an application that I have found that continually makes the
Thread-Test crash, its called TaskInfo (http://www.iarsn.com/taskinfo.html),
its basically a taskmanager on steroids. I use this application mainly to
check for memory leaks of applications running over long periods of time.
This application has a facility called Free RAM, if I have Thread-Test
running and I run Free RAM of taskinfo, the Thread-Test will crash. However
no other application on my systems crash.

I've attempted running Thread-Test with the snapshot dlls on the following
OSs and system configurations:

1.) Win2k (SP4) - P4-2.4Ghz 1024RAM
2.) WinXP (SP4) - P4-2.4Ghz 1024RAM
3.) WinNT (SP6) - P3-1.4Ghz  768RAM
3.) Win98 (SE)  - P3-1.0Ghz  512RAM


In all instances they will run fine if there are no other applications
running in the background, and only the basic windows applications are
loaded into memory, and of course no one actually uses the PC, if however
the PC begins running other tasks the Thread-Test goes to hell.


I think getting the cygwin's pthreads implementation stable is a very
important task because I know of many applications under *nixs that require
stable threading to perform their operations properly, and well it would be
nice to see some of those applications running on windows using the cygwin
unix emulation.

So keep up the good work, at least now the test can get pass 250k thread
creations even though you can't really use the system, which is more than
what was happening before.




Regards



Arash Partow

PS: Thanx to Ross Johnson's feed-back I've updated the Thread-Test's
    garbage collector to work a bit more efficiently than it did before.
    The source code can be downloaded from:
    http://www.partow.net/downloads/ThreadTest-Prototype.zip





>
>Hi Arash,
>
>i can confirm that this is a bug in cygwin.
>I will apply my changes when your testcase runs successfull over night, 
>therefore this should be fixed one of the next snapshots.
>
>Thanks for your stress test.
>
>Thomas
>
>
>>Arash Partow wrote:
>>The prototype initially creates 700 threads all of which are contained in 
>>a
>>vector (threadList), each thread does some "simple" string processing
>>(basically tokenize a string) and then exists. On completion of the 
>>thread,
>>the thread sets its own state in the thread-class to dead.
>>
>>Another thread called GarbageCollector, which is created before the other
>>threads are created, is continually traversing the threadList, looking for
>>dead threads, once a dead thread is found, it deletes the pointer pointing
>>to the thread class in the list, creates a new thread and adds it to the
>>end of the list. Hence continually maintaining the number of threads being
>>run at any one time, (hopefully)
>>

_________________________________________________________________
Get less junk mail with ninemsn Premium. Click here  
http://ninemsn.com.au/premium/landing.asp


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/