* RE: Handle leak ? @ 2000-07-20 3:28 Steve Croall 2000-07-20 16:56 ` David Baggett 0 siblings, 1 reply; 7+ messages in thread From: Steve Croall @ 2000-07-20 3:28 UTC (permalink / raw) To: 'Paul Redondo', ML Pthreads-Win32 Paul, Do you "join"/"detach" to/from the thread after creating it? If not the resources used by the thread are not cleared up. Look at pthread_join() and pthread_detach() MAN pages. Steve. -----Original Message----- From: Paul Redondo [ mailto:paul@matchvision.com ] Sent: 20 July 2000 11:16 To: ML Pthreads-Win32 Subject: Handle leak ? Hi there. Did anybody noticed a huge handle leak when creating lots of detached threads ? I've been creating lots of detached threads during 24 hours, and the NT Task Manager shows me an incredible amount of opened handles. Ok, my code may be wrong ;-P But, it may not. So, anybody else encountered this problem ? C.U. -- Paul Redondo - paul@matchvision.com - +33 6 16300303 / +33 6 16309014 MatchVision's R&D Team - http://www.matchvision.com ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Handle leak ? 2000-07-20 3:28 Handle leak ? Steve Croall @ 2000-07-20 16:56 ` David Baggett 2000-07-21 0:02 ` pthreads comments and suggestions [was: Re: Handle leak ?] Tristan Savatier 0 siblings, 1 reply; 7+ messages in thread From: David Baggett @ 2000-07-20 16:56 UTC (permalink / raw) To: 'Paul Redondo', ML Pthreads-Win32 Yup, I spent two weeks trying to track this down. There's a bug in the current code (unless it has been fixed in the last month or so) that causes not only the handle leakage you're seeing, but alsoinfrequent crashes. The relevant code is in dll.c: /* * Detached threads have their resources automatically * cleaned up upon exit (others must be 'joined' */ if (self != NULL && self->detachState == PTHREAD_CREATE_DETACHED) { if (self->threadH) { pthread_setspecific (_pthread_selfThreadKey, NULL); _pthread_threadDestroy (self); } } If you don't have the "if (self->threadH)" guard (which I've added here), you'll see handle leakage and very low frequency crashes. I reported this to this list some time ago but didn't get much response, so I doubt the distribution has been updated to fix this. I was not able to track down the bug that this guard prevents. In other words, I could not figure out why it was getting to that point in the code with self->threadH equaling zero. I only found this by spending a long time putting assertions throughout the code. Running full bore for 24 hours, you'll still see a very tiny amount of handle leakage even with this workaround. I don't know what causes that, but it won't be a problem in practice. (You have to create enormous numbers of threads to even detect it.) Dave > -----Original Message----- > From: Paul Redondo [ mailto:paul@matchvision.com ] > Sent: 20 July 2000 11:16 > To: ML Pthreads-Win32 > Subject: Handle leak ? > > > Hi there. > > Did anybody noticed a huge handle leak when creating lots of detached > threads ? > I've been creating lots of detached threads during 24 hours, and the NT Task > Manager shows me an incredible amount of opened handles. Ok, my code may be > wrong ;-P But, it may not. > > So, anybody else encountered this problem ? > > C.U. > -- > Paul Redondo - paul@matchvision.com - +33 6 16300303 / +33 6 16309014 > MatchVision's R&D Team - http://www.matchvision.com ^ permalink raw reply [flat|nested] 7+ messages in thread
* pthreads comments and suggestions [was: Re: Handle leak ?] 2000-07-20 16:56 ` David Baggett @ 2000-07-21 0:02 ` Tristan Savatier 2000-07-21 14:09 ` David Baggett 0 siblings, 1 reply; 7+ messages in thread From: Tristan Savatier @ 2000-07-21 0:02 UTC (permalink / raw) To: David Baggett, pthreads-win32 David Baggett wrote: > > Yup, I spent two weeks trying to track this down. > > There's a bug in the current code (unless it has been fixed in the last > month or so) that causes not only the handle leakage you're seeing, > but alsoinfrequent crashes. The relevant code is in dll.c: > > /* > * Detached threads have their resources automatically > * cleaned up upon exit (others must be 'joined' > */ > if (self != NULL && self->detachState == PTHREAD_CREATE_DETACHED) { > if (self->threadH) { > pthread_setspecific (_pthread_selfThreadKey, NULL); > _pthread_threadDestroy (self); > } > } If you look in _pthread_threadDestroy (in private.c), you'll see that there is already a guard against the case where threadH == 0. In fact this guard looks un-necessary: if I look in create.c, I don't see any situation where thread->threadH can be 0. If it was zero, then pthread_create would return EAGAIN (of course there is no way to really know why there was an error...). thread->threadH = 0 means that the thread structure has been trashed... By the way, I think that it would be a good idea to create a TSD variable in which pthread routines would store the Win32 error code indicating the reason of the most recent failure. A special routine, say GetLastPthreadError(), could be called to get this win32 error code. So if I call a pthread function and get some meaningless UNIX error code (e.g. pthread_create returns EAGAIN), there would be a way to know what is the real error. Also, it would be nice if there was a routine to get the win32 thread handle from the pthread_t id. In certain cases, even though we use pthread_win32, we want to refine the control the we have on threads. For example, on Windows-CE 3.0, it is possible to set the thread quantum with CeSetThreadQuantum(). But for that we need the thread handle. It is easy to get if for the current thread (with GetCurrentThread()), but there is no way to get it for another thread if we only know the pthread id... -t ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: pthreads comments and suggestions [was: Re: Handle leak ?] 2000-07-21 0:02 ` pthreads comments and suggestions [was: Re: Handle leak ?] Tristan Savatier @ 2000-07-21 14:09 ` David Baggett 2000-07-23 18:01 ` David Baggett 2000-07-24 0:37 ` Paul Redondo 0 siblings, 2 replies; 7+ messages in thread From: David Baggett @ 2000-07-21 14:09 UTC (permalink / raw) To: Tristan Savatier, pthreads-win32 >If you look in _pthread_threadDestroy (in private.c), you'll see that >there is already a guard against the case where threadH == 0. > >In fact this guard looks un-necessary: Ah, but there is no such guard in pthread_setspecific, which is where I gather it's crashing. (I'm going from memory, but I promise you that guard had an effect in the distribution code from October.) The best thing to do is try it: write a program that does nothing but create and destroy threads. Let it run for hours. It will leak handles. It will crash. I tested this empirically on three different NT machines. I got this same reaction when I notified this list about the bug before: nobody believed there was a bug. TRY IT. (I posted sample code to exhibit the bug a while back, but don't have it handy now. It is trivial to rewrite, though. It is literally a ten line program.) This is indicative of a major problem in the DLL and with that bug you cannot deploy a production application with the DLL. You may or may not get random infrequent crashes -- not what you want in production code... I regret I did not have the time to find and fix the actual bug, but I've at least gotten you a good start. :) Dave ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: pthreads comments and suggestions [was: Re: Handle leak ?] 2000-07-21 14:09 ` David Baggett @ 2000-07-23 18:01 ` David Baggett 2000-07-24 0:37 ` Paul Redondo 1 sibling, 0 replies; 7+ messages in thread From: David Baggett @ 2000-07-23 18:01 UTC (permalink / raw) To: Tristan Savatier, pthreads-win32 ----- Original Message ----- From: "Tristan Savatier" <tristan@mpegtv.com> To: "David Baggett" <dmb@itasoftware.com> Sent: Friday, July 21, 2000 6:19 PM Subject: Re: [pthread-win32] Re: pthreads comments and suggestions [was: Re: Handle leak ?] I suspect that, for the most part, we are in violent agreement. > Yes, but there is no way that threadH == 0 when pthread_create > returns with success. Right. I agree that, looking at the source, there is no obvious way that could happen. Furthermore, I'm pretty sure I put asserts in pthread_create to trap this case, and didn't see it happen even in cases where the guard was triggered. While this doesn't exonerate pthread_create, it certainly makes it unlikely that it is at fault. > So this only thing that is possible is that it gets > trashed later by the application (or by a bug in ANY > library). Since a tiny program with no other libraries or dependencies can exhibit the bug, I don't really suspect anything but the DLL itself, although I suppose it could be the compiler. I only tried this with MSVC++5 and MSVC++6. Let me be clear that it's quite possible that the bug is not *only* the fault of the DLL. It could be the interaction of aspects of or bugs in Windows NT with the DLL. (Remember that putting severe stress on the O/S causes the bug to manifest itself much more frequently.) I'll also point out two things that may or may not be relevant, but which are certainly scary about NT: 1) Run a test program that creates lots of threads. Call up the task manager. While your program is running, grab a slider on a window and slide it back and forth rapidly. Notice how thread creation drops to zero as the system hiccups. What's up with that? I have no idea, but it doesn't make me feel too warm and fuzzy. 2) If you grep Deja News at www.deja.com you will find messages from knowledgeable people claiming that the NT scheduler has various problems under high-load situations. Obviously many multithreaded apps do work under NT in practice. But perhaps the pthreads DLL is doing something that typical applications do not, which is getting NT into states that have not been extensively tested. I don't pretend to know what the bug might be; I'm just pointing out that there are possible problems at many levels. > If that happens, it is VERY BAD, and you little fix would probably > not improve the situation significantly. I agree with your sentiment that it looks like a memory corruption bug, that my workaround is not something you want to put in place and then forget about, and that it is at best ameliorating a VERY BAD bug. However, the fact is that in practice the guard either stops or significantly slows the handle leakage and crashes, which allowed me to run my (larger) application for long periods of time without any of the ill-effects seen previously, so in practice it does actually improve things significantly. This could be because the memory corruption is occuring very soon before the guarded code. Without knowing the source of the bug, it is impossible to draw any sound conclusions. At the least, it's a useful stopgap until the bug can be found and fixed, and may provide valuable clues about where the bug lies. And as I said, I wish I'd had the time and wherewithall to actually track the bug down and fix it, but I didn't. This was for a commercial application so the schedule was too rigid to allow further debugging (after two weeks of putting assertions in the code and trying random things). I moved the application to Linux rather than agonize further over it. But I'm glad someone else is concerned about it. That may mean it will get fixed, which would be a very good thing. You can grep the logs of the mailing list for my email address to find my previous postings on this topic, where I posted code and more detailed findings. Dave ^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: pthreads comments and suggestions [was: Re: Handle leak ?] 2000-07-21 14:09 ` David Baggett 2000-07-23 18:01 ` David Baggett @ 2000-07-24 0:37 ` Paul Redondo 2000-07-24 0:50 ` Ross Johnson 1 sibling, 1 reply; 7+ messages in thread From: Paul Redondo @ 2000-07-24 0:37 UTC (permalink / raw) To: pthreads-win32 > I got this same reaction when I notified this list about the bug > before: nobody believed there was a bug. TRY IT. I do confirm. There IS a bug. I'm gonna try your fix (the null testing), and relaunch the whole test (creation of detached threads during 24 hours). Feedback will be given to the list. C.U. -- Paul Redondo - paul@matchvision.com - +33 6 16300303 / +33 6 16309014 MatchVision's R&D Team - http://www.matchvision.com ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: pthreads comments and suggestions [was: Re: Handle leak ?] 2000-07-24 0:37 ` Paul Redondo @ 2000-07-24 0:50 ` Ross Johnson 0 siblings, 0 replies; 7+ messages in thread From: Ross Johnson @ 2000-07-24 0:50 UTC (permalink / raw) To: Paul Redondo; +Cc: pthreads-win32 Paul Redondo wrote: > > > I got this same reaction when I notified this list about the bug > > before: nobody believed there was a bug. TRY IT. > > I do confirm. There IS a bug. I'm gonna try your fix (the null testing), and > relaunch the whole test (creation of detached threads during 24 hours). David Baggett is currently testing another patch, this time in pthread_create(). The theory is that _beginthreadex() should start threads suspended so that the HANDLE it returns can be assigned to thread-threadH before the thread gets too far or ends (if it's a very short thread). Seems obvious now, but ... That is, the patch below: thread->threadH = (HANDLE) _beginthreadex ( (void *) NULL, /* No security info */ (unsigned) stackSize, /* default stack size */ (unsigned (PT_STDCALL *) (void *)) _pthread_threadStart, parms, (unsigned) CREATE_SUSPENDED, (unsigned *) &(thread->thread)); if (thread->threadH != (HANDLE) 0 && run) { ResumeThread(thread->threadH); } -- +----------------------+---+ | Ross Johnson | | E-Mail: rpj@ise.canberra.edu.au | Info Sciences and Eng|___| | University of Canberra | FAX: +61 6 2015227 | PO Box 1 | | Belconnen ACT 2616 | WWW: http://willow.canberra.edu.au/~rpj/ | AUSTRALIA | +--------------------------+ ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2000-07-24 0:50 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2000-07-20 3:28 Handle leak ? Steve Croall 2000-07-20 16:56 ` David Baggett 2000-07-21 0:02 ` pthreads comments and suggestions [was: Re: Handle leak ?] Tristan Savatier 2000-07-21 14:09 ` David Baggett 2000-07-23 18:01 ` David Baggett 2000-07-24 0:37 ` Paul Redondo 2000-07-24 0:50 ` Ross Johnson
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).