How to "bisect" Cygwin?

How to "bisect" Cygwin?

[Ryan C. Underwood]

[-- Attachment #1: Type: text/plain, Size: 421 bytes --]


How can I "bisect" Cygwin i.e. by setting a certain date at which to
install packages that were current as of that date, and checking if
problem remains, so as to track down the date a problem/different
behavior was introduced sometime in the past year?  Actually, if I
could start with the Cygwin DLLs themselves, that would probably get
me to the bottom of it.

-- 
Ryan C. Underwood, <nemesis@icequake.net>

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: How to "bisect" Cygwin?

[Ryan C. Underwood]

[-- Attachment #1: Type: text/plain, Size: 669 bytes --]


On Thu, May 31, 2012 at 11:51:53PM +0200, Cyrille Lefevre wrote:
> 
> well, have you tryed the cygwin time machine :
> 
> http://www.fruitbat.org/Cygwin/index.html#cygwincirca

This indeed appears to be what I was looking for:

  The mirror I maintain contains every version of every package of
  Cygwin, as of the date given (March of 2003). Each time I sync, I also
  keep a revision of setup.ini to match the download. This represents a
  snapshot in time. I've setup a directory hierarchy, based on
  datestamp, which allows you to run setup against and get the packages
  of that "era".

Thanks,

-- 
Ryan C. Underwood, <nemesis@icequake.net>

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: How to "bisect" Cygwin?

[Corinna Vinschen]

On May 31 19:15, Ryan C. Underwood wrote:
> 
> On Thu, May 31, 2012 at 11:51:53PM +0200, Cyrille Lefevre wrote:
> > 
> > well, have you tryed the cygwin time machine :
> > 
> > http://www.fruitbat.org/Cygwin/index.html#cygwincirca
> 
> This indeed appears to be what I was looking for:

Better: Use the developer snapshots from http://cygwin.com/snapshots/
This allows more finegrained bisecting.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: How to "bisect" Cygwin?

[Ryan C. Underwood]

[-- Attachment #1: Type: text/plain, Size: 781 bytes --]

On Thu, May 31, 2012 at 11:51:53PM +0200, Cyrille Lefevre wrote:
> 
> well, have you tryed the cygwin time machine :
> 
> http://www.fruitbat.org/Cygwin/index.html#cygwincirca

I must not be understanding how to use this.  For example, if I take
the URL for 2/13:

ftp://www.fruitbat.org/pub/cygwin/circa/2012/02/13/150019

If I put that URL into Cygwin setup.exe, it complains that it can't
find setup.bz2.sig.  If I view in my browser, there are no files
visible.  I guess the file listing could be disabled to avoid mass
leeching.  But then I just don't know what I'm doing wrong, having not
used this service before.  I tried a handful of other URLs with the
same results.  Has anyone else used this recently?

-- 
Ryan C. Underwood, <nemesis@icequake.net>

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

RE: How to "bisect" Cygwin?

[Thrall, Bryan]

Ryan C. Underwood wrote on 2012-06-01: 
> On Thu, May 31, 2012 at 11:51:53PM +0200, Cyrille Lefevre wrote:
>> 
>> well, have you tryed the cygwin time machine :
>> 
>> http://www.fruitbat.org/Cygwin/index.html#cygwincirca
> 
> I must not be understanding how to use this.  For example, if I take
> the URL for 2/13:
> 
> ftp://www.fruitbat.org/pub/cygwin/circa/2012/02/13/150019
> 
> If I put that URL into Cygwin setup.exe, it complains that it can't
> find setup.bz2.sig.  If I view in my browser, there are no files
> visible.  I guess the file listing could be disabled to avoid mass
> leeching.  But then I just don't know what I'm doing wrong, having not
> used this service before.  I tried a handful of other URLs with the
> same results.  Has anyone else used this recently?
>

See the "Old Update" from 8/5/2008 under
http://www.fruitbat.org/Cygwin/index.html#cygwincirca: You need to
specify -X when running Cygwin setup.exe.

--
Bryan Thrall
Principal Software Engineer
FlightSafety International
bryan.thrall@flightsafety.com



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: How to "bisect" Cygwin?

[Ryan C. Underwood]

[-- Attachment #1: Type: text/plain, Size: 403 bytes --]


On Fri, Jun 01, 2012 at 10:28:09AM -0500, Thrall, Bryan wrote:
> 
> See the "Old Update" from 8/5/2008 under
> http://www.fruitbat.org/Cygwin/index.html#cygwincirca: You need to
> specify -X when running Cygwin setup.exe.

Thanks, I'm not sure what about "Old Update" screamed "obsolete info"
to me at first, but this does indeed work so far.

-- 
Ryan C. Underwood, <nemesis@icequake.net>

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: How to "bisect" Cygwin?

[Ryan C. Underwood]

[-- Attachment #1: Type: text/plain, Size: 1027 bytes --]


On Thu, May 31, 2012 at 04:01:43PM -0500, Ryan C. Underwood wrote:
> 
> How can I "bisect" Cygwin i.e. by setting a certain date at which to
> install packages that were current as of that date, and checking if
> problem remains, so as to track down the date a problem/different
> behavior was introduced sometime in the past year?  Actually, if I
> could start with the Cygwin DLLs themselves, that would probably get
> me to the bottom of it.

I located the problem, it came in between the Time Machine 2012/02/04
and 2012/02/05 archives.  Unfortunately, this looks like a major
release for Cygwin happened on that day:

good:
CYGWIN_NT-6.1 MYPC 1.7.9(0.237/5/3) 2011-03-29 10:10 i686 Cygwin
bad:
CYGWIN_NT-6.1 MYPC 1.7.10(0.259/5/3) 2012-02-05 12:36 i686 Cygwin

I tried diffing the CVS by tags but it doesn't look like 1.7.9 was
tagged, or else the tags were removed as obsolete.  Any suggestions as
to the best way to dig into the differences here?

-- 
Ryan C. Underwood, <nemesis@icequake.net>

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: How to "bisect" Cygwin?

[Corinna Vinschen]

On Jun  1 12:46, Ryan C. Underwood wrote:
> 
> On Thu, May 31, 2012 at 04:01:43PM -0500, Ryan C. Underwood wrote:
> > 
> > How can I "bisect" Cygwin i.e. by setting a certain date at which to
> > install packages that were current as of that date, and checking if
> > problem remains, so as to track down the date a problem/different
> > behavior was introduced sometime in the past year?  Actually, if I
> > could start with the Cygwin DLLs themselves, that would probably get
> > me to the bottom of it.
> 
> I located the problem, it came in between the Time Machine 2012/02/04
> and 2012/02/05 archives.  Unfortunately, this looks like a major
> release for Cygwin happened on that day:
> 
> good:
> CYGWIN_NT-6.1 MYPC 1.7.9(0.237/5/3) 2011-03-29 10:10 i686 Cygwin
> bad:
> CYGWIN_NT-6.1 MYPC 1.7.10(0.259/5/3) 2012-02-05 12:36 i686 Cygwin
> 
> I tried diffing the CVS by tags but it doesn't look like 1.7.9 was
> tagged, or else the tags were removed as obsolete.  Any suggestions as
> to the best way to dig into the differences here?

http://cygwin.com/ml/cygwin/2012-06/msg00008.html

And on an even more important note, you never mentioned in this thread
which problem you actually have: http://cygwin.com/problems.html


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: How to "bisect" Cygwin?

[Ryan C. Underwood]

[-- Attachment #1: Type: text/plain, Size: 624 bytes --]

On Fri, Jun 01, 2012 at 08:18:52PM +0200, Corinna Vinschen wrote:
> 
> http://cygwin.com/ml/cygwin/2012-06/msg00008.html

Thanks, I have no idea why I did not receive that mail.  I looked for
"nightly builds" but that would have been why I didn't find such.

> And on an even more important note, you never mentioned in this thread
> which problem you actually have: http://cygwin.com/problems.html

It's a long story and I think it is better for everyone if I find the
offending change first, then decide if the problem is Cygwin or my own
code :-)

Thanks,

-- 
Ryan C. Underwood, <nemesis@icequake.net>

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: How to "bisect" Cygwin?

[Corinna Vinschen]

On Jun  1 13:40, Ryan C. Underwood wrote:
> On Fri, Jun 01, 2012 at 08:18:52PM +0200, Corinna Vinschen wrote:
> > 
> > http://cygwin.com/ml/cygwin/2012-06/msg00008.html
> 
> Thanks, I have no idea why I did not receive that mail.  I looked for
> "nightly builds" but that would have been why I didn't find such.
> 
> > And on an even more important note, you never mentioned in this thread
> > which problem you actually have: http://cygwin.com/problems.html
> 
> It's a long story and I think it is better for everyone if I find the
> offending change first, then decide if the problem is Cygwin or my own
> code :-)

If you can press a long story into a short testcase in plain C with the
bare minimum of code to reproduce the behaviour, it would be much
appreciated.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: How to "bisect" Cygwin?

[Ryan C. Underwood]

[-- Attachment #1: Type: text/plain, Size: 1327 bytes --]

On Fri, Jun 01, 2012 at 08:59:06PM +0200, Corinna Vinschen wrote:
> > 
> > Thanks, I have no idea why I did not receive that mail.  I looked for
> > "nightly builds" but that would have been why I didn't find such.

-20111216, the oldest snapshot, already produces the problem, so I
guess that at least tells me that my problem came in somewhere between
1.7.9 and the end of 2011.  Are there older snapshots archived?

> If you can press a long story into a short testcase in plain C with the
> bare minimum of code to reproduce the behaviour, it would be much
> appreciated.

The basic issue is that sem_wait() is being kicked out with EINTR
extremely frequently (9 out of 10 times or more), which slows my code
to a crawl as it repeatedly retries sem_wait() until it finally
returns zero.  In 1.7.9, it does not appear that sem_wait() is
preempted in this fashion; the thread would simply wait on the
semaphore until it was sem_post()'d and then continue as expected.

Like I said, I am not exactly sure my code is not to blame at this
point, but as a starting point for analysis I wanted to see if this
change in semantics was deliberate or not and if there was any mailing
list discussion or code comments about it at the time the change was
made.

-- 
Ryan C. Underwood, <nemesis@icequake.net>

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: How to "bisect" Cygwin?

[Warren Young]

[-- Attachment #1: Type: text/plain, Size: 730 bytes --]

On 6/1/2012 1:51 PM, Ryan C. Underwood wrote:
> On Fri, Jun 01, 2012 at 08:59:06PM +0200, Corinna Vinschen wrote:
>
>> If you can press a long story into a short testcase in plain C with the
>> bare minimum of code to reproduce the behaviour, it would be much
>> appreciated.
>
> The basic issue is that sem_wait() is being kicked out with EINTR
> extremely frequently (9 out of 10 times or more),

Does the attached program vaguely resemble what your program is trying 
to do?  It does get EINTR, but only because I'm using alarm(2) to add a 
timeout to the sem_wait() call.

You should just get an infinite series of "Child timed out normally 
waiting for the semaphore." messages from it after initialization, one 
per second.

[-- Attachment #2: semtest.c --]
[-- Type: text/plain, Size: 1736 bytes --]

#include <semaphore.h>
#include <sys/types.h>

#include <errno.h>
#include <fcntl.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>

static int timed_out = 0;

void timeout(int sig)
{
	timed_out = 1;
}

int main()
{
	static const char* sem_name = "/myhappysemaphore";
	sem_t* ps = sem_open(sem_name, O_RDWR);
	if (ps) sem_close(ps);	// clean up old semaphore; we probably crashed
	ps = sem_open(sem_name, O_CREAT, 0600, 0);
	if (ps == SEM_FAILED) {
		perror("sem_open");
		return 1;
	}
	puts("Created semamphore.");

	pid_t pid = fork();
	switch (pid) {
		case -1:	// error
			perror("fork");
			return 2;

		case 0:		// child
			signal(SIGALRM, timeout);
			while (1) {
				alarm(1);
				int ret = sem_wait(ps);
				if (ret == -1) {
					if (timed_out) {
						if (errno == EINTR) {
							puts("Child timed out normally waiting "
									"for the semaphore.");
							timed_out = 0;
						}
						else {
							puts("Child timed out ABNORMALLY waiting "
									"for the semaphore!");
							exit(99);
						}
					}
					else {
						perror("sem_wait");
						exit(69);
					}
				}
				else {
					puts("Child got the semaphore!  "
							"How the spit did that happen?");
					exit(42);
				}
			}
			break;

		default: {	// parent
			puts("Created child.  Posting the deathwatch.");
			int status;
			while (1) {
				pid_t ret = waitpid(pid, &status, 0);
				if (ret > 0) {
					if (WIFEXITED(status)) {
						printf("Child %d died peacefully, code %d.\n",
								ret, WEXITSTATUS(status));
						break;
					}
					else {
						printf("Child %d died screaming, status 0x%08X.\n",
								ret, status);
						return 3;
					}
				}
				else {
					perror("waitpid");
					return 4;
				}
			}
		}
	}

	return 0;
}


[-- Attachment #3: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: How to "bisect" Cygwin?

[Ryan C. Underwood]

[-- Attachment #1: Type: text/plain, Size: 1384 bytes --]

On Fri, Jun 01, 2012 at 03:50:19PM -0600, Warren Young wrote:
> >
> >The basic issue is that sem_wait() is being kicked out with EINTR
> >extremely frequently (9 out of 10 times or more),
> 
> Does the attached program vaguely resemble what your program is
> trying to do?  It does get EINTR, but only because I'm using
> alarm(2) to add a timeout to the sem_wait() call.

Yes.  In fact it is very close but with a significant difference.

My (inherited) program is using setitimer() as a master timer, and I
am not sure that the signals are being masked correctly per thread.
In fact, the code is a real mess, but what I have observed is that
with 1.7.9, the SIGALRM which was set by thread 1 does not interrupt
thread 2's sem_wait() system call, but with 1.7.10, the SIGALRM does
interrupt thread 2 (but without delivering the SIGALRM to thread 2,
apparently?  Need to test some more)

Basically, I was hoping to dig down and find some significant changes
in the way signals and threads interact from the 'working' Cygwin to
the 'non-working', which would convince me that I really need to get
to rewriting this stuff if the current Cygwin is considered to be
proper in that respect.  I also wanted to ensure that a bug had not
crept in to Cygwin that would hammer other people with legacy code.

Thanks,

-- 
Ryan C. Underwood, <nemesis@icequake.net>

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

sem_wait frequently returning with EINTR [Was: Re: How to "bisect" Cygwin?]

[Otto Meta]

> The basic issue is that sem_wait() is being kicked out with EINTR
> extremely frequently (9 out of 10 times or more), which slows my code
> to a crawl as it repeatedly retries sem_wait() until it finally
> returns zero.  In 1.7.9, it does not appear that sem_wait() is
> preempted in this fashion; the thread would simply wait on the
> semaphore until it was sem_post()'d and then continue as expected.

Are you using signals or functions that use signals internally? Signals are
a bit wonky in 1.7.9 and they seem to be pretty much broken in 1.7.12 and
newer (I didn’t try any snapshots). Have a look at the problem I reported:
http://cygwin.com/ml/cygwin/2012-05/msg00186.html
While the cancelling part seems to be mostly fixed by now, the signal part
is still unchanged.

Maybe you could try test 3 from my test program attached to the above
mentioned post. If its output changes along with your program’s behaviour
when you try different Cygwin versions, your problem could be signal-related.

Warren Young wrote in http://cygwin.com/ml/cygwin/2012-06/msg00032.html:
> You should just get an infinite series of "Child timed out normally waiting
> for the semaphore." messages from it after initialization, one per second.

This works for me out of the box, but if I create one or more pthreads before
the alarm loop it just fails: Some other thread gets woken up and the main
thread stays stuck in sem_wait.

If Ryan had told us more about his program, we’d know whether he uses multiple
threads or multiple processes...

Cheers,
Otto

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: sem_wait frequently returning with EINTR [Was: Re: How to "bisect" Cygwin?]

[Ryan C. Underwood]

[-- Attachment #1: Type: text/plain, Size: 1866 bytes --]

On Sat, Jun 02, 2012 at 12:07:50AM +0200, Otto Meta wrote:
> 
> Are you using signals or functions that use signals internally? Signals are
> a bit wonky in 1.7.9 and they seem to be pretty much broken in 1.7.12 and
> newer (I didn’t try any snapshots). Have a look at the problem I reported:
> http://cygwin.com/ml/cygwin/2012-05/msg00186.html
> While the cancelling part seems to be mostly fixed by now, the signal part
> is still unchanged.
> 
> Maybe you could try test 3 from my test program attached to the above
> mentioned post. If its output changes along with your program’s behaviour
> when you try different Cygwin versions, your problem could be signal-related.

I will give that a try.

> If Ryan had told us more about his program, we’d know whether he uses multiple
> threads or multiple processes...

I'm really sorry for that, I just find that on mailing lists it's easy
to brain-dump and then people's eyes glaze over in response to a wall
of combined text, questions and log traces.  I tend to want to
research my end out first and then get to the point where I can ask
direct questions.

Here is a bird's-eye description of the program:

The program has a master setitimer() timer, a SIGALRM handler, and
five pthreads which each 1) wait on POSIX semaphores, 2) go do work
when semaphore is released and then wait again.

The semaphores are released in the SIGALRM handler based on derived
clocks from the master timer (to achieve 8 Hz, 20 Hz, etc).  This code
was ported from an old embedded system and each thread used to be an
individual timer interrupt handler.

A Linux port was done (against NPTL rather than LinuxThreads
thankfully) and now I am porting the Linux port to Cygwin to reflect
changing circumstances of the system it is hosted on.

-- 
Ryan C. Underwood, <nemesis@icequake.net>

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]