Anecdotal: Rebase and Visual Studio 2015 and /etc

Anecdotal: Rebase and Visual Studio 2015 and /etc

[KARL BOTTS]

A couple of weeks ago I installed Visual Studio 2015 (aka VS14) onto a machine
that was already running Windows 10 and had VS2013 (aka VS12) on it.  The
machine was once updated from Windows 7, but that was months ago, had been
fine.  It is a huge install -- 20GB disk space, more than an hour, a couple of
reboots.

After that, I had the now semi-familiar problems with cygwin -- "cannot fork,
cannot load dlls", that kind of stuff.

I went thru several variations of the rebase procedures, following various
advice on cygwin web site, and emails from this list.  I could not get cygwin
back to fully normal.

Eventually, I reinstalled a fresh cygwin, right after a reboot, rebooted
again.  Now all has been good for a couple weeks.

I have a couple of anecdotal impressions, from this experience and others:

1) After you do the full rebase, before you even start anything cygwin, reboot
Windows, then start a bash or something.  Reboot Windows, early and often,
whenever upgrading anything.  This has helped me before.  I suspect that I
forgot it this last time.  I _suspect_ that had I followed that, I would not
have had to reinstall cygwin.  Maybe.

2) I always have cygdrive-prefix set to /, so that I can do, e.g. "cd /c". 
But when you reinstall cygwin, you must do it again with "mount -c".  And then
immediately do "mount -m > /etc/fstab", so that it sticks.  Then, you should
patch up the symlinks in /etc/{hosts,services,couple-more}; find them with
readlink.  There are some more in some fonts dir, but I ignore those: don't
care.

3) Once you have a cygwin you like, you can _usually_ copy it to other hosts. 
Use cygwin cp or tar.  Do not use XCOPY: does not handle links right. 
ROBOCOPY can be used in a pinch.  What goes wrong, maybe one time in four, is
rebase troubles: goto (1).  Once you know you can copy cygwin from host A to
host B, you can generally do it again, to keep cygwin up to date across hosts.
 I _think_ that once you have a DLL-load-address compatible install of cygwin
across several hosts, you can freely copy between them.

4) Keep yourself a list of cygwin packages you know you need, keep it small if
you can.  Makes a cygwin install much easier.  I have maybe 30.  I just enter
then in to the cygwin-setup GUI: is painful, but reliable.  I have had trouble
with the "setup -P", sometimes, but I think you could get that to work.

5) Again, reboot Windows between any steps.  Maybe not always necessary, but
can't hurt: you want to know that it _will_ reboot, anyhow.  And, it moves
DLLs around, even installs more, on the way back up.  You want to be sure it
has done all that.

6) Once you get this all working, it is pretty reliable: I have to fuss with
this stuff maybe a couple times a year.


Cygwin is worth it.  I still don't understand why MS did not just help cygwin
get a better installation/maintenance procedure, and fix the damn fork
problems, instead of going down the UbuntuOnWindows path.  But that's another
discussion... 


---
Karl Botts, kdbotts@usa.net


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Anecdotal: Rebase and Visual Studio 2015 and /etc

[Eliot Moss]

On 6/29/2016 8:24 AM, KARL BOTTS wrote:

> 2) I always have cygdrive-prefix set to /, so that I can do, e.g. "cd /c".
> But when you reinstall cygwin, you must do it again with "mount -c".  And then
> immediately do "mount -m > /etc/fstab", so that it sticks.  Then, you should
> patch up the symlinks in /etc/{hosts,services,couple-more}; find them with
> readlink.  There are some more in some fonts dir, but I ignore those: don't
> care.

A minor suggestion here:

I have had less trouble by linking individual drives, e.g.,

/c -> /cygdrive/c

This does not get you every one of them, but you can set more
(/d -> /cygdrive/d, etc.) and cover them over time.

May not be for everyone, but setting that prefix to / seems
to be potentially problematic.

More broadly, and this may be something for the cygwin / rebaseall
maintainers, it would have helped me to have clear instructions as
to how to blow away the rebase database so that I can force it to
be rebuilt.  Installations and upgrades do sometimes get me into
rebase issues, and sometimes it seems that rebuilding the database
(and (often) rebooting as well) are needed to fix it.  Having those
instructions in the documentation (man page, etc.) for rebase and
friends would have saved me time figuring it out.  The docs seem
somewhat to pre-date the advent of the rebase database and incremental
rebasing.

Regards -- Eliot Moss

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Anecdotal: Rebase and Visual Studio 2015 and /etc

[Andrey Repin]

Greetings, KARL BOTTS!

> 2) I always have cygdrive-prefix set to /, so that I can do, e.g. "cd /c".
> But when you reinstall cygwin, you must do it again with "mount -c".  And then
> immediately do "mount -m > /etc/fstab", so that it sticks.  Then, you should
> patch up the symlinks in /etc/{hosts,services,couple-more}; find them with
> readlink.  There are some more in some fonts dir, but I ignore those: don't
> care.

You can supply fstab and nsswitch.conf in advance of a new install, I presume.


-- 
With best regards,
Andrey Repin
Friday, July 1, 2016 09:32:44

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Anecdotal: Rebase and Visual Studio 2015 and /etc

[Warren Young]

On Jun 29, 2016, at 6:24 AM, KARL BOTTS <kdbotts@usa.net> wrote:
> 
> A couple of weeks ago I installed Visual Studio 2015...It is a huge install -- 20GB disk space, more than an hour, a couple of reboots.

In a world where main storage is measured in 100s of MB per second, installing 20 gigs should not take hours.  (Yes, hours: the recent upgrade to VS 2015 SP3 took two hours here.)

Maddening.

> After that, I had the now semi-familiar problems with cygwin -- "cannot fork,
> cannot load dlls", that kind of stuff.

A VS installation shouldn’t affect the rebase setup of Cygwin.

Is this a 32-bit system?  If so, and you have lots of Cygwin libraries installed, it can starve rebase of suitable library loading points.  That gives two obvious solutions: 1) Go 64-bit; or 2) Remove stuff you don’t actually need.

> Eventually, I reinstalled a fresh cygwin

Did you install all the same things, or a minimal install, which you built on top of, installing only things you need right now?  If the latter, perhaps the actual fix was installing fewer Cygwin-based DLLs, thereby giving rebase more freedom to place DLLs.

> 1) After you do the full rebase, before you even start anything cygwin, reboot
> Windows, then start a bash or something.  Reboot Windows, early and often,
> whenever upgrading anything.  This has helped me before.  I suspect that I
> forgot it this last time.  I _suspect_ that had I followed that, I would not
> have had to reinstall cygwin.  Maybe.

I’ve never had to resort to such voodoo to keep Cygwin going.  The standard schedule of reboots due to Patch Tuesday has been sufficient.

> 2) I always have cygdrive-prefix set to /

Me, too.

> But when you reinstall cygwin, you must do it again with "mount -c".  And then
> immediately do "mount -m > /etc/fstab", so that it sticks.

You’re doing it the hard way.

After a fresh install, just edit /etc/fstab.  The last line in the stock version is:

    none /cygdrive cygdrive binary,posix=0,user 0 0

Just change the second field to / and restart Cygwin.

> Then, you should
> patch up the symlinks in /etc/{hosts,services,couple-more}

I’ve never manually done that, primarily because I rarely use such files via Cygwin.

You can force Cygwin to fix such things up for you by reinstalling the base-files package.  That’s doubtless why my symlinks are in order: some upgrade along the line fixed them for me.

> 3) Once you have a cygwin you like, you can _usually_ copy it to other hosts. 
> Use cygwin cp or tar.

rsync -a should also work.

> 4) Keep yourself a list of cygwin packages you know you need,

You don’t need to; Cygwin itself remembers what you have installed.

To clone an existing install using setup.exe:

  $ /path/to/setup-x86_64 -R 'c:\cygwin-clone' -q -L \
    -P $(tail -n+2 installed.db | cut -f1 -d' ' | tr '\n' ,)

The installed.db file is copied from the machine whose Cygwin installation you want to clone.  It lives in /etc/setup.

If you don’t need many packages, you can prune the long list produced by that $() construct way down, so that you install only the “root” packages that cause all the others to be installed as dependencies.

> I have had trouble
> with the "setup -P", sometimes, but I think you could get that to work.

I’ve just tested the above command here to create a local clone.  I encountered no trouble once I had the command debugged.

> 5) Again, reboot Windows between any steps.

If that is needed after setup.exe exits, setup.exe itself will tell you.  If it doesn’t and you get no postinstall script errors, you can safely assume that your new Cygwin installation is ready to use.

> you want to know that it _will_ reboot, anyhow.

Cygwin should never cause a Windows box to become unbootable.  It simply doesn’t get its hooks into the system deeply enough to cause such trouble,

> And, it moves
> DLLs around, even installs more, on the way back up.

“It” being Windows, or Cygwin?

If the former, new Windows DLLs shouldn’t interfere with Cygwin DLLs, unless by some wild coincidence there is an overlap in memory load spaces.  And again, that’s solved by either using 64-bit Windows and 64-bit Cygwin or not installing so much stuff that conflicts are near-inevitable.

If the latter, Cygwin doesn’t install new things on bootup.  Once setup.exe exits, your Cygwin installation should be stable until you fire setup.exe up again.

> I still don't understand why MS did not just help cygwin
> get a better installation/maintenance procedure, and fix the damn fork
> problems, instead of going down the UbuntuOnWindows path.

Because Satya Nadella has been in charge for only about two years now.  The time to help Cygwin with the fork() problem was about 15 years ago.  Ballmer and his predecessors had no interest in helping Windows work with non-Windows platforms.  Nadella’s coming along now and bailing like mad, trying to keep the ship afloat despite their two major cash cows (Office and Windows) shrinking fast in profitability.  Things like WSL are just side effects of that mad activity.

The new WSL is also architecturally more in line with the way the NT kernel was designed.  From that perspective, Cygwin is something of a hack.

Not that Cygwin had any choice; the NT subsystem interface is basically undocumented.  Plus, implementing Cygwin that way would have prevented it from supporting Windows 95 thru ME, and presents other interop problems besides.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Anecdotal: Rebase and Visual Studio 2015 and /etc

[Warren Young]

[-- Attachment #1: Type: text/plain, Size: 771 bytes --]

On Jul 1, 2016, at 1:35 PM, Warren Young wrote:
> 
> To clone an existing install using setup.exe:
> 
>  $ /path/to/setup-x86_64 -R 'c:\cygwin-clone' -q -L \
>    -P $(tail -n+2 installed.db | cut -f1 -d' ' | tr '\n' ,)

[snip]

> ...you can prune the long list produced by that $() construct way down

I’ve written a script to do that automatically.  (Attached.)

It takes the raw list parsed from installed.db using the scheme above and a copy of the setup.ini file downloaded by setup.exe and removes all “non-root” packages, being those that will be installed indirectly by some other package on that list.

It cut my largest local Cygwin installation’s package list down to about a fifth the size spewed out by the command above.

Enjoy!


[-- Attachment #2: find-cyg-roots --]
[-- Type: application/octet-stream, Size: 3001 bytes --]

#!/usr/bin/perl

use strict;
use warnings;

my $prgname = $0;


#### parse_command_line ################################################
# Return a digested form of the command line arguments

sub parse_command_line {
	my ($pn, $inifile) = @_;

	usage() unless defined($inifile);
	my @pkgnames = split ',', $pn;
	usage("no packages given") if @pkgnames == 0;
	usage("cannot read INI file") unless -r $inifile;

	return ($inifile, \@pkgnames);
}


#### parse_cygwin_setup_ini_file #######################################
# Extract dependency info from the Cygwin setup.ini file.

sub parse_cygwin_setup_ini_file {
	my ($inifile, $piref) = @_;

	open my $ini, '<', $inifile
			or die "Cannot read INI file $inifile: $!\n";

	# Skip to first package entry
	while (<$ini>) { last if /^@/; }

	# Parse package entries
	my %deps;
	while (defined $_) {
		chomp;
		my $p = substr $_, 2;
		my $obs = 0;

		while (<$ini>) {
			if (/^@/) {
				# Found next package entry; restart outer loop
				last;
			}
			elsif (/^category: Base$/) {
				# Mark this one as a special sort of root package: one
				# we're going to install regardless of user selection,
				# so we need not list it in our output.
				$piref->{$p} = 2;
			}
			elsif (/^category: _obsolete$/) {
				# Select this package's replacement instead below.
				$piref->{$p} = 0;
				$obs = 1;
			}
			elsif (/^requires:/) {
				# Save this package's requirements as its dependents list.
				my ($junk, @deps) = split;
				$deps{$p} = \@deps;

				# If this package was marked obsolete above, select its
				# replacement as provisionally to-be-installed.  That
				# package still might end up removed from our output list
				# if it in turn is a dependent of one of the packages we 
				# consider a "root" package at the end.
				$piref->{$deps[0]} = 1 if $obs;
			}
		}
	}

	close $ini;
	return \%deps;
}


#### usage #############################################################
# Print usage message plus optional error string, then exit

sub usage {
	my ($error) = @_;
	print "ERROR: $error\n\n" if length($error);

	print <<"USAGE";
usage: $prgname packages ini-path

    packages is a comma-separated list of Cygwin package names, as
    produced by:
	
      \$ tail -n+2 /etc/setup/installed.db | cut -f1 -d' ' | tr '\\n' ,

    ini-path is the path to a Cygwin setup.ini file.
USAGE
	exit ($error ? 1 : 0);
}


#### main ##############################################################

my ($inifile, $pkgnames) = parse_command_line(@ARGV);

# Convert package list to a hash so we can mark them non-root by name
my %packages = map { $_ => 1 } @$pkgnames;

my $deps = parse_cygwin_setup_ini_file($inifile, \%packages);

# For each given package name, mark any of its dependencies also found
# on the command line as as non-root.
for my $p (@$pkgnames) {
	my $pdref = $deps->{$p};
	for my $d (@$pdref) {
		$packages{$d} = 0;
	}
}

# Collect list of root packages and print it out
print join ',', sort(grep { $packages{$_} == 1 } @$pkgnames);


[-- Attachment #3: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Anecdotal: Rebase and Visual Studio 2015 and /etc

[Warren Young]

[-- Attachment #1: Type: text/plain, Size: 409 bytes --]

On Jul 1, 2016, at 4:40 PM, Warren Young wrote:
> 
> I’ve written a script to do that automatically.

I’ve improved the script so that it no longer requires any parameters.  It finds the last-used setup.ini file and extracts the list of currently-installed packages, all on its own.

Thus, calling this script is now as simple as:

    $ /path/to/setup*.exe -P $(/path/to/find-cyg-roots) ...



[-- Attachment #2: find-cyg-roots --]
[-- Type: application/octet-stream, Size: 4087 bytes --]

#!/usr/bin/perl

use strict;
use warnings;

use URI::Escape;

my $prgname = $0;


#### find_setup_ini_file ###############################################
# Parse Cygwin's setup.rc file to find the last setup.ini file it used.

sub find_setup_ini_file {
	open my $rc, '<', '/etc/setup/setup.rc'
			or usage("could not read setup.rc file: $!");

	my ($path, $mirror);
	while (<$rc>) {
		chomp;

		if ($_ eq 'last-cache') {
			$path = <$rc>;
			chomp $path;
			$path =~ s/^\s+//;
			open my $cp, '-|', "cygpath -u '$path'";
			$path = <$cp>;
			chomp $path;
			close $cp;
		}
		elsif ($_ eq 'last-mirror') {
			$mirror = <$rc>;
			chomp $mirror;
			$mirror =~ s/^\s+//;
			$mirror = uri_escape($mirror);
		}
	}

	close $rc;

	usage("could not find last Cygwin cache dir") unless $path;
	usage("could not find last Cygwin DL mirror") unless $mirror;

	for my $parent (glob("$path/$mirror/x86*")) {
		my $path = "$parent/setup.ini";
		return $path if -r $path;
	}
	
	usage("could not find setup.ini");
	return;
}


#### get_installed_package_list ########################################
# Return a list of names of installed packages

sub get_installed_package_list {
	open my $db, '<', '/etc/setup/installed.db'
			or usage("failed to read installed package DB file: $!");

	my $header = <$db>;
	my @pkgnames;
	while (<$db>) {
		my ($name) = split;
		push @pkgnames, $name;
	}

	return \@pkgnames;
}


#### parse_cygwin_setup_ini_file #######################################
# Extract dependency info from the Cygwin setup.ini file.

sub parse_cygwin_setup_ini_file {
	my ($inifile, $piref) = @_;

	open my $ini, '<', $inifile
			or die "Cannot read INI file $inifile: $!\n";

	# Skip to first package entry
	while (<$ini>) { last if /^@/; }

	# Parse package entries
	my %deps;
	while (defined $_) {
		chomp;
		my $p = substr $_, 2;
		my $obs = 0;

		while (<$ini>) {
			if (/^@/) {
				# Found next package entry; restart outer loop
				last;
			}
			elsif (/^category: Base$/) {
				# Mark this one as a special sort of root package: one
				# we're going to install regardless of user selection,
				# so we need not list it in our output.
				$piref->{$p} = 2;
			}
			elsif (/^category: _obsolete$/) {
				# Select this package's replacement instead below.
				$piref->{$p} = 0;
				$obs = 1;
			}
			elsif (/^requires:/) {
				# Save this package's requirements as its dependents list.
				my ($junk, @deps) = split;
				$deps{$p} = \@deps;

				# If this package was marked obsolete above, select its
				# replacement as provisionally to-be-installed.  That
				# package still might end up removed from our output list
				# if it in turn is a dependent of one of the packages we 
				# consider a "root" package at the end.
				$piref->{$deps[0]} = 1 if $obs;
			}
		}
	}

	close $ini;
	return \%deps;
}


#### usage #############################################################
# Print usage message plus optional error string, then exit

sub usage {
	my ($error) = @_;
	print "ERROR: $error\n\n" if length($error);

	print <<"USAGE";
usage: $prgname

    Finds the last-used Cygwin setup.ini file, then uses the
    package dependency info found within it to pare the list of
    currently-installed Cygwin packages down to a "root" set,
    being those that will implicitly install all of the others
    as dependencies.
    
    The output is a list suitable for passing to setup.exe -P.
USAGE
	exit ($error ? 1 : 0);
}


#### main ##############################################################

my $inifile = find_setup_ini_file;

# Convert package list to a hash so we can mark them non-root by name
my $pkgnames = get_installed_package_list;
my %packages = map { $_ => 1 } @$pkgnames;

my $deps = parse_cygwin_setup_ini_file($inifile, \%packages);

# For each given package name, mark any of its dependencies also found
# on the command line as as non-root.
for my $p (@$pkgnames) {
	my $pdref = $deps->{$p};
	for my $d (@$pdref) {
		$packages{$d} = 0;
	}
}

# Collect list of root packages and print it out
print join ',', sort(grep { $packages{$_} == 1 } @$pkgnames);


[-- Attachment #3: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Anecdotal: Rebase and Visual Studio 2015 and /etc

[Ken Brown]

On 7/1/2016 7:38 PM, Warren Young wrote:
> On Jul 1, 2016, at 4:40 PM, Warren Young wrote:
>>
>> IÂ’ve written a script to do that automatically.
>
> IÂ’ve improved the script so that it no longer requires any parameters.  It finds the last-used setup.ini file and extracts the list of currently-installed packages, all on its own.

This script has a couple of problems.  First, it doesn't take dependency 
loops into account.  For example, if package p requires q and q requires 
p, then both will be marked as non-root.  Second, if the mirror was used 
for both an x86 and x86_64 installation, it always uses the x86 
setup.ini, regardless of the current architecture.

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Anecdotal: Rebase and Visual Studio 2015 and /etc

[Ken Brown]

On 7/3/2016 10:02 AM, Ken Brown wrote:
> On 7/1/2016 7:38 PM, Warren Young wrote:
>> On Jul 1, 2016, at 4:40 PM, Warren Young wrote:
>>>
>>> IÂ’ve written a script to do that automatically.
>>
>> IÂ’ve improved the script so that it no longer requires any
>> parameters.  It finds the last-used setup.ini file and extracts the
>> list of currently-installed packages, all on its own.
>
> This script has a couple of problems.  First, it doesn't take dependency
> loops into account.  For example, if package p requires q and q requires
> p, then both will be marked as non-root.  Second, if the mirror was used
> for both an x86 and x86_64 installation, it always uses the x86
> setup.ini, regardless of the current architecture.

The second problem is easy to fix.  I think you can fix the first 
problem by using the dependency order computed by setup and recorded in 
/var/log/setup.log.full:

$ grep "Dependency order" /var/log/setup.log.full
Dependency order of packages: base-cygwin cygwin libdbus1_3 libiconv2 
libintl8 libffi6 libpcre1 liblzma5 libgcc1 libstdc++6 terminfo...

You should only mark a package as a non-root if it is required by a 
package that occurs later in the list.

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Anecdotal: Rebase and Visual Studio 2015 and /etc

[Ken Brown]

On 7/5/2016 9:59 AM, Ken Brown wrote:
> On 7/3/2016 10:02 AM, Ken Brown wrote:

>> This script has a couple of problems.  First, it doesn't take dependency
>> loops into account.  For example, if package p requires q and q requires
>> p, then both will be marked as non-root.  Second, if the mirror was used
>> for both an x86 and x86_64 installation, it always uses the x86
>> setup.ini, regardless of the current architecture.
>
> The second problem is easy to fix.  I think you can fix the first
> problem by using the dependency order computed by setup and recorded in
> /var/log/setup.log.full:
>
> $ grep "Dependency order" /var/log/setup.log.full
> Dependency order of packages: base-cygwin cygwin libdbus1_3 libiconv2
> libintl8 libffi6 libpcre1 liblzma5 libgcc1 libstdc++6 terminfo...
>
> You should only mark a package as a non-root if it is required by a
> package that occurs later in the list.

Sorry, this isn't quite right, at least if your goal is to produce a 
minimal set of roots.  [It will, however, produce a set of roots that is 
not necessarily minimal.]  To get a minimal set you would have to find 
the strongly-connected components of the dependency graph.  [This is 
what is done internally by setup.exe in order to compute the dependency 
order.]  Then when a package is marked as a non-root, all packages in 
the same component should also be marked as non-root.

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Anecdotal: Rebase and Visual Studio 2015 and /etc

[Warren Young]

[-- Attachment #1: Type: text/plain, Size: 735 bytes --]

On Jul 3, 2016, at 8:02 AM, Ken Brown wrote:
> 
> it doesn't take dependency loops into account

I’ve fixed that using your proposed “Dependency order” solution.  I haven’t analyzed the output, but it is a bit longer than the last time, so I assume it has saved me from dropping all packages in a loop.

If this is also found to be insufficient, I think I’d have to build the dependency graph from setup.ini and learn to walk it, rather than continue with this present linear algorithm.

> if the mirror was used for both an x86 and x86_64 installation, it always uses the x86 setup.ini, regardless of the current architecture.

Good catch!  The script now uses the output of uname -m to select the mirror subdirectory.



[-- Attachment #2: find-cyg-roots --]
[-- Type: application/octet-stream, Size: 5644 bytes --]

#!/usr/bin/perl

use strict;
use warnings;

use URI::Escape;

my $prgname = $0;


#### find_setup_ini_file ###############################################
# Parse Cygwin's setup.rc file to find the last setup.ini file it used.

sub find_setup_ini_file {
	open my $rc, '<', '/etc/setup/setup.rc'
			or usage("could not read setup.rc file: $!");

	my ($path, $mirror);
	while (<$rc>) {
		chomp;

		if ($_ eq 'last-cache') {
			$path = <$rc>;
			chomp $path;
			$path =~ s/^\s+//;
			open my $cp, '-|', "cygpath -u '$path'";
			$path = <$cp>;
			chomp $path;
			close $cp;
		}
		elsif ($_ eq 'last-mirror') {
			$mirror = <$rc>;
			chomp $mirror;
			$mirror =~ s/^\s+//;
			$mirror = uri_escape($mirror);
		}
	}

	close $rc;

	usage("could not find last Cygwin cache dir") unless $path;
	usage("could not find last Cygwin DL mirror") unless $mirror;

	open my $uname, '-|', 'uname -m' or die "uname -m failed: $!\n";
	my $plat = <$uname>;
	chomp $plat;
	close $uname;

	$path .= "/$mirror/$plat/setup.ini";
	usage("could not find setup.ini") unless -r $path;
	return $path;
}


#### get_dependency_order ##############################################
# Return a hash mapping package names to their index on the "Dependency
# order" line written to /var/log/setup.log.full by setup.exe.  Lower
# numbers mean they're farther to the left and hence more depended-upon,
# so the index of "base-cygwin" is 0, "cygwin" is 1, etc.

sub get_dependency_order {
	open my $log, '<', '/var/log/setup.log.full'
			or usage("failed to read setup log: $!");
	my @lines = grep { /^Dependency order/ } <$log>;
	usage("no dependency order line in setup log") if @lines == 0;
	usage("multiple dependency order lines found") unless @lines == 1;
	
	my ($preamble, $deporder) = split ':', $lines[0];
	my @packages = split ' ', $deporder;

	my $i = 0;
	my %deps;
	for my $p (@packages) {
		$deps{$p} = $i++;
	}

	return \%deps;
}


#### get_installed_package_list ########################################
# Return a list of names of installed packages

sub get_installed_package_list {
	open my $db, '<', '/etc/setup/installed.db'
			or usage("failed to read installed package DB file: $!");

	my $header = <$db>;
	my @pkgnames;
	while (<$db>) {
		my ($name) = split;
		push @pkgnames, $name;
	}

	return \@pkgnames;
}


#### parse_cygwin_setup_ini_file #######################################
# Extract dependency info from the Cygwin setup.ini file.

sub parse_cygwin_setup_ini_file {
	my ($inifile, $piref) = @_;

	open my $ini, '<', $inifile
			or die "Cannot read INI file $inifile: $!\n";

	# Skip to first package entry
	while (<$ini>) { last if /^@/; }

	# Parse package entries
	my %deps;
	while (defined $_) {
		chomp;
		my $p = substr $_, 2;
		my $obs = 0;

		while (<$ini>) {
			if (/^@/) {
				# Found next package entry; restart outer loop
				last;
			}
			elsif (/^category: Base$/) {
				# Mark this one as a special sort of root package: one
				# we're going to install regardless of user selection,
				# so we need not list it in our output.
				$piref->{$p} = 2;
			}
			elsif (/^category: _obsolete$/) {
				# Select this package's replacement instead below.
				$piref->{$p} = 0;
				$obs = 1;
			}
			elsif (/^requires:/) {
				# Save this package's requirements as its dependents list.
				my ($junk, @deps) = split;
				$deps{$p} = \@deps;

				# If this package was marked obsolete above, select its
				# replacement as provisionally to-be-installed.  That
				# package still might end up removed from our output list
				# if it in turn is a dependent of one of the packages we 
				# consider a "root" package at the end.
				$piref->{$deps[0]} = 1 if $obs;
			}
		}
	}

	close $ini;
	return \%deps;
}


#### usage #############################################################
# Print usage message plus optional error string, then exit

sub usage {
	my ($error) = @_;
	print "ERROR: $error\n\n" if length($error);

	print <<"USAGE";
usage: $prgname

    Finds the last-used Cygwin setup.ini file, then uses the
    package dependency info found within it to pare the list of
    currently-installed Cygwin packages down to a "root" set,
    being those that will implicitly install all of the others
    as dependencies.
    
    The output is a list suitable for passing to setup.exe -P.
USAGE
	exit ($error ? 1 : 0);
}


#### main ##############################################################

my $inifile = find_setup_ini_file;

# Convert package list to a hash so we can mark them non-root by name
my $pkgnames = get_installed_package_list;
my %packages = map { $_ => 1 } @$pkgnames;

my $deps = parse_cygwin_setup_ini_file($inifile, \%packages);
my $deporder = get_dependency_order;

# For each installed package, mark all of its dependencies as non-root
# since those will be installed if the requiring package is installed.
for my $p (@$pkgnames) {
	my $pdref = $deps->{$p};
	for my $d (@$pdref) {
		# Package $p depends on $d, but only mark it as non-root if $p
		# was to the right of $d in the dependency order list written by
		# setup.exe on the last install.  Otherwise, setup.exe is saying
		# $p is more depended-upon than $d, which means we're looking at
		# a dependency graph cycle.  That means we always call the least
		# depended-upon package in that loop as the "root," but that
		# choice is not important.  What that matters is that we avoid
		# marking all packages in that loop as non-root, since then none
		# of them get installed.
		$packages{$d} = 0 if $deporder->{$p} > $deporder->{$d};
	}
}

# Collect list of root packages and print it out
print join ',', sort(grep { $packages{$_} == 1 } @$pkgnames);


[-- Attachment #3: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Anecdotal: Rebase and Visual Studio 2015 and /etc

[Vlado]

On 6.7.2016 7:17, Warren Young wrote:
>> if the mirror was used for both an x86 and x86_64 installation, it always uses the x86 setup.ini, regardless of the current architecture.
> Good catch!  The script now uses the output of uname -m to select the mirror subdirectory.
>

Hi,

this is not allways right. I still use 32-bit Cygwin on 64-bit Windows.

$ uname -m
i686

$ ls 'Downloads/Cygwin/http%3A%2F%2Fgd.tuwien.ac.at%2Fgnu%2Fcygwin%2F/'
noarch  x86

Vlado

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Anecdotal: Rebase and Visual Studio 2015 and /etc

[Vlado]

Hi.

One more sugestion: use -e test instead of -r, because Perl -r is not 
allways reliable in mixed Cygwin/Windows environment.

(Cygwin terminal)
$ perl -e "if( -r 
'Cygwin/http%3A%2F%2Fgd.tuwien.ac.at%2Fgnu%2Fcygwin%2F/x86/setup.ini'){print 
'readable'}else{print 'cant read'}"
cant read

$ head 'Cygwin/http%3A%2F%2Fgd.tuwien.ac.at%2Fgnu%2Fcygwin%2F/x86/setup.ini'
# This file is automatically generated.  If you edit it, your
# edits will be discarded next time the file is generated.
# See http://cygwin.com/setup.html for details.
#
release: cygwin
arch: x86
setup-timestamp: 1466876494
setup-version: 2.874

$ getfacl 
'Cygwin/http%3A%2F%2Fgd.tuwien.ac.at%2Fgnu%2Fcygwin%2F/x86/setup.ini'
# file: Cygwin/http%3A%2F%2Fgd.tuwien.ac.at%2Fgnu%2Fcygwin%2F/x86/setup.ini
# owner: POVB+User(1124)
# group: POVB+Group(513)
user::---
group::---
group:SYSTEM:rwx
group:Administrators:rwx
group:VLADOB+Own:rwx
mask:rwx
other:---

$ perl -e "if( -e 
'Cygwin/http%3A%2F%2Fgd.tuwien.ac.at%2Fgnu%2Fcygwin%2F/x86/setup.ini'){print 
'exists'}else{print 'doesnt exists'}"
exists

(Windows Command prompt)
Cygwin\http%3a%2f%2fgd.tuwien.ac.at%2fgnu%2fcygwin%2f> icacls x86\setup.ini
x86\setup.ini NT AUTHORITY\SYSTEM:(I)(F)
               BUILTIN\Administrators:(I)(F)
               VLADOB\Own:(I)(F)

Tail of sub find_setup_ini_file will be:
	open my $uname, '-|', 'uname -m' or die "uname -m failed: $!\n";
	my $plat = <$uname>;
	chomp $plat;
	close $uname;

	my $setup_path = "$path/$mirror/$plat/setup.ini";
	return $setup_path if -e $setup_path;
	
	# If 32-bit Cygwin on 64-bit OS, the above doesn't works
	$setup_path = "$path/$mirror/x86/setup.ini";
	return $setup_path if -e $setup_path;
	
	# Definitely not found
	usage("could not find setup.ini");
}

This works for me.

Vlado

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Anecdotal: Rebase and Visual Studio 2015 and /etc

[Warren Young]

On Jul 5, 2016, at 6:11 AM, KARL BOTTS <kdbotts@usa.net> wrote:
> 
> I have been using 32-bit Cygwin for at least 15
> years, and being without it throws me into a tizzy.

64-bit Cygwin installs in parallel to 32-bit Cygwin, not over the top of it.  If your 64-bit adventure is a complete failure, your 32-bit installation will still be there.

> Are there any gotchas I should know about?

I answered that here:

  http://stackoverflow.com/q/18329233

> Maybe eventually we can get
> your work up on the Cygwin web site?

It’s already there:

  https://cygwin.com/ml/cygwin/2016-07/msg00025.html
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Anecdotal: Rebase and Visual Studio 2015 and /etc

[Achim Gratz]

KARL BOTTS <kdbotts <at> usa.net> writes:
> I am just a little nervous.  I have been using 32-bit Cygwin for at least 15
> years, and being without it throws me into a tizzy.  (I barely know how to use
> WindowsExplorer.)  Are there any gotchas I should know about?

Here's another way to do it.

http://permalink.gmane.org/gmane.os.cygwin/154766


Regards,
Achim.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Anecdotal: Rebase and Visual Studio 2015 and /etc

[KARL BOTTS]

Karl Botts>> I use Cygwin32 on Windows-64.

Warren Young> Then you’re artificially making rebase’s job harder.
> The list of 32-bit-only Cygwin packages is tiny these days, and you’ve
just
> rebuilt your Cygwin environment. With my new find-cyg-roots script, you
could
> rebuild your current 32-bit environment under Cygwin 64 quickly. 
> That should prevent the reoccurrence of the rebase problem.


Your recommendation makes total sense.  I will do it soon: not today, but
soon.

I am just a little nervous.  I have been using 32-bit Cygwin for at least 15
years, and being without it throws me into a tizzy.  (I barely know how to use
WindowsExplorer.)  Are there any gotchas I should know about?


Please keep us up to date with your valuable scripting efforts.  I have
checked your script, emails and the followups into revision control, for my
own purposes.  I'll let you know how it goes.  Maybe eventually we can get
your work up on the Cygwin web site?


---
Karl Botts, kdbotts@usa.net


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Anecdotal: Rebase and Visual Studio 2015 and /etc

[Warren Young]

On Jul 1, 2016, at 4:12 PM, KARL BOTTS <kdbotts@usa.net> wrote:
> 
> I use Cygwin32 on Windows-64.

Then you’re artificially making rebase’s job harder.

The list of 32-bit-only Cygwin packages is tiny these days, and you’ve just rebuilt your Cygwin environment.  With my new find-cyg-roots script, you could rebuild your current 32-bit environment under Cygwin 64 quickly.  

That should prevent the reoccurrence of the rebase problem.

> Windows itself also
> uses lots of 32-bit components even under Win-64.  In fact, VS itself is
> a (very large) 32-bit app.

Both for legacy reasons, neither of which apply to Cygwin.

> 32-bit software runs so smoothly under an opsys running on
> Intel64, is one of the latter's best features.

64-bit software runs on 64-bit CPUs pretty well, too. :)

>> Because Satya Nadella has been in charge for only about two years now.
> 
> You think he's in charge?

Microsoft under Nadella feels a whole lot different than Microsoft under Ballmer to me.  Sure he’s steering a large ship, but he *is* moving it.
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Anecdotal: Rebase and Visual Studio 2015 and /etc

[KARL BOTTS]

> A VS installation shouldn’t affect the rebase setup of Cygwin.

I'm with you.  But it did.  I am not blaming Cygwin; I am a friend of Cygwin.


>> Eventually, I reinstalled a fresh cygwin

> Did you install all the same things, or a minimal install,

Most of the same packages, but a from-scratch install.  And it is a pretty
light Cygwin: no gcc, no X, no Apache, no databases, no LaTeX...


>> 1) After you do the full rebase, before you even start anything cygwin,
reboot
>> Windows, then start a bash or something. Reboot Windows, early and often,
>> whenever upgrading anything.

> I’ve never had to resort to such voodoo to keep Cygwin going. The
standard
> schedule of reboots due to Patch Tuesday has been sufficient.

I resort to such voodoo to keep Windows going.  Windows is much better than
it used to be, but I still say: whenever you feel nervous about the state of
Windows, reboot it.  You will feel better, even if it doesn't.
Voodoo is not always sufficient: silver crosses and garlic help, too.


BTW: I use Cygwin32 on Windows-64.  I'm happy with that.  Windows itself also
uses lots of 32-bit components even under Win-64.  In fact, VS itself is
a (very large) 32-bit app.  There are good reasons for that, for them and for
us.
Basically, that 32-bit software runs so smoothly under an opsys running on
Intel64,
is one of the latter's best features.


> Cygwin should never cause a Windows box to become unbootable. It simply
> doesn’t get its hooks into the system deeply enough to cause such
trouble,

I concur wholeheartedly: Cygwin is delightfully non-intrusive, given what it
does.
I have never seen Cygwin hose Windows: only the reverse.
MS should take a lesson.  In fact, I think they are trying: they claim that
the
next VS will be "XCOPY deployable", which means what sane software, such as
Cygwin,
has always been.  Mostly, it means they have to rip out all the dependencies
on
the goddamn Registry.  Which will require many thousand kiddy-coder-hours,
methinks.


>> And, it moves DLLs around, even installs more, on the way back up.

> “It” being Windows, or Cygwin?
> If the former, new Windows DLLs shouldn’t interfere with Cygwin DLLs,
unless
> by some wild coincidence there is an overlap in memory load spaces.

"It" being Windows.  That is why you get the screen after
install-stuff-then-reboot,
"Configuring windows, please don't turn off your machine", and so forth.
Among other things, they are delay-replacing DLLs (for which there are
special
Windows syscalls).  Also, they "rebase" lots of DLLs, notably .NET.Framework
parts, to try to optimize load time, by avoiding runtime fixups: they don't
really use PIC code, if at all.  Given all that, it does not seem to me that
some load-address conflicts would be a "wild coincidence".


> Because Satya Nadella has been in charge for only about two years now.

You think he's in charge?  Ha-ha, I bet so did he, when he took the job.
But he is trying to change a corporate culture, without breaking too many
eggs.
Good luck to him.  Heroin might help.
 

---
Karl Botts, kdbotts@usa.net


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple