setup.ini dependency graph?

setup.ini dependency graph?

[Charles Wilson]

Does anybody have a script or a tool that can parse a setup.ini and 
generate a dependency graph?  I'm using pmcyg to create a stripped-down 
standalone installation CD and it's too big, so I'm trying to figure out 
where the culprit is that's pulling in so much stuff...

--
Chuck

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: setup.ini dependency graph?

[Ryan Johnson]

On 25/10/2013 12:12 PM, Charles Wilson wrote:
> Does anybody have a script or a tool that can parse a setup.ini and 
> generate a dependency graph?  I'm using pmcyg to create a 
> stripped-down standalone installation CD and it's too big, so I'm 
> trying to figure out where the culprit is that's pulling in so much 
> stuff...
I threw one together a while back when dealing with packages trying to 
pull in latex: http://cygwin.com/ml/cygwin/2012-03/msg00242.html

It doesn't build a full graph, but if you name the offending package it 
tells you who depends on it.

HTH,
Ryan


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: setup.ini dependency graph?

[David Stacey]

[-- Attachment #1: Type: text/plain, Size: 1197 bytes --]

On 25/10/13 17:12, Charles Wilson wrote:
> Does anybody have a script or a tool that can parse a setup.ini and 
> generate a dependency graph?  I'm using pmcyg to create a 
> stripped-down standalone installation CD and it's too big, so I'm 
> trying to figure out where the culprit is that's pulling in so much 
> stuff... 

Oooo - this sounds like fun. I've knocked up some (very bad) perl that 
gives you what you need. It generates a graphviz file that you can pipe 
to 'dot' to generate the dependency graph in whatever format you 
require. Put the perl script and your 'setup.ini' file in the same 
directory and type:

     ./graph_setup_ini.pl | dot -Tpdf -osetup.pdf

I did this in Fedora 19; if you want this to work in Cygwin then you'll 
need to install graphviz from CygwinPorts.

Your problem here is Big Data: Cygwin has 3041 packages, and any 
dependency graph with this number of nodes is going to look a mess. It 
also takes a while to process the data. Oh, and some PDF viewers won't 
display the output file (LibreOffice Draw was the only tool I have that 
managed it). However, if your starting point is a stripped down Cygwin 
then you might be OK.

Hope this helps,

Dave.


[-- Attachment #2: graph_setup_ini.pl --]
[-- Type: application/x-perl, Size: 578 bytes --]

[-- Attachment #3: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

why do psutils & libtool packages need ~25+MB of latex (was Re: setup.ini dependency graph?)

[Linda Walsh]

On 10/25/2013 5:29 PM, Ryan Johnson wrote:
> On 25/10/2013 12:12 PM, Charles Wilson wrote:
>> Does anybody have a script or a tool that can parse a setup.ini and 
>> generate a dependency graph?  I'm using pmcyg to create a 
>> stripped-down standalone installation CD and it's too big, so I'm 
>> trying to figure out where the culprit is that's pulling in so much 
>> stuff...
> I threw one together a while back when dealing with packages trying to 
> pull in latex: http://cygwin.com/ml/cygwin/2012-03/msg00242.html
---

I keep trying to get rid of latex and it keeps coming back...

I think I figured out why... psutils?   libtool?

Both need latex?  I think I'm must really be missing some features in
psutils -- and doesn't libtool put together binary libraries?  Why would
it need latex in order to run?


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: why do psutils & libtool packages need ~25+MB of latex (was Re: setup.ini dependency graph?)

[Ken Brown]

On 10/27/2013 1:00 AM, Linda Walsh wrote:
> On 10/25/2013 5:29 PM, Ryan Johnson wrote:
>> On 25/10/2013 12:12 PM, Charles Wilson wrote:
>>> Does anybody have a script or a tool that can parse a setup.ini and
>>> generate a dependency graph?  I'm using pmcyg to create a
>>> stripped-down standalone installation CD and it's too big, so I'm
>>> trying to figure out where the culprit is that's pulling in so much
>>> stuff...
>> I threw one together a while back when dealing with packages trying to
>> pull in latex: http://cygwin.com/ml/cygwin/2012-03/msg00242.html
> ---
>
> I keep trying to get rid of latex and it keeps coming back...
>
> I think I figured out why... psutils?   libtool?

If you look at setup.ini, you'll see that psutils is an obsolete 
package, which has been replaced by texlive-collection-fontutils.  You 
must have once installed psutils for some reason.  If you don't need its 
functionality, uninstall it and all the texlive packages that it brought 
in.  You may have other obsolete packages installed that are also 
bringing in texlive packages.  Just run setup and uncheck the "Hide 
obsolete packages" box.

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: setup.ini dependency graph?

[Charles Wilson]

On 10/26/2013 5:40 AM, David Stacey wrote:
> On 25/10/13 17:12, Charles Wilson wrote:
> Oooo - this sounds like fun. I've knocked up some (very bad) perl that
> gives you what you need. It generates a graphviz file that you can pipe
> to 'dot' to generate the dependency graph in whatever format you
> require. Put the perl script and your 'setup.ini' file in the same
> directory and type:
>
>      ./graph_setup_ini.pl | dot -Tpdf -osetup.pdf

Thanks, that worked well.

> Your problem here is Big Data: Cygwin has 3041 packages, and any
> dependency graph with this number of nodes is going to look a mess. It
> also takes a while to process the data. Oh, and some PDF viewers won't
> display the output file (LibreOffice Draw was the only tool I have that
> managed it). However, if your starting point is a stripped down Cygwin
> then you might be OK.

Yeah; even for my stripped-down version, I need to pre-process the 
setup.ini and remove all mentions of cygwin, libstdc++6, libgcc1, etc. 
The ncurses DLLs are also a huge nexus.  (It's probably easier to 
exclude those nodes by mucking with the perl, but...)

--
Chuck



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: setup.ini dependency graph?

[Ryan Johnson]

On 30/10/2013 8:48 AM, Charles Wilson wrote:
> On 10/26/2013 5:40 AM, David Stacey wrote:
>> On 25/10/13 17:12, Charles Wilson wrote:
>> Oooo - this sounds like fun. I've knocked up some (very bad) perl that
>> gives you what you need. It generates a graphviz file that you can pipe
>> to 'dot' to generate the dependency graph in whatever format you
>> require. Put the perl script and your 'setup.ini' file in the same
>> directory and type:
>>
>>      ./graph_setup_ini.pl | dot -Tpdf -osetup.pdf
>
> Thanks, that worked well.
>
>> Your problem here is Big Data: Cygwin has 3041 packages, and any
>> dependency graph with this number of nodes is going to look a mess. It
>> also takes a while to process the data. Oh, and some PDF viewers won't
>> display the output file (LibreOffice Draw was the only tool I have that
>> managed it). However, if your starting point is a stripped down Cygwin
>> then you might be OK.
>
> Yeah; even for my stripped-down version, I need to pre-process the 
> setup.ini and remove all mentions of cygwin, libstdc++6, libgcc1, etc. 
> The ncurses DLLs are also a huge nexus.  (It's probably easier to 
> exclude those nodes by mucking with the perl, but...)
Quick question: do you have 1+ known-big-unwanted packages and need to 
know who's pulling them in, or are you hoping to take some cut of the 
graph that gets as many desirable packages as possible given the space 
constraints? The graph-building script here is good for the latter, but 
I had the impression you were doing the former; if so, my script might 
get you to an answer faster by avoiding information overload.

Ryan


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: setup.ini dependency graph?

[Charles Wilson]

On 10/30/2013 9:51 AM, Ryan Johnson wrote:
> On 30/10/2013 8:48 AM, Charles Wilson wrote:
>> Yeah; even for my stripped-down version, I need to pre-process the
>> setup.ini and remove all mentions of cygwin, libstdc++6, libgcc1, etc.
>> The ncurses DLLs are also a huge nexus.  (It's probably easier to
>> exclude those nodes by mucking with the perl, but...)
> Quick question: do you have 1+ known-big-unwanted packages and need to
> know who's pulling them in, or are you hoping to take some cut of the
> graph that gets as many desirable packages as possible given the space
> constraints? The graph-building script here is good for the latter, but
> I had the impression you were doing the former; if so, my script might
> get you to an answer faster by avoiding information overload.

A combination of the two, actually.  I've used both David's script and 
yours in concert.  In addition, I've modified David's script to color 
the nodes based on origination, and to exclude or collapse 'Base' and/or 
'required-by-Base' packages.

I've got a few cleanups, and then I'll share the result.  It's already 
helped me generate a few re-packaging requests I plan to post over on 
cygwin-apps...

--
Chuck



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: setup.ini dependency graph?

[Christopher Faylor]

On Mon, Nov 04, 2013 at 09:32:10AM -0500, Charles Wilson wrote:
>On 10/30/2013 9:51 AM, Ryan Johnson wrote:
>> On 30/10/2013 8:48 AM, Charles Wilson wrote:
>>> Yeah; even for my stripped-down version, I need to pre-process the
>>> setup.ini and remove all mentions of cygwin, libstdc++6, libgcc1, etc.
>>> The ncurses DLLs are also a huge nexus.  (It's probably easier to
>>> exclude those nodes by mucking with the perl, but...)
>> Quick question: do you have 1+ known-big-unwanted packages and need to
>> know who's pulling them in, or are you hoping to take some cut of the
>> graph that gets as many desirable packages as possible given the space
>> constraints? The graph-building script here is good for the latter, but
>> I had the impression you were doing the former; if so, my script might
>> get you to an answer faster by avoiding information overload.
>
>A combination of the two, actually.  I've used both David's script and 
>yours in concert.  In addition, I've modified David's script to color 
>the nodes based on origination, and to exclude or collapse 'Base' and/or 
>'required-by-Base' packages.
>
>I've got a few cleanups, and then I'll share the result.  It's already 
>helped me generate a few re-packaging requests I plan to post over on 
>cygwin-apps...

Is this packagable?  It sounds pretty interesting.

Would it be crazy to generate this and make it available on the cygwin
web site?  Or would the dependency graph generation overload
sourceware.org?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: setup.ini dependency graph?

[Ryan Johnson]

On 04/11/2013 11:00 AM, Christopher Faylor wrote:
> On Mon, Nov 04, 2013 at 09:32:10AM -0500, Charles Wilson wrote:
>> On 10/30/2013 9:51 AM, Ryan Johnson wrote:
>>> On 30/10/2013 8:48 AM, Charles Wilson wrote:
>>>> Yeah; even for my stripped-down version, I need to pre-process the
>>>> setup.ini and remove all mentions of cygwin, libstdc++6, libgcc1, etc.
>>>> The ncurses DLLs are also a huge nexus.  (It's probably easier to
>>>> exclude those nodes by mucking with the perl, but...)
>>> Quick question: do you have 1+ known-big-unwanted packages and need to
>>> know who's pulling them in, or are you hoping to take some cut of the
>>> graph that gets as many desirable packages as possible given the space
>>> constraints? The graph-building script here is good for the latter, but
>>> I had the impression you were doing the former; if so, my script might
>>> get you to an answer faster by avoiding information overload.
>> A combination of the two, actually.  I've used both David's script and
>> yours in concert.  In addition, I've modified David's script to color
>> the nodes based on origination, and to exclude or collapse 'Base' and/or
>> 'required-by-Base' packages.
>>
>> I've got a few cleanups, and then I'll share the result.  It's already
>> helped me generate a few re-packaging requests I plan to post over on
>> cygwin-apps...
> Is this packagable?  It sounds pretty interesting.
>
> Would it be crazy to generate this and make it available on the cygwin
> web site?  Or would the dependency graph generation overload
> sourceware.org?
Throwing graphviz at a full cygwin package dependency graph would make a 
pretty effective DoS attack. Smaller graphs are cheaper, but still 
consume non-trivial compute. Given how slowly the online regexp package 
search goes, I'd hesitate to give users more ways to overload the server...

What about calling out to graphviz from setup.exe (if found in %PATH%), 
as a replacement/supplement for the flat list of dependencies it 
currently reports? That would put all processing on the client, and 
limit the "big data" problem, as the graph only contains packages a user 
is currently trying to install.

Alternatively, cygcheck could gain a new -g option to dump subsets of a 
dependency graph, extracted from setup.ini, in some appropriate format 
like .dot:

`cygcheck -D -g python' would emit the graph of packages that python 
depends on
`cygcheck -R -g texlive,xorg-server' would build a braph of packages 
that pull in either of texlive or xorg-server (reverse dependencies)
-D -R -g would follow dependencies in both directions, and -g would be 
shorthand for -D -g; probably -D or -R by itself implies -g.

The actual work could be done by calling out to a scripting language 
that ships with cygwin. Awk would probably be able to, and perl 
certainly could.

/daydreaming

Ryan



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: setup.ini dependency graph?

[Charles Wilson]

[-- Attachment #1: Type: text/plain, Size: 5739 bytes --]

On 11/4/2013 11:00 AM, Christopher Faylor wrote:
> On Mon, Nov 04, 2013 at 09:32:10AM -0500, Charles Wilson wrote:
>> I've got a few cleanups, and then I'll share the result.  It's already
>> helped me generate a few re-packaging requests I plan to post over on
>> cygwin-apps...
>
> Is this packagable?  It sounds pretty interesting.

Probably. I could put it in cygutils, or standalone (like the new 
cygcheck-leaves package is a standalone utility).

> Would it be crazy to generate this and make it available on the cygwin
> web site?  Or would the dependency graph generation overload
> sourceware.org?

The basic processing to generate a .dot file is pretty simple really; 
just string comparisons and hash manipulations. But as Ryan already 
pointed out, generating the actual graph in whatever format, is probably 
compute intensive.

See attached...

The svg files in the tarball correspond to each of the --mode options. 
The tool was invoked as:
    setup-ini-graph -i setup.ini -o $dotfile --mode=$mode netpbm
followed by (on linux)
    dot -Tsvg $dotfile > $svgfile

Help output:
> Usage:
>     setup_ini_graph [options] [packagename [packagename ...]]
>
>      Options:
>        --help|-?                brief help message
>        --man                    full documentation
>        --version                show version information
>        --mode STRING            specify the output mode. Possible values:
>                                   strip-base-and-requirements <default>
>                                   strip-base-only
>                                   collapse-base-and-requirements
>                                   collapse-base-only
>                                   show-only-base-and-requirements
>                                   show-base-requirements-only
>                                   show-all
>        --color STRING           set color of specified packages <lightblue>
>        --color-req STRING       set color of package requirements <palegreen>
>        --color-base STRING      set color of Base packages <plum>
>        --color-base-req STRING  set color of Base requirements <pink>
>        --input|-i FILE          specify input setup.ini file <STDIN>
>        --output|-o FILE         specify output .dot file <STDOUT>
>        --verbose                turn on debugging output
>
>     If packagenames are specified, then only the specified packages and
>     their dependencies are analyzed. If no packages are specified, then all
>     packages in the setup.ini are analyzed.
>
> Options:
>     --help  Print a brief help message and exits.
>
>     --man   Prints the manual page and exits.
>
>     --version
>             Prints version information and exits.
>
>     --mode STRING
>             Specify the output mode. The STRING may be one of the following
>             options:
>
>             strip-base-and-requirements
>                 All packages in the Base category, and packages on which
>                 they depend, are completely removed from the output. This is
>                 the default mode.
>
>             strip-base-only
>                 All packages in the Base category are completely removed
>                 from the output.
>
>             collapse-base-and-requirements
>                 All packages in the Base category, and packages on which
>                 they depend, are replaced by a single node 'Base'.
>
>             collapse-base-only
>                 All packages in the Base category are replaced by a single
>                 node 'Base'.
>
>             show-only-base-and-requirements
>                 Ignore all packages that are not in the Base category, or
>                 are not required by packages in the Base category.
>
>             show-base-requirements-only
>                 As above, but the packages which are members of the Base
>                 category are replaced by a single node 'Base'.
>
>             show-all
>                 Chart the entire dependency tree.
>
>     --color STRING
>             Sets the color to be used for nodes that represent the packages
>             listed on the command line (or all packages extracted from
>             setup.ini that do not fall into one of the other categories
>             below). May be an RGB color name such as palegoldenrod, or an
>             RGB triple of the form #DA70D6. Note that in the latter case,
>             the # must be escaped as \#DA70D6 or '#DA70D6'. The default
>             value is lightblue (#ADD8E6).
>
>     --color-req STRING
>             Sets the color to be used for nodes that represent packages that
>             are required by the ones listed on the command line, but which
>             do not fall into one of the other categories below. The default
>             value is palegreen (#98FB98). See --color for more information.
>
>     --color-base STRING
>             Sets the color to be used for nodes that represent packages that
>             are in the Base category. It is also used to color the 'Base'
>             node in the collapse --mode options. The default value is plum
>             (#DDA0DD). See --color for more information.
>
>     --color-base-req STRING
>             Sets the color to be used for nodes that represent packages
>             which are required by packages in the Base category, but are not
>             in the Base category themselves. The default value is pink
>             (#FFC0CB).
>
>     --input FILENAME
>             Use the specified input file. Defaults to <STDIN>.
>
>     --output FILENAME
>             Use the specified output file. Defaults to <STDOUT>.
>
>     --verbose
>             Turn on verbose output (to STDERR).

--
Chuck


[-- Attachment #2: netpbm-dependency-chart-svg.tar.xz --]
[-- Type: application/octet-stream, Size: 74472 bytes --]

[-- Attachment #3: setup-ini-graph --]
[-- Type: text/plain, Size: 21798 bytes --]

#!/usr/bin/perl -w
# -*- cperl -*-
# setup-ini-graph - Generate a graph of cygwin dependencies culled
#                   from setup.hint
# Copyright (C) 2013 Charles Wilson, David Stacey

# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

use strict;
use version;
use Getopt::Long qw(:config gnu_getopt auto_version);
use Pod::Usage;
use Storable;

$main::VERSION = version->new("0.9");
my $man = 0;
my $help = 0;
my $base_mode = \&strip_base_and_rqmnts;
my $opt_verbose = 0; # default value = false
my $base_color      = 'plum';
my $base_reqs_color = 'pink';
my $key_color       = 'lightblue';
my $key_reqs_color  = 'palegreen';
my $infile = '-';
my $IFH = \*STDIN;
my $outfile = '-';
my $OFH = \*STDOUT;
my %valid_mode = (
 'strip-base-and-requirements'     => \&strip_base_and_rqmnts,
 'strip-base-only'                 => \&strip_base_only,
 'collapse-base-and-requirements'  => \&collapse_base_and_rqmnts,
 'collapse-base-only'              => \&collapse_base_only,
 'show-only-base-and-requirements' => \&show_only_base_and_rqmnts,
 'show-base-requirements-only'     => \&show_base_rqmnts_only,
 'show-all'                        => \&show_all,
);

GetOptions ('help|?'         => \$help,
            'man'            => \$man,
            'mode=s'         => \&mode_handler,
            'color'          => \$key_color,
            'color-req'      => \$key_reqs_color,
            'color-base'     => \$base_color,
            'color-base-req' => \$base_reqs_color,
            'verbose+'       => \$opt_verbose,
            'input|i=s'      => \&input_handler,
            'output|o=s'     => \&output_handler) or pod2usage(2);
pod2usage(1) if $help;
pod2usage(-exitval => 0, -verbose => 2) if $man;


sub mode_handler {
  my ($opt_name, $opt_value) = @_;
  die "invalid value for $opt_name option: '$opt_value'" unless (exists($valid_mode{$opt_value}));
  $base_mode = $valid_mode{$opt_value};
}

sub input_handler {
  my ($opt_name, $opt_value) = @_;
  # if we've already opened a file for input (other than STDIN), close it
  if ($IFH and not $IFH == \*STDIN) {
    close $IFH;
  }
  if ($opt_value eq '-') {
    $IFH = \*STDIN;
    $infile=$opt_value;
  }
  elsif ($opt_value) {
    open $IFH, '<', $opt_value or die $!;
    $infile=$opt_value;
  }
}

sub output_handler {
  my ($opt_name, $opt_value) = @_;
  # if we've already opened a file for output (other than STDOUT), close it
  if ($OFH and not $OFH == \*STDOUT) {
    close $OFH;
  }
  if ($opt_value eq '-') {
    $OFH = \*STDOUT;
    $outfile=$opt_value;
  }
  elsif ($opt_value) {
    open $OFH, '>', $opt_value or die $!;
    $outfile=$opt_value;
  }
}

sub collect_key_pkgs {
  my ($KeyPkgsRef, $ArgvRef, $LinesRef) = @_;
  my $package_name = '';
  my $numArgs = scalar @{$ArgvRef};
  if ($numArgs == 0)
  {
    # parse all @ lines from setup.ini
    foreach my $line (@{$LinesRef})
    {
      $line =~ s/\R//g;
      if ($line =~ /^@ /)
      {
        # and add them all to KeyPkgs
        ($package_name = $line) =~ s/^@ //;
        $KeyPkgsRef->{$package_name} = ();
      }
    }
  }
  else
  {
    my %pkgArgs;
    @pkgArgs{@{$ArgvRef}} = ();

    # parse all @ lines from setup.ini
    foreach my $line (@{$LinesRef})
    {
      $line =~ s/\R//g;
      if ($line =~ /^@ /)
      {
        ($package_name = $line) =~ s/^@ //;

        # check if it is in %pkgArgs, and if so add it
        if (exists($pkgArgs{$package_name}))
        {
          $KeyPkgsRef->{$package_name} = ();
          # and remove from pkgArgs
          delete $pkgArgs{$package_name};
        }
      }
    }
    # Check that %pkgArgs is empty; if not, then the user
    # specified packages that do not exist
    if (not scalar keys %pkgArgs == 0)
    {
      my $errMsg = ','.join(keys(%pkgArgs));
      die 'Error: some packages specified do not exist in the setup.ini:\n\t$errMsg'
    }
  }
}

sub get_base {
  my ($BaseRef, $LinesRef) = @_;
  my $package_name = '';
  foreach my $line (@{$LinesRef})
  {
    $line =~ s/\R//g;
    if ($line =~ /^@ /)
    {
      ($package_name = $line) =~ s/^@ //;
    }
    elsif ($line =~ /^category: /)
    {
      (my $categories = $line) =~ s/^category: //;
      my %cats;
      my @keys = split(/ /, $categories);
      @cats{@keys} = ();
      if (exists($cats{'Base'}))
      {
        $BaseRef->{$package_name} = ();
      }
    }
  }
}

sub get_requires {
  my ($PlusReqsRef, $ReqsRef, $PkgSetRef, $LinesRef) = @_;
  %{$PlusReqsRef} = %{ Storable::dclone ($PkgSetRef) };
  my $package_name = '';
  my $oldsetsize = 0;
  my $newsetsize = scalar keys %{$PlusReqsRef};

  while ($oldsetsize != $newsetsize)
  {
    $oldsetsize = $newsetsize;
    foreach my $line (@{$LinesRef})
    {
      $line =~ s/\R//g;
      if ($line =~ /^@ /)
      {
        ($package_name = $line) =~ s/^@ //;
      }
      if (exists($PlusReqsRef->{$package_name}))
      {
        if ($line =~ /^requires: /)
        {
          (my $requires = $line) =~ s/^requires: //;
          my @rqts = split(/ /, $requires);
          foreach my $rqt (@rqts)
          {
            if (not exists($PlusReqsRef->{$rqt}))
            {
              $PlusReqsRef->{$rqt} = ();
            }
          }
        }
      }
    }
    $newsetsize = scalar keys %{$PlusReqsRef};
  }
  %{$ReqsRef} = %{ Storable::dclone ($PlusReqsRef) };
  foreach my $key (keys %{$PkgSetRef})
  {
    delete $ReqsRef->{$key};
  }
}


sub strip_base_and_rqmnts {
  my ($NodeOrEdgeMode,
	  $BaseRef, $BasePlusReqsRef, $BaseReqsRef,
	  $KeyPkgsRef, $KeyPkgsPlusReqsRef, $KeyPkgsReqsRef,
	  $HasBaseRqmntRef, $PackageName, $line) = @_;
  if ($NodeOrEdgeMode == 0)
  {
	# color keypkg nodes light goldenrod yellow
	color_nodes ($KeyPkgsRef, $key_color);
	# color nodes required by keypkgs
	# color_nodes ($KeyPkgsReqsRef, $key_reqs_color);
  }
  else
  {
	# if it is not in our "universe", skip it
	return if (not exists($KeyPkgsPlusReqsRef->{$PackageName}) and
		       not exists($BasePlusReqsRef->{$PackageName}));

    # don't do anything for packages in Base or packages required
    # by Base.
    return if (exists($BasePlusReqsRef->{$PackageName}));

    (my $requires = $line) =~ s/^requires: //;
    my @rqts = split(/ /, $requires);
    foreach my $rqt (@rqts)
    {
      if (not exists($BasePlusReqsRef->{$rqt}))
      {
        print $OFH "\t\"$PackageName\" -> \"$rqt\";\n";
      }
    }
  }
}


sub strip_base_only {
  my ($NodeOrEdgeMode,
	  $BaseRef, $BasePlusReqsRef, $BaseReqsRef,
	  $KeyPkgsRef, $KeyPkgsPlusReqsRef, $KeyPkgsReqsRef,
	  $HasBaseRqmntRef, $PackageName, $line) = @_;
  if ($NodeOrEdgeMode == 0)
  {
	# color nodes required by base powderblue
	color_nodes ($BaseReqsRef, $base_reqs_color);
	# color keypkg nodes light goldenrod yellow
	color_nodes ($KeyPkgsRef, $key_color);
	# color nodes required by keypkgs
	# color_nodes ($KeyPkgsReqsRef, $key_reqs_color);
  }
  else
  {
	# if it is not in our "universe", skip it
	return if (not exists($KeyPkgsPlusReqsRef->{$PackageName}) and
		       not exists($BasePlusReqsRef->{$PackageName}));

    # don't do anything for packages in Base
    return if (exists($BaseRef->{$PackageName}));

    (my $requires = $line) =~ s/^requires: //;
    my @rqts = split(/ /, $requires);
    foreach my $rqt (@rqts)
    {
      if (not exists($BaseRef->{$rqt}))
      {
        print $OFH "\t\"$PackageName\" -> \"$rqt\";\n";
      }
    }
  }
}


sub collapse_base_and_rqmnts {
  my ($NodeOrEdgeMode,
	  $BaseRef, $BasePlusReqsRef, $BaseReqsRef,
	  $KeyPkgsRef, $KeyPkgsPlusReqsRef, $KeyPkgsReqsRef,
	  $HasBaseRqmntRef, $PackageName, $line) = @_;
  if ($NodeOrEdgeMode == 0)
  {
	my %CollapsedBase;
	$CollapsedBase{'Base'} = ();
	# color 'Base' node
	color_nodes (\%CollapsedBase, $base_color);
	# color keypkg nodes
	color_nodes ($KeyPkgsRef, $key_color);
	# color nodes required by keypkgs
	# color_nodes ($KeyPkgsReqsRef, $key_reqs_color);
  }
  else
  {
	# if it is not in our "universe", skip it
	return if (not exists($KeyPkgsPlusReqsRef->{$PackageName}) and
		       not exists($BasePlusReqsRef->{$PackageName}));

    # don't do anything for packages in Base or packages required
    # by Base.
    return if (exists($BasePlusReqsRef->{$PackageName}));

    (my $requires = $line) =~ s/^requires: //;
    my @rqts = split(/ /, $requires);
    foreach my $rqt (@rqts)
    {
      if (exists($BasePlusReqsRef->{$rqt}))
      {
        # This requirement is one of the packages in Base, or is
        # one of the requirements OF one of the packages in Base.
        # Replace it with a requirement labeled "Base" but ensure
        # we only have one such for each.
        if (not $$HasBaseRqmntRef)
        {
          print $OFH "\t\"$PackageName\" -> \"Base\";\n";
          $$HasBaseRqmntRef = 1;
        }
      }
      else
      {
        print $OFH "\t\"$PackageName\" -> \"$rqt\";\n";
      }
    }
  }
}


sub collapse_base_only {
  my ($NodeOrEdgeMode,
	  $BaseRef, $BasePlusReqsRef, $BaseReqsRef,
	  $KeyPkgsRef, $KeyPkgsPlusReqsRef, $KeyPkgsReqsRef,
	  $HasBaseRqmntRef, $PackageName, $line) = @_;
  if ($NodeOrEdgeMode == 0)
  {
	my %CollapsedBase;
	$CollapsedBase{'Base'} = ();
	# color 'Base' node
	color_nodes (\%CollapsedBase, $base_color);
	# color nodes required by base
	color_nodes ($BaseReqsRef, $base_reqs_color);
	# color keypkg nodes
	color_nodes ($KeyPkgsRef, $key_color);
	# color nodes required by keypkgs
	# color_nodes ($KeyPkgsReqsRef, $key_reqs_color);
  }
  else
  {
	# if it is not in our "universe", skip it
	return if (not exists($KeyPkgsPlusReqsRef->{$PackageName}) and
		       not exists($BasePlusReqsRef->{$PackageName}));

    # don't do anything for packages in Base
    return if (exists($BaseRef->{$PackageName}));

    (my $requires = $line) =~ s/^requires: //;
    my @rqts = split(/ /, $requires);
    foreach my $rqt (@rqts)
    {
      if (exists($BaseRef->{$rqt}))
      {
        # This requirement is one of the packages in Base
        # Replace it with a requirement labeled "Base" but ensure
        # we only have one such for each.
        if (not $$HasBaseRqmntRef)
        {
          print $OFH "\t\"$PackageName\" -> \"Base\";\n";
          $$HasBaseRqmntRef = 1;
        }
      }
      else
      {
        print $OFH "\t\"$PackageName\" -> \"$rqt\";\n";
      }
    }
  }
}


sub show_only_base_and_rqmnts {
  my ($NodeOrEdgeMode,
	  $BaseRef, $BasePlusReqsRef, $BaseReqsRef,
	  $KeyPkgsRef, $KeyPkgsPlusReqsRef, $KeyPkgsReqsRef,
	  $HasBaseRqmntRef, $PackageName, $line) = @_;
  if ($NodeOrEdgeMode == 0)
  {
	# color base nodes
	color_nodes ($BaseRef, $base_color);
	# color nodes required by base
	color_nodes ($BaseReqsRef, $base_reqs_color);
  }
  else
  {
	# if it is not in our "universe", skip it
	return if (not exists($KeyPkgsPlusReqsRef->{$PackageName}) and
		       not exists($BasePlusReqsRef->{$PackageName}));

    # don't do anything for packages that are not either (a) in
    # Base or (b) required by a package in Base.
    return if (not exists($BasePlusReqsRef->{$PackageName}));

    (my $requires = $line) =~ s/^requires: //;
    my @rqts = split(/ /, $requires);
    foreach my $rqt (@rqts)
    {
      print $OFH "\t\"$PackageName\" -> \"$rqt\";\n";
    }
  }
}


sub show_base_rqmnts_only {
  my ($NodeOrEdgeMode,
	  $BaseRef, $BasePlusReqsRef, $BaseReqsRef,
	  $KeyPkgsRef, $KeyPkgsPlusReqsRef, $KeyPkgsReqsRef,
	  $HasBaseRqmntRef, $PackageName, $line) = @_;
  if ($NodeOrEdgeMode == 0)
  {
	my %CollapsedBase;
	$CollapsedBase{'Base'} = ();
	# color 'Base' node
	color_nodes (\%CollapsedBase, $base_color);
	# color nodes required by base
	color_nodes ($BaseReqsRef, $base_reqs_color);
  }
  else
  {
	# if it is not in our "universe", skip it
	return if (not exists($KeyPkgsPlusReqsRef->{$PackageName}) and
		       not exists($BasePlusReqsRef->{$PackageName}));

    # don't do anything for packages that are not a requirement
    # of a package in Base.
    return if (not exists($BaseReqsRef->{$PackageName}));

    (my $requires = $line) =~ s/^requires: //;
    my @rqts = split(/ /, $requires);
    foreach my $rqt (@rqts)
    {
      if (exists($BaseRef->{$rqt}))
      {
        # This requirement is one of the packages in Base
        # Replace it with a requirement labeled "Base" but ensure
        # we only have one such for each.
        if (not $$HasBaseRqmntRef)
        {
          print $OFH "\t\"$PackageName\" -> \"Base\";\n";
          $$HasBaseRqmntRef = 1;
        }
      }
      else
      {
        print $OFH "\t\"$PackageName\" -> \"$rqt\";\n";
      }
    }
  }
}


sub show_all {
  my ($NodeOrEdgeMode,
	  $BaseRef, $BasePlusReqsRef, $BaseReqsRef,
	  $KeyPkgsRef, $KeyPkgsPlusReqsRef, $KeyPkgsReqsRef,
	  $HasBaseRqmntRef, $PackageName, $line) = @_;
  if ($NodeOrEdgeMode == 0)
  {
	# color base nodes skyblue
	color_nodes ($BaseRef, $base_color);
	# color nodes required by base
	color_nodes ($BaseReqsRef, $base_reqs_color);
	# color keypkg nodes
	color_nodes ($KeyPkgsRef, $key_color);
	# color nodes required by keypkgs
	color_nodes ($KeyPkgsReqsRef, $key_reqs_color);
  }
  else
  {
	# if it is not in our "universe", skip it
	return if (not exists($KeyPkgsPlusReqsRef->{$PackageName}) and
		       not exists($BasePlusReqsRef->{$PackageName}));

    (my $requires = $line) =~ s/^requires: //;
    my @rqts = split(/ /, $requires);
    foreach my $rqt (@rqts)
    {
      print $OFH "\t\"$PackageName\" -> \"$rqt\";\n";
    }
  }
}

sub color_nodes {
  my ($PkgsRef, $ColorName) = @_;
  print $OFH "\tnode [fillcolor=$ColorName];\n";
  foreach my $pkg (keys %{$PkgsRef})
  {
    print $OFH "\t\"$pkg\";\n";
  }
  print $OFH "\tnode [fillcolor=$key_reqs_color];\n";
}

sub generate_output {
  my ($BaseRef, $BasePlusReqsRef, $BaseReqsRef,
	  $KeyPkgsRef, $KeyPkgsPlusReqsRef, $KeyPkgsReqsRef,
	  $LinesRef) = @_;
  my $package_name = '';
  my $hasBaseRqmnt = 0;

  print $OFH "digraph cygwin_setup_ini {\n";
  print $OFH "\tnode [shape = ellipse, style=filled, fillcolor=$key_reqs_color];\n";
  $base_mode->(0,
               $BaseRef, $BasePlusReqsRef, $BaseReqsRef,
               $KeyPkgsRef, $KeyPkgsPlusReqsRef, $KeyPkgsReqsRef,
               \$hasBaseRqmnt, '', '');
  foreach my $line (@{$LinesRef})
  {
    $hasBaseRqmnt = 0;
    $line =~ s/\R//g;
    if ($line =~ /^@ /)
    {
      ($package_name = $line) =~ s/^@ //;
      $hasBaseRqmnt = 0;
    }
    elsif ($line =~ /^requires: /)
    {
      $base_mode->(1, # draw edges mode
                   $BaseRef, $BasePlusReqsRef, $BaseReqsRef,
                   $KeyPkgsRef, $KeyPkgsPlusReqsRef, $KeyPkgsReqsRef,
                   \$hasBaseRqmnt, $package_name, $line);
    }
  }
  print $OFH "}\n";
}

my @lines = <$IFH>;
close $IFH if (not $IFH == \*STDIN);


# first, determine the packages we're requesting in the
# graph.  If not specified in @ARGV, use all packages found
# in the setup.ini.
my %KeyPkgs = ();
my %KeyPkgsPlusReqs = ();
my %KeyPkgsReqs = ();
collect_key_pkgs (\%KeyPkgs, \@ARGV, \@lines);

# Now, collect (recursively) all the "requires" of these items
get_requires (\%KeyPkgsPlusReqs, \%KeyPkgsReqs, \%KeyPkgs, \@lines);

if ($opt_verbose)
{
  my $setsize = scalar keys %KeyPkgs;
  my @lbase = sort(keys(%KeyPkgs));
  print STDERR "KeyPkgs($setsize) = ",join(" ",@lbase),"\n";
  $setsize = scalar keys %KeyPkgsPlusReqs;
  @lbase = sort(keys(%KeyPkgsPlusReqs));
  print STDERR "KeyPkgsPlusReqs($setsize) = ",join(" ",@lbase),"\n";
  $setsize = scalar keys %KeyPkgsReqs;
  @lbase = sort(keys(%KeyPkgsReqs));
  print STDERR "KeyPkgsReqs($setsize) = ",join(" ",@lbase),"\n";
}

# then build database of Base elements
my %Base = ();
my %BasePlusReqs = ();
my %BaseReqs = ();
get_base (\%Base, \@lines);

# And collect (recursively) all the "requires" of Base.
get_requires (\%BasePlusReqs, \%BaseReqs, \%Base, \@lines);

if ($opt_verbose)
{
  my $setsize = scalar keys %Base;
  my @lbase = sort(keys(%Base));
  print STDERR "Base($setsize) = ",join(" ",@lbase),"\n";
  $setsize = scalar keys %BasePlusReqs;
  @lbase = sort(keys(%BasePlusReqs));
  print STDERR "BasePlusReqs($setsize) = ",join(" ",@lbase),"\n";
  $setsize = scalar keys %BaseReqs;
  @lbase = sort(keys(%BaseReqs));
  print STDERR "BaseReqs($setsize) = ",join(" ",@lbase),"\n";
}

generate_output (\%Base, \%BasePlusReqs, \%BaseReqs,
                 \%KeyPkgs, \%KeyPkgsPlusReqs, \%KeyPkgsReqs,
                 \@lines);
close $OFH if (not $OFH == \*STDOUT);

__END__

=head1 NAME

setup_ini_graph - Chart cygwin package dependencies

=head1 SYNOPSIS

setup_ini_graph [options] [packagename [packagename ...]]

 Options:
   --help|-?                brief help message
   --man                    full documentation
   --version                show version information
   --mode STRING            specify the output mode. Possible values:
                              strip-base-and-requirements <default>
                              strip-base-only
                              collapse-base-and-requirements
                              collapse-base-only
                              show-only-base-and-requirements
                              show-base-requirements-only
                              show-all
   --color STRING           set color of specified packages <lightblue>
   --color-req STRING       set color of package requirements <palegreen>
   --color-base STRING      set color of Base packages <plum>
   --color-base-req STRING  set color of Base requirements <pink>
   --input|-i FILE          specify input setup.ini file <STDIN>
   --output|-o FILE         specify output .dot file <STDOUT>
   --verbose                turn on debugging output

If packagenames are specified, then only the specified packages and
their dependencies are analyzed. If no packages are specified, then
all packages in the setup.ini are analyzed.

=head1 OPTIONS

=over 8

=item B<--help>

Print a brief help message and exits.

=item B<--man>

Prints the manual page and exits.

=item B<--version>

Prints version information and exits.

=item B<--mode STRING>

Specify the output mode. The STRING may be one of the following options:

=over

=item B<strip-base-and-requirements>

All packages in the Base category, and packages on which they depend,
are completely removed from the output. This is the default mode.

=item B<strip-base-only>

All packages in the Base category are completely removed from the output.

=item B<collapse-base-and-requirements>

All packages in the Base category, and packages on which they depend, are
replaced by a single node 'Base'.

=item B<collapse-base-only>

All packages in the Base category are replaced by a single node 'Base'.

=item B<show-only-base-and-requirements>

Ignore all packages that are not in the Base category, or are not required
by packages in the Base category.

=item B<show-base-requirements-only>

As above, but the packages which are members of the Base category are
replaced by a single node 'Base'.

=item B<show-all>

Chart the entire dependency tree.

=back

=item B<--color STRING>

Sets the color to be used for nodes that represent the
packages listed on the command line (or all packages
extracted from setup.ini that do not fall into one of
the other categories below).  May be an RGB color name
such as palegoldenrod, or an RGB triple of the form
#DA70D6. Note that in the latter case, the # must be
escaped as \#DA70D6 or '#DA70D6'.  The default value
is lightblue (#ADD8E6).

=item B<--color-req STRING>

Sets the color to be used for nodes that represent packages
that are required by the ones listed on the command line,
but which do not fall into one of the other categories below.
The default value is palegreen (#98FB98). See L<--color> for
more information.

=item B<--color-base STRING>

Sets the color to be used for nodes that represent packages
that are in the Base category. It is also used to color the
'Base' node in the collapse L<--mode> options. The default
value is plum (#DDA0DD). See L<--color> for more information.

=item B<--color-base-req STRING>

Sets the color to be used for nodes that represent packages
which are required by packages in the Base category, but are
not in the Base category themselves. The default value is
pink (#FFC0CB).

=item B<--input FILENAME>

Use the specified input file. Defaults to <STDIN>.

=item B<--output FILENAME>

Use the specified output file. Defaults to <STDOUT>.

=item B<--verbose>

Turn on verbose output (to STDERR).

=back

=head1 DESCRIPTION

B<setup_ini_graph> will process a cygwin setup.ini file and generate a dependency
chart in dot format. Use C<dot -Tsvg output-dot-file E<gt> cygwin-deps.svg>
to process the result. See the C<dot> manpage for more information; note that
C<dot> is not available in stock cygwin, and must be installed from
L<cygwin-ports|http://sourceware.org/cygwinports/>.

=cut
# vim: set ff=unix ts=2 sw=2 et:


[-- Attachment #4: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: setup.ini dependency graph?

[David Stacey]

On 04/11/13 17:30, Charles Wilson wrote:
> On 11/4/2013 11:00 AM, Christopher Faylor wrote:
>> On Mon, Nov 04, 2013 at 09:32:10AM -0500, Charles Wilson wrote:
>>> I've got a few cleanups, and then I'll share the result.  It's already
>>> helped me generate a few re-packaging requests I plan to post over on
>>> cygwin-apps...
>>
>> Is this packagable?  It sounds pretty interesting.
>
> Probably. I could put it in cygutils, or standalone (like the new 
> cygcheck-leaves package is a standalone utility).
>
>> Would it be crazy to generate this and make it available on the cygwin
>> web site?  Or would the dependency graph generation overload
>> sourceware.org?
>
> The basic processing to generate a .dot file is pretty simple really; 
> just string comparisons and hash manipulations. But as Ryan already 
> pointed out, generating the actual graph in whatever format, is 
> probably compute intensive.
>
> See attached...

My goodness - you have been busy! You've taken some hacky kludgy perl 
that I threw together in five minutes flat and turned it into a proper 
programme. Tip of the hat to you, sir...

Dave.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: setup.ini dependency graph?

[Achim Gratz]

Charles Wilson writes:
> On 11/4/2013 11:00 AM, Christopher Faylor wrote:
>> On Mon, Nov 04, 2013 at 09:32:10AM -0500, Charles Wilson wrote:
>>> I've got a few cleanups, and then I'll share the result.  It's already
>>> helped me generate a few re-packaging requests I plan to post over on
>>> cygwin-apps...
>>
>> Is this packagable?  It sounds pretty interesting.
>
> Probably. I could put it in cygutils, or standalone (like the new
> cygcheck-leaves package is a standalone utility).

Interesting, indeed.  I've been working on another script that creates
install directories for offline installation (using multiple package
repositories and a config file to control package selection) that I hope
to get polished for public release.  It doesn't explicitly produce a
dependency graph (I do have some optional debug output that could be
used for this), but it wouldn't be difficult to bolt on (I already have
a hash for the forward dependencies).

>> Would it be crazy to generate this and make it available on the cygwin
>> web site?  Or would the dependency graph generation overload
>> sourceware.org?
>
> The basic processing to generate a .dot file is pretty simple really;
> just string comparisons and hash manipulations. But as Ryan already
> pointed out, generating the actual graph in whatever format, is
> probably compute intensive.

It might be easier to keep things a bit more simple: the most common
question is to get the direct dependencies of a package and the direct
dependents (who is requiring this package).  This graph is almost
trivial and could be followed interactively up or down the dependency
chain with little effort.  It wouldn't look as pretty as a full
dependency graph on an 8k display, but it'd be very useful.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple