Package choosing algorithm ...

Package choosing algorithm ...

[Houder]

Hi Corinna,

A year ago you changed the "package choosing algorithm" ...

https://sourceware.org/ml/cygwin/2015-02/msg00034.html
  - [ANNOUNCEMENT] Updated: setup.exe (Release 2.864) -- by Corinna 
Vinschen
Here C. announces the modification of the "package choosing algorithm"
(and the introduction of the "mirror mode")

When I started setup (2.873) and selected "Download without Installing", 
it
offered me to download version 2.5.0-0.8 of cygwin (installed: 
2.5.0-0.7).

Splendid!

However when I restarted setup and selected "Install from Local 
Directory",
it did NOT offer me to install version 2.5.0-0.8 ...

(Meaning, I had to restart setup AGAIN, and force it to install the last
  version of cygwin - a bit annoying, as you wrote in [1])

Did your modification only apply to the  "Download without Installing" 
part
of setup? (i.e. downloading from internet)

(honestly, I do not remember whether or not setup acted consistently in 
both
  modes after your modification)

Did I miss something?

Regards,

Henri

[1]
https://cygwin.com/ml/cygwin-apps/2015-01/msg00126.html
  - Setup patch to keep test version if test version installed - by 
Corinna
    Vinschen
Start of discussion ...

=====

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Package choosing algorithm ...

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1346 bytes --]

On Mar 20 17:54, Houder wrote:
> Hi Corinna,
> 
> A year ago you changed the "package choosing algorithm" ...
> 
> https://sourceware.org/ml/cygwin/2015-02/msg00034.html
>  - [ANNOUNCEMENT] Updated: setup.exe (Release 2.864) -- by Corinna Vinschen
> Here C. announces the modification of the "package choosing algorithm"
> (and the introduction of the "mirror mode")
> 
> When I started setup (2.873) and selected "Download without Installing", it
> offered me to download version 2.5.0-0.8 of cygwin (installed: 2.5.0-0.7).
> 
> Splendid!
> 
> However when I restarted setup and selected "Install from Local Directory",
> it did NOT offer me to install version 2.5.0-0.8 ...
> 
> (Meaning, I had to restart setup AGAIN, and force it to install the last
>  version of cygwin - a bit annoying, as you wrote in [1])
> 
> Did your modification only apply to the  "Download without Installing" part
> of setup? (i.e. downloading from internet)

I'm using "Install from Local Directory: all the time (having a local
mirror dir) and it works fine for me.  No idea what has gone different
in your scenario.  If there's a bug, we're certainly open to get fixes :)


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Package choosing algorithm ...

[Houder]

On 2016-03-21 11:00, Corinna Vinschen wrote:
> On Mar 20 17:54, Houder wrote:
>> Hi Corinna,
>> 
>> A year ago you changed the "package choosing algorithm" ...

[snip]

> I'm using "Install from Local Directory: all the time (having a local
> mirror dir) and it works fine for me.  No idea what has gone different
> in your scenario.  If there's a bug, we're certainly open to get fixes 
> :)

... it works for you ... that is odd (yes, I am surprised, as I first 
did
look for the cause of the problem at my side)

 From the above I understand that you specify -m option when calling 
setup
... Do you?

Is that the only option that you use? (just asking, because I 
experimented
with a different (higher) version number for the cygwin tar ball when, 
but
it did not make a difference).

Anyone using setup in the same way as I do and not experiencing my 
problem?

Regards,

Henri


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Package choosing algorithm ...

[Houder]

On 2016-03-21 11:00, Corinna Vinschen wrote:
> On Mar 20 17:54, Houder wrote:
>> Hi Corinna,
>> 
>> A year ago you changed the "package choosing algorithm" ...

[snip]

> I'm using "Install from Local Directory: all the time (having a local
> mirror dir) and it works fine for me.  No idea what has gone different
> in your scenario.  If there's a bug, we're certainly open to get fixes 
> :)

Hi Corinna, Achim, Yaakov ... (did I miss anyone? :-)

Did some more experimenting ...

All the time using "Install from Local Directory" ...

  - copied vim-minimal-7.4.1558-1.tar.xz to vim-minimal-7.4.1600-1.tar.xz
  - copied cygwin-2.5.0-0.8.tar.xz to cygwin-2.5.0-0.9.tar.xz
  - modified setup.ini
vim-minimal: substituted 1600 for current
cygwin: substituted 0.9 for TEST <====

Invoked setup:
  - vim-minimal is offered for install
  - cygwin is NOT (0.8 is shown with a interrogation sign (?) for size 
...

==> Again:
  - modified setup.ini
cygwin: substituted 0.9 for CURRENT <====

Invoked setup:
  - vim-minimal is offered for install
  - cygwin is ** NOW ** offered for install ...

==> Again:
  - modified setup.ini
cygwin: substituted 0.9 for TEST <==== (as in the 1st test)

Invoked setup: BUT THIS TIME SPECIFYING the -m option
  - vim-minimal is offered for install
  - cygwin is ** NOW ** offered for install ...

Apparently specifying the -m option makes a difference ...

Regards,

Henri

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Package choosing algorithm ...

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1266 bytes --]

On Mar 21 13:40, Houder wrote:
> All the time using "Install from Local Directory" ...
> 
>  - copied vim-minimal-7.4.1558-1.tar.xz to vim-minimal-7.4.1600-1.tar.xz
>  - copied cygwin-2.5.0-0.8.tar.xz to cygwin-2.5.0-0.9.tar.xz
>  - modified setup.ini
> vim-minimal: substituted 1600 for current
> cygwin: substituted 0.9 for TEST <====
> 
> Invoked setup:
>  - vim-minimal is offered for install
>  - cygwin is NOT (0.8 is shown with a interrogation sign (?) for size ...
> 
> ==> Again:
>  - modified setup.ini
> cygwin: substituted 0.9 for CURRENT <====
> 
> Invoked setup:
>  - vim-minimal is offered for install
>  - cygwin is ** NOW ** offered for install ...
> 
> ==> Again:
>  - modified setup.ini
> cygwin: substituted 0.9 for TEST <==== (as in the 1st test)
> 
> Invoked setup: BUT THIS TIME SPECIFYING the -m option
>  - vim-minimal is offered for install
>  - cygwin is ** NOW ** offered for install ...
> 
> Apparently specifying the -m option makes a difference ...

It shouldn't but apparently it does, oh well.

https://cygwin.com/acronyms#PGA anybody?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Package choosing algorithm ...

[Ian Lambert]

On March 22, 2016 5:52:50 AM EDT, Corinna Vinschen <corinna-cygwin@cygwin.com> wrote:
>On Mar 21 13:40, Houder wrote:
>> 
>> Apparently specifying the -m option makes a difference ...

>>>Anyone using setup in the same way as I do and not experiencing my 
problem?



Since:
 http://cygwin.com/ml/cygwin/2016-02/msg00250.html

I have been struggling with using apt-cyg to download package updates, 
because it uses wget, and running setup local.
Also using no-admin, without -m, I had similar problems
 getting the latest or test versions to download and install. My latest trick was deletes in /etc/setup/installed.db, and a small mod to apt-cyg.
I got cygwin 2.5.0-0.8 and mintty 2.3.3 this way.


>It shouldn't but apparently it does, oh well.
>
>https://cygwin.com/acronyms#PGA anybody?
>
>
>Corinna

I took a shot at git mirror of setup, to see if I could add user/password, or user-agent, options to setup,
 but getting git to download anything has so far failed for me. 




--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Package choosing algorithm ...

[Houder]

On 2016-03-22 10:52, Corinna Vinschen wrote:
> On Mar 21 13:40, Houder wrote:

[snip]

>> Apparently specifying the -m option makes a difference ...
> 
> It shouldn't but apparently it does, oh well.
> 
> https://cygwin.com/acronyms#PGA anybody?

Hi Corinna,

More experimenting ... including building and testing setup.exe

All the time using "Install from Local Directory" ...

  - copied cygwin-2.5.0-0.8.tar.xz to cygwin-2.5.0-0.9.tar.xz
  - modified setup.ini
cygwin: substituted 0.9 for TEST

  - using my test version of setup.exe, I found out that the current 
version of
    the cygwin tarball (setup.ini) HAD to be in my repo ... or else setup 
would
    not offer the new test version for install ...
  - I modified package_meta.h accordingly - see below
    (my new test version is offered for install after the modification)
  - next I downloaded the current version of the cygwin tarball and moved
    the file to appropriate place in my repo.
    Using the official version of setup, I verified that my test version 
is now
    offered for install.

Please, keep in mind: I am NOT announcing here that I found the bug ... 
Please,
keep that in mind (inexperienced C++ developer and all that).

I am sure more testing is required ...

Regards,

Henri

======

package_meta.h:

   std::string action_caption () const;
   packageversion trustp (bool _default, trusts const t) const
   {
     /* If the user chose "test" and a "test" version is available, 
return it. */
     if (t == TRUST_TEST && exp)
       return exp;
     /* Are we looking for the default version and does the installed 
version
        have a higher version number than the "curr" package?  This means 
the
        user has installed a "test" version, or built her own version 
newer
        than "curr".  Rather than pulling the user back to "curr", we 
install
        "test" if a "test" version is available and the version number is 
higher,
        or we stick to "installed" if not.  This reflects the behaviour 
of
        `yum update' on Fedora. */
// Henri
//Log (LOG_PLAIN) << "HENRI:" << "\t" << this->name << "\t" << _default 
<< "\t" << prev << curr << exp << "\t" << installed << endLog;
Log (LOG_PLAIN) << "HENRI:" << "\t" << this->name << " " \
         << curr << " " << installed << " cmp(c,i): " << 
packageversion::compareVersions (curr, installed) << " " \
         << installed << " " << exp << " cmp(i,e): " << 
packageversion::compareVersions (installed, exp) << endLog;

// Henri: Oops, current (setup.ini) is NOT present in repo in my case 
...
//    if (_default && curr && installed
     if (_default && installed
         && packageversion::compareVersions (curr, installed) < 0)
       {
         if (exp && packageversion::compareVersions (installed, exp) < 0) 
{
Log (LOG_PLAIN) << "HENRI!" << "\t" << this->name << endLog;
           return exp;
         }
         return installed;
       }
     /* Otherwise, if a "curr" version exists, return "curr". */
     if (curr)
       return curr;
     /* Otherwise return the installed version. */
     return installed;
   }

=====

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Package choosing algorithm ...

[Houder]

[-- Attachment #1: Type: text/plain, Size: 4033 bytes --]

Title: Re: Package choosing agorithm ...

Hi Corinna,

(Of course I am formally addressing the project leader, but Achim, Jon, 
Yaakov,
  others? are welcome to correct me if I am wrong below).

D w/o I = Download without Install
I f LD  = Install from Local Directory

As a reminder: I reported that a new test release of Cygwin is NOT 
offered
for install (having installed the previous test release) in my case.

     https://cygwin.com/ml/cygwin/2016-03/msg00425.html

Relevant in my case is:

  - I am not using a local mirror ...
      - but a "partially populated Local Directory"
  - I do not invoke setup.exe with the -m option
  - my local directory does not have the current release of Cygwin

As you replied "it works for me" and "not using the -m option should not
make a difference", I decided to study the source code of setup.exe.

As I had never seen the source code of setup before, I started my study 
where
you had "left off" modifying (about a year ago).

Subsequently, I reported back to the list, announcing that I found the 
bug,
but was not sure I had.

    https://cygwin.com/ml/cygwin/2016-03/msg00480.html

I was wrong.

My patch did work, but for the wrong reasons.

So does your modification of packagemeta::trustp ...

Here is another attempt of mine (attempt? I seriously doubt whether 
there is
someone which fully understands the source code in all its details ...)

You will find 2 patch files attached. These patch files modify 
(correct):

  1. packagemeta::ScanDownLoadedFiles and
  2. packagemeta::trustp

(yes, I do realize, that this correction to setup, is NOT important 
enough to
  most of you - using a local mirror and all that; consequently, do 
whatever you
  like with my post - I am merely providing info to the list).

My argument for the correction is as follows:

  - packagedb::packages, a class variable, is the "package database" that 
is
    used by setup.
  - basically, it is a list of pairs (mapping between the packagename and 
its
    entry in setup.ini) (one where installed.db has been "integrated").
  - formally:
     - typedef std::map <std::string, packagemeta *> packagecollection;
     - static packagecollection packages;
  - packagemeta is the type that describes a "package entry" from 
setup.ini
  - an object of type packagemeta does not only describe the previous, 
current
    and experimental (if applicable) lines from a package entry in 
setup.ini,
    it also describes the installed version of a package, as if 
installed.db is
    part of setup.ini.
--
  - bottom-line: the prev, curr, exp and installed data member of 
packagemeta
    are "static": these data members represent the info from setup.ini 
(and
    installed.db) and they should never be "nullified".
  - yet, that is exactly what happens in 
metapackage::ScanDownLoadedFiles.
--
  - packagemeta ALSO has a "not so static" data member, called versions; 
like
    packages, the contents of versions is "variable"
     - versions, a set of type packageversion, contains the versions of 
the
       package that are available (the source code speaks about 
"accessible")
     - available (in my simplistic view) means whether or not the tarball 
is
       present in the (local) repo
     - unfortunately, though versions specifies which versions are 
available,
       it does not tell us whether the previous, current or experimental
       version of a package is available or not
--
  - bottom-line: version numbers can be compared if that info is present 
in
    setup.ini; however, although the info may be present, it does not 
tell us
    that the associated tarball is available (neither does versions).
  - consequently, metapackage:trustp does not only have to verify that
    setup.ini has the required info, it also has to verify that the 
tarball
    is available, before it can give the "go-ahead".

Lastly: No, I have not tested (been able) my modification for each 
possible
configuration out there (for instance, I do not install source 
tarballs).

Regards,
Henri

=====

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: package_meta_cc.patch --]
[-- Type: text/x-diff; name=package_meta_cc.patch, Size: 869 bytes --]

diff --git a/package_meta.cc b/package_meta.cc
index 34ff78c..ff787a8 100644
--- a/package_meta.cc
+++ b/../src2/package_meta.cc
@@ -683,15 +683,18 @@ packagemeta::ScanDownloadedFiles (bool mirror_mode)
 	  /* For local installs, if there is no src and no bin, the version
 	   * is unavailable
 	   */
-	  if (!i->accessible () && !pkgsrcver.accessible ()
-	      && *i != pkg.installed)
+	  if (*i != pkg.installed
+	      && !i->accessible () && !pkgsrcver.accessible () )
 	    {
+// Henri: innihilate the info that has been collected from setup.ini? DO NOT!
+	      #if 0
 	      if (pkg.prev == *i)
 		pkg.prev = packageversion ();
 	      if (pkg.curr == *i)
 		pkg.curr = packageversion ();
 	      if (pkg.exp == *i)
 		pkg.exp = packageversion ();
+	      #endif
 	      pkg.versions.erase (i++);
 	      /* For now, leave the source version alone */
 	    }

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: package_meta_h.patch --]
[-- Type: text/x-diff; name=package_meta_h.patch, Size: 856 bytes --]

diff --git a/package_meta.h b/package_meta.h
index b24d4fc..3d2b775 100644
--- a/package_meta.h
+++ b/../src2/package_meta.h
@@ -105,12 +105,14 @@ public:
     if (_default && curr && installed
 	&& packageversion::compareVersions (curr, installed) < 0)
       {
-	if (exp && packageversion::compareVersions (installed, exp) < 0)
+// Henri: no, no, no ... first check whether the associated tarball is in the repo or not!
+	if (exp.accessible() && exp && packageversion::compareVersions (installed, exp) < 0)
 	  return exp;
 	return installed;
       }
     /* Otherwise, if a "curr" version exists, return "curr". */
-    if (curr)
+// Henri: no, no, no ... first check whether the associated tarball is in the repo or not!
+    if (curr.accessible() && curr)
       return curr;
     /* Otherwise return the installed version. */
     return installed;

[-- Attachment #4: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Package choosing algorithm ... more clarification

[Houder]

[-- Attachment #1: Type: text/plain, Size: 79 bytes --]

For those interested, see attached file (chooser_overview.txt).

Regards,
Henri

[-- Attachment #2: chooser_overview.txt --]
[-- Type: text/plain, Size: 5697 bytes --]

chooser_overview.txt:

2016-04-05 // Another attempt to create an overview ...

 = precondition
 - assume a "database" which reflects the info from setup.ini ... (packagedb::packages)
    - it list each package and every version of that pkg ...
 - this database even tells us what version has been installed if that is the case (reflecting installed.db)
 - furthermore we (setup.exe) are able to ascertain whether the tarball, associated with a version, is available
   or not -- the source code of setup.exe uses the term "accessible" for availability

 = D w/o I (Download without Installing)
 - setup.exe should show us ALL the pkgs and every version of that pkg ...
    - assume all tarballs are available (stable mirror)

 = I f LD (Install from Local Directory)
 - setup.exe should show us the installed pkgs and the versions of that pkg that are not installed, but are available
   in the repo ...
    - the tarball of an installed version may or may not be available in the repo ...
 - furthermore it should show us each non-installed pkg if at least one version of that pkg is available in the repo

ChooserPage::OnInit has the following call flow in both cases (D w/o I and I f LD)

ChooserPage::OnInit
  packagemeta::ScanDownLoadedFiles
  packagedb::setExistence
  packagemeta::trustp(true, ...

 = D w/o I

 // (pkg.installed == version) => pkg is installed
 - ScanDownloadedFiles # method name is a misnomer here
    ! (pkg is installed || a version is acccessible) => erase version from packagemeta.versions
    - here all versions will be kept (stable mirror) in packagemeta.versions
 - setExistence
    ! (pkg is installed || a version is acccessible) => erase pkg from packages
    - the test whether the pkg is installed or not, is useless here ...
    - ... as all pkgs (i.e. all versions of each pkg) are available (accessible): UNmodified database
 End result: setup.up will show us all the pkgs and every version of each pkg ...

 = I f LD

 // (pkg.installed == version) => pkg is installed
 - ScanDownloadedFiles
    ! (pkg is installed || a version is accessible) => erase version from packagemeta.versions
    - not-installed pkgs: available versions will be kept in packagemeta.versions
    - installed pkgs: the installed version will be kept
    - installed pkgs: non-available versions will be removed from packagemeta.versions
 - setExistence
    ! (pkg is installed || a version is accessible) => erase pkg from packages
    - a not-installed pkg, of which at least one version is available: this pkg will be kept in the database
      (versions will show the available versions)
    - installed pkgs: these pkgs will be kept in the database
      (versions will show the installed version and available versions that are not installed)
 End result: setup.exe will show us all the available versions of each pkg, if it has not been installed yet, and all
 the pkgs (both the installed version and available versions that are not installed) which have been installed ...

----- Ignoring the issue of source tarballs ...

Each "package entry" (p, c and/or t) in setup.ini is represeneted by an object of type packagemeta ...

 - class variable packages (of type packagedb): std::map <std::string, *packagemeta>
   or: a mapping from a packagename to a pointer to an object of type packagemeta

A simplified view of the data members of class packagemeta is as follows:

 - prev of type packageversion
 - curr ditto
 - exp ditto
 - installed ditto
 - versions of type std::set <packageversion>
-
prev, curr, exp and installed reflect the "static" part of a "package entry": these data members reflect the info from both
setup.ini and installed.db ...
"Nullifying" these data members, as currently happens in ScanDownLoadedFiles is ... stupid?

versions tells what versions of the pkg are available (accessible); versions is managed by ScanDownLoadedFiles ...

Attempting to grasp the general flow of computation:

packagemeta::ScanDownLoadedFiles =>
for each pkg // from packagedb::packages
  for each version // from metapackage::versions
    [ <this>.scan() ] // packageversion::scan() -- simplified!
    // (pkg.installed == version) => pkg is installed
    if ! (pkg is installed || version is accessible) // packagemeta::installed, packageversion::accessible()
        erase version from versions
Note: a version is NOT erased if the version is installed OR if the version is available

packagemeta::accessible => // called by setExistence
for each version // from metapackage::versions
  if version is accessible // packageversion::accessible()
    return true // pkg is available in (local) repo
  return false

packageversion::accessible =>
_packageversion::accessible =>
...

packagedb::setExistence
for each pkg // from packagedb::packages
  // (pkg.installed == version) => pkg is installed
  if ! (pkg is installed || a version is accessible) // packagemeta::installed, packagemeta::accessible()
    erase pkg from packages
Note: a pkg is NOT erased if it is installed, OR if at least one version is available if the pkg is not installed

packagemeta:trustp =>
... only the default trust for the moment ...
if (pkg is installed)
  if (installed version > current version) // requires curr to be properly initialized, i.e. setup.ini MUST have
                                           // a current entry for this package
    if (the experimental version is available
        && experimental version > installed version) // requires exp to be properly initialized ...
          return exp;
    else
      return installed;
else
  if (the current version is available) // requires curr to be properly initialized ...
    return curr;
return installed;

=====

[-- Attachment #3: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple