setup.hint documentation issues

setup.hint documentation issues

[Jon Turney]

While I've been looking at replacement/improvement for the current upset 
script, I've come across some minor issues related to 
under-specification or under-documentation of setup.hint:

* The encoding of setup.hint is unspecified.

Historically both ISO-8859-1 and UTF-8 have been used. (e.g. libspiro 
used 'bézier' with an ISO-8859-1 e-acute, whereas calligra-l10n-nb uses 
'Bokmål' with an UTF-8 a-ring. Various other hints use UTF-8 punctuation 
marks)

I think currently UTF-8 displays correctly in the HTML package pages, 
but neither encoding displays correctly in setup.

I'd suggest that we specify UTF-8 and eventually fix setup to handle that.

* 'sdesc' text is mangled in setup.ini (but not the HTML package list)

In particular, it is forced to start with a capital letter (which is 
incorrect when the sdesc starts with a command name which is properly 
lower-case, e.g. "dash shell", etc.), and any text up to and including 
the first colon is removed, presumably in an effort to prevent people 
writing the package name again, (which mangles perl and ruby module 
names in the description, e.g. "Ruby Net::HTTP persistent connection 
support", ""Perl Math::Int64 distribution", etc.)

I'd suggest this mangling is removed, and sdesc starting "packagename:" 
is explicitly reported.

* Handling of double-quoted text seems over-complicated

A multi-line double-quoted value is terminated only by a double-quote at 
the end of the line, and embedded double-quotes are silently transformed 
to single-quotes (e.g proj had a sdesc of ""The PROJ Cartographic 
Projections Software (utilities)", where the erroneous nested 
double-quote was being transformed to a single-quote)

There is no escaping of embedded double-quotes, and no way to represent one.

Additionally, spaces after the leading quote are magically removed.

Additionally, genini requires that sdesc and ldesc are double-quoted, 
but upset does not.

I'd suggest that double-quoting of those keys is made mandatory, and 
embedded double-quotes are forbidden, as this permits simpler processing 
of this text, lexing character by character.

* It's not very clear what 'skip' represents

The description "The skip line indicates that that package should not 
appear in setup. It is intended for directories that exist in the 
hierarchy that should not be considered." is a bit vague to me.

It's not totally clear if it's intended for indicating directories which 
should be empty, source-only packages, or something else.

upset knows enough to omit packages which have no install tarfiles (i.e. 
are source-only) from from setup.ini, irrespective of 'skip'.

However, the presence of 'skip' also causes the package to be omitted 
from the HTML package list.

I think cygport's behaviour has changed over time, but currently will 
mark source-packages as 'skip', however there are several packages that 
are source-only (e.g. attica), that are missing 'skip'.

Re: setup.hint documentation issues

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 3555 bytes --]

On Feb  9 13:18, Jon Turney wrote:
> 
> While I've been looking at replacement/improvement for the current upset
> script, I've come across some minor issues related to under-specification or
> under-documentation of setup.hint:
> 
> * The encoding of setup.hint is unspecified.
> 
> Historically both ISO-8859-1 and UTF-8 have been used. (e.g. libspiro used
> 'bézier' with an ISO-8859-1 e-acute, whereas calligra-l10n-nb uses 'Bokmål'
> with an UTF-8 a-ring. Various other hints use UTF-8 punctuation marks)
> 
> I think currently UTF-8 displays correctly in the HTML package pages, but
> neither encoding displays correctly in setup.
> 
> I'd suggest that we specify UTF-8 and eventually fix setup to handle that.

ACK

> * 'sdesc' text is mangled in setup.ini (but not the HTML package list)
> 
> In particular, it is forced to start with a capital letter (which is
> incorrect when the sdesc starts with a command name which is properly
> lower-case, e.g. "dash shell", etc.), and any text up to and including the
> first colon is removed, presumably in an effort to prevent people writing
> the package name again, (which mangles perl and ruby module names in the
> description, e.g. "Ruby Net::HTTP persistent connection support", ""Perl
> Math::Int64 distribution", etc.)
> 
> I'd suggest this mangling is removed, and sdesc starting "packagename:" is
> explicitly reported.

Sounds good, but where is this mangling performed?  Upset?

> * Handling of double-quoted text seems over-complicated
> 
> A multi-line double-quoted value is terminated only by a double-quote at the
> end of the line, and embedded double-quotes are silently transformed to
> single-quotes (e.g proj had a sdesc of ""The PROJ Cartographic Projections
> Software (utilities)", where the erroneous nested double-quote was being
> transformed to a single-quote)
> 
> There is no escaping of embedded double-quotes, and no way to represent one.
> 
> Additionally, spaces after the leading quote are magically removed.
> 
> Additionally, genini requires that sdesc and ldesc are double-quoted, but
> upset does not.
> 
> I'd suggest that double-quoting of those keys is made mandatory, and
> embedded double-quotes are forbidden, as this permits simpler processing of
> this text, lexing character by character.

What about existing packages?

> * It's not very clear what 'skip' represents
> 
> The description "The skip line indicates that that package should not appear
> in setup. It is intended for directories that exist in the hierarchy that
> should not be considered." is a bit vague to me.
> 
> It's not totally clear if it's intended for indicating directories which
> should be empty, source-only packages, or something else.
> 
> upset knows enough to omit packages which have no install tarfiles (i.e. are
> source-only) from from setup.ini, irrespective of 'skip'.
> 
> However, the presence of 'skip' also causes the package to be omitted from
> the HTML package list.
> 
> I think cygport's behaviour has changed over time, but currently will mark
> source-packages as 'skip', however there are several packages that are
> source-only (e.g. attica), that are missing 'skip'.

IMHO we don't need "skip".  A source-only package should be
automatically skipped anyway.  What other reason do we need to ignore
a package?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: setup.hint documentation issues

[Achim Gratz]

Jon Turney writes:
> I think currently UTF-8 displays correctly in the HTML package pages,
> but neither encoding displays correctly in setup.
>
> I'd suggest that we specify UTF-8 and eventually fix setup to handle that.

UTF-8 these days, please.

> I'd suggest this mangling is removed, and sdesc starting
> "packagename:" is explicitly reported.

OK.

> I'd suggest that double-quoting of those keys is made mandatory, and
> embedded double-quotes are forbidden, as this permits simpler
> processing of this text, lexing character by character.

OK, although we might need some sort of escaping in the long run.

> upset knows enough to omit packages which have no install tarfiles
> (i.e. are source-only) from from setup.ini, irrespective of 'skip'.
>
> However, the presence of 'skip' also causes the package to be omitted
> from the HTML package list.

I could be wrong, but it seems it may have been intended to deal with
packages that became obsolete, but the previous version was still kept.

> I think cygport's behaviour has changed over time, but currently will
> mark source-packages as 'skip', however there are several packages
> that are source-only (e.g. attica), that are missing 'skip'.

So the proposal is to remove skip or make it mandatory for source-only
packages?


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: setup.hint documentation issues

[Jon Turney]

On 09/02/2016 17:10, Corinna Vinschen wrote:
> On Feb  9 13:18, Jon Turney wrote:
>> * 'sdesc' text is mangled in setup.ini (but not the HTML package list)
>>
>> In particular, it is forced to start with a capital letter (which is
>> incorrect when the sdesc starts with a command name which is properly
>> lower-case, e.g. "dash shell", etc.), and any text up to and including the
>> first colon is removed, presumably in an effort to prevent people writing
>> the package name again, (which mangles perl and ruby module names in the
>> description, e.g. "Ruby Net::HTTP persistent connection support", ""Perl
>> Math::Int64 distribution", etc.)
>>
>> I'd suggest this mangling is removed, and sdesc starting "packagename:" is
>> explicitly reported.
>
> Sounds good, but where is this mangling performed?  Upset?

Yes, upset currently does this mangling.

>> * Handling of double-quoted text seems over-complicated
>>
>> A multi-line double-quoted value is terminated only by a double-quote at the
>> end of the line, and embedded double-quotes are silently transformed to
>> single-quotes (e.g proj had a sdesc of ""The PROJ Cartographic Projections
>> Software (utilities)", where the erroneous nested double-quote was being
>> transformed to a single-quote)
>>
>> There is no escaping of embedded double-quotes, and no way to represent one.
>>
>> Additionally, spaces after the leading quote are magically removed.
>>
>> Additionally, genini requires that sdesc and ldesc are double-quoted, but
>> upset does not.
>>
>> I'd suggest that double-quoting of those keys is made mandatory, and
>> embedded double-quotes are forbidden, as this permits simpler processing of
>> this text, lexing character by character.
>
> What about existing packages?

I've fixed the existing uses on sourceware (which were proj and I think 
one other package I unfortunately didn't take note of)

>> * It's not very clear what 'skip' represents
>>
>> The description "The skip line indicates that that package should not appear
>> in setup. It is intended for directories that exist in the hierarchy that
>> should not be considered." is a bit vague to me.
>>
>> It's not totally clear if it's intended for indicating directories which
>> should be empty, source-only packages, or something else.
>>
>> upset knows enough to omit packages which have no install tarfiles (i.e. are
>> source-only) from from setup.ini, irrespective of 'skip'.
>>
>> However, the presence of 'skip' also causes the package to be omitted from
>> the HTML package list.
>>
>> I think cygport's behaviour has changed over time, but currently will mark
>> source-packages as 'skip', however there are several packages that are
>> source-only (e.g. attica), that are missing 'skip'.
>
> IMHO we don't need "skip".  A source-only package should be
> automatically skipped anyway.  What other reason do we need to ignore
> a package?

Yes, this is why I ask the question.

I don't see what 'skip' adds above automatically noticing that there are 
no install tar files, only source tar files.

Re: setup.hint documentation issues

[Jon Turney]

On 09/02/2016 19:31, Achim Gratz wrote:
> Jon Turney writes:
>> I'd suggest that double-quoting of those keys is made mandatory, and
>> embedded double-quotes are forbidden, as this permits simpler
>> processing of this text, lexing character by character.
>
> OK, although we might need some sort of escaping in the long run.

Yes, I have no problem with later adding escaping e.g. using '\', if 
needed, since that can written with a character by character lexer.

But writing a replacement setup.hint parser, while maintaining 
bug-for-bug compatibility with the existing behaviour, requires a 
line-by-line processing with various transformations...

>> upset knows enough to omit packages which have no install tarfiles
>> (i.e. are source-only) from from setup.ini, irrespective of 'skip'.
>>
>> However, the presence of 'skip' also causes the package to be omitted
>> from the HTML package list.
>
> I could be wrong, but it seems it may have been intended to deal with
> packages that became obsolete, but the previous version was still kept.

Any package marked 'skip' is completely omitted from setup.ini, so setup 
won't do anything with it, so I don't think that can be the use for it.

>> I think cygport's behaviour has changed over time, but currently will
>> mark source-packages as 'skip', however there are several packages
>> that are source-only (e.g. attica), that are missing 'skip'.
>
> So the proposal is to remove skip or make it mandatory for source-only
> packages?

I'm not sure.  I think I tend towards removing it, since it doesn't add 
any information.

But currently cygport generates a setup.hint for source-only package 
which only contains 'skip'.  I'm not sure that's the best idea, as 
having no sdesc, etc. means we can't describe the source package.

Re: setup.hint documentation issues

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 584 bytes --]

On Feb 17 14:23, Jon Turney wrote:
> On 09/02/2016 17:10, Corinna Vinschen wrote:
> >IMHO we don't need "skip".  A source-only package should be
> >automatically skipped anyway.  What other reason do we need to ignore
> >a package?
> 
> Yes, this is why I ask the question.
> 
> I don't see what 'skip' adds above automatically noticing that there are no
> install tar files, only source tar files.

Nuke it.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: setup.hint documentation issues

[Achim Gratz]

Jon Turney writes:
>> OK, although we might need some sort of escaping in the long run.
>
> Yes, I have no problem with later adding escaping e.g. using '\', if
> needed, since that can written with a character by character lexer.

Indeed and it should also be possible to parse via regex.  IN any case,
that's not a problem we need to solve now.

>> So the proposal is to remove skip or make it mandatory for source-only
>> packages?
>
> I'm not sure.  I think I tend towards removing it, since it doesn't
> add any information.

OK.

> But currently cygport generates a setup.hint for source-only package
> which only contains 'skip'.  I'm not sure that's the best idea, as
> having no sdesc, etc. means we can't describe the source package.

Hmm.  I guess Yaakov perhaps knows what that was supposed to achieve.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: setup.hint documentation issues

[Jon Turney]

On 17/02/2016 14:23, Jon Turney wrote:
> On 09/02/2016 17:10, Corinna Vinschen wrote:
>> On Feb  9 13:18, Jon Turney wrote:
>>> * 'sdesc' text is mangled in setup.ini (but not the HTML package
>>> list)
>>>
>>> In particular, it is forced to start with a capital letter (which
>>> is incorrect when the sdesc starts with a command name which is
>>> properly lower-case, e.g. "dash shell", etc.), and any text up to
>>> and including the first colon is removed, presumably in an effort
>>> to prevent people writing the package name again, (which mangles
>>> perl and ruby module names in the description, e.g. "Ruby
>>> Net::HTTP persistent connection support", ""Perl Math::Int64
>>> distribution", etc.)
>>>
>>> I'd suggest this mangling is removed, and sdesc starting
>>> "packagename:" is explicitly reported.
>>
>> Sounds good, but where is this mangling performed?  Upset?
>
> Yes, upset currently does this mangling.

This mangling is no longer done.

Instead there is an attempt to enforce the rule (from [1]) that, "the 
package name should not be part of the description", by detecting a 
sdesc which starts "packagename:" or "packagename -", but this is 
probably fairly easy to defeat.

[1] https://cygwin.com/setup.html