New upload method: Deleting old versions

New upload method: Deleting old versions

[David Stacey]

Thanks to CGF for the effort that has gone into the new upload process - 
I've just uploaded a new package called 'icoutils' and it was really 
quite painless.

One thing I haven't grasped yet is the mechanism by which old versions 
are deleted. If I were to upload a new version of an existing package, 
how would I go about removing the oldest?

For instance, for the hypothetical package 'bob', I have 'bob-0.2-1' as 
current and 'bob-0.1-1' as previous. When a new version comes along, I 
use the new mechanism to upload 'bob-0.3-1', and this becomes current. 
'bob-0.2-1' is relegated to the previous release. However, 'bob-0.1-1' 
is not automatically removed. If I wanted to delete this, how would I go 
about it?

Dave.

Re: New upload method: Deleting old versions

[Christopher Faylor]

On Tue, Oct 22, 2013 at 11:31:34PM +0100, David Stacey wrote:
>Thanks to CGF for the effort that has gone into the new upload process - 
>I've just uploaded a new package called 'icoutils' and it was really 
>quite painless.
>
>One thing I haven't grasped yet is the mechanism by which old versions 
>are deleted. If I were to upload a new version of an existing package, 
>how would I go about removing the oldest?

Eventually old packages will be deleted (actually archived) automatically.
I haven't yet added that functionality to upset though.  Maybe this weekend
if I have the time.

I'm currently working on getting rpm building though.  What a (*#$ pain
that is.

cgf

Deleting old versions of packages

[Christopher Faylor]

On Tue, Oct 22, 2013 at 10:14:52PM -0400, Christopher Faylor wrote:
>On Tue, Oct 22, 2013 at 11:31:34PM +0100, David Stacey wrote:
>>Thanks to CGF for the effort that has gone into the new upload process - 
>>I've just uploaded a new package called 'icoutils' and it was really 
>>quite painless.
>>
>>One thing I haven't grasped yet is the mechanism by which old versions 
>>are deleted. If I were to upload a new version of an existing package, 
>>how would I go about removing the oldest?
>
>Eventually old packages will be deleted (actually archived) automatically.
>I haven't yet added that functionality to upset though.  Maybe this weekend
>if I have the time.

The problem with this method is that it forces you to use a setup.hint
file to delete packages.  For instance, I just had a case where I wanted
make-4.0-2 to be curr and make-3.82.90-1 to be prev.  I really don't
want to have to change setup.hint to enforce that because that's error
prone.  If that's exactly what people were trying to warn me about then
I apologize for being dense.

I don't like the clunkiness of creating anti-files in the sftp directory
to force deletions but I can't think of any other way to do this short
of actually making it look like the entire release directory is in the
user's sftp area.

cgf

Re: Deleting old versions of packages

[Achim Gratz]

Christopher Faylor writes:
> I don't like the clunkiness of creating anti-files in the sftp directory
> to force deletions but I can't think of any other way to do this short
> of actually making it look like the entire release directory is in the
> user's sftp area.

What we've got is what I'd call a transactional procedure for
maintaining our packages.  The positive part is implemented: if the
transaction gets committed (!ready) then any files get transferred to
the release directory.

Removing files is difficult for two reasons: a) we don't easily know the
current state of the release directory and b) setup.hint may allow
multiple states depending on what is already in the release directory.

Keeping the deletion part a manual operation is possible, but probably
goes against your intentions and a !delete file is iffy.  So my proposal
would be that the current state of the release directory is mirrored
with (zero-sized?) files in the sftp area.  A maintainer could then just
delete any of those placeholders and the corresponding file would be
deleted from the release directory.  Crucially, the upload and the
deletion would not necessarily need to happen with the same transaction.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

DIY Stuff:
http://Synth.Stromeko.net/DIY.html

Re: Deleting old versions of packages

[Christopher Faylor]

On Mon, Oct 28, 2013 at 09:21:49PM +0100, Achim Gratz wrote:
>Christopher Faylor writes:
>> I don't like the clunkiness of creating anti-files in the sftp directory
>> to force deletions but I can't think of any other way to do this short
>> of actually making it look like the entire release directory is in the
>> user's sftp area.
>
>What we've got is what I'd call a transactional procedure for
>maintaining our packages.  The positive part is implemented: if the
>transaction gets committed (!ready) then any files get transferred to
>the release directory.
>
>Removing files is difficult for two reasons: a) we don't easily know the
>current state of the release directory and b) setup.hint may allow
>multiple states depending on what is already in the release directory.
>
>Keeping the deletion part a manual operation is possible, but probably
>goes against your intentions and a !delete file is iffy.  So my proposal
>would be that the current state of the release directory is mirrored
>with (zero-sized?) files in the sftp area.  A maintainer could then just
>delete any of those placeholders and the corresponding file would be
>deleted from the release directory.  Crucially, the upload and the
>deletion would not necessarily need to happen with the same transaction.

So your proposal is basically to do "I can't think of any way to do this
short of..."

That's not a very inspired proposal given the fact that I already
mentioned the idea and implied that I wasn't keen on doing it that way.

cgf

Re: Deleting old versions of packages

[Achim Gratz]

Christopher Faylor writes:
> So your proposal is basically to do "I can't think of any way to do this
> short of..."

Well, it was you who was taking options from the table.

> That's not a very inspired proposal given the fact that I already
> mentioned the idea and implied that I wasn't keen on doing it that way.

Sorry for failing to be inspired or inspiring or whatever it was you
were expecting.  Two proposals shot down, here's the third:

Leave everything as is at the upload side, but add two cleanup passes on
the release directory; a first that deletes zero-sized files and a
second that removes empty directories.  That way a maintainer can
"upload" a file that he wants to be deleted.  Then upset will see the
cleaned up release directory and setup.ini won't have to specify tzhe
exact versions of files most of the time.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: Deleting old versions of packages

[Christopher Faylor]

On Tue, Oct 29, 2013 at 09:02:56PM +0100, Achim Gratz wrote:
>Leave everything as is at the upload side, but add two cleanup passes on
>the release directory; a first that deletes zero-sized files and a
>second that removes empty directories.  That way a maintainer can
>"upload" a file that he wants to be deleted.  Then upset will see the
>cleaned up release directory and setup.ini won't have to specify tzhe
>exact versions of files most of the time.

I don't think I have to worry about empty directories.  Those are easy
to purge.  I was trying to avoid the zero-length file scenario but maybe
that's the best that can be done.  The ordering would have to be right
so that upset doesn't first reference a file and then delete it of
course.

In another discussion someone suggested a web-based form for deletion
but if I was going to use that, I might as well use that method for
upload too.  I don't think I'm ready to scrap sftp now that this is
implemented.

Re: Deleting old versions of packages

[Christopher Faylor]

On Tue, Oct 29, 2013 at 04:50:38PM -0400, Christopher Faylor wrote:
>On Tue, Oct 29, 2013 at 09:02:56PM +0100, Achim Gratz wrote:
>>Leave everything as is at the upload side, but add two cleanup passes on
>>the release directory; a first that deletes zero-sized files and a
>>second that removes empty directories.  That way a maintainer can
>>"upload" a file that he wants to be deleted.  Then upset will see the
>>cleaned up release directory and setup.ini won't have to specify tzhe
>>exact versions of files most of the time.
>
>I don't think I have to worry about empty directories.  Those are easy
>to purge.  I was trying to avoid the zero-length file scenario but maybe
>that's the best that can be done.  The ordering would have to be right
>so that upset doesn't first reference a file and then delete it of
>course.

In further consideration in the thinking room, two other ideas presented
themselves:

1) Create files with a leading '-' character to flag deletion.

2) Create a !deleteme directory containing files or directories that
should be deleted.  (Although empty directories can be automatically
detected and are not really an issue)

I like either of these better than the zero-length file because it
will be obvious from simple directory scanning when a file needs to
be deleted.

I'm thinking that the leading '-' is the easiest to implement since
it is stateless.

cgf

Re: Deleting old versions of packages

[Achim Gratz]

Christopher Faylor writes:
> In further consideration in the thinking room, two other ideas presented
> themselves:
>
> 1) Create files with a leading '-' character to flag deletion.

That should be workable and relatively painless on all sides.

> 2) Create a !deleteme directory containing files or directories that
> should be deleted.  (Although empty directories can be automatically
> detected and are not really an issue)

I like '-' prefixed files alongside the files to be uploaded better than
yet another directory hierarchy to traverse.  But if you think this is
easier to implement, I can adapt.

> I like either of these better than the zero-length file because it
> will be obvious from simple directory scanning when a file needs to
> be deleted.
>
> I'm thinking that the leading '-' is the easiest to implement since
> it is stateless.

It is still zero-sized though or do you envision to have these files
some content?


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Deleting old versions of packages

[Christopher Faylor]

On Thu, Nov 14, 2013 at 09:32:25PM +0100, Achim Gratz wrote:
>Christopher Faylor writes:
>> In further consideration in the thinking room, two other ideas presented
>> themselves:
>>
>> 1) Create files with a leading '-' character to flag deletion.
>
>That should be workable and relatively painless on all sides.
>
>> 2) Create a !deleteme directory containing files or directories that
>> should be deleted.  (Although empty directories can be automatically
>> detected and are not really an issue)
>
>I like '-' prefixed files alongside the files to be uploaded better than
>yet another directory hierarchy to traverse.  But if you think this is
>easier to implement, I can adapt.
>
>> I like either of these better than the zero-length file because it
>> will be obvious from simple directory scanning when a file needs to
>> be deleted.
>>
>> I'm thinking that the leading '-' is the easiest to implement since
>> it is stateless.
>
>It is still zero-sized though or do you envision to have these files
>some content?

I don't see any need for content but it actually won't matter what
people put in the files.

cgf

Re: Deleting old versions of packages

[Damien Doligez]

Hi,

On 2013-10-28, at 20:34, Christopher Faylor wrote:

> The problem with this method is that it forces you to use a setup.hint
> file to delete packages.  For instance, I just had a case where I wanted
> make-4.0-2 to be curr and make-3.82.90-1 to be prev.  I really don't
> want to have to change setup.hint to enforce that because that's error
> prone.  If that's exactly what people were trying to warn me about then
> I apologize for being dense.

Maybe I'm too naive or too late, but it seems to me that you could
use the same procedure we used when uploading was done manually:
you'd upload the new version and send a message to the maintainers
saying "upload 4.0-2, keep 3.82.90-1 as prev, and delete all other
versions".

With the automatic system, this could take the form of a file (for
example !keep) that lists all the versions (other than the uploaded
one) that should be kept, with all other versions to be archived
and removed.

This is almost the same as changing setup.hint, but much less
error-prone because it's outside the package. You should also do a
few sanity checks: that all the versions mentioned in !keep are
present, and that all the versions mentioned in setup.hint are also
in !keep. If one of these conditions is false, you don't remove
anything.

-- Damien