transition git-hooks on sourceware.org to Python 3.x

transition git-hooks on sourceware.org to Python 3.x

[Joel Brobecker]

[-- Attachment #1: Type: text/plain, Size: 3142 bytes --]

Hello,

The current version of the git-hooks being used on sourceware
is a version which uses Python 2.7. I have since then migrated
to Python 3, where the minimum version is currently Python 3.8.
That version has been running on AdaCore's server for a few weeks,
now, so I think it's time to think about transitioning our repos
on sourceware.

The main improvement, besides the transition away from Python 2.x,
comes from Python 3.x itself and the handling of strings vs bytes,
so charsets are handled a bit better.

The one small difficulty comes from making sure a recent version
of Python 3.x gets used. Ideally, we should use the same Python
I'm using on sourceware to run the git-hooks testsuite to make sure
there aren't any compatibility issues.

For this, what I'd like to propose is that we use a set of wrapper
scripts which provide the same hooks as the git-hooks, and which
only do the following: add the correct Python to the PATH, and
then call the real hook script in git-hooks.

I've created a small prototype and did some testing with it,
it seems to work well. In a nutshell, we have:

  <wrappers root>/
      - hooks/           <-- directory to use as hooks in Git bare repos
      - git-hooks/       <-- clone of https://github.com/adacore/git-hooks
      - python3-for-git-hooks/  <-- Python 3.x venv to be used by the git-hooks

To transition a repository to updated hooks, we simply replace
the hooks symbolic link to point to <wrappers root>/hooks.

My proptotype is at...

    /home/gdbadmin/brobecke/py3-git-hooks/git-hooks-wrapper

... for those who are curious and can log onto sourceware.org.
There is a README.md file that gives a bit more information
(I'll attach to this message as well)

If agreed with the approach, I think we'll want to decide where
we want this to live. Maybe /sourceware/projects/src-home/?
The main issue with this location is that I worry about shared
access and in particular maintenance over time. I'd like to be
able to make changes there if there is a need to fix something.
I'd like also to be able to continue updating the git-hooks version
being used as I continue to push fixes and enhancements.

I can take care of the git-hooks-wrapper setup once the location
is settled.

In terms of the projects themselves, I've identified the following
repositories as using the git-hooks:

  - binutils-gdb.git/hooks
  - gcc.git/hooks
  - glibc.git/hooks
  - newlib-cygwin.git/hooks
  - valgrind.git/hooks

This is based on searching for repositories on sourcware.org
whose hooks dir is a symlink to the git-hooks. Not sure if there are
others.

I've Cc'ed people I know have asked me about the git-hooks for
GCC, glibc and newlib. Not sure about valgrind.

What we could do is give the binutils & GDB maintainers a quick
heads up and then transition the binutils-gdb repository. If all
goes well, I think we can transition the remaining ones.

Thoughts?

PS: If a project needs to stay on Python 2.7 for the time being,
    it is easy to accomodate it with the current scheme. But they
    will need to use a dedicated branch, as the latest no longer
    supports Python 2.x.
-- 
Joel

[-- Attachment #2: README.md --]
[-- Type: text/markdown, Size: 2320 bytes --]

Git-Hooks Wrapper
=================

This repository provices a hooks/ directory which contain the same set of
scripts as the AdaCore git-hooks (https://github.com/adacore/git-hooks).
Those scripts simply act as wrappers allowing us to set the environment
up prior to calling the real git-hooks hooks. In particular, our wrappers
allow us to make sure we use a controlled version of Python to run the hooks.

Wrapper Setup Overview
----------------------

The wrappers expect the following structure:

    <wrappers root>/
        - hooks/           <-- directory to use as hooks in Git bare repos
        - git-hooks/       <-- clone of https://github.com/adacore/git-hooks
        - python3-for-git-hooks/  <-- Python 3.x venv to be used by the git-hooks

The wrappers provides wrapper scripts which will do the following:

  - set the environment up to put the python3-for-git-hooks virtual env
    in its PATH;

  - Call the corresponding hook in git-hooks/hooks/.


Git-Hooks Setup
---------------

The wrappers expect a clone of the git-hooks to be located at
the root of this repository, named git-hooks.


Python Setup
------------

The wrappers expect a Python 3 Virtual Enviroment (venv) to be set up
at the root of this repository, named python3-for-git-hooks.

The venv should be set up as follow:

   # Create the venv, and activate it...

   $ python3 -m venv python3-for-git-hooks
   $ . ./python3-for-git-hooks/bin/activate

   # Make sure we use the latest version of pip

   $ python3 -m pip install pip --upgrade

   # Install all dependencies requried by the git-hooks.
   #
   # Note that the exact list of dependencies may change
   # over time. If unsure, run the git-hooks testsuite
   # as follow:
   #
   #      $ cd path/to/git-hooks/testsuite
   #      $ ./run-validation-tests
   #
   # The testsuite will abort with an error if any dependency
   # is missing.

   $ python3 -m pip install black e3-core flake8 pre-commit pytest pytest-cov pytest-xdist

Check the git-hooks' README.md file for the required minimum version
of Python that must be used.

Activating the Git-Hooks Wrapper
--------------------------------

To use these wrappers in a Git repository, replace the hooks/ directory
inside that Git repository by a symbolic link to the hooks/ directory
inside the git-hooks-wrapper

Re: transition git-hooks on sourceware.org to Python 3.x

[Mark Wielaard]

Hi,

I added Ivo to the CC who setup the valgrind hooks. If he isn't
available I'll take care of them.

Cheers,

Mark

On Sun, Dec 12, 2021 at 03:32:47PM +0400, Joel Brobecker via Overseers wrote:
> Hello,
> 
> The current version of the git-hooks being used on sourceware
> is a version which uses Python 2.7. I have since then migrated
> to Python 3, where the minimum version is currently Python 3.8.
> That version has been running on AdaCore's server for a few weeks,
> now, so I think it's time to think about transitioning our repos
> on sourceware.
> 
> The main improvement, besides the transition away from Python 2.x,
> comes from Python 3.x itself and the handling of strings vs bytes,
> so charsets are handled a bit better.
> 
> The one small difficulty comes from making sure a recent version
> of Python 3.x gets used. Ideally, we should use the same Python
> I'm using on sourceware to run the git-hooks testsuite to make sure
> there aren't any compatibility issues.
> 
> For this, what I'd like to propose is that we use a set of wrapper
> scripts which provide the same hooks as the git-hooks, and which
> only do the following: add the correct Python to the PATH, and
> then call the real hook script in git-hooks.
> 
> I've created a small prototype and did some testing with it,
> it seems to work well. In a nutshell, we have:
> 
>   <wrappers root>/
>       - hooks/           <-- directory to use as hooks in Git bare repos
>       - git-hooks/       <-- clone of https://github.com/adacore/git-hooks
>       - python3-for-git-hooks/  <-- Python 3.x venv to be used by the git-hooks
> 
> To transition a repository to updated hooks, we simply replace
> the hooks symbolic link to point to <wrappers root>/hooks.
> 
> My proptotype is at...
> 
>     /home/gdbadmin/brobecke/py3-git-hooks/git-hooks-wrapper
> 
> ... for those who are curious and can log onto sourceware.org.
> There is a README.md file that gives a bit more information
> (I'll attach to this message as well)
> 
> If agreed with the approach, I think we'll want to decide where
> we want this to live. Maybe /sourceware/projects/src-home/?
> The main issue with this location is that I worry about shared
> access and in particular maintenance over time. I'd like to be
> able to make changes there if there is a need to fix something.
> I'd like also to be able to continue updating the git-hooks version
> being used as I continue to push fixes and enhancements.
> 
> I can take care of the git-hooks-wrapper setup once the location
> is settled.
> 
> In terms of the projects themselves, I've identified the following
> repositories as using the git-hooks:
> 
>   - binutils-gdb.git/hooks
>   - gcc.git/hooks
>   - glibc.git/hooks
>   - newlib-cygwin.git/hooks
>   - valgrind.git/hooks
> 
> This is based on searching for repositories on sourcware.org
> whose hooks dir is a symlink to the git-hooks. Not sure if there are
> others.
> 
> I've Cc'ed people I know have asked me about the git-hooks for
> GCC, glibc and newlib. Not sure about valgrind.
> 
> What we could do is give the binutils & GDB maintainers a quick
> heads up and then transition the binutils-gdb repository. If all
> goes well, I think we can transition the remaining ones.
> 
> Thoughts?
> 
> PS: If a project needs to stay on Python 2.7 for the time being,
>     it is easy to accomodate it with the current scheme. But they
>     will need to use a dedicated branch, as the latest no longer
>     supports Python 2.x.
> -- 
> Joel

> Git-Hooks Wrapper
> =================
> 
> This repository provices a hooks/ directory which contain the same set of
> scripts as the AdaCore git-hooks (https://github.com/adacore/git-hooks).
> Those scripts simply act as wrappers allowing us to set the environment
> up prior to calling the real git-hooks hooks. In particular, our wrappers
> allow us to make sure we use a controlled version of Python to run the hooks.
> 
> Wrapper Setup Overview
> ----------------------
> 
> The wrappers expect the following structure:
> 
>     <wrappers root>/
>         - hooks/           <-- directory to use as hooks in Git bare repos
>         - git-hooks/       <-- clone of https://github.com/adacore/git-hooks
>         - python3-for-git-hooks/  <-- Python 3.x venv to be used by the git-hooks
> 
> The wrappers provides wrapper scripts which will do the following:
> 
>   - set the environment up to put the python3-for-git-hooks virtual env
>     in its PATH;
> 
>   - Call the corresponding hook in git-hooks/hooks/.
> 
> 
> Git-Hooks Setup
> ---------------
> 
> The wrappers expect a clone of the git-hooks to be located at
> the root of this repository, named git-hooks.
> 
> 
> Python Setup
> ------------
> 
> The wrappers expect a Python 3 Virtual Enviroment (venv) to be set up
> at the root of this repository, named python3-for-git-hooks.
> 
> The venv should be set up as follow:
> 
>    # Create the venv, and activate it...
> 
>    $ python3 -m venv python3-for-git-hooks
>    $ . ./python3-for-git-hooks/bin/activate
> 
>    # Make sure we use the latest version of pip
> 
>    $ python3 -m pip install pip --upgrade
> 
>    # Install all dependencies requried by the git-hooks.
>    #
>    # Note that the exact list of dependencies may change
>    # over time. If unsure, run the git-hooks testsuite
>    # as follow:
>    #
>    #      $ cd path/to/git-hooks/testsuite
>    #      $ ./run-validation-tests
>    #
>    # The testsuite will abort with an error if any dependency
>    # is missing.
> 
>    $ python3 -m pip install black e3-core flake8 pre-commit pytest pytest-cov pytest-xdist
> 
> Check the git-hooks' README.md file for the required minimum version
> of Python that must be used.
> 
> Activating the Git-Hooks Wrapper
> --------------------------------
> 
> To use these wrappers in a Git repository, replace the hooks/ directory
> inside that Git repository by a symbolic link to the hooks/ directory
> inside the git-hooks-wrapper

Re: transition git-hooks on sourceware.org to Python 3.x

[Corinna Vinschen]

On Dec 12 15:32, Joel Brobecker via Overseers wrote:
> Hello,
> 
> The current version of the git-hooks being used on sourceware
> is a version which uses Python 2.7. I have since then migrated
> to Python 3, where the minimum version is currently Python 3.8.
> That version has been running on AdaCore's server for a few weeks,
> now, so I think it's time to think about transitioning our repos
> on sourceware.
> 
> The main improvement, besides the transition away from Python 2.x,
> comes from Python 3.x itself and the handling of strings vs bytes,
> so charsets are handled a bit better.
> 
> The one small difficulty comes from making sure a recent version
> of Python 3.x gets used. Ideally, we should use the same Python
> I'm using on sourceware to run the git-hooks testsuite to make sure
> there aren't any compatibility issues.

Not sure I understand this... wouldn't it make sense to run the
hooks and the testsuite using the default python3 release 3.6.8
installed on sware in the first place?

> For this, what I'd like to propose is that we use a set of wrapper
> scripts which provide the same hooks as the git-hooks, and which
> only do the following: add the correct Python to the PATH, and
> then call the real hook script in git-hooks.
> 
> I've created a small prototype and did some testing with it,
> it seems to work well. In a nutshell, we have:
> 
>   <wrappers root>/
>       - hooks/           <-- directory to use as hooks in Git bare repos
>       - git-hooks/       <-- clone of https://github.com/adacore/git-hooks
>       - python3-for-git-hooks/  <-- Python 3.x venv to be used by the git-hooks

That would also avoid this wrapper stuff, which seems fragile.

> [...]
> In terms of the projects themselves, I've identified the following
> repositories as using the git-hooks:
> 
>   - binutils-gdb.git/hooks
>   - gcc.git/hooks
>   - glibc.git/hooks
>   - newlib-cygwin.git/hooks
>   - valgrind.git/hooks
> 
> This is based on searching for repositories on sourcware.org
> whose hooks dir is a symlink to the git-hooks. Not sure if there are
> others.
> 
> I've Cc'ed people I know have asked me about the git-hooks for
> GCC, glibc and newlib. Not sure about valgrind.
> 
> What we could do is give the binutils & GDB maintainers a quick
> heads up and then transition the binutils-gdb repository. If all
> goes well, I think we can transition the remaining ones.

Given the hooks are shared anyway, I don't see much of a problem,
except for the non-shared local scripts in hooks-bin.  These are
mostly shell scripts and shouldn't be affected.

Only email_to.py is a python2 script. 2to3(*) only requests 2 simple
changes.  Is there more to it than that?  Does it even matter?


Corinna

(*) Would it make sense to install 2to3 on sware?

Re: transition git-hooks on sourceware.org to Python 3.x

[Joel Brobecker]

> > The current version of the git-hooks being used on sourceware
> > is a version which uses Python 2.7. I have since then migrated
> > to Python 3, where the minimum version is currently Python 3.8.
> > That version has been running on AdaCore's server for a few weeks,
> > now, so I think it's time to think about transitioning our repos
> > on sourceware.
> > 
> > The main improvement, besides the transition away from Python 2.x,
> > comes from Python 3.x itself and the handling of strings vs bytes,
> > so charsets are handled a bit better.
> > 
> > The one small difficulty comes from making sure a recent version
> > of Python 3.x gets used. Ideally, we should use the same Python
> > I'm using on sourceware to run the git-hooks testsuite to make sure
> > there aren't any compatibility issues.
> 
> Not sure I understand this... wouldn't it make sense to run the
> hooks and the testsuite using the default python3 release 3.6.8
> installed on sware in the first place?

This version of Python is too old. The testing needs e3-core,
which only supports 3.8.

> > For this, what I'd like to propose is that we use a set of wrapper
> > scripts which provide the same hooks as the git-hooks, and which
> > only do the following: add the correct Python to the PATH, and
> > then call the real hook script in git-hooks.
> > 
> > I've created a small prototype and did some testing with it,
> > it seems to work well. In a nutshell, we have:
> > 
> >   <wrappers root>/
> >       - hooks/           <-- directory to use as hooks in Git bare repos
> >       - git-hooks/       <-- clone of https://github.com/adacore/git-hooks
> >       - python3-for-git-hooks/  <-- Python 3.x venv to be used by the git-hooks
> 
> That would also avoid this wrapper stuff, which seems fragile.
> 
> > [...]
> > In terms of the projects themselves, I've identified the following
> > repositories as using the git-hooks:
> > 
> >   - binutils-gdb.git/hooks
> >   - gcc.git/hooks
> >   - glibc.git/hooks
> >   - newlib-cygwin.git/hooks
> >   - valgrind.git/hooks
> > 
> > This is based on searching for repositories on sourcware.org
> > whose hooks dir is a symlink to the git-hooks. Not sure if there are
> > others.
> > 
> > I've Cc'ed people I know have asked me about the git-hooks for
> > GCC, glibc and newlib. Not sure about valgrind.
> > 
> > What we could do is give the binutils & GDB maintainers a quick
> > heads up and then transition the binutils-gdb repository. If all
> > goes well, I think we can transition the remaining ones.
> 
> Given the hooks are shared anyway, I don't see much of a problem,
> except for the non-shared local scripts in hooks-bin.  These are
> mostly shell scripts and shouldn't be affected.
> 
> Only email_to.py is a python2 script. 2to3(*) only requests 2 simple
> changes.  Is there more to it than that?  Does it even matter?

Good point. We'll need to look at this script. I believe we can
do that prior to the transition.

-- 
Joel

Re: transition git-hooks on sourceware.org to Python 3.x

[Joseph Myers]

On Sun, 12 Dec 2021, Joel Brobecker via Overseers wrote:

> For this, what I'd like to propose is that we use a set of wrapper
> scripts which provide the same hooks as the git-hooks, and which
> only do the following: add the correct Python to the PATH, and
> then call the real hook script in git-hooks.

Adding to PATH, or other such variables, seems like it would also affect 
external programs in hooks-bin run via configuration settings in the hooks 
configuration (where those use "#! /usr/bin/env python3" or "#! 
/usr/bin/env python", for example).  Have you concluded that this will 
work OK for the existing hooks-bin code for the various repositories 
(including the use of packages in 
/home/gccadmin/.local/lib/python3.6/site-packages from GCC's 
commit_checker script)?

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: transition git-hooks on sourceware.org to Python 3.x

[Joel Brobecker]

> > For this, what I'd like to propose is that we use a set of wrapper
> > scripts which provide the same hooks as the git-hooks, and which
> > only do the following: add the correct Python to the PATH, and
> > then call the real hook script in git-hooks.
> 
> Adding to PATH, or other such variables, seems like it would also affect 
> external programs in hooks-bin run via configuration settings in the hooks 
> configuration (where those use "#! /usr/bin/env python3" or "#! 
> /usr/bin/env python", for example).  Have you concluded that this will 
> work OK for the existing hooks-bin code for the various repositories 
> (including the use of packages in 
> /home/gccadmin/.local/lib/python3.6/site-packages from GCC's 
> commit_checker script)?

I haven't no. I can do some testing if you tell me what testing
needs to be done. For now, I haven't been able to see the scripts
because they are in a location I do not have permissions to read.
Or, alternatively, we can agree on the location of the git-wrappers,
I can prepare a first version of the venv, and let you know when
it's ready, so you can do some testing, and then migrate to that
when you have done sufficient testing.

For the .local/lib/python3.6/site-packages, I propose we include
them into the venv that we create for the wrapper. The README.md
gives a list of all the Python packages we want to have, so I can
add those used by GCC.

-- 
Joel

Re: transition git-hooks on sourceware.org to Python 3.x

[Ivo Raisr]

Dear,

Thank you for the heads up.
Indeed, I did set up the git-hooks some time ago for the Valgrind
repository.
But I don't remember the exact details; I assume it was a very simple setup.
Is there any action required from my side at this point?

Kind regards,
Ivo Raisr



Il giorno lun 13 dic 2021 alle ore 10:23 Mark Wielaard <mark@klomp.org> ha
scritto:

> Hi,
>
> I added Ivo to the CC who setup the valgrind hooks. If he isn't
> available I'll take care of them.
>
> Cheers,
>
> Mark
>
> On Sun, Dec 12, 2021 at 03:32:47PM +0400, Joel Brobecker via Overseers
> wrote:
> > Hello,
> >
> > The current version of the git-hooks being used on sourceware
> > is a version which uses Python 2.7. I have since then migrated
> > to Python 3, where the minimum version is currently Python 3.8.
> > That version has been running on AdaCore's server for a few weeks,
> > now, so I think it's time to think about transitioning our repos
> > on sourceware.
> >
> > The main improvement, besides the transition away from Python 2.x,
> > comes from Python 3.x itself and the handling of strings vs bytes,
> > so charsets are handled a bit better.
> >
> > The one small difficulty comes from making sure a recent version
> > of Python 3.x gets used. Ideally, we should use the same Python
> > I'm using on sourceware to run the git-hooks testsuite to make sure
> > there aren't any compatibility issues.
> >
> > For this, what I'd like to propose is that we use a set of wrapper
> > scripts which provide the same hooks as the git-hooks, and which
> > only do the following: add the correct Python to the PATH, and
> > then call the real hook script in git-hooks.
> >
> > I've created a small prototype and did some testing with it,
> > it seems to work well. In a nutshell, we have:
> >
> >   <wrappers root>/
> >       - hooks/           <-- directory to use as hooks in Git bare repos
> >       - git-hooks/       <-- clone of
> https://github.com/adacore/git-hooks
> >       - python3-for-git-hooks/  <-- Python 3.x venv to be used by the
> git-hooks
> >
> > To transition a repository to updated hooks, we simply replace
> > the hooks symbolic link to point to <wrappers root>/hooks.
> >
> > My proptotype is at...
> >
> >     /home/gdbadmin/brobecke/py3-git-hooks/git-hooks-wrapper
> >
> > ... for those who are curious and can log onto sourceware.org.
> > There is a README.md file that gives a bit more information
> > (I'll attach to this message as well)
> >
> > If agreed with the approach, I think we'll want to decide where
> > we want this to live. Maybe /sourceware/projects/src-home/?
> > The main issue with this location is that I worry about shared
> > access and in particular maintenance over time. I'd like to be
> > able to make changes there if there is a need to fix something.
> > I'd like also to be able to continue updating the git-hooks version
> > being used as I continue to push fixes and enhancements.
> >
> > I can take care of the git-hooks-wrapper setup once the location
> > is settled.
> >
> > In terms of the projects themselves, I've identified the following
> > repositories as using the git-hooks:
> >
> >   - binutils-gdb.git/hooks
> >   - gcc.git/hooks
> >   - glibc.git/hooks
> >   - newlib-cygwin.git/hooks
> >   - valgrind.git/hooks
> >
> > This is based on searching for repositories on sourcware.org
> > whose hooks dir is a symlink to the git-hooks. Not sure if there are
> > others.
> >
> > I've Cc'ed people I know have asked me about the git-hooks for
> > GCC, glibc and newlib. Not sure about valgrind.
> >
> > What we could do is give the binutils & GDB maintainers a quick
> > heads up and then transition the binutils-gdb repository. If all
> > goes well, I think we can transition the remaining ones.
> >
> > Thoughts?
> >
> > PS: If a project needs to stay on Python 2.7 for the time being,
> >     it is easy to accomodate it with the current scheme. But they
> >     will need to use a dedicated branch, as the latest no longer
> >     supports Python 2.x.
> > --
> > Joel
>
> > Git-Hooks Wrapper
> > =================
> >
> > This repository provices a hooks/ directory which contain the same set of
> > scripts as the AdaCore git-hooks (https://github.com/adacore/git-hooks).
> > Those scripts simply act as wrappers allowing us to set the environment
> > up prior to calling the real git-hooks hooks. In particular, our wrappers
> > allow us to make sure we use a controlled version of Python to run the
> hooks.
> >
> > Wrapper Setup Overview
> > ----------------------
> >
> > The wrappers expect the following structure:
> >
> >     <wrappers root>/
> >         - hooks/           <-- directory to use as hooks in Git bare
> repos
> >         - git-hooks/       <-- clone of
> https://github.com/adacore/git-hooks
> >         - python3-for-git-hooks/  <-- Python 3.x venv to be used by the
> git-hooks
> >
> > The wrappers provides wrapper scripts which will do the following:
> >
> >   - set the environment up to put the python3-for-git-hooks virtual env
> >     in its PATH;
> >
> >   - Call the corresponding hook in git-hooks/hooks/.
> >
> >
> > Git-Hooks Setup
> > ---------------
> >
> > The wrappers expect a clone of the git-hooks to be located at
> > the root of this repository, named git-hooks.
> >
> >
> > Python Setup
> > ------------
> >
> > The wrappers expect a Python 3 Virtual Enviroment (venv) to be set up
> > at the root of this repository, named python3-for-git-hooks.
> >
> > The venv should be set up as follow:
> >
> >    # Create the venv, and activate it...
> >
> >    $ python3 -m venv python3-for-git-hooks
> >    $ . ./python3-for-git-hooks/bin/activate
> >
> >    # Make sure we use the latest version of pip
> >
> >    $ python3 -m pip install pip --upgrade
> >
> >    # Install all dependencies requried by the git-hooks.
> >    #
> >    # Note that the exact list of dependencies may change
> >    # over time. If unsure, run the git-hooks testsuite
> >    # as follow:
> >    #
> >    #      $ cd path/to/git-hooks/testsuite
> >    #      $ ./run-validation-tests
> >    #
> >    # The testsuite will abort with an error if any dependency
> >    # is missing.
> >
> >    $ python3 -m pip install black e3-core flake8 pre-commit pytest
> pytest-cov pytest-xdist
> >
> > Check the git-hooks' README.md file for the required minimum version
> > of Python that must be used.
> >
> > Activating the Git-Hooks Wrapper
> > --------------------------------
> >
> > To use these wrappers in a Git repository, replace the hooks/ directory
> > inside that Git repository by a symbolic link to the hooks/ directory
> > inside the git-hooks-wrapper
>
>

Re: transition git-hooks on sourceware.org to Python 3.x

[Joel Brobecker]

Hi Ivo, [All, anyone remember who wrote or who maintains irkerhook.py?]

> Thank you for the heads up.
> Indeed, I did set up the git-hooks some time ago for the Valgrind
> repository.
> But I don't remember the exact details; I assume it was a very simple setup.
> Is there any action required from my side at this point?

I took a look at your repository's configuration, and as far
as I can tell, you have no Python involved. Or at least not
directly involved: There is a dependency on the irkerhook.py
script, which is written in Python. However, the shebang explicitly
says /usr/bin/env python2, so we can transition the git-hooks
and that irkerhook.py script independently.

-- 
Joel

Re: transition git-hooks on sourceware.org to Python 3.x

[Joel Brobecker]

Hello,

> I've created a small prototype and did some testing with it,
> it seems to work well. In a nutshell, we have:
> 
>   <wrappers root>/
>       - hooks/           <-- directory to use as hooks in Git bare repos
>       - git-hooks/       <-- clone of https://github.com/adacore/git-hooks
>       - python3-for-git-hooks/  <-- Python 3.x venv to be used by the git-hooks
> 
> To transition a repository to updated hooks, we simply replace
> the hooks symbolic link to point to <wrappers root>/hooks.
> 
> My proptotype is at...
> 
>     /home/gdbadmin/brobecke/py3-git-hooks/git-hooks-wrapper
> 
> ... for those who are curious and can log onto sourceware.org.
> There is a README.md file that gives a bit more information
> (I'll attach to this message as well)
> 
> If agreed with the approach, I think we'll want to decide where
> we want this to live. Maybe /sourceware/projects/src-home/?

If there are no opinions, I'll use gdbadmin's home. I don't think
this is the ideal location, but that's one where I currently have
access to.

> The main issue with this location is that I worry about shared
> access and in particular maintenance over time. I'd like to be
> able to make changes there if there is a need to fix something.
> I'd like also to be able to continue updating the git-hooks version
> being used as I continue to push fixes and enhancements.

Thank you,
-- 
Joel

Re: transition git-hooks on sourceware.org to Python 3.x

[Joel Brobecker]

Quick update:

I have set up the hooks wrappers at the following location:

    /home/gdbadmin/shared-scripts/git-hooks-wrapper

For the record, here is the list of commands I used to create it:

    # Clone the git-hooks-wrapper scrips

      $ mkdir shared-scripts
      $ cd shared-scripts/
      $ git clone $HOME/GDBADMIN-BARE-REPOS/git-hooks-wrapper/
      $ cd git-hooks-wrapper/

    # Add a Python 3.9 install, and then the venv for the wrappers:

      $ tar xf ~/brobecke/py3-git-hooks/python3.9-3.9.7-x86_64-linux-bin.tar.gz
      $ mv python3.9-3.9.7-x86_64-linux-bin/ python-3.9.7
      $ export PATH=`pwd`/python-3.9.7/bin:$PATH
      $ python3 -m venv python3-for-git-hooks
      $ . ./python3-for-git-hooks/bin/activate
      $ python3 -m pip install pip --upgrade
      $ python3 -m pip install black e3-core flake8 pre-commit \
          pytest pytest-cov pytest-xdist

    # Clone the git-hooks, and run the validation test to make sure
    # the Python venv has everything the git-hooks need (note that
    # this doesn't include the wrappers in the testing, unfortunately).
    #
    # I might look at enhancing the testsuite to include the use of
    # those wrappers instead of the hooks themselves, at some point,
    # but it's a bit of work, so only if we find the regular need to
    # test the wrappers.

      $ git clone https://github.com/adacore/git-hooks
      $ cd git-hooks/testsuite/
      $ ./run-validation-tests

My plan is to transition the binutils-gdb repository to it on Monday
morning (European timezone). To do so, what I will do is:

    $ /git/binutils-gdb.git
    $ mv hooks{,.legacy-python2} \
       && ln -s /home/gdbadmin/shared-scripts/git-hooks-wrapper/hooks hooks

I'll monitor the hooks behavior throughout the end of the year,
and if all goes well, I suggest other projects who are ready
(after reviewing their python scripts used by their project's config)
do the same. I can provide assistance towards the transition,
if you'd like.

-- 
Joel

Re: transition git-hooks on sourceware.org to Python 3.x

[Joel Brobecker]

> My plan is to transition the binutils-gdb repository to it on Monday
> morning (European timezone). To do so, what I will do is:
> 
>     $ /git/binutils-gdb.git
>     $ mv hooks{,.legacy-python2} \
>        && ln -s /home/gdbadmin/shared-scripts/git-hooks-wrapper/hooks hooks

This is now done.

I tested with a first simple patch push, and realized I had forgotten
to update ...

    /git/binutils-gdb.git/hooks-bin

... first (missing Python3-ization of email_to.py). So I got an error
during the emailing, while the push went through.

I therefore did...

    | % cd /git/binutils-gdb.git/hooks-bin/
    | % git pull --rebase
    | remote: Enumerating objects: 5, done.
    | remote: Counting objects: 100% (5/5), done.
    | remote: Compressing objects: 100% (3/3), done.
    | remote: Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
    | Unpacking objects: 100% (3/3), 329 bytes | 41.00 KiB/s, done.
    | From /home/gdbadmin/GDBADMIN-BARE-REPOS/hooks-bin
    |    0018300..0d31c4b  master     -> origin/master
    | Updating 0018300..0d31c4b
    | Fast-forward
    |  email_to.py | 4 ++--
    |  1 file changed, 2 insertions(+), 2 deletions(-)
    | Current branch master is up to date.

... found another trivial change I could make, and pushed it,
this time with no error.

I double-checked the email for my commit has been sent to gdb-cvs:
https://sourceware.org/pipermail/gdb-cvs/2021-December/052054.html

I'll warn the users on gdb-patches to let me know if they see any issue.

-- 
Joel

Re: transition git-hooks on sourceware.org to Python 3.x

[Joel Brobecker]

Hi everyone,

Happy New Year!

For binutils-gdb, the transition has been made a few weeks ago,
and so far so good :). So I'm inviting you all to transition
this new version at your convenience.

Remember that, if your project's configuration calls some Python
hooks, you'll probably want to double-check that they are compatible
with Python 3.x!

Cheers :)

On Sun, Dec 19, 2021 at 08:45:39AM +0400, Joel Brobecker wrote:
> Quick update:
> 
> I have set up the hooks wrappers at the following location:
> 
>     /home/gdbadmin/shared-scripts/git-hooks-wrapper
> 
> For the record, here is the list of commands I used to create it:
> 
>     # Clone the git-hooks-wrapper scrips
> 
>       $ mkdir shared-scripts
>       $ cd shared-scripts/
>       $ git clone $HOME/GDBADMIN-BARE-REPOS/git-hooks-wrapper/
>       $ cd git-hooks-wrapper/
> 
>     # Add a Python 3.9 install, and then the venv for the wrappers:
> 
>       $ tar xf ~/brobecke/py3-git-hooks/python3.9-3.9.7-x86_64-linux-bin.tar.gz
>       $ mv python3.9-3.9.7-x86_64-linux-bin/ python-3.9.7
>       $ export PATH=`pwd`/python-3.9.7/bin:$PATH
>       $ python3 -m venv python3-for-git-hooks
>       $ . ./python3-for-git-hooks/bin/activate
>       $ python3 -m pip install pip --upgrade
>       $ python3 -m pip install black e3-core flake8 pre-commit \
>           pytest pytest-cov pytest-xdist
> 
>     # Clone the git-hooks, and run the validation test to make sure
>     # the Python venv has everything the git-hooks need (note that
>     # this doesn't include the wrappers in the testing, unfortunately).
>     #
>     # I might look at enhancing the testsuite to include the use of
>     # those wrappers instead of the hooks themselves, at some point,
>     # but it's a bit of work, so only if we find the regular need to
>     # test the wrappers.
> 
>       $ git clone https://github.com/adacore/git-hooks
>       $ cd git-hooks/testsuite/
>       $ ./run-validation-tests
> 
> My plan is to transition the binutils-gdb repository to it on Monday
> morning (European timezone). To do so, what I will do is:
> 
>     $ /git/binutils-gdb.git
>     $ mv hooks{,.legacy-python2} \
>        && ln -s /home/gdbadmin/shared-scripts/git-hooks-wrapper/hooks hooks
> 
> I'll monitor the hooks behavior throughout the end of the year,
> and if all goes well, I suggest other projects who are ready
> (after reviewing their python scripts used by their project's config)
> do the same. I can provide assistance towards the transition,
> if you'd like.
> 
> -- 
> Joel

-- 
Joel

Re: transition git-hooks on sourceware.org to Python 3.x

[Corinna Vinschen]

On Jan 16 14:06, Joel Brobecker via Overseers wrote:
> Hi everyone,
> 
> Happy New Year!
> 
> For binutils-gdb, the transition has been made a few weeks ago,
> and so far so good :). So I'm inviting you all to transition
> this new version at your convenience.
> 
> Remember that, if your project's configuration calls some Python
> hooks, you'll probably want to double-check that they are compatible
> with Python 3.x!

I did the same for newlib-cygwin.git just a couple of minutes ago.
Hopefully this works as nicely as for biunutils-gdb :}


Corinna

> 
> Cheers :)
> 
> On Sun, Dec 19, 2021 at 08:45:39AM +0400, Joel Brobecker wrote:
> > Quick update:
> > 
> > I have set up the hooks wrappers at the following location:
> > 
> >     /home/gdbadmin/shared-scripts/git-hooks-wrapper
> > 
> > For the record, here is the list of commands I used to create it:
> > 
> >     # Clone the git-hooks-wrapper scrips
> > 
> >       $ mkdir shared-scripts
> >       $ cd shared-scripts/
> >       $ git clone $HOME/GDBADMIN-BARE-REPOS/git-hooks-wrapper/
> >       $ cd git-hooks-wrapper/
> > 
> >     # Add a Python 3.9 install, and then the venv for the wrappers:
> > 
> >       $ tar xf ~/brobecke/py3-git-hooks/python3.9-3.9.7-x86_64-linux-bin.tar.gz
> >       $ mv python3.9-3.9.7-x86_64-linux-bin/ python-3.9.7
> >       $ export PATH=`pwd`/python-3.9.7/bin:$PATH
> >       $ python3 -m venv python3-for-git-hooks
> >       $ . ./python3-for-git-hooks/bin/activate
> >       $ python3 -m pip install pip --upgrade
> >       $ python3 -m pip install black e3-core flake8 pre-commit \
> >           pytest pytest-cov pytest-xdist
> > 
> >     # Clone the git-hooks, and run the validation test to make sure
> >     # the Python venv has everything the git-hooks need (note that
> >     # this doesn't include the wrappers in the testing, unfortunately).
> >     #
> >     # I might look at enhancing the testsuite to include the use of
> >     # those wrappers instead of the hooks themselves, at some point,
> >     # but it's a bit of work, so only if we find the regular need to
> >     # test the wrappers.
> > 
> >       $ git clone https://github.com/adacore/git-hooks
> >       $ cd git-hooks/testsuite/
> >       $ ./run-validation-tests
> > 
> > My plan is to transition the binutils-gdb repository to it on Monday
> > morning (European timezone). To do so, what I will do is:
> > 
> >     $ /git/binutils-gdb.git
> >     $ mv hooks{,.legacy-python2} \
> >        && ln -s /home/gdbadmin/shared-scripts/git-hooks-wrapper/hooks hooks
> > 
> > I'll monitor the hooks behavior throughout the end of the year,
> > and if all goes well, I suggest other projects who are ready
> > (after reviewing their python scripts used by their project's config)
> > do the same. I can provide assistance towards the transition,
> > if you'd like.
> > 
> > -- 
> > Joel
> 
> -- 
> Joel

Re: transition git-hooks on sourceware.org to Python 3.x

[Joel Brobecker]

> > For binutils-gdb, the transition has been made a few weeks ago,
> > and so far so good :). So I'm inviting you all to transition
> > this new version at your convenience.
> > 
> > Remember that, if your project's configuration calls some Python
> > hooks, you'll probably want to double-check that they are compatible
> > with Python 3.x!
> 
> I did the same for newlib-cygwin.git just a couple of minutes ago.
> Hopefully this works as nicely as for biunutils-gdb :}

No news must be good news. Well done, Corinna :).

-- 
Joel

Re: transition git-hooks on sourceware.org to Python 3.x

[Corinna Vinschen]

On Jan 20 09:27, Joel Brobecker via Overseers wrote:
> > > For binutils-gdb, the transition has been made a few weeks ago,
> > > and so far so good :). So I'm inviting you all to transition
> > > this new version at your convenience.
> > > 
> > > Remember that, if your project's configuration calls some Python
> > > hooks, you'll probably want to double-check that they are compatible
> > > with Python 3.x!
> > 
> > I did the same for newlib-cygwin.git just a couple of minutes ago.
> > Hopefully this works as nicely as for biunutils-gdb :}
> 
> No news must be good news. Well done, Corinna :).

Actually, looks good so far.  We only had a single, local python2
script which I 2to3-d and appears to work as well..  Thanks for a
smooth transition!


Corinna

Re: transition git-hooks on sourceware.org to Python 3.x

[Joel Brobecker]

> Actually, looks good so far.  We only had a single, local python2
> script which I 2to3-d and appears to work as well..  Thanks for a
> smooth transition!

Nice to hear. Thanks Corinna!

-- 
Joel