[PATCH] Add github actions to support workflows

[PATCH] Add github actions to support workflows

[Ben Woodard]

From: vsoch <vsoch@users.noreply.github.com>

To facilitate collaboration with developers working on the BUILD-SI
project who are using github, I have been maintaining a clone of the
libabigail repo there. Having this repo also allows us to leverage
some of the tooling that github provides, in particular the ability to
have github actions test patches to verify that the builds succeed,
and all the regression tests run successfully. This will allow it to
better integrate with the normal agile workflow used by at least this
community of developers.

While containerizing the workflow we found that fedabipkgdiff had
a subtle implicit dependency on system provided mimetypes. We found
it expedient to add logic that eliminated that dependency.

	* add .github directory
	* add .github/workflows directory
	* add .github/workflows/build-container.yaml
	* add .github/workflows/libabigail.yaml
	* add .github/workflows/test.yaml
	* add docker directory
	* add docker/Dockerfile.fedora
	* add docker/Dockerfile.ubuntu
	* add docker/Dockerfile.fedora-base
	* add docker/Dockerfile.test
	* add README-DOCKER.md
	* tools/fedabipkgdiff

Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Reviewed-by: Ben Woodard <woodard@redhat.com>
---
 .github/workflows/build-container.yaml |  99 +++++++++++++++++++
 .github/workflows/libabigail.yaml      | 126 +++++++++++++++++++++++++
 .github/workflows/test.yaml            |  34 +++++++
 README-DOCKER.md                       |  66 +++++++++++++
 docker/Dockerfile.fedora               |  38 ++++++++
 docker/Dockerfile.fedora-base          |  21 +++++
 docker/Dockerfile.test                 |  28 ++++++
 docker/Dockerfile.ubuntu               |  30 ++++++
 tools/fedabipkgdiff                    |   4 +
 9 files changed, 446 insertions(+)
 create mode 100644 .github/workflows/build-container.yaml
 create mode 100644 .github/workflows/libabigail.yaml
 create mode 100644 .github/workflows/test.yaml
 create mode 100644 README-DOCKER.md
 create mode 100644 docker/Dockerfile.fedora
 create mode 100644 docker/Dockerfile.fedora-base
 create mode 100644 docker/Dockerfile.test
 create mode 100644 docker/Dockerfile.ubuntu

diff --git a/.github/workflows/build-container.yaml b/.github/workflows/build-container.yaml
new file mode 100644
index 00000000..b2b50e27
--- /dev/null
+++ b/.github/workflows/build-container.yaml
@@ -0,0 +1,99 @@
+name: Build Deploy Container
+
+on:
+
+  # Always have a base image ready to go - this is a nightly build
+  schedule:
+    - cron: 0 3 * * *
+
+  # Allow manual trigger of a build
+  workflow_dispatch:
+    inputs:
+      whatever:
+        description: 'This variable does not matter, its a GHA bug.'
+
+  # On push to main we build and deploy images
+  push:
+    branches:
+      - master
+
+  # Do build to test works on PR
+  pull_request: []
+
+  # Publish packages on release
+  release:
+    types: [published]
+
+jobs:
+  build:
+    permissions:
+      packages: write
+    strategy:
+      fail-fast: false
+      matrix:
+
+        # Dockerfiles to build, container names to use, and to tag as libabigail:latest?
+        container: [["Dockerfile.fedora", "ghcr.io/woodard/libabigail-fedora", true],
+                    ["Dockerfile.ubuntu", "ghcr.io/woodard/libabigail-ubuntu-22.04", false],
+                    ["Dockerfile.fedora-base", "ghcr.io/woodard/libabigail-fedora-base", false]]
+
+    runs-on: ubuntu-latest
+    name: Build
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+
+      - name: Make Space For Build
+        run: |
+          sudo rm -rf /usr/share/dotnet
+          sudo rm -rf /opt/ghc
+
+       # It's easier to reference named variables than indexes of the matrix
+      - name: Set Environment
+        env:
+          dockerfile: ${{ matrix.container[0] }}
+          uri: ${{ matrix.container[1] }}
+          isLatest: ${{ matrix.container[2] }}
+        run: |
+          echo "dockerfile=$dockerfile" >> $GITHUB_ENV
+          echo "uri=$uri" >> $GITHUB_ENV
+          echo "isLatest=$isLatest" >> $GITHUB_ENV
+
+      - name: Pull previous layers for cache
+        run: docker pull ${uri}:latest || echo "No container to pull"
+
+      - name: Build Container
+        run: |
+           container=$uri:latest
+           docker build -f docker/${dockerfile} -t ${container} .
+           if [[ "${isLatest}" == "true" ]]; then               docker tag ${container} ghcr.io/woodard/libabigail:latest
+           fi
+           echo "container=$container" >> $GITHUB_ENV
+
+      - name: GHCR Login
+        if: (github.event_name != 'pull_request')
+        uses: docker/login-action@v1
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Deploy
+        if: (github.event_name != 'pull_request')
+        run: |
+            docker push ${container}
+            if [[ "${isLatest}" == "true" ]]; then
+               docker push ghcr.io/woodard/libabigail:latest
+            fi
+
+      - name: Tag and Push Release
+        if: (github.event_name == 'release')
+        run: |
+            tag=${GITHUB_REF#refs/tags/}
+            echo "Tagging and releasing ${uri}:${tag}"
+            docker tag ${uri}:latest ${uri}:${tag}
+            docker push ${uri}:${tag}
+            if [[ "${isLatest}" == "true" ]]; then
+                docker tag ${uri}:latest ghcr.io/woodard/libabigail:${tag}
+                docker push ghcr.io/woodard/libabigail:${tag}
+            fi
diff --git a/.github/workflows/libabigail.yaml b/.github/workflows/libabigail.yaml
new file mode 100644
index 00000000..a7c40d27
--- /dev/null
+++ b/.github/workflows/libabigail.yaml
@@ -0,0 +1,126 @@
+name: Libabigail ABI Checks
+on:
+  pull_request: []
+
+jobs:
+
+#  get-release:
+#    container: ghcr.io/woodard/libabigail-ubuntu-22.04:latest
+#    runs-on: ubuntu-latest
+#    steps:
+#    - name: Organize Files
+#      run: |
+#        mkdir -p /abi
+#        cp /opt/abigail-env/.spack-env/view/lib/libabigail.so /abi/
+#        cp /lib/x86_64-linux-gnu/libc.so.6 /abi/
+#        cp /lib/x86_64-linux-gnu/libstdc++.so.6 /abi/
+#        cp /lib/x86_64-linux-gnu/libsystemd.so.0 /abi/
+
+#    - name: Upload Libs
+#      uses: actions/upload-artifact@v3
+#      with:
+#        name: release-libs
+#        path: |
+#          /abi/libabigail.so
+#          /abi/libc.so.6
+#          /abi/libstdc++.so.6
+#          /abi/libsystemd.so.0
+
+
+  get-latest:
+    container: ghcr.io/woodard/libabigail-ubuntu-22.04:latest
+    runs-on: ubuntu-latest
+    steps:
+    - name: Organize Files
+      run: |
+        mkdir -p /abi
+        cp /opt/abigail-env/.spack-env/view/lib/libabigail.so /abi/
+        cp /lib/x86_64-linux-gnu/libc.so.6 /abi/
+        cp /lib/x86_64-linux-gnu/libstdc++.so.6 /abi/
+        cp /lib/x86_64-linux-gnu/libsystemd.so.0 /abi/
+
+    - name: Upload Libs
+      uses: actions/upload-artifact@v3
+      with:
+        name: latest-libs
+        path: |
+          /abi/libabigail.so
+          /abi/libc.so.6
+          /abi/libstdc++.so.6
+          /abi/libsystemd.so.0
+
+  get-pr:
+    container: ghcr.io/woodard/libabigail-ubuntu-22.04:latest
+    runs-on: ubuntu-latest
+    steps:
+    - name: Build Pull Request
+      uses: actions/checkout@v3
+    - name: Build
+      run: |
+         rm -rf /src
+         cp -R $PWD /src
+         ls /src
+         . /opt/spack/share/spack/setup-env.sh
+         cd /opt/abigail-env
+         spack install
+
+    - name: Organize Files
+      run: |
+        mkdir -p /abi
+        cp /opt/abigail-env/.spack-env/view/lib/libabigail.so /abi/
+        cp /lib/x86_64-linux-gnu/libc.so.6 /abi/
+        cp /lib/x86_64-linux-gnu/libstdc++.so.6 /abi/
+        cp /lib/x86_64-linux-gnu/libsystemd.so.0 /abi/
+
+    - name: Upload Libs
+      uses: actions/upload-artifact@v3
+      with:
+        name: pr-libs
+        path: |
+          /abi/libabigail.so
+          /abi/libc.so.6
+          /abi/libstdc++.so.6
+          /abi/libsystemd.so.0
+
+  abi:
+    runs-on: ubuntu-latest
+    needs: [get-latest, get-pr] # get-release
+    strategy:
+      fail-fast: false
+      matrix:
+
+        # Testing every paired library for release vs pr and main vs. pr
+        libs: ["libabigail.so",
+               "libc.so.6",
+               "libstdc++.so.6",
+               "libsystemd.so.0"]
+
+        # Artifact pairs (named) for comparison)
+        artifacts: [["pr-libs", "latest-libs"]]
+                    #["pr-libs", "release-libs"]]
+
+    steps:
+    - name: Download Previous Version
+      uses: actions/download-artifact@v2
+      with:
+        name: ${{ matrix.artifacts[1] }}
+        path: previous/
+
+    - name: Download Pull Request Version
+      uses: actions/download-artifact@v2
+      with:
+        name: ${{ matrix.artifacts[0] }}
+        path: current/
+
+    - name: Show Files
+      run: |
+        ls current/
+        ls previous/
+
+    - name: Run Libabigail
+      uses: buildsi/libabigail-action@0.0.11
+      env:
+        lib: ${{ matrix.libs }}
+      with:
+        abidiff: previous/${{ env.lib }} current/${{ env.lib }}
+        abidw: "--abidiff current/${{ env.lib }}"
diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml
new file mode 100644
index 00000000..f26f08cc
--- /dev/null
+++ b/.github/workflows/test.yaml
@@ -0,0 +1,34 @@
+name: Test Libabigail
+
+on:
+  pull_request: []
+  push:
+    branches:
+      - master
+
+jobs:
+  build:
+    permissions:
+      packages: read
+    strategy:
+      fail-fast: false
+      matrix:
+
+        # Testing containers (build and run make check)
+        dockerfile: ["Dockerfile.test"]
+
+    runs-on: ubuntu-latest
+    name: Build
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+
+      - name: Make Space For Build
+        run: |
+          sudo rm -rf /usr/share/dotnet
+          sudo rm -rf /opt/ghc
+
+      - name: Build Test Container
+        env:
+          dockerfile: ${{ matrix.dockerfile }}
+        run: docker build -f docker/${dockerfile} -t libabigail-test .
diff --git a/README-DOCKER.md b/README-DOCKER.md
new file mode 100644
index 00000000..ad2dd8e1
--- /dev/null
+++ b/README-DOCKER.md
@@ -0,0 +1,66 @@
+# Libabigail Docker
+
+Libabigail comes with two Dockerfile in [docker](docker) to build each of:
+
+ - a Fedora base image (recommended)
+ - an Ubuntu base image.
+
+These containers are built and deployed on merges to the main branch and releases.
+
+### Usage
+
+Here is how to build the containers. Note that we build so it belongs to the same
+namespace as the repository here. "ghcr.io" means "GitHub Container Registry" and
+is the [GitHub packages](https://github.com/features/packages) registry that supports
+ Docker images and other OCI artifacts.
+
+```bash
+$ docker build -f docker/Dockerfile.fedora -t ghcr.io/woodard/libabigail-fedora .
+```
+```bash
+$ docker build -f docker/Dockerfile.ubuntu -t ghcr.io/woodard/libabigail-ubuntu-22.04 .
+```
+
+Note that currently the fedora image is deployed to `ghcr.io/woodard/libabigail:latest`.
+
+### Shell
+
+To shell into a container (here is an example with ubuntu):
+
+```bash
+$ docker run -it ghcr.io/woodard/libabigail-ubuntu-22.04 bash
+```
+
+Off the bat, you can find the abi executables:
+
+```bash
+# which abidiff
+/opt/abigail-env/.spack-env/view/bin/abidiff
+```
+
+Since the ubuntu base uses spack, you can interact with spack.
+You can go to the environment in `/opt/abigail-env` and (given you
+have the source code bound to `/src`) build and test again.
+
+```bash
+$ spack install
+```
+
+And that's it! This workflow should make it easy to install development versions of libabigail with spack.
+We will also use the "production" containers to grab libraries in:
+
+```
+$ ls /opt/abigail-env/.spack-env/view/
+bin  include  lib  share
+```
+
+Note that the fedora container does not come with spack.
+
+### Testing
+
+We provide a testing container, which will use a fedora base and add new code to
+compile, and then run `make check`. You can do this as follows on your local machine:
+
+```bash
+$ docker build -f docker/Dockerfile.test -t test .
+```
diff --git a/docker/Dockerfile.fedora b/docker/Dockerfile.fedora
new file mode 100644
index 00000000..0257e99e
--- /dev/null
+++ b/docker/Dockerfile.fedora
@@ -0,0 +1,38 @@
+FROM ghcr.io/woodard/libabigail-fedora-base as builder
+
+# docker build -f docker/Dockerfile.fedora -t ghcr.io/woodard/libabigail-fedora .
+
+WORKDIR /code
+COPY . /code
+
+RUN mkdir -p build && \
+    autoreconf -i && \
+    cd build && \
+    CXXFLAGS="-g3 -fvar-tracking-assignments \
+    -gstatement-frontiers -gvariable-location-views -grecord-gcc-switches -pipe -Wall \
+    -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -Wp,-D_GLIBCXX_ASSERTIONS \
+    -fstack-protector-strong -fstack-clash-protection -fcf-protection \
+    -fasynchronous-unwind-tables -O2" CFLAGS="-g3 -fvar-tracking-assignments \
+    -gstatement-frontiers -gvariable-location-views -grecord-gcc-switches -pipe -Wall \
+    -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -Wp,-D_GLIBCXX_ASSERTIONS \
+    -fstack-protector-strong -fstack-clash-protection -fcf-protection \
+    -fasynchronous-unwind-tables -O2" LDFLAGS="-Wl,--no-undefined" \
+    ../configure --prefix=/opt/libabigail --enable-rpm=yes && \
+    make -j4 && \
+    make install
+
+FROM fedora:latest
+COPY --from=builder /opt/libabigail /opt/libabigail
+ENV PATH=/opt/libabigail/bin:$PATH
+
+# Runtime dependencies
+RUN dnf install -y \
+    python3-koji \
+    python3-mock \
+    python3-pyxdg \
+    elfutils-libelf \
+    elfutils-libs \
+    libstdc++ \
+    lbzip2 \
+    shared-mime-info \
+    six
diff --git a/docker/Dockerfile.fedora-base b/docker/Dockerfile.fedora-base
new file mode 100644
index 00000000..f803dd7f
--- /dev/null
+++ b/docker/Dockerfile.fedora-base
@@ -0,0 +1,21 @@
+ARG fedora_version=latest
+# ARG fedora_version=35
+FROM fedora:${fedora_version} as builder
+
+# docker build -f docker/Dockerfile.fedora-base -t ghcr.io/woodard/libabigail-fedora-base .
+
+RUN dnf install -y \
+    autoconf \
+    automake \
+    cpio \
+    elfutils-devel \
+    gcc-c++ \
+    lbzip2 \
+    libtool \
+    libxml2-devel \
+    python3-koji \
+    python3-mock \
+    python3-pyxdg \
+    shared-mime-info \
+    six \
+    wget
diff --git a/docker/Dockerfile.test b/docker/Dockerfile.test
new file mode 100644
index 00000000..52c6abd1
--- /dev/null
+++ b/docker/Dockerfile.test
@@ -0,0 +1,28 @@
+FROM ghcr.io/woodard/libabigail-fedora-base
+
+# docker build -f docker/Dockerfile.test -t test .
+
+RUN dnf install -y \
+    file \
+    diffutils \
+    lbzip2 \
+    elfutils-libs
+
+WORKDIR /code
+COPY . /code
+
+RUN mkdir -p build && \
+    autoreconf -i && \
+    cd build && \
+    CXXFLAGS="-g3 -fvar-tracking-assignments \
+    -gstatement-frontiers -gvariable-location-views -grecord-gcc-switches -pipe -Wall \
+    -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -Wp,-D_GLIBCXX_ASSERTIONS \
+    -fstack-protector-strong -fstack-clash-protection -fcf-protection \
+    -fasynchronous-unwind-tables -O2" CFLAGS="-g3 -fvar-tracking-assignments \
+    -gstatement-frontiers -gvariable-location-views -grecord-gcc-switches -pipe -Wall \
+    -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -Wp,-D_GLIBCXX_ASSERTIONS \
+    -fstack-protector-strong -fstack-clash-protection -fcf-protection \
+    -fasynchronous-unwind-tables -O2" LDFLAGS="-Wl,--no-undefined" \
+    ../configure --prefix=/opt/libabigail --enable-rpm=yes && \
+    make -j4 && \
+    make check || (cat /tests/test-suite.log && exit 1)
diff --git a/docker/Dockerfile.ubuntu b/docker/Dockerfile.ubuntu
new file mode 100644
index 00000000..cb1dc166
--- /dev/null
+++ b/docker/Dockerfile.ubuntu
@@ -0,0 +1,30 @@
+ARG ubuntu_version=22.04
+ARG gcc_version=10.3.0
+FROM ghcr.io/rse-ops/gcc-ubuntu-${ubuntu_version}:gcc-${gcc_version}
+
+# docker build -t ghcr.io/woodard/libabigail-ubuntu-22.04 .
+
+# Install Libabigail to its own view
+WORKDIR /opt/abigail-env
+RUN . /opt/spack/share/spack/setup-env.sh && \
+    spack env create -d . && \
+    spack env activate . && \
+    spack add libabigail@master  && \
+    spack --debug install
+
+# Prepare a source extraction of libabigail at /src (intended to be overwritten by user)
+COPY . /src
+
+# Second run - should have deps cached
+RUN . /opt/spack/share/spack/setup-env.sh && \
+
+    # This adds metadata for libabigail to spack.yaml
+    spack develop --path /src libabigail@master && \
+    spack --debug install
+    # At this point you can spack install, and bind the code to /code to develop
+
+# ensure libabigail stuffs always on the path
+RUN cd /opt/abigail-env && \
+    spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh
+
+ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l", "-c"]
diff --git a/tools/fedabipkgdiff b/tools/fedabipkgdiff
index dc80a6ef..c05bd8b1 100755
--- a/tools/fedabipkgdiff
+++ b/tools/fedabipkgdiff
@@ -213,6 +213,10 @@ def is_rpm_file(filename):
     mimetype = mimetypes.guess_type(filename)[0] if isfile else None
     isrpm = (mimetype == 'application/x-redhat-package-manager'
              or mimetype == 'application/x-rpm')
+
+    # Most systems won't have rpm defined as a mimetype
+    if not mimetype and filename.endswith('.rpm'):
+        isrpm = True
     logger.debug('is_rpm_file(\'%s\'): isfile=%s, mimetype=\'%s\', isrpm=%s',
                  filename, isfile, mimetype, isrpm)
     return isrpm
-- 
2.35.1

Re: [PATCH] Add github actions to support workflows

[Mark Wielaard]

Hi Ben,

On Fri, Apr 22, 2022 at 09:17:16AM -0700, Ben Woodard via Libabigail wrote:
[ ... application/octet-stream ...]

Aha, there are those scripts and container files. But sending the
patches as application/octet-stream causes them to be hidden in my mua
(mutt).

I'll leave the "workflow" stuff to someone who doesn't think github is
a major PITA. But I think you should submit the tools/fedabipkgdiff
change separately. It looks correct to me.

Thanks,

Mark

Re: [PATCH] Add github actions to support workflows

[Ben Woodard]

> On Apr 30, 2022, at 1:08 PM, Mark Wielaard <mark@klomp.org> wrote:
> 
> Hi Ben,
> 
> On Fri, Apr 22, 2022 at 09:17:16AM -0700, Ben Woodard via Libabigail wrote:
> [ ... application/octet-stream ...]
> 
> Aha, there are those scripts and container files. But sending the
> patches as application/octet-stream causes them to be hidden in my mua
> (mutt).
> 

Yeah I talked to Frank about that and he thinks that was a bug in Fedora 36 that day and it mime-ified the output of git send-email. I got the impression that he had seen it somewhere else earlier in the day.

I was planning to resend the patch after I do a bit more work on it.

> I'll leave the "workflow" stuff to someone who doesn't think github is
> a major PITA. But I think you should submit the tools/fedabipkgdiff
> change separately. It looks correct to me.
> 
I’ll split that out and resend it.

-ben
> Thanks,
> 
> Mark
> 

Re: [PATCH] Add github actions to support workflows

[David Marchand]

On Tue, May 3, 2022 at 2:18 AM Ben Woodard via Libabigail
<libabigail@sourceware.org> wrote:
>
>
> > On Apr 30, 2022, at 1:08 PM, Mark Wielaard <mark@klomp.org> wrote:
> >
> > Hi Ben,
> >
> > On Fri, Apr 22, 2022 at 09:17:16AM -0700, Ben Woodard via Libabigail wrote:
> > [ ... application/octet-stream ...]
> >
> > Aha, there are those scripts and container files. But sending the
> > patches as application/octet-stream causes them to be hidden in my mua
> > (mutt).
> >
>
> Yeah I talked to Frank about that and he thinks that was a bug in Fedora 36 that day and it mime-ified the output of git send-email. I got the impression that he had seen it somewhere else earlier in the day.

My two cents.

If using RH servers for mail, there was an issue, for some time like a
month, with Mimecast adding a silly "Content-type:
application/octet-stream" for mails which initially contained no such
header (additionnally, I noticed the issue only for mails which
contained a From: field in the body of the message).
This happened when both sending mails outside of RH, or when receiving
mails from outside in my experience.

The problem has been fixed (Mimecast rolled back the thing) around 04/25-04/26.

-- 
David Marchand