From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: by sourceware.org (Postfix, from userid 7840) id 7B3C83846035; Fri, 16 Apr 2021 20:36:36 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 7B3C83846035 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Eugene Rozenfeld To: gcc-cvs@gcc.gnu.org Subject: [gcc(refs/vendors/microsoft/heads/main)] Refactor gcc build and test steps (#38) X-Act-Checkin: gcc X-Git-Author: vitong <53017530+vitong@users.noreply.github.com> X-Git-Refname: refs/vendors/microsoft/heads/main X-Git-Oldrev: 1aef839c6ebcf3c5244c1d970ae2c5144e3be25d X-Git-Newrev: ff046e366cfe82197ae69ed74a4a865aa514e9c7 Message-Id: <20210416203636.7B3C83846035@sourceware.org> Date: Fri, 16 Apr 2021 20:36:36 +0000 (GMT) X-BeenThere: gcc-cvs@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-cvs mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Apr 2021 20:36:36 -0000 https://gcc.gnu.org/g:ff046e366cfe82197ae69ed74a4a865aa514e9c7 commit ff046e366cfe82197ae69ed74a4a865aa514e9c7 Author: vitong <53017530+vitong@users.noreply.github.com> Date: Fri Feb 19 10:38:58 2021 -0800 Refactor gcc build and test steps (#38) * Bring over workflow yamls and scripts to build and test gcc * Remove hack to find a particular build * Update configure script to work with new repo layout * update path to validate_failures.py * Add xfail file * Disable more failing ASAN tests * Update x86_64-pc-linux-gnu.xfail * Remove SHA paramter from DownloadBuildArtifact * Fix objdir paths * Update build-gcc with objdir path * Update objdir path when uploading artifact * Update build.yaml * Update gccWorkflow.py * Update build.yaml * Disable build-gcc.yaml * Test matrix * Add temp hack to get faster results * Add space for test-gcc.sh argument passing * Disable failfast * Add more gcc-c failures * Update x86_64-pc-linux-gnu.xfail * Temp remove hack * Update build.yaml * Move wait loop to a separate job so the test run doesn't exceed the 6h limit * Fix syntax erros * Fix missing logger def * Fix accesstoken * Debugging * More debug * Add hack to shortcut waiting on gcc build * Fix Reload method overloading * More debugigng * Don't print access token for future jobs * Zero out string in place with replace function * Fix printing * Remove shortcut hack * Remove commented out shortcut hack * Remove debugging prints * Address PR feedback * Fix error detection and reduce sleep times * Use composite for build step * Add composite branch for testing * Fix path to action * Add test composite * Temporarily add fetch-rebase to PR testing * Add test composite actioin * Temporarily comment out current ref checkout * Add python 3.7 and requests installation * Update fetch-rebase-test.yaml * Update config.py * Update action.yaml * Update action.yaml * Update test-gcc.yaml * Update test-gcc.yaml * Update gccWorkflow.py * Update fetch-rebase-test.yaml * Update build.yaml * Update build.yaml * Update test-gcc.yaml * Update config.py * Fix build being downloaded * Fix syntax * Fix inputs in yaml * Update inputs * Add missing slash * Print buildDownloaded * Add matrix chunking to fetch-rebase-test * Add dependencies yaml * Create action.yaml * Update action.yaml * Update build.yaml * Update fetch-rebase-test.yaml * Update test-gcc.yaml * Update fetch-rebase-test.yaml * Update action.yaml * Update gccWorkflow.py * Debug * Add outputs variable in fetch-rebase-test * Convert string param into bool * Increase artifact wait time to 30 minutes * Add unzipping of gcc build in fetch-rebase * Update downloadBuildArtifact.py * Debug * Add rename * Debug artifact downloading * Debug 2g * More debug * Test * Cleanup * Update branch ref * Address PR feedback * Add shell line in composite * Remove TODOs Diff: --- .github/actions/build-composite/action.yaml | 19 +++++ .github/actions/rebase-gcc-master/action.yaml | 24 ++++++ .github/actions/test-composite/action.yaml | 36 ++++++++ .github/scripts/config.py | 6 +- .github/scripts/downloadBuildArtifact.py | 21 +++-- .github/scripts/gccWorkflow.py | 44 +++++++--- .github/workflows/build.yaml | 20 ++--- .github/workflows/fetch-rebase-test.yaml | 118 +++++++++++++++++++++----- .github/workflows/test-gcc.yaml | 31 +++---- 9 files changed, 239 insertions(+), 80 deletions(-) diff --git a/.github/actions/build-composite/action.yaml b/.github/actions/build-composite/action.yaml new file mode 100644 index 00000000000..1857886a994 --- /dev/null +++ b/.github/actions/build-composite/action.yaml @@ -0,0 +1,19 @@ +# Composite yaml that's re-usable in multiple workflows to build gcc +inputs: + configjson: + description: 'Config object converted into json' + required: true + default: '' +runs: + using: "composite" + steps: + - name: Build gcc + run: | + chmod +x .github/scripts/gccWorkflow.py + echo "$PYTHONPATH" + export PYTHONPATH=${PYTHONPATH}:${PWD}/.github/scripts + echo "$PYTHONPATH" + python -c 'import sys; from gccWorkflow import *; GccWorkflow.Build(sys.argv[1])' "${CONFIG_JSON}" + shell: bash + env: + CONFIG_JSON: ${{ inputs.configJson }} \ No newline at end of file diff --git a/.github/actions/rebase-gcc-master/action.yaml b/.github/actions/rebase-gcc-master/action.yaml new file mode 100644 index 00000000000..2236536f501 --- /dev/null +++ b/.github/actions/rebase-gcc-master/action.yaml @@ -0,0 +1,24 @@ +# Composite yaml to rebase to upstream gcc master +runs: + using: "composite" + steps: + - name: Add remote + run: | + git remote add gcc git://gcc.gnu.org/git/gcc.git + shell: bash + - name: Fetch + run: | + git fetch gcc master + shell: bash + - name: Rebase + run: | + git log gcc/master -1 + git config --global user.email "erozen@microsoft.com" + git config --global user.name "Eugene Rozenfeld" + git rebase gcc/master + shell: bash + - name: Check commits + run: | + git status + git log -15 + shell: bash diff --git a/.github/actions/test-composite/action.yaml b/.github/actions/test-composite/action.yaml new file mode 100644 index 00000000000..b6f8b458fc5 --- /dev/null +++ b/.github/actions/test-composite/action.yaml @@ -0,0 +1,36 @@ +# Composite yaml that's re-usable in multiple workflows to run gcc tests +inputs: + configjson: + description: 'Config object converted into json' + required: true + default: '' + testSet: + description: 'gcc test set to run' + required: true + default: '' + githubtoken: + description: 'GitHub auth token' + required: true + default: '' + buildDownloaded: + description: 'Whether the build is already available on the machine this composite action is running on' + required: true + default: true +runs: + using: "composite" + steps: + - name: Download build and run tests + run: | + chmod +x .github/scripts/gccWorkflow.py + echo "$PYTHONPATH" + echo "$CONFIG_JSON" + export PYTHONPATH=${PYTHONPATH}:${PWD}/.github/scripts + echo "$PYTHONPATH" + echo "$BUILD_DOWNLOADED" + python -c 'import sys; from gccWorkflow import *; GccWorkflow.Test(sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4])' "${CONFIG_JSON}" "${TEST_SET}" "${GITHUB_TOKEN}" "${BUILD_DOWNLOADED}" + shell: bash + env: + CONFIG_JSON: ${{ inputs.configJson }} + TEST_SET: ${{ inputs.testSet }} + GITHUB_TOKEN: ${{ inputs.githubtoken }} + BUILD_DOWNLOADED: ${{ inputs.buildDownloaded }} diff --git a/.github/scripts/config.py b/.github/scripts/config.py index 44d01526540..96e81a3fc8d 100644 --- a/.github/scripts/config.py +++ b/.github/scripts/config.py @@ -19,7 +19,7 @@ class Config(object): logger.info('creating an instance of Config') @staticmethod - def Setup(githubObject, accessToken): + def Setup(githubObject, accessToken, isBuild): logger = GetLogger() logger.debug("In Setup") @@ -34,8 +34,8 @@ class Config(object): logger.info("SHA = " + commit) workflowName = githubJson["workflow"] - # If this isn't the GCC build, wait for the GCC build to complete with the needed artifacts - if (workflowName != GetGccBuildName()): + # If this isn't a GCC build, wait for the GCC build to complete with the needed artifacts + if (not isBuild): gccBuildArtifactID = WaitOnGccBuild(commit, accessToken) else: gccBuildArtifactID = 0 diff --git a/.github/scripts/downloadBuildArtifact.py b/.github/scripts/downloadBuildArtifact.py index 1dd8626d05d..51405afad54 100644 --- a/.github/scripts/downloadBuildArtifact.py +++ b/.github/scripts/downloadBuildArtifact.py @@ -55,13 +55,13 @@ def WaitOnGccBuild(commit, accessToken): RaiseWorkflowError("Build run ID " + str(latestBuildId) + " failed") else: # Sleep for 30 seconds - sleepTime = 30 - logger.error("Sleeping for " + str(sleepTime) + " seconds while waiting on build number " + str(latestBuildId) + " to finish") - time.sleep(sleepTime) # Sleep takes parameter in seconds + sleepTimeSeconds = 30 + logger.error("Sleeping for " + str(sleepTimeSeconds) + " seconds while waiting on build number " + str(latestBuildId) + " to finish") + time.sleep(sleepTimeSeconds) logger.info("Workflow run ID = " + str(workflowRunID)) - numArtifactRetries = 1 + numArtifactRetries = 60 # Add error checking for the number of artifacts while True: res = requests.get("https://api.github.com/repos/microsoft/test-gcc/actions/runs/" + str(workflowRunID) + "/artifacts", headers={'Authorization': "token " + accessToken}) @@ -81,12 +81,12 @@ def WaitOnGccBuild(commit, accessToken): numArtifactRetries = numArtifactRetries-1 # We need this sleep here because even if the build job status is "completed", the gccBuild artifacts aren't - # immediately ready. If for some reason they aren't, sleep for a bit and retry once more. + # immediately ready. If for some reason they aren't, sleep for a bit and retry. - # Sleep for 3 minutes - sleepTime = 3 - logger.info("Sleeping for " + str(sleepTime) + " minutes while waiting on artifacts for workflow run ID " + str(workflowRunID)) - time.sleep(sleepTime * 60) # Sleep takes parameter in seconds + # Sleep for 30 seconds + sleepTimeSeconds = 30 + logger.info("Sleeping for " + str(sleepTimeSeconds) + " seconds while waiting on artifacts for workflow run ID " + str(workflowRunID)) + time.sleep(sleepTimeSeconds) logger.info("GCC Build artifact ID = " + str(gccBuildArtifactID)) @@ -95,7 +95,6 @@ def WaitOnGccBuild(commit, accessToken): def DownloadBuildArtifact(): logger = GetLogger() gccBuildArtifactID = globals.configObj.gccBuildArtifactID - # TODO: Support downloading build artifact without a config object setup so the script can be used outside of workflows by developers res = requests.get("https://api.github.com/repos/microsoft/test-gcc/actions/artifacts/" + str(gccBuildArtifactID) +"/zip", headers={'Authorization': "token " + globals.configObj.accessToken}) @@ -105,6 +104,6 @@ def DownloadBuildArtifact(): logger.info("Unzipping zip") with zipfile.ZipFile('gccBuild.zip', 'r') as zip_ref: - zip_ref.extractall('gccBuild') + zip_ref.extractall('gccBuild') logger.info("Done downloading") diff --git a/.github/scripts/gccWorkflow.py b/.github/scripts/gccWorkflow.py index e0b82413dd1..f686e3f6ce2 100644 --- a/.github/scripts/gccWorkflow.py +++ b/.github/scripts/gccWorkflow.py @@ -4,14 +4,15 @@ from downloadBuildArtifact import * import sys import globals from common import * +import os.path class GccWorkflow(object): # Setup the config object and wait for any runs necessary to finish before proceeding onto the next job section # to avoid exceeding the 6 hr limit on Github Actions that run on Github machines @staticmethod - def Init(githubContext, accessToken): - Config.Setup(githubContext, accessToken) + def Init(githubContext, accessToken, isInitForBuild=False): + Config.Setup(githubContext, accessToken, isInitForBuild) # Runs the configure script to set up gcc configuration environment prior to building and running tests # Creates the objdir directory as part of this process @@ -47,26 +48,43 @@ class GccWorkflow(object): sys.exit(res.returncode) @staticmethod - def Test(configJson, testSet, accessToken): + def Test(configJson, testSet, accessToken, buildDownloadedStr): logger = GetLogger() logger.info("Gcc Test start") + + # Convert the input string (because that's how it's passed via the composite template) into a boolean + buildDownloaded = buildDownloadedStr == 'True' or buildDownloadedStr == 'true' + Config.Reload(configJson, accessToken) - Config.Reload(configJson, accessToken) - - logger.info("Downloading build artifact...") - try: - DownloadBuildArtifact() - except: - logger.error("Could not download build artifact") - RaiseWorkflowError("Error downloading build artifact for GCC workflow") + if (not buildDownloaded): + logger.info("Downloading gccBuild artifact from 'build' workflow...") + try: + DownloadBuildArtifact() + except: + logger.error("Could not download build artifact") + RaiseWorkflowError("Error downloading build artifact for GCC workflow") + + GccWorkflow.Configure() + res = subprocess.run(''' + mv gccBuild/* ../objdir -f + ''', + shell=True, check=False, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) + + logger.info("mv cmd output = " + str(res.stdout, 'utf-8')) + + if (res.returncode != 0): + logger.error("Failed to copy gccBuild into objdir directory") + logging.shutdown() + sys.exit(res.returncode) - GccWorkflow.Configure() + # The gcc build should be in the objdir folder one level above + currentDir = os.getcwd() + assert os.path.isdir(currentDir + "/../objdir") # Copy over downloaded build artifact into objdir directory that was created in the configure script # For more details on the subprocess function and its parameters, # see https://stackoverflow.com/questions/4256107/running-bash-commands-in-python res = subprocess.run(''' - mv gccBuild/* ../objdir -f chmod +x .github/scripts/test-gcc.sh ''' + ".github/scripts/test-gcc.sh " + testSet, diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 8e2da1b82a2..89ceaae0c42 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -33,10 +33,8 @@ jobs: uses: actions/setup-python@v1 with: python-version: 3.7 - + # Install requests package which is used in downloadBuildArtifact.py - # We need this since python will import packages and even though we aren't using this script in this workflow, - # the GCC workflow relies on the download script so the script will end up getting pulled in - name: Pip Install Requests run: | python -m pip install requests @@ -51,23 +49,15 @@ jobs: export PYTHONPATH=${PYTHONPATH}:${PWD}/.github/scripts echo "$PYTHONPATH" echo "${GITHUB_CONTEXT}" - python -c 'import sys; from gccWorkflow import *; GccWorkflow.Init(sys.argv[1], sys.argv[2])' "${GITHUB_CONTEXT}" "${GITHUB_TOKEN}" + python -c 'import sys; from gccWorkflow import *; GccWorkflow.Init(sys.argv[1], sys.argv[2], True)' "${GITHUB_CONTEXT}" "${GITHUB_TOKEN}" shell: bash env: GITHUB_CONTEXT: ${{ toJson(github) }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - # build.yaml creates the gccBuild artifact - - name: Build gcc - run: | - chmod +x .github/scripts/gccWorkflow.py - echo "$PYTHONPATH" - export PYTHONPATH=${PYTHONPATH}:${PWD}/.github/scripts - echo "$PYTHONPATH" - python -c 'import sys; from gccWorkflow import *; GccWorkflow.Build(sys.argv[1])' "${CONFIG_JSON}" - shell: bash - env: - CONFIG_JSON: ${{ steps.setupconfig.outputs.configJson }} + - uses: ./.github/actions/build-composite + with: + configjson: ${{ steps.setupconfig.outputs.configJson }} - name: Move objdir to be in repo so it can be uploaded run: mv ../objdir objdir diff --git a/.github/workflows/fetch-rebase-test.yaml b/.github/workflows/fetch-rebase-test.yaml index 85133ec749f..ef9abc703bb 100644 --- a/.github/workflows/fetch-rebase-test.yaml +++ b/.github/workflows/fetch-rebase-test.yaml @@ -7,8 +7,10 @@ on: workflow_dispatch: jobs: - fetch-rebase-test: + fetch-rebase-build: runs-on: ubuntu-18.04 + outputs: + config: ${{ steps.printNoSecretJson.outputs.noSecretConfigJson }} steps: - name: Checkout uses: actions/checkout@v2 @@ -16,26 +18,104 @@ jobs: fetch-depth: 0 submodules: recursive lfs: true - ref: current - - name: Add remote + ref: current + + - name: Setup Python 3.7 + uses: actions/setup-python@v1 + with: + python-version: 3.7 + + # Install requests package which is used in downloadBuildArtifact.py + - name: Pip Install Requests run: | - git remote add gcc git://gcc.gnu.org/git/gcc.git - - name: Fetch + python -m pip install requests + shell: bash + + - uses: ./.github/actions/rebase-gcc-master + + # Setup config + - name: Setup config + id: setupconfig run: | - git fetch gcc master - - name: Rebase + chmod +x .github/scripts/gccWorkflow.py + echo "$PYTHONPATH" + export PYTHONPATH=${PYTHONPATH}:${PWD}/.github/scripts + echo "$PYTHONPATH" + echo "${GITHUB_CONTEXT}" + python -c 'import sys; from gccWorkflow import *; GccWorkflow.Init(sys.argv[1], sys.argv[2], True)' "${GITHUB_CONTEXT}" "${GITHUB_TOKEN}" + shell: bash + env: + GITHUB_CONTEXT: ${{ toJson(github) }} + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + + - uses: ./.github/actions/build-composite + with: + configjson: ${{ steps.setupconfig.outputs.configJson }} + + - name: Move objdir to be in repo so it can be uploaded + run: mv ../objdir objdir + + - name: Upload build output + uses: actions/upload-artifact@v2 + with: + name: gccBuild + path: objdir + + # This should be the last step on this machine since it will clear out fields in the config json + - name: Print No Secret Json + id: printNoSecretJson run: | - git log gcc/master -1 - git config --global user.email "erozen@microsoft.com" - git config --global user.name "Eugene Rozenfeld" - git rebase gcc/master - - name: Check commits + chmod +x .github/scripts/config.py + export PYTHONPATH=${PYTHONPATH}:${PWD}/.github/scripts + python -c 'import sys; from config import *; Config.PrintNoSecretConfigJson(sys.argv[1])' "${CONFIG_JSON}" + shell: bash + env: + CONFIG_JSON: ${{ steps.setupconfig.outputs.configJson }} + + fetch-rebase-test: + needs: fetch-rebase-build + strategy: + matrix: + # If this test is updated, make sure you update the duplicate line in test-gcc.yaml + testSet: [check-target-libstdc++-v3, check-gcc-c++, check-gcc-c, check-target-libgomp, check-target-libitm, check-target-libatomic] + # Avoid cancelling other matrix chunks even if one fails + fail-fast: false + runs-on: ubuntu-18.04 + steps: + - name: Checkout + uses: actions/checkout@v2 + with: + fetch-depth: 0 + submodules: recursive + lfs: true + ref: current + + - name: Setup Python 3.7 + uses: actions/setup-python@v1 + with: + python-version: 3.7 + + # Install requests package which is used in downloadBuildArtifact.py + - name: Pip Install Requests run: | - git status - git log -15 - - - name: Run build script - run: | - chmod +x .github/scripts/build-gcc.sh - .github/scripts/build-gcc.sh + python -m pip install requests shell: bash + + - uses: ./.github/actions/rebase-gcc-master + + - name: Download gccBuild from fetch-rebase-build step + uses: actions/download-artifact@v2 + with: + name: gccBuild + path: objdir + + - name: Move objdir up one directory + run: mv objdir ../objdir + + # build.yaml creates the gccBuild artifact + - uses: ./.github/actions/test-composite + with: + configjson: ${{needs.fetch-rebase-build.outputs.config}} + testSet: ${{ matrix.testSet }} + githubtoken: ${{ secrets.GITHUB_TOKEN }} + buildDownloaded: True diff --git a/.github/workflows/test-gcc.yaml b/.github/workflows/test-gcc.yaml index a994e1ec56c..831b171a2d2 100644 --- a/.github/workflows/test-gcc.yaml +++ b/.github/workflows/test-gcc.yaml @@ -43,7 +43,7 @@ jobs: with: python-version: 3.7 - # Install requests package so we can use it for downloading the build artifacts in downloadBuildArtifact.py + # Install requests package which is used in downloadBuildArtifact.py - name: Pip Install Requests run: | python -m pip install requests @@ -55,7 +55,7 @@ jobs: run: | chmod +x .github/scripts/gccWorkflow.py export PYTHONPATH=${PYTHONPATH}:${PWD}/.github/scripts - python -c 'import sys; from gccWorkflow import *; GccWorkflow.Init(sys.argv[1], sys.argv[2])' "${GITHUB_CONTEXT}" "${GITHUB_TOKEN}" + python -c 'import sys; from gccWorkflow import *; GccWorkflow.Init(sys.argv[1], sys.argv[2], False)' "${GITHUB_CONTEXT}" "${GITHUB_TOKEN}" shell: bash env: GITHUB_CONTEXT: ${{ toJson(github) }} @@ -75,6 +75,7 @@ jobs: needs: init strategy: matrix: + # If this test is updated, make sure you update the duplicate line in fetch-rebase-test.yaml testSet: [check-target-libstdc++-v3, check-gcc-c++, check-gcc-c, check-target-libgomp, check-target-libitm, check-target-libatomic] # Avoid cancelling other matrix chunks even if one fails fail-fast: false @@ -95,26 +96,18 @@ jobs: uses: actions/setup-python@v1 with: python-version: 3.7 - - # Install requests package so we can use it for downloading the build artifacts in downloadBuildArtifact.py + + # Install requests package which is used in downloadBuildArtifact.py - name: Pip Install Requests run: | python -m pip install requests shell: bash # build.yaml creates the gccBuild artifact - - name: Download build and run tests - run: | - chmod +x .github/scripts/gccWorkflow.py - echo ${{needs.init.outputs.config}} - echo "$PYTHONPATH" - echo "$CONFIG_JSON" - export PYTHONPATH=${PYTHONPATH}:${PWD}/.github/scripts - echo "$PYTHONPATH" - python -c 'import sys; from gccWorkflow import *; GccWorkflow.Test(sys.argv[1], sys.argv[2], sys.argv[3])' "${CONFIG_JSON}" "${TEST_SET}" "${GITHUB_TOKEN}" - shell: bash - env: - CONFIG_JSON: ${{needs.init.outputs.config}} - TEST_SET: ${{ matrix.testSet }} - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - \ No newline at end of file + - uses: ./.github/actions/test-composite + with: + configjson: ${{needs.init.outputs.config}} + testSet: ${{ matrix.testSet }} + githubtoken: ${{ secrets.GITHUB_TOKEN }} + buildDownloaded: False +