From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: by sourceware.org (Postfix, from userid 7840) id D09A33857C6B; Tue, 5 Oct 2021 21:33:36 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org D09A33857C6B Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Eugene Rozenfeld To: gcc-cvs@gcc.gnu.org Subject: [gcc(refs/vendors/microsoft/heads/main)] This change fixes the git commands that update-main was printing so that they now work. The git comm X-Act-Checkin: gcc X-Git-Author: Victor Tong X-Git-Refname: refs/vendors/microsoft/heads/main X-Git-Oldrev: 3bbfbe9828429a2dfb6efa07c09f60bff5b5c7a0 X-Git-Newrev: bb54570de56f33a3554e50fe81a3a22aefb6d9e4 Message-Id: <20211005213336.D09A33857C6B@sourceware.org> Date: Tue, 5 Oct 2021 21:33:36 +0000 (GMT) X-BeenThere: gcc-cvs@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-cvs mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Oct 2021 21:33:36 -0000 https://gcc.gnu.org/g:bb54570de56f33a3554e50fe81a3a22aefb6d9e4 commit bb54570de56f33a3554e50fe81a3a22aefb6d9e4 Author: Victor Tong Date: Thu Aug 19 18:47:32 2021 -0700 This change fixes the git commands that update-main was printing so that they now work. The git commands have been put into a script to push to a specified vendor branch using a branch generated from the update-main.yaml workflow. The script will add the upstream gcc remote if none is found in the repository. The script will also create a temporary local branch to perform the merge. I also added a script that can be used by workflows and the script above to create an upstream gcc Git remote. I wasn't able to test the final push command since I don't have access to push to the GCC repository. Update-main test run with highlighted warning text for the command to stand out: https://github.com/microsoft/gcc/runs/3664797181?check_suite_focus=true#step:6:9 Diff: --- .github/actions/pull-master-upstream/action.yaml | 12 +++--- .github/actions/rebase-gcc-master/action.yaml | 7 +++- .github/actions/setup-vendor-branches/action.yaml | 18 +++++---- .github/scripts/common.py | 16 +++++++- .github/scripts/gccWorkflow.py | 17 ++------ .github/scripts/init.sh | 37 ++++++++++++++++++ .github/scripts/pushMergedVendorBranch.sh | 47 +++++++++++++++++++++++ .github/workflows/update-main.yaml | 9 +++-- .github/workflows/update-mirror-branches.yaml | 7 +++- 9 files changed, 135 insertions(+), 35 deletions(-) diff --git a/.github/actions/pull-master-upstream/action.yaml b/.github/actions/pull-master-upstream/action.yaml index 0d603e26c71..0b589c87e7a 100644 --- a/.github/actions/pull-master-upstream/action.yaml +++ b/.github/actions/pull-master-upstream/action.yaml @@ -35,12 +35,14 @@ inputs: runs: using: "composite" steps: - - name: Add remote and checkout from ${{ inputs.masterRef }} + - name: Initialize environment and checkout from ${{ inputs.masterRef }} run: | - git remote add gcc git://gcc.gnu.org/git/gcc.git - git fetch gcc master - chmod +x .github/scripts/checkout-refs.sh - .github/scripts/checkout-refs.sh ${MASTER_REF} ${SCRIPTS_REF} + python -m pip install requests + chmod +x .github/scripts/common.py + export PYTHONPATH=${PYTHONPATH}:${PWD}/.github/scripts + python -c 'from common import *; RunScript(".github/scripts/init.sh", True)' + git fetch gcc master + python -c 'import sys; from common import *; RunScript(".github/scripts/checkout-refs.sh", True, sys.argv[1], sys.argv[2])' "${MASTER_REF}" "${SCRIPTS_REF}" git log -1 shell: bash env: diff --git a/.github/actions/rebase-gcc-master/action.yaml b/.github/actions/rebase-gcc-master/action.yaml index cbf478e94d7..941ae85a822 100644 --- a/.github/actions/rebase-gcc-master/action.yaml +++ b/.github/actions/rebase-gcc-master/action.yaml @@ -25,9 +25,12 @@ runs: using: "composite" steps: - - name: Add remote + - name: Initialize environment run: | - git remote add gcc git://gcc.gnu.org/git/gcc.git + python -m pip install requests + chmod +x .github/scripts/common.py + export PYTHONPATH=${PYTHONPATH}:${PWD}/.github/scripts + python -c 'from common import *; RunScript(".github/scripts/init.sh", True)' shell: bash - name: Fetch run: | diff --git a/.github/actions/setup-vendor-branches/action.yaml b/.github/actions/setup-vendor-branches/action.yaml index e40d23c4bc3..415165a1b78 100644 --- a/.github/actions/setup-vendor-branches/action.yaml +++ b/.github/actions/setup-vendor-branches/action.yaml @@ -35,15 +35,17 @@ inputs: runs: using: "composite" steps: - - name: Add remote and checkout from vendor branch + - name: Initialize environment and checkout from vendor branch run: | - git remote add gcc git://gcc.gnu.org/git/gcc.git - echo -e "dummyName\ndummyEmail\ngcc\ndummyName\nme\nyes" > dummyGitCustomization.txt - cat "dummyGitCustomization.txt" | contrib/gcc-git-customization.sh - contrib/git-fetch-vendor.sh microsoft - git branch -a - chmod +x .github/scripts/checkout-refs.sh - .github/scripts/checkout-refs.sh ${VENDOR_REF} ${SCRIPTS_REF} + python -m pip install requests + chmod +x .github/scripts/common.py + export PYTHONPATH=${PYTHONPATH}:${PWD}/.github/scripts + python -c 'from common import *; RunScript(".github/scripts/init.sh", True)' + echo -e "MS Automation\ngnutools@microsoft.com\ngcc\nMSAutomation\nme\nyes" > gitCustomization.txt + cat "gitCustomization.txt" | contrib/gcc-git-customization.sh + python -c 'from common import *; RunScript("contrib/git-fetch-vendor.sh", True, "microsoft")' + git branch -a + python -c 'import sys; from common import *; RunScript(".github/scripts/checkout-refs.sh", True, sys.argv[1], sys.argv[2])' "${VENDOR_REF}" "${SCRIPTS_REF}" shell: bash env: SCRIPTS_REF: ${{ inputs.scriptsRef }} diff --git a/.github/scripts/common.py b/.github/scripts/common.py index 8f1264930df..58de82132d9 100644 --- a/.github/scripts/common.py +++ b/.github/scripts/common.py @@ -67,4 +67,18 @@ def SendGetRestCmd(restCmd, restArgs, restHeader): retryCount = retryCount - 1 if retryCount == 0: - RaiseWorkflowError("Max retry count hit with " + restCmd) \ No newline at end of file + RaiseWorkflowError("Max retry count hit with " + restCmd) + +# Run the provided script with the provided arguments +def RunScript(scriptPath, failOnNonZeroReturn, *args): + argsStr = "" + for a in args: + argsStr += " " + argsStr += a + + res = subprocess.run(''' + chmod +x ''' + scriptPath + ''' + ''' + scriptPath + argsStr, + shell=True, check=failOnNonZeroReturn, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) + + return res \ No newline at end of file diff --git a/.github/scripts/gccWorkflow.py b/.github/scripts/gccWorkflow.py index f63fd45b241..411aac30434 100644 --- a/.github/scripts/gccWorkflow.py +++ b/.github/scripts/gccWorkflow.py @@ -50,11 +50,7 @@ class GccWorkflow(object): @staticmethod def Configure(): logger = GetLogger() - res = subprocess.run(''' - chmod +x .github/scripts/configure-gcc.sh - .github/scripts/configure-gcc.sh''', - shell=True, check=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) - + res = RunScript(".github/scripts/configure-gcc.sh", False) logger.info("output = " + str(res.stdout, 'utf-8')) @staticmethod @@ -78,10 +74,7 @@ class GccWorkflow(object): GccWorkflow.MakeObjDir() # Build - res = subprocess.run(''' - chmod +x .github/scripts/build-gcc.sh - .github/scripts/build-gcc.sh''', - shell=True, check=False, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) + res = RunScript(".github/scripts/build-gcc.sh", False) logger.info("build output = " + str(res.stdout, 'utf-8')) # TODO: Add better handling of errors to display @@ -117,11 +110,7 @@ class GccWorkflow(object): # Copy over downloaded build artifact into objdir directory that was created in the configure script # For more details on the subprocess function and its parameters, # see https://stackoverflow.com/questions/4256107/running-bash-commands-in-python - res = subprocess.run(''' - chmod +x .github/scripts/test-gcc.sh - ''' - + ".github/scripts/test-gcc.sh " + testSet, - shell=True, check=False, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) + res = RunScript(".github/scripts/test-gcc.sh", False, testSet) logger.info("test output = " + str(res.stdout, 'utf-8')) # TODO: Add better handling of errors to display diff --git a/.github/scripts/init.sh b/.github/scripts/init.sh new file mode 100644 index 00000000000..7a94c321be2 --- /dev/null +++ b/.github/scripts/init.sh @@ -0,0 +1,37 @@ +# Script to run basic initialization for developers and GitHub workflows + +# Copyright (c) Microsoft Corporation. + +# MIT License + +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: + +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. + +# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +# Setup upstream gcc remote +git ls-remote --exit-code --quiet gcc > /dev/null 2>&1 +if test $? != 0; then + echo "Adding gcc upstream remote" + git remote add gcc git://gcc.gnu.org/git/gcc.git +else + # Check to make sure the gcc remote URL is the one we're expecting + gccUrl="$(git config remote.gcc.url)" + if [[ $gccUrl != "git://gcc.gnu.org/git/gcc.git" ]]; then + echo "ERROR: gcc upstream remote was found but it's not pointing to git://gcc.gnu.org/git/gcc.git" + exit -1 + fi +fi \ No newline at end of file diff --git a/.github/scripts/pushMergedVendorBranch.sh b/.github/scripts/pushMergedVendorBranch.sh new file mode 100644 index 00000000000..d9850e78938 --- /dev/null +++ b/.github/scripts/pushMergedVendorBranch.sh @@ -0,0 +1,47 @@ +# This script pushes the merged branch +# $1 is the temporary local branch to create +# $2 is the merge branch produced by the update-main.yaml workflow + +# Copyright (c) Microsoft Corporation. + +# MIT License + +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: + +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. + +# THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +BASEDIR=$(dirname $0) + +chmod +x ${BASEDIR}/init.sh +${BASEDIR}/init.sh + +echo " +---- READ THIS ---- +When prompted for the 'local name for upstream repository', +enter 'gcc'. The other fields can be the default +---- END OF READ THIS ---- +" + +${BASEDIR}/../../contrib/gcc-git-customization.sh +${BASEDIR}/../../contrib/git-fetch-vendor.sh microsoft +git fetch +git fetch gcc +git fetch origin +git fetch vendors/microsoft +git checkout -b $1 remotes/vendors/microsoft/main +git merge origin/$2 +git push remotes/vendors/microsoft/main diff --git a/.github/workflows/update-main.yaml b/.github/workflows/update-main.yaml index 55c58e46bdd..ea39ce29ca7 100644 --- a/.github/workflows/update-main.yaml +++ b/.github/workflows/update-main.yaml @@ -379,10 +379,11 @@ jobs: mergeBranch: ${{ needs.parse-failures.outputs.newBranchName }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - - name: Print git commands to use to merge and push manually. + # The text below is marked with a warning header so that it's colored yellow in the logs and will stand out + # I couldn't find any documentation on setting text color in printf statements in a GitHub Yaml file so we're using this for now + - name: Run the following command to push the merge manually run: | - printf "Run the following command. It assumes that you have the microsoft/gcc repository at origin and the upstream GCC Microsoft vendor branches set up at gcc" - printf "git fetch gcc \ngit checkout gcc vendors/microsoft/main \ngit merge origin ${mergeBranch} \ngit push gcc vendors/microsoft/main \n" + printf "::warning::Run the following command to create temporary local git branch and push the changes to our vendor branch: \n" + printf ".github/scripts/pushMergedVendorBranch.sh ${mergeBranch}" env: - masterSHA: ${{ needs.gcc-master-build.outputs.masterSHA }} mergeBranch: ${{ needs.parse-failures.outputs.newBranchName }} diff --git a/.github/workflows/update-mirror-branches.yaml b/.github/workflows/update-mirror-branches.yaml index 8ab9796c17b..24f299a6704 100644 --- a/.github/workflows/update-mirror-branches.yaml +++ b/.github/workflows/update-mirror-branches.yaml @@ -30,7 +30,7 @@ on: schedule: - cron: "0,10,20,30,40,50 * * * *" workflow_dispatch: - + jobs: sync: runs-on: ubuntu-18.04 @@ -63,6 +63,11 @@ jobs: lfs: true token: ${{ secrets.VICTORPAT }} # The basic ${{ github.token }} doesn't include "workflows" write permission access to modify workflows in the .github directory + - name: Setup Python 3.7 + uses: actions/setup-python@v1 + with: + python-version: 3.7 + # Add upstream gcc as remote, setup vendor branches and checkout matrix vendor branch - name: Setup and checkout vendors/microsoft/${{ matrix.upstreamBranch }} uses: ./.github/actions/setup-vendor-branches