2022-11-23T01:29:49.1204168Z Requested labels: self-hosted, linux.large
2022-11-23T01:29:49.1204291Z Job defined at: pytorch/pytorch/.github/workflows/_rocm-test.yml@refs/heads/master
2022-11-23T01:29:49.1204332Z Waiting for a runner to pick up this job...
2022-11-23T01:29:49.3238959Z Job is about to start running on the runner: i-03ba778d7d12c320d (organization)
2022-11-23T01:29:54.2200443Z Current runner version: '2.299.1'
2022-11-23T01:29:54.2205774Z Runner name: 'i-03ba778d7d12c320d'
2022-11-23T01:29:54.2206230Z Runner group name: 'Default'
2022-11-23T01:29:54.2206860Z Machine name: 'ip-10-0-0-28'
2022-11-23T01:29:54.2208762Z ##[group]GITHUB_TOKEN Permissions
2022-11-23T01:29:54.2209417Z Actions: write
2022-11-23T01:29:54.2209711Z Checks: write
2022-11-23T01:29:54.2210053Z Contents: write
2022-11-23T01:29:54.2210396Z Deployments: write
2022-11-23T01:29:54.2210696Z Discussions: write
2022-11-23T01:29:54.2210973Z Issues: write
2022-11-23T01:29:54.2211245Z Metadata: read
2022-11-23T01:29:54.2211536Z Packages: write
2022-11-23T01:29:54.2211782Z Pages: write
2022-11-23T01:29:54.2212072Z PullRequests: write
2022-11-23T01:29:54.2212421Z RepositoryProjects: write
2022-11-23T01:29:54.2212758Z SecurityEvents: write
2022-11-23T01:29:54.2213012Z Statuses: write
2022-11-23T01:29:54.2213292Z ##[endgroup]
2022-11-23T01:29:54.2216344Z Secret source: Actions
2022-11-23T01:29:54.2216968Z Prepare workflow directory
2022-11-23T01:29:54.5701990Z Prepare all required actions
2022-11-23T01:29:54.5874933Z Getting action download info
2022-11-23T01:29:54.7699949Z Download action repository 'pytorch/pytorch@master' (SHA:1cfd3858ac54fe3883534309081631a0a892ba3f)
2022-11-23T01:29:58.0581819Z Getting action download info
2022-11-23T01:29:58.2720934Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076)
2022-11-23T01:29:58.4836258Z Uses: pytorch/pytorch/.github/workflows/_rocm-test.yml
2022-11-23T01:29:58.4837716Z ##[group] Inputs
2022-11-23T01:29:58.4838001Z   build-environment: linux-focal-rocm5.2-py3.8
2022-11-23T01:29:58.4838382Z   test-matrix: { include: [
  { config: "distributed", shard: 1, num_shards: 2, runner: "linux.rocm.gpu" },
  { config: "distributed", shard: 2, num_shards: 2, runner: "linux.rocm.gpu" },
]}

2022-11-23T01:29:58.4838901Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm5.2-py3.8:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:29:58.4839244Z   sync-tag: 
2022-11-23T01:29:58.4839407Z ##[endgroup]
2022-11-23T01:29:58.4839838Z Complete job name: linux-focal-rocm5.2-py3.8-distributed / filter
2022-11-23T01:29:58.5454052Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2022-11-23T01:29:58.5454352Z with:
2022-11-23T01:29:58.5454517Z   fetch-depth: 1
2022-11-23T01:29:58.5454701Z   submodules: false
2022-11-23T01:29:58.5454879Z env:
2022-11-23T01:29:58.5455055Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:29:58.5455242Z ##[endgroup]
2022-11-23T01:29:58.5670736Z ##[group]Run retry () {
2022-11-23T01:29:58.5670998Z retry () {
2022-11-23T01:29:58.5671215Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2022-11-23T01:29:58.5671434Z }
2022-11-23T01:29:58.5671622Z echo "${GITHUB_WORKSPACE}"
2022-11-23T01:29:58.5671836Z if [ -z "${NO_SUDO}" ]; then
2022-11-23T01:29:58.5672063Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2022-11-23T01:29:58.5672312Z else
2022-11-23T01:29:58.5672509Z   retry rm -rf "${GITHUB_WORKSPACE}"
2022-11-23T01:29:58.5672689Z fi
2022-11-23T01:29:58.5672921Z mkdir "${GITHUB_WORKSPACE}"
2022-11-23T01:29:58.5688349Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:29:58.5688592Z env:
2022-11-23T01:29:58.5688814Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:29:58.5689000Z   NO_SUDO: 
2022-11-23T01:29:58.5689158Z ##[endgroup]
2022-11-23T01:29:58.5876524Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:29:58.8569966Z ##[group]Run malfet/checkout@silent-checkout
2022-11-23T01:29:58.8570192Z with:
2022-11-23T01:29:58.8570388Z   ref: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:29:58.8570581Z   fetch-depth: 1
2022-11-23T01:29:58.8570759Z   submodules: false
2022-11-23T01:29:58.8570941Z   quiet-checkout: true
2022-11-23T01:29:58.8571124Z   repository: pytorch/pytorch
2022-11-23T01:29:58.8571484Z   token: ***
2022-11-23T01:29:58.8571766Z   ssh-strict: true
2022-11-23T01:29:58.8571967Z   persist-credentials: true
2022-11-23T01:29:58.8572144Z   clean: true
2022-11-23T01:29:58.8572306Z   lfs: false
2022-11-23T01:29:58.8572495Z   set-safe-directory: true
2022-11-23T01:29:58.8572659Z env:
2022-11-23T01:29:58.8572825Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:29:58.8573007Z ##[endgroup]
2022-11-23T01:29:58.9887628Z Syncing repository: pytorch/pytorch
2022-11-23T01:29:58.9889894Z ##[group]Getting Git version info
2022-11-23T01:29:58.9890573Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-11-23T01:29:58.9891383Z [command]/usr/bin/git version
2022-11-23T01:29:58.9891717Z git version 2.37.1
2022-11-23T01:29:58.9892707Z ##[endgroup]
2022-11-23T01:29:58.9916287Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/96e222ba-558a-4fc2-ba74-f10145a41266' before making global git config changes
2022-11-23T01:29:58.9917019Z Adding repository directory to the temporary git global config as a safe directory
2022-11-23T01:29:58.9920589Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:29:58.9955464Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-11-23T01:29:58.9962020Z ##[group]Initializing the repository
2022-11-23T01:29:58.9966594Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:29:59.0000864Z hint: Using 'master' as the name for the initial branch. This default branch name
2022-11-23T01:29:59.0001453Z hint: is subject to change. To configure the initial branch name to use in all
2022-11-23T01:29:59.0002020Z hint: of your new repositories, which will suppress this warning, call:
2022-11-23T01:29:59.0002435Z hint: 
2022-11-23T01:29:59.0002928Z hint: 	git config --global init.defaultBranch <name>
2022-11-23T01:29:59.0003520Z hint: 
2022-11-23T01:29:59.0004056Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2022-11-23T01:29:59.0004720Z hint: 'development'. The just-created branch can be renamed via this command:
2022-11-23T01:29:59.0005110Z hint: 
2022-11-23T01:29:59.0005518Z hint: 	git branch -m <name>
2022-11-23T01:29:59.0019354Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2022-11-23T01:29:59.0020063Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2022-11-23T01:29:59.0042176Z ##[endgroup]
2022-11-23T01:29:59.0042853Z ##[group]Disabling automatic garbage collection
2022-11-23T01:29:59.0047944Z [command]/usr/bin/git config --local gc.auto 0
2022-11-23T01:29:59.0074821Z ##[endgroup]
2022-11-23T01:29:59.0075878Z ##[group]Setting up auth
2022-11-23T01:29:59.0083218Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-11-23T01:29:59.0111195Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-11-23T01:29:59.0344060Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-11-23T01:29:59.0372945Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-11-23T01:29:59.0603524Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-11-23T01:29:59.0646231Z ##[endgroup]
2022-11-23T01:29:59.0648660Z ##[group]Fetching the repository
2022-11-23T01:29:59.0657060Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --quiet --no-recurse-submodules --depth=1 origin 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:30:07.2320836Z ##[endgroup]
2022-11-23T01:30:07.2331074Z ##[group]Determining the checkout info
2022-11-23T01:30:07.2331814Z ##[endgroup]
2022-11-23T01:30:07.2332430Z ##[group]Checking out the ref
2022-11-23T01:30:07.2333408Z [command]/usr/bin/git checkout --quiet --force 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:30:08.3876351Z ##[endgroup]
2022-11-23T01:30:08.3922550Z [command]/usr/bin/git log -1 --format='%H'
2022-11-23T01:30:08.3953428Z '1cfd3858ac54fe3883534309081631a0a892ba3f'
2022-11-23T01:30:08.4180807Z Prepare all required actions
2022-11-23T01:30:08.4181120Z Getting action download info
2022-11-23T01:30:08.5920029Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2022-11-23T01:30:08.7601837Z ##[group]Run ./.github/actions/filter-test-configs
2022-11-23T01:30:08.7602037Z with:
2022-11-23T01:30:08.7602551Z   github-token: ***
2022-11-23T01:30:08.7602897Z   test-matrix: { include: [
  { config: "distributed", shard: 1, num_shards: 2, runner: "linux.rocm.gpu" },
  { config: "distributed", shard: 2, num_shards: 2, runner: "linux.rocm.gpu" },
]}

2022-11-23T01:30:08.7603211Z env:
2022-11-23T01:30:08.7603378Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:30:08.7603557Z ##[endgroup]
2022-11-23T01:30:08.7628462Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2022-11-23T01:30:08.7628693Z with:
2022-11-23T01:30:08.7628842Z   shell: bash
2022-11-23T01:30:08.7629045Z   timeout_minutes: 10
2022-11-23T01:30:08.7629211Z   max_attempts: 5
2022-11-23T01:30:08.7629389Z   retry_wait_seconds: 30
2022-11-23T01:30:08.7629634Z   command: set -eux
python3 -m pip install requests==2.26.0 pyyaml==6.0

2022-11-23T01:30:08.7629880Z   polling_interval_seconds: 1
2022-11-23T01:30:08.7630057Z   warning_on_retry: true
2022-11-23T01:30:08.7630243Z   continue_on_error: false
2022-11-23T01:30:08.7630413Z env:
2022-11-23T01:30:08.7630571Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:30:08.7630930Z   GITHUB_TOKEN: ***
2022-11-23T01:30:08.7631102Z ##[endgroup]
2022-11-23T01:30:08.8132213Z + python3 -m pip install requests==2.26.0 pyyaml==6.0
2022-11-23T01:30:09.0261255Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T01:30:09.0445812Z Requirement already satisfied: requests==2.26.0 in /home/ec2-user/.local/lib/python3.7/site-packages (2.26.0)
2022-11-23T01:30:09.0665236Z Requirement already satisfied: pyyaml==6.0 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0)
2022-11-23T01:30:09.0677408Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (1.26.12)
2022-11-23T01:30:09.0954503Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2.0.12)
2022-11-23T01:30:09.0983580Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2022.9.24)
2022-11-23T01:30:09.0997876Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (3.4)
2022-11-23T01:30:09.8095521Z Command completed after 1 attempt(s).
2022-11-23T01:30:09.8134212Z ##[group]Run .github/scripts/parse_ref.py
2022-11-23T01:30:09.8134460Z .github/scripts/parse_ref.py
2022-11-23T01:30:09.8145546Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:30:09.8146122Z env:
2022-11-23T01:30:09.8146296Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:30:09.8146485Z ##[endgroup]
2022-11-23T01:30:09.8419755Z ##[group]Run .github/scripts/filter_test_configs.py \
2022-11-23T01:30:09.8420081Z .github/scripts/filter_test_configs.py \
2022-11-23T01:30:09.8420364Z   --test-matrix "{ include: [
2022-11-23T01:30:09.8420676Z   { config: "distributed", shard: 1, num_shards: 2, runner: "linux.rocm.gpu" },
2022-11-23T01:30:09.8421046Z   { config: "distributed", shard: 2, num_shards: 2, runner: "linux.rocm.gpu" },
2022-11-23T01:30:09.8421287Z ]}
2022-11-23T01:30:09.8421502Z " \
2022-11-23T01:30:09.8421701Z   --pr-number "" \
2022-11-23T01:30:09.8459505Z   --tag "" \
2022-11-23T01:30:09.8459817Z   --event-name "schedule"
2022-11-23T01:30:09.8470749Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:30:09.8470966Z env:
2022-11-23T01:30:09.8471212Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:30:09.8471629Z   GITHUB_TOKEN: ***
2022-11-23T01:30:09.8471794Z ##[endgroup]
2022-11-23T01:30:09.9705764Z ##[group]Run echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}]}"
2022-11-23T01:30:09.9707115Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}]}"
2022-11-23T01:30:09.9718703Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:30:09.9718934Z env:
2022-11-23T01:30:09.9719107Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:30:09.9719280Z ##[endgroup]
2022-11-23T01:30:09.9745032Z {include: [{config: distributed, shard: 1, num_shards: 2, runner: linux.rocm.gpu, mem_leak_check: mem_leak_check}, {config: distributed, shard: 1, num_shards: 2, runner: linux.rocm.gpu, rerun_disabled_tests: rerun_disabled_tests}, {config: distributed, shard: 2, num_shards: 2, runner: linux.rocm.gpu, mem_leak_check: mem_leak_check}, {config: distributed, shard: 2, num_shards: 2, runner: linux.rocm.gpu, rerun_disabled_tests: rerun_disabled_tests}]}
2022-11-23T01:30:09.9838289Z Post job cleanup.
2022-11-23T01:30:09.9865844Z Post job cleanup.
2022-11-23T01:30:10.1061188Z [command]/usr/bin/git version
2022-11-23T01:30:10.1104478Z git version 2.37.1
2022-11-23T01:30:10.1175692Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/32af02af-b8fe-44b3-981b-9cedd8a3ae1a' before making global git config changes
2022-11-23T01:30:10.1180067Z Adding repository directory to the temporary git global config as a safe directory
2022-11-23T01:30:10.1191220Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:30:10.1280543Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-11-23T01:30:10.1328791Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-11-23T01:30:10.1702552Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-11-23T01:30:10.1745234Z http.https://github.com/.extraheader
2022-11-23T01:30:10.1767528Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-11-23T01:30:10.1790654Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-11-23T01:30:10.2247058Z Evaluate and set job outputs
2022-11-23T01:30:10.2257821Z Set output 'test-matrix'
2022-11-23T01:30:10.2258589Z Set output 'is-test-matrix-empty'
2022-11-23T01:30:10.2259183Z Cleaning up orphan processes