2024-08-07T17:43:07.9643896Z Current runner version: '2.318.0' 2024-08-07T17:43:07.9652848Z Runner name: 'i-028914771a8277c83' 2024-08-07T17:43:07.9654122Z Runner group name: 'Default' 2024-08-07T17:43:07.9655436Z Machine name: 'ip-10-0-48-24' 2024-08-07T17:43:07.9662090Z ##[group]GITHUB_TOKEN Permissions 2024-08-07T17:43:07.9665276Z Actions: read 2024-08-07T17:43:07.9666161Z Attestations: read 2024-08-07T17:43:07.9667095Z Checks: read 2024-08-07T17:43:07.9667909Z Contents: read 2024-08-07T17:43:07.9668690Z Deployments: read 2024-08-07T17:43:07.9669588Z Discussions: read 2024-08-07T17:43:07.9670434Z Issues: read 2024-08-07T17:43:07.9671238Z Metadata: read 2024-08-07T17:43:07.9672123Z Packages: read 2024-08-07T17:43:07.9672956Z Pages: read 2024-08-07T17:43:07.9673726Z PullRequests: read 2024-08-07T17:43:07.9674641Z RepositoryProjects: read 2024-08-07T17:43:07.9675613Z SecurityEvents: read 2024-08-07T17:43:07.9676455Z Statuses: read 2024-08-07T17:43:07.9677309Z ##[endgroup] 2024-08-07T17:43:07.9681916Z Secret source: Actions 2024-08-07T17:43:07.9683237Z Prepare workflow directory 2024-08-07T17:43:08.3573014Z Prepare all required actions 2024-08-07T17:43:08.3625721Z Getting action download info 2024-08-07T17:43:08.5676554Z Download action repository 'pytorch/test-infra@main' (SHA:a1f5a89251fc4258ab59806094fe3108f7d6741a) 2024-08-07T17:43:08.9525417Z Download action repository 'pytorch/pytorch@main' (SHA:a62710c82039b798befd86e938d2137af3978c93) 2024-08-07T17:43:12.5357741Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-07T17:43:12.7006434Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-07T17:43:12.9619789Z Getting action download info 2024-08-07T17:43:13.0585379Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-07T17:43:13.2708456Z Getting action download info 2024-08-07T17:43:13.3563528Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-07T17:43:13.5106863Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/131248/merge (f779f6b7738020e244184bded4026b37de3f9f24) 2024-08-07T17:43:13.5109989Z ##[group] Inputs 2024-08-07T17:43:13.5110609Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2024-08-07T17:43:13.5112302Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2024-08-07T17:43:13.5114422Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:43:13.5115580Z sync-tag: 2024-08-07T17:43:13.5116873Z timeout-minutes: 360 2024-08-07T17:43:13.5117451Z use-gha: 2024-08-07T17:43:13.5117964Z dashboard-tag: 2024-08-07T17:43:13.5118409Z s3-bucket: gha-artifacts 2024-08-07T17:43:13.5118942Z aws-role-to-assume: 2024-08-07T17:43:13.5119486Z ##[endgroup] 2024-08-07T17:43:13.5120228Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:43:13.5873153Z A job started hook has been configured by the self-hosted runner administrator 2024-08-07T17:43:13.6025973Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-07T17:43:13.6038043Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:43:13.6039159Z ##[endgroup] 2024-08-07T17:43:15.1653080Z Runner Type: linux.8xlarge.nvidia.gpu 2024-08-07T17:43:15.1653734Z Instance Type: g3.8xlarge 2024-08-07T17:43:15.1654182Z AMI Name: amzn2-ami-hvm-2.0.20240306.2-x86_64-ebs 2024-08-07T17:43:15.1654676Z AMI ID: ami-0ce0c36d7a00b20e2 2024-08-07T17:43:20.0698690Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-07T17:43:20.0699523Z with: 2024-08-07T17:43:20.0700320Z github-secret: *** 2024-08-07T17:43:20.0701241Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-07T17:43:20.0702167Z activate-with-label: false 2024-08-07T17:43:20.0702521Z label: with-ssh 2024-08-07T17:43:20.0702855Z remove-existing-keys: true 2024-08-07T17:43:20.0703227Z fail-silently: true 2024-08-07T17:43:20.0703538Z env: 2024-08-07T17:43:20.0703832Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:43:20.0704197Z ##[endgroup] 2024-08-07T17:43:20.1796419Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-07T17:43:20.5339608Z Grabbing public ssh keys from https://github.com/zdevito.keys 2024-08-07T17:43:20.6084883Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-07T17:43:20.6107118Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-07T17:43:20.6149521Z Login using: ssh ec2-user@ec2-54-87-226-243.compute-1.amazonaws.com 2024-08-07T17:43:20.6150673Z All testing is done inside the container, to start an interactive session run: 2024-08-07T17:43:20.6151540Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-07T17:43:20.6378745Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-07T17:43:20.6379718Z with: 2024-08-07T17:43:20.6380035Z submodules: recursive 2024-08-07T17:43:20.6380373Z fetch-depth: 0 2024-08-07T17:43:20.6380689Z env: 2024-08-07T17:43:20.6380982Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:43:20.6381321Z ##[endgroup] 2024-08-07T17:43:20.6485329Z ##[group]Run retry () { 2024-08-07T17:43:20.6485731Z retry () { 2024-08-07T17:43:20.6486168Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-07T17:43:20.6486663Z } 2024-08-07T17:43:20.6487001Z echo "${GITHUB_WORKSPACE}" 2024-08-07T17:43:20.6487461Z if [ -z "${NO_SUDO}" ]; then 2024-08-07T17:43:20.6487903Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-07T17:43:20.6488337Z else 2024-08-07T17:43:20.6488664Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-07T17:43:20.6489076Z fi 2024-08-07T17:43:20.6489443Z mkdir "${GITHUB_WORKSPACE}" 2024-08-07T17:43:20.6497605Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:43:20.6498094Z env: 2024-08-07T17:43:20.6498849Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:43:20.6499209Z NO_SUDO: 2024-08-07T17:43:20.6499499Z ##[endgroup] 2024-08-07T17:43:20.6525581Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-07T17:43:23.8569316Z ##[group]Run malfet/checkout@silent-checkout 2024-08-07T17:43:23.8569755Z with: 2024-08-07T17:43:23.8570067Z ref: 016588f53c6904b840aa56aa86f95460b4d9c996 2024-08-07T17:43:23.8570501Z fetch-depth: 0 2024-08-07T17:43:23.8570825Z submodules: recursive 2024-08-07T17:43:23.8571178Z quiet-checkout: true 2024-08-07T17:43:23.8571533Z repository: pytorch/pytorch 2024-08-07T17:43:23.8572035Z token: *** 2024-08-07T17:43:23.8572344Z ssh-strict: true 2024-08-07T17:43:23.8572657Z persist-credentials: true 2024-08-07T17:43:23.8573017Z clean: true 2024-08-07T17:43:23.8573342Z sparse-checkout-cone-mode: true 2024-08-07T17:43:23.8573731Z lfs: false 2024-08-07T17:43:23.8574039Z set-safe-directory: true 2024-08-07T17:43:23.8574369Z env: 2024-08-07T17:43:23.8574666Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:43:23.8575009Z ##[endgroup] 2024-08-07T17:43:23.9851810Z Syncing repository: pytorch/pytorch 2024-08-07T17:43:23.9853641Z ##[group]Getting Git version info 2024-08-07T17:43:23.9854255Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-07T17:43:23.9855118Z [command]/usr/bin/git version 2024-08-07T17:43:23.9855501Z git version 2.40.1 2024-08-07T17:43:23.9880042Z ##[endgroup] 2024-08-07T17:43:23.9902694Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/aa0440be-e1d6-4589-afee-19e9cee5bb66' before making global git config changes 2024-08-07T17:43:23.9904178Z Adding repository directory to the temporary git global config as a safe directory 2024-08-07T17:43:23.9909332Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-07T17:43:23.9951871Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-07T17:43:23.9958039Z ##[group]Initializing the repository 2024-08-07T17:43:23.9961731Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-07T17:43:23.9991997Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-07T17:43:23.9993368Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-07T17:43:23.9994057Z hint: of your new repositories, which will suppress this warning, call: 2024-08-07T17:43:23.9994599Z hint: 2024-08-07T17:43:23.9995619Z hint: git config --global init.defaultBranch 2024-08-07T17:43:23.9996525Z hint: 2024-08-07T17:43:23.9996949Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-07T17:43:23.9997663Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-07T17:43:23.9998440Z hint: 2024-08-07T17:43:23.9998772Z hint: git branch -m 2024-08-07T17:43:23.9999423Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-07T17:43:24.0005137Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-07T17:43:24.0037725Z ##[endgroup] 2024-08-07T17:43:24.0038334Z ##[group]Disabling automatic garbage collection 2024-08-07T17:43:24.0041416Z [command]/usr/bin/git config --local gc.auto 0 2024-08-07T17:43:24.0074540Z ##[endgroup] 2024-08-07T17:43:24.0075098Z ##[group]Setting up auth 2024-08-07T17:43:24.0083589Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-07T17:43:24.0117451Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-07T17:43:24.0395254Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-07T17:43:24.0424421Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-07T17:43:24.0704518Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-07T17:43:24.0750295Z ##[endgroup] 2024-08-07T17:43:24.0751236Z ##[group]Fetching the repository 2024-08-07T17:43:24.0758841Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-07T17:43:26.5306427Z remote: Enumerating objects: 1008255 2024-08-07T17:43:26.5306985Z remote: Enumerating objects: 1011035, done. 2024-08-07T17:43:26.5309070Z remote: Counting objects: 0% (1/2780) 2024-08-07T17:43:26.5311189Z remote: Counting objects: 1% (28/2780) 2024-08-07T17:43:26.5313653Z remote: Counting objects: 2% (56/2780) 2024-08-07T17:43:26.5314150Z remote: Counting objects: 3% (84/2780) 2024-08-07T17:43:26.5314633Z remote: Counting objects: 4% (112/2780) 2024-08-07T17:43:26.5315187Z remote: Counting objects: 5% (139/2780) 2024-08-07T17:43:26.5316126Z remote: Counting objects: 6% (167/2780) 2024-08-07T17:43:26.5316619Z remote: Counting objects: 7% (195/2780) 2024-08-07T17:43:26.5317100Z remote: Counting objects: 8% (223/2780) 2024-08-07T17:43:26.5317563Z remote: Counting objects: 9% (251/2780) 2024-08-07T17:43:26.5318045Z remote: Counting objects: 10% (278/2780) 2024-08-07T17:43:26.5318948Z remote: Counting objects: 11% (306/2780) 2024-08-07T17:43:26.5319421Z remote: Counting objects: 12% (334/2780) 2024-08-07T17:43:26.5319930Z remote: Counting objects: 13% (362/2780) 2024-08-07T17:43:26.5320407Z remote: Counting objects: 14% (390/2780) 2024-08-07T17:43:26.5320877Z remote: Counting objects: 15% (417/2780) 2024-08-07T17:43:26.5321366Z remote: Counting objects: 16% (445/2780) 2024-08-07T17:43:26.5321847Z remote: Counting objects: 17% (473/2780) 2024-08-07T17:43:26.5322308Z remote: Counting objects: 18% (501/2780) 2024-08-07T17:43:26.5322785Z remote: Counting objects: 19% (529/2780) 2024-08-07T17:43:26.5323264Z remote: Counting objects: 20% (556/2780) 2024-08-07T17:43:26.5323723Z remote: Counting objects: 21% (584/2780) 2024-08-07T17:43:26.5324203Z remote: Counting objects: 22% (612/2780) 2024-08-07T17:43:26.5324687Z remote: Counting objects: 23% (640/2780) 2024-08-07T17:43:26.5325164Z remote: Counting objects: 24% (668/2780) 2024-08-07T17:43:26.5325650Z remote: Counting objects: 25% (695/2780) 2024-08-07T17:43:26.5326131Z remote: Counting objects: 26% (723/2780) 2024-08-07T17:43:26.5326681Z remote: Counting objects: 27% (751/2780) 2024-08-07T17:43:26.5327344Z remote: Counting objects: 28% (779/2780) 2024-08-07T17:43:26.5327860Z remote: Counting objects: 29% (807/2780) 2024-08-07T17:43:26.5328342Z remote: Counting objects: 30% (834/2780) 2024-08-07T17:43:26.5328804Z remote: Counting objects: 31% (862/2780) 2024-08-07T17:43:26.5329285Z remote: Counting objects: 32% (890/2780) 2024-08-07T17:43:26.5329769Z remote: Counting objects: 33% (918/2780) 2024-08-07T17:43:26.5330231Z remote: Counting objects: 34% (946/2780) 2024-08-07T17:43:26.5330706Z remote: Counting objects: 35% (973/2780) 2024-08-07T17:43:26.5331190Z remote: Counting objects: 36% (1001/2780) 2024-08-07T17:43:26.5331672Z remote: Counting objects: 37% (1029/2780) 2024-08-07T17:43:26.5332153Z remote: Counting objects: 38% (1057/2780) 2024-08-07T17:43:26.5332631Z remote: Counting objects: 39% (1085/2780) 2024-08-07T17:43:26.5333104Z remote: Counting objects: 40% (1112/2780) 2024-08-07T17:43:26.5333598Z remote: Counting objects: 41% (1140/2780) 2024-08-07T17:43:26.5334083Z remote: Counting objects: 42% (1168/2780) 2024-08-07T17:43:26.5334545Z remote: Counting objects: 43% (1196/2780) 2024-08-07T17:43:26.5335028Z remote: Counting objects: 44% (1224/2780) 2024-08-07T17:43:26.5335618Z remote: Counting objects: 45% (1251/2780) 2024-08-07T17:43:26.5336119Z remote: Counting objects: 46% (1279/2780) 2024-08-07T17:43:26.5336602Z remote: Counting objects: 47% (1307/2780) 2024-08-07T17:43:26.5337083Z remote: Counting objects: 48% (1335/2780) 2024-08-07T17:43:26.5337549Z remote: Counting objects: 49% (1363/2780) 2024-08-07T17:43:26.5338043Z remote: Counting objects: 50% (1390/2780) 2024-08-07T17:43:26.5338954Z remote: Counting objects: 51% (1418/2780) 2024-08-07T17:43:26.5339425Z remote: Counting objects: 52% (1446/2780) 2024-08-07T17:43:26.5339971Z remote: Counting objects: 53% (1474/2780) 2024-08-07T17:43:26.5340458Z remote: Counting objects: 54% (1502/2780) 2024-08-07T17:43:26.5340934Z remote: Counting objects: 55% (1529/2780) 2024-08-07T17:43:26.5341403Z remote: Counting objects: 56% (1557/2780) 2024-08-07T17:43:26.5341886Z remote: Counting objects: 57% (1585/2780) 2024-08-07T17:43:26.5342365Z remote: Counting objects: 58% (1613/2780) 2024-08-07T17:43:26.5342835Z remote: Counting objects: 59% (1641/2780) 2024-08-07T17:43:26.5343313Z remote: Counting objects: 60% (1668/2780) 2024-08-07T17:43:26.5343791Z remote: Counting objects: 61% (1696/2780) 2024-08-07T17:43:26.5344264Z remote: Counting objects: 62% (1724/2780) 2024-08-07T17:43:26.5344958Z remote: Counting objects: 63% (1752/2780) 2024-08-07T17:43:26.5345442Z remote: Counting objects: 64% (1780/2780) 2024-08-07T17:43:26.5345921Z remote: Counting objects: 65% (1807/2780) 2024-08-07T17:43:26.5346392Z remote: Counting objects: 66% (1835/2780) 2024-08-07T17:43:26.5346879Z remote: Counting objects: 67% (1863/2780) 2024-08-07T17:43:26.5347360Z remote: Counting objects: 68% (1891/2780) 2024-08-07T17:43:26.5347823Z remote: Counting objects: 69% (1919/2780) 2024-08-07T17:43:26.5348304Z remote: Counting objects: 70% (1946/2780) 2024-08-07T17:43:26.5348795Z remote: Counting objects: 71% (1974/2780) 2024-08-07T17:43:26.5349257Z remote: Counting objects: 72% (2002/2780) 2024-08-07T17:43:26.5349740Z remote: Counting objects: 73% (2030/2780) 2024-08-07T17:43:26.5350222Z remote: Counting objects: 74% (2058/2780) 2024-08-07T17:43:26.5350696Z remote: Counting objects: 75% (2085/2780) 2024-08-07T17:43:26.5351175Z remote: Counting objects: 76% (2113/2780) 2024-08-07T17:43:26.5351649Z remote: Counting objects: 77% (2141/2780) 2024-08-07T17:43:26.5352107Z remote: Counting objects: 78% (2169/2780) 2024-08-07T17:43:26.5352709Z remote: Counting objects: 79% (2197/2780) 2024-08-07T17:43:26.5353212Z remote: Counting objects: 80% (2224/2780) 2024-08-07T17:43:26.5353680Z remote: Counting objects: 81% (2252/2780) 2024-08-07T17:43:26.5354165Z remote: Counting objects: 82% (2280/2780) 2024-08-07T17:43:26.5354647Z remote: Counting objects: 83% (2308/2780) 2024-08-07T17:43:26.5355106Z remote: Counting objects: 84% (2336/2780) 2024-08-07T17:43:26.5355586Z remote: Counting objects: 85% (2363/2780) 2024-08-07T17:43:26.5356066Z remote: Counting objects: 86% (2391/2780) 2024-08-07T17:43:26.5356529Z remote: Counting objects: 87% (2419/2780) 2024-08-07T17:43:26.5357017Z remote: Counting objects: 88% (2447/2780) 2024-08-07T17:43:26.5357505Z remote: Counting objects: 89% (2475/2780) 2024-08-07T17:43:26.5357967Z remote: Counting objects: 90% (2502/2780) 2024-08-07T17:43:26.5358451Z remote: Counting objects: 91% (2530/2780) 2024-08-07T17:43:26.5358938Z remote: Counting objects: 92% (2558/2780) 2024-08-07T17:43:26.5359402Z remote: Counting objects: 93% (2586/2780) 2024-08-07T17:43:26.5359885Z remote: Counting objects: 94% (2614/2780) 2024-08-07T17:43:26.5360365Z remote: Counting objects: 95% (2641/2780) 2024-08-07T17:43:26.5360827Z remote: Counting objects: 96% (2669/2780) 2024-08-07T17:43:26.5361307Z remote: Counting objects: 97% (2697/2780) 2024-08-07T17:43:26.5361809Z remote: Counting objects: 98% (2725/2780) 2024-08-07T17:43:26.5362270Z remote: Counting objects: 99% (2753/2780) 2024-08-07T17:43:26.5362752Z remote: Counting objects: 100% (2780/2780) 2024-08-07T17:43:26.5363276Z remote: Counting objects: 100% (2780/2780), done. 2024-08-07T17:43:26.5725458Z remote: Compressing objects: 0% (1/1516) 2024-08-07T17:43:26.6094280Z remote: Compressing objects: 1% (16/1516) 2024-08-07T17:43:26.7277052Z remote: Compressing objects: 2% (31/1516) 2024-08-07T17:43:26.9618174Z remote: Compressing objects: 3% (46/1516) 2024-08-07T17:43:27.1299810Z remote: Compressing objects: 4% (61/1516) 2024-08-07T17:43:27.2384443Z remote: Compressing objects: 5% (76/1516) 2024-08-07T17:43:27.3396956Z remote: Compressing objects: 6% (91/1516) 2024-08-07T17:43:27.4076581Z remote: Compressing objects: 7% (107/1516) 2024-08-07T17:43:27.4670885Z remote: Compressing objects: 8% (122/1516) 2024-08-07T17:43:27.5055172Z remote: Compressing objects: 9% (137/1516) 2024-08-07T17:43:27.5395751Z remote: Compressing objects: 10% (152/1516) 2024-08-07T17:43:27.5398119Z remote: Compressing objects: 10% (166/1516) 2024-08-07T17:43:27.5728243Z remote: Compressing objects: 11% (167/1516) 2024-08-07T17:43:27.6072489Z remote: Compressing objects: 12% (182/1516) 2024-08-07T17:43:27.6295518Z remote: Compressing objects: 13% (198/1516) 2024-08-07T17:43:27.6609849Z remote: Compressing objects: 14% (213/1516) 2024-08-07T17:43:27.6843792Z remote: Compressing objects: 15% (228/1516) 2024-08-07T17:43:27.6974565Z remote: Compressing objects: 16% (243/1516) 2024-08-07T17:43:27.7077847Z remote: Compressing objects: 17% (258/1516) 2024-08-07T17:43:27.7162183Z remote: Compressing objects: 18% (273/1516) 2024-08-07T17:43:27.7239334Z remote: Compressing objects: 19% (289/1516) 2024-08-07T17:43:27.7285568Z remote: Compressing objects: 20% (304/1516) 2024-08-07T17:43:27.7315570Z remote: Compressing objects: 21% (319/1516) 2024-08-07T17:43:27.7329015Z remote: Compressing objects: 22% (334/1516) 2024-08-07T17:43:27.7346754Z remote: Compressing objects: 23% (349/1516) 2024-08-07T17:43:27.7382256Z remote: Compressing objects: 24% (364/1516) 2024-08-07T17:43:27.7404980Z remote: Compressing objects: 25% (379/1516) 2024-08-07T17:43:27.7428162Z remote: Compressing objects: 26% (395/1516) 2024-08-07T17:43:27.7447656Z remote: Compressing objects: 27% (410/1516) 2024-08-07T17:43:27.7469283Z remote: Compressing objects: 28% (425/1516) 2024-08-07T17:43:27.7487868Z remote: Compressing objects: 29% (440/1516) 2024-08-07T17:43:27.7508585Z remote: Compressing objects: 30% (455/1516) 2024-08-07T17:43:27.7542153Z remote: Compressing objects: 31% (470/1516) 2024-08-07T17:43:27.7560245Z remote: Compressing objects: 32% (486/1516) 2024-08-07T17:43:27.7577868Z remote: Compressing objects: 33% (501/1516) 2024-08-07T17:43:27.7593132Z remote: Compressing objects: 34% (516/1516) 2024-08-07T17:43:27.7607232Z remote: Compressing objects: 35% (531/1516) 2024-08-07T17:43:27.7618461Z remote: Compressing objects: 36% (546/1516) 2024-08-07T17:43:27.7630679Z remote: Compressing objects: 37% (561/1516) 2024-08-07T17:43:27.7642510Z remote: Compressing objects: 38% (577/1516) 2024-08-07T17:43:27.7658883Z remote: Compressing objects: 39% (592/1516) 2024-08-07T17:43:27.7666443Z remote: Compressing objects: 40% (607/1516) 2024-08-07T17:43:27.7683501Z remote: Compressing objects: 41% (622/1516) 2024-08-07T17:43:27.7694049Z remote: Compressing objects: 42% (637/1516) 2024-08-07T17:43:27.7705005Z remote: Compressing objects: 43% (652/1516) 2024-08-07T17:43:27.7714836Z remote: Compressing objects: 44% (668/1516) 2024-08-07T17:43:27.7722704Z remote: Compressing objects: 45% (683/1516) 2024-08-07T17:43:27.7729237Z remote: Compressing objects: 46% (698/1516) 2024-08-07T17:43:27.7740798Z remote: Compressing objects: 47% (713/1516) 2024-08-07T17:43:27.7745948Z remote: Compressing objects: 48% (728/1516) 2024-08-07T17:43:27.7751837Z remote: Compressing objects: 49% (743/1516) 2024-08-07T17:43:27.7758642Z remote: Compressing objects: 50% (758/1516) 2024-08-07T17:43:27.7763769Z remote: Compressing objects: 51% (774/1516) 2024-08-07T17:43:27.7769389Z remote: Compressing objects: 52% (789/1516) 2024-08-07T17:43:27.7774698Z remote: Compressing objects: 53% (804/1516) 2024-08-07T17:43:27.7778160Z remote: Compressing objects: 54% (819/1516) 2024-08-07T17:43:27.7781894Z remote: Compressing objects: 55% (834/1516) 2024-08-07T17:43:27.7784585Z remote: Compressing objects: 56% (849/1516) 2024-08-07T17:43:27.7789702Z remote: Compressing objects: 57% (865/1516) 2024-08-07T17:43:27.7792372Z remote: Compressing objects: 58% (880/1516) 2024-08-07T17:43:27.7795009Z remote: Compressing objects: 59% (895/1516) 2024-08-07T17:43:27.7795507Z remote: Compressing objects: 60% (910/1516) 2024-08-07T17:43:27.7796237Z remote: Compressing objects: 61% (925/1516) 2024-08-07T17:43:27.7797286Z remote: Compressing objects: 62% (940/1516) 2024-08-07T17:43:27.7797776Z remote: Compressing objects: 63% (956/1516) 2024-08-07T17:43:27.7800138Z remote: Compressing objects: 64% (971/1516) 2024-08-07T17:43:27.7800655Z remote: Compressing objects: 65% (986/1516) 2024-08-07T17:43:27.7827109Z remote: Compressing objects: 66% (1001/1516) 2024-08-07T17:43:27.7841025Z remote: Compressing objects: 67% (1016/1516) 2024-08-07T17:43:27.7853844Z remote: Compressing objects: 68% (1031/1516) 2024-08-07T17:43:27.7865843Z remote: Compressing objects: 69% (1047/1516) 2024-08-07T17:43:27.7875750Z remote: Compressing objects: 70% (1062/1516) 2024-08-07T17:43:27.7883176Z remote: Compressing objects: 71% (1077/1516) 2024-08-07T17:43:27.7890344Z remote: Compressing objects: 72% (1092/1516) 2024-08-07T17:43:27.7902493Z remote: Compressing objects: 73% (1107/1516) 2024-08-07T17:43:27.7907926Z remote: Compressing objects: 74% (1122/1516) 2024-08-07T17:43:27.7916619Z remote: Compressing objects: 75% (1137/1516) 2024-08-07T17:43:27.7919421Z remote: Compressing objects: 76% (1153/1516) 2024-08-07T17:43:27.7924728Z remote: Compressing objects: 77% (1168/1516) 2024-08-07T17:43:27.7930290Z remote: Compressing objects: 78% (1183/1516) 2024-08-07T17:43:27.7936241Z remote: Compressing objects: 79% (1198/1516) 2024-08-07T17:43:27.7941611Z remote: Compressing objects: 80% (1213/1516) 2024-08-07T17:43:27.7946738Z remote: Compressing objects: 81% (1228/1516) 2024-08-07T17:43:27.7951927Z remote: Compressing objects: 82% (1244/1516) 2024-08-07T17:43:27.7956418Z remote: Compressing objects: 83% (1259/1516) 2024-08-07T17:43:27.7960416Z remote: Compressing objects: 84% (1274/1516) 2024-08-07T17:43:27.7965358Z remote: Compressing objects: 85% (1289/1516) 2024-08-07T17:43:27.7973843Z remote: Compressing objects: 86% (1304/1516) 2024-08-07T17:43:27.7979152Z remote: Compressing objects: 87% (1319/1516) 2024-08-07T17:43:27.7983175Z remote: Compressing objects: 88% (1335/1516) 2024-08-07T17:43:27.7988013Z remote: Compressing objects: 89% (1350/1516) 2024-08-07T17:43:27.7988539Z remote: Compressing objects: 90% (1365/1516) 2024-08-07T17:43:27.7992809Z remote: Compressing objects: 91% (1380/1516) 2024-08-07T17:43:27.7995925Z remote: Compressing objects: 92% (1395/1516) 2024-08-07T17:43:27.7996440Z remote: Compressing objects: 93% (1410/1516) 2024-08-07T17:43:27.8000157Z remote: Compressing objects: 94% (1426/1516) 2024-08-07T17:43:27.8004776Z remote: Compressing objects: 95% (1441/1516) 2024-08-07T17:43:27.8005302Z remote: Compressing objects: 96% (1456/1516) 2024-08-07T17:43:27.8008775Z remote: Compressing objects: 97% (1471/1516) 2024-08-07T17:43:27.8014083Z remote: Compressing objects: 98% (1486/1516) 2024-08-07T17:43:27.8014605Z remote: Compressing objects: 99% (1501/1516) 2024-08-07T17:43:27.8015110Z remote: Compressing objects: 100% (1516/1516) 2024-08-07T17:43:27.8015640Z remote: Compressing objects: 100% (1516/1516), done. 2024-08-07T17:43:55.1371151Z remote: Total 1011035 (delta 1758), reused 2106 (delta 1257), pack-reused 1008255 2024-08-07T17:44:17.4470772Z [command]/usr/bin/git rev-parse --verify --quiet 016588f53c6904b840aa56aa86f95460b4d9c996^{object} 2024-08-07T17:44:17.4497582Z 016588f53c6904b840aa56aa86f95460b4d9c996 2024-08-07T17:44:17.4503418Z ##[endgroup] 2024-08-07T17:44:17.4562841Z ##[group]Determining the checkout info 2024-08-07T17:44:17.4563564Z ##[endgroup] 2024-08-07T17:44:17.4564109Z ##[group]Checking out the ref 2024-08-07T17:44:17.4564696Z [command]/usr/bin/git checkout --quiet --force 016588f53c6904b840aa56aa86f95460b4d9c996 2024-08-07T17:44:19.2435148Z ##[endgroup] 2024-08-07T17:44:19.2435743Z ##[group]Setting up auth for fetching submodules 2024-08-07T17:44:19.2441506Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-07T17:44:19.2495636Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-07T17:44:19.2527278Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-07T17:44:19.2558945Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-07T17:44:19.2587357Z ##[endgroup] 2024-08-07T17:44:19.2587885Z ##[group]Fetching submodules 2024-08-07T17:44:19.2592630Z [command]/usr/bin/git submodule sync --recursive 2024-08-07T17:44:19.2903214Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-07T17:44:19.3198503Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-07T17:44:19.3201004Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-07T17:44:19.3204295Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-07T17:44:19.3207278Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-07T17:44:19.3210887Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-07T17:44:19.3213940Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-07T17:44:19.3217514Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-07T17:44:19.3221947Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-07T17:44:19.3225718Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-07T17:44:19.3229839Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-07T17:44:19.3233823Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-07T17:44:19.3238015Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-07T17:44:19.3242454Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-07T17:44:19.3247032Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-07T17:44:19.3251576Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-07T17:44:19.3256333Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-08-07T17:44:19.3262080Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-07T17:44:19.3266886Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-07T17:44:19.3272139Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-07T17:44:19.3277287Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-07T17:44:19.3282717Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-07T17:44:19.3288335Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-07T17:44:19.3294363Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-07T17:44:19.3301227Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-07T17:44:19.3307150Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-07T17:44:19.3313165Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-07T17:44:19.3319653Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-07T17:44:19.3325815Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-07T17:44:19.3332282Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-07T17:44:19.3339318Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-07T17:44:19.3346497Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-07T17:44:19.3353191Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-07T17:44:19.3360055Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-07T17:44:19.3367114Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-07T17:44:19.3374436Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-07T17:44:19.3401458Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-07T17:44:19.6534760Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-07T17:44:19.8228455Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-07T17:44:20.0319519Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-07T17:44:20.2735439Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-07T17:44:22.3793364Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-07T17:44:35.1605772Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-07T17:44:35.5842721Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-07T17:44:35.9999858Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-07T17:44:36.6884202Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-07T17:44:38.2222421Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-07T17:44:40.5403409Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-07T17:44:47.6157442Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-07T17:44:50.4149655Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-07T17:44:52.2627897Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-07T17:44:53.5030371Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-08-07T17:44:53.6598486Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-07T17:44:54.1323203Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-07T17:44:54.4768604Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-07T17:44:55.6359376Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-07T17:44:55.9644683Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-07T17:44:56.2368010Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-07T17:44:58.0644597Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-07T17:44:58.8504408Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-07T17:44:59.5060481Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-07T17:45:06.3430759Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-07T17:45:08.4767220Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-07T17:45:12.2913717Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-07T17:45:12.5200967Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-07T17:45:22.4464055Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-07T17:45:22.6067657Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-07T17:45:22.8570294Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-07T17:45:23.8071194Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-07T17:45:24.1256563Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-07T17:45:24.7743981Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-07T17:45:25.2463781Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-07T17:45:25.2586988Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-07T17:45:25.2677228Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-07T17:45:25.2950449Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-07T17:45:25.3367171Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-07T17:45:26.4932422Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-07T17:45:26.5193003Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-07T17:45:26.5728200Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-07T17:45:26.6934702Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-07T17:45:26.7317312Z Submodule path 'third_party/cudnn_frontend': checked out '98ca4e1941fe3263f128f74f10063a3ea35c7019' 2024-08-07T17:45:27.3855119Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-07T17:45:27.6994194Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-07T17:45:27.7962922Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-07T17:45:27.7980639Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-07T17:45:27.7983789Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T17:45:27.7987108Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-07T17:45:27.7990552Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-07T17:45:27.7994277Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T17:45:27.8020125Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-07T17:45:28.9407100Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-07T17:45:29.6607144Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-07T17:45:32.0272330Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-07T17:45:33.2003060Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-07T17:45:33.5673388Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-07T17:45:33.6848957Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-07T17:45:34.1909255Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-07T17:45:34.2658532Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-07T17:45:34.2788366Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-07T17:45:34.4139677Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-07T17:45:34.4597475Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-07T17:45:34.4692890Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-08-07T17:45:34.5143590Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-07T17:45:34.5425440Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-07T17:45:34.5949932Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-07T17:45:34.6083927Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-07T17:45:34.6100269Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-07T17:45:34.6124880Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-07T17:45:48.9439264Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-07T17:45:48.9640865Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-07T17:45:49.0767731Z Submodule path 'third_party/kineto': checked out 'da2f2682cabaf95d601fa2a9b7e0979f84fe7667' 2024-08-07T17:45:49.0786180Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T17:45:49.0789083Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T17:45:49.0792410Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T17:45:49.0817864Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-07T17:45:49.6548571Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-07T17:45:50.9019765Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-07T17:45:52.1994401Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-07T17:45:52.2011048Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T17:45:52.2014086Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T17:45:52.2017277Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T17:45:52.2021186Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T17:45:52.2024737Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T17:45:52.2028148Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T17:45:52.2031788Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T17:45:52.2035598Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T17:45:52.2062966Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-07T17:45:53.2132569Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-07T17:45:53.6374484Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-07T17:45:54.8931696Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-07T17:45:55.1887074Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-07T17:45:55.8533926Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-07T17:45:58.2209298Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-07T17:46:06.3642869Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-07T17:46:06.7926917Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-07T17:46:06.8130891Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-07T17:46:06.8554201Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-07T17:46:06.8698688Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-07T17:46:06.8714391Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T17:46:06.8740003Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-07T17:46:07.1899109Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-07T17:46:07.2107413Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-07T17:46:07.2584431Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-07T17:46:07.3850996Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-07T17:46:07.4028469Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-07T17:46:07.4483075Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-07T17:46:07.5166777Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-07T17:46:07.5583992Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-07T17:46:07.5852923Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-07T17:46:07.7121443Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-07T17:46:08.1428253Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-07T17:46:08.1464249Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-07T17:46:08.1467124Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-07T17:46:08.1493757Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-07T17:46:08.5826407Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-07T17:46:09.5490701Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-07T17:46:09.5880565Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-07T17:46:09.6660248Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-07T17:46:09.6678861Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T17:46:09.6681880Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T17:46:09.6684918Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T17:46:09.6688188Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T17:46:09.6691737Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T17:46:09.6695287Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T17:46:09.6699009Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T17:46:09.6702794Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T17:46:09.6728549Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-07T17:46:10.0866794Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-07T17:46:11.2354247Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-07T17:46:11.6443587Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-07T17:46:18.4773084Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-07T17:46:18.8302214Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-07T17:46:19.0197832Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-07T17:46:19.3360845Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-07T17:46:26.3460997Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-07T17:46:26.3931047Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-07T17:46:26.4104692Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-07T17:46:26.5345394Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-07T17:46:26.5483414Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-07T17:46:26.5641049Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-07T17:46:26.5805139Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-07T17:46:26.5821548Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T17:46:26.5824875Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T17:46:26.5850745Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-07T17:46:28.6670710Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-07T17:46:30.1273099Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-07T17:46:30.1804128Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-07T17:46:30.7343081Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-07T17:46:30.7465688Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-07T17:46:31.0659704Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-07T17:46:31.0681855Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-07T17:46:31.0684880Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-07T17:46:31.0710888Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-07T17:46:31.5303049Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-07T17:46:32.6868907Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-07T17:46:32.7726643Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-07T17:46:32.7817043Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-07T17:46:32.7944506Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-07T17:46:32.8372233Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-07T17:46:32.8692348Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-07T17:46:32.9178778Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-07T17:46:32.9478959Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-07T17:46:32.9496123Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-07T17:46:32.9499158Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-07T17:46:32.9502847Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-07T17:46:32.9506210Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T17:46:32.9531169Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-07T17:46:34.1478168Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-07T17:46:34.4131284Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-07T17:46:35.6804561Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-07T17:46:36.6721980Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-07T17:46:36.6887724Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-07T17:46:36.7721978Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-07T17:46:36.8043572Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-07T17:46:36.8059083Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T17:46:36.8084745Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-07T17:46:36.9835963Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-07T17:46:36.9870274Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-07T17:46:37.0181014Z Entering 'android/libs/fbjni' 2024-08-07T17:46:37.0221529Z Entering 'third_party/FP16' 2024-08-07T17:46:37.0261278Z Entering 'third_party/FXdiv' 2024-08-07T17:46:37.0301179Z Entering 'third_party/NNPACK' 2024-08-07T17:46:37.0341290Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-07T17:46:37.0382306Z Entering 'third_party/XNNPACK' 2024-08-07T17:46:37.0439748Z Entering 'third_party/benchmark' 2024-08-07T17:46:37.0481347Z Entering 'third_party/cpp-httplib' 2024-08-07T17:46:37.0521344Z Entering 'third_party/cpuinfo' 2024-08-07T17:46:37.0562045Z Entering 'third_party/cudnn_frontend' 2024-08-07T17:46:37.0602470Z Entering 'third_party/cutlass' 2024-08-07T17:46:37.0651767Z Entering 'third_party/eigen' 2024-08-07T17:46:37.0694247Z Entering 'third_party/fbgemm' 2024-08-07T17:46:37.0734995Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-07T17:46:37.0775394Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T17:46:37.0815349Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-07T17:46:37.0861671Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-07T17:46:37.0901006Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T17:46:37.0941522Z Entering 'third_party/flatbuffers' 2024-08-07T17:46:37.0984927Z Entering 'third_party/fmt' 2024-08-07T17:46:37.1024599Z Entering 'third_party/foxi' 2024-08-07T17:46:37.1064531Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-07T17:46:37.1104258Z Entering 'third_party/gloo' 2024-08-07T17:46:37.1144200Z Entering 'third_party/googletest' 2024-08-07T17:46:37.1184471Z Entering 'third_party/ideep' 2024-08-07T17:46:37.1223465Z Entering 'third_party/ideep/mkl-dnn' 2024-08-07T17:46:37.1270158Z Entering 'third_party/ittapi' 2024-08-07T17:46:37.1309671Z Entering 'third_party/kineto' 2024-08-07T17:46:37.1349805Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T17:46:37.1388982Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T17:46:37.1430582Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T17:46:37.1469842Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T17:46:37.1509078Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T17:46:37.1547374Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T17:46:37.1588517Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T17:46:37.1627919Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T17:46:37.1666935Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T17:46:37.1707782Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T17:46:37.1747929Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T17:46:37.1787146Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T17:46:37.1827772Z Entering 'third_party/mimalloc' 2024-08-07T17:46:37.1867650Z Entering 'third_party/nccl/nccl' 2024-08-07T17:46:37.1907992Z Entering 'third_party/nlohmann' 2024-08-07T17:46:37.1949318Z Entering 'third_party/onnx' 2024-08-07T17:46:37.2005465Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-07T17:46:37.2045665Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-07T17:46:37.2087623Z Entering 'third_party/opentelemetry-cpp' 2024-08-07T17:46:37.2131267Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T17:46:37.2170728Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T17:46:37.2210698Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T17:46:37.2249900Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T17:46:37.2290841Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T17:46:37.2329716Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T17:46:37.2368533Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T17:46:37.2407066Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T17:46:37.2448323Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T17:46:37.2488986Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T17:46:37.2550452Z Entering 'third_party/pocketfft' 2024-08-07T17:46:37.2590116Z Entering 'third_party/protobuf' 2024-08-07T17:46:37.2634157Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-07T17:46:37.2673664Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-07T17:46:37.2713666Z Entering 'third_party/psimd' 2024-08-07T17:46:37.2754146Z Entering 'third_party/pthreadpool' 2024-08-07T17:46:37.2794065Z Entering 'third_party/pybind11' 2024-08-07T17:46:37.2834502Z Entering 'third_party/python-peachpy' 2024-08-07T17:46:37.2874444Z Entering 'third_party/sleef' 2024-08-07T17:46:37.2914856Z Entering 'third_party/tensorpipe' 2024-08-07T17:46:37.2955595Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-07T17:46:37.2994099Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-07T17:46:37.3032559Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-07T17:46:37.3071391Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T17:46:37.3109298Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T17:46:37.3164545Z ##[endgroup] 2024-08-07T17:46:37.3166482Z ##[group]Persisting credentials for submodules 2024-08-07T17:46:37.3172476Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-07T17:46:37.3473939Z Entering 'android/libs/fbjni' 2024-08-07T17:46:37.3523798Z Entering 'third_party/FP16' 2024-08-07T17:46:37.3573804Z Entering 'third_party/FXdiv' 2024-08-07T17:46:37.3624321Z Entering 'third_party/NNPACK' 2024-08-07T17:46:37.3674562Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-07T17:46:37.3724841Z Entering 'third_party/XNNPACK' 2024-08-07T17:46:37.3792998Z Entering 'third_party/benchmark' 2024-08-07T17:46:37.3842809Z Entering 'third_party/cpp-httplib' 2024-08-07T17:46:37.3892514Z Entering 'third_party/cpuinfo' 2024-08-07T17:46:37.3944577Z Entering 'third_party/cudnn_frontend' 2024-08-07T17:46:37.3993369Z Entering 'third_party/cutlass' 2024-08-07T17:46:37.4051294Z Entering 'third_party/eigen' 2024-08-07T17:46:37.4103156Z Entering 'third_party/fbgemm' 2024-08-07T17:46:37.4153436Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-07T17:46:37.4203803Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T17:46:37.4252718Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-07T17:46:37.4308255Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-07T17:46:37.4356786Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T17:46:37.4406936Z Entering 'third_party/flatbuffers' 2024-08-07T17:46:37.4459647Z Entering 'third_party/fmt' 2024-08-07T17:46:37.4509271Z Entering 'third_party/foxi' 2024-08-07T17:46:37.4558538Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-07T17:46:37.4607487Z Entering 'third_party/gloo' 2024-08-07T17:46:37.4657741Z Entering 'third_party/googletest' 2024-08-07T17:46:37.4707662Z Entering 'third_party/ideep' 2024-08-07T17:46:37.4755518Z Entering 'third_party/ideep/mkl-dnn' 2024-08-07T17:46:37.4811179Z Entering 'third_party/ittapi' 2024-08-07T17:46:37.4860872Z Entering 'third_party/kineto' 2024-08-07T17:46:37.4911521Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T17:46:37.4961103Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T17:46:37.5012142Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T17:46:37.5062554Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T17:46:37.5111608Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T17:46:37.5159307Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T17:46:37.5210457Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T17:46:37.5260811Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T17:46:37.5310376Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T17:46:37.5363264Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T17:46:37.5414401Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T17:46:37.5463088Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T17:46:37.5512894Z Entering 'third_party/mimalloc' 2024-08-07T17:46:37.5563555Z Entering 'third_party/nccl/nccl' 2024-08-07T17:46:37.5613588Z Entering 'third_party/nlohmann' 2024-08-07T17:46:37.5665938Z Entering 'third_party/onnx' 2024-08-07T17:46:37.5732404Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-07T17:46:37.5782087Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-07T17:46:37.5833662Z Entering 'third_party/opentelemetry-cpp' 2024-08-07T17:46:37.5884713Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T17:46:37.5933165Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T17:46:37.5982903Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T17:46:37.6030711Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T17:46:37.6081217Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T17:46:37.6130080Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T17:46:37.6178729Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T17:46:37.6227248Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T17:46:37.6277596Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T17:46:37.6328829Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T17:46:37.6397615Z Entering 'third_party/pocketfft' 2024-08-07T17:46:37.6447269Z Entering 'third_party/protobuf' 2024-08-07T17:46:37.6499316Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-07T17:46:37.6548046Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-07T17:46:37.6598236Z Entering 'third_party/psimd' 2024-08-07T17:46:37.6647430Z Entering 'third_party/pthreadpool' 2024-08-07T17:46:37.6696022Z Entering 'third_party/pybind11' 2024-08-07T17:46:37.6746421Z Entering 'third_party/python-peachpy' 2024-08-07T17:46:37.6795668Z Entering 'third_party/sleef' 2024-08-07T17:46:37.6845081Z Entering 'third_party/tensorpipe' 2024-08-07T17:46:37.6894581Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-07T17:46:37.6944821Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-07T17:46:37.6992614Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-07T17:46:37.7041650Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T17:46:37.7090078Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T17:46:37.7156123Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-07T17:46:37.7454881Z Entering 'android/libs/fbjni' 2024-08-07T17:46:37.7502060Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-07T17:46:37.7519659Z Entering 'third_party/FP16' 2024-08-07T17:46:37.7566667Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-07T17:46:37.7585673Z Entering 'third_party/FXdiv' 2024-08-07T17:46:37.7631457Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-07T17:46:37.7649389Z Entering 'third_party/NNPACK' 2024-08-07T17:46:37.7696086Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-07T17:46:37.7715559Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-07T17:46:37.7761795Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-07T17:46:37.7780398Z Entering 'third_party/XNNPACK' 2024-08-07T17:46:37.7826246Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-07T17:46:37.7862427Z Entering 'third_party/benchmark' 2024-08-07T17:46:37.7908047Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-07T17:46:37.7925862Z Entering 'third_party/cpp-httplib' 2024-08-07T17:46:37.7972449Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-07T17:46:37.7990881Z Entering 'third_party/cpuinfo' 2024-08-07T17:46:37.8036445Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-07T17:46:37.8055679Z Entering 'third_party/cudnn_frontend' 2024-08-07T17:46:37.8101976Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-07T17:46:37.8119837Z Entering 'third_party/cutlass' 2024-08-07T17:46:37.8165075Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-07T17:46:37.8191884Z Entering 'third_party/eigen' 2024-08-07T17:46:37.8237063Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-07T17:46:37.8257196Z Entering 'third_party/fbgemm' 2024-08-07T17:46:37.8304923Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-07T17:46:37.8322957Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-07T17:46:37.8368008Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-07T17:46:37.8386038Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T17:46:37.8431354Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-07T17:46:37.8449272Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-07T17:46:37.8494947Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-07T17:46:37.8519205Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-07T17:46:37.8564446Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-07T17:46:37.8582613Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T17:46:37.8628617Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-07T17:46:37.8647314Z Entering 'third_party/flatbuffers' 2024-08-07T17:46:37.8692783Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-07T17:46:37.8713969Z Entering 'third_party/fmt' 2024-08-07T17:46:37.8760059Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-07T17:46:37.8778145Z Entering 'third_party/foxi' 2024-08-07T17:46:37.8825074Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-08-07T17:46:37.8843300Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-07T17:46:37.8889361Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-07T17:46:37.8908511Z Entering 'third_party/gloo' 2024-08-07T17:46:37.8953671Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-07T17:46:37.8972040Z Entering 'third_party/googletest' 2024-08-07T17:46:37.9019355Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-07T17:46:37.9037975Z Entering 'third_party/ideep' 2024-08-07T17:46:37.9084130Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-07T17:46:37.9101655Z Entering 'third_party/ideep/mkl-dnn' 2024-08-07T17:46:37.9147453Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-07T17:46:37.9171697Z Entering 'third_party/ittapi' 2024-08-07T17:46:37.9218335Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-07T17:46:37.9236569Z Entering 'third_party/kineto' 2024-08-07T17:46:37.9282574Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-07T17:46:37.9301412Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T17:46:37.9348392Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-07T17:46:37.9365512Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T17:46:37.9411534Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-07T17:46:37.9430882Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T17:46:37.9477332Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-07T17:46:37.9494828Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T17:46:37.9543304Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-07T17:46:37.9560917Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T17:46:37.9607094Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-07T17:46:37.9624069Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T17:46:37.9671438Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-07T17:46:37.9690419Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T17:46:37.9736938Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-07T17:46:37.9754238Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T17:46:37.9800473Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-07T17:46:37.9818127Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T17:46:37.9865641Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-07T17:46:37.9884197Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T17:46:37.9929911Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-07T17:46:37.9949216Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T17:46:37.9994743Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-07T17:46:38.0011938Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T17:46:38.0058071Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-07T17:46:38.0077720Z Entering 'third_party/mimalloc' 2024-08-07T17:46:38.0123318Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-07T17:46:38.0142116Z Entering 'third_party/nccl/nccl' 2024-08-07T17:46:38.0187684Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-07T17:46:38.0206272Z Entering 'third_party/nlohmann' 2024-08-07T17:46:38.0251908Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-07T17:46:38.0271590Z Entering 'third_party/onnx' 2024-08-07T17:46:38.0318499Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-07T17:46:38.0354147Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-07T17:46:38.0400314Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-07T17:46:38.0417428Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-07T17:46:38.0465110Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-07T17:46:38.0484792Z Entering 'third_party/opentelemetry-cpp' 2024-08-07T17:46:38.0531312Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-07T17:46:38.0550776Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T17:46:38.0597194Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-07T17:46:38.0614439Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T17:46:38.0660350Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-07T17:46:38.0678429Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T17:46:38.0725211Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-07T17:46:38.0742596Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T17:46:38.0788789Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-07T17:46:38.0807906Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T17:46:38.0854035Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-07T17:46:38.0871348Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T17:46:38.0916554Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-07T17:46:38.0933973Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T17:46:38.0979798Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-07T17:46:38.0996298Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T17:46:38.1042029Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-07T17:46:38.1062224Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T17:46:38.1107746Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-07T17:46:38.1126965Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T17:46:38.1171980Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-07T17:46:38.1210793Z Entering 'third_party/pocketfft' 2024-08-07T17:46:38.1257107Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-07T17:46:38.1275112Z Entering 'third_party/protobuf' 2024-08-07T17:46:38.1321682Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-07T17:46:38.1343138Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-07T17:46:38.1388415Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-07T17:46:38.1405750Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-07T17:46:38.1451560Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-07T17:46:38.1472519Z Entering 'third_party/psimd' 2024-08-07T17:46:38.1519521Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-07T17:46:38.1536952Z Entering 'third_party/pthreadpool' 2024-08-07T17:46:38.1583604Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-07T17:46:38.1601540Z Entering 'third_party/pybind11' 2024-08-07T17:46:38.1647513Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-07T17:46:38.1665908Z Entering 'third_party/python-peachpy' 2024-08-07T17:46:38.1711920Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-07T17:46:38.1729730Z Entering 'third_party/sleef' 2024-08-07T17:46:38.1775901Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-07T17:46:38.1794333Z Entering 'third_party/tensorpipe' 2024-08-07T17:46:38.1840156Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-07T17:46:38.1857510Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-07T17:46:38.1903743Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-07T17:46:38.1921803Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-07T17:46:38.1967928Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-07T17:46:38.1985569Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-07T17:46:38.2030833Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-07T17:46:38.2048635Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T17:46:38.2094086Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-07T17:46:38.2110935Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T17:46:38.2156740Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-07T17:46:38.3173061Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-07T17:46:38.3473646Z Entering 'android/libs/fbjni' 2024-08-07T17:46:38.3513031Z Entering 'third_party/FP16' 2024-08-07T17:46:38.3553231Z Entering 'third_party/FXdiv' 2024-08-07T17:46:38.3593002Z Entering 'third_party/NNPACK' 2024-08-07T17:46:38.3634395Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-07T17:46:38.3674213Z Entering 'third_party/XNNPACK' 2024-08-07T17:46:38.3731725Z Entering 'third_party/benchmark' 2024-08-07T17:46:38.3771644Z Entering 'third_party/cpp-httplib' 2024-08-07T17:46:38.3811791Z Entering 'third_party/cpuinfo' 2024-08-07T17:46:38.3852578Z Entering 'third_party/cudnn_frontend' 2024-08-07T17:46:38.3892623Z Entering 'third_party/cutlass' 2024-08-07T17:46:38.3940153Z Entering 'third_party/eigen' 2024-08-07T17:46:38.3998101Z Entering 'third_party/fbgemm' 2024-08-07T17:46:38.4038241Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-07T17:46:38.4077180Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T17:46:38.4117349Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-07T17:46:38.4162509Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-07T17:46:38.4203124Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T17:46:38.4242868Z Entering 'third_party/flatbuffers' 2024-08-07T17:46:38.4286047Z Entering 'third_party/fmt' 2024-08-07T17:46:38.4325570Z Entering 'third_party/foxi' 2024-08-07T17:46:38.4365680Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-07T17:46:38.4405696Z Entering 'third_party/gloo' 2024-08-07T17:46:38.4446328Z Entering 'third_party/googletest' 2024-08-07T17:46:38.4486607Z Entering 'third_party/ideep' 2024-08-07T17:46:38.4525367Z Entering 'third_party/ideep/mkl-dnn' 2024-08-07T17:46:38.4571692Z Entering 'third_party/ittapi' 2024-08-07T17:46:38.4611178Z Entering 'third_party/kineto' 2024-08-07T17:46:38.4650610Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T17:46:38.4690524Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T17:46:38.4731315Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T17:46:38.4771781Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T17:46:38.4811748Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T17:46:38.4851898Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T17:46:38.4893781Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T17:46:38.4935424Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T17:46:38.4975087Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T17:46:38.5016305Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T17:46:38.5059348Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T17:46:38.5098449Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T17:46:38.5139251Z Entering 'third_party/mimalloc' 2024-08-07T17:46:38.5179889Z Entering 'third_party/nccl/nccl' 2024-08-07T17:46:38.5220134Z Entering 'third_party/nlohmann' 2024-08-07T17:46:38.5261646Z Entering 'third_party/onnx' 2024-08-07T17:46:38.5318740Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-07T17:46:38.5358584Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-07T17:46:38.5401169Z Entering 'third_party/opentelemetry-cpp' 2024-08-07T17:46:38.5444760Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T17:46:38.5484600Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T17:46:38.5523756Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T17:46:38.5563060Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T17:46:38.5603686Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T17:46:38.5642492Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T17:46:38.5682151Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T17:46:38.5720501Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T17:46:38.5763091Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T17:46:38.5803296Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T17:46:38.5863613Z Entering 'third_party/pocketfft' 2024-08-07T17:46:38.5903078Z Entering 'third_party/protobuf' 2024-08-07T17:46:38.5945984Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-07T17:46:38.5985257Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-07T17:46:38.6026809Z Entering 'third_party/psimd' 2024-08-07T17:46:38.6066570Z Entering 'third_party/pthreadpool' 2024-08-07T17:46:38.6106564Z Entering 'third_party/pybind11' 2024-08-07T17:46:38.6146767Z Entering 'third_party/python-peachpy' 2024-08-07T17:46:38.6186762Z Entering 'third_party/sleef' 2024-08-07T17:46:38.6226259Z Entering 'third_party/tensorpipe' 2024-08-07T17:46:38.6266091Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-07T17:46:38.6305893Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-07T17:46:38.6345443Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-07T17:46:38.6384874Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T17:46:38.6422861Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T17:46:38.6478684Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-07T17:46:38.6777675Z Entering 'android/libs/fbjni' 2024-08-07T17:46:38.6819347Z Entering 'third_party/FP16' 2024-08-07T17:46:38.6861132Z Entering 'third_party/FXdiv' 2024-08-07T17:46:38.6903391Z Entering 'third_party/NNPACK' 2024-08-07T17:46:38.6943921Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-07T17:46:38.6984420Z Entering 'third_party/XNNPACK' 2024-08-07T17:46:38.7041582Z Entering 'third_party/benchmark' 2024-08-07T17:46:38.7083104Z Entering 'third_party/cpp-httplib' 2024-08-07T17:46:38.7123731Z Entering 'third_party/cpuinfo' 2024-08-07T17:46:38.7164647Z Entering 'third_party/cudnn_frontend' 2024-08-07T17:46:38.7205588Z Entering 'third_party/cutlass' 2024-08-07T17:46:38.7254262Z Entering 'third_party/eigen' 2024-08-07T17:46:38.7297404Z Entering 'third_party/fbgemm' 2024-08-07T17:46:38.7337328Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-07T17:46:38.7377188Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T17:46:38.7418087Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-07T17:46:38.7464401Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-07T17:46:38.7503818Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T17:46:38.7544140Z Entering 'third_party/flatbuffers' 2024-08-07T17:46:38.7587457Z Entering 'third_party/fmt' 2024-08-07T17:46:38.7628350Z Entering 'third_party/foxi' 2024-08-07T17:46:38.7668500Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-07T17:46:38.7708166Z Entering 'third_party/gloo' 2024-08-07T17:46:38.7748062Z Entering 'third_party/googletest' 2024-08-07T17:46:38.7789100Z Entering 'third_party/ideep' 2024-08-07T17:46:38.7827902Z Entering 'third_party/ideep/mkl-dnn' 2024-08-07T17:46:38.7875399Z Entering 'third_party/ittapi' 2024-08-07T17:46:38.7915614Z Entering 'third_party/kineto' 2024-08-07T17:46:38.7955760Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T17:46:38.7995365Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T17:46:38.8036678Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T17:46:38.8077412Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T17:46:38.8118414Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T17:46:38.8157079Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T17:46:38.8198563Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T17:46:38.8238792Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T17:46:38.8279296Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T17:46:38.8320434Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T17:46:38.8361316Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T17:46:38.8401118Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T17:46:38.8443137Z Entering 'third_party/mimalloc' 2024-08-07T17:46:38.8485127Z Entering 'third_party/nccl/nccl' 2024-08-07T17:46:38.8525688Z Entering 'third_party/nlohmann' 2024-08-07T17:46:38.8567360Z Entering 'third_party/onnx' 2024-08-07T17:46:38.8625487Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-07T17:46:38.8665180Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-07T17:46:38.8707158Z Entering 'third_party/opentelemetry-cpp' 2024-08-07T17:46:38.8750765Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T17:46:38.8790050Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T17:46:38.8829799Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T17:46:38.8869557Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T17:46:38.8910514Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T17:46:38.8949796Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T17:46:38.8989082Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T17:46:38.9028298Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T17:46:38.9069545Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T17:46:38.9110797Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T17:46:38.9172025Z Entering 'third_party/pocketfft' 2024-08-07T17:46:38.9212336Z Entering 'third_party/protobuf' 2024-08-07T17:46:38.9255239Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-07T17:46:38.9295342Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-07T17:46:38.9336708Z Entering 'third_party/psimd' 2024-08-07T17:46:38.9377586Z Entering 'third_party/pthreadpool' 2024-08-07T17:46:38.9418025Z Entering 'third_party/pybind11' 2024-08-07T17:46:38.9457505Z Entering 'third_party/python-peachpy' 2024-08-07T17:46:38.9497902Z Entering 'third_party/sleef' 2024-08-07T17:46:38.9537579Z Entering 'third_party/tensorpipe' 2024-08-07T17:46:38.9577972Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-07T17:46:38.9617005Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-07T17:46:38.9656151Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-07T17:46:38.9696122Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T17:46:38.9734665Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T17:46:38.9786780Z ##[endgroup] 2024-08-07T17:46:38.9827451Z [command]/usr/bin/git log -1 --format='%H' 2024-08-07T17:46:38.9860103Z '016588f53c6904b840aa56aa86f95460b4d9c996' 2024-08-07T17:46:39.0039332Z Prepare all required actions 2024-08-07T17:46:39.0040015Z Getting action download info 2024-08-07T17:46:39.1413566Z ##[group]Run ./.github/actions/setup-linux 2024-08-07T17:46:39.1413995Z env: 2024-08-07T17:46:39.1414277Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:39.1414627Z ##[endgroup] 2024-08-07T17:46:39.1499685Z ##[group]Run set -euo pipefail 2024-08-07T17:46:39.1500123Z set -euo pipefail 2024-08-07T17:46:39.1500510Z function get_ec2_metadata() { 2024-08-07T17:46:39.1500973Z  # Pulled from instance metadata endpoint for EC2 2024-08-07T17:46:39.1501745Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-07T17:46:39.1502436Z  category=$1 2024-08-07T17:46:39.1502882Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-07T17:46:39.1503427Z  runner_name_str=i-028914771a8277c83 2024-08-07T17:46:39.1503945Z  if [[ -f /.inarc ]]; then 2024-08-07T17:46:39.1504375Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-07T17:46:39.1504869Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-07T17:46:39.1505451Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-07T17:46:39.1505967Z  else 2024-08-07T17:46:39.1506407Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-07T17:46:39.1506929Z  fi 2024-08-07T17:46:39.1507206Z } 2024-08-07T17:46:39.1507550Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-07T17:46:39.1508085Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-07T17:46:39.1508663Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-07T17:46:39.1509186Z echo "system info $(uname -a)" 2024-08-07T17:46:39.1517265Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:46:39.1517734Z env: 2024-08-07T17:46:39.1518031Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:39.1518376Z ##[endgroup] 2024-08-07T17:46:39.1617299Z ami-id: ami-0ce0c36d7a00b20e2 2024-08-07T17:46:39.1682620Z instance-id: i-028914771a8277c83 2024-08-07T17:46:39.1745956Z instance-type: g3.8xlarge 2024-08-07T17:46:39.1754139Z system info Linux ip-10-0-48-24.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-07T17:46:39.1784073Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-07T17:46:39.1784884Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-07T17:46:39.1792423Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:46:39.1792899Z env: 2024-08-07T17:46:39.1793189Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:39.1793514Z ##[endgroup] 2024-08-07T17:46:39.1877097Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-07T17:46:39.1877696Z if systemctl is-active --quiet docker; then 2024-08-07T17:46:39.1878174Z  echo "Docker daemon is running..."; 2024-08-07T17:46:39.1878596Z else 2024-08-07T17:46:39.1879052Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-07T17:46:39.1879565Z fi 2024-08-07T17:46:39.1886858Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:46:39.1887342Z env: 2024-08-07T17:46:39.1887615Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:39.1887961Z ##[endgroup] 2024-08-07T17:46:39.1933828Z Docker daemon is running... 2024-08-07T17:46:39.1987049Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-07T17:46:39.1987561Z with: 2024-08-07T17:46:39.1987845Z shell: bash 2024-08-07T17:46:39.1988148Z timeout_minutes: 5 2024-08-07T17:46:39.1988454Z max_attempts: 3 2024-08-07T17:46:39.1988773Z retry_wait_seconds: 30 2024-08-07T17:46:39.1990206Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-08-07T17:46:39.1991653Z polling_interval_seconds: 1 2024-08-07T17:46:39.1992010Z warning_on_retry: true 2024-08-07T17:46:39.1992355Z continue_on_error: false 2024-08-07T17:46:39.1992692Z env: 2024-08-07T17:46:39.1992956Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:39.1993302Z AWS_RETRY_MODE: standard 2024-08-07T17:46:39.1993648Z AWS_MAX_ATTEMPTS: 5 2024-08-07T17:46:39.1993969Z AWS_DEFAULT_REGION: us-east-1 2024-08-07T17:46:39.1994324Z ##[endgroup] 2024-08-07T17:46:40.1915349Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-07T17:46:40.1916096Z Configure a credential helper to remove this warning. See 2024-08-07T17:46:40.1916819Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-07T17:46:40.1917335Z 2024-08-07T17:46:40.1917504Z Login Succeeded 2024-08-07T17:46:40.2687816Z Command completed after 1 attempt(s). 2024-08-07T17:46:40.2748895Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-07T17:46:40.2749531Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-07T17:46:40.2750112Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-07T17:46:40.2759181Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:46:40.2759644Z env: 2024-08-07T17:46:40.2759936Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:40.2760287Z ##[endgroup] 2024-08-07T17:46:40.2839925Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-07T17:46:40.2840631Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-07T17:46:40.2841153Z # shellcheck disable=SC2046 2024-08-07T17:46:40.2841590Z docker stop $(docker ps -q) || true 2024-08-07T17:46:40.2842056Z # Prune all of the docker images 2024-08-07T17:46:40.2842483Z docker system prune -af 2024-08-07T17:46:40.2849782Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:46:40.2850264Z env: 2024-08-07T17:46:40.2850539Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:40.2850887Z ##[endgroup] 2024-08-07T17:46:40.3176124Z "docker stop" requires at least 1 argument. 2024-08-07T17:46:40.3176591Z See 'docker stop --help'. 2024-08-07T17:46:40.3176841Z 2024-08-07T17:46:40.3177060Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-07T17:46:40.3177398Z 2024-08-07T17:46:40.3177570Z Stop one or more running containers 2024-08-07T17:46:40.3356196Z Total reclaimed space: 0B 2024-08-07T17:46:40.3395025Z ##[group]Run set +e 2024-08-07T17:46:40.3395405Z set +e 2024-08-07T17:46:40.3395703Z set -x 2024-08-07T17:46:40.3396011Z  2024-08-07T17:46:40.3396340Z PT_DOMAIN=download.pytorch.org 2024-08-07T17:46:40.3397054Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-07T17:46:40.3397990Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-07T17:46:40.3398643Z # one is returned at random 2024-08-07T17:46:40.3399148Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-07T17:46:40.3399615Z  2024-08-07T17:46:40.3399934Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-07T17:46:40.3400483Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-07T17:46:40.3401120Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-07T17:46:40.3401617Z  2024-08-07T17:46:40.3401935Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-07T17:46:40.3402406Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-07T17:46:40.3402871Z  exit 1 2024-08-07T17:46:40.3403186Z  fi 2024-08-07T17:46:40.3403641Z fi 2024-08-07T17:46:40.3404069Z  2024-08-07T17:46:40.3404429Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-07T17:46:40.3404888Z  # Clean up any old records first 2024-08-07T17:46:40.3405353Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-07T17:46:40.3405775Z fi 2024-08-07T17:46:40.3406043Z  2024-08-07T17:46:40.3406451Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-07T17:46:40.3406955Z cat /etc/hosts 2024-08-07T17:46:40.3414527Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:46:40.3415005Z env: 2024-08-07T17:46:40.3415296Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:40.3415627Z ##[endgroup] 2024-08-07T17:46:40.3439009Z + PT_DOMAIN=download.pytorch.org 2024-08-07T17:46:40.3444168Z ++ dig -4 +short download.pytorch.org 2024-08-07T17:46:40.3445178Z ++ tail -n1 2024-08-07T17:46:40.3604586Z + RESOLVED_IP=18.160.10.22 2024-08-07T17:46:40.3604970Z + '[' -z 18.160.10.22 ']' 2024-08-07T17:46:40.3605367Z + grep -r download.pytorch.org /etc/hosts 2024-08-07T17:46:40.3613880Z 18.160.10.22 download.pytorch.org 2024-08-07T17:46:40.3614941Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-08-07T17:46:40.3733627Z + echo '18.160.10.22 download.pytorch.org' 2024-08-07T17:46:40.3734461Z + sudo tee -a /etc/hosts 2024-08-07T17:46:40.3822969Z 18.160.10.22 download.pytorch.org 2024-08-07T17:46:40.3838300Z + cat /etc/hosts 2024-08-07T17:46:40.3844386Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-07T17:46:40.3849581Z ::1 localhost6 localhost6.localdomain6 2024-08-07T17:46:40.3850042Z 18.160.10.22 download.pytorch.org 2024-08-07T17:46:40.4004355Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-07T17:46:40.4004983Z with: 2024-08-07T17:46:40.4005883Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:40.4006886Z docker-build-dir: .ci/docker 2024-08-07T17:46:40.4007266Z working-directory: . 2024-08-07T17:46:40.4007715Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:40.4008203Z force-push: false 2024-08-07T17:46:40.4008516Z env: 2024-08-07T17:46:40.4008805Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:40.4009137Z ##[endgroup] 2024-08-07T17:46:40.4042435Z ##[group]Run set -ex 2024-08-07T17:46:40.4042828Z set -ex 2024-08-07T17:46:40.4043131Z  2024-08-07T17:46:40.4043629Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-07T17:46:40.4044482Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-07T17:46:40.4045216Z # job could then download the pre-built image as usual 2024-08-07T17:46:40.4045860Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-07T17:46:40.4046456Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-07T17:46:40.4047018Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-07T17:46:40.4047513Z  2024-08-07T17:46:40.4047959Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-07T17:46:40.4048509Z  exit 0 2024-08-07T17:46:40.4048797Z else 2024-08-07T17:46:40.4049143Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-07T17:46:40.4049565Z fi 2024-08-07T17:46:40.4049834Z  2024-08-07T17:46:40.4050266Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-07T17:46:40.4051013Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-07T17:46:40.4051657Z  # use it as it is, but first let's extract the tag 2024-08-07T17:46:40.4052256Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-07T17:46:40.4053065Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-07T17:46:40.4053654Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-07T17:46:40.4054166Z else 2024-08-07T17:46:40.4054572Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-07T17:46:40.4055135Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-07T17:46:40.4055911Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-07T17:46:40.4056594Z fi 2024-08-07T17:46:40.4064618Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:46:40.4065091Z env: 2024-08-07T17:46:40.4065386Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:40.4065726Z REPO_NAME: pytorch 2024-08-07T17:46:40.4066646Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:40.4067649Z DOCKER_BUILD_DIR: .ci/docker 2024-08-07T17:46:40.4068124Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:40.4068604Z ##[endgroup] 2024-08-07T17:46:40.4091403Z + [[ ! -d .ci/docker ]] 2024-08-07T17:46:40.4091772Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-07T17:46:40.4092125Z + echo skip=false 2024-08-07T17:46:40.4093355Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-07T17:46:40.4098712Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:40.4099699Z ++ awk -F '[:,]' '{print $2}' 2024-08-07T17:46:40.4109664Z + DOCKER_TAG=02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:40.4110240Z + echo docker-tag=02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:40.4111553Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:40.4147578Z ##[group]Run set +e 2024-08-07T17:46:40.4147989Z set +e 2024-08-07T17:46:40.4148284Z set -x 2024-08-07T17:46:40.4148586Z  2024-08-07T17:46:40.4148875Z login() { 2024-08-07T17:46:40.4149452Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-07T17:46:40.4150096Z } 2024-08-07T17:46:40.4150392Z  2024-08-07T17:46:40.4150673Z retry () { 2024-08-07T17:46:40.4151040Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-07T17:46:40.4151456Z } 2024-08-07T17:46:40.4151738Z  2024-08-07T17:46:40.4152056Z retry login "${DOCKER_REGISTRY}" 2024-08-07T17:46:40.4152460Z  2024-08-07T17:46:40.4152879Z # Check if image already exists, if it does then skip building it 2024-08-07T17:46:40.4153496Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-07T17:46:40.4153962Z  exit 0 2024-08-07T17:46:40.4154249Z fi 2024-08-07T17:46:40.4154536Z  2024-08-07T17:46:40.4154985Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-07T17:46:40.4155670Z # be empty. The default action would be to continue rebuild the image 2024-08-07T17:46:40.4156314Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-07T17:46:40.4156891Z  # if we're on the base branch then use the parent commit 2024-08-07T17:46:40.4157387Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-07T17:46:40.4157803Z else 2024-08-07T17:46:40.4158224Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-07T17:46:40.4158800Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-07T17:46:40.4159431Z fi 2024-08-07T17:46:40.4159718Z  2024-08-07T17:46:40.4160017Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-07T17:46:40.4160479Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-07T17:46:40.4160907Z  2024-08-07T17:46:40.4161471Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-07T17:46:40.4162145Z  exit 0 2024-08-07T17:46:40.4162451Z fi 2024-08-07T17:46:40.4162722Z  2024-08-07T17:46:40.4163116Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-07T17:46:40.4163946Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-07T17:46:40.4164658Z  exit 1 2024-08-07T17:46:40.4164944Z fi 2024-08-07T17:46:40.4165227Z  2024-08-07T17:46:40.4165689Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-07T17:46:40.4166467Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-07T17:46:40.4167185Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-07T17:46:40.4168004Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-07T17:46:40.4168900Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-07T17:46:40.4169468Z fi 2024-08-07T17:46:40.4169752Z  2024-08-07T17:46:40.4170211Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-07T17:46:40.4177305Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:46:40.4177782Z env: 2024-08-07T17:46:40.4178058Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:40.4178723Z DOCKER_BUILD_DIR: .ci/docker 2024-08-07T17:46:40.4179176Z BASE_REVISION: 6ce09a9bb33e4011761558032e2165ad7b49fb68 2024-08-07T17:46:40.4180184Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:40.4181225Z DOCKER_TAG: 02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:40.4181793Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:40.4182285Z ##[endgroup] 2024-08-07T17:46:40.4204623Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:40.4205193Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:40.4207053Z + aws ecr get-login-password --region us-east-1 2024-08-07T17:46:40.4208084Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:40.8871538Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-07T17:46:40.8872273Z Configure a credential helper to remove this warning. See 2024-08-07T17:46:40.8873019Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-07T17:46:40.8873513Z 2024-08-07T17:46:40.8874103Z Login Succeeded 2024-08-07T17:46:40.8886623Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:41.1093037Z { 2024-08-07T17:46:41.1093399Z "schemaVersion": 2, 2024-08-07T17:46:41.1093920Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-07T17:46:41.1094462Z "config": { 2024-08-07T17:46:41.1094924Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-07T17:46:41.1095441Z "size": 47454, 2024-08-07T17:46:41.1095955Z "digest": "sha256:8e4e19ae0377c273bc603b24bc49d406d6f755072d4534d040c86afbe761a129" 2024-08-07T17:46:41.1096549Z }, 2024-08-07T17:46:41.1096826Z "layers": [ 2024-08-07T17:46:41.1097096Z { 2024-08-07T17:46:41.1097522Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1098699Z "size": 28580681, 2024-08-07T17:46:41.1099238Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-07T17:46:41.1099829Z }, 2024-08-07T17:46:41.1100098Z { 2024-08-07T17:46:41.1100503Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1101022Z "size": 7943353, 2024-08-07T17:46:41.1101546Z "digest": "sha256:09d415c238d76b32a7ea4a6e6add9542db9a5641f7f183af70aae185d0709e58" 2024-08-07T17:46:41.1102241Z }, 2024-08-07T17:46:41.1102621Z { 2024-08-07T17:46:41.1103242Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1103916Z "size": 56226389, 2024-08-07T17:46:41.1104493Z "digest": "sha256:9fe6e2e61518cba6844870c03b285737daec35e62baf25ae7744629ed3a7b470" 2024-08-07T17:46:41.1105175Z }, 2024-08-07T17:46:41.1105903Z { 2024-08-07T17:46:41.1106452Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1107053Z "size": 187, 2024-08-07T17:46:41.1107667Z "digest": "sha256:41f16248e682693ff20b3032c1d5e5541cc87c5af898ae2ff9b24d2940e59100" 2024-08-07T17:46:41.1108354Z }, 2024-08-07T17:46:41.1108675Z { 2024-08-07T17:46:41.1109185Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1109802Z "size": 6885, 2024-08-07T17:46:41.1110424Z "digest": "sha256:95d7b781703928cf3c4eece39d800cccb76728c375fedf51ecd83833fb25e458" 2024-08-07T17:46:41.1111079Z }, 2024-08-07T17:46:41.1111427Z { 2024-08-07T17:46:41.1111906Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1112690Z "size": 1383662475, 2024-08-07T17:46:41.1141688Z "digest": "sha256:8f6c9048534734f4c873935293b7296225846ceb31c1a158400a67ea170dde7f" 2024-08-07T17:46:41.1142307Z }, 2024-08-07T17:46:41.1142838Z + exit 0 2024-08-07T17:46:41.1143156Z { 2024-08-07T17:46:41.1143584Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1144138Z "size": 62282, 2024-08-07T17:46:41.1144639Z "digest": "sha256:ab17245097e491b9368790714f9d90ed447bf0973bd677cfe6f2456d62b72a13" 2024-08-07T17:46:41.1145255Z }, 2024-08-07T17:46:41.1145526Z { 2024-08-07T17:46:41.1145935Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1146466Z "size": 1684, 2024-08-07T17:46:41.1146993Z "digest": "sha256:dfecd7e9912b76ed460b8edd5a85f1943666e38a973ab5458177cf2c7c3110e3" 2024-08-07T17:46:41.1147632Z }, 2024-08-07T17:46:41.1147903Z { 2024-08-07T17:46:41.1148323Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1148833Z "size": 1522, 2024-08-07T17:46:41.1149338Z "digest": "sha256:464a8f74544589bf7b57f9a4cadcb6681e5ed00758f6c35025e691df4e88e890" 2024-08-07T17:46:41.1149912Z }, 2024-08-07T17:46:41.1150153Z { 2024-08-07T17:46:41.1150571Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1151093Z "size": 2445884909, 2024-08-07T17:46:41.1151604Z "digest": "sha256:d67c111fa58886d8904335b5122039816c8e51af5844cb6fe4e14154d9a7c016" 2024-08-07T17:46:41.1152182Z }, 2024-08-07T17:46:41.1152446Z { 2024-08-07T17:46:41.1152850Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1153375Z "size": 85882, 2024-08-07T17:46:41.1153891Z "digest": "sha256:0723abde19c09431db80b5d0f0e003541f6232c2da5c4f35c635c861875937ae" 2024-08-07T17:46:41.1154455Z }, 2024-08-07T17:46:41.1154716Z { 2024-08-07T17:46:41.1155136Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1155644Z "size": 1820, 2024-08-07T17:46:41.1156168Z "digest": "sha256:ceceb076946ee95300822daccb5d3e4e45e6857d34996ef89305d9a078410f8e" 2024-08-07T17:46:41.1156755Z }, 2024-08-07T17:46:41.1157003Z { 2024-08-07T17:46:41.1157424Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1157932Z "size": 654860881, 2024-08-07T17:46:41.1158661Z "digest": "sha256:889eb8142c5187639e02cdc07adc75024843fd85a1f5c77d24c1a1123f5246bb" 2024-08-07T17:46:41.1159241Z }, 2024-08-07T17:46:41.1159488Z { 2024-08-07T17:46:41.1159909Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1160435Z "size": 542, 2024-08-07T17:46:41.1160942Z "digest": "sha256:af8081143f008c5cb1bd61f6d92fde22553cfc007b6a77c2fd2aaf6a9f0088a9" 2024-08-07T17:46:41.1161534Z }, 2024-08-07T17:46:41.1161795Z { 2024-08-07T17:46:41.1162198Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1162722Z "size": 1284, 2024-08-07T17:46:41.1163249Z "digest": "sha256:cc64a79090ad4059e2f3dbc1949b010b2733ab614bc9518ff6256ce2bed52115" 2024-08-07T17:46:41.1163811Z }, 2024-08-07T17:46:41.1164077Z { 2024-08-07T17:46:41.1164500Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1165010Z "size": 483, 2024-08-07T17:46:41.1165517Z "digest": "sha256:ad26542e0d01609840496a12691483f2c50eb8e23002a205becb5b695f9159fb" 2024-08-07T17:46:41.1166105Z }, 2024-08-07T17:46:41.1166355Z { 2024-08-07T17:46:41.1166779Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1167310Z "size": 91711921, 2024-08-07T17:46:41.1167823Z "digest": "sha256:7b42d8ae23671f21f6523a7e71cc50d5f87c4e993027a9bd924985cda6b57de9" 2024-08-07T17:46:41.1168407Z }, 2024-08-07T17:46:41.1168672Z { 2024-08-07T17:46:41.1169078Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1169604Z "size": 3229, 2024-08-07T17:46:41.1170245Z "digest": "sha256:6d46e3b97d9fd2bd0e7ceec44e4884756b26c594a1c1c51d1426ddd13f749db5" 2024-08-07T17:46:41.1170841Z }, 2024-08-07T17:46:41.1171092Z { 2024-08-07T17:46:41.1171509Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1172018Z "size": 1907, 2024-08-07T17:46:41.1172535Z "digest": "sha256:91f9dddffbee46981d9e74382704c1b53c87f1808350cf1dd518687805d3aa59" 2024-08-07T17:46:41.1173127Z }, 2024-08-07T17:46:41.1173378Z { 2024-08-07T17:46:41.1173802Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1174328Z "size": 699, 2024-08-07T17:46:41.1174836Z "digest": "sha256:03fb78f207e3cbcfedfc7619a1d60fa576d3293d508a7e97dc09a5e7a9c65ab0" 2024-08-07T17:46:41.1175424Z }, 2024-08-07T17:46:41.1175668Z { 2024-08-07T17:46:41.1176089Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1176615Z "size": 2813413845, 2024-08-07T17:46:41.1177151Z "digest": "sha256:eeedca475f8cafec096dfa0abc5c6f08938eb47c5d9d96fb001fd7f13a2531ca" 2024-08-07T17:46:41.1177760Z }, 2024-08-07T17:46:41.1178028Z { 2024-08-07T17:46:41.1178737Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1179272Z "size": 378, 2024-08-07T17:46:41.1179796Z "digest": "sha256:202c71391ade27762d5e9cafde793db67b4d5e4d6869debe2643d5092101c461" 2024-08-07T17:46:41.1180373Z }, 2024-08-07T17:46:41.1180637Z { 2024-08-07T17:46:41.1181054Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1181563Z "size": 12893, 2024-08-07T17:46:41.1182085Z "digest": "sha256:5af882ef27a2db3ee5e89c8730de161c0718b951f3f8477e63fe21517c9167f7" 2024-08-07T17:46:41.1182676Z }, 2024-08-07T17:46:41.1182927Z { 2024-08-07T17:46:41.1183348Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1183872Z "size": 802, 2024-08-07T17:46:41.1184380Z "digest": "sha256:bac001ac2fee5df3367ff9332bc1fcd706a8de3ad05a1885903f338197619520" 2024-08-07T17:46:41.1184970Z }, 2024-08-07T17:46:41.1185237Z { 2024-08-07T17:46:41.1185649Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1186183Z "size": 106, 2024-08-07T17:46:41.1186700Z "digest": "sha256:05896f1d52f7a8e2c4a53777e5575da3dc8fca8a786d14075b5bb204bcb4cf04" 2024-08-07T17:46:41.1187273Z }, 2024-08-07T17:46:41.1187681Z { 2024-08-07T17:46:41.1188108Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1188618Z "size": 502, 2024-08-07T17:46:41.1189134Z "digest": "sha256:49a0e418a175ac12f1b4b503e80e22679bc321e6163ed5b203af848204f0606a" 2024-08-07T17:46:41.1189719Z }, 2024-08-07T17:46:41.1189972Z { 2024-08-07T17:46:41.1190393Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1190931Z "size": 121477364, 2024-08-07T17:46:41.1191441Z "digest": "sha256:b5632121ff7a146647683888294bcca31252dd90ca681099dfb81d75c4710d52" 2024-08-07T17:46:41.1192015Z }, 2024-08-07T17:46:41.1192282Z { 2024-08-07T17:46:41.1192692Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1193218Z "size": 109, 2024-08-07T17:46:41.1193710Z "digest": "sha256:deceb46806b535404e025b9b532009eb211e79f85021171997af825253327d1a" 2024-08-07T17:46:41.1194289Z }, 2024-08-07T17:46:41.1194556Z { 2024-08-07T17:46:41.1194969Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1195497Z "size": 487, 2024-08-07T17:46:41.1196015Z "digest": "sha256:a2630e5fe166ae8ff146e4c14db0bdc1299ed2b6120108351c961adcd2375288" 2024-08-07T17:46:41.1196595Z }, 2024-08-07T17:46:41.1196867Z { 2024-08-07T17:46:41.1197281Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1197780Z "size": 297, 2024-08-07T17:46:41.1198290Z "digest": "sha256:603261fb379fff6cf0eb63de1787e7c98932f81e44e6af3f1645c9a977e0ca8c" 2024-08-07T17:46:41.1198874Z }, 2024-08-07T17:46:41.1199119Z { 2024-08-07T17:46:41.1199654Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1200200Z "size": 103, 2024-08-07T17:46:41.1200692Z "digest": "sha256:d8ac71337d62da7c4624f267380da27d834fcf174b873c8c53a963e6cc7193a4" 2024-08-07T17:46:41.1201271Z }, 2024-08-07T17:46:41.1201535Z { 2024-08-07T17:46:41.1201939Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1202467Z "size": 1470, 2024-08-07T17:46:41.1202991Z "digest": "sha256:2c2e35b979f5fc0ecccef6efc157a97f33db01a77958fa33855dc367544a5c46" 2024-08-07T17:46:41.1203566Z }, 2024-08-07T17:46:41.1203831Z { 2024-08-07T17:46:41.1204249Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1204943Z "size": 424146101, 2024-08-07T17:46:41.1205480Z "digest": "sha256:e559222c0ec62d839f78e4215ae553de028be354b7031e7393d1c6be6e91fd70" 2024-08-07T17:46:41.1206058Z }, 2024-08-07T17:46:41.1206304Z { 2024-08-07T17:46:41.1206723Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1207251Z "size": 159, 2024-08-07T17:46:41.1207758Z "digest": "sha256:c4dcb7e9d189eca2e414c757b5cf6d158055b29b0e71dbd082a5fb3585dba665" 2024-08-07T17:46:41.1208348Z }, 2024-08-07T17:46:41.1208615Z { 2024-08-07T17:46:41.1209027Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1209561Z "size": 563, 2024-08-07T17:46:41.1210060Z "digest": "sha256:2b9731d295d60278e361d491fc274e5d558a89ec5adeb9fdf342d7a8cc17bfec" 2024-08-07T17:46:41.1210648Z }, 2024-08-07T17:46:41.1210914Z { 2024-08-07T17:46:41.1211318Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1211846Z "size": 35807525, 2024-08-07T17:46:41.1212368Z "digest": "sha256:19da43312ebac2831028913760327e53d9812e80ab6b806cd7a920c549a3b554" 2024-08-07T17:46:41.1212929Z }, 2024-08-07T17:46:41.1213200Z { 2024-08-07T17:46:41.1213624Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1214133Z "size": 104, 2024-08-07T17:46:41.1214651Z "digest": "sha256:b846263d066405189d21552b7288e3efbc47e6cd2f236aa39f39d88ca576a33a" 2024-08-07T17:46:41.1215234Z }, 2024-08-07T17:46:41.1215485Z { 2024-08-07T17:46:41.1215908Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1216434Z "size": 424, 2024-08-07T17:46:41.1217044Z "digest": "sha256:3886e8f57dc887a87bb710fbda080ef4fd521f1ed2e560a895b000c05f6e9b0a" 2024-08-07T17:46:41.1217634Z }, 2024-08-07T17:46:41.1217901Z { 2024-08-07T17:46:41.1218602Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1219146Z "size": 20262048, 2024-08-07T17:46:41.1219679Z "digest": "sha256:58e13e7b6d7b78400d84805c843f08fb8f1d6583bccdb3373c8122b317e4d15a" 2024-08-07T17:46:41.1220242Z }, 2024-08-07T17:46:41.1220504Z { 2024-08-07T17:46:41.1220926Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1221432Z "size": 438, 2024-08-07T17:46:41.1221958Z "digest": "sha256:0eb80a0ac0006a45aa6b4581263cc1613ddf9545bd4a1d5a544310efe898ad81" 2024-08-07T17:46:41.1222543Z }, 2024-08-07T17:46:41.1222789Z { 2024-08-07T17:46:41.1223206Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1223726Z "size": 699, 2024-08-07T17:46:41.1224229Z "digest": "sha256:03fb78f207e3cbcfedfc7619a1d60fa576d3293d508a7e97dc09a5e7a9c65ab0" 2024-08-07T17:46:41.1224825Z }, 2024-08-07T17:46:41.1225089Z { 2024-08-07T17:46:41.1225493Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1226014Z "size": 141, 2024-08-07T17:46:41.1226534Z "digest": "sha256:a9f11b2b3622f5530c5cfefe973e96e028323fab1ff02e7a6337eb86fadebeb5" 2024-08-07T17:46:41.1227105Z }, 2024-08-07T17:46:41.1227366Z { 2024-08-07T17:46:41.1227765Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1228289Z "size": 135, 2024-08-07T17:46:41.1228937Z "digest": "sha256:d7e94279f8b2fe549e3f23637275d9dca8c831563ce84f1d44117eb297ddc871" 2024-08-07T17:46:41.1229524Z }, 2024-08-07T17:46:41.1229791Z { 2024-08-07T17:46:41.1230209Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1230714Z "size": 32, 2024-08-07T17:46:41.1231245Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-07T17:46:41.1231846Z }, 2024-08-07T17:46:41.1232097Z { 2024-08-07T17:46:41.1232520Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1233045Z "size": 190, 2024-08-07T17:46:41.1233548Z "digest": "sha256:0a3560fcd26c387cb3ff0638718fa9da02d3b08ad75b23c434b06d6a6d21cb55" 2024-08-07T17:46:41.1234144Z }, 2024-08-07T17:46:41.1234411Z { 2024-08-07T17:46:41.1234814Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1235347Z "size": 562, 2024-08-07T17:46:41.1235869Z "digest": "sha256:adf9d181cfbca5b0544f23797995d4ea6500cc4a635db9f7e18954c9044e9f4a" 2024-08-07T17:46:41.1236445Z }, 2024-08-07T17:46:41.1236718Z { 2024-08-07T17:46:41.1237139Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1237649Z "size": 43163050, 2024-08-07T17:46:41.1238181Z "digest": "sha256:bc18dd8315979255e24cf91e83757510baeb22f29a64d7579b02aca642215b58" 2024-08-07T17:46:41.1238768Z }, 2024-08-07T17:46:41.1239015Z { 2024-08-07T17:46:41.1239437Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1239962Z "size": 106, 2024-08-07T17:46:41.1240473Z "digest": "sha256:d62f2a6086195c833ef952a43eed18216c6cde79b07f8deeb4fb3aeea648f868" 2024-08-07T17:46:41.1241066Z }, 2024-08-07T17:46:41.1241328Z { 2024-08-07T17:46:41.1241733Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1242305Z "size": 1210, 2024-08-07T17:46:41.1242824Z "digest": "sha256:719a1c2abcfd7e4113aa2f03adeb54350a5e381c720688ea53bc0e18401ad33d" 2024-08-07T17:46:41.1243397Z }, 2024-08-07T17:46:41.1243659Z { 2024-08-07T17:46:41.1244085Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1244598Z "size": 699, 2024-08-07T17:46:41.1245116Z "digest": "sha256:03fb78f207e3cbcfedfc7619a1d60fa576d3293d508a7e97dc09a5e7a9c65ab0" 2024-08-07T17:46:41.1245755Z }, 2024-08-07T17:46:41.1246005Z { 2024-08-07T17:46:41.1246543Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1247049Z "size": 138, 2024-08-07T17:46:41.1247556Z "digest": "sha256:8e1355d516c5e0527c25402c664fc648a8a2b814090e6ee0fdd60f49a8826ac1" 2024-08-07T17:46:41.1248134Z }, 2024-08-07T17:46:41.1248382Z { 2024-08-07T17:46:41.1248790Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1249314Z "size": 117, 2024-08-07T17:46:41.1249803Z "digest": "sha256:701898e66f54fcf3b57d837ab651a876666391d9f9b51b9969ec842a23fd86f0" 2024-08-07T17:46:41.1250382Z }, 2024-08-07T17:46:41.1250648Z { 2024-08-07T17:46:41.1251058Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1251584Z "size": 1916643487, 2024-08-07T17:46:41.1252119Z "digest": "sha256:129b38cfed12360bb7c4d882141b5fb7f7337791383b9f75e4ffed521548f060" 2024-08-07T17:46:41.1252683Z }, 2024-08-07T17:46:41.1252946Z { 2024-08-07T17:46:41.1253364Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1253886Z "size": 172, 2024-08-07T17:46:41.1254390Z "digest": "sha256:1751651a072f518b34289de9dfa631c4e82c7958a5331061379684f9b9b11720" 2024-08-07T17:46:41.1254960Z }, 2024-08-07T17:46:41.1255209Z { 2024-08-07T17:46:41.1255625Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1256154Z "size": 905, 2024-08-07T17:46:41.1256659Z "digest": "sha256:fc956bf4e5b49119cab7cc7bda867c61bed76c76c1e113dfa7c7b184251f8173" 2024-08-07T17:46:41.1257252Z }, 2024-08-07T17:46:41.1257515Z { 2024-08-07T17:46:41.1257915Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1258883Z "size": 699, 2024-08-07T17:46:41.1259447Z "digest": "sha256:03fb78f207e3cbcfedfc7619a1d60fa576d3293d508a7e97dc09a5e7a9c65ab0" 2024-08-07T17:46:41.1260023Z }, 2024-08-07T17:46:41.1260287Z { 2024-08-07T17:46:41.1260707Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1261226Z "size": 134, 2024-08-07T17:46:41.1261738Z "digest": "sha256:874491333fb5b81d9cbbd71520163a6baf07e6f011dc148b1634b94e301972b8" 2024-08-07T17:46:41.1262320Z }, 2024-08-07T17:46:41.1262570Z { 2024-08-07T17:46:41.1262989Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1263514Z "size": 32, 2024-08-07T17:46:41.1264022Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-07T17:46:41.1264609Z }, 2024-08-07T17:46:41.1264848Z { 2024-08-07T17:46:41.1265267Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1265788Z "size": 156, 2024-08-07T17:46:41.1266305Z "digest": "sha256:365b4a0eec21fb9dfc58baf7b43a229ae3f8b638b17e9b4db5ee1cfaecb82de4" 2024-08-07T17:46:41.1266904Z }, 2024-08-07T17:46:41.1267166Z { 2024-08-07T17:46:41.1267569Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1268092Z "size": 1839, 2024-08-07T17:46:41.1268616Z "digest": "sha256:c1d4cac78640fc7d364011da807a363d6b248a2c46ccf99a6e1488c4974c17e3" 2024-08-07T17:46:41.1269184Z }, 2024-08-07T17:46:41.1269446Z { 2024-08-07T17:46:41.1269871Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1270380Z "size": 7529788, 2024-08-07T17:46:41.1270906Z "digest": "sha256:29afe2181ca6174db390fd0d9e8c4b8675fb99bb961c93936d4080ae92c8af57" 2024-08-07T17:46:41.1271499Z }, 2024-08-07T17:46:41.1271748Z { 2024-08-07T17:46:41.1272171Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1272697Z "size": 161, 2024-08-07T17:46:41.1273193Z "digest": "sha256:190dccfa687dc1e698f27965d8065a183168d7c66361505f368765456ff9f4bc" 2024-08-07T17:46:41.1273773Z }, 2024-08-07T17:46:41.1274038Z { 2024-08-07T17:46:41.1274442Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1274969Z "size": 7942, 2024-08-07T17:46:41.1275610Z "digest": "sha256:b01bea4b6634f47c51ede2b4a95ceade4e61356ca29551168b493ae6d4492f44" 2024-08-07T17:46:41.1276189Z }, 2024-08-07T17:46:41.1276454Z { 2024-08-07T17:46:41.1276872Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1277382Z "size": 8066, 2024-08-07T17:46:41.1277907Z "digest": "sha256:f13a9cc5968861df73c96e4ad67eef7f58576eeba8cdbee3d39e42ed3d6fe96d" 2024-08-07T17:46:41.1278511Z }, 2024-08-07T17:46:41.1278762Z { 2024-08-07T17:46:41.1279188Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1279722Z "size": 302, 2024-08-07T17:46:41.1280227Z "digest": "sha256:96d6dcd01b9f6c5557ba405d028442d808dd329f4f93e75f2c3ba0554e703a52" 2024-08-07T17:46:41.1280815Z }, 2024-08-07T17:46:41.1281062Z { 2024-08-07T17:46:41.1281482Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1282012Z "size": 7629762, 2024-08-07T17:46:41.1282518Z "digest": "sha256:fe134554f1b8a0509798c1f49e6f973f9b8b7f954eb1fda146f35a37c9c71599" 2024-08-07T17:46:41.1283107Z }, 2024-08-07T17:46:41.1283374Z { 2024-08-07T17:46:41.1283773Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1284300Z "size": 108, 2024-08-07T17:46:41.1284816Z "digest": "sha256:8068d364963fe87857fbb7c3aad42ddee962351a57eddc01b4282be12c69a074" 2024-08-07T17:46:41.1285386Z }, 2024-08-07T17:46:41.1285648Z { 2024-08-07T17:46:41.1286072Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1286579Z "size": 54145777, 2024-08-07T17:46:41.1287106Z "digest": "sha256:f307f545c1a5e086f18e23247c890ddd5b135b1cb7157807aeeaef38ceb3b7dc" 2024-08-07T17:46:41.1287770Z }, 2024-08-07T17:46:41.1288024Z { 2024-08-07T17:46:41.1288446Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1288973Z "size": 471, 2024-08-07T17:46:41.1289463Z "digest": "sha256:51c0b7eab50584e37e08f303a4bf3ba3141998800971452b2024a8b897858877" 2024-08-07T17:46:41.1290045Z }, 2024-08-07T17:46:41.1290303Z { 2024-08-07T17:46:41.1290701Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1291219Z "size": 1373229322, 2024-08-07T17:46:41.1291741Z "digest": "sha256:60aba96f31c695ec1e1433578783bab1257233794a6e8b5cc5c6365fdf68669e" 2024-08-07T17:46:41.1292297Z }, 2024-08-07T17:46:41.1292558Z { 2024-08-07T17:46:41.1292971Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1293475Z "size": 106, 2024-08-07T17:46:41.1293987Z "digest": "sha256:46db894a106ccd6c28198b9236ee84ac2973b37ae608caeb380dfaef91f13961" 2024-08-07T17:46:41.1294566Z }, 2024-08-07T17:46:41.1294822Z { 2024-08-07T17:46:41.1295243Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1295761Z "size": 557, 2024-08-07T17:46:41.1296234Z "digest": "sha256:0052583728e40920c255902250fe4a615b10ff00a5c0e7ed51a0075121d42fad" 2024-08-07T17:46:41.1296797Z }, 2024-08-07T17:46:41.1297068Z { 2024-08-07T17:46:41.1297476Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1298000Z "size": 36532090, 2024-08-07T17:46:41.1298808Z "digest": "sha256:8c6d352a3b51edbb546310cc3d97ded6aa8fd29a7d610e4d13fc54b5828fdd05" 2024-08-07T17:46:41.1299393Z }, 2024-08-07T17:46:41.1299658Z { 2024-08-07T17:46:41.1300067Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1300593Z "size": 111, 2024-08-07T17:46:41.1301114Z "digest": "sha256:380c4e16278f4e3bf0eb6aee3ff4a730e3a1adb82aeb1cfb8e4114f1759a2bcf" 2024-08-07T17:46:41.1301691Z }, 2024-08-07T17:46:41.1301962Z { 2024-08-07T17:46:41.1302386Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1302895Z "size": 32, 2024-08-07T17:46:41.1303414Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-07T17:46:41.1304002Z }, 2024-08-07T17:46:41.1304253Z { 2024-08-07T17:46:41.1304807Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1305328Z "size": 32, 2024-08-07T17:46:41.1305828Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-07T17:46:41.1306417Z }, 2024-08-07T17:46:41.1306683Z { 2024-08-07T17:46:41.1307085Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1307608Z "size": 32, 2024-08-07T17:46:41.1308127Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-07T17:46:41.1308700Z }, 2024-08-07T17:46:41.1308968Z { 2024-08-07T17:46:41.1309394Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:41.1309898Z "size": 32, 2024-08-07T17:46:41.1310415Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-07T17:46:41.1311005Z } 2024-08-07T17:46:41.1311252Z ] 2024-08-07T17:46:41.1311514Z } 2024-08-07T17:46:41.1400566Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-07T17:46:41.1401066Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-07T17:46:41.1401585Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-07T17:46:41.1409499Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:46:41.1409961Z env: 2024-08-07T17:46:41.1410254Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:41.1411188Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:41.1412151Z ##[endgroup] 2024-08-07T17:46:41.1438191Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9-02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:41.1502367Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-07T17:46:41.1502948Z with: 2024-08-07T17:46:41.1503799Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:41.1504895Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:41.1505390Z env: 2024-08-07T17:46:41.1505662Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:41.1506011Z ##[endgroup] 2024-08-07T17:46:41.1531861Z ##[group]Run set -x 2024-08-07T17:46:41.1532236Z set -x 2024-08-07T17:46:41.1532550Z set +e 2024-08-07T17:46:41.1532836Z  2024-08-07T17:46:41.1533124Z login() { 2024-08-07T17:46:41.1533721Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-07T17:46:41.1534354Z } 2024-08-07T17:46:41.1534644Z  2024-08-07T17:46:41.1534978Z retry () { 2024-08-07T17:46:41.1535330Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-07T17:46:41.1535748Z } 2024-08-07T17:46:41.1536032Z  2024-08-07T17:46:41.1536331Z retry login "${DOCKER_REGISTRY}" 2024-08-07T17:46:41.1536743Z  2024-08-07T17:46:41.1537030Z set -e 2024-08-07T17:46:41.1537452Z # ignore output since only exit code is used for conditional 2024-08-07T17:46:41.1538068Z # only pull docker image if it's not available locally 2024-08-07T17:46:41.1539033Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-07T17:46:41.1539639Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-07T17:46:41.1540060Z fi 2024-08-07T17:46:41.1547121Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:46:41.1547589Z env: 2024-08-07T17:46:41.1547881Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:41.1548818Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:41.1549879Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:41.1550353Z ##[endgroup] 2024-08-07T17:46:41.1571948Z + set +e 2024-08-07T17:46:41.1572358Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:41.1572889Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:41.1575346Z + aws ecr get-login-password --region us-east-1 2024-08-07T17:46:41.1576396Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:41.6230177Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-07T17:46:41.6230919Z Configure a credential helper to remove this warning. See 2024-08-07T17:46:41.6231633Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-07T17:46:41.6232115Z 2024-08-07T17:46:41.6232309Z Login Succeeded 2024-08-07T17:46:41.6242115Z + set -e 2024-08-07T17:46:41.6243075Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:41.6413364Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:41.6414922Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:42.1178770Z 02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9 2024-08-07T17:46:42.1180372Z 7a2c55901189: Pulling fs layer 2024-08-07T17:46:42.1181514Z 09d415c238d7: Pulling fs layer 2024-08-07T17:46:42.1182192Z 9fe6e2e61518: Pulling fs layer 2024-08-07T17:46:42.1182870Z 41f16248e682: Pulling fs layer 2024-08-07T17:46:42.1183533Z 95d7b7817039: Pulling fs layer 2024-08-07T17:46:42.1184188Z 8f6c90485347: Pulling fs layer 2024-08-07T17:46:42.1184903Z ab17245097e4: Pulling fs layer 2024-08-07T17:46:42.1185522Z dfecd7e9912b: Pulling fs layer 2024-08-07T17:46:42.1185890Z 41f16248e682: Waiting 2024-08-07T17:46:42.1186273Z 464a8f745445: Pulling fs layer 2024-08-07T17:46:42.1186936Z d67c111fa588: Pulling fs layer 2024-08-07T17:46:42.1187457Z 95d7b7817039: Waiting 2024-08-07T17:46:42.1187795Z 0723abde19c0: Pulling fs layer 2024-08-07T17:46:42.1188149Z ceceb076946e: Pulling fs layer 2024-08-07T17:46:42.1188521Z ab17245097e4: Waiting 2024-08-07T17:46:42.1189101Z 889eb8142c51: Pulling fs layer 2024-08-07T17:46:42.1189459Z af8081143f00: Pulling fs layer 2024-08-07T17:46:42.1189818Z dfecd7e9912b: Waiting 2024-08-07T17:46:42.1190157Z cc64a79090ad: Pulling fs layer 2024-08-07T17:46:42.1190500Z d67c111fa588: Waiting 2024-08-07T17:46:42.1190821Z 889eb8142c51: Waiting 2024-08-07T17:46:42.1191137Z 8f6c90485347: Waiting 2024-08-07T17:46:42.1191462Z ad26542e0d01: Pulling fs layer 2024-08-07T17:46:42.1191836Z 7b42d8ae2367: Pulling fs layer 2024-08-07T17:46:42.1192356Z 464a8f745445: Waiting 2024-08-07T17:46:42.1192685Z 6d46e3b97d9f: Pulling fs layer 2024-08-07T17:46:42.1193192Z 91f9dddffbee: Pulling fs layer 2024-08-07T17:46:42.1193563Z 03fb78f207e3: Pulling fs layer 2024-08-07T17:46:42.1194040Z 7b42d8ae2367: Waiting 2024-08-07T17:46:42.1194384Z ceceb076946e: Waiting 2024-08-07T17:46:42.1194833Z eeedca475f8c: Pulling fs layer 2024-08-07T17:46:42.1195203Z 91f9dddffbee: Waiting 2024-08-07T17:46:42.1195542Z 202c71391ade: Pulling fs layer 2024-08-07T17:46:42.1195883Z eeedca475f8c: Waiting 2024-08-07T17:46:42.1196223Z 5af882ef27a2: Pulling fs layer 2024-08-07T17:46:42.1196597Z bac001ac2fee: Pulling fs layer 2024-08-07T17:46:42.1196956Z 05896f1d52f7: Pulling fs layer 2024-08-07T17:46:42.1197324Z 49a0e418a175: Pulling fs layer 2024-08-07T17:46:42.1197686Z 5af882ef27a2: Waiting 2024-08-07T17:46:42.1197991Z 05896f1d52f7: Waiting 2024-08-07T17:46:42.1198338Z b5632121ff7a: Pulling fs layer 2024-08-07T17:46:42.1198702Z af8081143f00: Waiting 2024-08-07T17:46:42.1199045Z deceb46806b5: Pulling fs layer 2024-08-07T17:46:42.1199417Z a2630e5fe166: Pulling fs layer 2024-08-07T17:46:42.1199787Z 603261fb379f: Pulling fs layer 2024-08-07T17:46:42.1200495Z a2630e5fe166: Waiting 2024-08-07T17:46:42.1200935Z d8ac71337d62: Pulling fs layer 2024-08-07T17:46:42.1201380Z ad26542e0d01: Waiting 2024-08-07T17:46:42.1201847Z 2c2e35b979f5: Pulling fs layer 2024-08-07T17:46:42.1202257Z e559222c0ec6: Pulling fs layer 2024-08-07T17:46:42.1202715Z c4dcb7e9d189: Pulling fs layer 2024-08-07T17:46:42.1203191Z 6d46e3b97d9f: Waiting 2024-08-07T17:46:42.1203651Z 2c2e35b979f5: Waiting 2024-08-07T17:46:42.1204045Z 2b9731d295d6: Pulling fs layer 2024-08-07T17:46:42.1204403Z 19da43312eba: Pulling fs layer 2024-08-07T17:46:42.1204776Z b846263d0664: Pulling fs layer 2024-08-07T17:46:42.1205145Z 3886e8f57dc8: Pulling fs layer 2024-08-07T17:46:42.1205489Z b5632121ff7a: Waiting 2024-08-07T17:46:42.1205823Z e559222c0ec6: Waiting 2024-08-07T17:46:42.1206145Z 2b9731d295d6: Waiting 2024-08-07T17:46:42.1206450Z c4dcb7e9d189: Waiting 2024-08-07T17:46:42.1206783Z 19da43312eba: Waiting 2024-08-07T17:46:42.1207120Z 58e13e7b6d7b: Pulling fs layer 2024-08-07T17:46:42.1207488Z 0eb80a0ac000: Pulling fs layer 2024-08-07T17:46:42.1207850Z 3886e8f57dc8: Waiting 2024-08-07T17:46:42.1208186Z a9f11b2b3622: Pulling fs layer 2024-08-07T17:46:42.1208531Z 0eb80a0ac000: Waiting 2024-08-07T17:46:42.1208862Z d7e94279f8b2: Pulling fs layer 2024-08-07T17:46:42.1209220Z 58e13e7b6d7b: Waiting 2024-08-07T17:46:42.1209522Z a9f11b2b3622: Waiting 2024-08-07T17:46:42.1209856Z 4f4fb700ef54: Pulling fs layer 2024-08-07T17:46:42.1210198Z d7e94279f8b2: Waiting 2024-08-07T17:46:42.1210519Z deceb46806b5: Waiting 2024-08-07T17:46:42.1210837Z 603261fb379f: Waiting 2024-08-07T17:46:42.1211156Z 0a3560fcd26c: Pulling fs layer 2024-08-07T17:46:42.1211675Z adf9d181cfbc: Pulling fs layer 2024-08-07T17:46:42.1212188Z bc18dd831597: Pulling fs layer 2024-08-07T17:46:42.1212670Z d62f2a608619: Pulling fs layer 2024-08-07T17:46:42.1213183Z adf9d181cfbc: Waiting 2024-08-07T17:46:42.1213511Z bc18dd831597: Waiting 2024-08-07T17:46:42.1213812Z 0a3560fcd26c: Waiting 2024-08-07T17:46:42.1214149Z 719a1c2abcfd: Pulling fs layer 2024-08-07T17:46:42.1214535Z 8e1355d516c5: Pulling fs layer 2024-08-07T17:46:42.1214884Z 701898e66f54: Pulling fs layer 2024-08-07T17:46:42.1215252Z 129b38cfed12: Pulling fs layer 2024-08-07T17:46:42.1215618Z 1751651a072f: Pulling fs layer 2024-08-07T17:46:42.1215973Z fc956bf4e5b4: Pulling fs layer 2024-08-07T17:46:42.1216326Z 1751651a072f: Waiting 2024-08-07T17:46:42.1216657Z 874491333fb5: Pulling fs layer 2024-08-07T17:46:42.1216993Z fc956bf4e5b4: Waiting 2024-08-07T17:46:42.1217328Z 365b4a0eec21: Pulling fs layer 2024-08-07T17:46:42.1217702Z c1d4cac78640: Pulling fs layer 2024-08-07T17:46:42.1218043Z 8e1355d516c5: Waiting 2024-08-07T17:46:42.1218803Z 29afe2181ca6: Pulling fs layer 2024-08-07T17:46:42.1219194Z 190dccfa687d: Pulling fs layer 2024-08-07T17:46:42.1219718Z b01bea4b6634: Pulling fs layer 2024-08-07T17:46:42.1220199Z f13a9cc59688: Pulling fs layer 2024-08-07T17:46:42.1220591Z b01bea4b6634: Waiting 2024-08-07T17:46:42.1221057Z 29afe2181ca6: Waiting 2024-08-07T17:46:42.1221453Z 96d6dcd01b9f: Pulling fs layer 2024-08-07T17:46:42.1222063Z fe134554f1b8: Pulling fs layer 2024-08-07T17:46:42.1222491Z 8068d364963f: Pulling fs layer 2024-08-07T17:46:42.1222850Z 190dccfa687d: Waiting 2024-08-07T17:46:42.1223171Z f307f545c1a5: Pulling fs layer 2024-08-07T17:46:42.1223532Z 8068d364963f: Waiting 2024-08-07T17:46:42.1223872Z 51c0b7eab505: Pulling fs layer 2024-08-07T17:46:42.1224229Z 60aba96f31c6: Pulling fs layer 2024-08-07T17:46:42.1224603Z 46db894a106c: Pulling fs layer 2024-08-07T17:46:42.1224972Z 0052583728e4: Pulling fs layer 2024-08-07T17:46:42.1225324Z 8c6d352a3b51: Pulling fs layer 2024-08-07T17:46:42.1225686Z 129b38cfed12: Waiting 2024-08-07T17:46:42.1226014Z 60aba96f31c6: Waiting 2024-08-07T17:46:42.1226334Z 380c4e16278f: Pulling fs layer 2024-08-07T17:46:42.1226704Z 0052583728e4: Waiting 2024-08-07T17:46:42.1227024Z 46db894a106c: Waiting 2024-08-07T17:46:42.1227327Z c1d4cac78640: Waiting 2024-08-07T17:46:42.1227646Z 51c0b7eab505: Waiting 2024-08-07T17:46:42.1227948Z 719a1c2abcfd: Waiting 2024-08-07T17:46:42.1228468Z 380c4e16278f: Waiting 2024-08-07T17:46:42.1228786Z 701898e66f54: Waiting 2024-08-07T17:46:42.2891708Z 09d415c238d7: Verifying Checksum 2024-08-07T17:46:42.2892554Z 09d415c238d7: Download complete 2024-08-07T17:46:42.3650439Z 41f16248e682: Verifying Checksum 2024-08-07T17:46:42.3651016Z 41f16248e682: Download complete 2024-08-07T17:46:42.4502400Z 7a2c55901189: Verifying Checksum 2024-08-07T17:46:42.4503188Z 7a2c55901189: Download complete 2024-08-07T17:46:42.4538663Z 95d7b7817039: Verifying Checksum 2024-08-07T17:46:42.4539445Z 95d7b7817039: Download complete 2024-08-07T17:46:42.5301243Z ab17245097e4: Download complete 2024-08-07T17:46:42.6004981Z dfecd7e9912b: Verifying Checksum 2024-08-07T17:46:42.6005442Z dfecd7e9912b: Download complete 2024-08-07T17:46:42.6827267Z 464a8f745445: Verifying Checksum 2024-08-07T17:46:42.6827979Z 464a8f745445: Download complete 2024-08-07T17:46:42.7353126Z 9fe6e2e61518: Verifying Checksum 2024-08-07T17:46:42.7353548Z 9fe6e2e61518: Download complete 2024-08-07T17:46:42.8236825Z 0723abde19c0: Verifying Checksum 2024-08-07T17:46:42.8237282Z 0723abde19c0: Download complete 2024-08-07T17:46:42.9050410Z ceceb076946e: Verifying Checksum 2024-08-07T17:46:42.9050954Z ceceb076946e: Download complete 2024-08-07T17:46:43.7271717Z 7a2c55901189: Pull complete 2024-08-07T17:46:44.0680671Z 09d415c238d7: Pull complete 2024-08-07T17:46:45.2163408Z 9fe6e2e61518: Pull complete 2024-08-07T17:46:45.2627108Z 41f16248e682: Pull complete 2024-08-07T17:46:45.3325107Z 95d7b7817039: Pull complete 2024-08-07T17:46:49.5174530Z 889eb8142c51: Verifying Checksum 2024-08-07T17:46:49.5174996Z 889eb8142c51: Download complete 2024-08-07T17:46:49.5911745Z af8081143f00: Verifying Checksum 2024-08-07T17:46:49.5912161Z af8081143f00: Download complete 2024-08-07T17:46:49.6674190Z cc64a79090ad: Verifying Checksum 2024-08-07T17:46:49.6674634Z cc64a79090ad: Download complete 2024-08-07T17:46:49.7456964Z ad26542e0d01: Verifying Checksum 2024-08-07T17:46:49.7457357Z ad26542e0d01: Download complete 2024-08-07T17:46:50.7225651Z 7b42d8ae2367: Verifying Checksum 2024-08-07T17:46:50.7226139Z 7b42d8ae2367: Download complete 2024-08-07T17:46:50.8111047Z 6d46e3b97d9f: Verifying Checksum 2024-08-07T17:46:50.8111490Z 6d46e3b97d9f: Download complete 2024-08-07T17:46:50.8761932Z 91f9dddffbee: Download complete 2024-08-07T17:46:50.9497823Z 03fb78f207e3: Verifying Checksum 2024-08-07T17:46:50.9498522Z 03fb78f207e3: Download complete 2024-08-07T17:46:56.3405393Z 8f6c90485347: Verifying Checksum 2024-08-07T17:46:56.3405875Z 8f6c90485347: Download complete 2024-08-07T17:46:56.4190041Z 202c71391ade: Verifying Checksum 2024-08-07T17:46:56.4190490Z 202c71391ade: Download complete 2024-08-07T17:46:56.4975667Z 5af882ef27a2: Verifying Checksum 2024-08-07T17:46:56.4976176Z 5af882ef27a2: Download complete 2024-08-07T17:46:56.5814773Z bac001ac2fee: Verifying Checksum 2024-08-07T17:46:56.5815235Z bac001ac2fee: Download complete 2024-08-07T17:46:56.6679388Z 05896f1d52f7: Verifying Checksum 2024-08-07T17:46:56.6679845Z 05896f1d52f7: Download complete 2024-08-07T17:46:56.7516611Z 49a0e418a175: Verifying Checksum 2024-08-07T17:46:56.7517032Z 49a0e418a175: Download complete 2024-08-07T17:46:58.0318829Z b5632121ff7a: Verifying Checksum 2024-08-07T17:46:58.0319282Z b5632121ff7a: Download complete 2024-08-07T17:46:58.1164840Z deceb46806b5: Verifying Checksum 2024-08-07T17:46:58.1165310Z deceb46806b5: Download complete 2024-08-07T17:46:58.1956528Z a2630e5fe166: Verifying Checksum 2024-08-07T17:46:58.1956952Z a2630e5fe166: Download complete 2024-08-07T17:46:58.2747767Z 603261fb379f: Verifying Checksum 2024-08-07T17:46:58.2748618Z 603261fb379f: Download complete 2024-08-07T17:46:58.3657251Z d8ac71337d62: Verifying Checksum 2024-08-07T17:46:58.3657709Z d8ac71337d62: Download complete 2024-08-07T17:46:58.4282079Z 2c2e35b979f5: Verifying Checksum 2024-08-07T17:46:58.4282507Z 2c2e35b979f5: Download complete 2024-08-07T17:47:02.7809257Z e559222c0ec6: Verifying Checksum 2024-08-07T17:47:02.7809744Z e559222c0ec6: Download complete 2024-08-07T17:47:02.8552064Z c4dcb7e9d189: Verifying Checksum 2024-08-07T17:47:02.8552845Z c4dcb7e9d189: Download complete 2024-08-07T17:47:02.9312706Z 2b9731d295d6: Download complete 2024-08-07T17:47:03.3413187Z 19da43312eba: Verifying Checksum 2024-08-07T17:47:03.3413883Z 19da43312eba: Download complete 2024-08-07T17:47:03.5034622Z b846263d0664: Verifying Checksum 2024-08-07T17:47:03.5035256Z b846263d0664: Download complete 2024-08-07T17:47:03.5912901Z 3886e8f57dc8: Verifying Checksum 2024-08-07T17:47:03.5913327Z 3886e8f57dc8: Download complete 2024-08-07T17:47:03.8464241Z 58e13e7b6d7b: Verifying Checksum 2024-08-07T17:47:03.8464799Z 58e13e7b6d7b: Download complete 2024-08-07T17:47:03.9250842Z 0eb80a0ac000: Download complete 2024-08-07T17:47:04.0024111Z a9f11b2b3622: Download complete 2024-08-07T17:47:04.0892442Z d7e94279f8b2: Download complete 2024-08-07T17:47:04.0962598Z 4f4fb700ef54: Verifying Checksum 2024-08-07T17:47:04.0963334Z 4f4fb700ef54: Download complete 2024-08-07T17:47:04.1704801Z 0a3560fcd26c: Verifying Checksum 2024-08-07T17:47:04.1705283Z 0a3560fcd26c: Download complete 2024-08-07T17:47:04.2438754Z adf9d181cfbc: Verifying Checksum 2024-08-07T17:47:04.2439180Z adf9d181cfbc: Download complete 2024-08-07T17:47:04.7842000Z bc18dd831597: Verifying Checksum 2024-08-07T17:47:04.7842536Z bc18dd831597: Download complete 2024-08-07T17:47:04.8761626Z d62f2a608619: Verifying Checksum 2024-08-07T17:47:04.8762095Z d62f2a608619: Download complete 2024-08-07T17:47:04.9606210Z 719a1c2abcfd: Verifying Checksum 2024-08-07T17:47:04.9606716Z 719a1c2abcfd: Download complete 2024-08-07T17:47:05.0556204Z 8e1355d516c5: Verifying Checksum 2024-08-07T17:47:05.0556693Z 8e1355d516c5: Download complete 2024-08-07T17:47:05.1351182Z 701898e66f54: Download complete 2024-08-07T17:47:07.1953694Z d67c111fa588: Verifying Checksum 2024-08-07T17:47:07.1954211Z d67c111fa588: Download complete 2024-08-07T17:47:07.2662089Z 1751651a072f: Download complete 2024-08-07T17:47:07.3689576Z fc956bf4e5b4: Verifying Checksum 2024-08-07T17:47:07.3690071Z fc956bf4e5b4: Download complete 2024-08-07T17:47:07.4501923Z 874491333fb5: Verifying Checksum 2024-08-07T17:47:07.4502378Z 874491333fb5: Download complete 2024-08-07T17:47:07.5307600Z 365b4a0eec21: Verifying Checksum 2024-08-07T17:47:07.5308088Z 365b4a0eec21: Download complete 2024-08-07T17:47:07.6084490Z c1d4cac78640: Verifying Checksum 2024-08-07T17:47:07.6085016Z c1d4cac78640: Download complete 2024-08-07T17:47:07.7441671Z 29afe2181ca6: Verifying Checksum 2024-08-07T17:47:07.7442170Z 29afe2181ca6: Download complete 2024-08-07T17:47:07.8226178Z 190dccfa687d: Verifying Checksum 2024-08-07T17:47:07.8226611Z 190dccfa687d: Download complete 2024-08-07T17:47:07.9053806Z b01bea4b6634: Download complete 2024-08-07T17:47:07.9718557Z f13a9cc59688: Verifying Checksum 2024-08-07T17:47:07.9719033Z f13a9cc59688: Download complete 2024-08-07T17:47:08.0646318Z 96d6dcd01b9f: Verifying Checksum 2024-08-07T17:47:08.0646889Z 96d6dcd01b9f: Download complete 2024-08-07T17:47:08.1929854Z fe134554f1b8: Verifying Checksum 2024-08-07T17:47:08.1930346Z fe134554f1b8: Download complete 2024-08-07T17:47:08.2647891Z 8068d364963f: Verifying Checksum 2024-08-07T17:47:08.2650128Z 8068d364963f: Download complete 2024-08-07T17:47:08.8888629Z f307f545c1a5: Verifying Checksum 2024-08-07T17:47:08.8889124Z f307f545c1a5: Download complete 2024-08-07T17:47:08.9582847Z 51c0b7eab505: Download complete 2024-08-07T17:47:16.3410615Z 8f6c90485347: Pull complete 2024-08-07T17:47:19.0015794Z ab17245097e4: Pull complete 2024-08-07T17:47:19.1571124Z eeedca475f8c: Verifying Checksum 2024-08-07T17:47:19.1571591Z eeedca475f8c: Download complete 2024-08-07T17:47:19.2594194Z 46db894a106c: Verifying Checksum 2024-08-07T17:47:19.2594650Z 46db894a106c: Download complete 2024-08-07T17:47:19.3408159Z 0052583728e4: Verifying Checksum 2024-08-07T17:47:19.3408625Z 0052583728e4: Download complete 2024-08-07T17:47:19.8511000Z 8c6d352a3b51: Verifying Checksum 2024-08-07T17:47:19.8511469Z 8c6d352a3b51: Download complete 2024-08-07T17:47:19.9243748Z 380c4e16278f: Download complete 2024-08-07T17:47:21.4325530Z dfecd7e9912b: Pull complete 2024-08-07T17:47:22.7456444Z 60aba96f31c6: Verifying Checksum 2024-08-07T17:47:22.7456928Z 60aba96f31c6: Download complete 2024-08-07T17:47:23.8427257Z 464a8f745445: Pull complete 2024-08-07T17:47:24.4088508Z 129b38cfed12: Verifying Checksum 2024-08-07T17:47:24.4088966Z 129b38cfed12: Download complete 2024-08-07T17:48:02.1741467Z d67c111fa588: Pull complete 2024-08-07T17:48:04.7026390Z 0723abde19c0: Pull complete 2024-08-07T17:48:07.2804350Z ceceb076946e: Pull complete 2024-08-07T17:48:23.7988542Z 889eb8142c51: Pull complete 2024-08-07T17:48:26.2723296Z af8081143f00: Pull complete 2024-08-07T17:48:28.6825832Z cc64a79090ad: Pull complete 2024-08-07T17:48:31.1253261Z ad26542e0d01: Pull complete 2024-08-07T17:48:36.4680179Z 7b42d8ae2367: Pull complete 2024-08-07T17:48:38.9083451Z 6d46e3b97d9f: Pull complete 2024-08-07T17:48:41.5627879Z 91f9dddffbee: Pull complete 2024-08-07T17:48:43.7412854Z 03fb78f207e3: Pull complete 2024-08-07T17:49:46.0953544Z eeedca475f8c: Pull complete 2024-08-07T17:49:46.1449895Z 202c71391ade: Pull complete 2024-08-07T17:49:46.1985727Z 5af882ef27a2: Pull complete 2024-08-07T17:49:46.2493842Z bac001ac2fee: Pull complete 2024-08-07T17:49:46.2895342Z 05896f1d52f7: Pull complete 2024-08-07T17:49:46.3375709Z 49a0e418a175: Pull complete 2024-08-07T17:49:50.3619714Z b5632121ff7a: Pull complete 2024-08-07T17:49:50.4087281Z deceb46806b5: Pull complete 2024-08-07T17:49:50.4599893Z a2630e5fe166: Pull complete 2024-08-07T17:49:50.4968386Z 603261fb379f: Pull complete 2024-08-07T17:49:50.5478619Z d8ac71337d62: Pull complete 2024-08-07T17:49:50.5923415Z 2c2e35b979f5: Pull complete 2024-08-07T17:50:03.0988484Z e559222c0ec6: Pull complete 2024-08-07T17:50:05.5998266Z c4dcb7e9d189: Pull complete 2024-08-07T17:50:08.5232901Z 2b9731d295d6: Pull complete 2024-08-07T17:50:11.9369560Z 19da43312eba: Pull complete 2024-08-07T17:50:15.3461667Z b846263d0664: Pull complete 2024-08-07T17:50:17.7350616Z 3886e8f57dc8: Pull complete 2024-08-07T17:50:20.8833225Z 58e13e7b6d7b: Pull complete 2024-08-07T17:50:23.6012902Z 0eb80a0ac000: Pull complete 2024-08-07T17:50:27.2001843Z a9f11b2b3622: Pull complete 2024-08-07T17:50:27.2556170Z d7e94279f8b2: Pull complete 2024-08-07T17:50:27.2991580Z 4f4fb700ef54: Pull complete 2024-08-07T17:50:27.3521958Z 0a3560fcd26c: Pull complete 2024-08-07T17:50:27.3985258Z adf9d181cfbc: Pull complete 2024-08-07T17:50:29.7797013Z bc18dd831597: Pull complete 2024-08-07T17:50:29.8292159Z d62f2a608619: Pull complete 2024-08-07T17:50:29.8945941Z 719a1c2abcfd: Pull complete 2024-08-07T17:50:30.0210221Z 8e1355d516c5: Pull complete 2024-08-07T17:50:30.0649888Z 701898e66f54: Pull complete 2024-08-07T17:51:14.1875664Z 129b38cfed12: Pull complete 2024-08-07T17:51:16.5953670Z 1751651a072f: Pull complete 2024-08-07T17:51:18.7872015Z fc956bf4e5b4: Pull complete 2024-08-07T17:51:23.6617403Z 874491333fb5: Pull complete 2024-08-07T17:51:28.5382793Z 365b4a0eec21: Pull complete 2024-08-07T17:51:30.9489131Z c1d4cac78640: Pull complete 2024-08-07T17:51:33.6084252Z 29afe2181ca6: Pull complete 2024-08-07T17:51:36.7928405Z 190dccfa687d: Pull complete 2024-08-07T17:51:39.3440105Z b01bea4b6634: Pull complete 2024-08-07T17:51:42.0298104Z f13a9cc59688: Pull complete 2024-08-07T17:51:44.6440439Z 96d6dcd01b9f: Pull complete 2024-08-07T17:51:46.8991952Z fe134554f1b8: Pull complete 2024-08-07T17:51:46.9408318Z 8068d364963f: Pull complete 2024-08-07T17:51:49.1745543Z f307f545c1a5: Pull complete 2024-08-07T17:51:49.2303425Z 51c0b7eab505: Pull complete 2024-08-07T17:52:05.8176227Z 60aba96f31c6: Pull complete 2024-08-07T17:52:05.8644798Z 46db894a106c: Pull complete 2024-08-07T17:52:05.9406269Z 0052583728e4: Pull complete 2024-08-07T17:52:06.6451401Z 8c6d352a3b51: Pull complete 2024-08-07T17:52:06.6896521Z 380c4e16278f: Pull complete 2024-08-07T17:52:06.9760086Z Digest: sha256:66181a134b3ebc35388dabcc46dc11e767152fd72514dad76711590f50b650f7 2024-08-07T17:52:06.9798545Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:52:06.9836282Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:52:06.9885728Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-07T17:52:06.9886604Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-07T17:52:06.9895586Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:52:06.9896065Z env: 2024-08-07T17:52:06.9896344Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:52:06.9896694Z ##[endgroup] 2024-08-07T17:52:07.0163641Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-07T17:52:07.0164194Z with: 2024-08-07T17:52:07.0164500Z driver-version: 550.54.15 2024-08-07T17:52:07.0164852Z env: 2024-08-07T17:52:07.0165129Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:52:07.0165475Z ##[endgroup] 2024-08-07T17:52:07.0215571Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-07T17:52:07.0216125Z with: 2024-08-07T17:52:07.0216432Z timeout_minutes: 10 2024-08-07T17:52:07.0216752Z max_attempts: 3 2024-08-07T17:52:07.0246860Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-07T17:52:07.0277296Z retry_wait_seconds: 10 2024-08-07T17:52:07.0277679Z polling_interval_seconds: 1 2024-08-07T17:52:07.0278040Z warning_on_retry: true 2024-08-07T17:52:07.0278400Z continue_on_error: false 2024-08-07T17:52:07.0278752Z env: 2024-08-07T17:52:07.0279029Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:52:07.0279390Z DRIVER_VERSION: 550.54.15 2024-08-07T17:52:07.0279825Z ##[endgroup] 2024-08-07T17:52:07.1032416Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-07T17:52:07.1034483Z + pre_install_nvidia_driver_amzn2 2024-08-07T17:52:07.1034952Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-07T17:52:07.6097824Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-08-07T17:52:07.6689678Z No Match for argument: nvidia-driver-latest-dkms 2024-08-07T17:52:07.7054267Z No Packages marked for removal 2024-08-07T17:52:07.7209277Z + install_nvidia_driver_common 2024-08-07T17:52:07.7212547Z + echo 'Before installing NVIDIA driver' 2024-08-07T17:52:07.7213161Z + lspci 2024-08-07T17:52:07.7215424Z Before installing NVIDIA driver 2024-08-07T17:52:07.7411305Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-08-07T17:52:07.7412004Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-07T17:52:07.7412696Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-08-07T17:52:07.7413618Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-08-07T17:52:07.7414189Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-08-07T17:52:07.7414793Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-07T17:52:07.7415528Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-07T17:52:07.7416285Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-07T17:52:07.7416997Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-08-07T17:52:07.7417687Z + lsmod 2024-08-07T17:52:07.7435512Z Module Size Used by 2024-08-07T17:52:07.7435940Z ib_core 266240 0 2024-08-07T17:52:07.7436306Z veth 16384 0 2024-08-07T17:52:07.7436669Z nvidia_modeset 1339392 0 2024-08-07T17:52:07.7437018Z nvidia_uvm 4599808 0 2024-08-07T17:52:07.7437458Z nvidia 53989376 11 nvidia_uvm,nvidia_modeset 2024-08-07T17:52:07.7437925Z drm 421888 1 nvidia 2024-08-07T17:52:07.7438319Z i2c_core 77824 2 nvidia,drm 2024-08-07T17:52:07.7438761Z backlight 16384 1 nvidia_modeset 2024-08-07T17:52:07.7439198Z xt_conntrack 16384 1 2024-08-07T17:52:07.7439555Z ipt_MASQUERADE 16384 1 2024-08-07T17:52:07.7439970Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-08-07T17:52:07.7440422Z nf_conntrack_netlink 49152 0 2024-08-07T17:52:07.7440834Z nfnetlink 16384 2 nf_conntrack_netlink 2024-08-07T17:52:07.7441289Z xfrm_user 45056 1 2024-08-07T17:52:07.7441668Z xfrm_algo 16384 1 xfrm_user 2024-08-07T17:52:07.7442059Z iptable_nat 16384 1 2024-08-07T17:52:07.7442428Z nf_conntrack_ipv4 16384 3 2024-08-07T17:52:07.7442849Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-08-07T17:52:07.7443291Z nf_nat_ipv4 16384 1 iptable_nat 2024-08-07T17:52:07.7443784Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-08-07T17:52:07.7444737Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-08-07T17:52:07.7445519Z xt_addrtype 16384 2 2024-08-07T17:52:07.7445893Z iptable_filter 16384 1 2024-08-07T17:52:07.7446266Z br_netfilter 24576 0 2024-08-07T17:52:07.7446640Z bridge 172032 1 br_netfilter 2024-08-07T17:52:07.7447065Z stp 16384 1 bridge 2024-08-07T17:52:07.7447477Z llc 16384 2 bridge,stp 2024-08-07T17:52:07.7447864Z overlay 86016 0 2024-08-07T17:52:07.7448227Z sunrpc 393216 1 2024-08-07T17:52:07.7448592Z dm_mirror 28672 0 2024-08-07T17:52:07.7448952Z dm_region_hash 20480 1 dm_mirror 2024-08-07T17:52:07.7449407Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-08-07T17:52:07.7449880Z dm_mod 143360 2 dm_log,dm_mirror 2024-08-07T17:52:07.7450281Z dax 69632 1 dm_mod 2024-08-07T17:52:07.7450666Z sb_edac 24576 0 2024-08-07T17:52:07.7451045Z crc32_pclmul 16384 0 2024-08-07T17:52:07.7451427Z ghash_clmulni_intel 16384 0 2024-08-07T17:52:07.7451782Z pcbc 16384 0 2024-08-07T17:52:07.7452144Z aesni_intel 188416 0 2024-08-07T17:52:07.7452521Z aes_x86_64 20480 1 aesni_intel 2024-08-07T17:52:07.7452931Z crypto_simd 16384 1 aesni_intel 2024-08-07T17:52:07.7453362Z glue_helper 16384 1 aesni_intel 2024-08-07T17:52:07.7453885Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-08-07T17:52:07.7454385Z mousedev 24576 0 2024-08-07T17:52:07.7454749Z evdev 20480 3 2024-08-07T17:52:07.7455110Z psmouse 32768 0 2024-08-07T17:52:07.7455459Z pcc_cpufreq 16384 0 2024-08-07T17:52:07.7455996Z button 16384 0 2024-08-07T17:52:07.7456350Z ena 139264 0 2024-08-07T17:52:07.7456697Z ptp 20480 1 ena 2024-08-07T17:52:07.7457086Z pps_core 20480 1 ptp 2024-08-07T17:52:07.7457453Z ata_piix 36864 0 2024-08-07T17:52:07.7457821Z xen_blkfront 49152 2 2024-08-07T17:52:07.7458198Z libata 266240 1 ata_piix 2024-08-07T17:52:07.7458950Z crc32c_intel 24576 0 2024-08-07T17:52:07.7459324Z scsi_mod 245760 1 libata 2024-08-07T17:52:07.7459715Z autofs4 49152 2 2024-08-07T17:52:07.7460056Z + modinfo nvidia 2024-08-07T17:52:07.7460730Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-08-07T17:52:07.7461397Z alias: char-major-195-* 2024-08-07T17:52:07.7461841Z version: 550.54.15 2024-08-07T17:52:07.7462175Z supported: external 2024-08-07T17:52:07.7462516Z license: NVIDIA 2024-08-07T17:52:07.7462897Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-07T17:52:07.7463340Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-07T17:52:07.7463779Z srcversion: 833721318DA517F0C2FEC97 2024-08-07T17:52:07.7464218Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-07T17:52:07.7464669Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-07T17:52:07.7465132Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-07T17:52:07.7465565Z depends: i2c-core,drm 2024-08-07T17:52:07.7465929Z retpoline: Y 2024-08-07T17:52:07.7466451Z name: nvidia 2024-08-07T17:52:07.7466899Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-08-07T17:52:07.7467499Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-07T17:52:07.7468202Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-07T17:52:07.7468772Z parm: NVreg_ResmanDebugLevel:int 2024-08-07T17:52:07.7469191Z parm: NVreg_RmLogonRC:int 2024-08-07T17:52:07.7469621Z parm: NVreg_ModifyDeviceFiles:int 2024-08-07T17:52:07.7470063Z parm: NVreg_DeviceFileUID:int 2024-08-07T17:52:07.7470468Z parm: NVreg_DeviceFileGID:int 2024-08-07T17:52:07.7470896Z parm: NVreg_DeviceFileMode:int 2024-08-07T17:52:07.7471376Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-07T17:52:07.7471906Z parm: NVreg_UsePageAttributeTable:int 2024-08-07T17:52:07.7472366Z parm: NVreg_EnablePCIeGen3:int 2024-08-07T17:52:07.7472767Z parm: NVreg_EnableMSI:int 2024-08-07T17:52:07.7473174Z parm: NVreg_TCEBypassMode:int 2024-08-07T17:52:07.7473626Z parm: NVreg_EnableStreamMemOPs:int 2024-08-07T17:52:07.7474110Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-07T17:52:07.7474652Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-07T17:52:07.7475173Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-07T17:52:07.7475715Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-07T17:52:07.7476279Z parm: NVreg_DynamicPowerManagement:int 2024-08-07T17:52:07.7476851Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-07T17:52:07.7477416Z parm: NVreg_EnableGpuFirmware:int 2024-08-07T17:52:07.7477871Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-07T17:52:07.7478377Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-07T17:52:07.7478890Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-07T17:52:07.7479349Z parm: NVreg_MemoryPoolSize:int 2024-08-07T17:52:07.7479804Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-07T17:52:07.7480271Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-07T17:52:07.7480708Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-07T17:52:07.7481145Z parm: NVreg_NvLinkDisable:int 2024-08-07T17:52:07.7481604Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-07T17:52:07.7482108Z parm: NVreg_RegisterPCIDriver:int 2024-08-07T17:52:07.7482712Z parm: NVreg_EnableResizableBar:int 2024-08-07T17:52:07.7483163Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-07T17:52:07.7483642Z parm: NVreg_EnableNonblockingOpen:int 2024-08-07T17:52:07.7484110Z parm: NVreg_RegistryDwords:charp 2024-08-07T17:52:07.7484563Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-07T17:52:07.7485020Z parm: NVreg_RmMsg:charp 2024-08-07T17:52:07.7485423Z parm: NVreg_GpuBlacklist:charp 2024-08-07T17:52:07.7485855Z parm: NVreg_TemporaryFilePath:charp 2024-08-07T17:52:07.7486394Z parm: NVreg_ExcludedGpus:charp 2024-08-07T17:52:07.7486868Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-07T17:52:07.7487309Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-07T17:52:07.7487767Z parm: NVreg_ImexChannelCount:int 2024-08-07T17:52:07.7488204Z parm: rm_firmware_active:charp 2024-08-07T17:52:07.7488592Z + HAS_NVIDIA_DRIVER=0 2024-08-07T17:52:07.7488946Z ++ command -v nvidia-smi 2024-08-07T17:52:07.7489308Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-07T17:52:07.7489657Z + set +e 2024-08-07T17:52:07.7490086Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-07T17:52:07.7720020Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-07T17:52:07.7720431Z + NVIDIA_SMI_STATUS=0 2024-08-07T17:52:07.7720772Z + '[' 0 -ne 0 ']' 2024-08-07T17:52:07.7721086Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-07T17:52:07.7721436Z + HAS_NVIDIA_DRIVER=1 2024-08-07T17:52:07.7722016Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-07T17:52:07.7722665Z + set -e 2024-08-07T17:52:07.7722940Z + '[' 1 -eq 0 ']' 2024-08-07T17:52:07.7723466Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-07T17:52:07.7724100Z + post_install_nvidia_driver_common 2024-08-07T17:52:07.7726554Z + sudo modprobe nvidia 2024-08-07T17:52:07.7844845Z + echo 'After installing NVIDIA driver' 2024-08-07T17:52:07.7845279Z + lspci 2024-08-07T17:52:07.7846123Z After installing NVIDIA driver 2024-08-07T17:52:07.8056763Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-08-07T17:52:07.8057478Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-07T17:52:07.8058171Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-08-07T17:52:07.8059109Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-08-07T17:52:07.8059696Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-08-07T17:52:07.8060328Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-07T17:52:07.8061085Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-07T17:52:07.8061852Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-07T17:52:07.8062583Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-08-07T17:52:07.8063121Z + lsmod 2024-08-07T17:52:07.8078693Z Module Size Used by 2024-08-07T17:52:07.8079115Z ib_core 266240 0 2024-08-07T17:52:07.8079481Z veth 16384 0 2024-08-07T17:52:07.8079835Z nvidia_modeset 1339392 0 2024-08-07T17:52:07.8080203Z nvidia_uvm 4599808 0 2024-08-07T17:52:07.8080629Z nvidia 53989376 11 nvidia_uvm,nvidia_modeset 2024-08-07T17:52:07.8081073Z drm 421888 1 nvidia 2024-08-07T17:52:07.8081491Z i2c_core 77824 2 nvidia,drm 2024-08-07T17:52:07.8081945Z backlight 16384 1 nvidia_modeset 2024-08-07T17:52:07.8082359Z xt_conntrack 16384 1 2024-08-07T17:52:07.8082728Z ipt_MASQUERADE 16384 1 2024-08-07T17:52:07.8083140Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-08-07T17:52:07.8083573Z nf_conntrack_netlink 49152 0 2024-08-07T17:52:07.8084000Z nfnetlink 16384 2 nf_conntrack_netlink 2024-08-07T17:52:07.8084661Z xfrm_user 45056 1 2024-08-07T17:52:07.8085018Z xfrm_algo 16384 1 xfrm_user 2024-08-07T17:52:07.8085423Z iptable_nat 16384 1 2024-08-07T17:52:07.8085775Z nf_conntrack_ipv4 16384 3 2024-08-07T17:52:07.8086184Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-08-07T17:52:07.8086641Z nf_nat_ipv4 16384 1 iptable_nat 2024-08-07T17:52:07.8087102Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-08-07T17:52:07.8088097Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-08-07T17:52:07.8088912Z xt_addrtype 16384 2 2024-08-07T17:52:07.8089293Z iptable_filter 16384 1 2024-08-07T17:52:07.8089648Z br_netfilter 24576 0 2024-08-07T17:52:07.8090036Z bridge 172032 1 br_netfilter 2024-08-07T17:52:07.8090452Z stp 16384 1 bridge 2024-08-07T17:52:07.8090846Z llc 16384 2 bridge,stp 2024-08-07T17:52:07.8091284Z overlay 86016 0 2024-08-07T17:52:07.8091647Z sunrpc 393216 1 2024-08-07T17:52:07.8091991Z dm_mirror 28672 0 2024-08-07T17:52:07.8092369Z dm_region_hash 20480 1 dm_mirror 2024-08-07T17:52:07.8092820Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-08-07T17:52:07.8093273Z dm_mod 143360 2 dm_log,dm_mirror 2024-08-07T17:52:07.8093703Z dax 69632 1 dm_mod 2024-08-07T17:52:07.8094092Z sb_edac 24576 0 2024-08-07T17:52:07.8094438Z crc32_pclmul 16384 0 2024-08-07T17:52:07.8094818Z ghash_clmulni_intel 16384 0 2024-08-07T17:52:07.8095193Z pcbc 16384 0 2024-08-07T17:52:07.8095541Z aesni_intel 188416 0 2024-08-07T17:52:07.8095926Z aes_x86_64 20480 1 aesni_intel 2024-08-07T17:52:07.8096353Z crypto_simd 16384 1 aesni_intel 2024-08-07T17:52:07.8096774Z glue_helper 16384 1 aesni_intel 2024-08-07T17:52:07.8097296Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-08-07T17:52:07.8097810Z mousedev 24576 0 2024-08-07T17:52:07.8098157Z evdev 20480 3 2024-08-07T17:52:07.8099272Z psmouse 32768 0 2024-08-07T17:52:07.8099666Z pcc_cpufreq 16384 0 2024-08-07T17:52:07.8100016Z button 16384 0 2024-08-07T17:52:07.8100374Z ena 139264 0 2024-08-07T17:52:07.8100732Z ptp 20480 1 ena 2024-08-07T17:52:07.8101100Z pps_core 20480 1 ptp 2024-08-07T17:52:07.8101602Z ata_piix 36864 0 2024-08-07T17:52:07.8101987Z xen_blkfront 49152 2 2024-08-07T17:52:07.8102345Z libata 266240 1 ata_piix 2024-08-07T17:52:07.8102747Z crc32c_intel 24576 0 2024-08-07T17:52:07.8103203Z scsi_mod 245760 1 libata 2024-08-07T17:52:07.8103581Z autofs4 49152 2 2024-08-07T17:52:07.8103945Z + modinfo nvidia 2024-08-07T17:52:07.8104451Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-08-07T17:52:07.8105036Z alias: char-major-195-* 2024-08-07T17:52:07.8105410Z version: 550.54.15 2024-08-07T17:52:07.8105757Z supported: external 2024-08-07T17:52:07.8106083Z license: NVIDIA 2024-08-07T17:52:07.8106455Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-07T17:52:07.8106919Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-07T17:52:07.8107343Z srcversion: 833721318DA517F0C2FEC97 2024-08-07T17:52:07.8107790Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-07T17:52:07.8108243Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-07T17:52:07.8108707Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-07T17:52:07.8109144Z depends: i2c-core,drm 2024-08-07T17:52:07.8109489Z retpoline: Y 2024-08-07T17:52:07.8109803Z name: nvidia 2024-08-07T17:52:07.8110406Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-08-07T17:52:07.8110984Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-07T17:52:07.8111661Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-07T17:52:07.8112234Z parm: NVreg_ResmanDebugLevel:int 2024-08-07T17:52:07.8112650Z parm: NVreg_RmLogonRC:int 2024-08-07T17:52:07.8113091Z parm: NVreg_ModifyDeviceFiles:int 2024-08-07T17:52:07.8113536Z parm: NVreg_DeviceFileUID:int 2024-08-07T17:52:07.8113945Z parm: NVreg_DeviceFileGID:int 2024-08-07T17:52:07.8114494Z parm: NVreg_DeviceFileMode:int 2024-08-07T17:52:07.8115014Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-07T17:52:07.8115530Z parm: NVreg_UsePageAttributeTable:int 2024-08-07T17:52:07.8115998Z parm: NVreg_EnablePCIeGen3:int 2024-08-07T17:52:07.8116421Z parm: NVreg_EnableMSI:int 2024-08-07T17:52:07.8116822Z parm: NVreg_TCEBypassMode:int 2024-08-07T17:52:07.8117268Z parm: NVreg_EnableStreamMemOPs:int 2024-08-07T17:52:07.8117773Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-07T17:52:07.8118297Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-07T17:52:07.8118822Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-07T17:52:07.8119384Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-07T17:52:07.8119937Z parm: NVreg_DynamicPowerManagement:int 2024-08-07T17:52:07.8120510Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-07T17:52:07.8121078Z parm: NVreg_EnableGpuFirmware:int 2024-08-07T17:52:07.8121534Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-07T17:52:07.8122046Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-07T17:52:07.8122562Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-07T17:52:07.8123019Z parm: NVreg_MemoryPoolSize:int 2024-08-07T17:52:07.8123479Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-07T17:52:07.8123945Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-07T17:52:07.8124382Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-07T17:52:07.8124817Z parm: NVreg_NvLinkDisable:int 2024-08-07T17:52:07.8125294Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-07T17:52:07.8125770Z parm: NVreg_RegisterPCIDriver:int 2024-08-07T17:52:07.8126229Z parm: NVreg_EnableResizableBar:int 2024-08-07T17:52:07.8126696Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-07T17:52:07.8127156Z parm: NVreg_EnableNonblockingOpen:int 2024-08-07T17:52:07.8127629Z parm: NVreg_RegistryDwords:charp 2024-08-07T17:52:07.8128104Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-07T17:52:07.8128542Z parm: NVreg_RmMsg:charp 2024-08-07T17:52:07.8128947Z parm: NVreg_GpuBlacklist:charp 2024-08-07T17:52:07.8129402Z parm: NVreg_TemporaryFilePath:charp 2024-08-07T17:52:07.8129849Z parm: NVreg_ExcludedGpus:charp 2024-08-07T17:52:07.8130287Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-07T17:52:07.8130746Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-07T17:52:07.8131186Z parm: NVreg_ImexChannelCount:int 2024-08-07T17:52:07.8131623Z parm: rm_firmware_active:charp 2024-08-07T17:52:07.8132027Z + set +e 2024-08-07T17:52:07.8132296Z + nvidia-smi 2024-08-07T17:52:07.8342195Z Wed Aug 7 17:52:07 2024 2024-08-07T17:52:07.8342702Z +-----------------------------------------------------------------------------------------+ 2024-08-07T17:52:07.8343376Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-07T17:52:07.8344044Z |-----------------------------------------+------------------------+----------------------+ 2024-08-07T17:52:07.8344721Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-07T17:52:07.8345430Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-07T17:52:07.8346223Z | | | MIG M. | 2024-08-07T17:52:07.8346695Z |=========================================+========================+======================| 2024-08-07T17:52:07.8432820Z | 0 Tesla M60 On | 00000000:00:1D.0 Off | 0 | 2024-08-07T17:52:07.8433441Z | N/A 32C P8 15W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-07T17:52:07.8434247Z | | | N/A | 2024-08-07T17:52:07.8434827Z +-----------------------------------------+------------------------+----------------------+ 2024-08-07T17:52:07.8498095Z | 1 Tesla M60 On | 00000000:00:1E.0 Off | 0 | 2024-08-07T17:52:07.8498935Z | N/A 38C P8 15W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-07T17:52:07.8499495Z | | | N/A | 2024-08-07T17:52:07.8500027Z +-----------------------------------------+------------------------+----------------------+ 2024-08-07T17:52:07.8500574Z 2024-08-07T17:52:07.8501112Z +-----------------------------------------------------------------------------------------+ 2024-08-07T17:52:07.8501705Z | Processes: | 2024-08-07T17:52:07.8502316Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-07T17:52:07.8502897Z | ID ID Usage | 2024-08-07T17:52:07.8503388Z |=========================================================================================| 2024-08-07T17:52:07.8504640Z | No running processes found | 2024-08-07T17:52:07.8505295Z +-----------------------------------------------------------------------------------------+ 2024-08-07T17:52:07.9616107Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-07T17:52:07.9802467Z Tesla M60 2024-08-07T17:52:07.9868411Z + NVIDIA_SMI_STATUS=0 2024-08-07T17:52:07.9868763Z + '[' 0 -eq 0 ']' 2024-08-07T17:52:07.9869115Z + echo 'INFO: Ignoring allowed status 0' 2024-08-07T17:52:07.9869505Z + set -e 2024-08-07T17:52:07.9870321Z INFO: Ignoring allowed status 0 2024-08-07T17:52:07.9875004Z == Installing nvidia container toolkit for amzn2 == 2024-08-07T17:52:07.9878555Z + sudo yum install -y yum-utils 2024-08-07T17:52:08.4742905Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-08-07T17:52:10.1551276Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-08-07T17:52:10.1551918Z Nothing to do 2024-08-07T17:52:10.2963947Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-08-07T17:52:10.2964594Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-08-07T17:52:10.2965512Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-08-07T17:52:10.8012039Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-08-07T17:52:10.8326957Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-08-07T17:52:10.8327982Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-08-07T17:52:10.8328874Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-08-07T17:52:10.8472991Z + sudo yum install -y nvidia-docker2 2024-08-07T17:52:11.3323086Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-08-07T17:52:12.9993001Z Package nvidia-docker2-2.14.0-1.noarch already installed and latest version 2024-08-07T17:52:12.9993630Z Nothing to do 2024-08-07T17:52:13.1375766Z + sudo systemctl restart docker 2024-08-07T17:52:32.2178752Z nvidia-persistenced failed to initialize. Check syslog for more details. 2024-08-07T17:52:32.2669965Z Wed Aug 7 17:52:32 2024 2024-08-07T17:52:32.2670475Z +-----------------------------------------------------------------------------------------+ 2024-08-07T17:52:32.2671169Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-07T17:52:32.2671834Z |-----------------------------------------+------------------------+----------------------+ 2024-08-07T17:52:32.2672767Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-07T17:52:32.2673534Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-07T17:52:32.2674129Z | | | MIG M. | 2024-08-07T17:52:32.2674605Z |=========================================+========================+======================| 2024-08-07T17:52:32.2754700Z | 0 Tesla M60 On | 00000000:00:1D.0 Off | 0 | 2024-08-07T17:52:32.2755322Z | N/A 32C P8 15W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-07T17:52:32.2755861Z | | | N/A | 2024-08-07T17:52:32.2756402Z +-----------------------------------------+------------------------+----------------------+ 2024-08-07T17:52:32.2821522Z | 1 Tesla M60 On | 00000000:00:1E.0 Off | 0 | 2024-08-07T17:52:32.2822217Z | N/A 38C P8 15W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-07T17:52:32.2822737Z | | | N/A | 2024-08-07T17:52:32.2823516Z +-----------------------------------------+------------------------+----------------------+ 2024-08-07T17:52:32.2824092Z 2024-08-07T17:52:32.2824631Z +-----------------------------------------------------------------------------------------+ 2024-08-07T17:52:32.2825203Z | Processes: | 2024-08-07T17:52:32.2825820Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-07T17:52:32.2826402Z | ID ID Usage | 2024-08-07T17:52:32.2826880Z |=========================================================================================| 2024-08-07T17:52:32.2827884Z | No running processes found | 2024-08-07T17:52:32.2828935Z +-----------------------------------------------------------------------------------------+ 2024-08-07T17:52:33.1282992Z Command completed after 1 attempt(s). 2024-08-07T17:52:33.1351415Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-07T17:52:33.1352200Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-07T17:52:33.1352823Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-07T17:52:33.1353432Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-07T17:52:33.1361505Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:52:33.1361981Z env: 2024-08-07T17:52:33.1362347Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:52:33.1362785Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:52:33.1363239Z ##[endgroup] 2024-08-07T17:52:33.4276991Z Defaulting to user installation because normal site-packages is not writeable 2024-08-07T17:52:33.4483726Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-08-07T17:52:33.4607940Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-08-07T17:52:33.5751372Z Prepare all required actions 2024-08-07T17:52:33.5752558Z Getting action download info 2024-08-07T17:52:33.6698048Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-07T17:52:33.9419519Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-07T17:52:34.0712549Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-07T17:52:34.0713013Z with: 2024-08-07T17:52:34.0713432Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-08-07T17:52:34.0713862Z s3-bucket: gha-artifacts 2024-08-07T17:52:34.0714199Z env: 2024-08-07T17:52:34.0714494Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:52:34.0714931Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:52:34.0715391Z ##[endgroup] 2024-08-07T17:52:34.0785470Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-07T17:52:34.0785942Z with: 2024-08-07T17:52:34.0786322Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-08-07T17:52:34.0786740Z s3-bucket: gha-artifacts 2024-08-07T17:52:34.0787102Z region: us-east-1 2024-08-07T17:52:34.0787413Z env: 2024-08-07T17:52:34.0787688Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:52:34.0788124Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:52:34.0788592Z ##[endgroup] 2024-08-07T17:52:34.6808895Z (node:72911) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-07T17:52:34.6809513Z 2024-08-07T17:52:34.6809773Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-07T17:52:34.6810484Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-07T17:52:34.6811199Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-07T17:52:34.7670449Z Found 1 objects with prefix pytorch/pytorch/10288745067/linux-focal-cuda11.8-py3.10-gcc9/ 2024-08-07T17:52:34.7671388Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-07T17:52:48.3551345Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-07T17:52:48.3561540Z Artifact download has finished successfully 2024-08-07T17:52:48.3775736Z ##[group]Run unzip -o artifacts.zip 2024-08-07T17:52:48.3776206Z unzip -o artifacts.zip 2024-08-07T17:52:48.3785036Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:52:48.3785529Z env: 2024-08-07T17:52:48.3785831Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:52:48.3786252Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:52:48.3786713Z ##[endgroup] 2024-08-07T17:52:48.3822940Z Archive: artifacts.zip 2024-08-07T17:52:48.3824783Z creating: dist/ 2024-08-07T17:52:50.8294773Z inflating: dist/torch-2.5.0a0+git016588f-cp310-cp310-linux_x86_64.whl 2024-08-07T17:52:50.8295389Z creating: build/custom_test_artifacts/ 2024-08-07T17:52:50.8295939Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-07T17:52:50.8296574Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-07T17:52:50.8297300Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-07T17:52:50.8305669Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-07T17:52:50.8306499Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-07T17:52:50.8307313Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-07T17:52:50.8308163Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-07T17:52:50.8309011Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-07T17:52:50.8310184Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-07T17:52:50.8312562Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-07T17:52:50.8313619Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-07T17:52:50.8314495Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-07T17:52:50.8315804Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-07T17:52:50.8317591Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-07T17:52:50.8320612Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-07T17:52:50.8321637Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-07T17:52:50.8322859Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-07T17:52:50.8323916Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-07T17:52:50.8324839Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-07T17:52:50.8325725Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-07T17:52:50.8383716Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-07T17:52:50.8384951Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-07T17:52:50.8386219Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-07T17:52:50.8387518Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-07T17:52:50.8388815Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-07T17:52:50.8390037Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-07T17:52:50.8391266Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-07T17:52:50.8392502Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-07T17:52:50.8393717Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-07T17:52:50.8439646Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-07T17:52:50.8489405Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-07T17:52:50.8490649Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-07T17:52:50.8491811Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-07T17:52:50.8492896Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-07T17:52:50.8493984Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-07T17:52:50.8495075Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-07T17:52:50.8496134Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-07T17:52:50.8497192Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-07T17:52:50.8573233Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-07T17:52:50.8648977Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-07T17:52:50.8650030Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-07T17:52:50.8650892Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-07T17:52:50.8651656Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-07T17:52:50.8652453Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-07T17:52:50.8653254Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-07T17:52:50.8654159Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-07T17:52:50.8655198Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-07T17:52:50.8656197Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-07T17:52:50.8657116Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-07T17:52:50.8658072Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-07T17:52:50.8659295Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-07T17:52:50.8660237Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-07T17:52:50.8661189Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-07T17:52:50.8662129Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-07T17:52:50.8683876Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-07T17:52:50.8863569Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-07T17:52:50.8864457Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-07T17:52:50.8865417Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-07T17:52:50.8866476Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-07T17:52:50.8867509Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-07T17:52:50.8868476Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-07T17:52:50.8869489Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-07T17:52:50.8870488Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-07T17:52:50.8871508Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-07T17:52:50.8872507Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-07T17:52:50.8873493Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-07T17:52:50.8896278Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-07T17:52:50.9000599Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-07T17:52:50.9001661Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-07T17:52:50.9002600Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-07T17:52:50.9003776Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-07T17:52:50.9004601Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-07T17:52:50.9005394Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-07T17:52:50.9006174Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-07T17:52:50.9008081Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-07T17:52:50.9009026Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-07T17:52:50.9009721Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-07T17:52:50.9159136Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-07T17:52:50.9237629Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-07T17:52:50.9238271Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-07T17:52:50.9238865Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-07T17:52:50.9239580Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-07T17:52:50.9247706Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-07T17:52:50.9248511Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-07T17:52:50.9249317Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-07T17:52:50.9250171Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-07T17:52:50.9251005Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-07T17:52:50.9251935Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-07T17:52:50.9253470Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-07T17:52:50.9254351Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-07T17:52:50.9255189Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-07T17:52:50.9257331Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-07T17:52:50.9259052Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-07T17:52:50.9261901Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-07T17:52:50.9262883Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-07T17:52:50.9264322Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-07T17:52:50.9265361Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-07T17:52:50.9266279Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-07T17:52:50.9267137Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-07T17:52:50.9324343Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-07T17:52:50.9325578Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-07T17:52:50.9326825Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-07T17:52:50.9328091Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-07T17:52:50.9329357Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-07T17:52:50.9330850Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-07T17:52:50.9332077Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-07T17:52:50.9333282Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-07T17:52:50.9334492Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-07T17:52:50.9380657Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-07T17:52:50.9430564Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-07T17:52:50.9431778Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-07T17:52:50.9432922Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-07T17:52:50.9434019Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-07T17:52:50.9435097Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-07T17:52:50.9436162Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-07T17:52:50.9437213Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-07T17:52:50.9438279Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-07T17:52:50.9513524Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-07T17:52:50.9589277Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-07T17:52:50.9590315Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-07T17:52:50.9591174Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-07T17:52:50.9591903Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-07T17:52:50.9592686Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-07T17:52:50.9593507Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-07T17:52:50.9594421Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-07T17:52:50.9595472Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-07T17:52:50.9596508Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-07T17:52:50.9597455Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-07T17:52:50.9598406Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-07T17:52:50.9599382Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-07T17:52:50.9600365Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-07T17:52:50.9601327Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-07T17:52:50.9602283Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-07T17:52:50.9624160Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-07T17:52:50.9704403Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-07T17:52:50.9705454Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-07T17:52:50.9706395Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-07T17:52:50.9707260Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-07T17:52:50.9708050Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-07T17:52:50.9708809Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-07T17:52:50.9709608Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-07T17:52:50.9711906Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-07T17:52:50.9712579Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-07T17:52:50.9713295Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-07T17:52:50.9775246Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-07T17:52:50.9775937Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-07T17:52:50.9776599Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-07T17:52:50.9777356Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-07T17:52:50.9785620Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-07T17:52:50.9786504Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-07T17:52:50.9787367Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-07T17:52:50.9788276Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-07T17:52:50.9789209Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-07T17:52:50.9790238Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-07T17:52:50.9791587Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-07T17:52:50.9792530Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-07T17:52:50.9793462Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-07T17:52:50.9795259Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-07T17:52:50.9797034Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-07T17:52:50.9799388Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-07T17:52:50.9800482Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-07T17:52:50.9801856Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-07T17:52:50.9802950Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-07T17:52:50.9803927Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-07T17:52:50.9804864Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-07T17:52:50.9862116Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-07T17:52:50.9863447Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-07T17:52:50.9865088Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-07T17:52:50.9866469Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-07T17:52:50.9867815Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-07T17:52:50.9869080Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-07T17:52:50.9870362Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-07T17:52:50.9871660Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-07T17:52:50.9872943Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-07T17:52:50.9918204Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-07T17:52:50.9968048Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-07T17:52:50.9969336Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-07T17:52:50.9970543Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-07T17:52:50.9971689Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-07T17:52:50.9972835Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-07T17:52:50.9974005Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-07T17:52:50.9975114Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-07T17:52:50.9976252Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-07T17:52:51.0050971Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-07T17:52:51.0126791Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-07T17:52:51.0127890Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-07T17:52:51.0128830Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-07T17:52:51.0129643Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-07T17:52:51.0130505Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-07T17:52:51.0131380Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-07T17:52:51.0132379Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-07T17:52:51.0133517Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-07T17:52:51.0134598Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-07T17:52:51.0135622Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-07T17:52:51.0136673Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-07T17:52:51.0138019Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-07T17:52:51.0139379Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-07T17:52:51.0140449Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-07T17:52:51.0141495Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-07T17:52:51.0142596Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-07T17:52:51.0299037Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-07T17:52:51.0301243Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-07T17:52:51.0302308Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-07T17:52:51.0303506Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-07T17:52:51.0304655Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-07T17:52:51.0305729Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-07T17:52:51.0306834Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-07T17:52:51.0307929Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-07T17:52:51.0309048Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-07T17:52:51.0310160Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-07T17:52:51.0311253Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-07T17:52:51.0332242Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-07T17:52:51.0401729Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-07T17:52:51.0402899Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-07T17:52:51.0403897Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-07T17:52:51.0404820Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-07T17:52:51.0405670Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-07T17:52:51.0406511Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-07T17:52:51.0407358Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-07T17:52:51.0409699Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-07T17:52:51.0410443Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-07T17:52:51.0411186Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-07T17:52:51.0540483Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-07T17:52:51.0593953Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-07T17:52:51.0594563Z creating: build/lib/ 2024-08-07T17:52:51.0606115Z inflating: build/lib/libpthreadpool.a 2024-08-07T17:52:51.0615829Z inflating: build/lib/libcpuinfo_internals.a 2024-08-07T17:52:51.0626281Z inflating: build/lib/libcpuinfo.a 2024-08-07T17:52:51.0627175Z inflating: build/lib/libclog.a 2024-08-07T17:52:51.0738121Z inflating: build/lib/libprotobuf-lite.a 2024-08-07T17:52:51.0741094Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-07T17:52:51.1308022Z inflating: build/lib/libprotobuf.a 2024-08-07T17:52:51.1388976Z inflating: build/lib/libgtest.a 2024-08-07T17:52:51.1481963Z inflating: build/lib/libbenchmark.a 2024-08-07T17:52:51.1504628Z inflating: build/lib/libnnpack.a 2024-08-07T17:52:51.1583773Z inflating: build/lib/libasmjit.a 2024-08-07T17:52:51.1593143Z inflating: build/lib/libittnotify.a 2024-08-07T17:52:51.1627739Z inflating: build/lib/libtensorpipe_uv.a 2024-08-07T17:52:51.1785939Z inflating: build/lib/libgloo.a 2024-08-07T17:52:51.1811873Z inflating: build/lib/libfmt.a 2024-08-07T17:52:51.1930292Z inflating: build/lib/libc10.so 2024-08-07T17:52:51.1932336Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-07T17:52:51.1932838Z inflating: build/lib/libfoxi_loader.a 2024-08-07T17:52:51.1935060Z inflating: build/lib/libtorch_global_deps.so 2024-08-07T17:52:51.1959229Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-07T17:52:51.1981824Z inflating: build/lib/libgmock.a 2024-08-07T17:52:51.2610321Z inflating: build/lib/libprotoc.a 2024-08-07T17:52:51.2610943Z inflating: build/lib/libgtest_main.a 2024-08-07T17:52:51.2611984Z inflating: build/lib/libbenchmark_main.a 2024-08-07T17:52:52.5219257Z inflating: build/lib/libdnnl.a 2024-08-07T17:52:52.5931308Z inflating: build/lib/libtensorpipe.a 2024-08-07T17:52:52.6002453Z inflating: build/lib/libc10_cuda.so 2024-08-07T17:52:52.6002927Z inflating: build/lib/libgmock_main.a 2024-08-07T17:52:52.7559654Z inflating: build/lib/libfbgemm.a 2024-08-07T17:52:52.8188144Z inflating: build/lib/libkineto.a 2024-08-07T17:52:52.8505638Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-07T17:52:52.8988594Z inflating: build/lib/libgloo_cuda.a 2024-08-07T17:52:52.9217535Z inflating: build/lib/libXNNPACK.a 2024-08-07T17:52:52.9270733Z inflating: build/lib/libonnx_proto.a 2024-08-07T17:52:53.0130511Z inflating: build/lib/libonnx.a 2024-08-07T17:52:56.0234860Z inflating: build/lib/libtorch_cpu.so 2024-08-07T17:52:56.0239810Z inflating: build/lib/libshm.so 2024-08-07T17:52:56.0245666Z inflating: build/lib/libunbox_lib.a 2024-08-07T17:52:58.4952197Z inflating: build/lib/libtorch_cuda.so 2024-08-07T17:52:58.4953222Z inflating: build/lib/libtorch.so 2024-08-07T17:52:58.4956590Z inflating: build/lib/libc10d_cuda_test.so 2024-08-07T17:52:59.6331534Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-07T17:52:59.6357344Z inflating: build/lib/libjitbackend_test.so 2024-08-07T17:52:59.6446981Z inflating: build/lib/libtorchbind_test.so 2024-08-07T17:52:59.6479434Z inflating: build/lib/libaoti_custom_ops.so 2024-08-07T17:52:59.6511200Z inflating: build/lib/libbackend_with_compiler.so 2024-08-07T17:52:59.8989513Z inflating: build/lib/libtorch_python.so 2024-08-07T17:52:59.9032362Z inflating: build/lib/libnnapi_backend.so 2024-08-07T17:52:59.9032901Z creating: build/bin/ 2024-08-07T17:52:59.9096083Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-07T17:52:59.9160330Z inflating: build/bin/c10_DeviceGuard_test 2024-08-07T17:52:59.9223188Z inflating: build/bin/c10_Device_test 2024-08-07T17:52:59.9296654Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-07T17:52:59.9363373Z inflating: build/bin/c10_Scalar_test 2024-08-07T17:52:59.9424602Z inflating: build/bin/c10_StreamGuard_test 2024-08-07T17:52:59.9487979Z inflating: build/bin/c10_SymInt_test 2024-08-07T17:52:59.9556188Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-07T17:52:59.9625636Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-07T17:52:59.9695627Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-07T17:52:59.9784288Z inflating: build/bin/c10_cow_test 2024-08-07T17:52:59.9850301Z inflating: build/bin/c10_Bitset_test 2024-08-07T17:52:59.9911910Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-07T17:52:59.9973596Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-07T17:53:00.0037144Z inflating: build/bin/c10_Half_test 2024-08-07T17:53:00.0107528Z inflating: build/bin/c10_LeftRight_test 2024-08-07T17:53:00.0176092Z inflating: build/bin/c10_Metaprogramming_test 2024-08-07T17:53:00.0238758Z inflating: build/bin/c10_Synchronized_test 2024-08-07T17:53:00.0308214Z inflating: build/bin/c10_ThreadLocal_test 2024-08-07T17:53:00.0372640Z inflating: build/bin/c10_TypeIndex_test 2024-08-07T17:53:00.0436503Z inflating: build/bin/c10_TypeList_test 2024-08-07T17:53:00.0497739Z inflating: build/bin/c10_TypeTraits_test 2024-08-07T17:53:00.0563033Z inflating: build/bin/c10_accumulate_test 2024-08-07T17:53:00.0632328Z inflating: build/bin/c10_bfloat16_test 2024-08-07T17:53:00.0695290Z inflating: build/bin/c10_bit_cast_test 2024-08-07T17:53:00.0765755Z inflating: build/bin/c10_complex_math_test 2024-08-07T17:53:00.0831545Z inflating: build/bin/c10_exception_test 2024-08-07T17:53:00.0901136Z inflating: build/bin/c10_complex_test 2024-08-07T17:53:00.0964139Z inflating: build/bin/c10_flags_test 2024-08-07T17:53:00.1026885Z inflating: build/bin/c10_generic_math_test 2024-08-07T17:53:00.1090332Z inflating: build/bin/c10_irange_test 2024-08-07T17:53:00.1296286Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-07T17:53:00.1363398Z inflating: build/bin/c10_lazy_test 2024-08-07T17:53:00.1434833Z inflating: build/bin/c10_logging_test 2024-08-07T17:53:00.1528515Z inflating: build/bin/c10_optional_test 2024-08-07T17:53:00.1606358Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-07T17:53:00.1673634Z inflating: build/bin/c10_registry_test 2024-08-07T17:53:00.1862769Z inflating: build/bin/c10_small_vector_test 2024-08-07T17:53:00.1927259Z inflating: build/bin/c10_ssize_test 2024-08-07T17:53:00.1992216Z inflating: build/bin/c10_string_util_test 2024-08-07T17:53:00.2054990Z inflating: build/bin/c10_tempfile_test 2024-08-07T17:53:00.2128349Z inflating: build/bin/c10_string_view_test 2024-08-07T17:53:00.2198413Z inflating: build/bin/c10_typeid_test 2024-08-07T17:53:00.2259006Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-07T17:53:00.2816084Z inflating: build/bin/protoc-3.13.0.0 2024-08-07T17:53:00.3373059Z inflating: build/bin/protoc 2024-08-07T17:53:00.3439111Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-07T17:53:00.3505363Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-07T17:53:00.3571026Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-07T17:53:00.3636111Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-07T17:53:00.3702163Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-07T17:53:00.3763331Z inflating: build/bin/c10_cuda_CUDATest 2024-08-07T17:53:00.3829258Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-07T17:53:00.3895170Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-07T17:53:00.4324641Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-07T17:53:00.4736604Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-07T17:53:00.5186277Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-07T17:53:00.5252493Z inflating: build/bin/FileStoreTest 2024-08-07T17:53:00.5319877Z inflating: build/bin/HashStoreTest 2024-08-07T17:53:00.5385667Z inflating: build/bin/BackoffTest 2024-08-07T17:53:00.5455168Z inflating: build/bin/TCPStoreTest 2024-08-07T17:53:00.5472263Z inflating: build/bin/ProcessGroupMPITest 2024-08-07T17:53:00.5477500Z inflating: build/bin/torch_shm_manager 2024-08-07T17:53:00.5545537Z inflating: build/bin/test_edge_op_registration 2024-08-07T17:53:00.5549324Z inflating: build/bin/example_allreduce 2024-08-07T17:53:00.5614142Z inflating: build/bin/Dimname_test 2024-08-07T17:53:00.5706318Z inflating: build/bin/Dict_test 2024-08-07T17:53:00.5787030Z inflating: build/bin/MaybeOwned_test 2024-08-07T17:53:00.5861465Z inflating: build/bin/apply_utils_test 2024-08-07T17:53:00.5932947Z inflating: build/bin/NamedTensor_test 2024-08-07T17:53:00.6006532Z inflating: build/bin/atest 2024-08-07T17:53:00.6085043Z inflating: build/bin/basic 2024-08-07T17:53:00.6153775Z inflating: build/bin/broadcast_test 2024-08-07T17:53:00.6216932Z inflating: build/bin/cpu_allocator_test 2024-08-07T17:53:00.6289548Z inflating: build/bin/cpu_generator_test 2024-08-07T17:53:00.6356159Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-07T17:53:00.6472647Z inflating: build/bin/cpu_rng_test 2024-08-07T17:53:00.6534884Z inflating: build/bin/dispatch_key_set_test 2024-08-07T17:53:00.6598037Z inflating: build/bin/dlconvertor_test 2024-08-07T17:53:00.6670897Z inflating: build/bin/extension_backend_test 2024-08-07T17:53:00.6739794Z inflating: build/bin/half_test 2024-08-07T17:53:00.6859227Z inflating: build/bin/ivalue_test 2024-08-07T17:53:00.6921199Z inflating: build/bin/lazy_tensor_test 2024-08-07T17:53:00.6989169Z inflating: build/bin/math_kernel_test 2024-08-07T17:53:00.7056613Z inflating: build/bin/memory_format_test 2024-08-07T17:53:00.7122964Z inflating: build/bin/memory_overlapping_test 2024-08-07T17:53:00.7189265Z inflating: build/bin/mobile_memory_cleanup 2024-08-07T17:53:00.7252393Z inflating: build/bin/operator_name_test 2024-08-07T17:53:00.7322319Z inflating: build/bin/native_test 2024-08-07T17:53:00.7386816Z inflating: build/bin/operators_test 2024-08-07T17:53:00.7451355Z inflating: build/bin/packedtensoraccessor_test 2024-08-07T17:53:00.7535316Z inflating: build/bin/pow_test 2024-08-07T17:53:00.7606665Z inflating: build/bin/quantized_test 2024-08-07T17:53:00.7669305Z inflating: build/bin/reduce_ops_test 2024-08-07T17:53:00.7732979Z inflating: build/bin/reportMemoryUsage_test 2024-08-07T17:53:00.7803489Z inflating: build/bin/scalar_tensor_test 2024-08-07T17:53:00.7876509Z inflating: build/bin/scalar_test 2024-08-07T17:53:00.7941349Z inflating: build/bin/StorageUtils_test 2024-08-07T17:53:00.8006809Z inflating: build/bin/stride_properties_test 2024-08-07T17:53:00.8105214Z inflating: build/bin/tensor_iterator_test 2024-08-07T17:53:00.8108847Z inflating: build/bin/thread_init_test 2024-08-07T17:53:00.8178231Z inflating: build/bin/type_ptr_test 2024-08-07T17:53:00.8246946Z inflating: build/bin/test_parallel 2024-08-07T17:53:00.8321922Z inflating: build/bin/type_test 2024-08-07T17:53:00.8387175Z inflating: build/bin/undefined_tensor_test 2024-08-07T17:53:00.8389155Z inflating: build/bin/verify_api_visibility 2024-08-07T17:53:00.8475749Z inflating: build/bin/legacy_vmap_test 2024-08-07T17:53:00.8540410Z inflating: build/bin/wrapdim_test 2024-08-07T17:53:00.8604854Z inflating: build/bin/weakref_test 2024-08-07T17:53:00.8669550Z inflating: build/bin/xla_tensor_test 2024-08-07T17:53:00.8744654Z inflating: build/bin/IListRef_test 2024-08-07T17:53:00.8877785Z inflating: build/bin/List_test 2024-08-07T17:53:00.9029904Z inflating: build/bin/kernel_function_legacy_test 2024-08-07T17:53:00.9150803Z inflating: build/bin/kernel_function_test 2024-08-07T17:53:00.9234123Z inflating: build/bin/KernelFunction_test 2024-08-07T17:53:00.9394229Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-07T17:53:00.9523536Z inflating: build/bin/kernel_lambda_test 2024-08-07T17:53:00.9599952Z inflating: build/bin/kernel_stackbased_test 2024-08-07T17:53:00.9720204Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-07T17:53:00.9783928Z inflating: build/bin/CppSignature_test 2024-08-07T17:53:00.9845044Z inflating: build/bin/op_allowlist_test 2024-08-07T17:53:00.9915182Z inflating: build/bin/backend_fallback_test 2024-08-07T17:53:01.0292552Z inflating: build/bin/op_registration_test 2024-08-07T17:53:01.0372059Z inflating: build/bin/inline_container_test 2024-08-07T17:53:01.0438183Z inflating: build/bin/cuda_apply_test 2024-08-07T17:53:01.0503269Z inflating: build/bin/cuda_allocator_test 2024-08-07T17:53:01.0571424Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-07T17:53:01.0646012Z inflating: build/bin/cuda_atomic_ops_test 2024-08-07T17:53:01.0734067Z inflating: build/bin/cuda_complex_math_test 2024-08-07T17:53:01.0807759Z inflating: build/bin/cuda_complex_test 2024-08-07T17:53:01.0869917Z inflating: build/bin/cuda_device_test 2024-08-07T17:53:01.0942353Z inflating: build/bin/cuda_cub_test 2024-08-07T17:53:01.1005712Z inflating: build/bin/cuda_dlconvertor_test 2024-08-07T17:53:01.1087507Z inflating: build/bin/cuda_distributions_test 2024-08-07T17:53:01.1151357Z inflating: build/bin/cuda_integer_divider_test 2024-08-07T17:53:01.1213680Z inflating: build/bin/cuda_half_test 2024-08-07T17:53:01.1284509Z inflating: build/bin/cuda_generator_test 2024-08-07T17:53:01.1346119Z inflating: build/bin/cuda_optional_test 2024-08-07T17:53:01.1408466Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-07T17:53:01.1484297Z inflating: build/bin/cuda_stream_test 2024-08-07T17:53:01.1550277Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-07T17:53:01.1614477Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-07T17:53:01.1676096Z inflating: build/bin/cuda_cudnn_test 2024-08-07T17:53:01.1741216Z inflating: build/bin/cuda_vectorized_test 2024-08-07T17:53:01.1823211Z inflating: build/bin/ProcessGroupGlooTest 2024-08-07T17:53:01.1841166Z inflating: build/bin/tutorial_tensorexpr 2024-08-07T17:53:01.1913022Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-07T17:53:01.1992539Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-07T17:53:01.2070152Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-07T17:53:01.2135654Z inflating: build/bin/ProcessGroupUCCTest 2024-08-07T17:53:01.2204285Z inflating: build/bin/test_dist_autograd 2024-08-07T17:53:01.2207520Z inflating: build/bin/parallel_benchmark 2024-08-07T17:53:01.2293739Z inflating: build/bin/test_cpp_rpc 2024-08-07T17:53:01.2377450Z inflating: build/bin/test_mobile_nnc 2024-08-07T17:53:01.2389379Z inflating: build/bin/aot_model_compiler_test 2024-08-07T17:53:01.2822050Z inflating: build/bin/test_lazy 2024-08-07T17:53:01.3852834Z inflating: build/bin/test_tensorexpr 2024-08-07T17:53:01.5325246Z inflating: build/bin/test_api 2024-08-07T17:53:01.6052370Z inflating: build/bin/test_jit 2024-08-07T17:53:01.6052798Z creating: .additional_ci_files/ 2024-08-07T17:53:01.6124836Z inflating: .additional_ci_files/test-times.json 2024-08-07T17:53:01.6410750Z inflating: .additional_ci_files/test-class-times.json 2024-08-07T17:53:01.6454271Z ##[group]Run rm artifacts.zip 2024-08-07T17:53:01.6454708Z rm artifacts.zip 2024-08-07T17:53:01.6463150Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:01.6463684Z env: 2024-08-07T17:53:01.6463985Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:01.6464397Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:01.6464854Z ##[endgroup] 2024-08-07T17:53:01.7660483Z ##[group]Run df -H 2024-08-07T17:53:01.7660832Z df -H 2024-08-07T17:53:01.7668763Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:01.7669250Z env: 2024-08-07T17:53:01.7669527Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:01.7669961Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:01.7670421Z ##[endgroup] 2024-08-07T17:53:01.7705210Z Filesystem Size Used Avail Use% Mounted on 2024-08-07T17:53:01.7706249Z devtmpfs 129G 0 129G 0% /dev 2024-08-07T17:53:01.7706962Z tmpfs 129G 0 129G 0% /dev/shm 2024-08-07T17:53:01.7707843Z tmpfs 129G 594k 129G 1% /run 2024-08-07T17:53:01.7708286Z tmpfs 129G 0 129G 0% /sys/fs/cgroup 2024-08-07T17:53:01.7708728Z /dev/xvda1 162G 38G 125G 23% / 2024-08-07T17:53:01.7709154Z tmpfs 26G 0 26G 0% /run/user/0 2024-08-07T17:53:01.7748833Z Prepare all required actions 2024-08-07T17:53:01.7749364Z Getting action download info 2024-08-07T17:53:01.9077808Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-07T17:53:01.9078699Z with: 2024-08-07T17:53:01.9079195Z env: 2024-08-07T17:53:01.9079705Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:01.9080513Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:01.9081408Z ##[endgroup] 2024-08-07T17:53:01.9152679Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-07T17:53:01.9153527Z with: 2024-08-07T17:53:01.9154067Z name: td_results 2024-08-07T17:53:01.9154646Z s3-bucket: gha-artifacts 2024-08-07T17:53:01.9155349Z region: us-east-1 2024-08-07T17:53:01.9155911Z env: 2024-08-07T17:53:01.9156409Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:01.9157236Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:01.9158142Z ##[endgroup] 2024-08-07T17:53:02.5162462Z (node:72931) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-07T17:53:02.5163077Z 2024-08-07T17:53:02.5163334Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-07T17:53:02.5164047Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-07T17:53:02.5164738Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-07T17:53:02.6147282Z Found 1 objects with prefix pytorch/pytorch/10288745067/td_results/ 2024-08-07T17:53:02.6148192Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-08-07T17:53:02.6882292Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-08-07T17:53:02.6892298Z Artifact download has finished successfully 2024-08-07T17:53:02.7079355Z ##[group]Run mkdir -p .additional_ci_files 2024-08-07T17:53:02.7079831Z mkdir -p .additional_ci_files 2024-08-07T17:53:02.7080351Z mv td_results.json .additional_ci_files/td_results.json 2024-08-07T17:53:02.7088520Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:02.7089003Z env: 2024-08-07T17:53:02.7089279Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:02.7089710Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:02.7090172Z ##[endgroup] 2024-08-07T17:53:02.7193619Z ##[group]Run .github/scripts/parse_ref.py 2024-08-07T17:53:02.7194219Z .github/scripts/parse_ref.py 2024-08-07T17:53:02.7201593Z shell: /usr/bin/bash -e {0} 2024-08-07T17:53:02.7201939Z env: 2024-08-07T17:53:02.7202232Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:02.7202667Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:02.7203134Z ##[endgroup] 2024-08-07T17:53:02.7533062Z Prepare all required actions 2024-08-07T17:53:02.7612817Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-07T17:53:02.7613655Z with: 2024-08-07T17:53:02.7614517Z github-token: *** 2024-08-07T17:53:02.7615078Z env: 2024-08-07T17:53:02.7615580Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:02.7616305Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:02.7617167Z ##[endgroup] 2024-08-07T17:53:02.7657182Z ##[group]Run set -eux 2024-08-07T17:53:02.7657840Z set -eux 2024-08-07T17:53:02.7659337Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-07T17:53:02.7671420Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:02.7672326Z env: 2024-08-07T17:53:02.7672836Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:02.7673624Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:02.7674759Z GITHUB_TOKEN: *** 2024-08-07T17:53:02.7675610Z ##[endgroup] 2024-08-07T17:53:02.7707687Z + python3 .github/scripts/get_workflow_job_id.py 10288745067 i-028914771a8277c83 2024-08-07T17:53:06.1583334Z setting job-id=28476145515 2024-08-07T17:53:06.1584062Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:53:06.1799082Z Prepare all required actions 2024-08-07T17:53:06.1799702Z Getting action download info 2024-08-07T17:53:06.2904983Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-07T17:53:06.2905420Z with: 2024-08-07T17:53:06.2905926Z github-token: *** 2024-08-07T17:53:06.2907203Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2024-08-07T17:53:06.2908801Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:53:06.2909443Z env: 2024-08-07T17:53:06.2909731Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:06.2910164Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:06.2910606Z ##[endgroup] 2024-08-07T17:53:06.2960438Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-07T17:53:06.2961005Z with: 2024-08-07T17:53:06.2961293Z shell: bash 2024-08-07T17:53:06.2961576Z timeout_minutes: 10 2024-08-07T17:53:06.2961913Z max_attempts: 5 2024-08-07T17:53:06.2962228Z retry_wait_seconds: 30 2024-08-07T17:53:06.2963156Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-07T17:53:06.2964194Z polling_interval_seconds: 1 2024-08-07T17:53:06.2964570Z warning_on_retry: true 2024-08-07T17:53:06.2964920Z continue_on_error: false 2024-08-07T17:53:06.2965266Z env: 2024-08-07T17:53:06.2965553Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:06.2965982Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:06.2966739Z GITHUB_TOKEN: *** 2024-08-07T17:53:06.2967075Z ##[endgroup] 2024-08-07T17:53:06.3683412Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-07T17:53:06.6548185Z Defaulting to user installation because normal site-packages is not writeable 2024-08-07T17:53:06.6749352Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-08-07T17:53:06.6929242Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-08-07T17:53:06.6939979Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.7.4) 2024-08-07T17:53:06.6951039Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-08-07T17:53:06.6967888Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-08-07T17:53:06.6993729Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-08-07T17:53:07.3678797Z Command completed after 1 attempt(s). 2024-08-07T17:53:07.3736343Z ##[group]Run set -x 2024-08-07T17:53:07.3736706Z set -x 2024-08-07T17:53:07.3736999Z  2024-08-07T17:53:07.3737489Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-07T17:53:07.3738092Z # in runner workspace 2024-08-07T17:53:07.3738904Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-07T17:53:07.3747027Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:07.3747512Z env: 2024-08-07T17:53:07.3747990Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:07.3748430Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:07.3748896Z ##[endgroup] 2024-08-07T17:53:07.3774182Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-07T17:53:07.4062834Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-07T17:53:07.4063452Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-07T17:53:07.4064103Z echo "Job name: ${JOB_NAME}" 2024-08-07T17:53:07.4064605Z  2024-08-07T17:53:07.4065132Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-07T17:53:07.4065836Z # in runner workspace 2024-08-07T17:53:07.4066449Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-07T17:53:07.4067104Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-07T17:53:07.4067640Z  --job-name "${JOB_NAME}" \ 2024-08-07T17:53:07.4085997Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" \ 2024-08-07T17:53:07.4087436Z  --selected-test-configs "" \ 2024-08-07T17:53:07.4087888Z  --pr-number "${PR_NUMBER}" \ 2024-08-07T17:53:07.4088285Z  --tag "${TAG}" \ 2024-08-07T17:53:07.4088668Z  --event-name "${EVENT_NAME}" \ 2024-08-07T17:53:07.4089096Z  --schedule "${SCHEDULE}" \ 2024-08-07T17:53:07.4089489Z  --branch "${HEAD_BRANCH}" 2024-08-07T17:53:07.4096850Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:07.4097334Z env: 2024-08-07T17:53:07.4097609Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:07.4098041Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:07.4098996Z GITHUB_TOKEN: *** 2024-08-07T17:53:07.4099565Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:53:07.4100215Z PR_NUMBER: 131248 2024-08-07T17:53:07.4100524Z TAG: 2024-08-07T17:53:07.4100794Z EVENT_NAME: pull_request 2024-08-07T17:53:07.4101148Z SCHEDULE: 2024-08-07T17:53:07.4101626Z HEAD_BRANCH: 2024-08-07T17:53:07.4101913Z ##[endgroup] 2024-08-07T17:53:07.4124403Z Workflow: pull 2024-08-07T17:53:07.4125004Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:53:07.6583894Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-07T17:53:07.8075154Z ##[group]Run echo "Filtered matrix:" 2024-08-07T17:53:07.8075608Z echo "Filtered matrix:" 2024-08-07T17:53:07.8076947Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" 2024-08-07T17:53:07.8078307Z  2024-08-07T17:53:07.8078591Z echo 2024-08-07T17:53:07.8078952Z echo "Is the current job unstable? False" 2024-08-07T17:53:07.8079396Z  2024-08-07T17:53:07.8079677Z echo 2024-08-07T17:53:07.8080003Z echo "Is keep-going label set? False" 2024-08-07T17:53:07.8080419Z  2024-08-07T17:53:07.8080692Z echo 2024-08-07T17:53:07.8080988Z echo "Renabled issues? " 2024-08-07T17:53:07.8088854Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:07.8089337Z env: 2024-08-07T17:53:07.8089616Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:07.8090046Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:07.8090504Z ##[endgroup] 2024-08-07T17:53:07.8114330Z Filtered matrix: 2024-08-07T17:53:07.8115799Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}]} 2024-08-07T17:53:07.8117216Z 2024-08-07T17:53:07.8117535Z Is the current job unstable? False 2024-08-07T17:53:07.8117826Z 2024-08-07T17:53:07.8117997Z Is keep-going label set? False 2024-08-07T17:53:07.8118239Z 2024-08-07T17:53:07.8118381Z Renabled issues? 2024-08-07T17:53:07.8181202Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-07T17:53:07.8181980Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-07T17:53:07.8189567Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:07.8190062Z env: 2024-08-07T17:53:07.8190345Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:07.8190785Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:07.8191270Z JOB_TIMEOUT: 360 2024-08-07T17:53:07.8191571Z ##[endgroup] 2024-08-07T17:53:07.8271177Z ##[group]Run set -x 2024-08-07T17:53:07.8271629Z set -x 2024-08-07T17:53:07.8271918Z  2024-08-07T17:53:07.8272255Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-07T17:53:07.8272761Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-07T17:53:07.8273257Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-07T17:53:07.8273720Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-07T17:53:07.8274114Z else 2024-08-07T17:53:07.8274433Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-07T17:53:07.8274842Z fi 2024-08-07T17:53:07.8275129Z  2024-08-07T17:53:07.8275547Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-07T17:53:07.8276211Z # TODO: Stop building test binaries as part of the build phase 2024-08-07T17:53:07.8276797Z # Used for GPU_FLAG since that doesn't play nice 2024-08-07T17:53:07.8277340Z # shellcheck disable=SC2086,SC2090 2024-08-07T17:53:07.8277769Z container_name=$(docker run \ 2024-08-07T17:53:07.8278178Z  ${GPU_FLAG:-} \ 2024-08-07T17:53:07.8278552Z  -e BUILD_ENVIRONMENT \ 2024-08-07T17:53:07.8278927Z  -e PR_NUMBER \ 2024-08-07T17:53:07.8279286Z  -e GITHUB_ACTIONS \ 2024-08-07T17:53:07.8279664Z  -e GITHUB_REPOSITORY \ 2024-08-07T17:53:07.8280038Z  -e GITHUB_WORKFLOW \ 2024-08-07T17:53:07.8280412Z  -e GITHUB_JOB \ 2024-08-07T17:53:07.8280769Z  -e GITHUB_RUN_ID \ 2024-08-07T17:53:07.8281124Z  -e GITHUB_RUN_NUMBER \ 2024-08-07T17:53:07.8281511Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-07T17:53:07.8281894Z  -e JOB_ID \ 2024-08-07T17:53:07.8282215Z  -e JOB_NAME \ 2024-08-07T17:53:07.8282561Z  -e BASE_SHA \ 2024-08-07T17:53:07.8282906Z  -e BRANCH \ 2024-08-07T17:53:07.8283222Z  -e SHA1 \ 2024-08-07T17:53:07.8283578Z  -e AWS_DEFAULT_REGION \ 2024-08-07T17:53:07.8283969Z  -e IN_WHEEL_TEST \ 2024-08-07T17:53:07.8284316Z  -e SHARD_NUMBER \ 2024-08-07T17:53:07.8284677Z  -e TEST_CONFIG \ 2024-08-07T17:53:07.8285043Z  -e NUM_TEST_SHARDS \ 2024-08-07T17:53:07.8285413Z  -e REENABLED_ISSUES \ 2024-08-07T17:53:07.8285804Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-07T17:53:07.8286213Z  -e VERBOSE_TEST_LOGS \ 2024-08-07T17:53:07.8286585Z  -e TEST_SHOWLOCALS \ 2024-08-07T17:53:07.8286964Z  -e NO_TEST_TIMEOUT \ 2024-08-07T17:53:07.8287331Z  -e NO_TD \ 2024-08-07T17:53:07.8287653Z  -e TD_DISTRIBUTED \ 2024-08-07T17:53:07.8288022Z  -e PR_LABELS \ 2024-08-07T17:53:07.8288409Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-07T17:53:07.8288826Z  -e SCCACHE_BUCKET \ 2024-08-07T17:53:07.8289206Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-07T17:53:07.8289772Z  -e XLA_CUDA \ 2024-08-07T17:53:07.8290150Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-07T17:53:07.8290617Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-07T17:53:07.8291089Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-07T17:53:07.8291544Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-07T17:53:07.8292002Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-07T17:53:07.8292410Z  -e DASHBOARD_TAG \ 2024-08-07T17:53:07.8292834Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-07T17:53:07.8293339Z  --security-opt seccomp=unconfined \ 2024-08-07T17:53:07.8293785Z  --cap-add=SYS_PTRACE \ 2024-08-07T17:53:07.8294158Z  --ipc=host \ 2024-08-07T17:53:07.8294515Z  --shm-size="${SHM_SIZE}" \ 2024-08-07T17:53:07.8294900Z  --tty \ 2024-08-07T17:53:07.8295206Z  --detach \ 2024-08-07T17:53:07.8295554Z  --name="${container_name}" \ 2024-08-07T17:53:07.8296066Z  --user jenkins \ 2024-08-07T17:53:07.8296517Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-07T17:53:07.8297025Z  -w /var/lib/jenkins/workspace \ 2024-08-07T17:53:07.8297439Z  "${DOCKER_IMAGE}" 2024-08-07T17:53:07.8297772Z ) 2024-08-07T17:53:07.8298153Z # Propagate download.pytorch.org IP to container 2024-08-07T17:53:07.8299250Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-07T17:53:07.8300101Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-07T17:53:07.8300921Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-07T17:53:07.8308201Z shell: /usr/bin/bash -e {0} 2024-08-07T17:53:07.8308550Z env: 2024-08-07T17:53:07.8308845Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:07.8309282Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:07.8309829Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2024-08-07T17:53:07.8310297Z PR_NUMBER: 131248 2024-08-07T17:53:07.8310644Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-07T17:53:07.8311030Z GITHUB_WORKFLOW: pull 2024-08-07T17:53:07.8311365Z GITHUB_JOB: test 2024-08-07T17:53:07.8311684Z GITHUB_RUN_ID: 10288745067 2024-08-07T17:53:07.8312033Z GITHUB_RUN_NUMBER: 234358 2024-08-07T17:53:07.8312389Z GITHUB_RUN_ATTEMPT: 1 2024-08-07T17:53:07.8312718Z JOB_ID: 28476145515 2024-08-07T17:53:07.8313282Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:53:07.8313949Z BRANCH: pull/131248 2024-08-07T17:53:07.8314315Z SHA1: 016588f53c6904b840aa56aa86f95460b4d9c996 2024-08-07T17:53:07.8314775Z BASE_SHA: 6ce09a9bb33e4011761558032e2165ad7b49fb68 2024-08-07T17:53:07.8315226Z TEST_CONFIG: distributed 2024-08-07T17:53:07.8315577Z SHARD_NUMBER: 3 2024-08-07T17:53:07.8315882Z NUM_TEST_SHARDS: 3 2024-08-07T17:53:07.8316215Z REENABLED_ISSUES: 2024-08-07T17:53:07.8316558Z CONTINUE_THROUGH_ERROR: False 2024-08-07T17:53:07.8316915Z VERBOSE_TEST_LOGS: False 2024-08-07T17:53:07.8317266Z TEST_SHOWLOCALS: False 2024-08-07T17:53:07.8317611Z NO_TEST_TIMEOUT: False 2024-08-07T17:53:07.8317927Z NO_TD: False 2024-08-07T17:53:07.8318233Z TD_DISTRIBUTED: False 2024-08-07T17:53:07.8318638Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-07T17:53:07.8319086Z SCCACHE_S3_KEY_PREFIX: pull 2024-08-07T17:53:07.8319443Z SHM_SIZE: 2g 2024-08-07T17:53:07.8320308Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:53:07.8321242Z XLA_CUDA: 2024-08-07T17:53:07.8321695Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-07T17:53:07.8322267Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-07T17:53:07.8322661Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-07T17:53:07.8323207Z DASHBOARD_TAG: 2024-08-07T17:53:07.8323527Z HUGGING_FACE_HUB_TOKEN: 2024-08-07T17:53:07.8323855Z ##[endgroup] 2024-08-07T17:53:07.8347135Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-08-07T17:53:07.8347608Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2024-08-07T17:53:07.8348087Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-07T17:53:07.8357588Z +++ nproc --ignore=2 2024-08-07T17:53:07.8372476Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10288745067 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:53:19.0959103Z + container_name=fd8fbd7c4999ec777ca8e9214dc76f6d75fb011239fd9300b6a357a13f0916b0 2024-08-07T17:53:19.0961080Z + grep download.pytorch.org /etc/hosts 2024-08-07T17:53:19.0963212Z + docker exec -i fd8fbd7c4999ec777ca8e9214dc76f6d75fb011239fd9300b6a357a13f0916b0 sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-07T17:53:19.1696746Z + echo DOCKER_CONTAINER_ID=fd8fbd7c4999ec777ca8e9214dc76f6d75fb011239fd9300b6a357a13f0916b0 2024-08-07T17:53:19.1700765Z ++ echo dist/torch-2.5.0a0+git016588f-cp310-cp310-linux_x86_64.whl 2024-08-07T17:53:19.1703435Z + docker exec -t fd8fbd7c4999ec777ca8e9214dc76f6d75fb011239fd9300b6a357a13f0916b0 sh -c 'pip install dist/torch-2.5.0a0+git016588f-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-07T17:53:19.6886231Z Processing ./dist/torch-2.5.0a0+git016588f-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git016588f) 2024-08-07T17:53:20.8127926Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (3.13.1) 2024-08-07T17:53:20.8134328Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (4.12.2) 2024-08-07T17:53:20.8139050Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (2.8.8) 2024-08-07T17:53:20.8144045Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (3.1.4) 2024-08-07T17:53:20.8148478Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (2024.6.1) 2024-08-07T17:53:20.8161328Z Requirement already satisfied: sympy>=1.13.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (1.13.1) 2024-08-07T17:53:20.8200973Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (3.3.0) 2024-08-07T17:53:20.8279048Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (1.21.2) 2024-08-07T17:53:20.8324716Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.0->torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (1.3.0) 2024-08-07T17:53:20.9579936Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (2.1.5) 2024-08-07T17:53:22.0808817Z Installing collected packages: torch 2024-08-07T17:53:34.6525849Z Successfully installed torch-2.5.0a0+git016588f 2024-08-07T17:53:34.7199430Z ++ dirname .ci/pytorch/test.sh 2024-08-07T17:53:34.7205460Z + source .ci/pytorch/common.sh 2024-08-07T17:53:34.7208955Z +++ dirname .ci/pytorch/common.sh 2024-08-07T17:53:34.7216556Z ++ source .ci/pytorch/common_utils.sh 2024-08-07T17:53:34.7218563Z +++ declare -f -t trap_add 2024-08-07T17:53:34.7225577Z ++ set -ex 2024-08-07T17:53:34.7228413Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-08-07T17:53:34.7229057Z ++ BUILD_TEST_LIBTORCH=0 2024-08-07T17:53:34.7229726Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-08-07T17:53:34.7230647Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-07T17:53:34.7241646Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-07T17:53:34.7242081Z + trap_add cleanup_workspace EXIT 2024-08-07T17:53:34.7242485Z + trap_add_cmd=cleanup_workspace 2024-08-07T17:53:34.7242853Z + shift 2024-08-07T17:53:34.7243132Z + for trap_add_name in "$@" 2024-08-07T17:53:34.7250530Z +++ trap -p EXIT 2024-08-07T17:53:34.7253510Z ++ eval 'extract_trap_cmd ' 2024-08-07T17:53:34.7253882Z +++ extract_trap_cmd 2024-08-07T17:53:34.7254366Z +++ printf '%s\n' '' 2024-08-07T17:53:34.7254982Z ++ printf '%s\n' cleanup_workspace 2024-08-07T17:53:34.7255923Z + trap -- ' 2024-08-07T17:53:34.7256239Z cleanup_workspace' EXIT 2024-08-07T17:53:34.7256649Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-07T17:53:35.2631588Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-07T17:53:35.2648299Z + echo 'Environment variables:' 2024-08-07T17:53:35.2649134Z Environment variables: 2024-08-07T17:53:35.2649727Z + env 2024-08-07T17:53:35.2658046Z INSTALLED_DB=yes 2024-08-07T17:53:35.2659016Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-08-07T17:53:35.2659810Z NVIDIA_VISIBLE_DEVICES=all 2024-08-07T17:53:35.2660543Z NV_NVML_DEV_VERSION=11.8.86-1 2024-08-07T17:53:35.2661262Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-07T17:53:35.2661849Z CONTINUE_THROUGH_ERROR=False 2024-08-07T17:53:35.2662266Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-08-07T17:53:35.2662749Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-08-07T17:53:35.2663232Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-08-07T17:53:35.2663676Z HOSTNAME=fd8fbd7c4999 2024-08-07T17:53:35.2664400Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_fa241ae0-bfb8-49cb-888a-1f8dd2baff42 2024-08-07T17:53:35.2665456Z GITHUB_ACTION=__self 2024-08-07T17:53:35.2666225Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-07T17:53:35.2668306Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-08-07T17:53:35.2670317Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-08-07T17:53:35.2670790Z NV_NVTX_VERSION=11.8.86-1 2024-08-07T17:53:35.2671129Z GITHUB_RUN_NUMBER=234358 2024-08-07T17:53:35.2671482Z TEST_CONFIG=distributed 2024-08-07T17:53:35.2671844Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-07T17:53:35.2672253Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-07T17:53:35.2672687Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-08-07T17:53:35.2673098Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-08-07T17:53:35.2674024Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-08-07T17:53:35.2674793Z GITHUB_TRIGGERING_ACTOR=zdevito 2024-08-07T17:53:35.2675533Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-07T17:53:35.2675975Z GITHUB_REF_TYPE=branch 2024-08-07T17:53:35.2676322Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-07T17:53:35.2676684Z NCCL_VERSION=2.16.2-1 2024-08-07T17:53:35.2677046Z BASE_SHA=6ce09a9bb33e4011761558032e2165ad7b49fb68 2024-08-07T17:53:35.2677471Z XLA_CUDA= 2024-08-07T17:53:35.2677766Z HUGGING_FACE_HUB_TOKEN= 2024-08-07T17:53:35.2678257Z *** 2024-08-07T17:53:35.2678551Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-07T17:53:35.2678940Z GITHUB_REPOSITORY_ID=65600975 2024-08-07T17:53:35.2679290Z GITHUB_ACTIONS=true 2024-08-07T17:53:35.2679621Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:35.2680040Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-08-07T17:53:35.2680544Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-08-07T17:53:35.2680973Z SHA1=016588f53c6904b840aa56aa86f95460b4d9c996 2024-08-07T17:53:35.2681425Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-07T17:53:35.2682053Z GITHUB_SHA=f779f6b7738020e244184bded4026b37de3f9f24 2024-08-07T17:53:35.2682720Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/131248/merge 2024-08-07T17:53:35.2683331Z UCC_HOME=/usr 2024-08-07T17:53:35.2683652Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-08-07T17:53:35.2684031Z VERBOSE_TEST_LOGS=False 2024-08-07T17:53:35.2684378Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-07T17:53:35.2684763Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-08-07T17:53:35.2685215Z GITHUB_REF=refs/pull/131248/merge 2024-08-07T17:53:35.2685607Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-08-07T17:53:35.2685960Z SHARD_NUMBER=3 2024-08-07T17:53:35.2686279Z GITHUB_REF_PROTECTED=false 2024-08-07T17:53:35.2686636Z HOME=/var/lib/jenkins 2024-08-07T17:53:35.2686986Z GITHUB_API_URL=https://api.github.com 2024-08-07T17:53:35.2687410Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-07T17:53:35.2687858Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-07T17:53:35.2688307Z SCCACHE_S3_KEY_PREFIX=pull 2024-08-07T17:53:35.2688667Z CUDA_VERSION=11.8.0 2024-08-07T17:53:35.2689039Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-08-07T17:53:35.2689455Z NUM_TEST_SHARDS=3 2024-08-07T17:53:35.2689765Z UCX_HOME=/usr 2024-08-07T17:53:35.2690208Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-08-07T17:53:35.2691124Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_fa241ae0-bfb8-49cb-888a-1f8dd2baff42 2024-08-07T17:53:35.2692175Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:53:35.2693191Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_fa241ae0-bfb8-49cb-888a-1f8dd2baff42 2024-08-07T17:53:35.2694155Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-07T17:53:35.2694793Z GITHUB_EVENT_NAME=pull_request 2024-08-07T17:53:35.2695163Z DASHBOARD_TAG= 2024-08-07T17:53:35.2695475Z GITHUB_RUN_ID=10288745067 2024-08-07T17:53:35.2695884Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-08-07T17:53:35.2696361Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-08-07T17:53:35.2697222Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_fa241ae0-bfb8-49cb-888a-1f8dd2baff42 2024-08-07T17:53:35.2698051Z GITHUB_ACTOR=zdevito 2024-08-07T17:53:35.2698634Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-08-07T17:53:35.2699000Z PR_NUMBER=131248 2024-08-07T17:53:35.2699323Z GITHUB_RUN_ATTEMPT=1 2024-08-07T17:53:35.2699667Z ANACONDA_PYTHON_VERSION=3.10 2024-08-07T17:53:35.2700090Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-07T17:53:35.2700541Z TERM=xterm 2024-08-07T17:53:35.2700853Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-08-07T17:53:35.2701237Z INSTALLED_VISION=yes 2024-08-07T17:53:35.2701563Z BRANCH=pull/131248 2024-08-07T17:53:35.2701891Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-07T17:53:35.2702265Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-07T17:53:35.2702897Z CUDA_PATH=/usr/local/cuda 2024-08-07T17:53:35.2703547Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-07T17:53:35.2704243Z GITHUB_SERVER_URL=https://github.com 2024-08-07T17:53:35.2704706Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-07T17:53:35.2705156Z REENABLED_ISSUES= 2024-08-07T17:53:35.2705447Z SHLVL=1 2024-08-07T17:53:35.2705722Z MAX_JOBS=30 2024-08-07T17:53:35.2706024Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-08-07T17:53:35.2706364Z NVARCH=x86_64 2024-08-07T17:53:35.2706667Z GITHUB_ACTOR_ID=370202 2024-08-07T17:53:35.2707091Z GITHUB_WORKFLOW_SHA=f779f6b7738020e244184bded4026b37de3f9f24 2024-08-07T17:53:35.2707569Z GITHUB_REF_NAME=131248/merge 2024-08-07T17:53:35.2707956Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-08-07T17:53:35.2708522Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-07T17:53:35.2709056Z GITHUB_JOB=test 2024-08-07T17:53:35.2709401Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-08-07T17:53:35.2710080Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-07T17:53:35.2710577Z NO_TEST_TIMEOUT=False 2024-08-07T17:53:35.2710912Z TD_DISTRIBUTED=False 2024-08-07T17:53:35.2711251Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-08-07T17:53:35.2711681Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-07T17:53:35.2712078Z NV_NVPROF_VERSION=11.8.87-1 2024-08-07T17:53:35.2712429Z GITHUB_RETENTION_DAYS=90 2024-08-07T17:53:35.2712787Z OPENSSL_DIR=/opt/openssl 2024-08-07T17:53:35.2713144Z GITHUB_ACTION_REPOSITORY= 2024-08-07T17:53:35.2714066Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-07T17:53:35.2715053Z GITHUB_BASE_REF=gh/zdevito/267/base 2024-08-07T17:53:35.2715463Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-07T17:53:35.2715810Z CI=true 2024-08-07T17:53:35.2716115Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-08-07T17:53:35.2716529Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-07T17:53:35.2716888Z JOB_ID=28476145515 2024-08-07T17:53:35.2717208Z INSTALLED_PROTOBUF=yes 2024-08-07T17:53:35.2717557Z GITHUB_HEAD_REF=gh/zdevito/267/head 2024-08-07T17:53:35.2717928Z GITHUB_ACTION_REF= 2024-08-07T17:53:35.2718309Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-07T17:53:35.2718771Z TEST_SHOWLOCALS=False 2024-08-07T17:53:35.2719089Z GITHUB_WORKFLOW=pull 2024-08-07T17:53:35.2719434Z DEBIAN_FRONTEND=noninteractive 2024-08-07T17:53:35.2720210Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_fa241ae0-bfb8-49cb-888a-1f8dd2baff42 2024-08-07T17:53:35.2720978Z NO_TD=False 2024-08-07T17:53:35.2721289Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-07T17:53:35.2721650Z _=/usr/bin/env 2024-08-07T17:53:35.2722039Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-07T17:53:35.2873056Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-07T17:53:35.2874473Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-07T17:53:35.2875941Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-07T17:53:35.2877066Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-07T17:53:35.2877634Z + BUILD_DIR=build 2024-08-07T17:53:35.2877945Z + BUILD_RENAMED_DIR=build_renamed 2024-08-07T17:53:35.2878327Z + BUILD_BIN_DIR=build/bin 2024-08-07T17:53:35.2878668Z + SHARD_NUMBER=3 2024-08-07T17:53:35.2878960Z + NUM_TEST_SHARDS=3 2024-08-07T17:53:35.2879281Z + export VALGRIND=ON 2024-08-07T17:53:35.2879605Z + VALGRIND=ON 2024-08-07T17:53:35.2880153Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2024-08-07T17:53:35.2881105Z + [[ 0 == \1 ]] 2024-08-07T17:53:35.2881523Z + [[ False == \1 ]] 2024-08-07T17:53:35.2881875Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2024-08-07T17:53:35.2882343Z ++ realpath build/custom_test_artifacts 2024-08-07T17:53:35.2889087Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-07T17:53:35.2890328Z + [[ -n '' ]] 2024-08-07T17:53:35.2890651Z + echo 'Environment variables' 2024-08-07T17:53:35.2891028Z Environment variables 2024-08-07T17:53:35.2891332Z + env 2024-08-07T17:53:35.2896463Z INSTALLED_DB=yes 2024-08-07T17:53:35.2897115Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-08-07T17:53:35.2897914Z NVIDIA_VISIBLE_DEVICES=all 2024-08-07T17:53:35.2898736Z NV_NVML_DEV_VERSION=11.8.86-1 2024-08-07T17:53:35.2899381Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-07T17:53:35.2900524Z CONTINUE_THROUGH_ERROR=False 2024-08-07T17:53:35.2901180Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-08-07T17:53:35.2901662Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-08-07T17:53:35.2902146Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-08-07T17:53:35.2902593Z HOSTNAME=fd8fbd7c4999 2024-08-07T17:53:35.2903476Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_fa241ae0-bfb8-49cb-888a-1f8dd2baff42 2024-08-07T17:53:35.2905451Z GITHUB_ACTION=__self 2024-08-07T17:53:35.2905853Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-07T17:53:35.2908989Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-08-07T17:53:35.2910975Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-08-07T17:53:35.2911431Z NV_NVTX_VERSION=11.8.86-1 2024-08-07T17:53:35.2911791Z GITHUB_RUN_NUMBER=234358 2024-08-07T17:53:35.2912144Z TEST_CONFIG=distributed 2024-08-07T17:53:35.2912487Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-07T17:53:35.2912910Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-07T17:53:35.2913336Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-08-07T17:53:35.2913748Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-08-07T17:53:35.2914142Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-08-07T17:53:35.2914518Z GITHUB_TRIGGERING_ACTOR=zdevito 2024-08-07T17:53:35.2914933Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-07T17:53:35.2915383Z GITHUB_REF_TYPE=branch 2024-08-07T17:53:35.2915727Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-07T17:53:35.2916069Z NCCL_VERSION=2.16.2-1 2024-08-07T17:53:35.2916443Z BASE_SHA=6ce09a9bb33e4011761558032e2165ad7b49fb68 2024-08-07T17:53:35.2916869Z XLA_CUDA= 2024-08-07T17:53:35.2917144Z HUGGING_FACE_HUB_TOKEN= 2024-08-07T17:53:35.2917565Z *** 2024-08-07T17:53:35.2917860Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-07T17:53:35.2918231Z GITHUB_REPOSITORY_ID=65600975 2024-08-07T17:53:35.2918594Z GITHUB_ACTIONS=true 2024-08-07T17:53:35.2918927Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:35.2919328Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-08-07T17:53:35.2919793Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-08-07T17:53:35.2920260Z SHA1=016588f53c6904b840aa56aa86f95460b4d9c996 2024-08-07T17:53:35.2920688Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-07T17:53:35.2921144Z GITHUB_SHA=f779f6b7738020e244184bded4026b37de3f9f24 2024-08-07T17:53:35.2921804Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/131248/merge 2024-08-07T17:53:35.2922396Z UCC_HOME=/usr 2024-08-07T17:53:35.2922711Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-08-07T17:53:35.2923106Z VERBOSE_TEST_LOGS=False 2024-08-07T17:53:35.2923433Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-07T17:53:35.2923841Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-08-07T17:53:35.2924298Z GITHUB_REF=refs/pull/131248/merge 2024-08-07T17:53:35.2924672Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-08-07T17:53:35.2925041Z SHARD_NUMBER=3 2024-08-07T17:53:35.2925361Z GITHUB_REF_PROTECTED=false 2024-08-07T17:53:35.2925701Z HOME=/var/lib/jenkins 2024-08-07T17:53:35.2926066Z GITHUB_API_URL=https://api.github.com 2024-08-07T17:53:35.2926480Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-07T17:53:35.2927097Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-07T17:53:35.2927590Z SCCACHE_S3_KEY_PREFIX=pull 2024-08-07T17:53:35.2927944Z CUDA_VERSION=11.8.0 2024-08-07T17:53:35.2928294Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-08-07T17:53:35.2928726Z NUM_TEST_SHARDS=3 2024-08-07T17:53:35.2929038Z UCX_HOME=/usr 2024-08-07T17:53:35.2929463Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-08-07T17:53:35.2930393Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_fa241ae0-bfb8-49cb-888a-1f8dd2baff42 2024-08-07T17:53:35.2931439Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:53:35.2932442Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_fa241ae0-bfb8-49cb-888a-1f8dd2baff42 2024-08-07T17:53:35.2933423Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-07T17:53:35.2934064Z GITHUB_EVENT_NAME=pull_request 2024-08-07T17:53:35.2934527Z DASHBOARD_TAG= 2024-08-07T17:53:35.2934833Z GITHUB_RUN_ID=10288745067 2024-08-07T17:53:35.2935235Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-08-07T17:53:35.2935697Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-08-07T17:53:35.2936547Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_fa241ae0-bfb8-49cb-888a-1f8dd2baff42 2024-08-07T17:53:35.2937388Z GITHUB_ACTOR=zdevito 2024-08-07T17:53:35.2937730Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-08-07T17:53:35.2938093Z PR_NUMBER=131248 2024-08-07T17:53:35.2938655Z GITHUB_RUN_ATTEMPT=1 2024-08-07T17:53:35.2938965Z VALGRIND=ON 2024-08-07T17:53:35.2939271Z ANACONDA_PYTHON_VERSION=3.10 2024-08-07T17:53:35.2939702Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-07T17:53:35.2940134Z TERM=xterm 2024-08-07T17:53:35.2940449Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-08-07T17:53:35.2940850Z INSTALLED_VISION=yes 2024-08-07T17:53:35.2941171Z BRANCH=pull/131248 2024-08-07T17:53:35.2941505Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-07T17:53:35.2941900Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-07T17:53:35.2942298Z CUDA_PATH=/usr/local/cuda 2024-08-07T17:53:35.2942951Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-07T17:53:35.2943669Z GITHUB_SERVER_URL=https://github.com 2024-08-07T17:53:35.2944111Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-07T17:53:35.2944556Z REENABLED_ISSUES= 2024-08-07T17:53:35.2944866Z SHLVL=1 2024-08-07T17:53:35.2945129Z MAX_JOBS=30 2024-08-07T17:53:35.2945426Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-08-07T17:53:35.2945781Z NVARCH=x86_64 2024-08-07T17:53:35.2946071Z GITHUB_ACTOR_ID=370202 2024-08-07T17:53:35.2946498Z GITHUB_WORKFLOW_SHA=f779f6b7738020e244184bded4026b37de3f9f24 2024-08-07T17:53:35.2946995Z GITHUB_REF_NAME=131248/merge 2024-08-07T17:53:35.2947369Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-08-07T17:53:35.2947936Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-07T17:53:35.2948500Z GITHUB_JOB=test 2024-08-07T17:53:35.2948834Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-08-07T17:53:35.2949356Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-07T17:53:35.2949907Z NO_TEST_TIMEOUT=False 2024-08-07T17:53:35.2950226Z TD_DISTRIBUTED=False 2024-08-07T17:53:35.2950581Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-08-07T17:53:35.2950991Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-07T17:53:35.2951385Z NV_NVPROF_VERSION=11.8.87-1 2024-08-07T17:53:35.2951751Z GITHUB_RETENTION_DAYS=90 2024-08-07T17:53:35.2952089Z OPENSSL_DIR=/opt/openssl 2024-08-07T17:53:35.2952450Z GITHUB_ACTION_REPOSITORY= 2024-08-07T17:53:35.2953405Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-07T17:53:35.2954375Z GITHUB_BASE_REF=gh/zdevito/267/base 2024-08-07T17:53:35.2954787Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-07T17:53:35.2955305Z CI=true 2024-08-07T17:53:35.2955599Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-08-07T17:53:35.2956009Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-07T17:53:35.2956380Z JOB_ID=28476145515 2024-08-07T17:53:35.2956683Z INSTALLED_PROTOBUF=yes 2024-08-07T17:53:35.2957045Z GITHUB_HEAD_REF=gh/zdevito/267/head 2024-08-07T17:53:35.2957710Z GITHUB_ACTION_REF= 2024-08-07T17:53:35.2958078Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-07T17:53:35.2958533Z TEST_SHOWLOCALS=False 2024-08-07T17:53:35.2958863Z GITHUB_WORKFLOW=pull 2024-08-07T17:53:35.2959193Z DEBIAN_FRONTEND=noninteractive 2024-08-07T17:53:35.2959966Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_fa241ae0-bfb8-49cb-888a-1f8dd2baff42 2024-08-07T17:53:35.2960749Z NO_TD=False 2024-08-07T17:53:35.2961042Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-07T17:53:35.2961406Z _=/usr/bin/env 2024-08-07T17:53:35.2961714Z + echo 'Testing pytorch' 2024-08-07T17:53:35.2962041Z Testing pytorch 2024-08-07T17:53:35.2962374Z + export LANG=C.UTF-8 2024-08-07T17:53:35.2962823Z + LANG=C.UTF-8 2024-08-07T17:53:35.2963129Z + PR_NUMBER=131248 2024-08-07T17:53:35.2963505Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-08-07T17:53:35.2963927Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-07T17:53:35.2964377Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-08-07T17:53:35.2964834Z + [[ distributed == \s\l\o\w ]] 2024-08-07T17:53:35.2965339Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2024-08-07T17:53:35.2965864Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-08-07T17:53:35.2966345Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-07T17:53:35.2966793Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-07T17:53:35.2967194Z + [[ distributed == *crossref* ]] 2024-08-07T17:53:35.2967623Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-08-07T17:53:35.2968111Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-08-07T17:53:35.2968808Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-08-07T17:53:35.2969310Z + pip_install --user ninja==1.10.2 2024-08-07T17:53:35.2969772Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-07T17:53:35.8067825Z Collecting ninja==1.10.2 2024-08-07T17:53:35.8263402Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-07T17:53:35.8438623Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-07T17:53:37.0331962Z Installing collected packages: ninja 2024-08-07T17:53:37.0427766Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-07T17:53:37.0428930Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-07T17:53:37.0480413Z Successfully installed ninja-1.10.2 2024-08-07T17:53:37.1212754Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-07T17:53:37.1214692Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-07T17:53:37.1215860Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-08-07T17:53:37.1216337Z + install_tlparse 2024-08-07T17:53:37.1216690Z + pip_install --user tlparse==0.3.7 2024-08-07T17:53:37.1217152Z + pip install --progress-bar off --user tlparse==0.3.7 2024-08-07T17:53:37.6256059Z Collecting tlparse==0.3.7 2024-08-07T17:53:37.6455773Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-08-07T17:53:37.6559696Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-07T17:53:38.8863650Z Installing collected packages: tlparse 2024-08-07T17:53:38.9354325Z Successfully installed tlparse-0.3.7 2024-08-07T17:53:39.0021756Z ++ python -m site --user-base 2024-08-07T17:53:39.0243358Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-07T17:53:39.0244948Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2024-08-07T17:53:39.0245775Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2024-08-07T17:53:39.0246311Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-08-07T17:53:39.0246988Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2024-08-07T17:53:39.0247837Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2024-08-07T17:53:39.0248780Z + cd test 2024-08-07T17:53:39.0249481Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-07T17:53:41.1370802Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-07T17:53:41.1371728Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-07T17:53:41.1375731Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-07T17:53:41.1376122Z + [[ distributed == *dynamo_eager* ]] 2024-08-07T17:53:41.1376549Z + [[ distributed == *aot_eager* ]] 2024-08-07T17:53:41.1376955Z + [[ distributed == *aot_inductor* ]] 2024-08-07T17:53:41.1377345Z + [[ distributed == *inductor* ]] 2024-08-07T17:53:41.1377739Z + [[ distributed == *dynamic* ]] 2024-08-07T17:53:41.1378122Z + [[ distributed == *cpu* ]] 2024-08-07T17:53:41.1379161Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-07T17:53:41.1417806Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-08-07T17:53:41.1419000Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2024-08-07T17:53:41.1421832Z + cd test 2024-08-07T17:53:41.1422410Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-07T17:53:42.9979713Z PyTorch built with: 2024-08-07T17:53:42.9980130Z - GCC 9.4 2024-08-07T17:53:42.9980462Z - C++ Version: 201703 2024-08-07T17:53:42.9981189Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-07T17:53:42.9982068Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-07T17:53:42.9982608Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-07T17:53:42.9983049Z - LAPACK is enabled (usually provided by MKL) 2024-08-07T17:53:42.9983483Z - NNPACK is enabled 2024-08-07T17:53:42.9983816Z - CPU capability usage: AVX2 2024-08-07T17:53:42.9984190Z - CUDA Runtime 11.8 2024-08-07T17:53:42.9984632Z - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52 2024-08-07T17:53:42.9985111Z - CuDNN 90.1 2024-08-07T17:53:42.9985415Z - Magma 2.6.1 2024-08-07T17:53:42.9991056Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-07T17:53:42.9997316Z 2024-08-07T17:53:43.2450250Z + cd test 2024-08-07T17:53:43.2450762Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-07T17:53:45.0832015Z ATen/Parallel: 2024-08-07T17:53:45.0832440Z at::get_num_threads() : 16 2024-08-07T17:53:45.0832848Z at::get_num_interop_threads() : 16 2024-08-07T17:53:45.0833265Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-07T17:53:45.0833642Z omp_get_max_threads() : 16 2024-08-07T17:53:45.0834345Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-07T17:53:45.0835082Z mkl_get_max_threads() : 16 2024-08-07T17:53:45.0835563Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-07T17:53:45.0836136Z std::thread::hardware_concurrency() : 32 2024-08-07T17:53:45.0836558Z Environment variables: 2024-08-07T17:53:45.0836892Z OMP_NUM_THREADS : [not set] 2024-08-07T17:53:45.0837261Z MKL_NUM_THREADS : [not set] 2024-08-07T17:53:45.0837678Z ATen parallel backend: OpenMP 2024-08-07T17:53:45.0838284Z 2024-08-07T17:53:45.3289142Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-08-07T17:53:45.3289677Z + [[ distributed == *backward* ]] 2024-08-07T17:53:45.3290103Z + [[ distributed == *xla* ]] 2024-08-07T17:53:45.3290473Z + [[ distributed == *executorch* ]] 2024-08-07T17:53:45.3290890Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-07T17:53:45.3291372Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-08-07T17:53:45.3291834Z + [[ distributed == distributed ]] 2024-08-07T17:53:45.3292223Z + test_distributed 2024-08-07T17:53:45.3292574Z + echo 'Testing distributed python tests' 2024-08-07T17:53:45.3292993Z Testing distributed python tests 2024-08-07T17:53:45.4564583Z + python test/run_test.py --distributed-tests --shard 3 3 --verbose 2024-08-07T17:53:45.4565786Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-07T17:53:45.4566837Z import pkg_resources 2024-08-07T17:53:49.8472816Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-08-07T17:53:49.9019244Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-07T17:53:50.0125881Z Ignoring disabled issues: [''] 2024-08-07T17:53:50.0265085Z Found test times from artifacts 2024-08-07T17:53:50.0790833Z Found test times from artifacts 2024-08-07T17:53:50.0807209Z Running 25% of tests based on TD 2024-08-07T17:53:50.0932138Z Running parallel tests on 2 processes 2024-08-07T17:53:50.0933053Z Name: tests to run (est. time: 11.98min) 2024-08-07T17:53:50.0933474Z Serial tests (14): 2024-08-07T17:53:50.0933863Z distributed/_tensor/debug/test_op_coverage 1/1 2024-08-07T17:53:50.0934342Z distributed/checkpoint/test_nested_dict 1/1 2024-08-07T17:53:50.0934842Z distributed/pipelining/test_transformer 1/1 2024-08-07T17:53:50.0935354Z distributed/fsdp/test_fsdp_core 1/2 2024-08-07T17:53:50.0935772Z distributed/_tensor/test_matrix_ops 1/1 2024-08-07T17:53:50.0936218Z distributed/_tensor/test_random_ops 1/1 2024-08-07T17:53:50.0936758Z distributed/_tensor/experimental/test_register_sharding 1/1 2024-08-07T17:53:50.0937318Z distributed/tensor/parallel/test_tp_examples 1/1 2024-08-07T17:53:50.0937813Z distributed/_tensor/test_common_rules 1/1 2024-08-07T17:53:50.0938547Z distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 2024-08-07T17:53:50.0939052Z distributed/checkpoint/test_dedup_tensors 1/1 2024-08-07T17:53:50.0939532Z distributed/_tools/test_memory_tracker 1/1 2024-08-07T17:53:50.0939980Z distributed/_tensor/test_init 1/1 2024-08-07T17:53:50.0940397Z distributed/_tensor/test_redistribute 1/1 2024-08-07T17:53:50.0940829Z Parallel tests (0): 2024-08-07T17:53:50.0941210Z Name: excluded (est. time: 72.59min) 2024-08-07T17:53:50.0941582Z Serial tests (51): 2024-08-07T17:53:50.0942251Z distributed/test_distributed_spawn 3/9 2024-08-07T17:53:50.0942700Z distributed/test_distributed_spawn 6/9 2024-08-07T17:53:50.0943128Z distributed/test_distributed_spawn 9/9 2024-08-07T17:53:50.0943559Z distributed/test_c10d_nccl 2/2 2024-08-07T17:53:50.0943980Z distributed/fsdp/test_fsdp_state_dict 2/2 2024-08-07T17:53:50.0944439Z distributed/fsdp/test_fsdp_mixed_precision 1/1 2024-08-07T17:53:50.0944902Z distributed/test_c10d_ucc 1/1 2024-08-07T17:53:50.0945347Z distributed/fsdp/test_fsdp_freezing_weights 1/1 2024-08-07T17:53:50.0945816Z distributed/fsdp/test_fsdp_comm_hooks 1/1 2024-08-07T17:53:50.0946290Z distributed/fsdp/test_fsdp_use_orig_params 1/1 2024-08-07T17:53:50.0946832Z distributed/_composable/test_replicate_with_compiler 1/1 2024-08-07T17:53:50.0947357Z distributed/fsdp/test_fsdp_unshard_params 1/1 2024-08-07T17:53:50.0947825Z distributed/fsdp/test_fsdp_misc 1/1 2024-08-07T17:53:50.0948317Z distributed/checkpoint/e2e/test_e2e_save_and_load 1/1 2024-08-07T17:53:50.0948954Z distributed/fsdp/test_fsdp_comm 1/1 2024-08-07T17:53:50.0949400Z distributed/test_c10d_spawn_nccl 1/1 2024-08-07T17:53:50.0949893Z distributed/tensor/parallel/test_fsdp_2d_parallel 1/1 2024-08-07T17:53:50.0950433Z distributed/tensor/parallel/test_parallelize_api 1/1 2024-08-07T17:53:50.0950917Z distributed/launcher/test_run 1/1 2024-08-07T17:53:50.0951334Z distributed/fsdp/test_fsdp_meta 1/1 2024-08-07T17:53:50.0951761Z distributed/_composable/test_replicate 1/1 2024-08-07T17:53:50.0952221Z distributed/fsdp/test_fsdp_fine_tune 1/1 2024-08-07T17:53:50.0952703Z distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 2024-08-07T17:53:50.0953293Z distributed/_composable/test_composability/test_2d_composability 1/1 2024-08-07T17:53:50.0953981Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 2024-08-07T17:53:50.0954535Z distributed/_tensor/test_utils 1/1 2024-08-07T17:53:50.0955005Z distributed/_tensor/experimental/test_local_map 1/1 2024-08-07T17:53:50.0955490Z distributed/test_c10d_logger 1/1 2024-08-07T17:53:50.0955946Z distributed/checkpoint/test_hsdp_checkpoint 1/1 2024-08-07T17:53:50.0956433Z distributed/fsdp/test_fsdp_tp_integration 1/1 2024-08-07T17:53:50.0956966Z distributed/_shard/sharding_spec/test_sharding_spec 1/1 2024-08-07T17:53:50.0957562Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 2024-08-07T17:53:50.0958151Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 2024-08-07T17:53:50.0958759Z distributed/_composable/fsdp/test_fully_shard_extensions 1/1 2024-08-07T17:53:50.0959436Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 1/1 2024-08-07T17:53:50.0960004Z distributed/test_c10d_ops_nccl 1/1 2024-08-07T17:53:50.0960425Z distributed/_shard/test_sharder 1/1 2024-08-07T17:53:50.0960868Z distributed/test_inductor_collectives 1/1 2024-08-07T17:53:50.0961387Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 1/1 2024-08-07T17:53:50.0961950Z distributed/elastic/test_control_plane 1/1 2024-08-07T17:53:50.0962453Z distributed/fsdp/test_fsdp_multiple_wrapping 1/1 2024-08-07T17:53:50.0962917Z distributed/fsdp/test_fsdp_uneven 1/1 2024-08-07T17:53:50.0963402Z distributed/checkpoint/test_dtensor_checkpoint 1/1 2024-08-07T17:53:50.0963886Z distributed/rpc/test_share_memory 1/1 2024-08-07T17:53:50.0964359Z distributed/tensor/parallel/test_ddp_2d_parallel 1/1 2024-08-07T17:53:50.0964871Z distributed/_tensor/test_pointwise_ops 1/1 2024-08-07T17:53:50.0965335Z distributed/_tensor/test_op_strategy 1/1 2024-08-07T17:53:50.0965756Z distributed/fsdp/test_utils 1/1 2024-08-07T17:53:50.0966229Z distributed/elastic/utils/util_test 1/1 2024-08-07T17:53:50.0966679Z distributed/nn/jit/test_instantiator 1/1 2024-08-07T17:53:50.0967161Z distributed/fsdp/test_distributed_checkpoint 1/1 2024-08-07T17:53:50.0967621Z Parallel tests (0): 2024-08-07T17:53:50.0968233Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-08-07T17:53:50.1002351Z Running distributed/_tensor/debug/test_op_coverage 1/1 ... [2024-08-07 17:53:50.099914] 2024-08-07T17:53:50.1007990Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/debug/test_op_coverage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 17:53:50.100410] 2024-08-07T17:53:54.1736774Z 2024-08-07T17:53:54.1738184Z distributed/_tensor/debug/test_op_coverage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.debug.test_op_coverage_1.1_6c844a9ef45fa185_.log 2024-08-07T17:53:54.1740361Z Running 1 items in this shard: test/distributed/_tensor/debug/test_op_coverage.py::TestOpCoverage::test_trace_with_inductor_decomp 2024-08-07T17:53:54.1741043Z 2024-08-07T17:53:54.1745728Z Running distributed/checkpoint/test_nested_dict 1/1 ... [2024-08-07 17:53:54.174246] 2024-08-07T17:53:54.1750023Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_nested_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 17:53:54.174667] 2024-08-07T17:53:57.8462731Z 2024-08-07T17:53:57.8464135Z distributed/checkpoint/test_nested_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_nested_dict_1.1_13ef616b1c9041f1_.log 2024-08-07T17:53:57.8466039Z Running 2 items in this shard: test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_flattening_round_trip, test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_mapping 2024-08-07T17:53:57.8467144Z 2024-08-07T17:53:57.8469940Z Running distributed/pipelining/test_transformer 1/1 ... [2024-08-07 17:53:57.846650] 2024-08-07T17:53:57.8474837Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_transformer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 17:53:57.847094] 2024-08-07T17:54:02.8212763Z 2024-08-07T17:54:02.8214248Z distributed/pipelining/test_transformer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_transformer_1.1_4fc060f0ab3e5e19_.log 2024-08-07T17:54:02.8215736Z Running 1 items in this shard: test/distributed/pipelining/test_transformer.py::TransformerTests::test_ir 2024-08-07T17:54:02.8216336Z 2024-08-07T17:54:02.8219592Z Running distributed/fsdp/test_fsdp_core 1/2 ... [2024-08-07 17:54:02.821610] 2024-08-07T17:54:03.1628481Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '--shard-id=1', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 17:54:03.162292] 2024-08-07T18:00:44.4734033Z 2024-08-07T18:00:44.4737729Z distributed/fsdp/test_fsdp_core 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_core_1.2_74957132fa890310_.log 2024-08-07T18:00:44.4750824Z Running 22 items in this shard: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_optim_step_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_optim_step_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_with_delay_before_free_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_with_delay_before_free_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_with_delay_before_free_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_always_wrap_model_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_transformer_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_transformer_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_transformer_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParamInit::test_param_change_after_init_mixed_precision_False, test/distributed/fsdp/test_fsdp_core.py::TestParamInit::test_param_change_after_init_mixed_precision_True, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_pre_backward_hook_registration_cuda_first_False, test/distributed/fsdp/test_fsdp_core.py::TestNoGrad::test_transformer_no_grad_mixed_precision_False, test/distributed/fsdp/test_fsdp_core.py::TestNoGrad::test_transformer_no_grad_mixed_precision_True, test/distributed/fsdp/test_fsdp_core.py::TestAutograd::test_unshard_params_as_tensors 2024-08-07T18:00:44.4763298Z 2024-08-07T18:00:44.4763642Z Running distributed/_tensor/test_matrix_ops 1/1 ... [2024-08-07 18:00:44.475448] 2024-08-07T18:00:44.4765138Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_matrix_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:00:44.475907] 2024-08-07T18:01:28.8820679Z 2024-08-07T18:01:28.8824116Z distributed/_tensor/test_matrix_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_matrix_ops_1.1_5cfce17d6585a1be_.log 2024-08-07T18:01:28.8828989Z Running 9 items in this shard: test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm_auto_redistribute, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm_empty_operand, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_baddbmm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_bmm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_mm, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_scaled_dot_product_attention, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_t, test/distributed/_tensor/test_matrix_ops.py::DistMatrixOpsTest::test_t_partial 2024-08-07T18:01:28.8832869Z 2024-08-07T18:01:28.8833209Z Running distributed/_tensor/test_random_ops 1/1 ... [2024-08-07 18:01:28.882359] 2024-08-07T18:01:28.8834698Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_random_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:01:28.882836] 2024-08-07T18:02:01.8143941Z 2024-08-07T18:02:01.8145182Z distributed/_tensor/test_random_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_random_ops_1.1_979c7ea3254001c5_.log 2024-08-07T18:02:01.8149335Z Running 7 items in this shard: test/distributed/_tensor/test_random_ops.py::DistTensorRandomInitTest::test_init_ops, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_dropout_1d, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_rand_1d, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_uniform_2d, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_manual_seed, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_meta_tensor_init, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_rng_tracker_init 2024-08-07T18:02:01.8153007Z 2024-08-07T18:02:01.8153443Z Running distributed/_tensor/experimental/test_register_sharding 1/1 ... [2024-08-07 18:02:01.814605] 2024-08-07T18:02:01.8155281Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/experimental/test_register_sharding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:02:01.815010] 2024-08-07T18:02:14.4045191Z 2024-08-07T18:02:14.4046620Z distributed/_tensor/experimental/test_register_sharding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.experimental.test_register_sharding_1.1_dd287bdb363c4bca_.log 2024-08-07T18:02:14.4051350Z Running 2 items in this shard: test/distributed/_tensor/experimental/test_register_sharding.py::TestRegisterSharding::test_argmax, test/distributed/_tensor/experimental/test_register_sharding.py::TestRegisterSharding::test_softmax_fwd 2024-08-07T18:02:14.4052795Z 2024-08-07T18:02:14.4053185Z Running distributed/tensor/parallel/test_tp_examples 1/1 ... [2024-08-07 18:02:14.404835] 2024-08-07T18:02:14.4056482Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_examples.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:02:14.405281] 2024-08-07T18:03:05.1213661Z 2024-08-07T18:03:05.1215200Z distributed/tensor/parallel/test_tp_examples 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_examples_1.1_31005c0582abb419_.log 2024-08-07T18:03:05.1222576Z Running 9 items in this shard: test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_loss_parallel, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_inference, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_False_recompute_activation_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_True_recompute_activation_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_False_float32, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_False_float64, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_True_float32, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_True_float64, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_weight_tying 2024-08-07T18:03:05.1228988Z 2024-08-07T18:03:05.1229329Z Running distributed/_tensor/test_common_rules 1/1 ... [2024-08-07 18:03:05.121786] 2024-08-07T18:03:05.1230915Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_common_rules.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:03:05.122314] 2024-08-07T18:03:46.1247886Z 2024-08-07T18:03:46.1249057Z distributed/_tensor/test_common_rules 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_common_rules_1.1_2b6461d8e9a404c5_.log 2024-08-07T18:03:46.1256976Z Running 10 items in this shard: test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_basic_propagation, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_errors, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_linearity, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_merge_sharding, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_pointwise_propagation, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_enforce_sharding_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_broadcasting, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_suggestion 2024-08-07T18:03:46.1262528Z 2024-08-07T18:03:46.1262919Z Running distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 ... [2024-08-07 18:03:46.125073] 2024-08-07T18:03:46.1264503Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/fsdp/test_fsdp_dsd.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:03:46.125466] 2024-08-07T18:04:04.6774772Z 2024-08-07T18:04:04.6776262Z distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.fsdp.test_fsdp_dsd_1.1_a1b67cba8c2bd740_.log 2024-08-07T18:04:04.6779693Z Running 3 items in this shard: test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_1d_fsdp_cpu_offload_full_model_state_dict, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_1d_fsdp_get_model_state_dict, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_fsdp1_and_load_with_fsdp2 2024-08-07T18:04:04.6781966Z 2024-08-07T18:04:04.6782355Z Running distributed/checkpoint/test_dedup_tensors 1/1 ... [2024-08-07 18:04:04.677770] 2024-08-07T18:04:04.6785791Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dedup_tensors.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:04:04.678217] 2024-08-07T18:04:08.4498845Z 2024-08-07T18:04:08.4500370Z distributed/checkpoint/test_dedup_tensors 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dedup_tensors_1.1_fc8cb8adc56761f0_.log 2024-08-07T18:04:08.4501937Z Running 1 items in this shard: test/distributed/checkpoint/test_dedup_tensors.py::TestDedupTensor::test_dedup_shards 2024-08-07T18:04:08.4502567Z 2024-08-07T18:04:08.4504920Z Running distributed/_tools/test_memory_tracker 1/1 ... [2024-08-07 18:04:08.450197] 2024-08-07T18:04:08.4509444Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_memory_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:04:08.450602] 2024-08-07T18:04:12.5731223Z 2024-08-07T18:04:12.5732422Z distributed/_tools/test_memory_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_memory_tracker_1.1_bfcee3e058409494_.log 2024-08-07T18:04:12.5734287Z Running 1 items in this shard: test/distributed/_tools/test_memory_tracker.py::TestMemoryTracker::test_local_model 2024-08-07T18:04:12.5734902Z 2024-08-07T18:04:12.5737281Z Running distributed/_tensor/test_init 1/1 ... [2024-08-07 18:04:12.573418] 2024-08-07T18:04:12.5742598Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:04:12.573813] 2024-08-07T18:04:43.0541603Z 2024-08-07T18:04:43.0542778Z distributed/_tensor/test_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_init_1.1_c0574ac6f04bef5f_.log 2024-08-07T18:04:43.0548821Z Running 7 items in this shard: test/distributed/_tensor/test_init.py::DTensorInitOpsTest::test_init_ops, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_empty, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_full, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_ones, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros_full_mesh, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros_submesh 2024-08-07T18:04:43.0551903Z 2024-08-07T18:04:43.0552272Z Running distributed/_tensor/test_redistribute 1/1 ... [2024-08-07 18:04:43.054355] 2024-08-07T18:04:43.0553779Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_redistribute.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:04:43.054752] 2024-08-07T18:05:33.5820792Z 2024-08-07T18:05:33.5823840Z distributed/_tensor/test_redistribute 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_redistribute_1.1_bd355ab8c4e0a4f0_.log 2024-08-07T18:05:33.5830628Z Running 12 items in this shard: test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_partial_to_replicate_forward_backward, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_partial_to_shard, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_redistribute_negative_shard_dim, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_redistribute_shard_dim_change, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_redistribute_uneven_sharding, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_local_partial_grad, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_partial, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_replicate_forward_backward, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_shard_forward_backward, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_shard_dim_alltoall, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_shard_to_replicate_forward_backward, test/distributed/_tensor/test_redistribute.py::MultiDimRedistributeTest::test_multi_dim_mesh 2024-08-07T18:05:33.5836766Z 2024-08-07T18:05:34.5315811Z 2024-08-07T18:05:34.5316716Z real 11m49.202s 2024-08-07T18:05:34.5317116Z user 21m18.266s 2024-08-07T18:05:34.5317413Z sys 8m31.195s 2024-08-07T18:05:34.5317736Z + assert_git_not_dirty 2024-08-07T18:05:34.5318134Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-08-07T18:05:34.5318643Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *xla* ]] 2024-08-07T18:05:34.5321971Z ++ git status --porcelain 2024-08-07T18:05:34.5322432Z ++ grep -v '?? third_party' 2024-08-07T18:05:37.3938536Z ++ true 2024-08-07T18:05:37.3939665Z + git_status= 2024-08-07T18:05:37.3940005Z + [[ -n '' ]] 2024-08-07T18:05:37.3940364Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-08-07T18:05:37.3941174Z + [[ 3 == 1 ]] 2024-08-07T18:05:37.3941479Z + [[ 3 == 1 ]] 2024-08-07T18:05:37.3942791Z + cleanup_workspace 2024-08-07T18:05:37.3943427Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-07T18:05:37.3944383Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-07T18:05:37.3945130Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-07T18:05:37.3945715Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-07T18:05:37.3946382Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-07T18:05:37.3947122Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-07T18:05:37.3947705Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-07T18:05:37.9457236Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-07T18:05:37.9457767Z cat test/**/*_toprint.log || true 2024-08-07T18:05:37.9466357Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:05:37.9466843Z env: 2024-08-07T18:05:37.9467121Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:05:37.9467551Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:05:37.9468242Z DOCKER_CONTAINER_ID: fd8fbd7c4999ec777ca8e9214dc76f6d75fb011239fd9300b6a357a13f0916b0 2024-08-07T18:05:37.9468848Z ##[endgroup] 2024-08-07T18:05:37.9539093Z cat: test/**/*_toprint.log: No such file or directory 2024-08-07T18:05:37.9580870Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-07T18:05:37.9581352Z kill "$MONITOR_SCRIPT_PID" 2024-08-07T18:05:37.9588710Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:05:37.9589194Z env: 2024-08-07T18:05:37.9589473Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:05:37.9589909Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:05:37.9590599Z DOCKER_CONTAINER_ID: fd8fbd7c4999ec777ca8e9214dc76f6d75fb011239fd9300b6a357a13f0916b0 2024-08-07T18:05:37.9591235Z MONITOR_SCRIPT_PID: 72895 2024-08-07T18:05:37.9591590Z ##[endgroup] 2024-08-07T18:05:37.9756818Z Prepare all required actions 2024-08-07T18:05:37.9757375Z Getting action download info 2024-08-07T18:05:38.1520961Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-07T18:05:38.3046062Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-07T18:05:38.3046509Z with: 2024-08-07T18:05:38.3046981Z file-suffix: test-distributed-3-3-linux.8xlarge.nvidia.gpu_28476145515 2024-08-07T18:05:38.3047567Z s3-bucket: gha-artifacts 2024-08-07T18:05:38.3047910Z env: 2024-08-07T18:05:38.3048204Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:05:38.3048624Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:05:38.3049320Z DOCKER_CONTAINER_ID: fd8fbd7c4999ec777ca8e9214dc76f6d75fb011239fd9300b6a357a13f0916b0 2024-08-07T18:05:38.3049953Z ##[endgroup] 2024-08-07T18:05:38.3098908Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-07T18:05:38.3099526Z # Remove any previous test jsons if they exist 2024-08-07T18:05:38.3100056Z rm -f test-jsons-*.zip 2024-08-07T18:05:38.3100548Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-07T18:05:38.3107892Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:05:38.3108379Z env: 2024-08-07T18:05:38.3108675Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:05:38.3109092Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:05:38.3109782Z DOCKER_CONTAINER_ID: fd8fbd7c4999ec777ca8e9214dc76f6d75fb011239fd9300b6a357a13f0916b0 2024-08-07T18:05:38.3110578Z FILE_SUFFIX: test-distributed-3-3-linux.8xlarge.nvidia.gpu_28476145515 2024-08-07T18:05:38.3111120Z ##[endgroup] 2024-08-07T18:05:38.3348430Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-07T18:05:38.3382897Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-07T18:05:38.3383814Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-07T18:05:38.3390781Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-07T18:05:38.3393164Z adding: test/test-reports/td_exclusions-0f3be12f7f3f19dc79f4.json (deflated 86%) 2024-08-07T18:05:38.3399292Z adding: test/.pytorch-slow-tests.json (deflated 81%) 2024-08-07T18:05:38.3414669Z adding: test/.pytorch-disabled-tests.json (deflated 89%) 2024-08-07T18:05:38.3475581Z ##[group]Run # Remove any previous test reports if they exist 2024-08-07T18:05:38.3476184Z # Remove any previous test reports if they exist 2024-08-07T18:05:38.3476661Z rm -f test-reports-*.zip 2024-08-07T18:05:38.3477210Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-07T18:05:38.3484445Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:05:38.3484912Z env: 2024-08-07T18:05:38.3485203Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:05:38.3485651Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:05:38.3486333Z DOCKER_CONTAINER_ID: fd8fbd7c4999ec777ca8e9214dc76f6d75fb011239fd9300b6a357a13f0916b0 2024-08-07T18:05:38.3487123Z FILE_SUFFIX: test-distributed-3-3-linux.8xlarge.nvidia.gpu_28476145515 2024-08-07T18:05:38.3487678Z ##[endgroup] 2024-08-07T18:05:38.3697096Z adding: test/test-reports/python-pytest/distributed._tensor.debug.test_op_coverage/distributed._tensor.debug.test_op_coverage-0372d9e2183305a8.xml (deflated 34%) 2024-08-07T18:05:38.3698942Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_nested_dict/distributed.checkpoint.test_nested_dict-20249e18413179f0.xml (deflated 50%) 2024-08-07T18:05:38.3700587Z adding: test/test-reports/python-pytest/distributed.pipelining.test_transformer/distributed.pipelining.test_transformer-fa64f929dfab7ec4.xml (deflated 39%) 2024-08-07T18:05:38.3702093Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-21eaf4605c329043.xml (deflated 82%) 2024-08-07T18:05:38.3703717Z adding: test/test-reports/python-pytest/distributed._tensor.test_matrix_ops/distributed._tensor.test_matrix_ops-f672678bdc368a30.xml (deflated 75%) 2024-08-07T18:05:38.3705196Z adding: test/test-reports/python-pytest/distributed._tensor.test_random_ops/distributed._tensor.test_random_ops-0663fc0dd6ac8d55.xml (deflated 72%) 2024-08-07T18:05:38.3706887Z adding: test/test-reports/python-pytest/distributed._tensor.experimental.test_register_sharding/distributed._tensor.experimental.test_register_sharding-f3ef3d6496a95eb4.xml (deflated 49%) 2024-08-07T18:05:38.3708942Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_examples/distributed.tensor.parallel.test_tp_examples-37d166db19806269.xml (deflated 78%) 2024-08-07T18:05:38.3710517Z adding: test/test-reports/python-pytest/distributed._tensor.test_common_rules/distributed._tensor.test_common_rules-895f951e16cf6369.xml (deflated 76%) 2024-08-07T18:05:38.3712095Z adding: test/test-reports/python-pytest/distributed.checkpoint.fsdp.test_fsdp_dsd/distributed.checkpoint.fsdp.test_fsdp_dsd-118b45f7fbd6bcf4.xml (deflated 57%) 2024-08-07T18:05:38.3713723Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dedup_tensors/distributed.checkpoint.test_dedup_tensors-518e0d2a8587d684.xml (deflated 37%) 2024-08-07T18:05:38.3715287Z adding: test/test-reports/python-pytest/distributed._tools.test_memory_tracker/distributed._tools.test_memory_tracker-66d2a26281e37ce9.xml (deflated 63%) 2024-08-07T18:05:38.3716692Z adding: test/test-reports/python-pytest/distributed._tensor.test_init/distributed._tensor.test_init-bd31fe34392d0cc4.xml (deflated 71%) 2024-08-07T18:05:38.3718122Z adding: test/test-reports/python-pytest/distributed._tensor.test_redistribute/distributed._tensor.test_redistribute-036ee6f3e1c6c009.xml (deflated 79%) 2024-08-07T18:05:38.3749288Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-07T18:05:38.3749865Z # Remove any previous usage logs if they exist 2024-08-07T18:05:38.3750511Z rm -f logs-*.zip 2024-08-07T18:05:38.3751084Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-07T18:05:38.3751756Z # so check to see if the file exists first 2024-08-07T18:05:38.3752215Z if [ -f 'usage_log.txt' ]; then 2024-08-07T18:05:38.3752673Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-07T18:05:38.3753122Z fi 2024-08-07T18:05:38.3753466Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-07T18:05:38.3753969Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-07T18:05:38.3754421Z fi 2024-08-07T18:05:38.3761314Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:05:38.3761779Z env: 2024-08-07T18:05:38.3762081Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:05:38.3762498Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:05:38.3763194Z DOCKER_CONTAINER_ID: fd8fbd7c4999ec777ca8e9214dc76f6d75fb011239fd9300b6a357a13f0916b0 2024-08-07T18:05:38.3763988Z FILE_SUFFIX: test-distributed-3-3-linux.8xlarge.nvidia.gpu_28476145515 2024-08-07T18:05:38.3764537Z ##[endgroup] 2024-08-07T18:05:38.3810510Z adding: usage_log.txt (deflated 92%) 2024-08-07T18:05:38.4025750Z adding: test/test-reports/distributed._tensor.debug.test_op_coverage_1.1_6c844a9ef45fa185_.log (deflated 55%) 2024-08-07T18:05:38.4026825Z adding: test/test-reports/distributed.checkpoint.test_nested_dict_1.1_13ef616b1c9041f1_.log (deflated 59%) 2024-08-07T18:05:38.4027864Z adding: test/test-reports/distributed.pipelining.test_transformer_1.1_4fc060f0ab3e5e19_.log (deflated 54%) 2024-08-07T18:05:38.4063866Z adding: test/test-reports/distributed.fsdp.test_fsdp_core_1.2_74957132fa890310_.log (deflated 96%) 2024-08-07T18:05:38.4064843Z adding: test/test-reports/distributed._tensor.test_matrix_ops_1.1_5cfce17d6585a1be_.log (deflated 73%) 2024-08-07T18:05:38.4065843Z adding: test/test-reports/distributed._tensor.test_random_ops_1.1_979c7ea3254001c5_.log (deflated 70%) 2024-08-07T18:05:38.4067171Z adding: test/test-reports/distributed._tensor.experimental.test_register_sharding_1.1_dd287bdb363c4bca_.log (deflated 60%) 2024-08-07T18:05:38.4068363Z adding: test/test-reports/distributed.tensor.parallel.test_tp_examples_1.1_31005c0582abb419_.log (deflated 78%) 2024-08-07T18:05:38.4069401Z adding: test/test-reports/distributed._tensor.test_common_rules_1.1_2b6461d8e9a404c5_.log (deflated 76%) 2024-08-07T18:05:38.4070465Z adding: test/test-reports/distributed.checkpoint.fsdp.test_fsdp_dsd_1.1_a1b67cba8c2bd740_.log (deflated 89%) 2024-08-07T18:05:38.4071520Z adding: test/test-reports/distributed.checkpoint.test_dedup_tensors_1.1_fc8cb8adc56761f0_.log (deflated 52%) 2024-08-07T18:05:38.4072561Z adding: test/test-reports/distributed._tools.test_memory_tracker_1.1_bfcee3e058409494_.log (deflated 54%) 2024-08-07T18:05:38.4073528Z adding: test/test-reports/distributed._tensor.test_init_1.1_c0574ac6f04bef5f_.log (deflated 69%) 2024-08-07T18:05:38.4074989Z adding: test/test-reports/distributed._tensor.test_redistribute_1.1_bd355ab8c4e0a4f0_.log (deflated 93%) 2024-08-07T18:05:38.4110933Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-07T18:05:38.4111572Z # Remove any previous debugging artifacts if they exist 2024-08-07T18:05:38.4112071Z rm -f debug-*.zip 2024-08-07T18:05:38.4112446Z if [ -d 'test/debug' ]; then 2024-08-07T18:05:38.4112896Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-07T18:05:38.4113312Z fi 2024-08-07T18:05:38.4120221Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:05:38.4120694Z env: 2024-08-07T18:05:38.4120965Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:05:38.4121386Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:05:38.4122064Z DOCKER_CONTAINER_ID: fd8fbd7c4999ec777ca8e9214dc76f6d75fb011239fd9300b6a357a13f0916b0 2024-08-07T18:05:38.4122993Z FILE_SUFFIX: test-distributed-3-3-linux.8xlarge.nvidia.gpu_28476145515 2024-08-07T18:05:38.4123521Z ##[endgroup] 2024-08-07T18:05:38.4212936Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-07T18:05:38.4213388Z with: 2024-08-07T18:05:38.4213670Z s3-bucket: gha-artifacts 2024-08-07T18:05:38.4214093Z s3-prefix: pytorch/pytorch/10288745067/1/artifact 2024-08-07T18:05:38.4214565Z retention-days: 14 2024-08-07T18:05:38.4214886Z if-no-files-found: warn 2024-08-07T18:05:38.4215245Z path: test-jsons-*.zip 2024-08-07T18:05:38.4215587Z name: artifact 2024-08-07T18:05:38.4215880Z region: us-east-1 2024-08-07T18:05:38.4216180Z env: 2024-08-07T18:05:38.4216450Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:05:38.4216868Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:05:38.4217593Z DOCKER_CONTAINER_ID: fd8fbd7c4999ec777ca8e9214dc76f6d75fb011239fd9300b6a357a13f0916b0 2024-08-07T18:05:38.4218631Z ##[endgroup] 2024-08-07T18:05:38.8533423Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-07T18:05:38.8534620Z With the provided path, there will be 1 file uploaded 2024-08-07T18:05:38.8535204Z Uploading to s3 prefix: pytorch/pytorch/10288745067/1/artifact 2024-08-07T18:05:38.8582138Z Starting upload of test-jsons-test-distributed-3-3-linux.8xlarge.nvidia.gpu_28476145515.zip 2024-08-07T18:05:39.0638817Z Finished upload of test-jsons-test-distributed-3-3-linux.8xlarge.nvidia.gpu_28476145515.zip 2024-08-07T18:05:39.0803184Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-07T18:05:39.0803630Z with: 2024-08-07T18:05:39.0803919Z s3-bucket: gha-artifacts 2024-08-07T18:05:39.0804364Z s3-prefix: pytorch/pytorch/10288745067/1/artifact 2024-08-07T18:05:39.0804825Z retention-days: 14 2024-08-07T18:05:39.0805156Z if-no-files-found: error 2024-08-07T18:05:39.0805529Z path: test-reports-*.zip 2024-08-07T18:05:39.0805887Z name: artifact 2024-08-07T18:05:39.0806186Z region: us-east-1 2024-08-07T18:05:39.0806503Z env: 2024-08-07T18:05:39.0806792Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:05:39.0807234Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:05:39.0808162Z DOCKER_CONTAINER_ID: fd8fbd7c4999ec777ca8e9214dc76f6d75fb011239fd9300b6a357a13f0916b0 2024-08-07T18:05:39.0808823Z ##[endgroup] 2024-08-07T18:05:39.5226427Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-07T18:05:39.5227308Z With the provided path, there will be 1 file uploaded 2024-08-07T18:05:39.5227890Z Uploading to s3 prefix: pytorch/pytorch/10288745067/1/artifact 2024-08-07T18:05:39.5274814Z Starting upload of test-reports-test-distributed-3-3-linux.8xlarge.nvidia.gpu_28476145515.zip 2024-08-07T18:05:39.6844070Z Finished upload of test-reports-test-distributed-3-3-linux.8xlarge.nvidia.gpu_28476145515.zip 2024-08-07T18:05:39.7009182Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-07T18:05:39.7009624Z with: 2024-08-07T18:05:39.7009914Z s3-bucket: gha-artifacts 2024-08-07T18:05:39.7010338Z s3-prefix: pytorch/pytorch/10288745067/1/artifact 2024-08-07T18:05:39.7010817Z retention-days: 14 2024-08-07T18:05:39.7011149Z if-no-files-found: ignore 2024-08-07T18:05:39.7011531Z path: logs-*.zip 2024-08-07T18:05:39.7011851Z name: artifact 2024-08-07T18:05:39.7012146Z region: us-east-1 2024-08-07T18:05:39.7012463Z env: 2024-08-07T18:05:39.7012754Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:05:39.7013173Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:05:39.7013873Z DOCKER_CONTAINER_ID: fd8fbd7c4999ec777ca8e9214dc76f6d75fb011239fd9300b6a357a13f0916b0 2024-08-07T18:05:39.7014500Z ##[endgroup] 2024-08-07T18:05:40.1440891Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-07T18:05:40.1441854Z With the provided path, there will be 1 file uploaded 2024-08-07T18:05:40.1442453Z Uploading to s3 prefix: pytorch/pytorch/10288745067/1/artifact 2024-08-07T18:05:40.1489201Z Starting upload of logs-test-distributed-3-3-linux.8xlarge.nvidia.gpu_28476145515.zip 2024-08-07T18:05:40.2967918Z Finished upload of logs-test-distributed-3-3-linux.8xlarge.nvidia.gpu_28476145515.zip 2024-08-07T18:05:40.3205960Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-07T18:05:40.3206422Z with: 2024-08-07T18:05:40.3206712Z s3-bucket: gha-artifacts 2024-08-07T18:05:40.3207137Z s3-prefix: pytorch/pytorch/10288745067/1/artifact 2024-08-07T18:05:40.3207592Z retention-days: 14 2024-08-07T18:05:40.3207921Z if-no-files-found: ignore 2024-08-07T18:05:40.3208284Z path: debug-*.zip 2024-08-07T18:05:40.3208601Z name: artifact 2024-08-07T18:05:40.3208893Z region: us-east-1 2024-08-07T18:05:40.3209204Z env: 2024-08-07T18:05:40.3209493Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:05:40.3209906Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:05:40.3210593Z DOCKER_CONTAINER_ID: fd8fbd7c4999ec777ca8e9214dc76f6d75fb011239fd9300b6a357a13f0916b0 2024-08-07T18:05:40.3211213Z ##[endgroup] 2024-08-07T18:05:40.7497617Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-07T18:05:40.7662216Z ##[group]Run # shellcheck disable=SC2156 2024-08-07T18:05:40.7662777Z # shellcheck disable=SC2156 2024-08-07T18:05:40.7663508Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-07T18:05:40.7671794Z shell: /usr/bin/bash -e {0} 2024-08-07T18:05:40.7672178Z env: 2024-08-07T18:05:40.7672475Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:05:40.7672897Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:05:40.7673600Z DOCKER_CONTAINER_ID: fd8fbd7c4999ec777ca8e9214dc76f6d75fb011239fd9300b6a357a13f0916b0 2024-08-07T18:05:40.7674228Z ##[endgroup] 2024-08-07T18:05:41.0641148Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-07T18:05:41.0641722Z with: 2024-08-07T18:05:41.0641987Z env: 2024-08-07T18:05:41.0642278Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:05:41.0642713Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:05:41.0643383Z DOCKER_CONTAINER_ID: fd8fbd7c4999ec777ca8e9214dc76f6d75fb011239fd9300b6a357a13f0916b0 2024-08-07T18:05:41.0644025Z ##[endgroup] 2024-08-07T18:05:41.0674708Z ##[group]Run set -eou pipefail 2024-08-07T18:05:41.0675132Z set -eou pipefail 2024-08-07T18:05:41.0675472Z  2024-08-07T18:05:41.0675943Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-07T18:05:41.0676519Z for _ in $(seq 1440); do 2024-08-07T18:05:41.0676931Z  # Break if no ssh session exists anymore 2024-08-07T18:05:41.0677381Z  if [ "$(who)" = "" ]; then 2024-08-07T18:05:41.0677770Z  break 2024-08-07T18:05:41.0678128Z  fi 2024-08-07T18:05:41.0678423Z  echo "." 2024-08-07T18:05:41.0678750Z  sleep 5 2024-08-07T18:05:41.0679068Z done 2024-08-07T18:05:41.0686713Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:05:41.0687202Z env: 2024-08-07T18:05:41.0687498Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:05:41.0687931Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:05:41.0688622Z DOCKER_CONTAINER_ID: fd8fbd7c4999ec777ca8e9214dc76f6d75fb011239fd9300b6a357a13f0916b0 2024-08-07T18:05:41.0689247Z ##[endgroup] 2024-08-07T18:05:41.0713611Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-07T18:05:41.0769503Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-07T18:05:41.0770346Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-07T18:05:41.0770990Z # shellcheck disable=SC2046 2024-08-07T18:05:41.0771519Z docker stop $(docker ps -q) || true 2024-08-07T18:05:41.0772077Z # Prune all of the docker images 2024-08-07T18:05:41.0772615Z docker system prune -af 2024-08-07T18:05:41.0798538Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:05:41.0799036Z env: 2024-08-07T18:05:41.0799339Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:05:41.0799760Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:05:41.0800734Z DOCKER_CONTAINER_ID: fd8fbd7c4999ec777ca8e9214dc76f6d75fb011239fd9300b6a357a13f0916b0 2024-08-07T18:05:41.0801366Z ##[endgroup] 2024-08-07T18:05:41.4648736Z fd8fbd7c4999 2024-08-07T18:05:42.0208920Z Deleted Containers: 2024-08-07T18:05:42.0209473Z fd8fbd7c4999ec777ca8e9214dc76f6d75fb011239fd9300b6a357a13f0916b0 2024-08-07T18:05:42.0209915Z 2024-08-07T18:05:48.6231395Z Deleted Images: 2024-08-07T18:05:48.6232636Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T18:05:48.6234296Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9@sha256:66181a134b3ebc35388dabcc46dc11e767152fd72514dad76711590f50b650f7 2024-08-07T18:05:48.6235594Z deleted: sha256:8e4e19ae0377c273bc603b24bc49d406d6f755072d4534d040c86afbe761a129 2024-08-07T18:05:48.6236387Z deleted: sha256:bee5df81b6fe1ee0aae7e536a9bda16351d0515be5b789ad50bf0ac60fd41e33 2024-08-07T18:05:48.6237224Z deleted: sha256:8ee1ee099b65b2d73d66dc3330f82f8be2dd6cbdccf3d718b448a5e3419c214e 2024-08-07T18:05:48.6238013Z deleted: sha256:aed7424cdc4428e32e398a33e98e6e252075e6280e7068e20df7626451fa312a 2024-08-07T18:05:48.6238772Z deleted: sha256:61b7d2994ed16481c3de07839b5a3bb779dc2f3f716ad49acb91537042c3b7d1 2024-08-07T18:05:48.6239551Z deleted: sha256:79999dfec9b5df258486b014aa99c30979900c4131effd2a7bc18251cd5fc372 2024-08-07T18:05:48.6240338Z deleted: sha256:9020e3bab07b431af26c788ac2d12cbca9a76ffb1c923ea45a38e872942e1f34 2024-08-07T18:05:48.6241371Z deleted: sha256:387e2f2f359df5f69c500ae9589c224d8283ae5418520509b3c9ef0a45d116fa 2024-08-07T18:05:48.6242190Z deleted: sha256:5ddfc73c14e2e0a762c51ddecfb2710258366e1d048a8bf56d70268486907b99 2024-08-07T18:05:48.6242979Z deleted: sha256:4dffb251634d1c8105e68310ef1b2ffb923dac4c4139f494be0938ad032a1bca 2024-08-07T18:05:48.6243744Z deleted: sha256:e0df7617281f0e4396253d2f0c40d1418ad604dc0912cae13f153fe22e316fa0 2024-08-07T18:05:48.6244542Z deleted: sha256:d239b3ccf05c2850364c3f73acbc6e0017a25f8faa4c3abaa8524e63dc9f9c29 2024-08-07T18:05:48.6245339Z deleted: sha256:918d0912faf4251f3be58f9a5aaa80f59a180ace4771b9ce903cedf94a57723a 2024-08-07T18:05:48.6246117Z deleted: sha256:09d5696392736fa88c52a0656d1d831d2bcb4015630810412565a5a086eb98ff 2024-08-07T18:05:48.6246866Z deleted: sha256:67542af70388f69b345fc7f0965e1fce3c552ee987689f804c8406536f16534f 2024-08-07T18:05:48.6247651Z deleted: sha256:350337bafbf2976e8d4e0b7ce3cc66b2cb43ebb8622fa6e538b02f561f929393 2024-08-07T18:05:48.6248465Z deleted: sha256:00fe9fe12fefe10d5dbfb2f855aa385e840f360deeccbc9ee92cae51c6eea4ba 2024-08-07T18:05:48.6249258Z deleted: sha256:9895af0586ff7d94936462a92d7cec0d8ec3b01df27b664f7e4ac277ce7bbe9e 2024-08-07T18:05:48.6250050Z deleted: sha256:7dd7dbc3ec792cf9d043ff867afe169d15fe414138097ad9247a5f388d3eeccf 2024-08-07T18:05:48.6250840Z deleted: sha256:e0a3ace1b87c67d71f5db76d211740431660a7fbcd80415333c9e7b7b9a2e89b 2024-08-07T18:05:48.6251612Z deleted: sha256:b9c56b5040d5d5998961b788563b85e5be12f678bc6b3976d089a81f1c93b7e8 2024-08-07T18:05:48.6252392Z deleted: sha256:9232fe8591973938b7bdb0a7a31dcdbc79f9667d6e011a2facf4e17fb77d4922 2024-08-07T18:05:48.6253183Z deleted: sha256:52f5fbe778d735123327825c6a16dcec9bc84c2623087bb4c7c094ac99764ae7 2024-08-07T18:05:48.6253964Z deleted: sha256:4b7c4121b76b04d5c7e4a0b77add435c0caa99cccff506ea704eba27fd2b2166 2024-08-07T18:05:48.6254754Z deleted: sha256:795dd765ace5b617808fc40cd576b449ae01b8671290e6df6b49d86ea79f4145 2024-08-07T18:05:48.6255758Z deleted: sha256:51cc651f1ffbeef49ad010ab2d4cf2d5ce5f8f5767c2133346a959c773d837bb 2024-08-07T18:05:48.6256541Z deleted: sha256:4c3019549b40f7a268d344eae162d7a7cc78855880db865944da136e01c190e9 2024-08-07T18:05:48.6257341Z deleted: sha256:855b789ec056edaab8cc8dad50a82e9dbfb9b20bcf39c70d9c3bd40080b2b65f 2024-08-07T18:05:48.6258141Z deleted: sha256:cef853cb7e0f5d6351692a4b9a0413a2496ea3c7bfe5d8063528ac2d6c17696f 2024-08-07T18:05:48.6259539Z deleted: sha256:4f7b163ef5ff693ce9f7380bdf7b526131f2567489342f0630949c9341e1c63f 2024-08-07T18:05:48.6260537Z deleted: sha256:09f12e5e102232b4ec9fd5168f4aa4b20ee0dfc755b053ac57690657e72e3fe3 2024-08-07T18:05:48.6261332Z deleted: sha256:c9962be393e03353af0d6a26f5478a94cacb5e528ee3f97609fac594c9a5d387 2024-08-07T18:05:48.6262127Z deleted: sha256:ac7aa4ec25b0d9a035c438effaa309071729de3f3a0f96e8e481b9839e7460b8 2024-08-07T18:05:48.6262895Z deleted: sha256:63ff2106018988888d022ee38d6ce1805aa29d8368a18dec1ab8eb587799ab44 2024-08-07T18:05:48.6263680Z deleted: sha256:3298c1acdeb701f11351366cfe680a84b8290666fdcd76af5ac79f226422fc0d 2024-08-07T18:05:48.6264485Z deleted: sha256:f076cc29eff5dae969bc7c05003d680b6e683eaef971c0a3f67a8a4d12693ee4 2024-08-07T18:05:48.6265258Z deleted: sha256:2604655e98c557e6de2e650d21cd0306c01f12204f3538386a9f374d003a4d70 2024-08-07T18:05:48.6266029Z deleted: sha256:fb942194d6d4fc6f9c8a4c8cba17330563877c92369892edc9b005b8b491f204 2024-08-07T18:05:48.6266815Z deleted: sha256:d349cc387708a540eee839ccd78494458dc5f1ccf989db608d25c5f798ec8c76 2024-08-07T18:05:48.6267967Z deleted: sha256:269864865f9b12ee399ca9ee3e15a16cedb86fd4889e01a83567afb357a774e6 2024-08-07T18:05:48.6268888Z deleted: sha256:236a91ce743cd500427338a78915ff834ae80331fcc06afccfca82a35bffb63c 2024-08-07T18:05:48.6269958Z deleted: sha256:de968bbc1d8bc2ad420d6a1c78a9a6cf8a763bb855ca3aefe7de6c5c52201f7a 2024-08-07T18:05:48.6270797Z deleted: sha256:0f15ecc513ffdac89965bd76cd9f8c7f4e37bbe7be0bfa086353133e2dd4f086 2024-08-07T18:05:48.6271618Z deleted: sha256:754bb9b5a5af40b6ad88f0ab14be37b9e512e25edccbfc9fd5e0c5a88ca09c0f 2024-08-07T18:05:48.6272589Z deleted: sha256:427104437dfa1883feec2c441eb4e4f825481883c6d3d31759361cbc924a27fe 2024-08-07T18:05:48.6273381Z deleted: sha256:0b404434cf2c3087ef28ea3cdc5813cc555f6e261c2514a7ab5bf5d484b5b575 2024-08-07T18:05:48.6274183Z deleted: sha256:bfb91e062bab9dafb63d0d4ab366026692fbd63e642da9ab28f04c8bba9b56a3 2024-08-07T18:05:48.6274975Z deleted: sha256:54617b36a396e9e82f2092257d2f621c90de460e0e54b1cf8df065573fbaa0ba 2024-08-07T18:05:48.6275755Z deleted: sha256:d8bdc337db1750bb3ebc7c8e4e1bd48e722218f261cb35ec2056ceaa6d3c4e9c 2024-08-07T18:05:48.6276582Z deleted: sha256:25cbacf4be7b2bbcbeaf7375d0eff0eaa33e94c5bef20ff0993bf074fdaa54eb 2024-08-07T18:05:48.6277392Z deleted: sha256:43744f05dcf6dc44c2f5e06533b13e085319a78b6b5e17eba5e7bd4419f5c905 2024-08-07T18:05:48.6278163Z deleted: sha256:2ea7d74f63d16a8d6e14a87e0e916f67f99536b21ac53e80b27786dcd1ce36fd 2024-08-07T18:05:48.6278950Z deleted: sha256:dbfb6031f18e50c89ff1a268c1e80e2fa88e727c48656718b326b60e8ef0d383 2024-08-07T18:05:48.6279756Z deleted: sha256:3eb3bb1755fcc3d4aba9abe2d2270ed2a55ec6254b3e87ca7688370e67e707a3 2024-08-07T18:05:48.6280561Z deleted: sha256:04605aba2d51924a85499584edda24c28390eb777c1c88eff2f4191b48e3da8c 2024-08-07T18:05:48.6281329Z deleted: sha256:117ff9e5eddba4582a39d416380eadf1c245237612c5ae0d6781d36c36bcb87f 2024-08-07T18:05:48.6282124Z deleted: sha256:ebf86c84eafa12a81b8eb66541a56c8056f4f9bc30e4be761771e53c6e1342aa 2024-08-07T18:05:48.6282922Z deleted: sha256:23d7f7e0c9d7e60acb59e60da9b4ee4e1840c1c406bf2082822c49430975529d 2024-08-07T18:05:48.6283705Z deleted: sha256:3e7e5583645195af5eeb60fe49024a4aa2f0eb3be28ff95c9859bb6233c29c4b 2024-08-07T18:05:48.6284510Z deleted: sha256:45b8f0dd78e72f0156b1fbc4dcb14200c61954f37a75129c753275e5a45ccd9b 2024-08-07T18:05:48.6285302Z deleted: sha256:f6b7a042384e2f2bb3401da2b9f4e9b9f942adb70be7e7f89c46fa81a42774d8 2024-08-07T18:05:48.6286085Z deleted: sha256:eb547a7b8fe732bffe8cc9a66aa19f2b41f6e0616a8a27040307c022d00fbc0d 2024-08-07T18:05:48.6286880Z deleted: sha256:20f729283bb4f2063f3a8baa0b2afc17911da345bf367e1918d8cc09ddd6d964 2024-08-07T18:05:48.6287679Z deleted: sha256:8ce8da9ab0319625a66c35d24ddadf8d2688d16c5024ba2345099431e0aa7ff8 2024-08-07T18:05:48.6288468Z deleted: sha256:e7a91181c74f879360530f0bbe957b8ff96a1ec19adde8a41d9e3fa2038d0d16 2024-08-07T18:05:48.6289257Z deleted: sha256:212b7a0fad7208624a2bc56768d8c489d2690142d0db0fa20f69eb648234e8d1 2024-08-07T18:05:48.6290042Z deleted: sha256:72315e37f51ba8bcc88fc5b56f92e5bb6264456d9a755afa01fd19f9128c1df7 2024-08-07T18:05:48.6290936Z deleted: sha256:bfc5ab07cfab1698147152988eabdc7ebbdeb35b6d9aefd9360dd64104cd3888 2024-08-07T18:05:48.6291725Z deleted: sha256:89edf8c906678c217b9a88160e1561991f53909c3fd27a4a5e1f0ca0f7ff7888 2024-08-07T18:05:48.6292526Z deleted: sha256:d5ba67f8f9b0c2b6da06d2d5d9d5fedc9f17cbe1a5e13a1a4033a4619f4528ba 2024-08-07T18:05:48.6293309Z deleted: sha256:5d61311e1a293c0c9642e34871719525dfeec7028e4b45e08344a469e3fe24bf 2024-08-07T18:05:48.6294068Z deleted: sha256:b89bb70f745a94c0edd5e974aee294e160735f76fd666a1a06a0b9b1be3caf63 2024-08-07T18:05:48.6294861Z deleted: sha256:9e5477f1f21a38c3dfe21a56a7d77d1b1a42453aed3e11729afcbb9d5731770b 2024-08-07T18:05:48.6295660Z deleted: sha256:b76f4af91cffe2bb7b3f09960f5bdbb26f741419defda65c451c4407fc5b0a92 2024-08-07T18:05:48.6296452Z deleted: sha256:3f8748a00ebd394c0e9d9d481972d764dff58896841ef0150b3f7efaac0f4a4a 2024-08-07T18:05:48.6297237Z deleted: sha256:13989c771e3ba4d40f618344e55e5cb7062a35041b5efd69e2f740ba819f46ee 2024-08-07T18:05:48.6298034Z deleted: sha256:9ebb27837f6c67ad8c17e9af4e63c008faae03fa226c3641dd8bbbfeefb106c4 2024-08-07T18:05:48.6299207Z deleted: sha256:8c16907d735950f8093582bc3cf392b70e002456bf9c4224de5a5b86b78b3ea2 2024-08-07T18:05:48.6299987Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-07T18:05:48.6300504Z 2024-08-07T18:05:48.6300692Z Total reclaimed space: 26.03GB 2024-08-07T18:05:48.6368803Z Post job cleanup. 2024-08-07T18:05:48.6427410Z Post job cleanup. 2024-08-07T18:05:48.7609708Z [command]/usr/bin/git version 2024-08-07T18:05:48.7664295Z git version 2.40.1 2024-08-07T18:05:48.7727694Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/1efe7a0b-c102-4624-8b13-814cb1b996ca' before making global git config changes 2024-08-07T18:05:48.7729033Z Adding repository directory to the temporary git global config as a safe directory 2024-08-07T18:05:48.7735740Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-07T18:05:48.7777606Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-07T18:05:48.7814173Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-07T18:05:48.8124739Z Entering 'android/libs/fbjni' 2024-08-07T18:05:48.8174910Z Entering 'third_party/FP16' 2024-08-07T18:05:48.8225682Z Entering 'third_party/FXdiv' 2024-08-07T18:05:48.8275252Z Entering 'third_party/NNPACK' 2024-08-07T18:05:48.8325594Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-07T18:05:48.8376339Z Entering 'third_party/XNNPACK' 2024-08-07T18:05:48.8445484Z Entering 'third_party/benchmark' 2024-08-07T18:05:48.8496231Z Entering 'third_party/cpp-httplib' 2024-08-07T18:05:48.8545373Z Entering 'third_party/cpuinfo' 2024-08-07T18:05:48.8595020Z Entering 'third_party/cudnn_frontend' 2024-08-07T18:05:48.8644688Z Entering 'third_party/cutlass' 2024-08-07T18:05:48.8702633Z Entering 'third_party/eigen' 2024-08-07T18:05:48.8754584Z Entering 'third_party/fbgemm' 2024-08-07T18:05:48.8805708Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-07T18:05:48.8854425Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T18:05:48.8904979Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-07T18:05:48.8961384Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-07T18:05:48.9010477Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T18:05:48.9060136Z Entering 'third_party/flatbuffers' 2024-08-07T18:05:48.9112140Z Entering 'third_party/fmt' 2024-08-07T18:05:48.9162816Z Entering 'third_party/foxi' 2024-08-07T18:05:48.9213652Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-07T18:05:48.9263722Z Entering 'third_party/gloo' 2024-08-07T18:05:48.9312647Z Entering 'third_party/googletest' 2024-08-07T18:05:48.9362369Z Entering 'third_party/ideep' 2024-08-07T18:05:48.9412277Z Entering 'third_party/ideep/mkl-dnn' 2024-08-07T18:05:48.9470015Z Entering 'third_party/ittapi' 2024-08-07T18:05:48.9519345Z Entering 'third_party/kineto' 2024-08-07T18:05:48.9570367Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T18:05:48.9620283Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T18:05:48.9671924Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T18:05:48.9723920Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T18:05:48.9772896Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T18:05:48.9821509Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T18:05:48.9871970Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T18:05:48.9925300Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T18:05:48.9978939Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T18:05:49.0031981Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T18:05:49.0083873Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T18:05:49.0134332Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T18:05:49.0187025Z Entering 'third_party/mimalloc' 2024-08-07T18:05:49.0238182Z Entering 'third_party/nccl/nccl' 2024-08-07T18:05:49.0289871Z Entering 'third_party/nlohmann' 2024-08-07T18:05:49.0342683Z Entering 'third_party/onnx' 2024-08-07T18:05:49.0410915Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-07T18:05:49.0463681Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-07T18:05:49.0515828Z Entering 'third_party/opentelemetry-cpp' 2024-08-07T18:05:49.0568372Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T18:05:49.0618083Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T18:05:49.0667591Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T18:05:49.0716981Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T18:05:49.0768524Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T18:05:49.0817774Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T18:05:49.0867217Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T18:05:49.0916754Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T18:05:49.0967686Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T18:05:49.1018938Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T18:05:49.1089817Z Entering 'third_party/pocketfft' 2024-08-07T18:05:49.1139755Z Entering 'third_party/protobuf' 2024-08-07T18:05:49.1193884Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-07T18:05:49.1244602Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-07T18:05:49.1296560Z Entering 'third_party/psimd' 2024-08-07T18:05:49.1346501Z Entering 'third_party/pthreadpool' 2024-08-07T18:05:49.1395564Z Entering 'third_party/pybind11' 2024-08-07T18:05:49.1446845Z Entering 'third_party/python-peachpy' 2024-08-07T18:05:49.1497677Z Entering 'third_party/sleef' 2024-08-07T18:05:49.1549590Z Entering 'third_party/tensorpipe' 2024-08-07T18:05:49.1599987Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-07T18:05:49.1651470Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-07T18:05:49.1701553Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-07T18:05:49.1751474Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T18:05:49.1799489Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T18:05:49.1870724Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-07T18:05:49.1896808Z http.https://github.com/.extraheader 2024-08-07T18:05:49.1909624Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-07T18:05:49.1945745Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-07T18:05:49.2245338Z Entering 'android/libs/fbjni' 2024-08-07T18:05:49.2277612Z http.https://github.com/.extraheader 2024-08-07T18:05:49.2311275Z Entering 'third_party/FP16' 2024-08-07T18:05:49.2343260Z http.https://github.com/.extraheader 2024-08-07T18:05:49.2375941Z Entering 'third_party/FXdiv' 2024-08-07T18:05:49.2408322Z http.https://github.com/.extraheader 2024-08-07T18:05:49.2442154Z Entering 'third_party/NNPACK' 2024-08-07T18:05:49.2473940Z http.https://github.com/.extraheader 2024-08-07T18:05:49.2507556Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-07T18:05:49.2539023Z http.https://github.com/.extraheader 2024-08-07T18:05:49.2572164Z Entering 'third_party/XNNPACK' 2024-08-07T18:05:49.2605267Z http.https://github.com/.extraheader 2024-08-07T18:05:49.2655447Z Entering 'third_party/benchmark' 2024-08-07T18:05:49.2687605Z http.https://github.com/.extraheader 2024-08-07T18:05:49.2721218Z Entering 'third_party/cpp-httplib' 2024-08-07T18:05:49.2753181Z http.https://github.com/.extraheader 2024-08-07T18:05:49.2785480Z Entering 'third_party/cpuinfo' 2024-08-07T18:05:49.2817246Z http.https://github.com/.extraheader 2024-08-07T18:05:49.2850210Z Entering 'third_party/cudnn_frontend' 2024-08-07T18:05:49.2882980Z http.https://github.com/.extraheader 2024-08-07T18:05:49.2915628Z Entering 'third_party/cutlass' 2024-08-07T18:05:49.2948464Z http.https://github.com/.extraheader 2024-08-07T18:05:49.2989196Z Entering 'third_party/eigen' 2024-08-07T18:05:49.3021129Z http.https://github.com/.extraheader 2024-08-07T18:05:49.3056649Z Entering 'third_party/fbgemm' 2024-08-07T18:05:49.3089208Z http.https://github.com/.extraheader 2024-08-07T18:05:49.3122334Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-07T18:05:49.3155032Z http.https://github.com/.extraheader 2024-08-07T18:05:49.3187881Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T18:05:49.3219617Z http.https://github.com/.extraheader 2024-08-07T18:05:49.3253012Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-07T18:05:49.3284413Z http.https://github.com/.extraheader 2024-08-07T18:05:49.3324810Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-07T18:05:49.3357093Z http.https://github.com/.extraheader 2024-08-07T18:05:49.3389947Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T18:05:49.3420768Z http.https://github.com/.extraheader 2024-08-07T18:05:49.3453484Z Entering 'third_party/flatbuffers' 2024-08-07T18:05:49.3486306Z http.https://github.com/.extraheader 2024-08-07T18:05:49.3522171Z Entering 'third_party/fmt' 2024-08-07T18:05:49.3554030Z http.https://github.com/.extraheader 2024-08-07T18:05:49.3586991Z Entering 'third_party/foxi' 2024-08-07T18:05:49.3618813Z http.https://github.com/.extraheader 2024-08-07T18:05:49.3650600Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-07T18:05:49.3682555Z http.https://github.com/.extraheader 2024-08-07T18:05:49.3714998Z Entering 'third_party/gloo' 2024-08-07T18:05:49.3748212Z http.https://github.com/.extraheader 2024-08-07T18:05:49.3780993Z Entering 'third_party/googletest' 2024-08-07T18:05:49.3812267Z http.https://github.com/.extraheader 2024-08-07T18:05:49.3845621Z Entering 'third_party/ideep' 2024-08-07T18:05:49.3877316Z http.https://github.com/.extraheader 2024-08-07T18:05:49.3909271Z Entering 'third_party/ideep/mkl-dnn' 2024-08-07T18:05:49.3940657Z http.https://github.com/.extraheader 2024-08-07T18:05:49.3979971Z Entering 'third_party/ittapi' 2024-08-07T18:05:49.4011491Z http.https://github.com/.extraheader 2024-08-07T18:05:49.4044787Z Entering 'third_party/kineto' 2024-08-07T18:05:49.4076936Z http.https://github.com/.extraheader 2024-08-07T18:05:49.4110457Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T18:05:49.4142420Z http.https://github.com/.extraheader 2024-08-07T18:05:49.4176846Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T18:05:49.4208412Z http.https://github.com/.extraheader 2024-08-07T18:05:49.4244082Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T18:05:49.4275782Z http.https://github.com/.extraheader 2024-08-07T18:05:49.4310186Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T18:05:49.4342394Z http.https://github.com/.extraheader 2024-08-07T18:05:49.4375360Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T18:05:49.4407123Z http.https://github.com/.extraheader 2024-08-07T18:05:49.4439141Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T18:05:49.4471143Z http.https://github.com/.extraheader 2024-08-07T18:05:49.4506537Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T18:05:49.4537858Z http.https://github.com/.extraheader 2024-08-07T18:05:49.4571057Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T18:05:49.4602841Z http.https://github.com/.extraheader 2024-08-07T18:05:49.4636056Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T18:05:49.4667398Z http.https://github.com/.extraheader 2024-08-07T18:05:49.4702735Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T18:05:49.4734041Z http.https://github.com/.extraheader 2024-08-07T18:05:49.4769068Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T18:05:49.4801073Z http.https://github.com/.extraheader 2024-08-07T18:05:49.4834479Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T18:05:49.4865381Z http.https://github.com/.extraheader 2024-08-07T18:05:49.4900481Z Entering 'third_party/mimalloc' 2024-08-07T18:05:49.4932544Z http.https://github.com/.extraheader 2024-08-07T18:05:49.4966143Z Entering 'third_party/nccl/nccl' 2024-08-07T18:05:49.4998518Z http.https://github.com/.extraheader 2024-08-07T18:05:49.5031502Z Entering 'third_party/nlohmann' 2024-08-07T18:05:49.5063932Z http.https://github.com/.extraheader 2024-08-07T18:05:49.5097695Z Entering 'third_party/onnx' 2024-08-07T18:05:49.5129493Z http.https://github.com/.extraheader 2024-08-07T18:05:49.5180149Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-07T18:05:49.5211792Z http.https://github.com/.extraheader 2024-08-07T18:05:49.5246026Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-07T18:05:49.5278870Z http.https://github.com/.extraheader 2024-08-07T18:05:49.5314461Z Entering 'third_party/opentelemetry-cpp' 2024-08-07T18:05:49.5346471Z http.https://github.com/.extraheader 2024-08-07T18:05:49.5382857Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T18:05:49.5413803Z http.https://github.com/.extraheader 2024-08-07T18:05:49.5447257Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T18:05:49.5478881Z http.https://github.com/.extraheader 2024-08-07T18:05:49.5512880Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T18:05:49.5545488Z http.https://github.com/.extraheader 2024-08-07T18:05:49.5577950Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T18:05:49.5609720Z http.https://github.com/.extraheader 2024-08-07T18:05:49.5644334Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T18:05:49.5675667Z http.https://github.com/.extraheader 2024-08-07T18:05:49.5709349Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T18:05:49.5740330Z http.https://github.com/.extraheader 2024-08-07T18:05:49.5772802Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T18:05:49.5804628Z http.https://github.com/.extraheader 2024-08-07T18:05:49.5837405Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T18:05:49.5868704Z http.https://github.com/.extraheader 2024-08-07T18:05:49.5903875Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T18:05:49.5934136Z http.https://github.com/.extraheader 2024-08-07T18:05:49.5969205Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T18:05:49.6001531Z http.https://github.com/.extraheader 2024-08-07T18:05:49.6057276Z Entering 'third_party/pocketfft' 2024-08-07T18:05:49.6089413Z http.https://github.com/.extraheader 2024-08-07T18:05:49.6122198Z Entering 'third_party/protobuf' 2024-08-07T18:05:49.6154223Z http.https://github.com/.extraheader 2024-08-07T18:05:49.6190266Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-07T18:05:49.6223205Z http.https://github.com/.extraheader 2024-08-07T18:05:49.6255500Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-07T18:05:49.6287738Z http.https://github.com/.extraheader 2024-08-07T18:05:49.6323858Z Entering 'third_party/psimd' 2024-08-07T18:05:49.6355245Z http.https://github.com/.extraheader 2024-08-07T18:05:49.6388426Z Entering 'third_party/pthreadpool' 2024-08-07T18:05:49.6421471Z http.https://github.com/.extraheader 2024-08-07T18:05:49.6453980Z Entering 'third_party/pybind11' 2024-08-07T18:05:49.6486163Z http.https://github.com/.extraheader 2024-08-07T18:05:49.6519549Z Entering 'third_party/python-peachpy' 2024-08-07T18:05:49.6552165Z http.https://github.com/.extraheader 2024-08-07T18:05:49.6586138Z Entering 'third_party/sleef' 2024-08-07T18:05:49.6617944Z http.https://github.com/.extraheader 2024-08-07T18:05:49.6650830Z Entering 'third_party/tensorpipe' 2024-08-07T18:05:49.6682966Z http.https://github.com/.extraheader 2024-08-07T18:05:49.6715976Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-07T18:05:49.6749426Z http.https://github.com/.extraheader 2024-08-07T18:05:49.6781912Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-07T18:05:49.6813235Z http.https://github.com/.extraheader 2024-08-07T18:05:49.6845798Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-07T18:05:49.6877440Z http.https://github.com/.extraheader 2024-08-07T18:05:49.6910726Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T18:05:49.6942412Z http.https://github.com/.extraheader 2024-08-07T18:05:49.6974285Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T18:05:49.7006503Z http.https://github.com/.extraheader 2024-08-07T18:05:49.7223992Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-07T18:05:49.7271420Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-07T18:05:49.7278404Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:05:49.7278892Z ##[endgroup] 2024-08-07T18:05:55.4859254Z Cleaning up orphan processes