2024-10-08T22:01:09.1444503Z Current runner version: '2.320.0' 2024-10-08T22:01:09.1453163Z Runner name: 'i-0324360b5e1830423' 2024-10-08T22:01:09.1454109Z Runner group name: 'Default' 2024-10-08T22:01:09.1455291Z Machine name: 'ip-10-0-4-60' 2024-10-08T22:01:09.1479092Z Testing runner upgrade compatibility 2024-10-08T22:01:09.2791109Z ##[group]GITHUB_TOKEN Permissions 2024-10-08T22:01:09.2793891Z Actions: read 2024-10-08T22:01:09.2794490Z Attestations: read 2024-10-08T22:01:09.2795231Z Checks: read 2024-10-08T22:01:09.2795812Z Contents: read 2024-10-08T22:01:09.2796349Z Deployments: read 2024-10-08T22:01:09.2797019Z Discussions: read 2024-10-08T22:01:09.2797629Z Issues: read 2024-10-08T22:01:09.2798195Z Metadata: read 2024-10-08T22:01:09.2798782Z Packages: read 2024-10-08T22:01:09.2799379Z Pages: read 2024-10-08T22:01:09.2799944Z PullRequests: read 2024-10-08T22:01:09.2800572Z RepositoryProjects: read 2024-10-08T22:01:09.2801273Z SecurityEvents: read 2024-10-08T22:01:09.2801892Z Statuses: read 2024-10-08T22:01:09.2802735Z ##[endgroup] 2024-10-08T22:01:09.2807432Z Secret source: Actions 2024-10-08T22:01:09.2808513Z Prepare workflow directory 2024-10-08T22:01:09.4233749Z Prepare all required actions 2024-10-08T22:01:09.4477756Z Getting action download info 2024-10-08T22:01:09.6924758Z Download action repository 'pytorch/test-infra@main' (SHA:a148ec45e978a6dd650f2bde8050ece3439ba579) 2024-10-08T22:01:10.1987642Z Download action repository 'pytorch/pytorch@main' (SHA:81c8a8ada657136be90e124f3d9f6c193a2b58c2) 2024-10-08T22:01:14.4746404Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-10-08T22:01:14.6199828Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-10-08T22:01:15.0148453Z Getting action download info 2024-10-08T22:01:15.1267403Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-10-08T22:01:15.3501872Z Getting action download info 2024-10-08T22:01:15.4579859Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-10-08T22:01:15.6391071Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/134247/merge (cbe39ffc96498585a83508a23de66f3eecb751d5) 2024-10-08T22:01:15.6394788Z ##[group] Inputs 2024-10-08T22:01:15.6395366Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2024-10-08T22:01:15.6397467Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2024-10-08T22:01:15.6400220Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:01:15.6401569Z sync-tag: 2024-10-08T22:01:15.6402826Z timeout-minutes: 360 2024-10-08T22:01:15.6403244Z use-gha: 2024-10-08T22:01:15.6403606Z dashboard-tag: 2024-10-08T22:01:15.6404016Z s3-bucket: gha-artifacts 2024-10-08T22:01:15.6404467Z aws-role-to-assume: 2024-10-08T22:01:15.6404881Z ##[endgroup] 2024-10-08T22:01:15.6406198Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-10-08T22:01:15.7389384Z A job started hook has been configured by the self-hosted runner administrator 2024-10-08T22:01:15.7597113Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-10-08T22:01:15.7609508Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:01:15.7610193Z ##[endgroup] 2024-10-08T22:01:18.5964187Z Runner Type: linux.8xlarge.nvidia.gpu 2024-10-08T22:01:18.5964889Z Instance Type: g3.8xlarge 2024-10-08T22:01:18.5965347Z AMI Name: unknown 2024-10-08T22:01:18.5966031Z AMI ID: ami-0fff1b9a61dec8a5f 2024-10-08T22:01:25.9472632Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-10-08T22:01:25.9473652Z with: 2024-10-08T22:01:25.9474482Z github-secret: *** 2024-10-08T22:01:25.9475706Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-10-08T22:01:25.9476979Z activate-with-label: false 2024-10-08T22:01:25.9477432Z label: with-ssh 2024-10-08T22:01:25.9477834Z remove-existing-keys: true 2024-10-08T22:01:25.9478272Z fail-silently: true 2024-10-08T22:01:25.9478669Z env: 2024-10-08T22:01:25.9479022Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:01:25.9479435Z ##[endgroup] 2024-10-08T22:01:26.0687420Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-10-08T22:01:26.4975195Z Grabbing public ssh keys from https://github.com/kwen2501.keys 2024-10-08T22:01:26.5683568Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-10-08T22:01:26.5706616Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-10-08T22:01:26.5748827Z Login using: ssh ec2-user@ec2-3-208-34-64.compute-1.amazonaws.com 2024-10-08T22:01:26.5749808Z All testing is done inside the container, to start an interactive session run: 2024-10-08T22:01:26.5750854Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-10-08T22:01:26.5906232Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-10-08T22:01:26.5906944Z with: 2024-10-08T22:01:26.5907290Z submodules: recursive 2024-10-08T22:01:26.5907722Z fetch-depth: 0 2024-10-08T22:01:26.5908062Z env: 2024-10-08T22:01:26.5908394Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:01:26.5908818Z ##[endgroup] 2024-10-08T22:01:26.6177496Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-10-08T22:01:26.6179142Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-10-08T22:01:26.6187613Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:01:26.6188266Z env: 2024-10-08T22:01:26.6188627Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:01:26.6189056Z ##[endgroup] 2024-10-08T22:01:26.6302669Z ##[group]Run retry () { 2024-10-08T22:01:26.6303190Z retry () { 2024-10-08T22:01:26.6303821Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-10-08T22:01:26.6304501Z } 2024-10-08T22:01:26.6304881Z echo "${GITHUB_WORKSPACE}" 2024-10-08T22:01:26.6305412Z if [ -z "${NO_SUDO}" ]; then 2024-10-08T22:01:26.6305998Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-10-08T22:01:26.6306567Z else 2024-10-08T22:01:26.6306991Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-10-08T22:01:26.6307523Z fi 2024-10-08T22:01:26.6307891Z mkdir "${GITHUB_WORKSPACE}" 2024-10-08T22:01:26.6314965Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:01:26.6315601Z env: 2024-10-08T22:01:26.6315951Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:01:26.6316567Z NO_SUDO: 2024-10-08T22:01:26.6316897Z ##[endgroup] 2024-10-08T22:01:26.6346493Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-10-08T22:01:29.9668277Z ##[group]Run malfet/checkout@silent-checkout 2024-10-08T22:01:29.9668956Z with: 2024-10-08T22:01:29.9669577Z ref: 47952c0a8001fd60c9ffe52b121d022d440bef8b 2024-10-08T22:01:29.9670226Z fetch-depth: 0 2024-10-08T22:01:29.9670735Z submodules: recursive 2024-10-08T22:01:29.9671347Z quiet-checkout: true 2024-10-08T22:01:29.9671936Z repository: pytorch/pytorch 2024-10-08T22:01:29.9672701Z token: *** 2024-10-08T22:01:29.9673303Z ssh-strict: true 2024-10-08T22:01:29.9673842Z persist-credentials: true 2024-10-08T22:01:29.9674409Z clean: true 2024-10-08T22:01:29.9675051Z sparse-checkout-cone-mode: true 2024-10-08T22:01:29.9675662Z lfs: false 2024-10-08T22:01:29.9676154Z set-safe-directory: true 2024-10-08T22:01:29.9676993Z env: 2024-10-08T22:01:29.9677452Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:01:29.9678015Z ##[endgroup] 2024-10-08T22:01:30.1068853Z Syncing repository: pytorch/pytorch 2024-10-08T22:01:30.1071115Z ##[group]Getting Git version info 2024-10-08T22:01:30.1072140Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-10-08T22:01:30.1073248Z [command]/usr/bin/git version 2024-10-08T22:01:30.1073713Z git version 2.40.1 2024-10-08T22:01:30.1075857Z ##[endgroup] 2024-10-08T22:01:30.1098569Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/6b418829-b4c8-48ef-b4e8-6898bc8a997e' before making global git config changes 2024-10-08T22:01:30.1100168Z Adding repository directory to the temporary git global config as a safe directory 2024-10-08T22:01:30.1105755Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-10-08T22:01:30.1155443Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-10-08T22:01:30.1161554Z ##[group]Initializing the repository 2024-10-08T22:01:30.1166132Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-10-08T22:01:30.1203472Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-10-08T22:01:30.1205282Z hint: is subject to change. To configure the initial branch name to use in all 2024-10-08T22:01:30.1206360Z hint: of your new repositories, which will suppress this warning, call: 2024-10-08T22:01:30.1207058Z hint: 2024-10-08T22:01:30.1207655Z hint: git config --global init.defaultBranch 2024-10-08T22:01:30.1208245Z hint: 2024-10-08T22:01:30.1208902Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-10-08T22:01:30.1209965Z hint: 'development'. The just-created branch can be renamed via this command: 2024-10-08T22:01:30.1210711Z hint: 2024-10-08T22:01:30.1211119Z hint: git branch -m 2024-10-08T22:01:30.1212097Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-10-08T22:01:30.1218890Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-10-08T22:01:30.1255758Z ##[endgroup] 2024-10-08T22:01:30.1256478Z ##[group]Disabling automatic garbage collection 2024-10-08T22:01:30.1260329Z [command]/usr/bin/git config --local gc.auto 0 2024-10-08T22:01:30.1297088Z ##[endgroup] 2024-10-08T22:01:30.1297737Z ##[group]Setting up auth 2024-10-08T22:01:30.1306685Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-10-08T22:01:30.1344476Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-10-08T22:01:30.1681579Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-10-08T22:01:30.1716996Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-10-08T22:01:30.2049610Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-10-08T22:01:30.2104523Z ##[endgroup] 2024-10-08T22:01:30.2105232Z ##[group]Fetching the repository 2024-10-08T22:01:30.2113618Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-10-08T22:01:33.1252794Z remote: Enumerating objects: 1010997 2024-10-08T22:01:33.1253561Z remote: Enumerating objects: 1012271, done. 2024-10-08T22:01:33.1254668Z remote: Counting objects: 0% (1/1274) 2024-10-08T22:01:33.1255336Z remote: Counting objects: 1% (13/1274) 2024-10-08T22:01:33.1256005Z remote: Counting objects: 2% (26/1274) 2024-10-08T22:01:33.1256667Z remote: Counting objects: 3% (39/1274) 2024-10-08T22:01:33.1257824Z remote: Counting objects: 4% (51/1274) 2024-10-08T22:01:33.1258475Z remote: Counting objects: 5% (64/1274) 2024-10-08T22:01:33.1259098Z remote: Counting objects: 6% (77/1274) 2024-10-08T22:01:33.1259749Z remote: Counting objects: 7% (90/1274) 2024-10-08T22:01:33.1260430Z remote: Counting objects: 8% (102/1274) 2024-10-08T22:01:33.1261111Z remote: Counting objects: 9% (115/1274) 2024-10-08T22:01:33.1261741Z remote: Counting objects: 10% (128/1274) 2024-10-08T22:01:33.1262781Z remote: Counting objects: 11% (141/1274) 2024-10-08T22:01:33.1263449Z remote: Counting objects: 12% (153/1274) 2024-10-08T22:01:33.1264187Z remote: Counting objects: 13% (166/1274) 2024-10-08T22:01:33.1264870Z remote: Counting objects: 14% (179/1274) 2024-10-08T22:01:33.1265525Z remote: Counting objects: 15% (192/1274) 2024-10-08T22:01:33.1266170Z remote: Counting objects: 16% (204/1274) 2024-10-08T22:01:33.1266835Z remote: Counting objects: 17% (217/1274) 2024-10-08T22:01:33.1267469Z remote: Counting objects: 18% (230/1274) 2024-10-08T22:01:33.1268128Z remote: Counting objects: 19% (243/1274) 2024-10-08T22:01:33.1269036Z remote: Counting objects: 20% (255/1274) 2024-10-08T22:01:33.1269733Z remote: Counting objects: 21% (268/1274) 2024-10-08T22:01:33.1270356Z remote: Counting objects: 22% (281/1274) 2024-10-08T22:01:33.1271019Z remote: Counting objects: 23% (294/1274) 2024-10-08T22:01:33.1271670Z remote: Counting objects: 24% (306/1274) 2024-10-08T22:01:33.1272316Z remote: Counting objects: 25% (319/1274) 2024-10-08T22:01:33.1272942Z remote: Counting objects: 26% (332/1274) 2024-10-08T22:01:33.1273591Z remote: Counting objects: 27% (344/1274) 2024-10-08T22:01:33.1274237Z remote: Counting objects: 28% (357/1274) 2024-10-08T22:01:33.1274885Z remote: Counting objects: 29% (370/1274) 2024-10-08T22:01:33.1275520Z remote: Counting objects: 30% (383/1274) 2024-10-08T22:01:33.1276169Z remote: Counting objects: 31% (395/1274) 2024-10-08T22:01:33.1276820Z remote: Counting objects: 32% (408/1274) 2024-10-08T22:01:33.1277478Z remote: Counting objects: 33% (421/1274) 2024-10-08T22:01:33.1278101Z remote: Counting objects: 34% (434/1274) 2024-10-08T22:01:33.1278748Z remote: Counting objects: 35% (446/1274) 2024-10-08T22:01:33.1279390Z remote: Counting objects: 36% (459/1274) 2024-10-08T22:01:33.1280030Z remote: Counting objects: 37% (472/1274) 2024-10-08T22:01:33.1280654Z remote: Counting objects: 38% (485/1274) 2024-10-08T22:01:33.1281306Z remote: Counting objects: 39% (497/1274) 2024-10-08T22:01:33.1281959Z remote: Counting objects: 40% (510/1274) 2024-10-08T22:01:33.1282606Z remote: Counting objects: 41% (523/1274) 2024-10-08T22:01:33.1283227Z remote: Counting objects: 42% (536/1274) 2024-10-08T22:01:33.1283884Z remote: Counting objects: 43% (548/1274) 2024-10-08T22:01:33.1284524Z remote: Counting objects: 44% (561/1274) 2024-10-08T22:01:33.1285159Z remote: Counting objects: 45% (574/1274) 2024-10-08T22:01:33.1285784Z remote: Counting objects: 46% (587/1274) 2024-10-08T22:01:33.1286436Z remote: Counting objects: 47% (599/1274) 2024-10-08T22:01:33.1287089Z remote: Counting objects: 48% (612/1274) 2024-10-08T22:01:33.1287738Z remote: Counting objects: 49% (625/1274) 2024-10-08T22:01:33.1288358Z remote: Counting objects: 50% (637/1274) 2024-10-08T22:01:33.1289007Z remote: Counting objects: 51% (650/1274) 2024-10-08T22:01:33.1289653Z remote: Counting objects: 52% (663/1274) 2024-10-08T22:01:33.1290296Z remote: Counting objects: 53% (676/1274) 2024-10-08T22:01:33.1290915Z remote: Counting objects: 54% (688/1274) 2024-10-08T22:01:33.1291562Z remote: Counting objects: 55% (701/1274) 2024-10-08T22:01:33.1292394Z remote: Counting objects: 56% (714/1274) 2024-10-08T22:01:33.1293019Z remote: Counting objects: 57% (727/1274) 2024-10-08T22:01:33.1293667Z remote: Counting objects: 58% (739/1274) 2024-10-08T22:01:33.1294319Z remote: Counting objects: 59% (752/1274) 2024-10-08T22:01:33.1294969Z remote: Counting objects: 60% (765/1274) 2024-10-08T22:01:33.1295592Z remote: Counting objects: 61% (778/1274) 2024-10-08T22:01:33.1296237Z remote: Counting objects: 62% (790/1274) 2024-10-08T22:01:33.1296911Z remote: Counting objects: 63% (803/1274) 2024-10-08T22:01:33.1297567Z remote: Counting objects: 64% (816/1274) 2024-10-08T22:01:33.1298211Z remote: Counting objects: 65% (829/1274) 2024-10-08T22:01:33.1298833Z remote: Counting objects: 66% (841/1274) 2024-10-08T22:01:33.1299673Z remote: Counting objects: 67% (854/1274) 2024-10-08T22:01:33.1300331Z remote: Counting objects: 68% (867/1274) 2024-10-08T22:01:33.1300980Z remote: Counting objects: 69% (880/1274) 2024-10-08T22:01:33.1301649Z remote: Counting objects: 70% (892/1274) 2024-10-08T22:01:33.1302279Z remote: Counting objects: 71% (905/1274) 2024-10-08T22:01:33.1303040Z remote: Counting objects: 72% (918/1274) 2024-10-08T22:01:33.1303713Z remote: Counting objects: 73% (931/1274) 2024-10-08T22:01:33.1304352Z remote: Counting objects: 74% (943/1274) 2024-10-08T22:01:33.1304972Z remote: Counting objects: 75% (956/1274) 2024-10-08T22:01:33.1305617Z remote: Counting objects: 76% (969/1274) 2024-10-08T22:01:33.1306268Z remote: Counting objects: 77% (981/1274) 2024-10-08T22:01:33.1306910Z remote: Counting objects: 78% (994/1274) 2024-10-08T22:01:33.1307542Z remote: Counting objects: 79% (1007/1274) 2024-10-08T22:01:33.1308214Z remote: Counting objects: 80% (1020/1274) 2024-10-08T22:01:33.1308903Z remote: Counting objects: 81% (1032/1274) 2024-10-08T22:01:33.1309578Z remote: Counting objects: 82% (1045/1274) 2024-10-08T22:01:33.1310231Z remote: Counting objects: 83% (1058/1274) 2024-10-08T22:01:33.1310895Z remote: Counting objects: 84% (1071/1274) 2024-10-08T22:01:33.1311570Z remote: Counting objects: 85% (1083/1274) 2024-10-08T22:01:33.1312244Z remote: Counting objects: 86% (1096/1274) 2024-10-08T22:01:33.1312896Z remote: Counting objects: 87% (1109/1274) 2024-10-08T22:01:33.1313558Z remote: Counting objects: 88% (1122/1274) 2024-10-08T22:01:33.1314223Z remote: Counting objects: 89% (1134/1274) 2024-10-08T22:01:33.1314864Z remote: Counting objects: 90% (1147/1274) 2024-10-08T22:01:33.1315530Z remote: Counting objects: 91% (1160/1274) 2024-10-08T22:01:33.1316186Z remote: Counting objects: 92% (1173/1274) 2024-10-08T22:01:33.1316847Z remote: Counting objects: 93% (1185/1274) 2024-10-08T22:01:33.1317497Z remote: Counting objects: 94% (1198/1274) 2024-10-08T22:01:33.1318157Z remote: Counting objects: 95% (1211/1274) 2024-10-08T22:01:33.1318818Z remote: Counting objects: 96% (1224/1274) 2024-10-08T22:01:33.1319480Z remote: Counting objects: 97% (1236/1274) 2024-10-08T22:01:33.1320130Z remote: Counting objects: 98% (1249/1274) 2024-10-08T22:01:33.1320806Z remote: Counting objects: 99% (1262/1274) 2024-10-08T22:01:33.1321467Z remote: Counting objects: 100% (1274/1274) 2024-10-08T22:01:33.1322174Z remote: Counting objects: 100% (1274/1274), done. 2024-10-08T22:01:33.1447324Z remote: Compressing objects: 0% (1/616) 2024-10-08T22:01:33.1885976Z remote: Compressing objects: 1% (7/616) 2024-10-08T22:01:33.2179835Z remote: Compressing objects: 2% (13/616) 2024-10-08T22:01:33.2322365Z remote: Compressing objects: 3% (19/616) 2024-10-08T22:01:33.2536375Z remote: Compressing objects: 4% (25/616) 2024-10-08T22:01:33.3107322Z remote: Compressing objects: 5% (31/616) 2024-10-08T22:01:33.3899863Z remote: Compressing objects: 6% (37/616) 2024-10-08T22:01:33.4353536Z remote: Compressing objects: 7% (44/616) 2024-10-08T22:01:33.4720122Z remote: Compressing objects: 8% (50/616) 2024-10-08T22:01:33.4896750Z remote: Compressing objects: 9% (56/616) 2024-10-08T22:01:33.5012243Z remote: Compressing objects: 10% (62/616) 2024-10-08T22:01:33.5112927Z remote: Compressing objects: 11% (68/616) 2024-10-08T22:01:33.5309454Z remote: Compressing objects: 12% (74/616) 2024-10-08T22:01:33.5476751Z remote: Compressing objects: 13% (81/616) 2024-10-08T22:01:33.5617404Z remote: Compressing objects: 14% (87/616) 2024-10-08T22:01:33.5715353Z remote: Compressing objects: 15% (93/616) 2024-10-08T22:01:33.5836803Z remote: Compressing objects: 16% (99/616) 2024-10-08T22:01:33.5922867Z remote: Compressing objects: 17% (105/616) 2024-10-08T22:01:33.6014149Z remote: Compressing objects: 18% (111/616) 2024-10-08T22:01:33.6080572Z remote: Compressing objects: 19% (118/616) 2024-10-08T22:01:33.6138023Z remote: Compressing objects: 20% (124/616) 2024-10-08T22:01:33.6178924Z remote: Compressing objects: 21% (130/616) 2024-10-08T22:01:33.6199263Z remote: Compressing objects: 22% (136/616) 2024-10-08T22:01:33.6219637Z remote: Compressing objects: 23% (142/616) 2024-10-08T22:01:33.6232673Z remote: Compressing objects: 24% (148/616) 2024-10-08T22:01:33.6238172Z remote: Compressing objects: 25% (154/616) 2024-10-08T22:01:33.6240567Z remote: Compressing objects: 26% (161/616) 2024-10-08T22:01:33.6242413Z remote: Compressing objects: 27% (167/616) 2024-10-08T22:01:33.6248931Z remote: Compressing objects: 28% (173/616) 2024-10-08T22:01:33.6256104Z remote: Compressing objects: 29% (179/616) 2024-10-08T22:01:33.6260333Z remote: Compressing objects: 30% (185/616) 2024-10-08T22:01:33.6265793Z remote: Compressing objects: 31% (191/616) 2024-10-08T22:01:33.6269910Z remote: Compressing objects: 32% (198/616) 2024-10-08T22:01:33.6275989Z remote: Compressing objects: 33% (204/616) 2024-10-08T22:01:33.6289279Z remote: Compressing objects: 34% (210/616) 2024-10-08T22:01:33.6295546Z remote: Compressing objects: 35% (216/616) 2024-10-08T22:01:33.6304511Z remote: Compressing objects: 36% (222/616) 2024-10-08T22:01:33.6312638Z remote: Compressing objects: 37% (228/616) 2024-10-08T22:01:33.6317984Z remote: Compressing objects: 38% (235/616) 2024-10-08T22:01:33.6324185Z remote: Compressing objects: 39% (241/616) 2024-10-08T22:01:33.6330715Z remote: Compressing objects: 40% (247/616) 2024-10-08T22:01:33.6335382Z remote: Compressing objects: 41% (253/616) 2024-10-08T22:01:33.6339153Z remote: Compressing objects: 42% (259/616) 2024-10-08T22:01:33.6343621Z remote: Compressing objects: 43% (265/616) 2024-10-08T22:01:33.6346530Z remote: Compressing objects: 44% (272/616) 2024-10-08T22:01:33.6350005Z remote: Compressing objects: 45% (278/616) 2024-10-08T22:01:33.6352275Z remote: Compressing objects: 46% (284/616) 2024-10-08T22:01:33.6355403Z remote: Compressing objects: 47% (290/616) 2024-10-08T22:01:33.6357983Z remote: Compressing objects: 48% (296/616) 2024-10-08T22:01:33.6360386Z remote: Compressing objects: 49% (302/616) 2024-10-08T22:01:33.6363648Z remote: Compressing objects: 50% (308/616) 2024-10-08T22:01:33.6366311Z remote: Compressing objects: 51% (315/616) 2024-10-08T22:01:33.6368844Z remote: Compressing objects: 52% (321/616) 2024-10-08T22:01:33.6370759Z remote: Compressing objects: 53% (327/616) 2024-10-08T22:01:33.6372121Z remote: Compressing objects: 54% (333/616) 2024-10-08T22:01:33.6374655Z remote: Compressing objects: 55% (339/616) 2024-10-08T22:01:33.6375679Z remote: Compressing objects: 56% (345/616) 2024-10-08T22:01:33.6378420Z remote: Compressing objects: 57% (352/616) 2024-10-08T22:01:33.6379094Z remote: Compressing objects: 58% (358/616) 2024-10-08T22:01:33.6379774Z remote: Compressing objects: 59% (364/616) 2024-10-08T22:01:33.6380418Z remote: Compressing objects: 60% (370/616) 2024-10-08T22:01:33.6386316Z remote: Compressing objects: 61% (376/616) 2024-10-08T22:01:33.6392012Z remote: Compressing objects: 62% (382/616) 2024-10-08T22:01:33.6392692Z remote: Compressing objects: 63% (389/616) 2024-10-08T22:01:33.6393334Z remote: Compressing objects: 64% (395/616) 2024-10-08T22:01:33.6393992Z remote: Compressing objects: 65% (401/616) 2024-10-08T22:01:33.6398345Z remote: Compressing objects: 66% (407/616) 2024-10-08T22:01:33.6404205Z remote: Compressing objects: 67% (413/616) 2024-10-08T22:01:33.6410616Z remote: Compressing objects: 68% (419/616) 2024-10-08T22:01:33.6415317Z remote: Compressing objects: 69% (426/616) 2024-10-08T22:01:33.6415992Z remote: Compressing objects: 70% (432/616) 2024-10-08T22:01:33.6417355Z remote: Compressing objects: 71% (438/616) 2024-10-08T22:01:33.6418195Z remote: Compressing objects: 72% (444/616) 2024-10-08T22:01:33.6418873Z remote: Compressing objects: 73% (450/616) 2024-10-08T22:01:33.6420880Z remote: Compressing objects: 74% (456/616) 2024-10-08T22:01:33.6423307Z remote: Compressing objects: 75% (462/616) 2024-10-08T22:01:33.6424607Z remote: Compressing objects: 76% (469/616) 2024-10-08T22:01:33.6426840Z remote: Compressing objects: 77% (475/616) 2024-10-08T22:01:33.6429122Z remote: Compressing objects: 78% (481/616) 2024-10-08T22:01:33.6430902Z remote: Compressing objects: 79% (487/616) 2024-10-08T22:01:33.6435220Z remote: Compressing objects: 80% (493/616) 2024-10-08T22:01:33.6437550Z remote: Compressing objects: 81% (499/616) 2024-10-08T22:01:33.6439320Z remote: Compressing objects: 82% (506/616) 2024-10-08T22:01:33.6442084Z remote: Compressing objects: 83% (512/616) 2024-10-08T22:01:33.6443759Z remote: Compressing objects: 84% (518/616) 2024-10-08T22:01:33.6445601Z remote: Compressing objects: 85% (524/616) 2024-10-08T22:01:33.6447499Z remote: Compressing objects: 86% (530/616) 2024-10-08T22:01:33.6449252Z remote: Compressing objects: 87% (536/616) 2024-10-08T22:01:33.6450571Z remote: Compressing objects: 88% (543/616) 2024-10-08T22:01:33.6452520Z remote: Compressing objects: 89% (549/616) 2024-10-08T22:01:33.6453690Z remote: Compressing objects: 90% (555/616) 2024-10-08T22:01:33.6455731Z remote: Compressing objects: 91% (561/616) 2024-10-08T22:01:33.6457063Z remote: Compressing objects: 92% (567/616) 2024-10-08T22:01:33.6458850Z remote: Compressing objects: 93% (573/616) 2024-10-08T22:01:33.6459992Z remote: Compressing objects: 94% (580/616) 2024-10-08T22:01:33.6462734Z remote: Compressing objects: 95% (586/616) 2024-10-08T22:01:33.6463819Z remote: Compressing objects: 96% (592/616) 2024-10-08T22:01:33.6464863Z remote: Compressing objects: 97% (598/616) 2024-10-08T22:01:33.6465930Z remote: Compressing objects: 98% (604/616) 2024-10-08T22:01:33.6467717Z remote: Compressing objects: 99% (610/616) 2024-10-08T22:01:33.6468389Z remote: Compressing objects: 100% (616/616) 2024-10-08T22:01:33.6469085Z remote: Compressing objects: 100% (616/616), done. 2024-10-08T22:01:59.9796988Z remote: Total 1012271 (delta 772), reused 1072 (delta 655), pack-reused 1010997 (from 1) 2024-10-08T22:02:21.9511727Z [command]/usr/bin/git rev-parse --verify --quiet 47952c0a8001fd60c9ffe52b121d022d440bef8b^{object} 2024-10-08T22:02:21.9545410Z 47952c0a8001fd60c9ffe52b121d022d440bef8b 2024-10-08T22:02:21.9550900Z ##[endgroup] 2024-10-08T22:02:21.9551563Z ##[group]Determining the checkout info 2024-10-08T22:02:21.9553874Z ##[endgroup] 2024-10-08T22:02:21.9556485Z ##[group]Checking out the ref 2024-10-08T22:02:21.9559002Z [command]/usr/bin/git checkout --quiet --force 47952c0a8001fd60c9ffe52b121d022d440bef8b 2024-10-08T22:02:23.9172109Z ##[endgroup] 2024-10-08T22:02:23.9172874Z ##[group]Setting up auth for fetching submodules 2024-10-08T22:02:23.9179468Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-10-08T22:02:23.9256502Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-10-08T22:02:23.9294957Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-10-08T22:02:23.9333624Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-10-08T22:02:23.9368601Z ##[endgroup] 2024-10-08T22:02:23.9369254Z ##[group]Fetching submodules 2024-10-08T22:02:23.9374321Z [command]/usr/bin/git submodule sync --recursive 2024-10-08T22:02:23.9735800Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-10-08T22:02:24.0087951Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-10-08T22:02:24.0090571Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-10-08T22:02:24.0094775Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-10-08T22:02:24.0098539Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-10-08T22:02:24.0102771Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2024-10-08T22:02:24.0107495Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-10-08T22:02:24.0111577Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-10-08T22:02:24.0116150Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-10-08T22:02:24.0120775Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-10-08T22:02:24.0125607Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-10-08T22:02:24.0130699Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-10-08T22:02:24.0135455Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-10-08T22:02:24.0140513Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-10-08T22:02:24.0145640Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-10-08T22:02:24.0151176Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-10-08T22:02:24.0156343Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-10-08T22:02:24.0165032Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-10-08T22:02:24.0170492Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-10-08T22:02:24.0176316Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-10-08T22:02:24.0182344Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-10-08T22:02:24.0188396Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-10-08T22:02:24.0194543Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-10-08T22:02:24.0200901Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-10-08T22:02:24.0207100Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-10-08T22:02:24.0213674Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-10-08T22:02:24.0220081Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-10-08T22:02:24.0227254Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-10-08T22:02:24.0233783Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-10-08T22:02:24.0240950Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-10-08T22:02:24.0247871Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-10-08T22:02:24.0255119Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-10-08T22:02:24.0262687Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-10-08T22:02:24.0270491Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-10-08T22:02:24.0279643Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-10-08T22:02:24.0287556Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-10-08T22:02:24.0320098Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-10-08T22:02:24.3902529Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-10-08T22:02:24.6128164Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-10-08T22:02:24.8638667Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-10-08T22:02:25.1652341Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2024-10-08T22:02:25.5230555Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-10-08T22:02:27.6649895Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-10-08T22:02:39.0030254Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-10-08T22:02:39.4873074Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-10-08T22:02:39.9991160Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-10-08T22:02:40.7329099Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-10-08T22:02:42.2150856Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-10-08T22:02:44.4057287Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-10-08T22:02:51.3116653Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-10-08T22:02:52.9855038Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-10-08T22:02:54.9543584Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-10-08T22:02:56.3720641Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-10-08T22:02:56.9048406Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-10-08T22:02:57.3155966Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-10-08T22:02:58.3989586Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-10-08T22:02:58.7937706Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-10-08T22:02:59.0761778Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-10-08T22:03:00.9608184Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-10-08T22:03:01.8072374Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-10-08T22:03:02.3121276Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-10-08T22:03:09.2192793Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-10-08T22:03:11.8076266Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-10-08T22:03:17.5214003Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-10-08T22:03:17.8283946Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-10-08T22:03:28.0793128Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-10-08T22:03:28.2820947Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-10-08T22:03:28.4949173Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-10-08T22:03:29.6553867Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-10-08T22:03:30.0578729Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-10-08T22:03:30.8752383Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-10-08T22:03:31.3604234Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-10-08T22:03:31.3752187Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-10-08T22:03:31.3863741Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-10-08T22:03:31.4199469Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-10-08T22:03:31.4625934Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1' 2024-10-08T22:03:31.5104229Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-10-08T22:03:32.6702632Z Submodule path 'third_party/XNNPACK': checked out '87ee0b46b834f67bad9025d4a82ed5654f3403d3' 2024-10-08T22:03:32.7007232Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-10-08T22:03:32.7603464Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-10-08T22:03:32.8863231Z Submodule path 'third_party/cpuinfo': checked out 'a5ff6df40ce528721cfc310c7ed43946d77404d5' 2024-10-08T22:03:32.9301738Z Submodule path 'third_party/cudnn_frontend': checked out '2533f5e5c1877fd76266133c1479ef1643ce3a8b' 2024-10-08T22:03:33.6215196Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-10-08T22:03:33.9565401Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-10-08T22:03:34.0719672Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-10-08T22:03:34.0740386Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-10-08T22:03:34.0744358Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-10-08T22:03:34.0748282Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-10-08T22:03:34.0752277Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-10-08T22:03:34.0756589Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-10-08T22:03:34.0788801Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-10-08T22:03:35.2879421Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-10-08T22:03:35.9682081Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-10-08T22:03:38.2072818Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-10-08T22:03:39.3265310Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-10-08T22:03:39.7207326Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-10-08T22:03:39.8449445Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-10-08T22:03:40.4120042Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-10-08T22:03:40.4927432Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-10-08T22:03:40.5082091Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-10-08T22:03:40.6702555Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-10-08T22:03:40.7174322Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-10-08T22:03:40.7682315Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-10-08T22:03:40.8042113Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-10-08T22:03:40.8628585Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-10-08T22:03:40.8793328Z Submodule path 'third_party/ideep': checked out '41d636c2bbcea6bff0faf97cdb65a48cdde987af' 2024-10-08T22:03:40.8812987Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-10-08T22:03:40.8843714Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-10-08T22:03:54.2450409Z Submodule path 'third_party/ideep/mkl-dnn': checked out '66f0cb9eb66affd2da3bf5f8d897376f04aae6af' 2024-10-08T22:03:54.2681199Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-10-08T22:03:54.3816899Z Submodule path 'third_party/kineto': checked out 'b5c85daac1ee123aa7f04eb6f2bc71363f429e68' 2024-10-08T22:03:54.3838625Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-10-08T22:03:54.3842059Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-10-08T22:03:54.3846778Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-10-08T22:03:54.3878379Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-10-08T22:03:54.9991196Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-10-08T22:03:56.4402065Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-10-08T22:03:57.6032222Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-10-08T22:03:57.6053638Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-10-08T22:03:57.6057859Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-10-08T22:03:57.6061924Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-10-08T22:03:57.6067130Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-10-08T22:03:57.6071106Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-10-08T22:03:57.6075743Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-10-08T22:03:57.6080300Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-10-08T22:03:57.6084945Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-10-08T22:03:57.6117471Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-10-08T22:03:58.6276706Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-10-08T22:03:59.0901446Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-10-08T22:04:00.5386445Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-10-08T22:04:00.9320296Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-10-08T22:04:01.6432193Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-10-08T22:04:02.8196156Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-10-08T22:04:09.9337183Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-10-08T22:04:10.4002968Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-10-08T22:04:10.4255587Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-10-08T22:04:10.4738901Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-10-08T22:04:10.4908647Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-10-08T22:04:10.4928489Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-10-08T22:04:10.4961709Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-10-08T22:04:10.8419052Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-10-08T22:04:10.8658380Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-10-08T22:04:10.9189819Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-10-08T22:04:11.0640594Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-10-08T22:04:11.0857810Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-10-08T22:04:11.1340937Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-10-08T22:04:11.2088995Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-10-08T22:04:11.2588091Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-10-08T22:04:11.2971426Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-10-08T22:04:11.4410806Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-10-08T22:04:11.9636880Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-10-08T22:04:11.9678247Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-10-08T22:04:11.9682578Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-10-08T22:04:11.9715259Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-10-08T22:04:12.5193019Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-10-08T22:04:13.7228397Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-10-08T22:04:13.7683867Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-10-08T22:04:13.8663075Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-10-08T22:04:13.8686188Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-10-08T22:04:13.8689991Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-10-08T22:04:13.8693800Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-10-08T22:04:13.8697892Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-10-08T22:04:13.8702389Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-10-08T22:04:13.8706193Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-10-08T22:04:13.8710861Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-10-08T22:04:13.8715021Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-10-08T22:04:13.8746896Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-10-08T22:04:14.3696855Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-10-08T22:04:15.4278975Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-10-08T22:04:15.8859768Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-10-08T22:04:22.8407609Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-10-08T22:04:23.1738840Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-10-08T22:04:23.4198442Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-10-08T22:04:23.7563844Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-10-08T22:04:31.4975581Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-10-08T22:04:31.5503176Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-10-08T22:04:31.5699688Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-10-08T22:04:31.7083463Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-10-08T22:04:31.7253720Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-10-08T22:04:31.7448394Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-10-08T22:04:31.7667001Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-10-08T22:04:31.7686732Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-10-08T22:04:31.7690764Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-10-08T22:04:31.7720476Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-10-08T22:04:33.7774530Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-10-08T22:04:35.2065662Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-10-08T22:04:35.2681528Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-10-08T22:04:35.9678240Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-10-08T22:04:35.9823478Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-10-08T22:04:36.3605637Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-10-08T22:04:36.3632937Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-10-08T22:04:36.3636852Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-10-08T22:04:36.3671780Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-10-08T22:04:36.9191461Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-10-08T22:04:38.0262870Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-10-08T22:04:38.1191486Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-10-08T22:04:38.1301237Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-10-08T22:04:38.1459549Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-10-08T22:04:38.1921989Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2024-10-08T22:04:38.2289232Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-10-08T22:04:38.2828977Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-10-08T22:04:38.3185720Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-10-08T22:04:38.3206371Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-10-08T22:04:38.3210136Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-10-08T22:04:38.3214160Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-10-08T22:04:38.3218221Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-10-08T22:04:38.3248680Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-10-08T22:04:39.4370456Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-10-08T22:04:39.6930995Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-10-08T22:04:41.1492264Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-10-08T22:04:42.3990490Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-10-08T22:04:42.4189143Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-10-08T22:04:42.5088334Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-10-08T22:04:42.5453542Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-10-08T22:04:42.5475645Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-10-08T22:04:42.5506592Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-10-08T22:04:42.7702187Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-10-08T22:04:42.7743352Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-10-08T22:04:42.8105618Z Entering 'android/libs/fbjni' 2024-10-08T22:04:42.8155707Z Entering 'third_party/FP16' 2024-10-08T22:04:42.8206782Z Entering 'third_party/FXdiv' 2024-10-08T22:04:42.8255258Z Entering 'third_party/NNPACK' 2024-10-08T22:04:42.8305372Z Entering 'third_party/NVTX' 2024-10-08T22:04:42.8355595Z Entering 'third_party/VulkanMemoryAllocator' 2024-10-08T22:04:42.8405422Z Entering 'third_party/XNNPACK' 2024-10-08T22:04:42.8476926Z Entering 'third_party/benchmark' 2024-10-08T22:04:42.8526001Z Entering 'third_party/cpp-httplib' 2024-10-08T22:04:42.8577708Z Entering 'third_party/cpuinfo' 2024-10-08T22:04:42.8627130Z Entering 'third_party/cudnn_frontend' 2024-10-08T22:04:42.8677610Z Entering 'third_party/cutlass' 2024-10-08T22:04:42.8735876Z Entering 'third_party/eigen' 2024-10-08T22:04:42.8789324Z Entering 'third_party/fbgemm' 2024-10-08T22:04:42.8840064Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-10-08T22:04:42.8889113Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-10-08T22:04:42.8938503Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-10-08T22:04:42.8997331Z Entering 'third_party/fbgemm/third_party/googletest' 2024-10-08T22:04:42.9044660Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-10-08T22:04:42.9097044Z Entering 'third_party/flatbuffers' 2024-10-08T22:04:42.9147997Z Entering 'third_party/fmt' 2024-10-08T22:04:42.9197501Z Entering 'third_party/gemmlowp/gemmlowp' 2024-10-08T22:04:42.9246061Z Entering 'third_party/gloo' 2024-10-08T22:04:42.9296163Z Entering 'third_party/googletest' 2024-10-08T22:04:42.9344431Z Entering 'third_party/ideep' 2024-10-08T22:04:42.9395171Z Entering 'third_party/ideep/mkl-dnn' 2024-10-08T22:04:42.9452700Z Entering 'third_party/ittapi' 2024-10-08T22:04:42.9502176Z Entering 'third_party/kineto' 2024-10-08T22:04:42.9550523Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-10-08T22:04:42.9600151Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-10-08T22:04:42.9651488Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-10-08T22:04:42.9703198Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-10-08T22:04:42.9753734Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-10-08T22:04:42.9804393Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-10-08T22:04:42.9855982Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-10-08T22:04:42.9907106Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-10-08T22:04:42.9957082Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-10-08T22:04:43.0007797Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-10-08T22:04:43.0059617Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-10-08T22:04:43.0108277Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-10-08T22:04:43.0157861Z Entering 'third_party/mimalloc' 2024-10-08T22:04:43.0208075Z Entering 'third_party/nccl/nccl' 2024-10-08T22:04:43.0257386Z Entering 'third_party/nlohmann' 2024-10-08T22:04:43.0308539Z Entering 'third_party/onnx' 2024-10-08T22:04:43.0376653Z Entering 'third_party/onnx/third_party/benchmark' 2024-10-08T22:04:43.0425885Z Entering 'third_party/onnx/third_party/pybind11' 2024-10-08T22:04:43.0478943Z Entering 'third_party/opentelemetry-cpp' 2024-10-08T22:04:43.0531087Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-10-08T22:04:43.0580346Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-10-08T22:04:43.0629066Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-10-08T22:04:43.0677465Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-10-08T22:04:43.0727794Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-10-08T22:04:43.0777588Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-10-08T22:04:43.0825656Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-10-08T22:04:43.0874457Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-10-08T22:04:43.0925668Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-10-08T22:04:43.0977146Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-10-08T22:04:43.1046158Z Entering 'third_party/pocketfft' 2024-10-08T22:04:43.1095048Z Entering 'third_party/protobuf' 2024-10-08T22:04:43.1148174Z Entering 'third_party/protobuf/third_party/benchmark' 2024-10-08T22:04:43.1196878Z Entering 'third_party/protobuf/third_party/googletest' 2024-10-08T22:04:43.1247592Z Entering 'third_party/psimd' 2024-10-08T22:04:43.1298193Z Entering 'third_party/pthreadpool' 2024-10-08T22:04:43.1346044Z Entering 'third_party/pybind11' 2024-10-08T22:04:43.1395740Z Entering 'third_party/python-peachpy' 2024-10-08T22:04:43.1444533Z Entering 'third_party/sleef' 2024-10-08T22:04:43.1495314Z Entering 'third_party/tensorpipe' 2024-10-08T22:04:43.1543953Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-10-08T22:04:43.1593468Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-10-08T22:04:43.1640618Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-10-08T22:04:43.1690990Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-10-08T22:04:43.1739122Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-10-08T22:04:43.1805751Z ##[endgroup] 2024-10-08T22:04:43.1810460Z ##[group]Persisting credentials for submodules 2024-10-08T22:04:43.1812855Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-10-08T22:04:43.2168543Z Entering 'android/libs/fbjni' 2024-10-08T22:04:43.2232749Z Entering 'third_party/FP16' 2024-10-08T22:04:43.2298141Z Entering 'third_party/FXdiv' 2024-10-08T22:04:43.2363221Z Entering 'third_party/NNPACK' 2024-10-08T22:04:43.2427821Z Entering 'third_party/NVTX' 2024-10-08T22:04:43.2493140Z Entering 'third_party/VulkanMemoryAllocator' 2024-10-08T22:04:43.2557792Z Entering 'third_party/XNNPACK' 2024-10-08T22:04:43.2641927Z Entering 'third_party/benchmark' 2024-10-08T22:04:43.2706572Z Entering 'third_party/cpp-httplib' 2024-10-08T22:04:43.2771800Z Entering 'third_party/cpuinfo' 2024-10-08T22:04:43.2835086Z Entering 'third_party/cudnn_frontend' 2024-10-08T22:04:43.2899919Z Entering 'third_party/cutlass' 2024-10-08T22:04:43.2974149Z Entering 'third_party/eigen' 2024-10-08T22:04:43.3040790Z Entering 'third_party/fbgemm' 2024-10-08T22:04:43.3105090Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-10-08T22:04:43.3169259Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-10-08T22:04:43.3231567Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-10-08T22:04:43.3302363Z Entering 'third_party/fbgemm/third_party/googletest' 2024-10-08T22:04:43.3364922Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-10-08T22:04:43.3428774Z Entering 'third_party/flatbuffers' 2024-10-08T22:04:43.3499162Z Entering 'third_party/fmt' 2024-10-08T22:04:43.3561605Z Entering 'third_party/gemmlowp/gemmlowp' 2024-10-08T22:04:43.3625286Z Entering 'third_party/gloo' 2024-10-08T22:04:43.3689938Z Entering 'third_party/googletest' 2024-10-08T22:04:43.3753180Z Entering 'third_party/ideep' 2024-10-08T22:04:43.3817121Z Entering 'third_party/ideep/mkl-dnn' 2024-10-08T22:04:43.3889406Z Entering 'third_party/ittapi' 2024-10-08T22:04:43.3952691Z Entering 'third_party/kineto' 2024-10-08T22:04:43.4016662Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-10-08T22:04:43.4079833Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-10-08T22:04:43.4144921Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-10-08T22:04:43.4209934Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-10-08T22:04:43.4274654Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-10-08T22:04:43.4338687Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-10-08T22:04:43.4405970Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-10-08T22:04:43.4470370Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-10-08T22:04:43.4534090Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-10-08T22:04:43.4599758Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-10-08T22:04:43.4666569Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-10-08T22:04:43.4729977Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-10-08T22:04:43.4796282Z Entering 'third_party/mimalloc' 2024-10-08T22:04:43.4859045Z Entering 'third_party/nccl/nccl' 2024-10-08T22:04:43.4923097Z Entering 'third_party/nlohmann' 2024-10-08T22:04:43.4987940Z Entering 'third_party/onnx' 2024-10-08T22:04:43.5069740Z Entering 'third_party/onnx/third_party/benchmark' 2024-10-08T22:04:43.5133579Z Entering 'third_party/onnx/third_party/pybind11' 2024-10-08T22:04:43.5200730Z Entering 'third_party/opentelemetry-cpp' 2024-10-08T22:04:43.5265840Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-10-08T22:04:43.5329080Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-10-08T22:04:43.5392269Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-10-08T22:04:43.5454524Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-10-08T22:04:43.5519978Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-10-08T22:04:43.5582596Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-10-08T22:04:43.5645818Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-10-08T22:04:43.5708845Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-10-08T22:04:43.5773795Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-10-08T22:04:43.5837775Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-10-08T22:04:43.5924256Z Entering 'third_party/pocketfft' 2024-10-08T22:04:43.5988879Z Entering 'third_party/protobuf' 2024-10-08T22:04:43.6055486Z Entering 'third_party/protobuf/third_party/benchmark' 2024-10-08T22:04:43.6119853Z Entering 'third_party/protobuf/third_party/googletest' 2024-10-08T22:04:43.6184932Z Entering 'third_party/psimd' 2024-10-08T22:04:43.6248433Z Entering 'third_party/pthreadpool' 2024-10-08T22:04:43.6312188Z Entering 'third_party/pybind11' 2024-10-08T22:04:43.6377447Z Entering 'third_party/python-peachpy' 2024-10-08T22:04:43.6440067Z Entering 'third_party/sleef' 2024-10-08T22:04:43.6504563Z Entering 'third_party/tensorpipe' 2024-10-08T22:04:43.6568071Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-10-08T22:04:43.6631337Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-10-08T22:04:43.6695168Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-10-08T22:04:43.6758080Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-10-08T22:04:43.6820196Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-10-08T22:04:43.6903911Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-10-08T22:04:43.7255021Z Entering 'android/libs/fbjni' 2024-10-08T22:04:43.7315191Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-10-08T22:04:43.7333791Z Entering 'third_party/FP16' 2024-10-08T22:04:43.7394071Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-10-08T22:04:43.7412737Z Entering 'third_party/FXdiv' 2024-10-08T22:04:43.7473729Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-10-08T22:04:43.7492156Z Entering 'third_party/NNPACK' 2024-10-08T22:04:43.7551132Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-10-08T22:04:43.7571484Z Entering 'third_party/NVTX' 2024-10-08T22:04:43.7630470Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2024-10-08T22:04:43.7650239Z Entering 'third_party/VulkanMemoryAllocator' 2024-10-08T22:04:43.7713002Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-10-08T22:04:43.7731727Z Entering 'third_party/XNNPACK' 2024-10-08T22:04:43.7791594Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-10-08T22:04:43.7829345Z Entering 'third_party/benchmark' 2024-10-08T22:04:43.7890026Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-10-08T22:04:43.7908960Z Entering 'third_party/cpp-httplib' 2024-10-08T22:04:43.7969197Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-10-08T22:04:43.7988059Z Entering 'third_party/cpuinfo' 2024-10-08T22:04:43.8047855Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-10-08T22:04:43.8068336Z Entering 'third_party/cudnn_frontend' 2024-10-08T22:04:43.8128991Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-10-08T22:04:43.8147901Z Entering 'third_party/cutlass' 2024-10-08T22:04:43.8208218Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-10-08T22:04:43.8236035Z Entering 'third_party/eigen' 2024-10-08T22:04:43.8295979Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-10-08T22:04:43.8317206Z Entering 'third_party/fbgemm' 2024-10-08T22:04:43.8381037Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-10-08T22:04:43.8399637Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-10-08T22:04:43.8460447Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-10-08T22:04:43.8479430Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-10-08T22:04:43.8538669Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-10-08T22:04:43.8557223Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-10-08T22:04:43.8618426Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-10-08T22:04:43.8644899Z Entering 'third_party/fbgemm/third_party/googletest' 2024-10-08T22:04:43.8704536Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-10-08T22:04:43.8722903Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-10-08T22:04:43.8782932Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-10-08T22:04:43.8802782Z Entering 'third_party/flatbuffers' 2024-10-08T22:04:43.8862868Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-10-08T22:04:43.8884595Z Entering 'third_party/fmt' 2024-10-08T22:04:43.8944367Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-10-08T22:04:43.8965338Z Entering 'third_party/gemmlowp/gemmlowp' 2024-10-08T22:04:43.9024381Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-10-08T22:04:43.9043416Z Entering 'third_party/gloo' 2024-10-08T22:04:43.9103174Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-10-08T22:04:43.9122920Z Entering 'third_party/googletest' 2024-10-08T22:04:43.9182595Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-10-08T22:04:43.9201677Z Entering 'third_party/ideep' 2024-10-08T22:04:43.9262084Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-10-08T22:04:43.9280544Z Entering 'third_party/ideep/mkl-dnn' 2024-10-08T22:04:43.9339472Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-10-08T22:04:43.9366687Z Entering 'third_party/ittapi' 2024-10-08T22:04:43.9425822Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-10-08T22:04:43.9444532Z Entering 'third_party/kineto' 2024-10-08T22:04:43.9505665Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-10-08T22:04:43.9524022Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-10-08T22:04:43.9597925Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-10-08T22:04:43.9615621Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-10-08T22:04:43.9676387Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-10-08T22:04:43.9696666Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-10-08T22:04:43.9757500Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-10-08T22:04:43.9777435Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-10-08T22:04:43.9837494Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-10-08T22:04:43.9855815Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-10-08T22:04:43.9918180Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-10-08T22:04:43.9935769Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-10-08T22:04:43.9999137Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-10-08T22:04:44.0019622Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-10-08T22:04:44.0079798Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-10-08T22:04:44.0098285Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-10-08T22:04:44.0158259Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-10-08T22:04:44.0177911Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-10-08T22:04:44.0237802Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-10-08T22:04:44.0258077Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-10-08T22:04:44.0320981Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-10-08T22:04:44.0341227Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-10-08T22:04:44.0403675Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-10-08T22:04:44.0421460Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-10-08T22:04:44.0483028Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-10-08T22:04:44.0503605Z Entering 'third_party/mimalloc' 2024-10-08T22:04:44.0563813Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-10-08T22:04:44.0582818Z Entering 'third_party/nccl/nccl' 2024-10-08T22:04:44.0643078Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-10-08T22:04:44.0662775Z Entering 'third_party/nlohmann' 2024-10-08T22:04:44.0723564Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-10-08T22:04:44.0743947Z Entering 'third_party/onnx' 2024-10-08T22:04:44.0806725Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-10-08T22:04:44.0842762Z Entering 'third_party/onnx/third_party/benchmark' 2024-10-08T22:04:44.0904269Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-10-08T22:04:44.0922961Z Entering 'third_party/onnx/third_party/pybind11' 2024-10-08T22:04:44.0984966Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-10-08T22:04:44.1005929Z Entering 'third_party/opentelemetry-cpp' 2024-10-08T22:04:44.1068206Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-10-08T22:04:44.1088895Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-10-08T22:04:44.1148690Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-10-08T22:04:44.1168835Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-10-08T22:04:44.1227788Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-10-08T22:04:44.1247050Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-10-08T22:04:44.1311854Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-10-08T22:04:44.1329968Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-10-08T22:04:44.1391428Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-10-08T22:04:44.1411299Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-10-08T22:04:44.1473965Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-10-08T22:04:44.1492590Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-10-08T22:04:44.1552439Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-10-08T22:04:44.1571331Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-10-08T22:04:44.1631359Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-10-08T22:04:44.1648761Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-10-08T22:04:44.1710904Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-10-08T22:04:44.1731283Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-10-08T22:04:44.1792447Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-10-08T22:04:44.1812482Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-10-08T22:04:44.1873726Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-10-08T22:04:44.1915680Z Entering 'third_party/pocketfft' 2024-10-08T22:04:44.1976408Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-10-08T22:04:44.1995178Z Entering 'third_party/protobuf' 2024-10-08T22:04:44.2054811Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-10-08T22:04:44.2078064Z Entering 'third_party/protobuf/third_party/benchmark' 2024-10-08T22:04:44.2137654Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-10-08T22:04:44.2155899Z Entering 'third_party/protobuf/third_party/googletest' 2024-10-08T22:04:44.2217828Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-10-08T22:04:44.2238146Z Entering 'third_party/psimd' 2024-10-08T22:04:44.2299027Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-10-08T22:04:44.2318124Z Entering 'third_party/pthreadpool' 2024-10-08T22:04:44.2379171Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-10-08T22:04:44.2397803Z Entering 'third_party/pybind11' 2024-10-08T22:04:44.2457514Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-10-08T22:04:44.2477738Z Entering 'third_party/python-peachpy' 2024-10-08T22:04:44.2537364Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-10-08T22:04:44.2556033Z Entering 'third_party/sleef' 2024-10-08T22:04:44.2616619Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-10-08T22:04:44.2635883Z Entering 'third_party/tensorpipe' 2024-10-08T22:04:44.2697104Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-10-08T22:04:44.2715492Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-10-08T22:04:44.2777439Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-10-08T22:04:44.2796110Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-10-08T22:04:44.2854983Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-10-08T22:04:44.2874090Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-10-08T22:04:44.2933172Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-10-08T22:04:44.2951855Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-10-08T22:04:44.3013095Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-10-08T22:04:44.3030540Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-10-08T22:04:44.3093453Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-10-08T22:04:44.6549222Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-10-08T22:04:44.6921081Z Entering 'android/libs/fbjni' 2024-10-08T22:04:44.6971522Z Entering 'third_party/FP16' 2024-10-08T22:04:44.7020955Z Entering 'third_party/FXdiv' 2024-10-08T22:04:44.7071181Z Entering 'third_party/NNPACK' 2024-10-08T22:04:44.7121732Z Entering 'third_party/NVTX' 2024-10-08T22:04:44.7172169Z Entering 'third_party/VulkanMemoryAllocator' 2024-10-08T22:04:44.7222117Z Entering 'third_party/XNNPACK' 2024-10-08T22:04:44.7290982Z Entering 'third_party/benchmark' 2024-10-08T22:04:44.7340364Z Entering 'third_party/cpp-httplib' 2024-10-08T22:04:44.7392127Z Entering 'third_party/cpuinfo' 2024-10-08T22:04:44.7441273Z Entering 'third_party/cudnn_frontend' 2024-10-08T22:04:44.7492035Z Entering 'third_party/cutlass' 2024-10-08T22:04:44.7548692Z Entering 'third_party/eigen' 2024-10-08T22:04:44.7601594Z Entering 'third_party/fbgemm' 2024-10-08T22:04:44.7651260Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-10-08T22:04:44.7699999Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-10-08T22:04:44.7748730Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-10-08T22:04:44.7805734Z Entering 'third_party/fbgemm/third_party/googletest' 2024-10-08T22:04:44.7853629Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-10-08T22:04:44.7904525Z Entering 'third_party/flatbuffers' 2024-10-08T22:04:44.7956477Z Entering 'third_party/fmt' 2024-10-08T22:04:44.8006561Z Entering 'third_party/gemmlowp/gemmlowp' 2024-10-08T22:04:44.8055439Z Entering 'third_party/gloo' 2024-10-08T22:04:44.8106851Z Entering 'third_party/googletest' 2024-10-08T22:04:44.8155939Z Entering 'third_party/ideep' 2024-10-08T22:04:44.8205170Z Entering 'third_party/ideep/mkl-dnn' 2024-10-08T22:04:44.8261873Z Entering 'third_party/ittapi' 2024-10-08T22:04:44.8311933Z Entering 'third_party/kineto' 2024-10-08T22:04:44.8361583Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-10-08T22:04:44.8410853Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-10-08T22:04:44.8463081Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-10-08T22:04:44.8512355Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-10-08T22:04:44.8561330Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-10-08T22:04:44.8611311Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-10-08T22:04:44.8666507Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-10-08T22:04:44.8715024Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-10-08T22:04:44.8767465Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-10-08T22:04:44.8818338Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-10-08T22:04:44.8869869Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-10-08T22:04:44.8917987Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-10-08T22:04:44.8969577Z Entering 'third_party/mimalloc' 2024-10-08T22:04:44.9018586Z Entering 'third_party/nccl/nccl' 2024-10-08T22:04:44.9068109Z Entering 'third_party/nlohmann' 2024-10-08T22:04:44.9117999Z Entering 'third_party/onnx' 2024-10-08T22:04:44.9185860Z Entering 'third_party/onnx/third_party/benchmark' 2024-10-08T22:04:44.9234177Z Entering 'third_party/onnx/third_party/pybind11' 2024-10-08T22:04:44.9287021Z Entering 'third_party/opentelemetry-cpp' 2024-10-08T22:04:44.9338058Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-10-08T22:04:44.9388382Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-10-08T22:04:44.9436231Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-10-08T22:04:44.9486166Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-10-08T22:04:44.9537370Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-10-08T22:04:44.9585630Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-10-08T22:04:44.9633301Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-10-08T22:04:44.9683485Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-10-08T22:04:44.9734798Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-10-08T22:04:44.9786012Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-10-08T22:04:44.9858847Z Entering 'third_party/pocketfft' 2024-10-08T22:04:44.9908491Z Entering 'third_party/protobuf' 2024-10-08T22:04:44.9961931Z Entering 'third_party/protobuf/third_party/benchmark' 2024-10-08T22:04:45.0011245Z Entering 'third_party/protobuf/third_party/googletest' 2024-10-08T22:04:45.0062074Z Entering 'third_party/psimd' 2024-10-08T22:04:45.0111430Z Entering 'third_party/pthreadpool' 2024-10-08T22:04:45.0161519Z Entering 'third_party/pybind11' 2024-10-08T22:04:45.0212316Z Entering 'third_party/python-peachpy' 2024-10-08T22:04:45.0261445Z Entering 'third_party/sleef' 2024-10-08T22:04:45.0313649Z Entering 'third_party/tensorpipe' 2024-10-08T22:04:45.0363498Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-10-08T22:04:45.0413239Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-10-08T22:04:45.0460624Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-10-08T22:04:45.0510019Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-10-08T22:04:45.0558478Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-10-08T22:04:45.0627755Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-10-08T22:04:45.0986339Z Entering 'android/libs/fbjni' 2024-10-08T22:04:45.1035607Z Entering 'third_party/FP16' 2024-10-08T22:04:45.1086449Z Entering 'third_party/FXdiv' 2024-10-08T22:04:45.1136382Z Entering 'third_party/NNPACK' 2024-10-08T22:04:45.1186799Z Entering 'third_party/NVTX' 2024-10-08T22:04:45.1237035Z Entering 'third_party/VulkanMemoryAllocator' 2024-10-08T22:04:45.1287115Z Entering 'third_party/XNNPACK' 2024-10-08T22:04:45.1355179Z Entering 'third_party/benchmark' 2024-10-08T22:04:45.1405397Z Entering 'third_party/cpp-httplib' 2024-10-08T22:04:45.1455253Z Entering 'third_party/cpuinfo' 2024-10-08T22:04:45.1505884Z Entering 'third_party/cudnn_frontend' 2024-10-08T22:04:45.1555474Z Entering 'third_party/cutlass' 2024-10-08T22:04:45.1614721Z Entering 'third_party/eigen' 2024-10-08T22:04:45.1666843Z Entering 'third_party/fbgemm' 2024-10-08T22:04:45.1716263Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-10-08T22:04:45.1764806Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-10-08T22:04:45.1812518Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-10-08T22:04:45.1867870Z Entering 'third_party/fbgemm/third_party/googletest' 2024-10-08T22:04:45.1915699Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-10-08T22:04:45.1965757Z Entering 'third_party/flatbuffers' 2024-10-08T22:04:45.2017106Z Entering 'third_party/fmt' 2024-10-08T22:04:45.2066141Z Entering 'third_party/gemmlowp/gemmlowp' 2024-10-08T22:04:45.2114800Z Entering 'third_party/gloo' 2024-10-08T22:04:45.2164721Z Entering 'third_party/googletest' 2024-10-08T22:04:45.2214045Z Entering 'third_party/ideep' 2024-10-08T22:04:45.2261589Z Entering 'third_party/ideep/mkl-dnn' 2024-10-08T22:04:45.2318963Z Entering 'third_party/ittapi' 2024-10-08T22:04:45.2368500Z Entering 'third_party/kineto' 2024-10-08T22:04:45.2417397Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-10-08T22:04:45.2467024Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-10-08T22:04:45.2517105Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-10-08T22:04:45.2568897Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-10-08T22:04:45.2618242Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-10-08T22:04:45.2668431Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-10-08T22:04:45.2719850Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-10-08T22:04:45.2769509Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-10-08T22:04:45.2818541Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-10-08T22:04:45.2869795Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-10-08T22:04:45.2920568Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-10-08T22:04:45.2969691Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-10-08T22:04:45.3019446Z Entering 'third_party/mimalloc' 2024-10-08T22:04:45.3070314Z Entering 'third_party/nccl/nccl' 2024-10-08T22:04:45.3119625Z Entering 'third_party/nlohmann' 2024-10-08T22:04:45.3171188Z Entering 'third_party/onnx' 2024-10-08T22:04:45.3238128Z Entering 'third_party/onnx/third_party/benchmark' 2024-10-08T22:04:45.3287477Z Entering 'third_party/onnx/third_party/pybind11' 2024-10-08T22:04:45.3337739Z Entering 'third_party/opentelemetry-cpp' 2024-10-08T22:04:45.3389492Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-10-08T22:04:45.3436705Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-10-08T22:04:45.3486003Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-10-08T22:04:45.3533158Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-10-08T22:04:45.3584313Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-10-08T22:04:45.3631455Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-10-08T22:04:45.3680530Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-10-08T22:04:45.3729491Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-10-08T22:04:45.3781406Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-10-08T22:04:45.3831704Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-10-08T22:04:45.3905420Z Entering 'third_party/pocketfft' 2024-10-08T22:04:45.3953581Z Entering 'third_party/protobuf' 2024-10-08T22:04:45.4006421Z Entering 'third_party/protobuf/third_party/benchmark' 2024-10-08T22:04:45.4054804Z Entering 'third_party/protobuf/third_party/googletest' 2024-10-08T22:04:45.4106229Z Entering 'third_party/psimd' 2024-10-08T22:04:45.4155232Z Entering 'third_party/pthreadpool' 2024-10-08T22:04:45.4204858Z Entering 'third_party/pybind11' 2024-10-08T22:04:45.4253892Z Entering 'third_party/python-peachpy' 2024-10-08T22:04:45.4303935Z Entering 'third_party/sleef' 2024-10-08T22:04:45.4353806Z Entering 'third_party/tensorpipe' 2024-10-08T22:04:45.4404316Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-10-08T22:04:45.4452160Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-10-08T22:04:45.4500172Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-10-08T22:04:45.4547901Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-10-08T22:04:45.4596349Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-10-08T22:04:45.4660445Z ##[endgroup] 2024-10-08T22:04:45.4710034Z [command]/usr/bin/git log -1 --format='%H' 2024-10-08T22:04:45.4750249Z '47952c0a8001fd60c9ffe52b121d022d440bef8b' 2024-10-08T22:04:45.4985247Z Prepare all required actions 2024-10-08T22:04:45.4986008Z Getting action download info 2024-10-08T22:04:45.6349120Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2024-10-08T22:04:45.7869576Z ##[group]Run ./.github/actions/setup-linux 2024-10-08T22:04:45.7870075Z env: 2024-10-08T22:04:45.7870391Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:04:45.7870799Z ##[endgroup] 2024-10-08T22:04:45.7954444Z ##[group]Run set -euo pipefail 2024-10-08T22:04:45.7954945Z set -euo pipefail 2024-10-08T22:04:45.7955413Z function get_ec2_metadata() { 2024-10-08T22:04:45.7956075Z  # Pulled from instance metadata endpoint for EC2 2024-10-08T22:04:45.7957201Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-10-08T22:04:45.7958162Z  category=$1 2024-10-08T22:04:45.7958788Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-10-08T22:04:45.7959595Z  runner_name_str=i-0324360b5e1830423 2024-10-08T22:04:45.7960154Z  if [[ -f /.inarc ]]; then 2024-10-08T22:04:45.7960767Z  echo "ARC Runner, no info on ec2 metadata" 2024-10-08T22:04:45.7961470Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-10-08T22:04:45.7962905Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-10-08T22:04:45.7963680Z  else 2024-10-08T22:04:45.7964297Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-10-08T22:04:45.7964988Z  fi 2024-10-08T22:04:45.7965328Z } 2024-10-08T22:04:45.7965762Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-10-08T22:04:45.7966473Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-10-08T22:04:45.7967262Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-10-08T22:04:45.7967953Z echo "system info $(uname -a)" 2024-10-08T22:04:45.7975043Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:04:45.7975652Z env: 2024-10-08T22:04:45.7975960Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:04:45.7976369Z ##[endgroup] 2024-10-08T22:04:45.8087006Z ami-id: ami-0fff1b9a61dec8a5f 2024-10-08T22:04:45.8148163Z instance-id: i-0324360b5e1830423 2024-10-08T22:04:45.8209220Z instance-type: g3.8xlarge 2024-10-08T22:04:45.8222750Z system info Linux ip-10-0-4-60.ec2.internal 6.1.109-118.189.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 10 08:59:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-10-08T22:04:45.8251772Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-10-08T22:04:45.8253398Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-10-08T22:04:45.8260445Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:04:45.8261074Z env: 2024-10-08T22:04:45.8261401Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:04:45.8261835Z ##[endgroup] 2024-10-08T22:04:45.8330821Z ##[group]Run if systemctl is-active --quiet docker; then 2024-10-08T22:04:45.8331572Z if systemctl is-active --quiet docker; then 2024-10-08T22:04:45.8332251Z  echo "Docker daemon is running..."; 2024-10-08T22:04:45.8332811Z else 2024-10-08T22:04:45.8333411Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-10-08T22:04:45.8334139Z fi 2024-10-08T22:04:45.8341086Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:04:45.8341917Z env: 2024-10-08T22:04:45.8342243Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:04:45.8342690Z ##[endgroup] 2024-10-08T22:04:45.8428773Z Docker daemon is running... 2024-10-08T22:04:45.8487818Z ##[group]Run nick-fields/retry@v3.0.0 2024-10-08T22:04:45.8488318Z with: 2024-10-08T22:04:45.8488646Z shell: bash 2024-10-08T22:04:45.8489197Z timeout_minutes: 5 2024-10-08T22:04:45.8489629Z max_attempts: 3 2024-10-08T22:04:45.8490018Z retry_wait_seconds: 30 2024-10-08T22:04:45.8493777Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-10-08T22:04:45.8497613Z polling_interval_seconds: 1 2024-10-08T22:04:45.8498085Z warning_on_retry: true 2024-10-08T22:04:45.8498521Z continue_on_error: false 2024-10-08T22:04:45.8498921Z env: 2024-10-08T22:04:45.8499257Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:04:45.8499707Z AWS_RETRY_MODE: standard 2024-10-08T22:04:45.8500126Z AWS_MAX_ATTEMPTS: 5 2024-10-08T22:04:45.8500554Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T22:04:45.8501019Z ##[endgroup] 2024-10-08T22:04:47.4129731Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-10-08T22:04:47.4130781Z Configure a credential helper to remove this warning. See 2024-10-08T22:04:47.4131876Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-10-08T22:04:47.4132540Z 2024-10-08T22:04:47.4132692Z Login Succeeded 2024-10-08T22:04:48.0767212Z Command completed after 1 attempt(s). 2024-10-08T22:04:48.0846137Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-10-08T22:04:48.0847043Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-10-08T22:04:48.0847861Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-10-08T22:04:48.0855718Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:04:48.0856333Z env: 2024-10-08T22:04:48.0856680Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:04:48.0857112Z ##[endgroup] 2024-10-08T22:04:48.0960367Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-10-08T22:04:48.0961331Z # ignore expansion of "docker ps -q" since it could be empty 2024-10-08T22:04:48.0962072Z # shellcheck disable=SC2046 2024-10-08T22:04:48.0963078Z docker stop $(docker ps -q) || true 2024-10-08T22:04:48.0963664Z # Prune all of the docker images 2024-10-08T22:04:48.0964225Z docker system prune -af 2024-10-08T22:04:48.0971209Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:04:48.0971851Z env: 2024-10-08T22:04:48.0972193Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:04:48.0972627Z ##[endgroup] 2024-10-08T22:04:48.1329198Z "docker stop" requires at least 1 argument. 2024-10-08T22:04:48.1330155Z See 'docker stop --help'. 2024-10-08T22:04:48.1330479Z 2024-10-08T22:04:48.1330776Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-10-08T22:04:48.1331249Z 2024-10-08T22:04:48.1331450Z Stop one or more running containers 2024-10-08T22:04:51.2343447Z Deleted Images: 2024-10-08T22:04:51.2345294Z untagged: us-central1-docker.pkg.dev/tpu-pytorch-releases/docker/development:3.10_cuda_12.1 2024-10-08T22:04:51.2347219Z untagged: us-central1-docker.pkg.dev/tpu-pytorch-releases/docker/development@sha256:6e7523903615eac9d6ff045da8cf1d46c77c1a70d1e52636fc397df8ea7a7030 2024-10-08T22:04:51.2348809Z deleted: sha256:a532ad67a5dc2b6c7d97486b5929efb880b318b41d427724c4cdd6b7496fd6d2 2024-10-08T22:04:51.2349900Z deleted: sha256:0363735bb2cbbffbf426a24c69d4ed073c0801d09009692ea433ca2cfc69bd02 2024-10-08T22:04:51.2351295Z deleted: sha256:77fcfcd10107b4aea41fd73e800b845827d38d8a3469df1ec9b66106f779a808 2024-10-08T22:04:51.2352358Z deleted: sha256:05f1b0b5b303248b8b7d92949ad6f4537f5452302744911d383ec5983b976876 2024-10-08T22:04:51.2353638Z deleted: sha256:9bc6024d9c99b34c243fa36942735efc4b22fc161d494cf28e83dfab08c25f29 2024-10-08T22:04:51.2354747Z deleted: sha256:e246b313af6d2d6fa787ea389a31c45f39239afd0d24a8a1fa6fa134bbd3201d 2024-10-08T22:04:51.2355834Z deleted: sha256:c2d5c8a5cd84fefdd669475ba1a9b7e50438bad3e48b3aa93d77dc7e2c096f1b 2024-10-08T22:04:51.2356907Z deleted: sha256:bc1118f3d53d88e5d83d94a0c5c48dadbdb58468a00fabfebb95c8944d06d632 2024-10-08T22:04:51.2357987Z deleted: sha256:8d7228f5dda113d6804f85d2e2ed5755dac0e11d054a6f6b2705ee871b45e51b 2024-10-08T22:04:51.2359070Z deleted: sha256:f0d2d3d9c91f87cc4df0b7ac99f62ad58eb47eeac99e3ee21850647ffabe8ac0 2024-10-08T22:04:51.2360146Z deleted: sha256:f406734d5dc2ea1df8eb0d2288e9be4a854640442a93a14084358ff1527beced 2024-10-08T22:04:51.2361201Z deleted: sha256:315317d32d9b3cadda3f89b1c2c172b6973b6fe080b131bf8d25252036d8d0f5 2024-10-08T22:04:51.2361860Z 2024-10-08T22:04:51.2394796Z Total reclaimed space: 12.95GB 2024-10-08T22:04:51.2439928Z ##[group]Run set +e 2024-10-08T22:04:51.2440366Z set +e 2024-10-08T22:04:51.2440747Z set -x 2024-10-08T22:04:51.2441106Z  2024-10-08T22:04:51.2493348Z PT_DOMAIN=download.pytorch.org 2024-10-08T22:04:51.2494382Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-10-08T22:04:51.2495731Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-10-08T22:04:51.2496684Z # one is returned at random 2024-10-08T22:04:51.2497360Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-10-08T22:04:51.2497996Z  2024-10-08T22:04:51.2498361Z if [ -z "${RESOLVED_IP}" ]; then 2024-10-08T22:04:51.2499150Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-10-08T22:04:51.2500094Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-10-08T22:04:51.2500781Z  2024-10-08T22:04:51.2501174Z  if [ -z "${RESOLVED_IP}" ]; then 2024-10-08T22:04:51.2501850Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-10-08T22:04:51.2502483Z  exit 1 2024-10-08T22:04:51.2502866Z  fi 2024-10-08T22:04:51.2503196Z fi 2024-10-08T22:04:51.2503535Z  2024-10-08T22:04:51.2503972Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-10-08T22:04:51.2504608Z  # Clean up any old records first 2024-10-08T22:04:51.2505216Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-10-08T22:04:51.2505774Z fi 2024-10-08T22:04:51.2506110Z  2024-10-08T22:04:51.2506640Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-10-08T22:04:51.2507313Z cat /etc/hosts 2024-10-08T22:04:51.2514639Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:04:51.2515259Z env: 2024-10-08T22:04:51.2515598Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:04:51.2516011Z ##[endgroup] 2024-10-08T22:04:51.2546358Z + PT_DOMAIN=download.pytorch.org 2024-10-08T22:04:51.2553049Z ++ dig -4 +short download.pytorch.org 2024-10-08T22:04:51.2553874Z ++ tail -n1 2024-10-08T22:04:51.3186015Z + RESOLVED_IP=18.160.10.22 2024-10-08T22:04:51.3186604Z + '[' -z 18.160.10.22 ']' 2024-10-08T22:04:51.3187120Z + grep -r download.pytorch.org /etc/hosts 2024-10-08T22:04:51.3200614Z 108.138.64.61 download.pytorch.org 2024-10-08T22:04:51.3202595Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-10-08T22:04:51.4977813Z + echo '18.160.10.22 download.pytorch.org' 2024-10-08T22:04:51.4978484Z + sudo tee -a /etc/hosts 2024-10-08T22:04:51.5523599Z 18.160.10.22 download.pytorch.org 2024-10-08T22:04:51.5542822Z + cat /etc/hosts 2024-10-08T22:04:51.5552726Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-10-08T22:04:51.5570838Z ::1 localhost6 localhost6.localdomain6 2024-10-08T22:04:51.5571466Z 18.160.10.22 download.pytorch.org 2024-10-08T22:04:51.5770103Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-10-08T22:04:51.5770898Z with: 2024-10-08T22:04:51.5772120Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:04:51.5773503Z docker-build-dir: .ci/docker 2024-10-08T22:04:51.5773974Z working-directory: . 2024-10-08T22:04:51.5774561Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T22:04:51.5775227Z force-push: false 2024-10-08T22:04:51.5775585Z env: 2024-10-08T22:04:51.5775923Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:04:51.5776355Z ##[endgroup] 2024-10-08T22:04:51.5803908Z ##[group]Run set -ex 2024-10-08T22:04:51.5804388Z set -ex 2024-10-08T22:04:51.5804762Z  2024-10-08T22:04:51.5805441Z # If the docker build directory or the build script doesn't exist, the action will 2024-10-08T22:04:51.5806697Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-10-08T22:04:51.5807713Z # job could then download the pre-built image as usual 2024-10-08T22:04:51.5808624Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-10-08T22:04:51.5809468Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-10-08T22:04:51.5810260Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-10-08T22:04:51.5810955Z  2024-10-08T22:04:51.5811575Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-10-08T22:04:51.5812357Z  exit 0 2024-10-08T22:04:51.5812726Z else 2024-10-08T22:04:51.5813177Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-10-08T22:04:51.5813726Z fi 2024-10-08T22:04:51.5814063Z  2024-10-08T22:04:51.5814641Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-10-08T22:04:51.5815710Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-10-08T22:04:51.5816670Z  # use it as it is, but first let's extract the tag 2024-10-08T22:04:51.5817560Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-10-08T22:04:51.5818465Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-10-08T22:04:51.5819339Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-10-08T22:04:51.5820018Z else 2024-10-08T22:04:51.5820560Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-10-08T22:04:51.5821390Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-10-08T22:04:51.5822509Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-10-08T22:04:51.5823456Z fi 2024-10-08T22:04:51.5831280Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:04:51.5831913Z env: 2024-10-08T22:04:51.5832250Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:04:51.5832673Z REPO_NAME: pytorch 2024-10-08T22:04:51.5833930Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:04:51.5835281Z DOCKER_BUILD_DIR: .ci/docker 2024-10-08T22:04:51.5835904Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T22:04:51.5836550Z ##[endgroup] 2024-10-08T22:04:51.5865790Z + [[ ! -d .ci/docker ]] 2024-10-08T22:04:51.5866322Z + [[ ! -f .ci/docker/build.sh ]] 2024-10-08T22:04:51.5866783Z + echo skip=false 2024-10-08T22:04:51.5869413Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:46f3eb402e47f9fae93ec1dc1f6772950f4498aa == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-10-08T22:04:51.5875590Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:04:51.5877006Z ++ awk -F '[:,]' '{print $2}' 2024-10-08T22:04:51.5899881Z + DOCKER_TAG=46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:04:51.5900737Z + echo docker-tag=46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:04:51.5902925Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:04:51.5938722Z ##[group]Run set +e 2024-10-08T22:04:51.5939190Z set +e 2024-10-08T22:04:51.5939564Z set -x 2024-10-08T22:04:51.5939925Z  2024-10-08T22:04:51.5940250Z login() { 2024-10-08T22:04:51.5941092Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-10-08T22:04:51.5941994Z } 2024-10-08T22:04:51.5942320Z  2024-10-08T22:04:51.5942675Z retry () { 2024-10-08T22:04:51.5943169Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-10-08T22:04:51.5943722Z } 2024-10-08T22:04:51.5944044Z  2024-10-08T22:04:51.5944431Z retry login "${DOCKER_REGISTRY}" 2024-10-08T22:04:51.5944966Z  2024-10-08T22:04:51.5945561Z # Check if image already exists, if it does then skip building it 2024-10-08T22:04:51.5946450Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-10-08T22:04:51.5947080Z  exit 0 2024-10-08T22:04:51.5947440Z fi 2024-10-08T22:04:51.5947780Z  2024-10-08T22:04:51.5948409Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-10-08T22:04:51.5949448Z # be empty. The default action would be to continue rebuild the image 2024-10-08T22:04:51.5950368Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-10-08T22:04:51.5951222Z  # if we're on the base branch then use the parent commit 2024-10-08T22:04:51.5951963Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-10-08T22:04:51.5952503Z else 2024-10-08T22:04:51.5953075Z  # otherwise we're on a PR, so use the most recent base commit 2024-10-08T22:04:51.5953936Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-10-08T22:04:51.5954570Z fi 2024-10-08T22:04:51.5954910Z  2024-10-08T22:04:51.5955288Z if [[ -z "${MERGE_BASE}" ]]; then 2024-10-08T22:04:51.5955924Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-10-08T22:04:51.5956490Z  2024-10-08T22:04:51.5957315Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-10-08T22:04:51.5958262Z  exit 0 2024-10-08T22:04:51.5958634Z fi 2024-10-08T22:04:51.5958976Z  2024-10-08T22:04:51.5959517Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-10-08T22:04:51.5960737Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-10-08T22:04:51.5961758Z  exit 1 2024-10-08T22:04:51.5962553Z fi 2024-10-08T22:04:51.5962912Z  2024-10-08T22:04:51.5963509Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-10-08T22:04:51.5964715Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-10-08T22:04:51.5965805Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-10-08T22:04:51.5967028Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-10-08T22:04:51.5968599Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-10-08T22:04:51.5969392Z fi 2024-10-08T22:04:51.5969735Z  2024-10-08T22:04:51.5970314Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-10-08T22:04:51.5976935Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:04:51.5977540Z env: 2024-10-08T22:04:51.5977882Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:04:51.5978342Z DOCKER_BUILD_DIR: .ci/docker 2024-10-08T22:04:51.5978905Z BASE_REVISION: 05ad01290ec4310464e5073140f6c94a10acf565 2024-10-08T22:04:51.5980344Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:04:51.5981764Z DOCKER_TAG: 46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:04:51.5982522Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T22:04:51.5983169Z ##[endgroup] 2024-10-08T22:04:51.6012045Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T22:04:51.6012879Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T22:04:51.6015247Z + aws ecr get-login-password --region us-east-1 2024-10-08T22:04:51.6017008Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T22:04:52.2837089Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-10-08T22:04:52.2838124Z Configure a credential helper to remove this warning. See 2024-10-08T22:04:52.2839225Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-10-08T22:04:52.2839915Z 2024-10-08T22:04:52.2840063Z Login Succeeded 2024-10-08T22:04:52.2853949Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:04:52.4949206Z { 2024-10-08T22:04:52.4949603Z "schemaVersion": 2, 2024-10-08T22:04:52.4950314Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-10-08T22:04:52.4951069Z "config": { 2024-10-08T22:04:52.4951660Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-10-08T22:04:52.4952343Z "size": 49266, 2024-10-08T22:04:52.4953017Z "digest": "sha256:afe1462db4393617da580c2f21c43484a88ab03cdc72dd8fbaa63722368879a8" 2024-10-08T22:04:52.4953793Z }, 2024-10-08T22:04:52.4954104Z "layers": [ 2024-10-08T22:04:52.4954446Z { 2024-10-08T22:04:52.4954968Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.4955666Z "size": 28580681, 2024-10-08T22:04:52.4956364Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-10-08T22:04:52.4957124Z }, 2024-10-08T22:04:52.4957423Z { 2024-10-08T22:04:52.4957971Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.4958686Z "size": 7943353, 2024-10-08T22:04:52.4959362Z "digest": "sha256:09d415c238d76b32a7ea4a6e6add9542db9a5641f7f183af70aae185d0709e58" 2024-10-08T22:04:52.4960162Z }, 2024-10-08T22:04:52.4960468Z { 2024-10-08T22:04:52.4961030Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.4961719Z "size": 56226389, 2024-10-08T22:04:52.4962960Z "digest": "sha256:9fe6e2e61518cba6844870c03b285737daec35e62baf25ae7744629ed3a7b470" 2024-10-08T22:04:52.4964147Z }, 2024-10-08T22:04:52.4964458Z { 2024-10-08T22:04:52.4964984Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.4965680Z "size": 187, 2024-10-08T22:04:52.4966370Z "digest": "sha256:41f16248e682693ff20b3032c1d5e5541cc87c5af898ae2ff9b24d2940e59100" 2024-10-08T22:04:52.4967166Z }, 2024-10-08T22:04:52.4967450Z { 2024-10-08T22:04:52.4967991Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.4968692Z "size": 6885, 2024-10-08T22:04:52.4969372Z "digest": "sha256:95d7b781703928cf3c4eece39d800cccb76728c375fedf51ecd83833fb25e458" 2024-10-08T22:04:52.4970577Z + exit 0 2024-10-08T22:04:52.4970903Z }, 2024-10-08T22:04:52.4971202Z { 2024-10-08T22:04:52.4971882Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.4972605Z "size": 1383662475, 2024-10-08T22:04:52.4973280Z "digest": "sha256:8f6c9048534734f4c873935293b7296225846ceb31c1a158400a67ea170dde7f" 2024-10-08T22:04:52.4974024Z }, 2024-10-08T22:04:52.4974306Z { 2024-10-08T22:04:52.4974844Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.4975537Z "size": 62282, 2024-10-08T22:04:52.4976205Z "digest": "sha256:ab17245097e491b9368790714f9d90ed447bf0973bd677cfe6f2456d62b72a13" 2024-10-08T22:04:52.4976943Z }, 2024-10-08T22:04:52.4977246Z { 2024-10-08T22:04:52.4977789Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.4978482Z "size": 1684, 2024-10-08T22:04:52.4979153Z "digest": "sha256:dfecd7e9912b76ed460b8edd5a85f1943666e38a973ab5458177cf2c7c3110e3" 2024-10-08T22:04:52.4979948Z }, 2024-10-08T22:04:52.4980249Z { 2024-10-08T22:04:52.4980769Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.4981469Z "size": 1522, 2024-10-08T22:04:52.4982156Z "digest": "sha256:464a8f74544589bf7b57f9a4cadcb6681e5ed00758f6c35025e691df4e88e890" 2024-10-08T22:04:52.4982929Z }, 2024-10-08T22:04:52.4983212Z { 2024-10-08T22:04:52.4983750Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.4984457Z "size": 2445884909, 2024-10-08T22:04:52.4985149Z "digest": "sha256:d67c111fa58886d8904335b5122039816c8e51af5844cb6fe4e14154d9a7c016" 2024-10-08T22:04:52.4985887Z }, 2024-10-08T22:04:52.4986191Z { 2024-10-08T22:04:52.4986783Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.4987481Z "size": 85882, 2024-10-08T22:04:52.4988148Z "digest": "sha256:0723abde19c09431db80b5d0f0e003541f6232c2da5c4f35c635c861875937ae" 2024-10-08T22:04:52.4988935Z }, 2024-10-08T22:04:52.4989236Z { 2024-10-08T22:04:52.4989755Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.4990450Z "size": 1822, 2024-10-08T22:04:52.4991146Z "digest": "sha256:bfa46deece966df100d20bd622a98fa6ff3770e47dd69751803beca8582f0648" 2024-10-08T22:04:52.4991930Z }, 2024-10-08T22:04:52.4992212Z { 2024-10-08T22:04:52.4992755Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.4993450Z "size": 655141581, 2024-10-08T22:04:52.4994159Z "digest": "sha256:d993aa29d70c9f463289c409f4a47017fbbeb211cc7c90d6aeb7246d05b51a81" 2024-10-08T22:04:52.4994920Z }, 2024-10-08T22:04:52.4995223Z { 2024-10-08T22:04:52.4995758Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.4996438Z "size": 543, 2024-10-08T22:04:52.4997108Z "digest": "sha256:3704d1191479183dc75b012d39a3acba92bf13f2cc7b3ce3f8bd39327302a22a" 2024-10-08T22:04:52.4997876Z }, 2024-10-08T22:04:52.4998182Z { 2024-10-08T22:04:52.4998701Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.4999394Z "size": 1258, 2024-10-08T22:04:52.5000089Z "digest": "sha256:59545c272cb5976b98cd6e0cb68bd4bd3179abccb6ccad318a0636b3dc029371" 2024-10-08T22:04:52.5000880Z }, 2024-10-08T22:04:52.5001162Z { 2024-10-08T22:04:52.5001700Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5002393Z "size": 485, 2024-10-08T22:04:52.5003083Z "digest": "sha256:efd563baaa50e28af9ce4b5d59f05f229869802459f90decdaf928b3a404e7b3" 2024-10-08T22:04:52.5003851Z }, 2024-10-08T22:04:52.5004151Z { 2024-10-08T22:04:52.5004687Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5005365Z "size": 91712693, 2024-10-08T22:04:52.5006063Z "digest": "sha256:5972678dbf61fff62de0d2c01732f16c617348ab0c30cf285ca4b3259bec76ef" 2024-10-08T22:04:52.5006829Z }, 2024-10-08T22:04:52.5007128Z { 2024-10-08T22:04:52.5007748Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5008441Z "size": 3482, 2024-10-08T22:04:52.5009194Z "digest": "sha256:b178ce45c4c4237129521d416bbcae354ef9f073d6e9f0d2af15d9805755d80b" 2024-10-08T22:04:52.5009984Z }, 2024-10-08T22:04:52.5010264Z { 2024-10-08T22:04:52.5010802Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5011497Z "size": 1959, 2024-10-08T22:04:52.5012156Z "digest": "sha256:fe973329b31bfca304ba2a7bf830c2dc5938e6565645a62165265ec4002d724b" 2024-10-08T22:04:52.5012923Z }, 2024-10-08T22:04:52.5013223Z { 2024-10-08T22:04:52.5013762Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5014440Z "size": 701, 2024-10-08T22:04:52.5015113Z "digest": "sha256:a1726b06f88ca301d64366b88bf29971fb7a274674afeb3ee3b7d36cc30fe0cf" 2024-10-08T22:04:52.5015891Z }, 2024-10-08T22:04:52.5016188Z { 2024-10-08T22:04:52.5016708Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5017418Z "size": 2903729516, 2024-10-08T22:04:52.5018133Z "digest": "sha256:d768ae8cece99a6294f71ca2ada8dc8b15b5bbf516341a352fc1a572374f8c99" 2024-10-08T22:04:52.5018926Z }, 2024-10-08T22:04:52.5019209Z { 2024-10-08T22:04:52.5019744Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5020439Z "size": 380, 2024-10-08T22:04:52.5021117Z "digest": "sha256:383dfd7fe39b18b3b5e70a75e23d14ed5b55fbba10e8f21b2ddeddfd788aeee6" 2024-10-08T22:04:52.5021908Z }, 2024-10-08T22:04:52.5022209Z { 2024-10-08T22:04:52.5022749Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5023430Z "size": 12867, 2024-10-08T22:04:52.5024112Z "digest": "sha256:b5da99ef195e280ee36e9c257b661733c62c44e1ff2c360bb7355684090932ba" 2024-10-08T22:04:52.5024885Z }, 2024-10-08T22:04:52.5025188Z { 2024-10-08T22:04:52.5025708Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5026412Z "size": 844, 2024-10-08T22:04:52.5027084Z "digest": "sha256:0cdd9f0e273c3834674343b9e9a60b34b5a8d09082cc67be4121c008edfb1b43" 2024-10-08T22:04:52.5027852Z }, 2024-10-08T22:04:52.5028137Z { 2024-10-08T22:04:52.5028681Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5029378Z "size": 106, 2024-10-08T22:04:52.5030020Z "digest": "sha256:3800219be7491131618644bbed03e4a4b0a68f6a642b7a78ac802674896997cd" 2024-10-08T22:04:52.5030774Z }, 2024-10-08T22:04:52.5031074Z { 2024-10-08T22:04:52.5031612Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5032289Z "size": 503, 2024-10-08T22:04:52.5032961Z "digest": "sha256:2f447e7302bf2b2d71004e162bd82dfdd23e4629d89f585d576e4cbf9f5e7fb9" 2024-10-08T22:04:52.5033735Z }, 2024-10-08T22:04:52.5034033Z { 2024-10-08T22:04:52.5034554Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5035250Z "size": 121477305, 2024-10-08T22:04:52.5035952Z "digest": "sha256:469c5d8e340380fcedc42524ea516107f16a48c2c9a2356087ebf208a9d80f51" 2024-10-08T22:04:52.5036700Z }, 2024-10-08T22:04:52.5036998Z { 2024-10-08T22:04:52.5037542Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5038232Z "size": 109, 2024-10-08T22:04:52.5038890Z "digest": "sha256:c789deff668cbd779f993d84765a6459f9d9e477e8df636f225eb34aa9294a3d" 2024-10-08T22:04:52.5039661Z }, 2024-10-08T22:04:52.5039960Z { 2024-10-08T22:04:52.5040500Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5041175Z "size": 490, 2024-10-08T22:04:52.5041837Z "digest": "sha256:ce8d6236277760c5075e99fa0535cee1c39d25194b085586d06fa4da00a4543b" 2024-10-08T22:04:52.5042601Z }, 2024-10-08T22:04:52.5042882Z { 2024-10-08T22:04:52.5043421Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5044114Z "size": 292, 2024-10-08T22:04:52.5044797Z "digest": "sha256:5327a09bc8e29a1baeae9044216efcea92ade118eb0b4127a221f4e87698bd9c" 2024-10-08T22:04:52.5045674Z }, 2024-10-08T22:04:52.5045975Z { 2024-10-08T22:04:52.5046580Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5047289Z "size": 103, 2024-10-08T22:04:52.5047939Z "digest": "sha256:857ec5393e372d8bf7811e8a195166bd66d1ef818cda22025609ae33d63a4503" 2024-10-08T22:04:52.5048713Z }, 2024-10-08T22:04:52.5049013Z { 2024-10-08T22:04:52.5049553Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5050232Z "size": 1473, 2024-10-08T22:04:52.5050914Z "digest": "sha256:3f118703ee211c73da4f0bad84b4ee44b68be9e3e4d82c58e0236721eae6d446" 2024-10-08T22:04:52.5051695Z }, 2024-10-08T22:04:52.5051979Z { 2024-10-08T22:04:52.5052515Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5053211Z "size": 426601805, 2024-10-08T22:04:52.5053894Z "digest": "sha256:9f276609669a12a243e22289f3579f0c1038d52ea4a1b98de5c2b223aef9c99e" 2024-10-08T22:04:52.5054641Z }, 2024-10-08T22:04:52.5054941Z { 2024-10-08T22:04:52.5055480Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5056177Z "size": 163, 2024-10-08T22:04:52.5056858Z "digest": "sha256:70fbeb15f40bca03f1370ae737cbddb5ef5a20a4d94dea369ee1bc7f92c1406e" 2024-10-08T22:04:52.5057650Z }, 2024-10-08T22:04:52.5057953Z { 2024-10-08T22:04:52.5058475Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5059170Z "size": 564, 2024-10-08T22:04:52.5059862Z "digest": "sha256:652ceee7bd72eb92a90cb5a8bef7fbe13b72b5b28945b7a7a506517851e0d9e1" 2024-10-08T22:04:52.5060656Z }, 2024-10-08T22:04:52.5060938Z { 2024-10-08T22:04:52.5061474Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5062661Z "size": 35806315, 2024-10-08T22:04:52.5063361Z "digest": "sha256:6342136239c9a49e1481c17d71047f46e270fee45502c57cc5a94c391b60689a" 2024-10-08T22:04:52.5064103Z }, 2024-10-08T22:04:52.5064403Z { 2024-10-08T22:04:52.5064940Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5065631Z "size": 104, 2024-10-08T22:04:52.5066283Z "digest": "sha256:f9b62b97f427848350c50dfb72d67fe67053e92d2b6934683a28fac470ad0553" 2024-10-08T22:04:52.5067048Z }, 2024-10-08T22:04:52.5067349Z { 2024-10-08T22:04:52.5067867Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5068561Z "size": 426, 2024-10-08T22:04:52.5069229Z "digest": "sha256:543c465a9a2093a7f95219edcf273a7455951f6fe6991da8ecdcb34abfb1929d" 2024-10-08T22:04:52.5069998Z }, 2024-10-08T22:04:52.5070281Z { 2024-10-08T22:04:52.5070829Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5071525Z "size": 20262182, 2024-10-08T22:04:52.5072218Z "digest": "sha256:db883be4ee2c953f66388249e2a02474247a8c30d2aa65f0b5b362f7bd2dd085" 2024-10-08T22:04:52.5072973Z }, 2024-10-08T22:04:52.5073291Z { 2024-10-08T22:04:52.5073826Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5074503Z "size": 438, 2024-10-08T22:04:52.5075158Z "digest": "sha256:81145445f0934368fd31437966f7173de951c962abba0bf15746dab649ad726f" 2024-10-08T22:04:52.5075911Z }, 2024-10-08T22:04:52.5076211Z { 2024-10-08T22:04:52.5076738Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5077429Z "size": 701, 2024-10-08T22:04:52.5078104Z "digest": "sha256:a1726b06f88ca301d64366b88bf29971fb7a274674afeb3ee3b7d36cc30fe0cf" 2024-10-08T22:04:52.5078879Z }, 2024-10-08T22:04:52.5079165Z { 2024-10-08T22:04:52.5079710Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5080407Z "size": 141, 2024-10-08T22:04:52.5081071Z "digest": "sha256:c07d4654f64a6b90353e30f2729d160f2d50623faa90725d4815cf15d1a09966" 2024-10-08T22:04:52.5081818Z }, 2024-10-08T22:04:52.5082119Z { 2024-10-08T22:04:52.5082807Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5083481Z "size": 135, 2024-10-08T22:04:52.5084282Z "digest": "sha256:a88f2dd09400478534f14aa5f2fb02bed1316b73a843feaa6e73c8dde829c85f" 2024-10-08T22:04:52.5085078Z }, 2024-10-08T22:04:52.5085377Z { 2024-10-08T22:04:52.5085896Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5086633Z "size": 32, 2024-10-08T22:04:52.5087318Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-10-08T22:04:52.5088099Z }, 2024-10-08T22:04:52.5088379Z { 2024-10-08T22:04:52.5088919Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5089617Z "size": 195, 2024-10-08T22:04:52.5090275Z "digest": "sha256:1229eaa069bffa97559c143685300c06e04baefb392bd6f7072f1c7033e03683" 2024-10-08T22:04:52.5091043Z }, 2024-10-08T22:04:52.5091347Z { 2024-10-08T22:04:52.5091885Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5092569Z "size": 565, 2024-10-08T22:04:52.5093237Z "digest": "sha256:d6193226e3b55a404fd0678e41ccb67d4d2eab691219d6d07396d9c416aee020" 2024-10-08T22:04:52.5094004Z }, 2024-10-08T22:04:52.5094315Z { 2024-10-08T22:04:52.5094836Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5095534Z "size": 43163384, 2024-10-08T22:04:52.5096227Z "digest": "sha256:d74819041b0de97d0a7c80de85c0d1849e3a281d953f2e22e8b435c1c61e613c" 2024-10-08T22:04:52.5096993Z }, 2024-10-08T22:04:52.5097278Z { 2024-10-08T22:04:52.5097817Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5098512Z "size": 106, 2024-10-08T22:04:52.5099165Z "digest": "sha256:ad425d313620217659c9f6ed5fe11fc4175efeb5c312ff62142de5186ac3eb26" 2024-10-08T22:04:52.5099935Z }, 2024-10-08T22:04:52.5100237Z { 2024-10-08T22:04:52.5100774Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5101495Z "size": 1402, 2024-10-08T22:04:52.5102189Z "digest": "sha256:faf3cf63111bb0125eddef73314e710c3d5be887cb1ee2a868acc592ffffbaa3" 2024-10-08T22:04:52.5102974Z }, 2024-10-08T22:04:52.5103276Z { 2024-10-08T22:04:52.5103805Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5104502Z "size": 701, 2024-10-08T22:04:52.5105171Z "digest": "sha256:a1726b06f88ca301d64366b88bf29971fb7a274674afeb3ee3b7d36cc30fe0cf" 2024-10-08T22:04:52.5105938Z }, 2024-10-08T22:04:52.5106222Z { 2024-10-08T22:04:52.5106764Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5107456Z "size": 136, 2024-10-08T22:04:52.5108113Z "digest": "sha256:db8c679abd5e3fdc5648375e71a865e9a30f1dab070127f392727346f3436f45" 2024-10-08T22:04:52.5108893Z }, 2024-10-08T22:04:52.5109195Z { 2024-10-08T22:04:52.5109735Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5110414Z "size": 120, 2024-10-08T22:04:52.5111110Z "digest": "sha256:d109ead9d4eb05d6638a820fbfc12dad473a3cc70f3496ce2a6f304b354b4f04" 2024-10-08T22:04:52.5111902Z }, 2024-10-08T22:04:52.5112206Z { 2024-10-08T22:04:52.5112734Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5113437Z "size": 4284084197, 2024-10-08T22:04:52.5114153Z "digest": "sha256:32653b9dad8fe12b5ed012e0d8f2143c5f2872785fc93fa1060b1aaf7fcfc2f2" 2024-10-08T22:04:52.5114926Z }, 2024-10-08T22:04:52.5115211Z { 2024-10-08T22:04:52.5115749Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5116440Z "size": 172, 2024-10-08T22:04:52.5117081Z "digest": "sha256:0204c99cfdf97511dc9109100c39d44398f07c65422e172673d23f789cb8af92" 2024-10-08T22:04:52.5117838Z }, 2024-10-08T22:04:52.5118137Z { 2024-10-08T22:04:52.5118676Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5119354Z "size": 907, 2024-10-08T22:04:52.5120016Z "digest": "sha256:03531a4b8530baafbbc558688a3c654b46ba176297111799479523ec05a7ba89" 2024-10-08T22:04:52.5120891Z }, 2024-10-08T22:04:52.5121192Z { 2024-10-08T22:04:52.5121713Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5122497Z "size": 701, 2024-10-08T22:04:52.5123185Z "digest": "sha256:a1726b06f88ca301d64366b88bf29971fb7a274674afeb3ee3b7d36cc30fe0cf" 2024-10-08T22:04:52.5123935Z }, 2024-10-08T22:04:52.5124238Z { 2024-10-08T22:04:52.5124779Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5125476Z "size": 135, 2024-10-08T22:04:52.5126119Z "digest": "sha256:1f862a05b848e33f85143015d8f32866057c122be1a1cc594182115b783c60a5" 2024-10-08T22:04:52.5126871Z }, 2024-10-08T22:04:52.5127172Z { 2024-10-08T22:04:52.5127707Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5128384Z "size": 32, 2024-10-08T22:04:52.5129057Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-10-08T22:04:52.5129850Z }, 2024-10-08T22:04:52.5130133Z { 2024-10-08T22:04:52.5130670Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5131364Z "size": 159, 2024-10-08T22:04:52.5132052Z "digest": "sha256:84cb101b0f6a6e42c4239f83af2dbe2dc4916f7ccb8dc701c6359d177eaa167c" 2024-10-08T22:04:52.5132825Z }, 2024-10-08T22:04:52.5133133Z { 2024-10-08T22:04:52.5133673Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5134371Z "size": 1841, 2024-10-08T22:04:52.5135040Z "digest": "sha256:c9cb49a615faf25a1a47606bd3f0dd3b44c9e9b7b43fd1924699b454889f68b8" 2024-10-08T22:04:52.5135823Z }, 2024-10-08T22:04:52.5136129Z { 2024-10-08T22:04:52.5136669Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5137347Z "size": 7529766, 2024-10-08T22:04:52.5138050Z "digest": "sha256:c51e13e07a2430b87fa4ada54cad6aff2277c57df3699c35ea92024b35176cc9" 2024-10-08T22:04:52.5138843Z }, 2024-10-08T22:04:52.5139127Z { 2024-10-08T22:04:52.5139669Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5140363Z "size": 163, 2024-10-08T22:04:52.5141041Z "digest": "sha256:4b5a281c9b3676d51a95d0497ccec3f3fc110cedac082f63138811315e1db9a0" 2024-10-08T22:04:52.5141799Z }, 2024-10-08T22:04:52.5142205Z { 2024-10-08T22:04:52.5142742Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5143434Z "size": 7943, 2024-10-08T22:04:52.5144100Z "digest": "sha256:d177450e7faf6f90e1fa6b4a23f82cd481f02b311fb57a90b426f12485eff4aa" 2024-10-08T22:04:52.5144881Z }, 2024-10-08T22:04:52.5145184Z { 2024-10-08T22:04:52.5145702Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5146394Z "size": 8070, 2024-10-08T22:04:52.5147076Z "digest": "sha256:7f15c9b24fdb4a1aec0f332541309cdc50b9548302163cf6e64ae77d20e2144c" 2024-10-08T22:04:52.5147847Z }, 2024-10-08T22:04:52.5148129Z { 2024-10-08T22:04:52.5148679Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5149374Z "size": 302, 2024-10-08T22:04:52.5150081Z "digest": "sha256:fe3a151b0fbcb438b32da96cba5ead50cbd1f155ba0b0e50cc850ae7410b96b8" 2024-10-08T22:04:52.5150860Z }, 2024-10-08T22:04:52.5151164Z { 2024-10-08T22:04:52.5151697Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5152395Z "size": 7630051, 2024-10-08T22:04:52.5153055Z "digest": "sha256:b59657f39c769a496cc11e1a1e912beb7a341e902605461b92c816cf57f8566f" 2024-10-08T22:04:52.5153817Z }, 2024-10-08T22:04:52.5154119Z { 2024-10-08T22:04:52.5154638Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5155332Z "size": 108, 2024-10-08T22:04:52.5156009Z "digest": "sha256:441c56cab1a6362d3ae1ca0c6f1968c33db611016d4a6cac532aef7022e7bc02" 2024-10-08T22:04:52.5156784Z }, 2024-10-08T22:04:52.5157066Z { 2024-10-08T22:04:52.5157602Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5158410Z "size": 54145664, 2024-10-08T22:04:52.5159121Z "digest": "sha256:f682cdeac3428b46112ddbc8fb0bfd058e390cd5dcd3c46150200e7ac93d3855" 2024-10-08T22:04:52.5159982Z }, 2024-10-08T22:04:52.5160296Z { 2024-10-08T22:04:52.5160839Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5161512Z "size": 474, 2024-10-08T22:04:52.5162644Z "digest": "sha256:34ada4a1554dfde9bcb57878a8451cac5d8d533fa7f6c891cf11bd67ef05193e" 2024-10-08T22:04:52.5163455Z }, 2024-10-08T22:04:52.5163757Z { 2024-10-08T22:04:52.5164282Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5164983Z "size": 1373229428, 2024-10-08T22:04:52.5165689Z "digest": "sha256:0a31332063ae77b99b025ce00aebd5497dbdd2ed5d6b0b458888a6fe97c3fb7c" 2024-10-08T22:04:52.5166464Z }, 2024-10-08T22:04:52.5166748Z { 2024-10-08T22:04:52.5167286Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5167995Z "size": 106, 2024-10-08T22:04:52.5168680Z "digest": "sha256:a129e7e090d5e1ca999c5a3ed36ae6922b7a7bc6900371f8bd8dde23a59a03b3" 2024-10-08T22:04:52.5169447Z }, 2024-10-08T22:04:52.5169757Z { 2024-10-08T22:04:52.5170304Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5170982Z "size": 568, 2024-10-08T22:04:52.5171649Z "digest": "sha256:4555e8c56fe056ad9245e1424353112ed83d24909e08ab3c26a2f79db0853d8e" 2024-10-08T22:04:52.5172424Z }, 2024-10-08T22:04:52.5172732Z { 2024-10-08T22:04:52.5173254Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5173951Z "size": 36532084, 2024-10-08T22:04:52.5174666Z "digest": "sha256:7e58c37a5557a6cbc4dd49e45a6fdf442bbda33e6c4260dfcadd3e2633b9a1a9" 2024-10-08T22:04:52.5175457Z }, 2024-10-08T22:04:52.5175743Z { 2024-10-08T22:04:52.5176282Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5176988Z "size": 111, 2024-10-08T22:04:52.5177630Z "digest": "sha256:30a8dd7295658dac778205182127dbfd382535629860fe0a3324cce6751b01b7" 2024-10-08T22:04:52.5178387Z }, 2024-10-08T22:04:52.5178689Z { 2024-10-08T22:04:52.5179230Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5179909Z "size": 531, 2024-10-08T22:04:52.5180594Z "digest": "sha256:df4f9872ec6d37b8f97e9e96efbcce157881a0d6fecd849ee5b45327f06cc335" 2024-10-08T22:04:52.5181384Z }, 2024-10-08T22:04:52.5181686Z { 2024-10-08T22:04:52.5182209Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5182904Z "size": 25837, 2024-10-08T22:04:52.5183589Z "digest": "sha256:c26c5b53fc08fec581e1f2a6f286703914c1968acf93fd08d7b9b497dd653230" 2024-10-08T22:04:52.5184365Z }, 2024-10-08T22:04:52.5184652Z { 2024-10-08T22:04:52.5185190Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5185884Z "size": 106, 2024-10-08T22:04:52.5186557Z "digest": "sha256:d03fc0c9e531605929c22a3dd4de1b6e7b0bf18f0fcfd70ab56f3227588c2bb3" 2024-10-08T22:04:52.5187387Z }, 2024-10-08T22:04:52.5187685Z { 2024-10-08T22:04:52.5188230Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5188910Z "size": 32, 2024-10-08T22:04:52.5189587Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-10-08T22:04:52.5190368Z }, 2024-10-08T22:04:52.5190671Z { 2024-10-08T22:04:52.5191197Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5191893Z "size": 32, 2024-10-08T22:04:52.5192573Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-10-08T22:04:52.5193356Z }, 2024-10-08T22:04:52.5193641Z { 2024-10-08T22:04:52.5194178Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5194868Z "size": 32, 2024-10-08T22:04:52.5195522Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-10-08T22:04:52.5196459Z }, 2024-10-08T22:04:52.5196759Z { 2024-10-08T22:04:52.5197300Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T22:04:52.5198101Z "size": 32, 2024-10-08T22:04:52.5198797Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-10-08T22:04:52.5199578Z } 2024-10-08T22:04:52.5199877Z ] 2024-10-08T22:04:52.5200159Z } 2024-10-08T22:04:52.5338187Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-10-08T22:04:52.5338825Z tag=${ECR_DOCKER_IMAGE##*/} 2024-10-08T22:04:52.5339511Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-10-08T22:04:52.5347279Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:04:52.5347906Z env: 2024-10-08T22:04:52.5348245Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:04:52.5349547Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:04:52.5350881Z ##[endgroup] 2024-10-08T22:04:52.5383471Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9-46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:04:52.5469229Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-10-08T22:04:52.5469957Z with: 2024-10-08T22:04:52.5471135Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:04:52.5472624Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T22:04:52.5473267Z env: 2024-10-08T22:04:52.5473588Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:04:52.5474021Z ##[endgroup] 2024-10-08T22:04:52.5498785Z ##[group]Run set -x 2024-10-08T22:04:52.5499206Z set -x 2024-10-08T22:04:52.5499582Z set +e 2024-10-08T22:04:52.5499940Z  2024-10-08T22:04:52.5500264Z login() { 2024-10-08T22:04:52.5501108Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-10-08T22:04:52.5502004Z } 2024-10-08T22:04:52.5502341Z  2024-10-08T22:04:52.5502723Z retry () { 2024-10-08T22:04:52.5503214Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-10-08T22:04:52.5503747Z } 2024-10-08T22:04:52.5504086Z  2024-10-08T22:04:52.5504470Z retry login "${DOCKER_REGISTRY}" 2024-10-08T22:04:52.5504966Z  2024-10-08T22:04:52.5505297Z set -e 2024-10-08T22:04:52.5505881Z # ignore output since only exit code is used for conditional 2024-10-08T22:04:52.5506757Z # only pull docker image if it's not available locally 2024-10-08T22:04:52.5507700Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-10-08T22:04:52.5508564Z  retry docker pull "${DOCKER_IMAGE}" 2024-10-08T22:04:52.5509107Z fi 2024-10-08T22:04:52.5515598Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:04:52.5516203Z env: 2024-10-08T22:04:52.5516548Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:04:52.5517822Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:04:52.5519304Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T22:04:52.5519954Z ##[endgroup] 2024-10-08T22:04:52.5548481Z + set +e 2024-10-08T22:04:52.5549404Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T22:04:52.5550219Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T22:04:52.5552737Z + aws ecr get-login-password --region us-east-1 2024-10-08T22:04:52.5554429Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T22:04:53.3463453Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-10-08T22:04:53.3465671Z Configure a credential helper to remove this warning. See 2024-10-08T22:04:53.3466770Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-10-08T22:04:53.3467454Z 2024-10-08T22:04:53.3467620Z Login Succeeded 2024-10-08T22:04:53.3478527Z + set -e 2024-10-08T22:04:53.3480702Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:04:53.3655595Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:04:53.3657987Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:04:53.8452890Z 46f3eb402e47f9fae93ec1dc1f6772950f4498aa: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9 2024-10-08T22:04:53.8454369Z 7a2c55901189: Pulling fs layer 2024-10-08T22:04:53.8455196Z 09d415c238d7: Pulling fs layer 2024-10-08T22:04:53.8455946Z 9fe6e2e61518: Pulling fs layer 2024-10-08T22:04:53.8456586Z 41f16248e682: Pulling fs layer 2024-10-08T22:04:53.8457604Z 95d7b7817039: Pulling fs layer 2024-10-08T22:04:53.8458324Z 8f6c90485347: Pulling fs layer 2024-10-08T22:04:53.8458823Z ab17245097e4: Pulling fs layer 2024-10-08T22:04:53.8459522Z dfecd7e9912b: Pulling fs layer 2024-10-08T22:04:53.8459998Z 464a8f745445: Pulling fs layer 2024-10-08T22:04:53.8460460Z d67c111fa588: Pulling fs layer 2024-10-08T22:04:53.8460911Z 0723abde19c0: Pulling fs layer 2024-10-08T22:04:53.8461392Z bfa46deece96: Pulling fs layer 2024-10-08T22:04:53.8461858Z d993aa29d70c: Pulling fs layer 2024-10-08T22:04:53.8462977Z 3704d1191479: Pulling fs layer 2024-10-08T22:04:53.8463526Z 59545c272cb5: Pulling fs layer 2024-10-08T22:04:53.8464056Z efd563baaa50: Pulling fs layer 2024-10-08T22:04:53.8464728Z 5972678dbf61: Pulling fs layer 2024-10-08T22:04:53.8465225Z b178ce45c4c4: Pulling fs layer 2024-10-08T22:04:53.8465758Z fe973329b31b: Pulling fs layer 2024-10-08T22:04:53.8466191Z 95d7b7817039: Waiting 2024-10-08T22:04:53.8466600Z a1726b06f88c: Pulling fs layer 2024-10-08T22:04:53.8467063Z d67c111fa588: Waiting 2024-10-08T22:04:53.8467442Z 8f6c90485347: Waiting 2024-10-08T22:04:53.8467834Z 0723abde19c0: Waiting 2024-10-08T22:04:53.8468251Z d768ae8cece9: Pulling fs layer 2024-10-08T22:04:53.8468722Z 383dfd7fe39b: Pulling fs layer 2024-10-08T22:04:53.8469159Z bfa46deece96: Waiting 2024-10-08T22:04:53.8469575Z b5da99ef195e: Pulling fs layer 2024-10-08T22:04:53.8470021Z ab17245097e4: Waiting 2024-10-08T22:04:53.8470413Z d993aa29d70c: Waiting 2024-10-08T22:04:53.8470783Z 3704d1191479: Waiting 2024-10-08T22:04:53.8471191Z 0cdd9f0e273c: Pulling fs layer 2024-10-08T22:04:53.8471648Z dfecd7e9912b: Waiting 2024-10-08T22:04:53.8472046Z 3800219be749: Pulling fs layer 2024-10-08T22:04:53.8472510Z 2f447e7302bf: Pulling fs layer 2024-10-08T22:04:53.8472952Z 464a8f745445: Waiting 2024-10-08T22:04:53.8473367Z 469c5d8e3403: Pulling fs layer 2024-10-08T22:04:53.8473794Z 59545c272cb5: Waiting 2024-10-08T22:04:53.8474184Z a1726b06f88c: Waiting 2024-10-08T22:04:53.8474575Z efd563baaa50: Waiting 2024-10-08T22:04:53.8474979Z 383dfd7fe39b: Waiting 2024-10-08T22:04:53.8475377Z c789deff668c: Pulling fs layer 2024-10-08T22:04:53.8475873Z 41f16248e682: Waiting 2024-10-08T22:04:53.8476545Z ce8d62362777: Pulling fs layer 2024-10-08T22:04:53.8477270Z b5da99ef195e: Waiting 2024-10-08T22:04:53.8478032Z 5972678dbf61: Waiting 2024-10-08T22:04:53.8478784Z d768ae8cece9: Waiting 2024-10-08T22:04:53.8479545Z 5327a09bc8e2: Pulling fs layer 2024-10-08T22:04:53.8480247Z 0cdd9f0e273c: Waiting 2024-10-08T22:04:53.8480666Z 857ec5393e37: Pulling fs layer 2024-10-08T22:04:53.8481136Z 3f118703ee21: Pulling fs layer 2024-10-08T22:04:53.8481563Z 3800219be749: Waiting 2024-10-08T22:04:53.8481951Z fe973329b31b: Waiting 2024-10-08T22:04:53.8482363Z 9f276609669a: Pulling fs layer 2024-10-08T22:04:53.8482816Z b178ce45c4c4: Waiting 2024-10-08T22:04:53.8483447Z 2f447e7302bf: Waiting 2024-10-08T22:04:53.8483858Z 70fbeb15f40b: Pulling fs layer 2024-10-08T22:04:53.8484308Z ce8d62362777: Waiting 2024-10-08T22:04:53.8484699Z 5327a09bc8e2: Waiting 2024-10-08T22:04:53.8485079Z 857ec5393e37: Waiting 2024-10-08T22:04:53.8485488Z 652ceee7bd72: Pulling fs layer 2024-10-08T22:04:53.8485936Z 469c5d8e3403: Waiting 2024-10-08T22:04:53.8486331Z 6342136239c9: Pulling fs layer 2024-10-08T22:04:53.8486839Z 3f118703ee21: Waiting 2024-10-08T22:04:53.8487237Z c789deff668c: Waiting 2024-10-08T22:04:53.8487655Z f9b62b97f427: Pulling fs layer 2024-10-08T22:04:53.8488112Z 543c465a9a20: Pulling fs layer 2024-10-08T22:04:53.8488562Z 9f276609669a: Waiting 2024-10-08T22:04:53.8488976Z db883be4ee2c: Pulling fs layer 2024-10-08T22:04:53.8489408Z 6342136239c9: Waiting 2024-10-08T22:04:53.8489811Z 70fbeb15f40b: Waiting 2024-10-08T22:04:53.8490205Z 652ceee7bd72: Waiting 2024-10-08T22:04:53.8490598Z f9b62b97f427: Waiting 2024-10-08T22:04:53.8491014Z 543c465a9a20: Waiting 2024-10-08T22:04:53.8491820Z 81145445f093: Pulling fs layer 2024-10-08T22:04:53.8492717Z c07d4654f64a: Pulling fs layer 2024-10-08T22:04:53.8493278Z a88f2dd09400: Pulling fs layer 2024-10-08T22:04:53.8493715Z db883be4ee2c: Waiting 2024-10-08T22:04:53.8494371Z 4f4fb700ef54: Pulling fs layer 2024-10-08T22:04:53.8494877Z 1229eaa069bf: Pulling fs layer 2024-10-08T22:04:53.8495324Z d6193226e3b5: Pulling fs layer 2024-10-08T22:04:53.8495790Z d74819041b0d: Pulling fs layer 2024-10-08T22:04:53.8496258Z ad425d313620: Pulling fs layer 2024-10-08T22:04:53.8496732Z faf3cf63111b: Pulling fs layer 2024-10-08T22:04:53.8497187Z db8c679abd5e: Pulling fs layer 2024-10-08T22:04:53.8497635Z 81145445f093: Waiting 2024-10-08T22:04:53.8498049Z d109ead9d4eb: Pulling fs layer 2024-10-08T22:04:53.8498496Z c07d4654f64a: Waiting 2024-10-08T22:04:53.8498873Z d74819041b0d: Waiting 2024-10-08T22:04:53.8499278Z 32653b9dad8f: Pulling fs layer 2024-10-08T22:04:53.8499729Z a88f2dd09400: Waiting 2024-10-08T22:04:53.8500106Z ad425d313620: Waiting 2024-10-08T22:04:53.8500511Z d6193226e3b5: Waiting 2024-10-08T22:04:53.8500899Z 4f4fb700ef54: Waiting 2024-10-08T22:04:53.8501295Z d109ead9d4eb: Waiting 2024-10-08T22:04:53.8501697Z 0204c99cfdf9: Pulling fs layer 2024-10-08T22:04:53.8502152Z 32653b9dad8f: Waiting 2024-10-08T22:04:53.8502565Z 03531a4b8530: Pulling fs layer 2024-10-08T22:04:53.8502997Z db8c679abd5e: Waiting 2024-10-08T22:04:53.8503410Z 1f862a05b848: Pulling fs layer 2024-10-08T22:04:53.8503878Z 84cb101b0f6a: Pulling fs layer 2024-10-08T22:04:53.8504350Z c9cb49a615fa: Pulling fs layer 2024-10-08T22:04:53.8504804Z c51e13e07a24: Pulling fs layer 2024-10-08T22:04:53.8505274Z 4b5a281c9b36: Pulling fs layer 2024-10-08T22:04:53.8505739Z d177450e7faf: Pulling fs layer 2024-10-08T22:04:53.8506203Z 7f15c9b24fdb: Pulling fs layer 2024-10-08T22:04:53.8506658Z fe3a151b0fbc: Pulling fs layer 2024-10-08T22:04:53.8507128Z b59657f39c76: Pulling fs layer 2024-10-08T22:04:53.8507589Z 441c56cab1a6: Pulling fs layer 2024-10-08T22:04:53.8508038Z f682cdeac342: Pulling fs layer 2024-10-08T22:04:53.8508518Z 34ada4a1554d: Pulling fs layer 2024-10-08T22:04:53.8509022Z 0a31332063ae: Pulling fs layer 2024-10-08T22:04:53.8509467Z 03531a4b8530: Waiting 2024-10-08T22:04:53.8509867Z a129e7e090d5: Pulling fs layer 2024-10-08T22:04:53.8510322Z fe3a151b0fbc: Waiting 2024-10-08T22:04:53.8510716Z 4b5a281c9b36: Waiting 2024-10-08T22:04:53.8511108Z d177450e7faf: Waiting 2024-10-08T22:04:53.8511501Z 4555e8c56fe0: Pulling fs layer 2024-10-08T22:04:53.8512076Z 7f15c9b24fdb: Waiting 2024-10-08T22:04:53.8512854Z 7e58c37a5557: Pulling fs layer 2024-10-08T22:04:53.8513723Z b59657f39c76: Waiting 2024-10-08T22:04:53.8514444Z 30a8dd729565: Pulling fs layer 2024-10-08T22:04:53.8515304Z 84cb101b0f6a: Waiting 2024-10-08T22:04:53.8516040Z 441c56cab1a6: Waiting 2024-10-08T22:04:53.8516746Z f682cdeac342: Waiting 2024-10-08T22:04:53.8517521Z df4f9872ec6d: Pulling fs layer 2024-10-08T22:04:53.8518381Z 34ada4a1554d: Waiting 2024-10-08T22:04:53.8519122Z c9cb49a615fa: Waiting 2024-10-08T22:04:53.8520064Z c26c5b53fc08: Pulling fs layer 2024-10-08T22:04:53.8520913Z 0a31332063ae: Waiting 2024-10-08T22:04:53.8521672Z d03fc0c9e531: Pulling fs layer 2024-10-08T22:04:53.8522536Z 4555e8c56fe0: Waiting 2024-10-08T22:04:53.8523234Z 30a8dd729565: Waiting 2024-10-08T22:04:53.8523979Z a129e7e090d5: Waiting 2024-10-08T22:04:53.8524720Z df4f9872ec6d: Waiting 2024-10-08T22:04:53.8525206Z c26c5b53fc08: Waiting 2024-10-08T22:04:53.8525603Z d03fc0c9e531: Waiting 2024-10-08T22:04:54.0805254Z 09d415c238d7: Verifying Checksum 2024-10-08T22:04:54.0805854Z 09d415c238d7: Download complete 2024-10-08T22:04:54.1631854Z 41f16248e682: Verifying Checksum 2024-10-08T22:04:54.1632385Z 41f16248e682: Download complete 2024-10-08T22:04:54.1885697Z 7a2c55901189: Download complete 2024-10-08T22:04:54.2341055Z 95d7b7817039: Verifying Checksum 2024-10-08T22:04:54.2341575Z 95d7b7817039: Download complete 2024-10-08T22:04:54.3116894Z ab17245097e4: Download complete 2024-10-08T22:04:54.3919848Z dfecd7e9912b: Download complete 2024-10-08T22:04:54.4667854Z 9fe6e2e61518: Verifying Checksum 2024-10-08T22:04:54.4668480Z 9fe6e2e61518: Download complete 2024-10-08T22:04:54.5003914Z 464a8f745445: Verifying Checksum 2024-10-08T22:04:54.5004463Z 464a8f745445: Download complete 2024-10-08T22:04:54.6004958Z 0723abde19c0: Download complete 2024-10-08T22:04:54.6787422Z bfa46deece96: Download complete 2024-10-08T22:04:55.5389266Z 7a2c55901189: Pull complete 2024-10-08T22:04:55.8865737Z 09d415c238d7: Pull complete 2024-10-08T22:04:56.9554265Z 9fe6e2e61518: Pull complete 2024-10-08T22:04:56.9882479Z 41f16248e682: Pull complete 2024-10-08T22:04:57.0204956Z 95d7b7817039: Pull complete 2024-10-08T22:05:01.4313912Z d993aa29d70c: Verifying Checksum 2024-10-08T22:05:01.4314530Z d993aa29d70c: Download complete 2024-10-08T22:05:01.5040590Z 3704d1191479: Verifying Checksum 2024-10-08T22:05:01.5041152Z 3704d1191479: Download complete 2024-10-08T22:05:01.5894223Z 59545c272cb5: Verifying Checksum 2024-10-08T22:05:01.5894970Z 59545c272cb5: Download complete 2024-10-08T22:05:01.6694727Z efd563baaa50: Verifying Checksum 2024-10-08T22:05:01.6695380Z efd563baaa50: Download complete 2024-10-08T22:05:02.6676016Z 5972678dbf61: Verifying Checksum 2024-10-08T22:05:02.6676621Z 5972678dbf61: Download complete 2024-10-08T22:05:02.7495224Z b178ce45c4c4: Download complete 2024-10-08T22:05:02.8315764Z fe973329b31b: Download complete 2024-10-08T22:05:08.0859574Z 8f6c90485347: Verifying Checksum 2024-10-08T22:05:08.0860212Z 8f6c90485347: Download complete 2024-10-08T22:05:08.1616363Z 383dfd7fe39b: Verifying Checksum 2024-10-08T22:05:08.1616958Z 383dfd7fe39b: Download complete 2024-10-08T22:05:08.2383416Z b5da99ef195e: Download complete 2024-10-08T22:05:08.3061135Z 0cdd9f0e273c: Download complete 2024-10-08T22:05:08.3758057Z 3800219be749: Verifying Checksum 2024-10-08T22:05:08.3759096Z 3800219be749: Download complete 2024-10-08T22:05:08.4545175Z 2f447e7302bf: Verifying Checksum 2024-10-08T22:05:08.4545839Z 2f447e7302bf: Download complete 2024-10-08T22:05:09.7209018Z 469c5d8e3403: Verifying Checksum 2024-10-08T22:05:09.7209668Z 469c5d8e3403: Download complete 2024-10-08T22:05:09.7899222Z c789deff668c: Download complete 2024-10-08T22:05:09.8816236Z ce8d62362777: Verifying Checksum 2024-10-08T22:05:09.8817383Z ce8d62362777: Download complete 2024-10-08T22:05:10.0182088Z 5327a09bc8e2: Verifying Checksum 2024-10-08T22:05:10.0183166Z 5327a09bc8e2: Download complete 2024-10-08T22:05:10.0999316Z 857ec5393e37: Verifying Checksum 2024-10-08T22:05:10.1000380Z 857ec5393e37: Download complete 2024-10-08T22:05:10.1780776Z 3f118703ee21: Verifying Checksum 2024-10-08T22:05:10.1781883Z 3f118703ee21: Download complete 2024-10-08T22:05:14.4942167Z 9f276609669a: Verifying Checksum 2024-10-08T22:05:14.4942767Z 9f276609669a: Download complete 2024-10-08T22:05:14.5822881Z 70fbeb15f40b: Verifying Checksum 2024-10-08T22:05:14.5823449Z 70fbeb15f40b: Download complete 2024-10-08T22:05:14.6666351Z 652ceee7bd72: Verifying Checksum 2024-10-08T22:05:14.6666910Z 652ceee7bd72: Download complete 2024-10-08T22:05:15.0839829Z 6342136239c9: Verifying Checksum 2024-10-08T22:05:15.0840754Z 6342136239c9: Download complete 2024-10-08T22:05:15.1982997Z f9b62b97f427: Verifying Checksum 2024-10-08T22:05:15.1984086Z f9b62b97f427: Download complete 2024-10-08T22:05:15.2771914Z 543c465a9a20: Download complete 2024-10-08T22:05:15.5310681Z db883be4ee2c: Verifying Checksum 2024-10-08T22:05:15.5311325Z db883be4ee2c: Download complete 2024-10-08T22:05:15.6141216Z 81145445f093: Download complete 2024-10-08T22:05:15.6972166Z c07d4654f64a: Verifying Checksum 2024-10-08T22:05:15.6973259Z c07d4654f64a: Download complete 2024-10-08T22:05:15.7892496Z a88f2dd09400: Verifying Checksum 2024-10-08T22:05:15.7893514Z a88f2dd09400: Download complete 2024-10-08T22:05:15.8022066Z 4f4fb700ef54: Download complete 2024-10-08T22:05:15.8824193Z 1229eaa069bf: Verifying Checksum 2024-10-08T22:05:15.8825242Z 1229eaa069bf: Download complete 2024-10-08T22:05:15.9629009Z d6193226e3b5: Verifying Checksum 2024-10-08T22:05:15.9629988Z d6193226e3b5: Download complete 2024-10-08T22:05:16.4501098Z d74819041b0d: Verifying Checksum 2024-10-08T22:05:16.5339063Z ad425d313620: Verifying Checksum 2024-10-08T22:05:16.5339668Z ad425d313620: Download complete 2024-10-08T22:05:16.6047165Z faf3cf63111b: Verifying Checksum 2024-10-08T22:05:16.6048099Z faf3cf63111b: Download complete 2024-10-08T22:05:16.6702622Z db8c679abd5e: Download complete 2024-10-08T22:05:16.7817183Z d109ead9d4eb: Download complete 2024-10-08T22:05:18.9764512Z d67c111fa588: Verifying Checksum 2024-10-08T22:05:18.9765578Z d67c111fa588: Download complete 2024-10-08T22:05:19.0803131Z 0204c99cfdf9: Verifying Checksum 2024-10-08T22:05:19.0803717Z 0204c99cfdf9: Download complete 2024-10-08T22:05:19.1975514Z 03531a4b8530: Verifying Checksum 2024-10-08T22:05:19.1976114Z 03531a4b8530: Download complete 2024-10-08T22:05:19.2715740Z 1f862a05b848: Verifying Checksum 2024-10-08T22:05:19.2716306Z 1f862a05b848: Download complete 2024-10-08T22:05:19.3529567Z 84cb101b0f6a: Verifying Checksum 2024-10-08T22:05:19.3530176Z 84cb101b0f6a: Download complete 2024-10-08T22:05:19.4274351Z c9cb49a615fa: Verifying Checksum 2024-10-08T22:05:19.4274943Z c9cb49a615fa: Download complete 2024-10-08T22:05:19.5626479Z c51e13e07a24: Verifying Checksum 2024-10-08T22:05:19.5627113Z c51e13e07a24: Download complete 2024-10-08T22:05:19.6478910Z 4b5a281c9b36: Download complete 2024-10-08T22:05:19.7219786Z d177450e7faf: Verifying Checksum 2024-10-08T22:05:19.7220367Z d177450e7faf: Download complete 2024-10-08T22:05:19.8001867Z 7f15c9b24fdb: Verifying Checksum 2024-10-08T22:05:19.8002460Z 7f15c9b24fdb: Download complete 2024-10-08T22:05:19.8734373Z fe3a151b0fbc: Verifying Checksum 2024-10-08T22:05:19.8735351Z fe3a151b0fbc: Download complete 2024-10-08T22:05:20.0106301Z b59657f39c76: Verifying Checksum 2024-10-08T22:05:20.0106879Z b59657f39c76: Download complete 2024-10-08T22:05:20.0973464Z 441c56cab1a6: Verifying Checksum 2024-10-08T22:05:20.0974068Z 441c56cab1a6: Download complete 2024-10-08T22:05:20.7070624Z f682cdeac342: Verifying Checksum 2024-10-08T22:05:20.7851292Z 34ada4a1554d: Download complete 2024-10-08T22:05:25.3093197Z 8f6c90485347: Pull complete 2024-10-08T22:05:25.7835387Z ab17245097e4: Pull complete 2024-10-08T22:05:26.2273231Z dfecd7e9912b: Pull complete 2024-10-08T22:05:26.6708709Z 464a8f745445: Pull complete 2024-10-08T22:05:32.0049930Z d768ae8cece9: Verifying Checksum 2024-10-08T22:05:32.0050565Z d768ae8cece9: Download complete 2024-10-08T22:05:32.0756911Z a129e7e090d5: Verifying Checksum 2024-10-08T22:05:32.0757525Z a129e7e090d5: Download complete 2024-10-08T22:05:32.1608553Z 4555e8c56fe0: Download complete 2024-10-08T22:05:32.6125954Z 7e58c37a5557: Verifying Checksum 2024-10-08T22:05:32.6127023Z 7e58c37a5557: Download complete 2024-10-08T22:05:32.6948128Z 30a8dd729565: Verifying Checksum 2024-10-08T22:05:32.6949169Z 30a8dd729565: Download complete 2024-10-08T22:05:32.7779370Z df4f9872ec6d: Verifying Checksum 2024-10-08T22:05:32.7779948Z df4f9872ec6d: Download complete 2024-10-08T22:05:32.8567014Z c26c5b53fc08: Verifying Checksum 2024-10-08T22:05:32.8567603Z c26c5b53fc08: Download complete 2024-10-08T22:05:34.5752273Z 0a31332063ae: Verifying Checksum 2024-10-08T22:05:34.5753359Z 0a31332063ae: Download complete 2024-10-08T22:05:59.6835925Z 32653b9dad8f: Verifying Checksum 2024-10-08T22:05:59.6836573Z 32653b9dad8f: Download complete 2024-10-08T22:06:00.9114186Z d67c111fa588: Pull complete 2024-10-08T22:06:01.3704670Z 0723abde19c0: Pull complete 2024-10-08T22:06:01.8293320Z bfa46deece96: Pull complete 2024-10-08T22:06:15.8360279Z d993aa29d70c: Pull complete 2024-10-08T22:06:16.3135599Z 3704d1191479: Pull complete 2024-10-08T22:06:16.8239675Z 59545c272cb5: Pull complete 2024-10-08T22:06:17.3093902Z efd563baaa50: Pull complete 2024-10-08T22:06:20.4185610Z 5972678dbf61: Pull complete 2024-10-08T22:06:20.8753461Z b178ce45c4c4: Pull complete 2024-10-08T22:06:21.3233971Z fe973329b31b: Pull complete 2024-10-08T22:06:21.6311366Z a1726b06f88c: Pull complete 2024-10-08T22:07:28.1667897Z d768ae8cece9: Pull complete 2024-10-08T22:07:28.6130585Z 383dfd7fe39b: Pull complete 2024-10-08T22:07:29.0558782Z b5da99ef195e: Pull complete 2024-10-08T22:07:29.1822144Z 0cdd9f0e273c: Pull complete 2024-10-08T22:07:29.3975238Z 3800219be749: Pull complete 2024-10-08T22:07:29.6328876Z 2f447e7302bf: Pull complete 2024-10-08T22:07:33.8196748Z 469c5d8e3403: Pull complete 2024-10-08T22:07:33.8417553Z c789deff668c: Pull complete 2024-10-08T22:07:33.8669606Z ce8d62362777: Pull complete 2024-10-08T22:07:33.8932586Z 5327a09bc8e2: Pull complete 2024-10-08T22:07:33.9173068Z 857ec5393e37: Pull complete 2024-10-08T22:07:33.9438859Z 3f118703ee21: Pull complete 2024-10-08T22:07:44.2916747Z 9f276609669a: Pull complete 2024-10-08T22:07:44.7207083Z 70fbeb15f40b: Pull complete 2024-10-08T22:07:45.0971552Z 652ceee7bd72: Pull complete 2024-10-08T22:07:46.5169168Z 6342136239c9: Pull complete 2024-10-08T22:07:46.9944680Z f9b62b97f427: Pull complete 2024-10-08T22:07:47.4782695Z 543c465a9a20: Pull complete 2024-10-08T22:07:48.2631675Z db883be4ee2c: Pull complete 2024-10-08T22:07:48.7557687Z 81145445f093: Pull complete 2024-10-08T22:07:49.7315431Z c07d4654f64a: Pull complete 2024-10-08T22:07:50.2198123Z a88f2dd09400: Pull complete 2024-10-08T22:07:50.7125840Z 4f4fb700ef54: Pull complete 2024-10-08T22:07:51.0420293Z 1229eaa069bf: Pull complete 2024-10-08T22:07:51.3570170Z d6193226e3b5: Pull complete 2024-10-08T22:07:54.4070079Z d74819041b0d: Pull complete 2024-10-08T22:07:54.8439410Z ad425d313620: Pull complete 2024-10-08T22:07:55.2729215Z faf3cf63111b: Pull complete 2024-10-08T22:07:56.2331664Z db8c679abd5e: Pull complete 2024-10-08T22:07:56.7200610Z d109ead9d4eb: Pull complete 2024-10-08T22:09:32.2273440Z 32653b9dad8f: Pull complete 2024-10-08T22:09:32.6713760Z 0204c99cfdf9: Pull complete 2024-10-08T22:09:33.1196668Z 03531a4b8530: Pull complete 2024-10-08T22:09:34.0270333Z 1f862a05b848: Pull complete 2024-10-08T22:09:34.9498911Z 84cb101b0f6a: Pull complete 2024-10-08T22:09:35.4109809Z c9cb49a615fa: Pull complete 2024-10-08T22:09:36.0434579Z c51e13e07a24: Pull complete 2024-10-08T22:09:36.5029270Z 4b5a281c9b36: Pull complete 2024-10-08T22:09:36.9586688Z d177450e7faf: Pull complete 2024-10-08T22:09:37.4290655Z 7f15c9b24fdb: Pull complete 2024-10-08T22:09:37.9043510Z fe3a151b0fbc: Pull complete 2024-10-08T22:09:39.6965844Z b59657f39c76: Pull complete 2024-10-08T22:09:40.1709957Z 441c56cab1a6: Pull complete 2024-10-08T22:09:43.0153987Z f682cdeac342: Pull complete 2024-10-08T22:09:43.4877294Z 34ada4a1554d: Pull complete 2024-10-08T22:09:59.0093234Z 0a31332063ae: Pull complete 2024-10-08T22:09:59.4747903Z a129e7e090d5: Pull complete 2024-10-08T22:09:59.9379663Z 4555e8c56fe0: Pull complete 2024-10-08T22:10:00.9603255Z 7e58c37a5557: Pull complete 2024-10-08T22:10:01.4140607Z 30a8dd729565: Pull complete 2024-10-08T22:10:01.8734691Z df4f9872ec6d: Pull complete 2024-10-08T22:10:02.3353360Z c26c5b53fc08: Pull complete 2024-10-08T22:10:02.7696275Z d03fc0c9e531: Pull complete 2024-10-08T22:10:04.7167737Z Digest: sha256:314907662aadd5875a7a36524f299773cbf917ac65385064018e080dc893f046 2024-10-08T22:10:04.8106960Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:10:04.8642542Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:10:04.8735422Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-10-08T22:10:04.8737112Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-10-08T22:10:04.8747408Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:10:04.8748037Z env: 2024-10-08T22:10:04.8748358Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:10:04.8748795Z ##[endgroup] 2024-10-08T22:10:04.9012173Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-10-08T22:10:04.9012896Z with: 2024-10-08T22:10:04.9013239Z driver-version: 550.54.15 2024-10-08T22:10:04.9013679Z env: 2024-10-08T22:10:04.9013996Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:10:04.9014428Z ##[endgroup] 2024-10-08T22:10:04.9256398Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-10-08T22:10:04.9257096Z with: 2024-10-08T22:10:04.9257429Z timeout_minutes: 10 2024-10-08T22:10:04.9257815Z max_attempts: 3 2024-10-08T22:10:04.9300335Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-10-08T22:10:04.9343004Z retry_wait_seconds: 10 2024-10-08T22:10:04.9343468Z polling_interval_seconds: 1 2024-10-08T22:10:04.9343916Z warning_on_retry: true 2024-10-08T22:10:04.9344362Z continue_on_error: false 2024-10-08T22:10:04.9344778Z env: 2024-10-08T22:10:04.9345117Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:10:04.9345542Z DRIVER_VERSION: 550.54.15 2024-10-08T22:10:04.9345978Z ##[endgroup] 2024-10-08T22:10:05.0138299Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-10-08T22:10:05.0140177Z + pre_install_nvidia_driver_amzn2 2024-10-08T22:10:05.0140863Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-10-08T22:10:05.4525024Z No match for argument: nvidia-driver-latest-dkms 2024-10-08T22:10:05.4528472Z No packages marked for removal. 2024-10-08T22:10:05.4610629Z Dependencies resolved. 2024-10-08T22:10:05.4624625Z Nothing to do. 2024-10-08T22:10:05.4626978Z Complete! 2024-10-08T22:10:05.5913616Z + install_nvidia_driver_common 2024-10-08T22:10:05.5917847Z + echo 'Before installing NVIDIA driver' 2024-10-08T22:10:05.5919758Z Before installing NVIDIA driver 2024-10-08T22:10:05.5920527Z + lspci 2024-10-08T22:10:05.6123715Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-10-08T22:10:05.6124968Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-10-08T22:10:05.6125967Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-10-08T22:10:05.6126924Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-10-08T22:10:05.6127751Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-10-08T22:10:05.6128615Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-10-08T22:10:05.6129626Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-10-08T22:10:05.6130688Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-10-08T22:10:05.6131926Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-10-08T22:10:05.6132691Z + lsmod 2024-10-08T22:10:05.6169081Z Module Size Used by 2024-10-08T22:10:05.6169618Z xt_nat 16384 0 2024-10-08T22:10:05.6170110Z ib_core 454656 0 2024-10-08T22:10:05.6170572Z veth 36864 0 2024-10-08T22:10:05.6171021Z nvidia_modeset 1351680 0 2024-10-08T22:10:05.6171544Z video 65536 1 nvidia_modeset 2024-10-08T22:10:05.6172094Z wmi 36864 1 video 2024-10-08T22:10:05.6172588Z nvidia_uvm 4706304 0 2024-10-08T22:10:05.6177501Z nvidia 54071296 11 nvidia_uvm,nvidia_modeset 2024-10-08T22:10:05.6178164Z drm 602112 1 nvidia 2024-10-08T22:10:05.6178754Z drm_panel_orientation_quirks 32768 1 drm 2024-10-08T22:10:05.6179393Z backlight 24576 3 video,drm,nvidia_modeset 2024-10-08T22:10:05.6180049Z i2c_core 106496 2 nvidia,drm 2024-10-08T22:10:05.6180578Z xt_conntrack 16384 1 2024-10-08T22:10:05.6181042Z nft_chain_nat 16384 3 2024-10-08T22:10:05.6181525Z xt_MASQUERADE 20480 1 2024-10-08T22:10:05.6182130Z nf_nat 57344 3 xt_nat,nft_chain_nat,xt_MASQUERADE 2024-10-08T22:10:05.6182781Z nf_conntrack_netlink 57344 0 2024-10-08T22:10:05.6183546Z nf_conntrack 184320 5 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-10-08T22:10:05.6184391Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-10-08T22:10:05.6184981Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-10-08T22:10:05.6185532Z xfrm_user 57344 1 2024-10-08T22:10:05.6186004Z xfrm_algo 16384 1 xfrm_user 2024-10-08T22:10:05.6186522Z xt_addrtype 16384 2 2024-10-08T22:10:05.6187014Z nft_compat 20480 4 2024-10-08T22:10:05.6187708Z nf_tables 311296 57 nft_compat,nft_chain_nat 2024-10-08T22:10:05.6188534Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-10-08T22:10:05.6189245Z br_netfilter 36864 0 2024-10-08T22:10:05.6189790Z bridge 323584 1 br_netfilter 2024-10-08T22:10:05.6190375Z stp 16384 1 bridge 2024-10-08T22:10:05.6190936Z llc 16384 2 bridge,stp 2024-10-08T22:10:05.6191472Z overlay 167936 0 2024-10-08T22:10:05.6191958Z tls 114688 0 2024-10-08T22:10:05.6192453Z nls_ascii 16384 1 2024-10-08T22:10:05.6192937Z nls_cp437 20480 1 2024-10-08T22:10:05.6193408Z vfat 24576 1 2024-10-08T22:10:05.6193896Z sunrpc 692224 1 2024-10-08T22:10:05.6194397Z fat 86016 1 vfat 2024-10-08T22:10:05.6194896Z ata_piix 45056 0 2024-10-08T22:10:05.6195409Z libata 401408 1 ata_piix 2024-10-08T22:10:05.6195967Z scsi_mod 290816 1 libata 2024-10-08T22:10:05.6196499Z ena 167936 0 2024-10-08T22:10:05.6197036Z scsi_common 16384 2 scsi_mod,libata 2024-10-08T22:10:05.6197627Z ghash_clmulni_intel 16384 0 2024-10-08T22:10:05.6198128Z i8042 45056 0 2024-10-08T22:10:05.6198618Z serio 28672 3 i8042 2024-10-08T22:10:05.6199375Z button 24576 0 2024-10-08T22:10:05.6199869Z sch_fq_codel 20480 9 2024-10-08T22:10:05.6200371Z fuse 163840 1 2024-10-08T22:10:05.6200842Z dm_mod 188416 0 2024-10-08T22:10:05.6201329Z loop 36864 0 2024-10-08T22:10:05.6201811Z configfs 57344 1 2024-10-08T22:10:05.6202327Z dax 45056 1 dm_mod 2024-10-08T22:10:05.6202839Z dmi_sysfs 20480 0 2024-10-08T22:10:05.6203333Z crc32_pclmul 16384 0 2024-10-08T22:10:05.6203833Z crc32c_intel 24576 0 2024-10-08T22:10:05.6204319Z + modinfo nvidia 2024-10-08T22:10:05.6205244Z filename: /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-10-08T22:10:05.6206357Z alias: char-major-195-* 2024-10-08T22:10:05.6206892Z version: 550.54.15 2024-10-08T22:10:05.6207355Z supported: external 2024-10-08T22:10:05.6207788Z license: NVIDIA 2024-10-08T22:10:05.6208296Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-10-08T22:10:05.6208940Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-10-08T22:10:05.6209534Z srcversion: 833721318DA517F0C2FEC97 2024-10-08T22:10:05.6210156Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-10-08T22:10:05.6210827Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-10-08T22:10:05.6211491Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-10-08T22:10:05.6212147Z depends: i2c-core,drm 2024-10-08T22:10:05.6212615Z retpoline: Y 2024-10-08T22:10:05.6213019Z name: nvidia 2024-10-08T22:10:05.6213857Z vermagic: 6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-10-08T22:10:05.6214789Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-10-08T22:10:05.6215659Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-10-08T22:10:05.6216458Z parm: NVreg_ResmanDebugLevel:int 2024-10-08T22:10:05.6217046Z parm: NVreg_RmLogonRC:int 2024-10-08T22:10:05.6217622Z parm: NVreg_ModifyDeviceFiles:int 2024-10-08T22:10:05.6218235Z parm: NVreg_DeviceFileUID:int 2024-10-08T22:10:05.6218825Z parm: NVreg_DeviceFileGID:int 2024-10-08T22:10:05.6219417Z parm: NVreg_DeviceFileMode:int 2024-10-08T22:10:05.6220104Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-10-08T22:10:05.6220857Z parm: NVreg_UsePageAttributeTable:int 2024-10-08T22:10:05.6221498Z parm: NVreg_EnablePCIeGen3:int 2024-10-08T22:10:05.6222055Z parm: NVreg_EnableMSI:int 2024-10-08T22:10:05.6222612Z parm: NVreg_TCEBypassMode:int 2024-10-08T22:10:05.6223288Z parm: NVreg_EnableStreamMemOPs:int 2024-10-08T22:10:05.6223970Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-10-08T22:10:05.6224699Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-10-08T22:10:05.6225413Z parm: NVreg_EnableS0ixPowerManagement:int 2024-10-08T22:10:05.6226188Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-10-08T22:10:05.6226939Z parm: NVreg_DynamicPowerManagement:int 2024-10-08T22:10:05.6227688Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-10-08T22:10:05.6228438Z parm: NVreg_EnableGpuFirmware:int 2024-10-08T22:10:05.6229057Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-10-08T22:10:05.6229745Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-10-08T22:10:05.6230414Z parm: NVreg_EnableUserNUMAManagement:int 2024-10-08T22:10:05.6231037Z parm: NVreg_MemoryPoolSize:int 2024-10-08T22:10:05.6231617Z parm: NVreg_KMallocHeapMaxSize:int 2024-10-08T22:10:05.6232220Z parm: NVreg_VMallocHeapMaxSize:int 2024-10-08T22:10:05.6232789Z parm: NVreg_IgnoreMMIOCheck:int 2024-10-08T22:10:05.6233355Z parm: NVreg_NvLinkDisable:int 2024-10-08T22:10:05.6233989Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-10-08T22:10:05.6234776Z parm: NVreg_RegisterPCIDriver:int 2024-10-08T22:10:05.6235354Z parm: NVreg_EnableResizableBar:int 2024-10-08T22:10:05.6235970Z parm: NVreg_EnableDbgBreakpoint:int 2024-10-08T22:10:05.6236598Z parm: NVreg_EnableNonblockingOpen:int 2024-10-08T22:10:05.6237211Z parm: NVreg_RegistryDwords:charp 2024-10-08T22:10:05.6237815Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-10-08T22:10:05.6238418Z parm: NVreg_RmMsg:charp 2024-10-08T22:10:05.6238935Z parm: NVreg_GpuBlacklist:charp 2024-10-08T22:10:05.6239520Z parm: NVreg_TemporaryFilePath:charp 2024-10-08T22:10:05.6240090Z parm: NVreg_ExcludedGpus:charp 2024-10-08T22:10:05.6240660Z parm: NVreg_DmaRemapPeerMmio:int 2024-10-08T22:10:05.6241354Z parm: NVreg_RmNvlinkBandwidth:charp 2024-10-08T22:10:05.6241956Z parm: NVreg_ImexChannelCount:int 2024-10-08T22:10:05.6242522Z parm: rm_firmware_active:charp 2024-10-08T22:10:05.6243049Z + HAS_NVIDIA_DRIVER=0 2024-10-08T22:10:05.6243542Z ++ command -v nvidia-smi 2024-10-08T22:10:05.6244023Z + '[' -x /usr/bin/nvidia-smi ']' 2024-10-08T22:10:05.6244482Z + set +e 2024-10-08T22:10:05.6245130Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-10-08T22:10:05.6480663Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-10-08T22:10:05.6481202Z + NVIDIA_SMI_STATUS=0 2024-10-08T22:10:05.6481736Z + '[' 0 -ne 0 ']' 2024-10-08T22:10:05.6482191Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-10-08T22:10:05.6482663Z + HAS_NVIDIA_DRIVER=1 2024-10-08T22:10:05.6483571Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-10-08T22:10:05.6484459Z + set -e 2024-10-08T22:10:05.6484843Z + '[' 1 -eq 0 ']' 2024-10-08T22:10:05.6485576Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-10-08T22:10:05.6486424Z + post_install_nvidia_driver_common 2024-10-08T22:10:05.6488187Z + sudo modprobe nvidia 2024-10-08T22:10:05.7973160Z + echo 'After installing NVIDIA driver' 2024-10-08T22:10:05.7973690Z + lspci 2024-10-08T22:10:05.7974061Z After installing NVIDIA driver 2024-10-08T22:10:05.8174921Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-10-08T22:10:05.8175941Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-10-08T22:10:05.8176929Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-10-08T22:10:05.8177858Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-10-08T22:10:05.8178673Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-10-08T22:10:05.8179539Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-10-08T22:10:05.8180597Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-10-08T22:10:05.8181655Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-10-08T22:10:05.8182701Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-10-08T22:10:05.8183459Z + lsmod 2024-10-08T22:10:05.8205723Z Module Size Used by 2024-10-08T22:10:05.8206234Z xt_nat 16384 0 2024-10-08T22:10:05.8206693Z ib_core 454656 0 2024-10-08T22:10:05.8207160Z veth 36864 0 2024-10-08T22:10:05.8207634Z nvidia_modeset 1351680 0 2024-10-08T22:10:05.8208135Z video 65536 1 nvidia_modeset 2024-10-08T22:10:05.8208690Z wmi 36864 1 video 2024-10-08T22:10:05.8209181Z nvidia_uvm 4706304 0 2024-10-08T22:10:05.8209752Z nvidia 54071296 11 nvidia_uvm,nvidia_modeset 2024-10-08T22:10:05.8210349Z drm 602112 1 nvidia 2024-10-08T22:10:05.8210909Z drm_panel_orientation_quirks 32768 1 drm 2024-10-08T22:10:05.8211591Z backlight 24576 3 video,drm,nvidia_modeset 2024-10-08T22:10:05.8212238Z i2c_core 106496 2 nvidia,drm 2024-10-08T22:10:05.8213043Z xt_conntrack 16384 1 2024-10-08T22:10:05.8213530Z nft_chain_nat 16384 3 2024-10-08T22:10:05.8214010Z xt_MASQUERADE 20480 1 2024-10-08T22:10:05.8214608Z nf_nat 57344 3 xt_nat,nft_chain_nat,xt_MASQUERADE 2024-10-08T22:10:05.8215235Z nf_conntrack_netlink 57344 0 2024-10-08T22:10:05.8216010Z nf_conntrack 184320 5 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-10-08T22:10:05.8216861Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-10-08T22:10:05.8217443Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-10-08T22:10:05.8217969Z xfrm_user 57344 1 2024-10-08T22:10:05.8218465Z xfrm_algo 16384 1 xfrm_user 2024-10-08T22:10:05.8218988Z xt_addrtype 16384 2 2024-10-08T22:10:05.8219459Z nft_compat 20480 4 2024-10-08T22:10:05.8221161Z nf_tables 311296 57 nft_compat,nft_chain_nat 2024-10-08T22:10:05.8221996Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-10-08T22:10:05.8222701Z br_netfilter 36864 0 2024-10-08T22:10:05.8223276Z bridge 323584 1 br_netfilter 2024-10-08T22:10:05.8223807Z stp 16384 1 bridge 2024-10-08T22:10:05.8224326Z llc 16384 2 bridge,stp 2024-10-08T22:10:05.8224858Z overlay 167936 0 2024-10-08T22:10:05.8225307Z tls 114688 0 2024-10-08T22:10:05.8225764Z nls_ascii 16384 1 2024-10-08T22:10:05.8226229Z nls_cp437 20480 1 2024-10-08T22:10:05.8226690Z vfat 24576 1 2024-10-08T22:10:05.8227133Z sunrpc 692224 1 2024-10-08T22:10:05.8227613Z fat 86016 1 vfat 2024-10-08T22:10:05.8228105Z ata_piix 45056 0 2024-10-08T22:10:05.8228597Z libata 401408 1 ata_piix 2024-10-08T22:10:05.8229114Z scsi_mod 290816 1 libata 2024-10-08T22:10:05.8229613Z ena 167936 0 2024-10-08T22:10:05.8230131Z scsi_common 16384 2 scsi_mod,libata 2024-10-08T22:10:05.8230709Z ghash_clmulni_intel 16384 0 2024-10-08T22:10:05.8231160Z i8042 45056 0 2024-10-08T22:10:05.8231677Z serio 28672 3 i8042 2024-10-08T22:10:05.8232168Z button 24576 0 2024-10-08T22:10:05.8232616Z sch_fq_codel 20480 9 2024-10-08T22:10:05.8233075Z fuse 163840 1 2024-10-08T22:10:05.8233534Z dm_mod 188416 0 2024-10-08T22:10:05.8233988Z loop 36864 0 2024-10-08T22:10:05.8234434Z configfs 57344 1 2024-10-08T22:10:05.8234910Z dax 45056 1 dm_mod 2024-10-08T22:10:05.8235415Z dmi_sysfs 20480 0 2024-10-08T22:10:05.8235917Z crc32_pclmul 16384 0 2024-10-08T22:10:05.8236360Z crc32c_intel 24576 0 2024-10-08T22:10:05.8236821Z + modinfo nvidia 2024-10-08T22:10:05.8237701Z filename: /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-10-08T22:10:05.8238606Z alias: char-major-195-* 2024-10-08T22:10:05.8239066Z version: 550.54.15 2024-10-08T22:10:05.8239502Z supported: external 2024-10-08T22:10:05.8239927Z license: NVIDIA 2024-10-08T22:10:05.8240396Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-10-08T22:10:05.8240989Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-10-08T22:10:05.8241561Z srcversion: 833721318DA517F0C2FEC97 2024-10-08T22:10:05.8242146Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-10-08T22:10:05.8242764Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-10-08T22:10:05.8243385Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-10-08T22:10:05.8244015Z depends: i2c-core,drm 2024-10-08T22:10:05.8244468Z retpoline: Y 2024-10-08T22:10:05.8244831Z name: nvidia 2024-10-08T22:10:05.8245635Z vermagic: 6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-10-08T22:10:05.8246539Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-10-08T22:10:05.8247474Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-10-08T22:10:05.8248206Z parm: NVreg_ResmanDebugLevel:int 2024-10-08T22:10:05.8248762Z parm: NVreg_RmLogonRC:int 2024-10-08T22:10:05.8249304Z parm: NVreg_ModifyDeviceFiles:int 2024-10-08T22:10:05.8249874Z parm: NVreg_DeviceFileUID:int 2024-10-08T22:10:05.8250400Z parm: NVreg_DeviceFileGID:int 2024-10-08T22:10:05.8250952Z parm: NVreg_DeviceFileMode:int 2024-10-08T22:10:05.8251613Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-10-08T22:10:05.8252326Z parm: NVreg_UsePageAttributeTable:int 2024-10-08T22:10:05.8252911Z parm: NVreg_EnablePCIeGen3:int 2024-10-08T22:10:05.8253448Z parm: NVreg_EnableMSI:int 2024-10-08T22:10:05.8254071Z parm: NVreg_TCEBypassMode:int 2024-10-08T22:10:05.8254649Z parm: NVreg_EnableStreamMemOPs:int 2024-10-08T22:10:05.8255316Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-10-08T22:10:05.8256061Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-10-08T22:10:05.8256780Z parm: NVreg_EnableS0ixPowerManagement:int 2024-10-08T22:10:05.8257529Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-10-08T22:10:05.8258291Z parm: NVreg_DynamicPowerManagement:int 2024-10-08T22:10:05.8259061Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-10-08T22:10:05.8259809Z parm: NVreg_EnableGpuFirmware:int 2024-10-08T22:10:05.8260406Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-10-08T22:10:05.8261077Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-10-08T22:10:05.8261767Z parm: NVreg_EnableUserNUMAManagement:int 2024-10-08T22:10:05.8262913Z parm: NVreg_MemoryPoolSize:int 2024-10-08T22:10:05.8263486Z parm: NVreg_KMallocHeapMaxSize:int 2024-10-08T22:10:05.8264097Z parm: NVreg_VMallocHeapMaxSize:int 2024-10-08T22:10:05.8264697Z parm: NVreg_IgnoreMMIOCheck:int 2024-10-08T22:10:05.8265253Z parm: NVreg_NvLinkDisable:int 2024-10-08T22:10:05.8265867Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-10-08T22:10:05.8266531Z parm: NVreg_RegisterPCIDriver:int 2024-10-08T22:10:05.8267137Z parm: NVreg_EnableResizableBar:int 2024-10-08T22:10:05.8267748Z parm: NVreg_EnableDbgBreakpoint:int 2024-10-08T22:10:05.8268353Z parm: NVreg_EnableNonblockingOpen:int 2024-10-08T22:10:05.8268961Z parm: NVreg_RegistryDwords:charp 2024-10-08T22:10:05.8269587Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-10-08T22:10:05.8270184Z parm: NVreg_RmMsg:charp 2024-10-08T22:10:05.8270683Z parm: NVreg_GpuBlacklist:charp 2024-10-08T22:10:05.8271272Z parm: NVreg_TemporaryFilePath:charp 2024-10-08T22:10:05.8271868Z parm: NVreg_ExcludedGpus:charp 2024-10-08T22:10:05.8272422Z parm: NVreg_DmaRemapPeerMmio:int 2024-10-08T22:10:05.8273037Z parm: NVreg_RmNvlinkBandwidth:charp 2024-10-08T22:10:05.8273633Z parm: NVreg_ImexChannelCount:int 2024-10-08T22:10:05.8274205Z parm: rm_firmware_active:charp 2024-10-08T22:10:05.8274686Z + set +e 2024-10-08T22:10:05.8275084Z + nvidia-smi 2024-10-08T22:10:05.8494616Z Tue Oct 8 22:10:05 2024 2024-10-08T22:10:05.8495466Z +-----------------------------------------------------------------------------------------+ 2024-10-08T22:10:05.8496504Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-10-08T22:10:05.8497513Z |-----------------------------------------+------------------------+----------------------+ 2024-10-08T22:10:05.8498548Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-10-08T22:10:05.8499715Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-10-08T22:10:05.8500596Z | | | MIG M. | 2024-10-08T22:10:05.8501536Z |=========================================+========================+======================| 2024-10-08T22:10:05.8583273Z | 0 Tesla M60 On | 00000000:00:1D.0 Off | 0 | 2024-10-08T22:10:05.8584190Z | N/A 25C P8 14W / 150W | 0MiB / 7680MiB | 0% Default | 2024-10-08T22:10:05.8584996Z | | | N/A | 2024-10-08T22:10:05.8585886Z +-----------------------------------------+------------------------+----------------------+ 2024-10-08T22:10:05.8648692Z | 1 Tesla M60 On | 00000000:00:1E.0 Off | 0 | 2024-10-08T22:10:05.8649817Z | N/A 29C P8 13W / 150W | 0MiB / 7680MiB | 0% Default | 2024-10-08T22:10:05.8650659Z | | | N/A | 2024-10-08T22:10:05.8651557Z +-----------------------------------------+------------------------+----------------------+ 2024-10-08T22:10:05.8652276Z 2024-10-08T22:10:05.8653135Z +-----------------------------------------------------------------------------------------+ 2024-10-08T22:10:05.8653933Z | Processes: | 2024-10-08T22:10:05.8654837Z | GPU GI CI PID Type Process name GPU Memory | 2024-10-08T22:10:05.8655696Z | ID ID Usage | 2024-10-08T22:10:05.8656416Z |=========================================================================================| 2024-10-08T22:10:05.8657251Z | No running processes found | 2024-10-08T22:10:05.8658244Z +-----------------------------------------------------------------------------------------+ 2024-10-08T22:10:05.9773852Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-10-08T22:10:05.9958652Z Tesla M60 2024-10-08T22:10:06.0024342Z + NVIDIA_SMI_STATUS=0 2024-10-08T22:10:06.0024870Z + '[' 0 -eq 0 ']' 2024-10-08T22:10:06.0025370Z + echo 'INFO: Ignoring allowed status 0' 2024-10-08T22:10:06.0025911Z + set -e 2024-10-08T22:10:06.0026277Z INFO: Ignoring allowed status 0 2024-10-08T22:10:06.0032379Z == Installing nvidia container toolkit for amzn2023 == 2024-10-08T22:10:06.0036485Z + sudo yum install -y yum-utils 2024-10-08T22:10:06.5263411Z Last metadata expiration check: 0:41:02 ago on Tue Oct 8 21:29:04 2024. 2024-10-08T22:10:06.5568139Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-10-08T22:10:06.6029109Z Dependencies resolved. 2024-10-08T22:10:06.6240499Z Nothing to do. 2024-10-08T22:10:06.6241598Z Complete! 2024-10-08T22:10:06.7600845Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-10-08T22:10:06.7602191Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-10-08T22:10:06.7603916Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-10-08T22:10:07.1137015Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-10-08T22:10:07.2011003Z + sudo yum install -y nvidia-docker2 2024-10-08T22:10:07.8755878Z nvidia-container-toolkit 8.4 kB/s | 833 B 00:00 2024-10-08T22:10:07.9070164Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-10-08T22:10:07.9532316Z Dependencies resolved. 2024-10-08T22:10:07.9748979Z Nothing to do. 2024-10-08T22:10:07.9749812Z Complete! 2024-10-08T22:10:08.1092780Z + sudo systemctl restart docker 2024-10-08T22:10:45.7344339Z nvidia-persistenced failed to initialize. Check syslog for more details. 2024-10-08T22:10:45.7642127Z Tue Oct 8 22:10:45 2024 2024-10-08T22:10:45.7643471Z +-----------------------------------------------------------------------------------------+ 2024-10-08T22:10:45.7645949Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-10-08T22:10:45.7647041Z |-----------------------------------------+------------------------+----------------------+ 2024-10-08T22:10:45.7648100Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-10-08T22:10:45.7649242Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-10-08T22:10:45.7650125Z | | | MIG M. | 2024-10-08T22:10:45.7650829Z |=========================================+========================+======================| 2024-10-08T22:10:45.7730774Z | 0 Tesla M60 On | 00000000:00:1D.0 Off | 0 | 2024-10-08T22:10:45.7731735Z | N/A 25C P8 14W / 150W | 0MiB / 7680MiB | 0% Default | 2024-10-08T22:10:45.7732545Z | | | N/A | 2024-10-08T22:10:45.7733457Z +-----------------------------------------+------------------------+----------------------+ 2024-10-08T22:10:45.7797464Z | 1 Tesla M60 On | 00000000:00:1E.0 Off | 0 | 2024-10-08T22:10:45.7798386Z | N/A 29C P8 13W / 150W | 0MiB / 7680MiB | 0% Default | 2024-10-08T22:10:45.7799204Z | | | N/A | 2024-10-08T22:10:45.7800094Z +-----------------------------------------+------------------------+----------------------+ 2024-10-08T22:10:45.7800849Z 2024-10-08T22:10:45.7801709Z +-----------------------------------------------------------------------------------------+ 2024-10-08T22:10:45.7802533Z | Processes: | 2024-10-08T22:10:45.7803422Z | GPU GI CI PID Type Process name GPU Memory | 2024-10-08T22:10:45.7804310Z | ID ID Usage | 2024-10-08T22:10:45.7805029Z |=========================================================================================| 2024-10-08T22:10:45.7805856Z | No running processes found | 2024-10-08T22:10:45.7806852Z +-----------------------------------------------------------------------------------------+ 2024-10-08T22:10:46.0595807Z Command completed after 1 attempt(s). 2024-10-08T22:10:46.0706796Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-10-08T22:10:46.0707810Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-10-08T22:10:46.0708683Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-10-08T22:10:46.0709543Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-10-08T22:10:46.0719513Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:10:46.0720123Z env: 2024-10-08T22:10:46.0720473Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:10:46.0721042Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:10:46.0721652Z ##[endgroup] 2024-10-08T22:10:46.4113157Z Defaulting to user installation because normal site-packages is not writeable 2024-10-08T22:10:46.4327980Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.9/site-packages (5.9.1) 2024-10-08T22:10:46.4336003Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.9/site-packages (11.525.84) 2024-10-08T22:10:46.6026862Z Prepare all required actions 2024-10-08T22:10:46.6028415Z Getting action download info 2024-10-08T22:10:46.7219400Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-10-08T22:10:46.9175887Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-10-08T22:10:47.0660954Z ##[group]Run ./.github/actions/download-build-artifacts 2024-10-08T22:10:47.0661565Z with: 2024-10-08T22:10:47.0661970Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-10-08T22:10:47.0663239Z s3-bucket: gha-artifacts 2024-10-08T22:10:47.0663681Z env: 2024-10-08T22:10:47.0664020Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:10:47.0664580Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:10:47.0665167Z ##[endgroup] 2024-10-08T22:10:47.0712359Z ##[group]Run seemethere/download-artifact-s3@v4 2024-10-08T22:10:47.0712903Z with: 2024-10-08T22:10:47.0738785Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-10-08T22:10:47.0739473Z s3-bucket: gha-artifacts 2024-10-08T22:10:47.0739912Z region: us-east-1 2024-10-08T22:10:47.0740291Z env: 2024-10-08T22:10:47.0740622Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:10:47.0741181Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:10:47.0741787Z ##[endgroup] 2024-10-08T22:10:47.7099426Z (node:412624) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-10-08T22:10:47.7100347Z 2024-10-08T22:10:47.7100697Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-10-08T22:10:47.7101858Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-10-08T22:10:47.7103896Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-10-08T22:10:47.8043853Z Found 1 objects with prefix pytorch/pytorch/11242664097/linux-focal-cuda11.8-py3.10-gcc9/ 2024-10-08T22:10:47.8045263Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-10-08T22:11:14.0192673Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-10-08T22:11:14.0203367Z Artifact download has finished successfully 2024-10-08T22:11:14.0493969Z ##[group]Run unzip -o artifacts.zip 2024-10-08T22:11:14.0494522Z unzip -o artifacts.zip 2024-10-08T22:11:14.0503406Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:11:14.0504044Z env: 2024-10-08T22:11:14.0504397Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:11:14.0504950Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:11:14.0505561Z ##[endgroup] 2024-10-08T22:11:14.0554826Z Archive: artifacts.zip 2024-10-08T22:11:14.0556141Z creating: dist/ 2024-10-08T22:11:16.5322214Z inflating: dist/torch-2.6.0a0+git47952c0-cp310-cp310-linux_x86_64.whl 2024-10-08T22:11:16.5323054Z creating: build/custom_test_artifacts/ 2024-10-08T22:11:16.5323819Z creating: build/custom_test_artifacts/custom-op-build/ 2024-10-08T22:11:16.5324771Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-10-08T22:11:16.5325867Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-10-08T22:11:16.5333366Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-10-08T22:11:16.5334607Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-10-08T22:11:16.5335840Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-10-08T22:11:16.5337140Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-10-08T22:11:16.5338414Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-10-08T22:11:16.5339834Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-10-08T22:11:16.5341660Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-10-08T22:11:16.5343051Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-10-08T22:11:16.5344364Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-10-08T22:11:16.5346105Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-10-08T22:11:16.5347607Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-10-08T22:11:16.5349515Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-10-08T22:11:16.5351062Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-10-08T22:11:16.5352600Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-10-08T22:11:16.5354143Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-10-08T22:11:16.5355520Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-10-08T22:11:16.5356832Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-10-08T22:11:16.5412113Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-10-08T22:11:16.5413979Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-10-08T22:11:16.5415888Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-10-08T22:11:16.5417826Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-10-08T22:11:16.5419728Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-10-08T22:11:16.5421524Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-10-08T22:11:16.5423354Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-10-08T22:11:16.5425178Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-10-08T22:11:16.5426997Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-10-08T22:11:16.5468461Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-10-08T22:11:16.5516249Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-10-08T22:11:16.5518069Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-10-08T22:11:16.5519771Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-10-08T22:11:16.5521404Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-10-08T22:11:16.5523005Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-10-08T22:11:16.5524597Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-10-08T22:11:16.5526148Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-10-08T22:11:16.5527747Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-10-08T22:11:16.5599322Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-10-08T22:11:16.5673369Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-10-08T22:11:16.5675187Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-10-08T22:11:16.5676470Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-10-08T22:11:16.5677613Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-10-08T22:11:16.5678809Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-10-08T22:11:16.5680013Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-10-08T22:11:16.5681332Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-10-08T22:11:16.5682836Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-10-08T22:11:16.5684291Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-10-08T22:11:16.5685661Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-10-08T22:11:16.5687075Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-10-08T22:11:16.5688471Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-10-08T22:11:16.5689887Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-10-08T22:11:16.5691289Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-10-08T22:11:16.5692676Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-10-08T22:11:16.5710205Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-10-08T22:11:16.5883866Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-10-08T22:11:16.5886512Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-10-08T22:11:16.5888030Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-10-08T22:11:16.5889634Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-10-08T22:11:16.5891165Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-10-08T22:11:16.5892592Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-10-08T22:11:16.5894048Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-10-08T22:11:16.5895560Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-10-08T22:11:16.5897046Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-10-08T22:11:16.5898525Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-10-08T22:11:16.5899992Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-10-08T22:11:16.5917574Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-10-08T22:11:16.6017906Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-10-08T22:11:16.6019475Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-10-08T22:11:16.6020890Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-10-08T22:11:16.6022408Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-10-08T22:11:16.6023638Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-10-08T22:11:16.6024822Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-10-08T22:11:16.6026158Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-10-08T22:11:16.6027486Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-10-08T22:11:16.6028662Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-10-08T22:11:16.6030147Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-10-08T22:11:16.6176220Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-10-08T22:11:16.6252665Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-10-08T22:11:16.6253620Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-10-08T22:11:16.6254541Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-10-08T22:11:16.6255614Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-10-08T22:11:16.6263224Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-10-08T22:11:16.6264465Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-10-08T22:11:16.6265660Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-10-08T22:11:16.6266933Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-10-08T22:11:16.6268188Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-10-08T22:11:16.6269608Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-10-08T22:11:16.6271024Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-10-08T22:11:16.6272323Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-10-08T22:11:16.6273610Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-10-08T22:11:16.6275122Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-10-08T22:11:16.6276847Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-10-08T22:11:16.6278967Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-10-08T22:11:16.6280452Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-10-08T22:11:16.6282121Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-10-08T22:11:16.6283872Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-10-08T22:11:16.6285252Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-10-08T22:11:16.6286558Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-10-08T22:11:16.6340728Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-10-08T22:11:16.6342570Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-10-08T22:11:16.6344417Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-10-08T22:11:16.6346321Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-10-08T22:11:16.6348207Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-10-08T22:11:16.6350167Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-10-08T22:11:16.6352019Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-10-08T22:11:16.6353965Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-10-08T22:11:16.6355740Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-10-08T22:11:16.6397327Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-10-08T22:11:16.6445167Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-10-08T22:11:16.6446984Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-10-08T22:11:16.6448650Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-10-08T22:11:16.6450243Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-10-08T22:11:16.6451825Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-10-08T22:11:16.6453414Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-10-08T22:11:16.6454962Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-10-08T22:11:16.6456540Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-10-08T22:11:16.6528308Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-10-08T22:11:16.6602697Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-10-08T22:11:16.6604248Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-10-08T22:11:16.6605541Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-10-08T22:11:16.6606657Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-10-08T22:11:16.6607809Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-10-08T22:11:16.6609035Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-10-08T22:11:16.6610422Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-10-08T22:11:16.6611969Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-10-08T22:11:16.6613485Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-10-08T22:11:16.6614865Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-10-08T22:11:16.6616314Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-10-08T22:11:16.6617758Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-10-08T22:11:16.6619214Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-10-08T22:11:16.6620638Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-10-08T22:11:16.6622075Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-10-08T22:11:16.6639719Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-10-08T22:11:16.6717534Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-10-08T22:11:16.6719119Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-10-08T22:11:16.6720663Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-10-08T22:11:16.6721905Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-10-08T22:11:16.6723091Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-10-08T22:11:16.6724256Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-10-08T22:11:16.6725421Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-10-08T22:11:16.6726712Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-10-08T22:11:16.6728043Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-10-08T22:11:16.6729140Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-10-08T22:11:16.6789787Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-10-08T22:11:16.6790781Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-10-08T22:11:16.6791775Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-10-08T22:11:16.6792930Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-10-08T22:11:16.6800058Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-10-08T22:11:16.6801338Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-10-08T22:11:16.6802655Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-10-08T22:11:16.6804029Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-10-08T22:11:16.6805387Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-10-08T22:11:16.6806897Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-10-08T22:11:16.6808436Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-10-08T22:11:16.6809825Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-10-08T22:11:16.6811201Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-10-08T22:11:16.6812797Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-10-08T22:11:16.6814399Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-10-08T22:11:16.6816266Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-10-08T22:11:16.6817906Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-10-08T22:11:16.6819521Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-10-08T22:11:16.6821148Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-10-08T22:11:16.6822570Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-10-08T22:11:16.6823975Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-10-08T22:11:16.6878376Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-10-08T22:11:16.6880343Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-10-08T22:11:16.6882518Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-10-08T22:11:16.6884598Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-10-08T22:11:16.6886734Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-10-08T22:11:16.6888637Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-10-08T22:11:16.6890544Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-10-08T22:11:16.6892467Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-10-08T22:11:16.6894374Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-10-08T22:11:16.6934343Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-10-08T22:11:16.6983055Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-10-08T22:11:16.6984975Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-10-08T22:11:16.6986749Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-10-08T22:11:16.6988461Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-10-08T22:11:16.6990141Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-10-08T22:11:16.6991817Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-10-08T22:11:16.6993468Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-10-08T22:11:16.6995162Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-10-08T22:11:16.7066674Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-10-08T22:11:16.7140838Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-10-08T22:11:16.7142455Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-10-08T22:11:16.7143836Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-10-08T22:11:16.7145046Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-10-08T22:11:16.7146325Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-10-08T22:11:16.7147647Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-10-08T22:11:16.7149126Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-10-08T22:11:16.7150775Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-10-08T22:11:16.7152376Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-10-08T22:11:16.7153874Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-10-08T22:11:16.7155414Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-10-08T22:11:16.7157150Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-10-08T22:11:16.7158760Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-10-08T22:11:16.7160459Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-10-08T22:11:16.7162000Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-10-08T22:11:16.7164061Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-10-08T22:11:16.7305511Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-10-08T22:11:16.7307052Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-10-08T22:11:16.7308624Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-10-08T22:11:16.7310358Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-10-08T22:11:16.7312048Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-10-08T22:11:16.7313633Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-10-08T22:11:16.7315244Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-10-08T22:11:16.7316868Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-10-08T22:11:16.7318507Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-10-08T22:11:16.7320132Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-10-08T22:11:16.7321747Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-10-08T22:11:16.7339511Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-10-08T22:11:16.7406818Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-10-08T22:11:16.7408530Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-10-08T22:11:16.7410033Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-10-08T22:11:16.7411392Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-10-08T22:11:16.7412634Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-10-08T22:11:16.7413890Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-10-08T22:11:16.7415156Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-10-08T22:11:16.7416362Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-10-08T22:11:16.7417475Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-10-08T22:11:16.7418588Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-10-08T22:11:16.7539990Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-10-08T22:11:16.7592402Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-10-08T22:11:16.7593215Z creating: build/lib/ 2024-10-08T22:11:16.7604487Z inflating: build/lib/libpthreadpool.a 2024-10-08T22:11:16.7614103Z inflating: build/lib/libcpuinfo_internals.a 2024-10-08T22:11:16.7624290Z inflating: build/lib/libcpuinfo.a 2024-10-08T22:11:16.7625406Z inflating: build/lib/libclog.a 2024-10-08T22:11:16.7629107Z inflating: build/lib/libnnpack_reference_layers.a 2024-10-08T22:11:16.7736012Z inflating: build/lib/libprotobuf-lite.a 2024-10-08T22:11:16.7814678Z inflating: build/lib/libgtest.a 2024-10-08T22:11:16.7906259Z inflating: build/lib/libbenchmark.a 2024-10-08T22:11:16.7915581Z inflating: build/lib/libittnotify.a 2024-10-08T22:11:16.8465884Z inflating: build/lib/libprotobuf.a 2024-10-08T22:11:16.8543188Z inflating: build/lib/libasmjit.a 2024-10-08T22:11:16.8577742Z inflating: build/lib/libtensorpipe_uv.a 2024-10-08T22:11:16.8731087Z inflating: build/lib/libgloo.a 2024-10-08T22:11:16.8753205Z inflating: build/lib/libnnpack.a 2024-10-08T22:11:16.8778923Z inflating: build/lib/libfmt.a 2024-10-08T22:11:16.8781112Z inflating: build/lib/libcaffe2_nvrtc.so 2024-10-08T22:11:16.8901796Z inflating: build/lib/libc10.so 2024-10-08T22:11:16.8903778Z inflating: build/lib/libtorch_global_deps.so 2024-10-08T22:11:16.8927405Z inflating: build/lib/libpytorch_qnnpack.a 2024-10-08T22:11:16.9536162Z inflating: build/lib/libprotoc.a 2024-10-08T22:11:16.9558261Z inflating: build/lib/libgmock.a 2024-10-08T22:11:16.9558977Z inflating: build/lib/libgtest_main.a 2024-10-08T22:11:16.9560189Z inflating: build/lib/libbenchmark_main.a 2024-10-08T22:11:18.1889546Z inflating: build/lib/libdnnl.a 2024-10-08T22:11:18.2578959Z inflating: build/lib/libtensorpipe.a 2024-10-08T22:11:18.2649413Z inflating: build/lib/libc10_cuda.so 2024-10-08T22:11:18.4154459Z inflating: build/lib/libfbgemm.a 2024-10-08T22:11:18.4155234Z inflating: build/lib/libgmock_main.a 2024-10-08T22:11:18.4771760Z inflating: build/lib/libkineto.a 2024-10-08T22:11:18.5080087Z inflating: build/lib/libtensorpipe_cuda.a 2024-10-08T22:11:18.5543197Z inflating: build/lib/libgloo_cuda.a 2024-10-08T22:11:18.5594717Z inflating: build/lib/libonnx_proto.a 2024-10-08T22:11:18.5830844Z inflating: build/lib/libXNNPACK.a 2024-10-08T22:11:18.6666786Z inflating: build/lib/libonnx.a 2024-10-08T22:11:21.6710022Z inflating: build/lib/libtorch_cpu.so 2024-10-08T22:11:21.6715242Z inflating: build/lib/libshm.so 2024-10-08T22:11:21.6721208Z inflating: build/lib/libunbox_lib.a 2024-10-08T22:11:24.1243967Z inflating: build/lib/libtorch_cuda.so 2024-10-08T22:11:24.1245362Z inflating: build/lib/libtorch.so 2024-10-08T22:11:24.1248792Z inflating: build/lib/libc10d_cuda_test.so 2024-10-08T22:11:25.2531454Z inflating: build/lib/libtorch_cuda_linalg.so 2024-10-08T22:11:25.2556088Z inflating: build/lib/libjitbackend_test.so 2024-10-08T22:11:25.2586451Z inflating: build/lib/libbackend_with_compiler.so 2024-10-08T22:11:25.2616440Z inflating: build/lib/libaoti_custom_ops.so 2024-10-08T22:11:25.2702786Z inflating: build/lib/libtorchbind_test.so 2024-10-08T22:11:25.5148057Z inflating: build/lib/libtorch_python.so 2024-10-08T22:11:25.5190374Z inflating: build/lib/libnnapi_backend.so 2024-10-08T22:11:25.5190974Z creating: build/bin/ 2024-10-08T22:11:25.5252311Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-10-08T22:11:25.5315388Z inflating: build/bin/c10_DeviceGuard_test 2024-10-08T22:11:25.5378963Z inflating: build/bin/c10_Device_test 2024-10-08T22:11:25.5450922Z inflating: build/bin/c10_DispatchKeySet_test 2024-10-08T22:11:25.5516692Z inflating: build/bin/c10_Scalar_test 2024-10-08T22:11:25.5577058Z inflating: build/bin/c10_StreamGuard_test 2024-10-08T22:11:25.5639986Z inflating: build/bin/c10_SymInt_test 2024-10-08T22:11:25.5706933Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-10-08T22:11:25.5775128Z inflating: build/bin/c10_InlineStreamGuard_test 2024-10-08T22:11:25.5843615Z inflating: build/bin/c10_SizesAndStrides_test 2024-10-08T22:11:25.5930577Z inflating: build/bin/c10_cow_test 2024-10-08T22:11:25.5995404Z inflating: build/bin/c10_Bitset_test 2024-10-08T22:11:25.6055154Z inflating: build/bin/c10_ConstexprCrc_test 2024-10-08T22:11:25.6116828Z inflating: build/bin/c10_DeadlockDetection_test 2024-10-08T22:11:25.6185387Z inflating: build/bin/c10_LeftRight_test 2024-10-08T22:11:25.6247131Z inflating: build/bin/c10_Half_test 2024-10-08T22:11:25.6314576Z inflating: build/bin/c10_Metaprogramming_test 2024-10-08T22:11:25.6379765Z inflating: build/bin/c10_NetworkFlow_test 2024-10-08T22:11:25.6440876Z inflating: build/bin/c10_Synchronized_test 2024-10-08T22:11:25.6504057Z inflating: build/bin/c10_TypeIndex_test 2024-10-08T22:11:25.6572303Z inflating: build/bin/c10_ThreadLocal_test 2024-10-08T22:11:25.6632245Z inflating: build/bin/c10_TypeTraits_test 2024-10-08T22:11:25.6695454Z inflating: build/bin/c10_TypeList_test 2024-10-08T22:11:25.6758830Z inflating: build/bin/c10_accumulate_test 2024-10-08T22:11:25.6827052Z inflating: build/bin/c10_bfloat16_test 2024-10-08T22:11:25.6889137Z inflating: build/bin/c10_bit_cast_test 2024-10-08T22:11:25.6957858Z inflating: build/bin/c10_complex_math_test 2024-10-08T22:11:25.7026029Z inflating: build/bin/c10_complex_test 2024-10-08T22:11:25.7091847Z inflating: build/bin/c10_exception_test 2024-10-08T22:11:25.7153239Z inflating: build/bin/c10_flags_test 2024-10-08T22:11:25.7215206Z inflating: build/bin/c10_generic_math_test 2024-10-08T22:11:25.7414859Z inflating: build/bin/c10_intrusive_ptr_test 2024-10-08T22:11:25.7477473Z inflating: build/bin/c10_irange_test 2024-10-08T22:11:25.7542554Z inflating: build/bin/c10_lazy_test 2024-10-08T22:11:25.7612516Z inflating: build/bin/c10_logging_test 2024-10-08T22:11:25.7704067Z inflating: build/bin/c10_optional_test 2024-10-08T22:11:25.7780330Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-10-08T22:11:25.7846023Z inflating: build/bin/c10_registry_test 2024-10-08T22:11:25.8029174Z inflating: build/bin/c10_small_vector_test 2024-10-08T22:11:25.8092348Z inflating: build/bin/c10_ssize_test 2024-10-08T22:11:25.8155841Z inflating: build/bin/c10_string_util_test 2024-10-08T22:11:25.8228049Z inflating: build/bin/c10_string_view_test 2024-10-08T22:11:25.8290142Z inflating: build/bin/c10_tempfile_test 2024-10-08T22:11:25.8358464Z inflating: build/bin/c10_typeid_test 2024-10-08T22:11:25.8418008Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-10-08T22:11:25.8963399Z inflating: build/bin/protoc-3.13.0.0 2024-10-08T22:11:25.9507695Z inflating: build/bin/protoc 2024-10-08T22:11:25.9572850Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-10-08T22:11:25.9637475Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-10-08T22:11:25.9701995Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-10-08T22:11:25.9766036Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-10-08T22:11:25.9826047Z inflating: build/bin/c10_cuda_CUDATest 2024-10-08T22:11:25.9890914Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-10-08T22:11:25.9955559Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-10-08T22:11:26.0020208Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-10-08T22:11:26.0473556Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-10-08T22:11:26.0957844Z inflating: build/bin/vec_test_all_types_AVX2 2024-10-08T22:11:26.1422690Z inflating: build/bin/vec_test_all_types_AVX512 2024-10-08T22:11:26.1491116Z inflating: build/bin/TCPStoreTest 2024-10-08T22:11:26.1556016Z inflating: build/bin/FileStoreTest 2024-10-08T22:11:26.1620602Z inflating: build/bin/BackoffTest 2024-10-08T22:11:26.1686177Z inflating: build/bin/HashStoreTest 2024-10-08T22:11:26.1691593Z inflating: build/bin/torch_shm_manager 2024-10-08T22:11:26.1708214Z inflating: build/bin/ProcessGroupMPITest 2024-10-08T22:11:26.1774270Z inflating: build/bin/test_edge_op_registration 2024-10-08T22:11:26.1838007Z inflating: build/bin/Dimname_test 2024-10-08T22:11:26.1841538Z inflating: build/bin/example_allreduce 2024-10-08T22:11:26.1920579Z inflating: build/bin/MaybeOwned_test 2024-10-08T22:11:26.1991328Z inflating: build/bin/NamedTensor_test 2024-10-08T22:11:26.2081531Z inflating: build/bin/Dict_test 2024-10-08T22:11:26.2153336Z inflating: build/bin/atest 2024-10-08T22:11:26.2225945Z inflating: build/bin/apply_utils_test 2024-10-08T22:11:26.2288225Z inflating: build/bin/cpu_allocator_test 2024-10-08T22:11:26.2365754Z inflating: build/bin/basic 2024-10-08T22:11:26.2432599Z inflating: build/bin/broadcast_test 2024-10-08T22:11:26.2504433Z inflating: build/bin/cpu_generator_test 2024-10-08T22:11:26.2569672Z inflating: build/bin/cpu_profiling_allocator_test 2024-10-08T22:11:26.2631233Z inflating: build/bin/dlconvertor_test 2024-10-08T22:11:26.2692863Z inflating: build/bin/dispatch_key_set_test 2024-10-08T22:11:26.2760078Z inflating: build/bin/half_test 2024-10-08T22:11:26.2831695Z inflating: build/bin/extension_backend_test 2024-10-08T22:11:26.2892788Z inflating: build/bin/lazy_tensor_test 2024-10-08T22:11:26.2958751Z inflating: build/bin/memory_format_test 2024-10-08T22:11:26.3025864Z inflating: build/bin/math_kernel_test 2024-10-08T22:11:26.3088095Z inflating: build/bin/operator_name_test 2024-10-08T22:11:26.3152982Z inflating: build/bin/memory_overlapping_test 2024-10-08T22:11:26.3267131Z inflating: build/bin/cpu_rng_test 2024-10-08T22:11:26.3330321Z inflating: build/bin/operators_test 2024-10-08T22:11:26.3399323Z inflating: build/bin/native_test 2024-10-08T22:11:26.3516424Z inflating: build/bin/ivalue_test 2024-10-08T22:11:26.3580401Z inflating: build/bin/packedtensoraccessor_test 2024-10-08T22:11:26.3650326Z inflating: build/bin/quantized_test 2024-10-08T22:11:26.3716590Z inflating: build/bin/mobile_memory_cleanup 2024-10-08T22:11:26.3779665Z inflating: build/bin/reportMemoryUsage_test 2024-10-08T22:11:26.3840756Z inflating: build/bin/reduce_ops_test 2024-10-08T22:11:26.3904803Z inflating: build/bin/StorageUtils_test 2024-10-08T22:11:26.3987099Z inflating: build/bin/pow_test 2024-10-08T22:11:26.4056088Z inflating: build/bin/scalar_tensor_test 2024-10-08T22:11:26.4124747Z inflating: build/bin/type_ptr_test 2024-10-08T22:11:26.4196891Z inflating: build/bin/scalar_test 2024-10-08T22:11:26.4263788Z inflating: build/bin/test_parallel 2024-10-08T22:11:26.4328309Z inflating: build/bin/stride_properties_test 2024-10-08T22:11:26.4332122Z inflating: build/bin/thread_init_test 2024-10-08T22:11:26.4396230Z inflating: build/bin/undefined_tensor_test 2024-10-08T22:11:26.4398287Z inflating: build/bin/verify_api_visibility 2024-10-08T22:11:26.4494986Z inflating: build/bin/tensor_iterator_test 2024-10-08T22:11:26.4568718Z inflating: build/bin/IListRef_test 2024-10-08T22:11:26.4631918Z inflating: build/bin/weakref_test 2024-10-08T22:11:26.4761158Z inflating: build/bin/List_test 2024-10-08T22:11:26.4846486Z inflating: build/bin/legacy_vmap_test 2024-10-08T22:11:26.4995502Z inflating: build/bin/kernel_function_legacy_test 2024-10-08T22:11:26.5113546Z inflating: build/bin/kernel_function_test 2024-10-08T22:11:26.5270045Z inflating: build/bin/kernel_lambda_legacy_test 2024-10-08T22:11:26.5333028Z inflating: build/bin/wrapdim_test 2024-10-08T22:11:26.5459493Z inflating: build/bin/kernel_lambda_test 2024-10-08T22:11:26.5534508Z inflating: build/bin/kernel_stackbased_test 2024-10-08T22:11:26.5652381Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-10-08T22:11:26.5716060Z inflating: build/bin/xla_tensor_test 2024-10-08T22:11:26.5779147Z inflating: build/bin/CppSignature_test 2024-10-08T22:11:26.5839128Z inflating: build/bin/op_allowlist_test 2024-10-08T22:11:26.5912787Z inflating: build/bin/type_test 2024-10-08T22:11:26.6278998Z inflating: build/bin/op_registration_test 2024-10-08T22:11:26.6357160Z inflating: build/bin/inline_container_test 2024-10-08T22:11:26.6422197Z inflating: build/bin/cuda_apply_test 2024-10-08T22:11:26.6495074Z inflating: build/bin/cuda_atomic_ops_test 2024-10-08T22:11:26.6581757Z inflating: build/bin/cuda_complex_math_test 2024-10-08T22:11:26.6654096Z inflating: build/bin/cuda_complex_test 2024-10-08T22:11:26.6725175Z inflating: build/bin/cuda_cub_test 2024-10-08T22:11:26.6787242Z inflating: build/bin/cuda_device_test 2024-10-08T22:11:26.6868495Z inflating: build/bin/KernelFunction_test 2024-10-08T22:11:26.6936773Z inflating: build/bin/backend_fallback_test 2024-10-08T22:11:26.7017887Z inflating: build/bin/cuda_distributions_test 2024-10-08T22:11:26.7081929Z inflating: build/bin/cuda_allocator_test 2024-10-08T22:11:26.7151458Z inflating: build/bin/cuda_generator_test 2024-10-08T22:11:26.7218914Z inflating: build/bin/cuda_caching_host_allocator_test 2024-10-08T22:11:26.7282357Z inflating: build/bin/cuda_integer_divider_test 2024-10-08T22:11:26.7343641Z inflating: build/bin/cuda_half_test 2024-10-08T22:11:26.7405363Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-10-08T22:11:26.7466276Z inflating: build/bin/cuda_optional_test 2024-10-08T22:11:26.7529845Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-10-08T22:11:26.7604374Z inflating: build/bin/cuda_stream_test 2024-10-08T22:11:26.7667257Z inflating: build/bin/cuda_dlconvertor_test 2024-10-08T22:11:26.7731021Z inflating: build/bin/cuda_vectorized_test 2024-10-08T22:11:26.7796032Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-10-08T22:11:26.7856138Z inflating: build/bin/cuda_cudnn_test 2024-10-08T22:11:26.7936794Z inflating: build/bin/ProcessGroupGlooTest 2024-10-08T22:11:26.8007118Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-10-08T22:11:26.8024877Z inflating: build/bin/tutorial_tensorexpr 2024-10-08T22:11:26.8103320Z inflating: build/bin/ProcessGroupNCCLTest 2024-10-08T22:11:26.8167851Z inflating: build/bin/ProcessGroupUCCTest 2024-10-08T22:11:26.8243564Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-10-08T22:11:26.8311249Z inflating: build/bin/test_dist_autograd 2024-10-08T22:11:26.8396210Z inflating: build/bin/test_cpp_rpc 2024-10-08T22:11:26.8399474Z inflating: build/bin/parallel_benchmark 2024-10-08T22:11:26.8410633Z inflating: build/bin/aot_model_compiler_test 2024-10-08T22:11:26.8492811Z inflating: build/bin/test_mobile_nnc 2024-10-08T22:11:26.8913219Z inflating: build/bin/test_lazy 2024-10-08T22:11:26.9920134Z inflating: build/bin/test_tensorexpr 2024-10-08T22:11:27.1354324Z inflating: build/bin/test_api 2024-10-08T22:11:27.2057583Z inflating: build/bin/test_jit 2024-10-08T22:11:27.2058155Z creating: .additional_ci_files/ 2024-10-08T22:11:27.2154438Z inflating: .additional_ci_files/test-times.json 2024-10-08T22:11:27.2536530Z inflating: .additional_ci_files/test-class-times.json 2024-10-08T22:11:27.2581016Z ##[group]Run rm artifacts.zip 2024-10-08T22:11:27.2581520Z rm artifacts.zip 2024-10-08T22:11:27.2588906Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:11:27.2589539Z env: 2024-10-08T22:11:27.2589884Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:11:27.2590454Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:11:27.2591057Z ##[endgroup] 2024-10-08T22:11:27.3304754Z ##[group]Run df -H 2024-10-08T22:11:27.3305162Z df -H 2024-10-08T22:11:27.3312073Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:11:27.3312707Z env: 2024-10-08T22:11:27.3313031Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:11:27.3313597Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:11:27.3314206Z ##[endgroup] 2024-10-08T22:11:27.3362975Z Filesystem Size Used Avail Use% Mounted on 2024-10-08T22:11:27.3363628Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-10-08T22:11:27.3364219Z tmpfs 129G 2.3M 129G 1% /dev/shm 2024-10-08T22:11:27.3365017Z tmpfs 52G 635k 52G 1% /run 2024-10-08T22:11:27.3366086Z /dev/xvda1 161G 62G 100G 39% / 2024-10-08T22:11:27.3366693Z tmpfs 129G 16G 113G 13% /tmp 2024-10-08T22:11:27.3367593Z /dev/xvda128 11M 1.4M 9.2M 13% /boot/efi 2024-10-08T22:11:27.3368189Z tmpfs 26G 0 26G 0% /run/user/0 2024-10-08T22:11:27.3417957Z Prepare all required actions 2024-10-08T22:11:27.3418612Z Getting action download info 2024-10-08T22:11:27.5030858Z ##[group]Run ./.github/actions/download-td-artifacts 2024-10-08T22:11:27.5031452Z with: 2024-10-08T22:11:27.5031750Z env: 2024-10-08T22:11:27.5032089Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:11:27.5032740Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:11:27.5033333Z ##[endgroup] 2024-10-08T22:11:27.5079250Z ##[group]Run seemethere/download-artifact-s3@v4 2024-10-08T22:11:27.5079822Z with: 2024-10-08T22:11:27.5080163Z name: td_results 2024-10-08T22:11:27.5080539Z s3-bucket: gha-artifacts 2024-10-08T22:11:27.5080978Z region: us-east-1 2024-10-08T22:11:27.5081348Z env: 2024-10-08T22:11:27.5081667Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:11:27.5082232Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:11:27.5082869Z ##[endgroup] 2024-10-08T22:11:28.1350203Z (node:412651) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-10-08T22:11:28.1351095Z 2024-10-08T22:11:28.1351447Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-10-08T22:11:28.1352346Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-10-08T22:11:28.1353455Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-10-08T22:11:28.2281348Z Found 1 objects with prefix pytorch/pytorch/11242664097/td_results/ 2024-10-08T22:11:28.2282628Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-10-08T22:11:28.3036686Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-10-08T22:11:28.3047008Z Artifact download has finished successfully 2024-10-08T22:11:28.3305263Z ##[group]Run mkdir -p .additional_ci_files 2024-10-08T22:11:28.3305872Z mkdir -p .additional_ci_files 2024-10-08T22:11:28.3306586Z mv td_results.json .additional_ci_files/td_results.json 2024-10-08T22:11:28.3315441Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:11:28.3316074Z env: 2024-10-08T22:11:28.3316421Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:11:28.3316992Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:11:28.3317584Z ##[endgroup] 2024-10-08T22:11:28.3418350Z ##[group]Run .github/scripts/parse_ref.py 2024-10-08T22:11:28.3418984Z .github/scripts/parse_ref.py 2024-10-08T22:11:28.3425512Z shell: /usr/bin/bash -e {0} 2024-10-08T22:11:28.3426246Z env: 2024-10-08T22:11:28.3426589Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:11:28.3427164Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:11:28.3427762Z ##[endgroup] 2024-10-08T22:11:28.3803251Z Prepare all required actions 2024-10-08T22:11:28.3855729Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-10-08T22:11:28.3856318Z with: 2024-10-08T22:11:28.3857310Z github-token: *** 2024-10-08T22:11:28.3857723Z env: 2024-10-08T22:11:28.3858038Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:11:28.3858603Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:11:28.3859199Z ##[endgroup] 2024-10-08T22:11:28.3890438Z ##[group]Run set -eux 2024-10-08T22:11:28.3890865Z set -eux 2024-10-08T22:11:28.3891606Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-10-08T22:11:28.3898391Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:11:28.3899003Z env: 2024-10-08T22:11:28.3899347Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:11:28.3899910Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:11:28.3900796Z GITHUB_TOKEN: *** 2024-10-08T22:11:28.3901186Z ##[endgroup] 2024-10-08T22:11:28.3930130Z + python3 .github/scripts/get_workflow_job_id.py 11242664097 i-0324360b5e1830423 2024-10-08T22:11:31.6650551Z setting job-id=31259716486 2024-10-08T22:11:31.6652041Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-10-08T22:11:31.6935883Z Prepare all required actions 2024-10-08T22:11:31.6936496Z Getting action download info 2024-10-08T22:11:31.8406621Z ##[group]Run ./.github/actions/filter-test-configs 2024-10-08T22:11:31.8407214Z with: 2024-10-08T22:11:31.8407892Z github-token: *** 2024-10-08T22:11:31.8409719Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2024-10-08T22:11:31.8412051Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-10-08T22:11:31.8412948Z env: 2024-10-08T22:11:31.8413291Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:11:31.8413879Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:11:31.8414481Z ##[endgroup] 2024-10-08T22:11:31.8473590Z ##[group]Run nick-fields/retry@v3.0.0 2024-10-08T22:11:31.8474120Z with: 2024-10-08T22:11:31.8474457Z shell: bash 2024-10-08T22:11:31.8474832Z timeout_minutes: 10 2024-10-08T22:11:31.8475245Z max_attempts: 5 2024-10-08T22:11:31.8475621Z retry_wait_seconds: 30 2024-10-08T22:11:31.8477031Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-10-08T22:11:31.8478517Z polling_interval_seconds: 1 2024-10-08T22:11:31.8478987Z warning_on_retry: true 2024-10-08T22:11:31.8479406Z continue_on_error: false 2024-10-08T22:11:31.8479830Z env: 2024-10-08T22:11:31.8480175Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:11:31.8480742Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:11:31.8481804Z GITHUB_TOKEN: *** 2024-10-08T22:11:31.8482222Z ##[endgroup] 2024-10-08T22:11:31.9575046Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-10-08T22:11:32.2809661Z Defaulting to user installation because normal site-packages is not writeable 2024-10-08T22:11:32.3011415Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.9/site-packages (2.27.1) 2024-10-08T22:11:32.3017490Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.9/site-packages (6.0.1) 2024-10-08T22:11:32.3173118Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2024.8.30) 2024-10-08T22:11:32.3187733Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2.0.12) 2024-10-08T22:11:32.3200304Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-10-08T22:11:32.3208600Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-10-08T22:11:32.9565187Z Command completed after 1 attempt(s). 2024-10-08T22:11:32.9641507Z ##[group]Run set -x 2024-10-08T22:11:32.9641927Z set -x 2024-10-08T22:11:32.9642292Z  2024-10-08T22:11:32.9642975Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-10-08T22:11:32.9643812Z # in runner workspace 2024-10-08T22:11:32.9644476Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-10-08T22:11:32.9651948Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:11:32.9652578Z env: 2024-10-08T22:11:32.9652906Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:11:32.9653478Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:11:32.9654089Z ##[endgroup] 2024-10-08T22:11:32.9687361Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-10-08T22:11:33.0012434Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-10-08T22:11:33.0013884Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-10-08T22:11:33.0014536Z echo "Job name: ${JOB_NAME}" 2024-10-08T22:11:33.0015033Z  2024-10-08T22:11:33.0015716Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-10-08T22:11:33.0016547Z # in runner workspace 2024-10-08T22:11:33.0017273Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-10-08T22:11:33.0018078Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-10-08T22:11:33.0018649Z  --job-name "${JOB_NAME}" \ 2024-10-08T22:11:33.0020661Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" \ 2024-10-08T22:11:33.0022697Z  --selected-test-configs "" \ 2024-10-08T22:11:33.0023258Z  --pr-number "${PR_NUMBER}" \ 2024-10-08T22:11:33.0023790Z  --tag "${TAG}" \ 2024-10-08T22:11:33.0024287Z  --event-name "${EVENT_NAME}" \ 2024-10-08T22:11:33.0024824Z  --schedule "${SCHEDULE}" \ 2024-10-08T22:11:33.0025355Z  --branch "${HEAD_BRANCH}" 2024-10-08T22:11:33.0032185Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:11:33.0032895Z env: 2024-10-08T22:11:33.0033244Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:11:33.0033824Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:11:33.0034779Z GITHUB_TOKEN: *** 2024-10-08T22:11:33.0035634Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-10-08T22:11:33.0036549Z PR_NUMBER: 134247 2024-10-08T22:11:33.0036947Z TAG: 2024-10-08T22:11:33.0037285Z EVENT_NAME: pull_request 2024-10-08T22:11:33.0037734Z SCHEDULE: 2024-10-08T22:11:33.0038090Z HEAD_BRANCH: 2024-10-08T22:11:33.0038467Z ##[endgroup] 2024-10-08T22:11:33.0068048Z Workflow: pull 2024-10-08T22:11:33.0069126Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-10-08T22:11:33.2858660Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-10-08T22:11:33.5017304Z ##[group]Run echo "Filtered matrix:" 2024-10-08T22:11:33.5017872Z echo "Filtered matrix:" 2024-10-08T22:11:33.5019809Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" 2024-10-08T22:11:33.5021702Z  2024-10-08T22:11:33.5022029Z echo 2024-10-08T22:11:33.5022502Z echo "Is the current job unstable? False" 2024-10-08T22:11:33.5023100Z  2024-10-08T22:11:33.5023417Z echo 2024-10-08T22:11:33.5023890Z echo "Is keep-going label set? False" 2024-10-08T22:11:33.5024439Z  2024-10-08T22:11:33.5024776Z echo 2024-10-08T22:11:33.5025150Z echo "Renabled issues? " 2024-10-08T22:11:33.5032461Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:11:33.5033169Z env: 2024-10-08T22:11:33.5033518Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:11:33.5034076Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:11:33.5034688Z ##[endgroup] 2024-10-08T22:11:33.5064774Z Filtered matrix: 2024-10-08T22:11:33.5066619Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}]} 2024-10-08T22:11:33.5068790Z 2024-10-08T22:11:33.5068982Z Is the current job unstable? False 2024-10-08T22:11:33.5069356Z 2024-10-08T22:11:33.5069933Z Is keep-going label set? False 2024-10-08T22:11:33.5070288Z 2024-10-08T22:11:33.5070454Z Renabled issues? 2024-10-08T22:11:33.5151797Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-10-08T22:11:33.5152732Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-10-08T22:11:33.5159351Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:11:33.5159981Z env: 2024-10-08T22:11:33.5160303Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:11:33.5160870Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:11:33.5161481Z JOB_TIMEOUT: 360 2024-10-08T22:11:33.5161867Z ##[endgroup] 2024-10-08T22:11:33.5265242Z ##[group]Run set -x 2024-10-08T22:11:33.5265786Z set -x 2024-10-08T22:11:33.5266140Z  2024-10-08T22:11:33.5266584Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-10-08T22:11:33.5267294Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-10-08T22:11:33.5268106Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-10-08T22:11:33.5268730Z  TEST_COMMAND=.ci/onnx/test.sh 2024-10-08T22:11:33.5269251Z else 2024-10-08T22:11:33.5269677Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-10-08T22:11:33.5270205Z fi 2024-10-08T22:11:33.5270530Z  2024-10-08T22:11:33.5271118Z # detached container should get cleaned up by teardown_ec2_linux 2024-10-08T22:11:33.5272076Z # TODO: Stop building test binaries as part of the build phase 2024-10-08T22:11:33.5272932Z # Used for GPU_FLAG since that doesn't play nice 2024-10-08T22:11:33.5273652Z # shellcheck disable=SC2086,SC2090 2024-10-08T22:11:33.5274241Z container_name=$(docker run \ 2024-10-08T22:11:33.5274748Z  ${GPU_FLAG:-} \ 2024-10-08T22:11:33.5275275Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2024-10-08T22:11:33.5275884Z  -e BUILD_ENVIRONMENT \ 2024-10-08T22:11:33.5276368Z  -e PR_NUMBER \ 2024-10-08T22:11:33.5276933Z  -e GITHUB_ACTIONS \ 2024-10-08T22:11:33.5277546Z  -e GITHUB_REPOSITORY \ 2024-10-08T22:11:33.5278051Z  -e GITHUB_WORKFLOW \ 2024-10-08T22:11:33.5278512Z  -e GITHUB_JOB \ 2024-10-08T22:11:33.5278960Z  -e GITHUB_RUN_ID \ 2024-10-08T22:11:33.5279437Z  -e GITHUB_RUN_NUMBER \ 2024-10-08T22:11:33.5279948Z  -e GITHUB_RUN_ATTEMPT \ 2024-10-08T22:11:33.5280425Z  -e JOB_ID \ 2024-10-08T22:11:33.5280844Z  -e JOB_NAME \ 2024-10-08T22:11:33.5281275Z  -e BASE_SHA \ 2024-10-08T22:11:33.5281699Z  -e BRANCH \ 2024-10-08T22:11:33.5282109Z  -e SHA1 \ 2024-10-08T22:11:33.5282539Z  -e AWS_DEFAULT_REGION \ 2024-10-08T22:11:33.5283048Z  -e IN_WHEEL_TEST \ 2024-10-08T22:11:33.5283505Z  -e SHARD_NUMBER \ 2024-10-08T22:11:33.5283976Z  -e TEST_CONFIG \ 2024-10-08T22:11:33.5284435Z  -e NUM_TEST_SHARDS \ 2024-10-08T22:11:33.5284931Z  -e REENABLED_ISSUES \ 2024-10-08T22:11:33.5285426Z  -e CONTINUE_THROUGH_ERROR \ 2024-10-08T22:11:33.5285955Z  -e VERBOSE_TEST_LOGS \ 2024-10-08T22:11:33.5286461Z  -e TEST_SHOWLOCALS \ 2024-10-08T22:11:33.5286948Z  -e NO_TEST_TIMEOUT \ 2024-10-08T22:11:33.5287395Z  -e NO_TD \ 2024-10-08T22:11:33.5287824Z  -e TD_DISTRIBUTED \ 2024-10-08T22:11:33.5288304Z  -e PR_LABELS \ 2024-10-08T22:11:33.5288814Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-10-08T22:11:33.5289367Z  -e SCCACHE_BUCKET \ 2024-10-08T22:11:33.5289857Z  -e SCCACHE_REGION \ 2024-10-08T22:11:33.5290353Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-10-08T22:11:33.5290861Z  -e XLA_CUDA \ 2024-10-08T22:11:33.5291348Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-10-08T22:11:33.5292189Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-10-08T22:11:33.5292847Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-10-08T22:11:33.5293492Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-10-08T22:11:33.5294065Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-10-08T22:11:33.5294634Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-10-08T22:11:33.5295197Z  -e DASHBOARD_TAG \ 2024-10-08T22:11:33.5295672Z  -e IS_A100_RUNNER \ 2024-10-08T22:11:33.5296252Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-10-08T22:11:33.5296952Z  --security-opt seccomp=unconfined \ 2024-10-08T22:11:33.5297541Z  --cap-add=SYS_PTRACE \ 2024-10-08T22:11:33.5298034Z  --ipc=host \ 2024-10-08T22:11:33.5298597Z  --shm-size="${SHM_SIZE}" \ 2024-10-08T22:11:33.5299123Z  --tty \ 2024-10-08T22:11:33.5299517Z  --detach \ 2024-10-08T22:11:33.5299945Z  --name="${container_name}" \ 2024-10-08T22:11:33.5300478Z  --user jenkins \ 2024-10-08T22:11:33.5301092Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-10-08T22:11:33.5301797Z  -w /var/lib/jenkins/workspace \ 2024-10-08T22:11:33.5302328Z  "${DOCKER_IMAGE}" 2024-10-08T22:11:33.5302765Z ) 2024-10-08T22:11:33.5303277Z # Propagate download.pytorch.org IP to container 2024-10-08T22:11:33.5304446Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-10-08T22:11:33.5305655Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-10-08T22:11:33.5306819Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-10-08T22:11:33.5313823Z shell: /usr/bin/bash -e {0} 2024-10-08T22:11:33.5314267Z env: 2024-10-08T22:11:33.5314610Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:11:33.5315183Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:11:33.5315913Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2024-10-08T22:11:33.5316519Z PR_NUMBER: 134247 2024-10-08T22:11:33.5316946Z GITHUB_REPOSITORY: pytorch/pytorch 2024-10-08T22:11:33.5317459Z GITHUB_WORKFLOW: pull 2024-10-08T22:11:33.5317861Z GITHUB_JOB: test 2024-10-08T22:11:33.5318261Z GITHUB_RUN_ID: 11242664097 2024-10-08T22:11:33.5318712Z GITHUB_RUN_NUMBER: 252617 2024-10-08T22:11:33.5319146Z GITHUB_RUN_ATTEMPT: 1 2024-10-08T22:11:33.5319566Z JOB_ID: 31259716486 2024-10-08T22:11:33.5320379Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-10-08T22:11:33.5321294Z BRANCH: pull/134247 2024-10-08T22:11:33.5321769Z SHA1: 47952c0a8001fd60c9ffe52b121d022d440bef8b 2024-10-08T22:11:33.5322421Z BASE_SHA: 05ad01290ec4310464e5073140f6c94a10acf565 2024-10-08T22:11:33.5323012Z TEST_CONFIG: distributed 2024-10-08T22:11:33.5323457Z SHARD_NUMBER: 2 2024-10-08T22:11:33.5323874Z NUM_TEST_SHARDS: 3 2024-10-08T22:11:33.5324289Z REENABLED_ISSUES: 2024-10-08T22:11:33.5324714Z CONTINUE_THROUGH_ERROR: False 2024-10-08T22:11:33.5325195Z VERBOSE_TEST_LOGS: False 2024-10-08T22:11:33.5325624Z TEST_SHOWLOCALS: False 2024-10-08T22:11:33.5326059Z NO_TEST_TIMEOUT: False 2024-10-08T22:11:33.5326478Z NO_TD: False 2024-10-08T22:11:33.5326838Z TD_DISTRIBUTED: False 2024-10-08T22:11:33.5327363Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-10-08T22:11:33.5327969Z SCCACHE_REGION: us-east-1 2024-10-08T22:11:33.5328432Z SCCACHE_S3_KEY_PREFIX: pull 2024-10-08T22:11:33.5328862Z SHM_SIZE: 2g 2024-10-08T22:11:33.5330069Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:11:33.5331380Z XLA_CUDA: 2024-10-08T22:11:33.5331981Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-10-08T22:11:33.5332760Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-10-08T22:11:33.5333485Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-10-08T22:11:33.5333997Z DASHBOARD_TAG: 2024-10-08T22:11:33.5334644Z HUGGING_FACE_HUB_TOKEN: *** 2024-10-08T22:11:33.5335131Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-10-08T22:11:33.5335584Z IS_A100_RUNNER: 0 2024-10-08T22:11:33.5335965Z ##[endgroup] 2024-10-08T22:11:33.5364327Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-10-08T22:11:33.5365196Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2024-10-08T22:11:33.5365815Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-10-08T22:11:33.5374850Z +++ nproc --ignore=2 2024-10-08T22:11:33.5394622Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_REGION -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e IS_A100_RUNNER --env-file=/tmp/github_env_11242664097 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:11:44.7953013Z + container_name=cd91a56406f3434120c824708028491b4d148dc6169c77fb24a60d78398a479f 2024-10-08T22:11:44.7957769Z + grep download.pytorch.org /etc/hosts 2024-10-08T22:11:44.7959815Z + docker exec -i cd91a56406f3434120c824708028491b4d148dc6169c77fb24a60d78398a479f sudo bash -c '/bin/cat >> /etc/hosts' 2024-10-08T22:11:44.9722027Z + echo DOCKER_CONTAINER_ID=cd91a56406f3434120c824708028491b4d148dc6169c77fb24a60d78398a479f 2024-10-08T22:11:44.9726950Z ++ echo dist/torch-2.6.0a0+git47952c0-cp310-cp310-linux_x86_64.whl 2024-10-08T22:11:44.9729753Z + docker exec -t cd91a56406f3434120c824708028491b4d148dc6169c77fb24a60d78398a479f sh -c 'pip install dist/torch-2.6.0a0+git47952c0-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-10-08T22:11:45.5108454Z Processing ./dist/torch-2.6.0a0+git47952c0-cp310-cp310-linux_x86_64.whl (from torch==2.6.0a0+git47952c0) 2024-10-08T22:11:46.6654854Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git47952c0->torch==2.6.0a0+git47952c0) (3.13.1) 2024-10-08T22:11:46.6659974Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git47952c0->torch==2.6.0a0+git47952c0) (4.12.2) 2024-10-08T22:11:46.6664266Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git47952c0->torch==2.6.0a0+git47952c0) (2.8.8) 2024-10-08T22:11:46.6669878Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git47952c0->torch==2.6.0a0+git47952c0) (3.1.4) 2024-10-08T22:11:46.6675004Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git47952c0->torch==2.6.0a0+git47952c0) (2024.6.1) 2024-10-08T22:11:46.6682795Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git47952c0->torch==2.6.0a0+git47952c0) (1.13.1) 2024-10-08T22:11:46.6706946Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.6.0a0+git47952c0->torch==2.6.0a0+git47952c0) (1.3.0) 2024-10-08T22:11:46.6728188Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git47952c0->torch==2.6.0a0+git47952c0) (3.3.0) 2024-10-08T22:11:46.6754798Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.6.0a0+git47952c0->torch==2.6.0a0+git47952c0) (1.21.2) 2024-10-08T22:11:46.7286767Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.6.0a0+git47952c0->torch==2.6.0a0+git47952c0) (2.1.5) 2024-10-08T22:11:47.1870956Z Installing collected packages: torch 2024-10-08T22:12:00.4872463Z Successfully installed torch-2.6.0a0+git47952c0 2024-10-08T22:12:00.5712118Z + export TERM=vt100 2024-10-08T22:12:00.5712587Z + TERM=vt100 2024-10-08T22:12:00.5715717Z ++ dirname .ci/pytorch/test.sh 2024-10-08T22:12:00.5724331Z + source .ci/pytorch/common.sh 2024-10-08T22:12:00.5728294Z +++ dirname .ci/pytorch/common.sh 2024-10-08T22:12:00.5737031Z ++ source .ci/pytorch/common_utils.sh 2024-10-08T22:12:00.5739075Z +++ declare -f -t trap_add 2024-10-08T22:12:00.5747169Z ++ set -ex 2024-10-08T22:12:00.5748127Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-10-08T22:12:00.5748741Z ++ BUILD_TEST_LIBTORCH=0 2024-10-08T22:12:00.5749378Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-10-08T22:12:00.5752463Z ++ stat -c %u /var/lib/jenkins/workspace 2024-10-08T22:12:00.5770165Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-10-08T22:12:00.5770881Z + trap_add cleanup_workspace EXIT 2024-10-08T22:12:00.5771414Z + trap_add_cmd=cleanup_workspace 2024-10-08T22:12:00.5771884Z + shift 2024-10-08T22:12:00.5773987Z + for trap_add_name in "$@" 2024-10-08T22:12:00.5778808Z +++ trap -p EXIT 2024-10-08T22:12:00.5781634Z ++ eval 'extract_trap_cmd ' 2024-10-08T22:12:00.5782096Z +++ extract_trap_cmd 2024-10-08T22:12:00.5782560Z +++ printf '%s\n' '' 2024-10-08T22:12:00.5783079Z ++ printf '%s\n' cleanup_workspace 2024-10-08T22:12:00.5784426Z + trap -- ' 2024-10-08T22:12:00.5784849Z cleanup_workspace' EXIT 2024-10-08T22:12:00.5785468Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-10-08T22:12:01.1895761Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-10-08T22:12:01.1918024Z + echo 'Environment variables:' 2024-10-08T22:12:01.1918550Z Environment variables: 2024-10-08T22:12:01.1918938Z + env 2024-10-08T22:12:01.1928789Z INSTALLED_DB=yes 2024-10-08T22:12:01.1929561Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-10-08T22:12:01.1930508Z NVIDIA_VISIBLE_DEVICES=all 2024-10-08T22:12:01.1931002Z NV_NVML_DEV_VERSION=11.8.86-1 2024-10-08T22:12:01.1932132Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-10-08T22:12:01.1933261Z CONTINUE_THROUGH_ERROR=False 2024-10-08T22:12:01.1933936Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-10-08T22:12:01.1934829Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-10-08T22:12:01.1935568Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-10-08T22:12:01.1936222Z HOSTNAME=cd91a56406f3 2024-10-08T22:12:01.1937341Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_9a0bf9fe-6037-458a-8068-898e688882a2 2024-10-08T22:12:01.1938373Z GITHUB_ACTION=__self 2024-10-08T22:12:01.1938821Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-10-08T22:12:01.1941529Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-10-08T22:12:01.1944504Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-10-08T22:12:01.1945159Z NV_NVTX_VERSION=11.8.86-1 2024-10-08T22:12:01.1945601Z GITHUB_RUN_NUMBER=252617 2024-10-08T22:12:01.1946348Z TEST_CONFIG=distributed 2024-10-08T22:12:01.1946801Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-10-08T22:12:01.1947457Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-10-08T22:12:01.1947983Z IS_A100_RUNNER=0 2024-10-08T22:12:01.1948432Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-10-08T22:12:01.1949588Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-10-08T22:12:01.1950559Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-10-08T22:12:01.1951623Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-10-08T22:12:01.1952625Z GITHUB_TRIGGERING_ACTOR=kwen2501 2024-10-08T22:12:01.1953214Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-10-08T22:12:01.1953809Z GITHUB_REF_TYPE=branch 2024-10-08T22:12:01.1954221Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-10-08T22:12:01.1954718Z NCCL_VERSION=2.16.2-1 2024-10-08T22:12:01.1955417Z BASE_SHA=05ad01290ec4310464e5073140f6c94a10acf565 2024-10-08T22:12:01.1956003Z XLA_CUDA= 2024-10-08T22:12:01.1956629Z HUGGING_FACE_HUB_TOKEN=*** 2024-10-08T22:12:01.1961198Z *** 2024-10-08T22:12:01.1961576Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-10-08T22:12:01.1962100Z GITHUB_REPOSITORY_ID=65600975 2024-10-08T22:12:01.1963171Z GITHUB_ACTIONS=true 2024-10-08T22:12:01.1963593Z NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:12:01.1964279Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-10-08T22:12:01.1964966Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-10-08T22:12:01.1965542Z SHA1=47952c0a8001fd60c9ffe52b121d022d440bef8b 2024-10-08T22:12:01.1966201Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-10-08T22:12:01.1966822Z GITHUB_SHA=cbe39ffc96498585a83508a23de66f3eecb751d5 2024-10-08T22:12:01.1967733Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/134247/merge 2024-10-08T22:12:01.1968524Z UCC_HOME=/usr 2024-10-08T22:12:01.1968962Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-10-08T22:12:01.1969525Z VERBOSE_TEST_LOGS=False 2024-10-08T22:12:01.1969985Z NVIDIA_PRODUCT_NAME=CUDA 2024-10-08T22:12:01.1970605Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-10-08T22:12:01.1971245Z GITHUB_REF=refs/pull/134247/merge 2024-10-08T22:12:01.1971821Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-10-08T22:12:01.1972305Z SHARD_NUMBER=2 2024-10-08T22:12:01.1972709Z GITHUB_REF_PROTECTED=false 2024-10-08T22:12:01.1973172Z HOME=/var/lib/jenkins 2024-10-08T22:12:01.1973665Z GITHUB_API_URL=https://api.github.com 2024-10-08T22:12:01.1974237Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-10-08T22:12:01.1974885Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-10-08T22:12:01.1975549Z SCCACHE_S3_KEY_PREFIX=pull 2024-10-08T22:12:01.1976022Z CUDA_VERSION=11.8.0 2024-10-08T22:12:01.1976597Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-10-08T22:12:01.1977189Z NUM_TEST_SHARDS=3 2024-10-08T22:12:01.1977586Z UCX_HOME=/usr 2024-10-08T22:12:01.1978278Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-10-08T22:12:01.1979832Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_9a0bf9fe-6037-458a-8068-898e688882a2 2024-10-08T22:12:01.1981483Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-10-08T22:12:01.1983141Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_9a0bf9fe-6037-458a-8068-898e688882a2 2024-10-08T22:12:01.1984671Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-10-08T22:12:01.1985572Z GITHUB_EVENT_NAME=pull_request 2024-10-08T22:12:01.1986049Z DASHBOARD_TAG= 2024-10-08T22:12:01.1986448Z GITHUB_RUN_ID=11242664097 2024-10-08T22:12:01.1987084Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-10-08T22:12:01.1987818Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-10-08T22:12:01.1989172Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_9a0bf9fe-6037-458a-8068-898e688882a2 2024-10-08T22:12:01.1990361Z GITHUB_ACTOR=kwen2501 2024-10-08T22:12:01.1990857Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-10-08T22:12:01.1991379Z PR_NUMBER=134247 2024-10-08T22:12:01.1991770Z GITHUB_RUN_ATTEMPT=1 2024-10-08T22:12:01.1992219Z ANACONDA_PYTHON_VERSION=3.10 2024-10-08T22:12:01.1993066Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-10-08T22:12:01.1993668Z TERM=vt100 2024-10-08T22:12:01.1994116Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-10-08T22:12:01.1994653Z INSTALLED_VISION=yes 2024-10-08T22:12:01.1995081Z BRANCH=pull/134247 2024-10-08T22:12:01.1995544Z SCCACHE_REGION=us-east-1 2024-10-08T22:12:01.1996027Z OPENSSL_ROOT_DIR=/opt/openssl 2024-10-08T22:12:01.1996564Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-10-08T22:12:01.1997111Z CUDA_PATH=/usr/local/cuda 2024-10-08T22:12:01.1998154Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-10-08T22:12:01.1999123Z GITHUB_SERVER_URL=https://github.com 2024-10-08T22:12:01.1999881Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-10-08T22:12:01.2000512Z REENABLED_ISSUES= 2024-10-08T22:12:01.2000891Z SHLVL=1 2024-10-08T22:12:01.2001213Z MAX_JOBS=30 2024-10-08T22:12:01.2001625Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-10-08T22:12:01.2002062Z NVARCH=x86_64 2024-10-08T22:12:01.2002432Z GITHUB_ACTOR_ID=6676466 2024-10-08T22:12:01.2003006Z GITHUB_WORKFLOW_SHA=cbe39ffc96498585a83508a23de66f3eecb751d5 2024-10-08T22:12:01.2003663Z GITHUB_REF_NAME=134247/merge 2024-10-08T22:12:01.2004210Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-10-08T22:12:01.2005070Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-10-08T22:12:01.2005797Z GITHUB_JOB=test 2024-10-08T22:12:01.2006296Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-10-08T22:12:01.2007007Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-10-08T22:12:01.2007655Z NO_TEST_TIMEOUT=False 2024-10-08T22:12:01.2008068Z TD_DISTRIBUTED=False 2024-10-08T22:12:01.2008532Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-10-08T22:12:01.2009097Z GITHUB_REPOSITORY=pytorch/pytorch 2024-10-08T22:12:01.2009625Z NV_NVPROF_VERSION=11.8.87-1 2024-10-08T22:12:01.2010079Z GITHUB_RETENTION_DAYS=90 2024-10-08T22:12:01.2010495Z OPENSSL_DIR=/opt/openssl 2024-10-08T22:12:01.2010948Z GITHUB_ACTION_REPOSITORY= 2024-10-08T22:12:01.2012278Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-10-08T22:12:01.2013684Z GITHUB_BASE_REF=gh/kwen2501/47/base 2024-10-08T22:12:01.2014189Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-10-08T22:12:01.2014649Z CI=true 2024-10-08T22:12:01.2015044Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-10-08T22:12:01.2015541Z GITHUB_REPOSITORY_OWNER=pytorch 2024-10-08T22:12:01.2016012Z JOB_ID=31259716486 2024-10-08T22:12:01.2016401Z INSTALLED_PROTOBUF=yes 2024-10-08T22:12:01.2016840Z GITHUB_HEAD_REF=gh/kwen2501/47/head 2024-10-08T22:12:01.2017315Z GITHUB_ACTION_REF= 2024-10-08T22:12:01.2017896Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-10-08T22:12:01.2018490Z TEST_SHOWLOCALS=False 2024-10-08T22:12:01.2018903Z GITHUB_WORKFLOW=pull 2024-10-08T22:12:01.2019315Z DEBIAN_FRONTEND=noninteractive 2024-10-08T22:12:01.2020501Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_9a0bf9fe-6037-458a-8068-898e688882a2 2024-10-08T22:12:01.2021549Z NO_TD=False 2024-10-08T22:12:01.2021905Z SKIP_SCCACHE_INITIALIZATION=1 2024-10-08T22:12:01.2022364Z _=/usr/bin/env 2024-10-08T22:12:01.2022980Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-10-08T22:12:01.2236494Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-10-08T22:12:01.2238006Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-10-08T22:12:01.2239139Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-10-08T22:12:01.2240282Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-10-08T22:12:01.2241064Z + BUILD_DIR=build 2024-10-08T22:12:01.2241478Z + BUILD_RENAMED_DIR=build_renamed 2024-10-08T22:12:01.2241949Z + BUILD_BIN_DIR=build/bin 2024-10-08T22:12:01.2242614Z + SHARD_NUMBER=2 2024-10-08T22:12:01.2243006Z + NUM_TEST_SHARDS=3 2024-10-08T22:12:01.2243602Z + export VALGRIND=ON 2024-10-08T22:12:01.2244001Z + VALGRIND=ON 2024-10-08T22:12:01.2244600Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2024-10-08T22:12:01.2245196Z + [[ 0 == \1 ]] 2024-10-08T22:12:01.2245553Z + [[ False == \1 ]] 2024-10-08T22:12:01.2246126Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2024-10-08T22:12:01.2246756Z ++ realpath build/custom_test_artifacts 2024-10-08T22:12:01.2255343Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-10-08T22:12:01.2256234Z + [[ -n '' ]] 2024-10-08T22:12:01.2256685Z + echo 'Environment variables' 2024-10-08T22:12:01.2257166Z Environment variables 2024-10-08T22:12:01.2257564Z + env 2024-10-08T22:12:01.2265522Z INSTALLED_DB=yes 2024-10-08T22:12:01.2266991Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-10-08T22:12:01.2268066Z NVIDIA_VISIBLE_DEVICES=all 2024-10-08T22:12:01.2269196Z NV_NVML_DEV_VERSION=11.8.86-1 2024-10-08T22:12:01.2270780Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-10-08T22:12:01.2271715Z CONTINUE_THROUGH_ERROR=False 2024-10-08T22:12:01.2272371Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-10-08T22:12:01.2273010Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-10-08T22:12:01.2273708Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-10-08T22:12:01.2274310Z HOSTNAME=cd91a56406f3 2024-10-08T22:12:01.2275643Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_9a0bf9fe-6037-458a-8068-898e688882a2 2024-10-08T22:12:01.2276827Z GITHUB_ACTION=__self 2024-10-08T22:12:01.2277271Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-10-08T22:12:01.2280161Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-10-08T22:12:01.2283034Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-10-08T22:12:01.2283666Z NV_NVTX_VERSION=11.8.86-1 2024-10-08T22:12:01.2284104Z GITHUB_RUN_NUMBER=252617 2024-10-08T22:12:01.2284541Z TEST_CONFIG=distributed 2024-10-08T22:12:01.2284987Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-10-08T22:12:01.2285571Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-10-08T22:12:01.2286085Z IS_A100_RUNNER=0 2024-10-08T22:12:01.2286525Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-10-08T22:12:01.2287080Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-10-08T22:12:01.2287564Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-10-08T22:12:01.2288059Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-10-08T22:12:01.2288536Z GITHUB_TRIGGERING_ACTOR=kwen2501 2024-10-08T22:12:01.2289091Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-10-08T22:12:01.2289688Z GITHUB_REF_TYPE=branch 2024-10-08T22:12:01.2290115Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-10-08T22:12:01.2290594Z NCCL_VERSION=2.16.2-1 2024-10-08T22:12:01.2291063Z BASE_SHA=05ad01290ec4310464e5073140f6c94a10acf565 2024-10-08T22:12:01.2291607Z XLA_CUDA= 2024-10-08T22:12:01.2292194Z HUGGING_FACE_HUB_TOKEN=*** 2024-10-08T22:12:01.2292688Z *** 2024-10-08T22:12:01.2293015Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-10-08T22:12:01.2293510Z GITHUB_REPOSITORY_ID=65600975 2024-10-08T22:12:01.2293971Z GITHUB_ACTIONS=true 2024-10-08T22:12:01.2294387Z NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:12:01.2295000Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-10-08T22:12:01.2295678Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-10-08T22:12:01.2296267Z SHA1=47952c0a8001fd60c9ffe52b121d022d440bef8b 2024-10-08T22:12:01.2296920Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-10-08T22:12:01.2297522Z GITHUB_SHA=cbe39ffc96498585a83508a23de66f3eecb751d5 2024-10-08T22:12:01.2298435Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/134247/merge 2024-10-08T22:12:01.2299238Z UCC_HOME=/usr 2024-10-08T22:12:01.2299673Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-10-08T22:12:01.2300377Z VERBOSE_TEST_LOGS=False 2024-10-08T22:12:01.2300808Z NVIDIA_PRODUCT_NAME=CUDA 2024-10-08T22:12:01.2301402Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-10-08T22:12:01.2301976Z GITHUB_REF=refs/pull/134247/merge 2024-10-08T22:12:01.2302508Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-10-08T22:12:01.2302980Z SHARD_NUMBER=2 2024-10-08T22:12:01.2303364Z GITHUB_REF_PROTECTED=false 2024-10-08T22:12:01.2303783Z HOME=/var/lib/jenkins 2024-10-08T22:12:01.2304241Z GITHUB_API_URL=https://api.github.com 2024-10-08T22:12:01.2304790Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-10-08T22:12:01.2305385Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-10-08T22:12:01.2305966Z SCCACHE_S3_KEY_PREFIX=pull 2024-10-08T22:12:01.2306404Z CUDA_VERSION=11.8.0 2024-10-08T22:12:01.2307054Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-10-08T22:12:01.2307603Z NUM_TEST_SHARDS=3 2024-10-08T22:12:01.2307981Z UCX_HOME=/usr 2024-10-08T22:12:01.2308693Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-10-08T22:12:01.2310107Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_9a0bf9fe-6037-458a-8068-898e688882a2 2024-10-08T22:12:01.2311665Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-10-08T22:12:01.2313222Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_9a0bf9fe-6037-458a-8068-898e688882a2 2024-10-08T22:12:01.2314639Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-10-08T22:12:01.2315483Z GITHUB_EVENT_NAME=pull_request 2024-10-08T22:12:01.2315945Z DASHBOARD_TAG= 2024-10-08T22:12:01.2316320Z GITHUB_RUN_ID=11242664097 2024-10-08T22:12:01.2316897Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-10-08T22:12:01.2317586Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-10-08T22:12:01.2318875Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_9a0bf9fe-6037-458a-8068-898e688882a2 2024-10-08T22:12:01.2319975Z GITHUB_ACTOR=kwen2501 2024-10-08T22:12:01.2320421Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-10-08T22:12:01.2320900Z PR_NUMBER=134247 2024-10-08T22:12:01.2321284Z GITHUB_RUN_ATTEMPT=1 2024-10-08T22:12:01.2321659Z VALGRIND=ON 2024-10-08T22:12:01.2322034Z ANACONDA_PYTHON_VERSION=3.10 2024-10-08T22:12:01.2322592Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-10-08T22:12:01.2323168Z TERM=vt100 2024-10-08T22:12:01.2323582Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-10-08T22:12:01.2324128Z INSTALLED_VISION=yes 2024-10-08T22:12:01.2324553Z BRANCH=pull/134247 2024-10-08T22:12:01.2325011Z SCCACHE_REGION=us-east-1 2024-10-08T22:12:01.2325493Z OPENSSL_ROOT_DIR=/opt/openssl 2024-10-08T22:12:01.2326049Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-10-08T22:12:01.2326611Z CUDA_PATH=/usr/local/cuda 2024-10-08T22:12:01.2327629Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-10-08T22:12:01.2328645Z GITHUB_SERVER_URL=https://github.com 2024-10-08T22:12:01.2329304Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-10-08T22:12:01.2329926Z REENABLED_ISSUES= 2024-10-08T22:12:01.2330296Z SHLVL=1 2024-10-08T22:12:01.2330632Z MAX_JOBS=30 2024-10-08T22:12:01.2331055Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-10-08T22:12:01.2331528Z NVARCH=x86_64 2024-10-08T22:12:01.2331898Z GITHUB_ACTOR_ID=6676466 2024-10-08T22:12:01.2332511Z GITHUB_WORKFLOW_SHA=cbe39ffc96498585a83508a23de66f3eecb751d5 2024-10-08T22:12:01.2333191Z GITHUB_REF_NAME=134247/merge 2024-10-08T22:12:01.2333770Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-10-08T22:12:01.2334682Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-10-08T22:12:01.2335447Z GITHUB_JOB=test 2024-10-08T22:12:01.2336002Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-10-08T22:12:01.2336765Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-10-08T22:12:01.2337416Z NO_TEST_TIMEOUT=False 2024-10-08T22:12:01.2337833Z TD_DISTRIBUTED=False 2024-10-08T22:12:01.2338437Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-10-08T22:12:01.2338969Z GITHUB_REPOSITORY=pytorch/pytorch 2024-10-08T22:12:01.2339497Z NV_NVPROF_VERSION=11.8.87-1 2024-10-08T22:12:01.2339946Z GITHUB_RETENTION_DAYS=90 2024-10-08T22:12:01.2340365Z OPENSSL_DIR=/opt/openssl 2024-10-08T22:12:01.2340801Z GITHUB_ACTION_REPOSITORY= 2024-10-08T22:12:01.2342116Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-10-08T22:12:01.2343484Z GITHUB_BASE_REF=gh/kwen2501/47/base 2024-10-08T22:12:01.2343989Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-10-08T22:12:01.2344451Z CI=true 2024-10-08T22:12:01.2344848Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-10-08T22:12:01.2345469Z GITHUB_REPOSITORY_OWNER=pytorch 2024-10-08T22:12:01.2345931Z JOB_ID=31259716486 2024-10-08T22:12:01.2346322Z INSTALLED_PROTOBUF=yes 2024-10-08T22:12:01.2346757Z GITHUB_HEAD_REF=gh/kwen2501/47/head 2024-10-08T22:12:01.2347248Z GITHUB_ACTION_REF= 2024-10-08T22:12:01.2347814Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-10-08T22:12:01.2348408Z TEST_SHOWLOCALS=False 2024-10-08T22:12:01.2348822Z GITHUB_WORKFLOW=pull 2024-10-08T22:12:01.2349229Z DEBIAN_FRONTEND=noninteractive 2024-10-08T22:12:01.2350412Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_9a0bf9fe-6037-458a-8068-898e688882a2 2024-10-08T22:12:01.2351443Z NO_TD=False 2024-10-08T22:12:01.2351816Z SKIP_SCCACHE_INITIALIZATION=1 2024-10-08T22:12:01.2352248Z _=/usr/bin/env 2024-10-08T22:12:01.2352681Z + echo 'Testing pytorch' 2024-10-08T22:12:01.2353108Z Testing pytorch 2024-10-08T22:12:01.2353515Z + export LANG=C.UTF-8 2024-10-08T22:12:01.2353927Z + LANG=C.UTF-8 2024-10-08T22:12:01.2354303Z + PR_NUMBER=134247 2024-10-08T22:12:01.2354726Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-10-08T22:12:01.2355273Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-10-08T22:12:01.2355990Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-10-08T22:12:01.2356599Z + [[ distributed == \s\l\o\w ]] 2024-10-08T22:12:01.2357316Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2024-10-08T22:12:01.2358100Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-10-08T22:12:01.2358753Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-10-08T22:12:01.2359341Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-10-08T22:12:01.2359878Z + [[ distributed == *crossref* ]] 2024-10-08T22:12:01.2360509Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-10-08T22:12:01.2361255Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-10-08T22:12:01.2362015Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-10-08T22:12:01.2363499Z + pip_install --user ninja==1.10.2 2024-10-08T22:12:01.2364176Z + pip install --progress-bar off --user ninja==1.10.2 2024-10-08T22:12:01.8041353Z Collecting ninja==1.10.2 2024-10-08T22:12:01.8284039Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-10-08T22:12:01.8412089Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-10-08T22:12:02.2883088Z Installing collected packages: ninja 2024-10-08T22:12:02.2990440Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-10-08T22:12:02.2992085Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-10-08T22:12:02.3058636Z Successfully installed ninja-1.10.2 2024-10-08T22:12:02.3826978Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-10-08T22:12:02.3829743Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-10-08T22:12:02.3831619Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-10-08T22:12:02.3832624Z + install_tlparse 2024-10-08T22:12:02.3833126Z + pip_install --user tlparse==0.3.25 2024-10-08T22:12:02.3833821Z + pip install --progress-bar off --user tlparse==0.3.25 2024-10-08T22:12:02.8942265Z Collecting tlparse==0.3.25 2024-10-08T22:12:02.9157701Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-10-08T22:12:02.9278552Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-10-08T22:12:03.3979450Z Installing collected packages: tlparse 2024-10-08T22:12:03.4402383Z Successfully installed tlparse-0.3.25 2024-10-08T22:12:03.5141034Z ++ python -m site --user-base 2024-10-08T22:12:03.5466068Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-10-08T22:12:03.5468483Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2024-10-08T22:12:03.5469288Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2024-10-08T22:12:03.5470078Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-10-08T22:12:03.5471218Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2024-10-08T22:12:03.5472578Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2024-10-08T22:12:03.5473430Z + cd test 2024-10-08T22:12:03.5474108Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-10-08T22:12:05.7583861Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-10-08T22:12:05.7584550Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-10-08T22:12:05.7589136Z + DYNAMO_BENCHMARK_FLAGS=() 2024-10-08T22:12:05.7590153Z + [[ distributed == *pr_time_benchmarks* ]] 2024-10-08T22:12:05.7591388Z + [[ distributed == *dynamo_eager* ]] 2024-10-08T22:12:05.7592044Z + [[ distributed == *aot_eager* ]] 2024-10-08T22:12:05.7592569Z + [[ distributed == *aot_inductor* ]] 2024-10-08T22:12:05.7593099Z + [[ distributed == *inductor* ]] 2024-10-08T22:12:05.7593606Z + [[ distributed == *dynamic* ]] 2024-10-08T22:12:05.7594099Z + [[ distributed == *cpu* ]] 2024-10-08T22:12:05.7594806Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-10-08T22:12:05.7632031Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-10-08T22:12:05.7632853Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2024-10-08T22:12:05.7635572Z + cd test 2024-10-08T22:12:05.7636657Z + python -c 'import torch; print(torch.__config__.show())' 2024-10-08T22:12:07.7399779Z PyTorch built with: 2024-10-08T22:12:07.7401139Z - GCC 9.4 2024-10-08T22:12:07.7401889Z - C++ Version: 201703 2024-10-08T22:12:07.7403024Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-10-08T22:12:07.7404427Z - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-10-08T22:12:07.7405308Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-10-08T22:12:07.7405968Z - LAPACK is enabled (usually provided by MKL) 2024-10-08T22:12:07.7406567Z - NNPACK is enabled 2024-10-08T22:12:07.7407062Z - CPU capability usage: AVX2 2024-10-08T22:12:07.7407583Z - CUDA Runtime 11.8 2024-10-08T22:12:07.7408240Z - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52 2024-10-08T22:12:07.7408949Z - CuDNN 90.1 2024-10-08T22:12:07.7409360Z - Magma 2.6.1 2024-10-08T22:12:07.7418179Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.6.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-10-08T22:12:07.7426553Z 2024-10-08T22:12:08.0292995Z + cd test 2024-10-08T22:12:08.0294274Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-10-08T22:12:09.9789014Z ATen/Parallel: 2024-10-08T22:12:09.9789846Z at::get_num_threads() : 16 2024-10-08T22:12:09.9790941Z at::get_num_interop_threads() : 16 2024-10-08T22:12:09.9791585Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-10-08T22:12:09.9792085Z omp_get_max_threads() : 16 2024-10-08T22:12:09.9793428Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-10-08T22:12:09.9794426Z mkl_get_max_threads() : 16 2024-10-08T22:12:09.9795240Z Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-10-08T22:12:09.9796045Z std::thread::hardware_concurrency() : 32 2024-10-08T22:12:09.9796586Z Environment variables: 2024-10-08T22:12:09.9797008Z OMP_NUM_THREADS : [not set] 2024-10-08T22:12:09.9797474Z MKL_NUM_THREADS : [not set] 2024-10-08T22:12:09.9797948Z ATen parallel backend: OpenMP 2024-10-08T22:12:09.9798273Z 2024-10-08T22:12:10.2479658Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-10-08T22:12:10.2480718Z + [[ distributed == *backward* ]] 2024-10-08T22:12:10.2481231Z + [[ distributed == *xla* ]] 2024-10-08T22:12:10.2481699Z + [[ distributed == *executorch* ]] 2024-10-08T22:12:10.2482269Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-10-08T22:12:10.2483040Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-10-08T22:12:10.2483676Z + [[ distributed == distributed ]] 2024-10-08T22:12:10.2484148Z + test_distributed 2024-10-08T22:12:10.2484664Z + echo 'Testing distributed python tests' 2024-10-08T22:12:10.2485231Z Testing distributed python tests 2024-10-08T22:12:10.2486021Z + python test/run_test.py --distributed-tests --shard 2 3 --verbose 2024-10-08T22:12:10.3836588Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-10-08T22:12:10.3838075Z import pkg_resources 2024-10-08T22:12:15.5841003Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-10-08T22:12:15.6365582Z Ignoring disabled issues: [''] 2024-10-08T22:12:15.6543902Z Found test times from artifacts 2024-10-08T22:12:15.7248514Z Found test times from artifacts 2024-10-08T22:12:15.7270228Z Running 25% of tests based on TD 2024-10-08T22:12:15.7394429Z Running parallel tests on 2 processes 2024-10-08T22:12:15.7395582Z Name: tests to run (est. time: 7.18min) 2024-10-08T22:12:15.7396726Z Serial tests (13): 2024-10-08T22:12:15.7397474Z distributed/_tensor/test_dtensor_compile 1/1 2024-10-08T22:12:15.7398193Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 2024-10-08T22:12:15.7398970Z distributed/tensor/parallel/test_tp_random_state 1/1 2024-10-08T22:12:15.7400085Z distributed/tensor/parallel/test_tp_style 1/1 2024-10-08T22:12:15.7401309Z distributed/_tensor/test_embedding_ops 1/1 2024-10-08T22:12:15.7402604Z distributed/_tensor/test_dtensor 1/1 2024-10-08T22:12:15.7403821Z distributed/_tensor/test_random_ops 1/1 2024-10-08T22:12:15.7404922Z distributed/_tensor/test_math_ops 1/1 2024-10-08T22:12:15.7405775Z distributed/pipelining/test_backward 1/1 2024-10-08T22:12:15.7406665Z distributed/pipelining/test_transformer 1/1 2024-10-08T22:12:15.7407270Z distributed/_tensor/test_view_ops 1/1 2024-10-08T22:12:15.7407889Z distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 2024-10-08T22:12:15.7408566Z distributed/checkpoint/test_save_load_api 1/1 2024-10-08T22:12:15.7409148Z Parallel tests (0): 2024-10-08T22:12:15.7409624Z Name: excluded (est. time: 83.71min) 2024-10-08T22:12:15.7410128Z Serial tests (57): 2024-10-08T22:12:15.7410569Z distributed/test_distributed_spawn 2/9 2024-10-08T22:12:15.7411157Z distributed/test_distributed_spawn 5/9 2024-10-08T22:12:15.7411736Z distributed/test_distributed_spawn 8/9 2024-10-08T22:12:15.7412294Z distributed/test_c10d_nccl 1/2 2024-10-08T22:12:15.7412966Z distributed/fsdp/test_fsdp_core 1/2 2024-10-08T22:12:15.7413593Z distributed/rpc/cuda/test_tensorpipe_agent 1/2 2024-10-08T22:12:15.7414246Z distributed/fsdp/test_fsdp_optim_state 1/1 2024-10-08T22:12:15.7414918Z distributed/optim/test_zero_redundancy_optimizer 1/1 2024-10-08T22:12:15.7415641Z distributed/fsdp/test_fsdp_freezing_weights 1/1 2024-10-08T22:12:15.7416293Z distributed/fsdp/test_fsdp_comm_hooks 1/1 2024-10-08T22:12:15.7416867Z distributed/fsdp/test_wrap 1/1 2024-10-08T22:12:15.7417390Z distributed/test_dynamo_distributed 1/1 2024-10-08T22:12:15.7418093Z distributed/_composable/test_replicate_with_compiler 1/1 2024-10-08T22:12:15.7418814Z distributed/fsdp/test_fsdp_checkpoint 1/1 2024-10-08T22:12:15.7419406Z distributed/fsdp/test_fsdp_misc 1/1 2024-10-08T22:12:15.7420078Z distributed/algorithms/quantization/test_quantization 1/1 2024-10-08T22:12:15.7420810Z distributed/fsdp/test_fsdp_flatten_params 1/1 2024-10-08T22:12:15.7421426Z distributed/test_c10d_spawn_nccl 1/1 2024-10-08T22:12:15.7421989Z distributed/algorithms/test_join 1/1 2024-10-08T22:12:15.7422522Z distributed/launcher/test_run 1/1 2024-10-08T22:12:15.7423095Z distributed/fsdp/test_fsdp_exec_order 1/1 2024-10-08T22:12:15.7423732Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 2024-10-08T22:12:15.7424466Z distributed/_composable/fully_shard/test_fully_shard_init 1/1 2024-10-08T22:12:15.7425273Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 2024-10-08T22:12:15.7426068Z distributed/checkpoint/test_file_system_checkpoint 1/1 2024-10-08T22:12:15.7426727Z distributed/test_c10d_spawn_ucc 1/1 2024-10-08T22:12:15.7427491Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 1/1 2024-10-08T22:12:15.7428241Z distributed/test_c10d_logger 1/1 2024-10-08T22:12:15.7428784Z distributed/fsdp/test_fsdp_apply 1/1 2024-10-08T22:12:15.7429413Z distributed/elastic/timer/local_timer_example 1/1 2024-10-08T22:12:15.7430159Z distributed/_shard/sharding_spec/test_sharding_spec 1/1 2024-10-08T22:12:15.7430878Z distributed/_shard/sharded_tensor/ops/test_init 1/1 2024-10-08T22:12:15.7431584Z distributed/checkpoint/test_fsdp_optim_state 1/1 2024-10-08T22:12:15.7432239Z distributed/checkpoint/test_fsspec 1/1 2024-10-08T22:12:15.7432796Z distributed/test_c10d_ops_nccl 1/1 2024-10-08T22:12:15.7433410Z distributed/checkpoint/test_fsdp_model_state 1/1 2024-10-08T22:12:15.7434203Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 1/1 2024-10-08T22:12:15.7434997Z distributed/fsdp/test_fsdp_backward_prefetch 1/1 2024-10-08T22:12:15.7435644Z distributed/elastic/test_control_plane 1/1 2024-10-08T22:12:15.7436349Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 2024-10-08T22:12:15.7437087Z distributed/fsdp/test_fsdp_multiple_wrapping 1/1 2024-10-08T22:12:15.7437800Z distributed/fsdp/test_fsdp_traversal 1/1 2024-10-08T22:12:15.7438521Z distributed/_composable/fsdp/test_fully_shard_grad_scaler 1/1 2024-10-08T22:12:15.7439264Z distributed/test_compute_comm_reordering 1/1 2024-10-08T22:12:15.7439902Z distributed/pipelining/test_unflatten 1/1 2024-10-08T22:12:15.7440536Z distributed/pipelining/test_microbatch 1/1 2024-10-08T22:12:15.7441238Z distributed/test_multi_threaded_pg 1/1 2024-10-08T22:12:15.7441827Z distributed/elastic/utils/util_test 1/1 2024-10-08T22:12:15.7442438Z distributed/_tools/test_memory_tracker 1/1 2024-10-08T22:12:15.7443074Z distributed/_composable/test_checkpoint 1/1 2024-10-08T22:12:15.7443648Z distributed/fsdp/test_fsdp_fx 1/1 2024-10-08T22:12:15.7444218Z distributed/nn/jit/test_instantiator 1/1 2024-10-08T22:12:15.7444909Z distributed/_composable/fsdp/test_fully_shard_state 1/1 2024-10-08T22:12:15.7445726Z distributed/_composable/fully_shard/test_fully_shard_compile 1/1 2024-10-08T22:12:15.7446459Z distributed/rpc/test_tensorpipe_agent 1/1 2024-10-08T22:12:15.7447152Z distributed/test_collective_utils 1/1 2024-10-08T22:12:15.7447730Z distributed/rpc/test_faulty_agent 1/1 2024-10-08T22:12:15.7448266Z Parallel tests (0): 2024-10-08T22:12:15.7477063Z Running distributed/_tensor/test_dtensor_compile 1/1 ... [2024-10-08 22:12:15.747355] 2024-10-08T22:12:15.7477919Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-10-08T22:12:15.7482398Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-10-08 22:12:15.747798] 2024-10-08T22:12:54.2493898Z 2024-10-08T22:12:54.2496224Z distributed/_tensor/test_dtensor_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_compile_1.1_4963609674cd5ad0_.log 2024-10-08T22:12:54.2517078Z Running 28 items in this shard: test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_device_mesh_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_attribute_access_on_intermediate, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_dynamo_disable, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_graph_break, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_contiguous_dtensor_noncontiguous_local_as_tangent, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_different_gradient_placement, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dont_recompile_on_same_placement_devicemesh, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamic, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamo_device_mesh_attrs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_noncontiguous_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_graph_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_redistribute_unbalanced_correct_strides, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_dynamic_shapes, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_recompile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs_forward_hook, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_fakify_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_graph_input_is_async, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_placement_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True 2024-10-08T22:12:54.2536219Z 2024-10-08T22:12:54.2537156Z Running distributed/tensor/parallel/test_micro_pipeline_tp 1/1 ... [2024-10-08 22:12:54.249451] 2024-10-08T22:12:54.2538085Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-10-08T22:12:54.2540431Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_micro_pipeline_tp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-10-08 22:12:54.249944] 2024-10-08T22:12:59.9265210Z 2024-10-08T22:12:59.9267360Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_297ec46988f18be7_.log 2024-10-08T22:12:59.9293289Z Running 29 items in this shard: test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_all_gather_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_reduce_scatter_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_get_unexposed_collectives 2024-10-08T22:12:59.9317847Z 2024-10-08T22:12:59.9318544Z Running distributed/tensor/parallel/test_tp_random_state 1/1 ... [2024-10-08 22:12:59.926830] 2024-10-08T22:12:59.9319452Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-10-08T22:12:59.9321750Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_random_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-10-08 22:12:59.927331] 2024-10-08T22:13:08.0582380Z 2024-10-08T22:13:08.0584532Z distributed/tensor/parallel/test_tp_random_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_random_state_1.1_eec74c4335c686fc_.log 2024-10-08T22:13:08.0587019Z Running 1 items in this shard: test/distributed/tensor/parallel/test_tp_random_state.py::TensorParallelRandomStateTests::test_model_init 2024-10-08T22:13:08.0588061Z 2024-10-08T22:13:08.0589258Z Running distributed/tensor/parallel/test_tp_style 1/1 ... [2024-10-08 22:13:08.058543] 2024-10-08T22:13:08.0590186Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-10-08T22:13:08.0595127Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_style.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-10-08 22:13:08.059046] 2024-10-08T22:14:00.4779900Z 2024-10-08T22:14:00.4784332Z distributed/tensor/parallel/test_tp_style 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_style_1.1_2df091510cb30e5c_.log 2024-10-08T22:14:00.4792208Z Running 9 items in this shard: test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_colwise_parallel_embedding, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_colwise_parallel_style, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_input, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_input_multiple_inputs, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_kwargs_input, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_output, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_rowwise_parallel_embedding, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_rowwise_parallel_style, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_sequence_parallel_style 2024-10-08T22:14:00.4799069Z 2024-10-08T22:14:00.4799642Z Running distributed/_tensor/test_embedding_ops 1/1 ... [2024-10-08 22:14:00.478271] 2024-10-08T22:14:00.4800461Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-10-08T22:14:00.4802667Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_embedding_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-10-08 22:14:00.478718] 2024-10-08T22:14:26.2436946Z 2024-10-08T22:14:26.2441690Z distributed/_tensor/test_embedding_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_embedding_ops_1.1_46ba640b65101b15_.log 2024-10-08T22:14:26.2445712Z Running 4 items in this shard: test/distributed/_tensor/test_embedding_ops.py::TestEmbeddingOp::test_multiple_embeddings_rowwise, test/distributed/_tensor/test_embedding_ops.py::TestEmbeddingOp::test_sharded_embedding_colwise, test/distributed/_tensor/test_embedding_ops.py::TestEmbeddingOp::test_sharded_embedding_colwise_max_norm_errors, test/distributed/_tensor/test_embedding_ops.py::TestEmbeddingOp::test_sharded_embedding_rowwise 2024-10-08T22:14:26.2448504Z 2024-10-08T22:14:26.2449072Z Running distributed/_tensor/test_dtensor 1/1 ... [2024-10-08 22:14:26.243964] 2024-10-08T22:14:26.2449860Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-10-08T22:14:26.2452020Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-10-08 22:14:26.244428] 2024-10-08T22:17:11.1590136Z 2024-10-08T22:17:11.1592228Z distributed/_tensor/test_dtensor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_1.1_7dbe0b012ae7438c_.log 2024-10-08T22:17:11.1610774Z Running 33 items in this shard: test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_async_output, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_constructor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_new_empty_strided, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_properties, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_save_load, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_hash, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_read_only_after_set, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_stride, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_negative_dim, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_then_to_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding_raise_error, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_grad_hint, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_sync, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_meta_dtensor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_modules_w_meta_dtensor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_to_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_to_local_grad_hint, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_auto_implicit_replication, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_default_value_sub_mesh, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_device_mesh_nd, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_2d_mesh, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_api_device_mesh_context_manager, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_device_mesh_device_conversion, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_spec_local_shard_offset, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_from_local_sub_mesh, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_implicit_replication, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_implicit_replication_for_foreach_ops, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_metadata_consistency_check, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_redistribute_sub_mesh, test/distributed/_tensor/test_dtensor.py::TestDTensorPlacementTypes::test_split_tensor_1D, test/distributed/_tensor/test_dtensor.py::DTensorLogTest::test_dtensor_log 2024-10-08T22:17:11.1628295Z 2024-10-08T22:17:11.1628897Z Running distributed/_tensor/test_random_ops 1/1 ... [2024-10-08 22:17:11.159344] 2024-10-08T22:17:11.1629721Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-10-08T22:17:11.1631887Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_random_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-10-08 22:17:11.159817] 2024-10-08T22:17:49.0495909Z 2024-10-08T22:17:49.0498091Z distributed/_tensor/test_random_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_random_ops_1.1_39a5da37a58adb35_.log 2024-10-08T22:17:49.0503596Z Running 7 items in this shard: test/distributed/_tensor/test_random_ops.py::DistTensorRandomInitTest::test_init_ops, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_dropout_1d, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_rand_1d, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_uniform_2d, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_manual_seed, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_meta_tensor_init, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_rng_tracker_init 2024-10-08T22:17:49.0507913Z 2024-10-08T22:17:49.0508474Z Running distributed/_tensor/test_math_ops 1/1 ... [2024-10-08 22:17:49.049827] 2024-10-08T22:17:49.0509247Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-10-08T22:17:49.0511423Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_math_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-10-08 22:17:49.050282] 2024-10-08T22:19:13.1322868Z 2024-10-08T22:19:13.1324914Z distributed/_tensor/test_math_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_math_ops_1.1_879561519c43e911_.log 2024-10-08T22:19:13.1336122Z Running 14 items in this shard: test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_foreach_norm, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_bwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_bwd_req_grad, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_fwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_linalg_eigh, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_linear_op_reductions, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_mean, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_nll_loss_and_cross_entropy, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_shard0_svd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_shard_math_ops, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_softmax_fwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_softmax_with_bwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_topk, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_upsampling 2024-10-08T22:19:13.1344070Z 2024-10-08T22:19:13.1344738Z Running distributed/pipelining/test_backward 1/1 ... [2024-10-08 22:19:13.132585] 2024-10-08T22:19:13.1345577Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-10-08T22:19:13.1347793Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_backward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-10-08 22:19:13.133075] 2024-10-08T22:19:17.6064668Z 2024-10-08T22:19:17.6067076Z distributed/pipelining/test_backward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_backward_1.1_723cec206fad25a5_.log 2024-10-08T22:19:17.6071128Z Running 4 items in this shard: test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward, test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward_input, test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward_weight, test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward_weight_multiple_iters 2024-10-08T22:19:17.6073942Z 2024-10-08T22:19:17.6074562Z Running distributed/pipelining/test_transformer 1/1 ... [2024-10-08 22:19:17.606782] 2024-10-08T22:19:17.6075450Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-10-08T22:19:17.6077690Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_transformer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-10-08 22:19:17.607292] 2024-10-08T22:19:22.7817981Z 2024-10-08T22:19:22.7820199Z distributed/pipelining/test_transformer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_transformer_1.1_6dd256c9c02c5576_.log 2024-10-08T22:19:22.7822303Z Running 1 items in this shard: test/distributed/pipelining/test_transformer.py::TransformerTests::test_ir 2024-10-08T22:19:22.7823134Z 2024-10-08T22:19:22.7824050Z Running distributed/_tensor/test_view_ops 1/1 ... [2024-10-08 22:19:22.782102] 2024-10-08T22:19:22.7824825Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-10-08T22:19:22.7830026Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_view_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-10-08 22:19:22.782573] 2024-10-08T22:19:58.5906638Z 2024-10-08T22:19:58.5908520Z distributed/_tensor/test_view_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_view_ops_1.1_aa445adfaa6fdcab_.log 2024-10-08T22:19:58.5912333Z Running 4 items in this shard: test/distributed/_tensor/test_view_ops.py::TestViewOps::test_complex_view_ops, test/distributed/_tensor/test_view_ops.py::TestViewOps::test_dtensor_view_op_uneven, test/distributed/_tensor/test_view_ops.py::TestViewOps::test_view_groups, test/distributed/_tensor/test_view_ops.py::TestViewOps::test_view_ops 2024-10-08T22:19:58.5914545Z 2024-10-08T22:19:58.5915269Z Running distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 ... [2024-10-08 22:19:58.591047] 2024-10-08T22:19:58.5916118Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-10-08T22:19:58.5918950Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_hsdp_dtensor_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-10-08 22:19:58.591500] 2024-10-08T22:20:34.8773702Z 2024-10-08T22:20:34.8775760Z distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_hsdp_dtensor_state_dict_1.1_c1b03a7faef2ce98_.log 2024-10-08T22:20:34.8785188Z Running 8 items in this shard: test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_False, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_True, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_False, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_True, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_False, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_True, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_hsdp_init_with_device_mesh, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensor::test_root_module_is_not_FSDP 2024-10-08T22:20:34.8793002Z 2024-10-08T22:20:34.8793628Z Running distributed/checkpoint/test_save_load_api 1/1 ... [2024-10-08 22:20:34.877725] 2024-10-08T22:20:34.8794496Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-10-08T22:20:34.8796799Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_save_load_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-10-08 22:20:34.878226] 2024-10-08T22:20:43.1089963Z 2024-10-08T22:20:43.1093188Z distributed/checkpoint/test_save_load_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_save_load_api_1.1_a5b57e5c68593cef_.log 2024-10-08T22:20:43.1095759Z Running 1 items in this shard: test/distributed/checkpoint/test_save_load_api.py::TestSaveAndLoadAPI::test_auto_detect 2024-10-08T22:20:43.1096712Z 2024-10-08T22:20:43.7832735Z Running test batch 'tests to run' cost 508.04 seconds 2024-10-08T22:20:44.2870375Z 2024-10-08T22:20:44.2870688Z real 8m34.039s 2024-10-08T22:20:44.2871080Z user 22m34.227s 2024-10-08T22:20:44.2871457Z sys 8m0.160s 2024-10-08T22:20:44.2873932Z + assert_git_not_dirty 2024-10-08T22:20:44.2874841Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-10-08T22:20:44.2875639Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *xla* ]] 2024-10-08T22:20:44.2878573Z ++ git status --porcelain 2024-10-08T22:20:44.2879507Z ++ grep -v '?? third_party' 2024-10-08T22:20:47.3203673Z ++ true 2024-10-08T22:20:47.3204823Z + git_status= 2024-10-08T22:20:47.3205458Z + [[ -n '' ]] 2024-10-08T22:20:47.3206097Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-10-08T22:20:47.3206692Z + [[ 2 == 1 ]] 2024-10-08T22:20:47.3207030Z + [[ 2 == 1 ]] 2024-10-08T22:20:47.3207418Z + cleanup_workspace 2024-10-08T22:20:47.3208449Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-10-08T22:20:47.3209836Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-10-08T22:20:47.3211011Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-10-08T22:20:47.3211822Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-10-08T22:20:47.3212870Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-10-08T22:20:47.3214031Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-10-08T22:20:47.3214933Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-10-08T22:20:47.9384684Z ##[group]Run cat test/**/*_toprint.log || true 2024-10-08T22:20:47.9385575Z cat test/**/*_toprint.log || true 2024-10-08T22:20:47.9396518Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:20:47.9397229Z env: 2024-10-08T22:20:47.9397564Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:20:47.9398142Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:20:47.9399061Z DOCKER_CONTAINER_ID: cd91a56406f3434120c824708028491b4d148dc6169c77fb24a60d78398a479f 2024-10-08T22:20:47.9399861Z ##[endgroup] 2024-10-08T22:20:47.9492310Z cat: 'test/**/*_toprint.log': No such file or directory 2024-10-08T22:20:47.9534229Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-10-08T22:20:47.9534874Z kill "$MONITOR_SCRIPT_PID" 2024-10-08T22:20:47.9541627Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:20:47.9542263Z env: 2024-10-08T22:20:47.9542613Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:20:47.9543192Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:20:47.9544109Z DOCKER_CONTAINER_ID: cd91a56406f3434120c824708028491b4d148dc6169c77fb24a60d78398a479f 2024-10-08T22:20:47.9544931Z MONITOR_SCRIPT_PID: 412590 2024-10-08T22:20:47.9545377Z ##[endgroup] 2024-10-08T22:20:47.9789747Z Prepare all required actions 2024-10-08T22:20:47.9790369Z Getting action download info 2024-10-08T22:20:48.2695278Z Download action repository 'actions/upload-artifact@v3' (SHA:ff15f0306b3f739f7b6fd43fb5d26cd321bd4de5) 2024-10-08T22:20:48.4697441Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-10-08T22:20:48.4698043Z with: 2024-10-08T22:20:48.4698652Z file-suffix: test-distributed-2-3-linux.8xlarge.nvidia.gpu_31259716486 2024-10-08T22:20:48.4699429Z s3-bucket: gha-artifacts 2024-10-08T22:20:48.4699834Z env: 2024-10-08T22:20:48.4700172Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:20:48.4700741Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:20:48.4701657Z DOCKER_CONTAINER_ID: cd91a56406f3434120c824708028491b4d148dc6169c77fb24a60d78398a479f 2024-10-08T22:20:48.4702456Z ##[endgroup] 2024-10-08T22:20:48.4742091Z ##[group]Run # Remove any previous test jsons if they exist 2024-10-08T22:20:48.4742961Z # Remove any previous test jsons if they exist 2024-10-08T22:20:48.4743610Z rm -f test-jsons-*.zip 2024-10-08T22:20:48.4744254Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-10-08T22:20:48.4751383Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:20:48.4752013Z env: 2024-10-08T22:20:48.4752357Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:20:48.4752922Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:20:48.4753844Z DOCKER_CONTAINER_ID: cd91a56406f3434120c824708028491b4d148dc6169c77fb24a60d78398a479f 2024-10-08T22:20:48.4754923Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_31259716486 2024-10-08T22:20:48.4755684Z ##[endgroup] 2024-10-08T22:20:48.5027606Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-10-08T22:20:48.5075810Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-10-08T22:20:48.5076693Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-10-08T22:20:48.5085795Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-10-08T22:20:48.5093677Z adding: test/slow_tests.json (deflated 82%) 2024-10-08T22:20:48.5096608Z adding: test/test-reports/td_exclusions-b664c8c994c0328d300e.json (deflated 86%) 2024-10-08T22:20:48.5116748Z adding: test/.pytorch-disabled-tests.json (deflated 88%) 2024-10-08T22:20:48.5154080Z ##[group]Run # Remove any previous test reports if they exist 2024-10-08T22:20:48.5177287Z # Remove any previous test reports if they exist 2024-10-08T22:20:48.5177985Z rm -f test-reports-*.zip 2024-10-08T22:20:48.5178724Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-10-08T22:20:48.5185889Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:20:48.5186522Z env: 2024-10-08T22:20:48.5186867Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:20:48.5187631Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:20:48.5188566Z DOCKER_CONTAINER_ID: cd91a56406f3434120c824708028491b4d148dc6169c77fb24a60d78398a479f 2024-10-08T22:20:48.5189634Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_31259716486 2024-10-08T22:20:48.5190376Z ##[endgroup] 2024-10-08T22:20:48.5414602Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor_compile/distributed._tensor.test_dtensor_compile-1e9ba20194124c26.xml (deflated 85%) 2024-10-08T22:20:48.5417114Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_micro_pipeline_tp/distributed.tensor.parallel.test_micro_pipeline_tp-a5402faced92be0a.xml (deflated 95%) 2024-10-08T22:20:48.5419717Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_random_state/distributed.tensor.parallel.test_tp_random_state-4fd834703964c22d.xml (deflated 51%) 2024-10-08T22:20:48.5422163Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_style/distributed.tensor.parallel.test_tp_style-fe9ca0b6fa8441b5.xml (deflated 78%) 2024-10-08T22:20:48.5424732Z adding: test/test-reports/python-pytest/distributed._tensor.test_embedding_ops/distributed._tensor.test_embedding_ops-c1dc068e9880843a.xml (deflated 64%) 2024-10-08T22:20:48.5427197Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor/distributed._tensor.test_dtensor-1c9e48fa51b1cde4.xml (deflated 82%) 2024-10-08T22:20:48.5429311Z adding: test/test-reports/python-pytest/distributed._tensor.test_random_ops/distributed._tensor.test_random_ops-465d56194514aa14.xml (deflated 73%) 2024-10-08T22:20:48.5431397Z adding: test/test-reports/python-pytest/distributed._tensor.test_math_ops/distributed._tensor.test_math_ops-28e125628482fed5.xml (deflated 79%) 2024-10-08T22:20:48.5433549Z adding: test/test-reports/python-pytest/distributed.pipelining.test_backward/distributed.pipelining.test_backward-e865f0a80ac43c23.xml (deflated 68%) 2024-10-08T22:20:48.5435833Z adding: test/test-reports/python-pytest/distributed.pipelining.test_transformer/distributed.pipelining.test_transformer-db2ad57734d67e40.xml (deflated 39%) 2024-10-08T22:20:48.5438021Z adding: test/test-reports/python-pytest/distributed._tensor.test_view_ops/distributed._tensor.test_view_ops-a77648f5532c3a27.xml (deflated 61%) 2024-10-08T22:20:48.5440255Z adding: test/test-reports/python-pytest/distributed.fsdp.test_hsdp_dtensor_state_dict/distributed.fsdp.test_hsdp_dtensor_state_dict-3900a9eac6210865.xml (deflated 86%) 2024-10-08T22:20:48.5442616Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_save_load_api/distributed.checkpoint.test_save_load_api-d94ffedf5f1f4545.xml (deflated 50%) 2024-10-08T22:20:48.5476325Z ##[group]Run # Remove any previous usage logs if they exist 2024-10-08T22:20:48.5477112Z # Remove any previous usage logs if they exist 2024-10-08T22:20:48.5477719Z rm -f logs-*.zip 2024-10-08T22:20:48.5478561Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-10-08T22:20:48.5479577Z # so check to see if the file exists first 2024-10-08T22:20:48.5480214Z if [ -f 'usage_log.txt' ]; then 2024-10-08T22:20:48.5480866Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-10-08T22:20:48.5481483Z fi 2024-10-08T22:20:48.5481932Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-10-08T22:20:48.5482658Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-10-08T22:20:48.5483262Z fi 2024-10-08T22:20:48.5489591Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:20:48.5490233Z env: 2024-10-08T22:20:48.5490574Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:20:48.5491132Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:20:48.5492053Z DOCKER_CONTAINER_ID: cd91a56406f3434120c824708028491b4d148dc6169c77fb24a60d78398a479f 2024-10-08T22:20:48.5493121Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_31259716486 2024-10-08T22:20:48.5494061Z ##[endgroup] 2024-10-08T22:20:48.5550435Z adding: usage_log.txt (deflated 92%) 2024-10-08T22:20:48.5794773Z adding: test/test-reports/distributed._tensor.test_dtensor_compile_1.1_4963609674cd5ad0_.log (deflated 84%) 2024-10-08T22:20:48.5796451Z adding: test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_297ec46988f18be7_.log (deflated 90%) 2024-10-08T22:20:48.5798241Z adding: test/test-reports/distributed.tensor.parallel.test_tp_random_state_1.1_eec74c4335c686fc_.log (deflated 52%) 2024-10-08T22:20:48.5799901Z adding: test/test-reports/distributed.tensor.parallel.test_tp_style_1.1_2df091510cb30e5c_.log (deflated 74%) 2024-10-08T22:20:48.5801463Z adding: test/test-reports/distributed._tensor.test_embedding_ops_1.1_46ba640b65101b15_.log (deflated 66%) 2024-10-08T22:20:48.5802946Z adding: test/test-reports/distributed._tensor.test_dtensor_1.1_7dbe0b012ae7438c_.log (deflated 90%) 2024-10-08T22:20:48.5804423Z adding: test/test-reports/distributed._tensor.test_random_ops_1.1_39a5da37a58adb35_.log (deflated 70%) 2024-10-08T22:20:48.5806190Z adding: test/test-reports/distributed._tensor.test_math_ops_1.1_879561519c43e911_.log (deflated 70%) 2024-10-08T22:20:48.5807741Z adding: test/test-reports/distributed.pipelining.test_backward_1.1_723cec206fad25a5_.log (deflated 66%) 2024-10-08T22:20:48.5809294Z adding: test/test-reports/distributed.pipelining.test_transformer_1.1_6dd256c9c02c5576_.log (deflated 54%) 2024-10-08T22:20:48.5810814Z adding: test/test-reports/distributed._tensor.test_view_ops_1.1_aa445adfaa6fdcab_.log (deflated 85%) 2024-10-08T22:20:48.5812364Z adding: test/test-reports/distributed.fsdp.test_hsdp_dtensor_state_dict_1.1_c1b03a7faef2ce98_.log (deflated 85%) 2024-10-08T22:20:48.5814231Z adding: test/test-reports/distributed.checkpoint.test_save_load_api_1.1_a5b57e5c68593cef_.log (deflated 50%) 2024-10-08T22:20:48.5846702Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-10-08T22:20:48.5847624Z # Remove any previous debugging artifacts if they exist 2024-10-08T22:20:48.5848308Z rm -f debug-*.zip 2024-10-08T22:20:48.5848788Z if [ -d 'test/debug' ]; then 2024-10-08T22:20:48.5849404Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-10-08T22:20:48.5850002Z fi 2024-10-08T22:20:48.5856409Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:20:48.5857040Z env: 2024-10-08T22:20:48.5857365Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:20:48.5857930Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:20:48.5858842Z DOCKER_CONTAINER_ID: cd91a56406f3434120c824708028491b4d148dc6169c77fb24a60d78398a479f 2024-10-08T22:20:48.5859912Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_31259716486 2024-10-08T22:20:48.5860647Z ##[endgroup] 2024-10-08T22:20:48.5968978Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-10-08T22:20:48.5969522Z with: 2024-10-08T22:20:48.5969882Z s3-bucket: gha-artifacts 2024-10-08T22:20:48.5970428Z s3-prefix: pytorch/pytorch/11242664097/1/artifact 2024-10-08T22:20:48.5971021Z retention-days: 14 2024-10-08T22:20:48.5971428Z if-no-files-found: warn 2024-10-08T22:20:48.5971883Z path: test-jsons-*.zip 2024-10-08T22:20:48.5972304Z name: artifact 2024-10-08T22:20:48.5972678Z region: us-east-1 2024-10-08T22:20:48.5973031Z env: 2024-10-08T22:20:48.5973367Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:20:48.5973935Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:20:48.5974828Z DOCKER_CONTAINER_ID: cd91a56406f3434120c824708028491b4d148dc6169c77fb24a60d78398a479f 2024-10-08T22:20:48.5975629Z ##[endgroup] 2024-10-08T22:20:49.0547201Z NOTE: s3-prefix specified, ignoring name parameter 2024-10-08T22:20:49.0548216Z With the provided path, there will be 1 file uploaded 2024-10-08T22:20:49.0549014Z Uploading to s3 prefix: pytorch/pytorch/11242664097/1/artifact 2024-10-08T22:20:49.0603140Z Starting upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_31259716486.zip 2024-10-08T22:20:49.1979078Z Finished upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_31259716486.zip 2024-10-08T22:20:49.2191217Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-10-08T22:20:49.2191772Z with: 2024-10-08T22:20:49.2192123Z s3-bucket: gha-artifacts 2024-10-08T22:20:49.2192671Z s3-prefix: pytorch/pytorch/11242664097/1/artifact 2024-10-08T22:20:49.2193268Z retention-days: 14 2024-10-08T22:20:49.2193670Z if-no-files-found: error 2024-10-08T22:20:49.2194128Z path: test-reports-*.zip 2024-10-08T22:20:49.2194571Z name: artifact 2024-10-08T22:20:49.2194938Z region: us-east-1 2024-10-08T22:20:49.2195313Z env: 2024-10-08T22:20:49.2195655Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:20:49.2196223Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:20:49.2197191Z DOCKER_CONTAINER_ID: cd91a56406f3434120c824708028491b4d148dc6169c77fb24a60d78398a479f 2024-10-08T22:20:49.2197999Z ##[endgroup] 2024-10-08T22:20:49.6742778Z NOTE: s3-prefix specified, ignoring name parameter 2024-10-08T22:20:49.6744028Z With the provided path, there will be 1 file uploaded 2024-10-08T22:20:49.6745235Z Uploading to s3 prefix: pytorch/pytorch/11242664097/1/artifact 2024-10-08T22:20:49.6798005Z Starting upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_31259716486.zip 2024-10-08T22:20:49.8181273Z Finished upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_31259716486.zip 2024-10-08T22:20:49.8392664Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-10-08T22:20:49.8393202Z with: 2024-10-08T22:20:49.8393551Z s3-bucket: gha-artifacts 2024-10-08T22:20:49.8394101Z s3-prefix: pytorch/pytorch/11242664097/1/artifact 2024-10-08T22:20:49.8394691Z retention-days: 14 2024-10-08T22:20:49.8395094Z if-no-files-found: ignore 2024-10-08T22:20:49.8395545Z path: logs-*.zip 2024-10-08T22:20:49.8395929Z name: artifact 2024-10-08T22:20:49.8396290Z region: us-east-1 2024-10-08T22:20:49.8396682Z env: 2024-10-08T22:20:49.8397024Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:20:49.8397674Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:20:49.8398592Z DOCKER_CONTAINER_ID: cd91a56406f3434120c824708028491b4d148dc6169c77fb24a60d78398a479f 2024-10-08T22:20:49.8399394Z ##[endgroup] 2024-10-08T22:20:50.2967916Z NOTE: s3-prefix specified, ignoring name parameter 2024-10-08T22:20:50.2968824Z With the provided path, there will be 1 file uploaded 2024-10-08T22:20:50.2969610Z Uploading to s3 prefix: pytorch/pytorch/11242664097/1/artifact 2024-10-08T22:20:50.3023082Z Starting upload of logs-test-distributed-2-3-linux.8xlarge.nvidia.gpu_31259716486.zip 2024-10-08T22:20:50.4416164Z Finished upload of logs-test-distributed-2-3-linux.8xlarge.nvidia.gpu_31259716486.zip 2024-10-08T22:20:50.4627332Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-10-08T22:20:50.4627878Z with: 2024-10-08T22:20:50.4628228Z s3-bucket: gha-artifacts 2024-10-08T22:20:50.4628775Z s3-prefix: pytorch/pytorch/11242664097/1/artifact 2024-10-08T22:20:50.4629383Z retention-days: 14 2024-10-08T22:20:50.4629791Z if-no-files-found: ignore 2024-10-08T22:20:50.4630243Z path: debug-*.zip 2024-10-08T22:20:50.4630646Z name: artifact 2024-10-08T22:20:50.4631002Z region: us-east-1 2024-10-08T22:20:50.4631375Z env: 2024-10-08T22:20:50.4631714Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:20:50.4632282Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:20:50.4633179Z DOCKER_CONTAINER_ID: cd91a56406f3434120c824708028491b4d148dc6169c77fb24a60d78398a479f 2024-10-08T22:20:50.4633980Z ##[endgroup] 2024-10-08T22:20:50.9097830Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-10-08T22:20:50.9307567Z ##[group]Run # shellcheck disable=SC2156 2024-10-08T22:20:50.9308209Z # shellcheck disable=SC2156 2024-10-08T22:20:50.9309218Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-10-08T22:20:50.9317173Z shell: /usr/bin/bash -e {0} 2024-10-08T22:20:50.9317624Z env: 2024-10-08T22:20:50.9317968Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:20:50.9318543Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:20:50.9319441Z DOCKER_CONTAINER_ID: cd91a56406f3434120c824708028491b4d148dc6169c77fb24a60d78398a479f 2024-10-08T22:20:50.9320232Z ##[endgroup] 2024-10-08T22:20:51.2080354Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-10-08T22:20:51.2081106Z with: 2024-10-08T22:20:51.2081422Z env: 2024-10-08T22:20:51.2081763Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:20:51.2082317Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:20:51.2083232Z DOCKER_CONTAINER_ID: cd91a56406f3434120c824708028491b4d148dc6169c77fb24a60d78398a479f 2024-10-08T22:20:51.2084029Z ##[endgroup] 2024-10-08T22:20:51.2111121Z ##[group]Run set -eou pipefail 2024-10-08T22:20:51.2111636Z set -eou pipefail 2024-10-08T22:20:51.2112076Z  2024-10-08T22:20:51.2112726Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-10-08T22:20:51.2113512Z for _ in $(seq 1440); do 2024-10-08T22:20:51.2114101Z  # Break if no ssh session exists anymore 2024-10-08T22:20:51.2114719Z  if [ "$(who)" = "" ]; then 2024-10-08T22:20:51.2115226Z  break 2024-10-08T22:20:51.2115654Z  fi 2024-10-08T22:20:51.2116016Z  echo "." 2024-10-08T22:20:51.2116420Z  sleep 5 2024-10-08T22:20:51.2116807Z done 2024-10-08T22:20:51.2124123Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:20:51.2124758Z env: 2024-10-08T22:20:51.2125102Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:20:51.2125665Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:20:51.2126568Z DOCKER_CONTAINER_ID: cd91a56406f3434120c824708028491b4d148dc6169c77fb24a60d78398a479f 2024-10-08T22:20:51.2127368Z ##[endgroup] 2024-10-08T22:20:51.2175738Z Holding runner for 2 hours until all ssh sessions have logged out 2024-10-08T22:20:51.2238266Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-10-08T22:20:51.2239236Z # ignore expansion of "docker ps -q" since it could be empty 2024-10-08T22:20:51.2239977Z # shellcheck disable=SC2046 2024-10-08T22:20:51.2240567Z docker stop $(docker ps -q) || true 2024-10-08T22:20:51.2241282Z # Prune all of the docker images 2024-10-08T22:20:51.2241833Z docker system prune -af 2024-10-08T22:20:51.2248582Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:20:51.2249207Z env: 2024-10-08T22:20:51.2249553Z GIT_DEFAULT_BRANCH: main 2024-10-08T22:20:51.2250102Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-10-08T22:20:51.2251014Z DOCKER_CONTAINER_ID: cd91a56406f3434120c824708028491b4d148dc6169c77fb24a60d78398a479f 2024-10-08T22:20:51.2251809Z ##[endgroup] 2024-10-08T22:20:51.9816038Z cd91a56406f3 2024-10-08T22:20:52.5089365Z Deleted Containers: 2024-10-08T22:20:52.5090302Z cd91a56406f3434120c824708028491b4d148dc6169c77fb24a60d78398a479f 2024-10-08T22:20:52.5090850Z 2024-10-08T22:20:58.7930280Z Deleted Images: 2024-10-08T22:20:58.7932041Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T22:20:58.7934632Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9@sha256:314907662aadd5875a7a36524f299773cbf917ac65385064018e080dc893f046 2024-10-08T22:20:58.7936425Z deleted: sha256:afe1462db4393617da580c2f21c43484a88ab03cdc72dd8fbaa63722368879a8 2024-10-08T22:20:58.7937514Z deleted: sha256:ef8bd83b9984ad2b04228ac09fbeb541ff4d9ba4102cfdb6395c7c61adf28a64 2024-10-08T22:20:58.7938589Z deleted: sha256:b26ae8dbef9541eb4aa069fb27492979f0d2514421a54dcc7570badd4a83af4a 2024-10-08T22:20:58.7939668Z deleted: sha256:1e76e77e99b942452170ec6a0262afdcabfa39bce14afa5674720f6f1c2c6985 2024-10-08T22:20:58.7941149Z deleted: sha256:5a6d02cf3cff3bc78e175a43c345fc0985c3324af328a7e55d9c8b68f0133592 2024-10-08T22:20:58.7942224Z deleted: sha256:0d7360d338fb982261464db035ec9b26aeeb37353ce0d5535dd58858ca8370d7 2024-10-08T22:20:58.7943290Z deleted: sha256:606dd0ca912fc40bb86b886cbf10c1161f5cd2a0dc945da0e42f0c8925a31cc7 2024-10-08T22:20:58.7945240Z deleted: sha256:bb81a2b33a03eb097d97279a3c7575a25b3e7d9ce8a1597825bbbd63c6031852 2024-10-08T22:20:58.7946337Z deleted: sha256:97e58782751868403737230032d1205c7abf65511fd6bfbd365cd87f44d7cbe3 2024-10-08T22:20:58.7947373Z deleted: sha256:6183b8b4678b754efced7885af6eb97c67b2fe8072bba00981ab1586c117879c 2024-10-08T22:20:58.7948435Z deleted: sha256:8cdd60f0a2f7e5b9d0ab9260122302a85066720ae6ddd8f94a0fa7f7805374b2 2024-10-08T22:20:58.7949486Z deleted: sha256:0930989bc95c1921ece76acf88487cc561d3c75b2a508fa732a944d81730dc5e 2024-10-08T22:20:58.7950541Z deleted: sha256:3959cc5f3c756ce3a388f83738b8a1a30610a0e7f5cfc661056196a62f9f05f5 2024-10-08T22:20:58.7951621Z deleted: sha256:dc4ae9b7f27e2c6c1558de5c5048c8fe80f36ce586c79e1078bf8850bd6e200d 2024-10-08T22:20:58.7952684Z deleted: sha256:c1000dad002d0d0c7d29794cd689bc50f95522a9429d9cd305698c8f13c48fe2 2024-10-08T22:20:58.7953732Z deleted: sha256:f43da7b9764ea2a583d1c63f7e743eef0fe654680c399487fae79edba6dd1845 2024-10-08T22:20:58.7954812Z deleted: sha256:3b12b8d2f5199d902dbca6ce70a19d526be95afac00011a3900b7e2425681f6c 2024-10-08T22:20:58.7955879Z deleted: sha256:6366e33e61695fdf02781e37776b854f156b40bec5f9e9d6a3e3b7ab8183ce60 2024-10-08T22:20:58.7956927Z deleted: sha256:fd8ef0494016467260b4ecca3adc551d58f31f23c88b299058b3bdb0732c037e 2024-10-08T22:20:58.7957950Z deleted: sha256:1455f08cb641e12f8a42d7f6446011a30a483513b0674634ee8f5921f400bd66 2024-10-08T22:20:58.7959020Z deleted: sha256:bdf8a4d21f4d5c651edd5f83aa53cd6bae256aac421b3ae4e1b2e6bbdc871b60 2024-10-08T22:20:58.7960100Z deleted: sha256:62afbcd31586549501bc56551bfea29e9bf1e93e181d161b3c4556d4dced7b6f 2024-10-08T22:20:58.7961166Z deleted: sha256:9262dd2d24a74723cb605404df62b6a3d5feaaa1c21a4e03e34b1e9d785455f8 2024-10-08T22:20:58.7962585Z deleted: sha256:88358e5713df7c4a40845dcf53cc90fcd7acff92c169f32d25defd5bd7c2ac95 2024-10-08T22:20:58.7963648Z deleted: sha256:4e7985a4e473afc0acb58028271e3d41623c7fedc227649738edd1b2f7d7deed 2024-10-08T22:20:58.7964695Z deleted: sha256:942381edef753b62e56ff26485f6f9588c5a135989d3f72828f19d6410e9a131 2024-10-08T22:20:58.7965761Z deleted: sha256:fd42ab1b7036dae71a2b1928fd7a322dcd73b59c350fd8c5600381d192aa411c 2024-10-08T22:20:58.7966832Z deleted: sha256:9a927087978af9df016ee6a7fba978e9aa17bf6298c6128da049aec0f4f44736 2024-10-08T22:20:58.7967883Z deleted: sha256:db1be15145ef9b6f4ab6f1e3c777947bb82bd6b88b96fda0b37a9d0455bc2fe3 2024-10-08T22:20:58.7968972Z deleted: sha256:4f4b9dc92afc77ccedb195a240b4a67f85835a0cf7a881a1144ec261ca32b38d 2024-10-08T22:20:58.7970047Z deleted: sha256:e83608da31fda60632cd2ba4e6dc63020b77af51416e42bee0824cd0d120a2db 2024-10-08T22:20:58.7971100Z deleted: sha256:e668552d30a457c79e20fe93a9e0687bc6d7412513258df5a5a79b76a70eb0c4 2024-10-08T22:20:58.7972165Z deleted: sha256:36b2fd6fb351223266236bfea21be064882aafd13dde8b2e378b15351d58ac50 2024-10-08T22:20:58.7973212Z deleted: sha256:3f59af073fce5e4fa9725354498f18d1818862a93ddd54e9323847d642355778 2024-10-08T22:20:58.7974261Z deleted: sha256:e2dc274e8ad7421e9b8384d7f369070716fe0ec8ccc4e72707bb174306c95a0c 2024-10-08T22:20:58.7975298Z deleted: sha256:a43bf89384077eae1f959820620828bd54264060f3255e34474477e771bc3e25 2024-10-08T22:20:58.7976311Z deleted: sha256:25176ae87768a658a3fac4d249129bb6486fda8a28c6dd05929e3d9f303547ea 2024-10-08T22:20:58.7977356Z deleted: sha256:2ee79aeb74a41713224e720e8ae159904225965ebf0a75635f27959fce0c3e89 2024-10-08T22:20:58.7978405Z deleted: sha256:fcf0c096f92469dee127ed87fce4619913919f190af836905da0d1d17187faef 2024-10-08T22:20:58.7979460Z deleted: sha256:1da10ad2ef76946acfc8029a03c84f92491d301ef01bbfdd4073691c852d4193 2024-10-08T22:20:58.7980499Z deleted: sha256:e80968c83a310db2236c6e8b13f3f5473b562d193b5b1506c7c0da13ad42b629 2024-10-08T22:20:58.7981719Z deleted: sha256:692bed0503b2a91f7b30b84d3dd331edb1208113c33df88b5629473b3dbf3238 2024-10-08T22:20:58.7982772Z deleted: sha256:30e9ee11b808505dbaab348d8ab63351c37d8a92a9b500c228253d5d29d94557 2024-10-08T22:20:58.7983817Z deleted: sha256:5f0b22975397213720ca20e32f2c786a6d918dede5eed926856ed8057406a9af 2024-10-08T22:20:58.7985024Z deleted: sha256:c2408b30cb74af3382a6fd70bce479e741483c00d3f58adcc67b505c3624d2f8 2024-10-08T22:20:58.7986100Z deleted: sha256:45d34f3b58dc7cfaccb50edd665c4ec7b6f10dc3136605800472a86504c563e9 2024-10-08T22:20:58.7987167Z deleted: sha256:05eb538878a32f76f4a80e69a5b0ba77b52b5ecf52df6e395db6778a4eb3d6be 2024-10-08T22:20:58.7988195Z deleted: sha256:63384492f701263d28c3781772f3a6aef489820e81a1350ce5b143a7222d3b4c 2024-10-08T22:20:58.7989238Z deleted: sha256:7000ca0027cbf1e4cb978b98aeb9b3a76bb379c03befe9b11ba5cb05b6ff1bbd 2024-10-08T22:20:58.7990291Z deleted: sha256:c18f5705574fbb54953a7a8ef39f073b2258b07bfffad18ec8bc156e94c9fc2f 2024-10-08T22:20:58.7991352Z deleted: sha256:edc9a596c49895b5d2311a7f0a9f1bd3fe1b9723131e5a75d343dfc2ec1cdbb7 2024-10-08T22:20:58.7992432Z deleted: sha256:aeba52c7c7635fd170fd7793c8c536860be5236dfaf5730d1601a639317e9c64 2024-10-08T22:20:58.7993490Z deleted: sha256:5cc64064de9051a68954cf7ef587c49c7bcc51ea4100f411f29d5b58d2fe75ea 2024-10-08T22:20:58.7994527Z deleted: sha256:e30fe32fd81558f404564519f977f8d3d783f617a6a6f3cb59954da237a5476f 2024-10-08T22:20:58.7995573Z deleted: sha256:109540d4af65b685f9bec0eb0d125c0bdf62a12400dd6427dab678263d749e72 2024-10-08T22:20:58.7996629Z deleted: sha256:f9d5e1d43d411cde8419c4e410d200588fd6eea970c375ad565cc7a704ccc631 2024-10-08T22:20:58.7997666Z deleted: sha256:e4ef213e9676c78f69c9c14087b370f99b151e9448133cf52224c2000f43c2df 2024-10-08T22:20:58.7998758Z deleted: sha256:f41fff2cd8c22c8fa5c709a3aecb9b55d6d1d6a56a97f27188211be0093aab26 2024-10-08T22:20:58.7999832Z deleted: sha256:68b84dec4ef794fb2f5c641e0df685fa8798bff487cc6722933fa3b5e5a58332 2024-10-08T22:20:58.8000879Z deleted: sha256:2bc01ff482136dfe9e8048b68a3f17b21426007795489331e73958a5f5ec2ab5 2024-10-08T22:20:58.8001931Z deleted: sha256:b32bcc84761193aa0f697b0f5051bb1d826dc7e4fe22384f434d7b5046e767aa 2024-10-08T22:20:58.8002965Z deleted: sha256:494a91e184e2b5d2a45d3b77d078bd0374924cab07a9c4f86f810ab144091355 2024-10-08T22:20:58.8004016Z deleted: sha256:e1d22aeca2dc5393de33338c232333858239e9ef9829bdb17fd157a1095dd113 2024-10-08T22:20:58.8005080Z deleted: sha256:c1cf67c156a427068488c83e6c7ca52c56db14d56aa223f7648eaac3fc91b95a 2024-10-08T22:20:58.8006132Z deleted: sha256:4907acb82438f79d084ee104f0ff46ee696dc53a789644bb40185390bab51ed2 2024-10-08T22:20:58.8007173Z deleted: sha256:9036a8f35f206e874b23b54d1cd8b67cf24f15066a7eff0d0e7b79b2c0b91da1 2024-10-08T22:20:58.8008196Z deleted: sha256:1f910448cfaf72ac16f01c3c81493411c44d2d8a399e3f768947789bab7b8f2f 2024-10-08T22:20:58.8009230Z deleted: sha256:71841876f8d0bea41701fae65627059a31e0a49314d1115225eb3a82ca935d8b 2024-10-08T22:20:58.8010276Z deleted: sha256:2b772d1953b8df339c75cf717e1a3a55ca34e10d1745afd22a207c741d06cc92 2024-10-08T22:20:58.8011335Z deleted: sha256:e83e86e1d89cea333e7034c12a74889caf9594b6990dce4f6171bed159609513 2024-10-08T22:20:58.8012363Z deleted: sha256:562c46ce6729e7ea17c6173e136b9ce715321c22a735d35cc48070ea6df51e2d 2024-10-08T22:20:58.8013401Z deleted: sha256:89edf8c906678c217b9a88160e1561991f53909c3fd27a4a5e1f0ca0f7ff7888 2024-10-08T22:20:58.8014472Z deleted: sha256:d5ba67f8f9b0c2b6da06d2d5d9d5fedc9f17cbe1a5e13a1a4033a4619f4528ba 2024-10-08T22:20:58.8015536Z deleted: sha256:5d61311e1a293c0c9642e34871719525dfeec7028e4b45e08344a469e3fe24bf 2024-10-08T22:20:58.8016567Z deleted: sha256:b89bb70f745a94c0edd5e974aee294e160735f76fd666a1a06a0b9b1be3caf63 2024-10-08T22:20:58.8017631Z deleted: sha256:9e5477f1f21a38c3dfe21a56a7d77d1b1a42453aed3e11729afcbb9d5731770b 2024-10-08T22:20:58.8018714Z deleted: sha256:b76f4af91cffe2bb7b3f09960f5bdbb26f741419defda65c451c4407fc5b0a92 2024-10-08T22:20:58.8019784Z deleted: sha256:3f8748a00ebd394c0e9d9d481972d764dff58896841ef0150b3f7efaac0f4a4a 2024-10-08T22:20:58.8020808Z deleted: sha256:13989c771e3ba4d40f618344e55e5cb7062a35041b5efd69e2f740ba819f46ee 2024-10-08T22:20:58.8021977Z deleted: sha256:9ebb27837f6c67ad8c17e9af4e63c008faae03fa226c3641dd8bbbfeefb106c4 2024-10-08T22:20:58.8023033Z deleted: sha256:8c16907d735950f8093582bc3cf392b70e002456bf9c4224de5a5b86b78b3ea2 2024-10-08T22:20:58.8024165Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-10-08T22:20:58.8024813Z 2024-10-08T22:20:58.8025018Z Total reclaimed space: 34.47GB 2024-10-08T22:20:58.8123340Z Post job cleanup. 2024-10-08T22:20:58.8191727Z Post job cleanup. 2024-10-08T22:20:58.9417069Z [command]/usr/bin/git version 2024-10-08T22:20:58.9477801Z git version 2.40.1 2024-10-08T22:20:58.9542391Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/971cc874-af6e-4934-a7c8-14eb5d4138e2' before making global git config changes 2024-10-08T22:20:58.9543978Z Adding repository directory to the temporary git global config as a safe directory 2024-10-08T22:20:58.9550318Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-10-08T22:20:58.9598720Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-10-08T22:20:58.9639815Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-10-08T22:20:59.0005465Z Entering 'android/libs/fbjni' 2024-10-08T22:20:59.0071887Z Entering 'third_party/FP16' 2024-10-08T22:20:59.0136693Z Entering 'third_party/FXdiv' 2024-10-08T22:20:59.0202411Z Entering 'third_party/NNPACK' 2024-10-08T22:20:59.0268166Z Entering 'third_party/NVTX' 2024-10-08T22:20:59.0333215Z Entering 'third_party/VulkanMemoryAllocator' 2024-10-08T22:20:59.0399595Z Entering 'third_party/XNNPACK' 2024-10-08T22:20:59.0484181Z Entering 'third_party/benchmark' 2024-10-08T22:20:59.0547752Z Entering 'third_party/cpp-httplib' 2024-10-08T22:20:59.0612615Z Entering 'third_party/cpuinfo' 2024-10-08T22:20:59.0678477Z Entering 'third_party/cudnn_frontend' 2024-10-08T22:20:59.0743600Z Entering 'third_party/cutlass' 2024-10-08T22:20:59.0820008Z Entering 'third_party/eigen' 2024-10-08T22:20:59.0888900Z Entering 'third_party/fbgemm' 2024-10-08T22:20:59.0951851Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-10-08T22:20:59.1016529Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-10-08T22:20:59.1080600Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-10-08T22:20:59.1149863Z Entering 'third_party/fbgemm/third_party/googletest' 2024-10-08T22:20:59.1213793Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-10-08T22:20:59.1282528Z Entering 'third_party/flatbuffers' 2024-10-08T22:20:59.1349923Z Entering 'third_party/fmt' 2024-10-08T22:20:59.1415638Z Entering 'third_party/gemmlowp/gemmlowp' 2024-10-08T22:20:59.1479531Z Entering 'third_party/gloo' 2024-10-08T22:20:59.1543933Z Entering 'third_party/googletest' 2024-10-08T22:20:59.1609058Z Entering 'third_party/ideep' 2024-10-08T22:20:59.1672226Z Entering 'third_party/ideep/mkl-dnn' 2024-10-08T22:20:59.1743328Z Entering 'third_party/ittapi' 2024-10-08T22:20:59.1807775Z Entering 'third_party/kineto' 2024-10-08T22:20:59.1871778Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-10-08T22:20:59.1934774Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-10-08T22:20:59.2002560Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-10-08T22:20:59.2070345Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-10-08T22:20:59.2133545Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-10-08T22:20:59.2195948Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-10-08T22:20:59.2261321Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-10-08T22:20:59.2326298Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-10-08T22:20:59.2392392Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-10-08T22:20:59.2457374Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-10-08T22:20:59.2524172Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-10-08T22:20:59.2591950Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-10-08T22:20:59.2656042Z Entering 'third_party/mimalloc' 2024-10-08T22:20:59.2721249Z Entering 'third_party/nccl/nccl' 2024-10-08T22:20:59.2786055Z Entering 'third_party/nlohmann' 2024-10-08T22:20:59.2850527Z Entering 'third_party/onnx' 2024-10-08T22:20:59.2935052Z Entering 'third_party/onnx/third_party/benchmark' 2024-10-08T22:20:59.2999820Z Entering 'third_party/onnx/third_party/pybind11' 2024-10-08T22:20:59.3066637Z Entering 'third_party/opentelemetry-cpp' 2024-10-08T22:20:59.3133573Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-10-08T22:20:59.3197329Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-10-08T22:20:59.3260422Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-10-08T22:20:59.3323770Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-10-08T22:20:59.3389371Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-10-08T22:20:59.3451842Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-10-08T22:20:59.3515415Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-10-08T22:20:59.3578274Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-10-08T22:20:59.3643311Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-10-08T22:20:59.3712915Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-10-08T22:20:59.3800977Z Entering 'third_party/pocketfft' 2024-10-08T22:20:59.3865402Z Entering 'third_party/protobuf' 2024-10-08T22:20:59.3933493Z Entering 'third_party/protobuf/third_party/benchmark' 2024-10-08T22:20:59.3998983Z Entering 'third_party/protobuf/third_party/googletest' 2024-10-08T22:20:59.4065158Z Entering 'third_party/psimd' 2024-10-08T22:20:59.4129690Z Entering 'third_party/pthreadpool' 2024-10-08T22:20:59.4194816Z Entering 'third_party/pybind11' 2024-10-08T22:20:59.4258816Z Entering 'third_party/python-peachpy' 2024-10-08T22:20:59.4324768Z Entering 'third_party/sleef' 2024-10-08T22:20:59.4390393Z Entering 'third_party/tensorpipe' 2024-10-08T22:20:59.4453405Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-10-08T22:20:59.4517405Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-10-08T22:20:59.4580935Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-10-08T22:20:59.4644081Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-10-08T22:20:59.4706897Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-10-08T22:20:59.4795070Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-10-08T22:20:59.4828675Z http.https://github.com/.extraheader 2024-10-08T22:20:59.4840684Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-10-08T22:20:59.4884015Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-10-08T22:20:59.5237924Z Entering 'android/libs/fbjni' 2024-10-08T22:20:59.5281939Z http.https://github.com/.extraheader 2024-10-08T22:20:59.5320751Z Entering 'third_party/FP16' 2024-10-08T22:20:59.5364929Z http.https://github.com/.extraheader 2024-10-08T22:20:59.5403588Z Entering 'third_party/FXdiv' 2024-10-08T22:20:59.5446802Z http.https://github.com/.extraheader 2024-10-08T22:20:59.5486746Z Entering 'third_party/NNPACK' 2024-10-08T22:20:59.5529997Z http.https://github.com/.extraheader 2024-10-08T22:20:59.5569878Z Entering 'third_party/NVTX' 2024-10-08T22:20:59.5615021Z http.https://github.com/.extraheader 2024-10-08T22:20:59.5654571Z Entering 'third_party/VulkanMemoryAllocator' 2024-10-08T22:20:59.5700170Z http.https://github.com/.extraheader 2024-10-08T22:20:59.5740264Z Entering 'third_party/XNNPACK' 2024-10-08T22:20:59.5784063Z http.https://github.com/.extraheader 2024-10-08T22:20:59.5843707Z Entering 'third_party/benchmark' 2024-10-08T22:20:59.5887223Z http.https://github.com/.extraheader 2024-10-08T22:20:59.5926012Z Entering 'third_party/cpp-httplib' 2024-10-08T22:20:59.5970977Z http.https://github.com/.extraheader 2024-10-08T22:20:59.6009722Z Entering 'third_party/cpuinfo' 2024-10-08T22:20:59.6052792Z http.https://github.com/.extraheader 2024-10-08T22:20:59.6093319Z Entering 'third_party/cudnn_frontend' 2024-10-08T22:20:59.6136861Z http.https://github.com/.extraheader 2024-10-08T22:20:59.6177334Z Entering 'third_party/cutlass' 2024-10-08T22:20:59.6220914Z http.https://github.com/.extraheader 2024-10-08T22:20:59.6270168Z Entering 'third_party/eigen' 2024-10-08T22:20:59.6314554Z http.https://github.com/.extraheader 2024-10-08T22:20:59.6356141Z Entering 'third_party/fbgemm' 2024-10-08T22:20:59.6400642Z http.https://github.com/.extraheader 2024-10-08T22:20:59.6440484Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-10-08T22:20:59.6483859Z http.https://github.com/.extraheader 2024-10-08T22:20:59.6522111Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-10-08T22:20:59.6566083Z http.https://github.com/.extraheader 2024-10-08T22:20:59.6605222Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-10-08T22:20:59.6648098Z http.https://github.com/.extraheader 2024-10-08T22:20:59.6695075Z Entering 'third_party/fbgemm/third_party/googletest' 2024-10-08T22:20:59.6737648Z http.https://github.com/.extraheader 2024-10-08T22:20:59.6776884Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-10-08T22:20:59.6819474Z http.https://github.com/.extraheader 2024-10-08T22:20:59.6859307Z Entering 'third_party/flatbuffers' 2024-10-08T22:20:59.6903955Z http.https://github.com/.extraheader 2024-10-08T22:20:59.6945459Z Entering 'third_party/fmt' 2024-10-08T22:20:59.6990651Z http.https://github.com/.extraheader 2024-10-08T22:20:59.7029450Z Entering 'third_party/gemmlowp/gemmlowp' 2024-10-08T22:20:59.7074175Z http.https://github.com/.extraheader 2024-10-08T22:20:59.7113749Z Entering 'third_party/gloo' 2024-10-08T22:20:59.7156633Z http.https://github.com/.extraheader 2024-10-08T22:20:59.7196634Z Entering 'third_party/googletest' 2024-10-08T22:20:59.7239269Z http.https://github.com/.extraheader 2024-10-08T22:20:59.7279597Z Entering 'third_party/ideep' 2024-10-08T22:20:59.7322398Z http.https://github.com/.extraheader 2024-10-08T22:20:59.7360577Z Entering 'third_party/ideep/mkl-dnn' 2024-10-08T22:20:59.7403610Z http.https://github.com/.extraheader 2024-10-08T22:20:59.7451061Z Entering 'third_party/ittapi' 2024-10-08T22:20:59.7495785Z http.https://github.com/.extraheader 2024-10-08T22:20:59.7535094Z Entering 'third_party/kineto' 2024-10-08T22:20:59.7581296Z http.https://github.com/.extraheader 2024-10-08T22:20:59.7620898Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-10-08T22:20:59.7664875Z http.https://github.com/.extraheader 2024-10-08T22:20:59.7704441Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-10-08T22:20:59.7747104Z http.https://github.com/.extraheader 2024-10-08T22:20:59.7789063Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-10-08T22:20:59.7831528Z http.https://github.com/.extraheader 2024-10-08T22:20:59.7872329Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-10-08T22:20:59.7915527Z http.https://github.com/.extraheader 2024-10-08T22:20:59.7955019Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-10-08T22:20:59.7999223Z http.https://github.com/.extraheader 2024-10-08T22:20:59.8038461Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-10-08T22:20:59.8082014Z http.https://github.com/.extraheader 2024-10-08T22:20:59.8124459Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-10-08T22:20:59.8167495Z http.https://github.com/.extraheader 2024-10-08T22:20:59.8206817Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-10-08T22:20:59.8250445Z http.https://github.com/.extraheader 2024-10-08T22:20:59.8290912Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-10-08T22:20:59.8333842Z http.https://github.com/.extraheader 2024-10-08T22:20:59.8375535Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-10-08T22:20:59.8417716Z http.https://github.com/.extraheader 2024-10-08T22:20:59.8458354Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-10-08T22:20:59.8502682Z http.https://github.com/.extraheader 2024-10-08T22:20:59.8541087Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-10-08T22:20:59.8584920Z http.https://github.com/.extraheader 2024-10-08T22:20:59.8625327Z Entering 'third_party/mimalloc' 2024-10-08T22:20:59.8672485Z http.https://github.com/.extraheader 2024-10-08T22:20:59.8711746Z Entering 'third_party/nccl/nccl' 2024-10-08T22:20:59.8754643Z http.https://github.com/.extraheader 2024-10-08T22:20:59.8795154Z Entering 'third_party/nlohmann' 2024-10-08T22:20:59.8838257Z http.https://github.com/.extraheader 2024-10-08T22:20:59.8880381Z Entering 'third_party/onnx' 2024-10-08T22:20:59.8924358Z http.https://github.com/.extraheader 2024-10-08T22:20:59.8982851Z Entering 'third_party/onnx/third_party/benchmark' 2024-10-08T22:20:59.9026503Z http.https://github.com/.extraheader 2024-10-08T22:20:59.9066525Z Entering 'third_party/onnx/third_party/pybind11' 2024-10-08T22:20:59.9110384Z http.https://github.com/.extraheader 2024-10-08T22:20:59.9153330Z Entering 'third_party/opentelemetry-cpp' 2024-10-08T22:20:59.9198988Z http.https://github.com/.extraheader 2024-10-08T22:20:59.9241300Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-10-08T22:20:59.9285181Z http.https://github.com/.extraheader 2024-10-08T22:20:59.9325913Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-10-08T22:20:59.9369739Z http.https://github.com/.extraheader 2024-10-08T22:20:59.9409382Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-10-08T22:20:59.9452008Z http.https://github.com/.extraheader 2024-10-08T22:20:59.9492124Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-10-08T22:20:59.9536034Z http.https://github.com/.extraheader 2024-10-08T22:20:59.9578582Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-10-08T22:20:59.9622311Z http.https://github.com/.extraheader 2024-10-08T22:20:59.9661077Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-10-08T22:20:59.9705355Z http.https://github.com/.extraheader 2024-10-08T22:20:59.9744379Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-10-08T22:20:59.9788224Z http.https://github.com/.extraheader 2024-10-08T22:20:59.9827008Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-10-08T22:20:59.9870945Z http.https://github.com/.extraheader 2024-10-08T22:20:59.9912755Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-10-08T22:20:59.9956346Z http.https://github.com/.extraheader 2024-10-08T22:21:00.0001202Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-10-08T22:21:00.0044708Z http.https://github.com/.extraheader 2024-10-08T22:21:00.0111476Z Entering 'third_party/pocketfft' 2024-10-08T22:21:00.0155868Z http.https://github.com/.extraheader 2024-10-08T22:21:00.0197006Z Entering 'third_party/protobuf' 2024-10-08T22:21:00.0240330Z http.https://github.com/.extraheader 2024-10-08T22:21:00.0284405Z Entering 'third_party/protobuf/third_party/benchmark' 2024-10-08T22:21:00.0327427Z http.https://github.com/.extraheader 2024-10-08T22:21:00.0367545Z Entering 'third_party/protobuf/third_party/googletest' 2024-10-08T22:21:00.0410072Z http.https://github.com/.extraheader 2024-10-08T22:21:00.0451150Z Entering 'third_party/psimd' 2024-10-08T22:21:00.0494872Z http.https://github.com/.extraheader 2024-10-08T22:21:00.0533243Z Entering 'third_party/pthreadpool' 2024-10-08T22:21:00.0577094Z http.https://github.com/.extraheader 2024-10-08T22:21:00.0615971Z Entering 'third_party/pybind11' 2024-10-08T22:21:00.0659266Z http.https://github.com/.extraheader 2024-10-08T22:21:00.0698639Z Entering 'third_party/python-peachpy' 2024-10-08T22:21:00.0742204Z http.https://github.com/.extraheader 2024-10-08T22:21:00.0781557Z Entering 'third_party/sleef' 2024-10-08T22:21:00.0824526Z http.https://github.com/.extraheader 2024-10-08T22:21:00.0863827Z Entering 'third_party/tensorpipe' 2024-10-08T22:21:00.0907215Z http.https://github.com/.extraheader 2024-10-08T22:21:00.0946517Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-10-08T22:21:00.0991058Z http.https://github.com/.extraheader 2024-10-08T22:21:00.1029852Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-10-08T22:21:00.1072912Z http.https://github.com/.extraheader 2024-10-08T22:21:00.1111562Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-10-08T22:21:00.1154376Z http.https://github.com/.extraheader 2024-10-08T22:21:00.1194008Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-10-08T22:21:00.1237085Z http.https://github.com/.extraheader 2024-10-08T22:21:00.1277023Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-10-08T22:21:00.1320149Z http.https://github.com/.extraheader 2024-10-08T22:21:00.1477164Z A job completed hook has been configured by the self-hosted runner administrator 2024-10-08T22:21:00.1502050Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-10-08T22:21:00.1508469Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T22:21:00.1509097Z ##[endgroup] 2024-10-08T22:21:09.5058795Z Cleaning up orphan processes