2024-08-07T17:43:08.1510454Z Current runner version: '2.318.0' 2024-08-07T17:43:08.1520581Z Runner name: 'i-0dcb26b5a502300df' 2024-08-07T17:43:08.1522004Z Runner group name: 'Default' 2024-08-07T17:43:08.1523512Z Machine name: 'ip-10-0-1-17' 2024-08-07T17:43:08.1530175Z ##[group]GITHUB_TOKEN Permissions 2024-08-07T17:43:08.1533651Z Actions: read 2024-08-07T17:43:08.1535139Z Attestations: read 2024-08-07T17:43:08.1536197Z Checks: read 2024-08-07T17:43:08.1537219Z Contents: read 2024-08-07T17:43:08.1538217Z Deployments: read 2024-08-07T17:43:08.1539193Z Discussions: read 2024-08-07T17:43:08.1540275Z Issues: read 2024-08-07T17:43:08.1541270Z Metadata: read 2024-08-07T17:43:08.1542230Z Packages: read 2024-08-07T17:43:08.1543259Z Pages: read 2024-08-07T17:43:08.1544236Z PullRequests: read 2024-08-07T17:43:08.1545285Z RepositoryProjects: read 2024-08-07T17:43:08.1546408Z SecurityEvents: read 2024-08-07T17:43:08.1547787Z Statuses: read 2024-08-07T17:43:08.1548760Z ##[endgroup] 2024-08-07T17:43:08.1553671Z Secret source: Actions 2024-08-07T17:43:08.1555180Z Prepare workflow directory 2024-08-07T17:43:08.5485724Z Prepare all required actions 2024-08-07T17:43:08.5541415Z Getting action download info 2024-08-07T17:43:08.7255184Z Download action repository 'pytorch/test-infra@main' (SHA:a1f5a89251fc4258ab59806094fe3108f7d6741a) 2024-08-07T17:43:09.1576577Z Download action repository 'pytorch/pytorch@main' (SHA:a62710c82039b798befd86e938d2137af3978c93) 2024-08-07T17:43:12.8734860Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-07T17:43:13.0505910Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-07T17:43:13.3162969Z Getting action download info 2024-08-07T17:43:13.4265695Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-07T17:43:13.6319474Z Getting action download info 2024-08-07T17:43:13.7274215Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-07T17:43:13.8739346Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/131248/merge (f779f6b7738020e244184bded4026b37de3f9f24) 2024-08-07T17:43:13.8742352Z ##[group] Inputs 2024-08-07T17:43:13.8742944Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2024-08-07T17:43:13.8744625Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2024-08-07T17:43:13.8746760Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:43:13.8748154Z sync-tag: 2024-08-07T17:43:13.8749480Z timeout-minutes: 360 2024-08-07T17:43:13.8750104Z use-gha: 2024-08-07T17:43:13.8750618Z dashboard-tag: 2024-08-07T17:43:13.8751073Z s3-bucket: gha-artifacts 2024-08-07T17:43:13.8751619Z aws-role-to-assume: 2024-08-07T17:43:13.8752123Z ##[endgroup] 2024-08-07T17:43:13.8752850Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:43:13.9534803Z A job started hook has been configured by the self-hosted runner administrator 2024-08-07T17:43:13.9692887Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-07T17:43:13.9705442Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:43:13.9706499Z ##[endgroup] 2024-08-07T17:43:15.5504170Z Runner Type: linux.8xlarge.nvidia.gpu 2024-08-07T17:43:15.5504928Z Instance Type: g3.8xlarge 2024-08-07T17:43:15.5505360Z AMI Name: amzn2-ami-hvm-2.0.20240306.2-x86_64-ebs 2024-08-07T17:43:15.5505830Z AMI ID: ami-0ce0c36d7a00b20e2 2024-08-07T17:43:20.5683448Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-07T17:43:20.5684242Z with: 2024-08-07T17:43:20.5685088Z github-secret: *** 2024-08-07T17:43:20.5686035Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-07T17:43:20.5686968Z activate-with-label: false 2024-08-07T17:43:20.5687323Z label: with-ssh 2024-08-07T17:43:20.5687662Z remove-existing-keys: true 2024-08-07T17:43:20.5688034Z fail-silently: true 2024-08-07T17:43:20.5688337Z env: 2024-08-07T17:43:20.5688632Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:43:20.5688992Z ##[endgroup] 2024-08-07T17:43:20.6810830Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-07T17:43:20.9973222Z Grabbing public ssh keys from https://github.com/zdevito.keys 2024-08-07T17:43:21.0735888Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-07T17:43:21.0757331Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-07T17:43:21.0799172Z Login using: ssh ec2-user@ec2-3-81-129-98.compute-1.amazonaws.com 2024-08-07T17:43:21.0799940Z All testing is done inside the container, to start an interactive session run: 2024-08-07T17:43:21.0800649Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-07T17:43:21.0932281Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-07T17:43:21.0932871Z with: 2024-08-07T17:43:21.0933161Z submodules: recursive 2024-08-07T17:43:21.0933524Z fetch-depth: 0 2024-08-07T17:43:21.0933835Z env: 2024-08-07T17:43:21.0934110Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:43:21.0934464Z ##[endgroup] 2024-08-07T17:43:21.1040612Z ##[group]Run retry () { 2024-08-07T17:43:21.1041009Z retry () { 2024-08-07T17:43:21.1041466Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-07T17:43:21.1041969Z } 2024-08-07T17:43:21.1042289Z echo "${GITHUB_WORKSPACE}" 2024-08-07T17:43:21.1042702Z if [ -z "${NO_SUDO}" ]; then 2024-08-07T17:43:21.1043145Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-07T17:43:21.1043578Z else 2024-08-07T17:43:21.1043903Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-07T17:43:21.1044314Z fi 2024-08-07T17:43:21.1044683Z mkdir "${GITHUB_WORKSPACE}" 2024-08-07T17:43:21.1052967Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:43:21.1053457Z env: 2024-08-07T17:43:21.1053741Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:43:21.1054091Z NO_SUDO: 2024-08-07T17:43:21.1054387Z ##[endgroup] 2024-08-07T17:43:21.1080420Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-07T17:43:24.3170873Z ##[group]Run malfet/checkout@silent-checkout 2024-08-07T17:43:24.3171313Z with: 2024-08-07T17:43:24.3171628Z ref: 016588f53c6904b840aa56aa86f95460b4d9c996 2024-08-07T17:43:24.3172051Z fetch-depth: 0 2024-08-07T17:43:24.3172371Z submodules: recursive 2024-08-07T17:43:24.3172718Z quiet-checkout: true 2024-08-07T17:43:24.3173072Z repository: pytorch/pytorch 2024-08-07T17:43:24.3173572Z token: *** 2024-08-07T17:43:24.3173883Z ssh-strict: true 2024-08-07T17:43:24.3174203Z persist-credentials: true 2024-08-07T17:43:24.3174559Z clean: true 2024-08-07T17:43:24.3174883Z sparse-checkout-cone-mode: true 2024-08-07T17:43:24.3175268Z lfs: false 2024-08-07T17:43:24.3175579Z set-safe-directory: true 2024-08-07T17:43:24.3175903Z env: 2024-08-07T17:43:24.3176191Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:43:24.3176531Z ##[endgroup] 2024-08-07T17:43:24.4470818Z Syncing repository: pytorch/pytorch 2024-08-07T17:43:24.4472644Z ##[group]Getting Git version info 2024-08-07T17:43:24.4473266Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-07T17:43:24.4474123Z [command]/usr/bin/git version 2024-08-07T17:43:24.4478278Z git version 2.40.1 2024-08-07T17:43:24.4514350Z ##[endgroup] 2024-08-07T17:43:24.4536842Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/d32ac4dd-199a-4f99-9cf0-f31c952bc740' before making global git config changes 2024-08-07T17:43:24.4538285Z Adding repository directory to the temporary git global config as a safe directory 2024-08-07T17:43:24.4544451Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-07T17:43:24.4590184Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-07T17:43:24.4596543Z ##[group]Initializing the repository 2024-08-07T17:43:24.4600552Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-07T17:43:24.4632120Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-07T17:43:24.4632875Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-07T17:43:24.4633563Z hint: of your new repositories, which will suppress this warning, call: 2024-08-07T17:43:24.4634085Z hint: 2024-08-07T17:43:24.4634488Z hint: git config --global init.defaultBranch 2024-08-07T17:43:24.4634927Z hint: 2024-08-07T17:43:24.4635363Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-07T17:43:24.4636063Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-07T17:43:24.4636800Z hint: 2024-08-07T17:43:24.4637131Z hint: git branch -m 2024-08-07T17:43:24.4637767Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-07T17:43:24.4645693Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-07T17:43:24.4679910Z ##[endgroup] 2024-08-07T17:43:24.4680457Z ##[group]Disabling automatic garbage collection 2024-08-07T17:43:24.4684479Z [command]/usr/bin/git config --local gc.auto 0 2024-08-07T17:43:24.4717733Z ##[endgroup] 2024-08-07T17:43:24.4718240Z ##[group]Setting up auth 2024-08-07T17:43:24.4727756Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-07T17:43:24.4762156Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-07T17:43:24.5051023Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-07T17:43:24.5081372Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-07T17:43:24.5367059Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-07T17:43:24.5415743Z ##[endgroup] 2024-08-07T17:43:24.5416297Z ##[group]Fetching the repository 2024-08-07T17:43:24.5424386Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-07T17:43:26.5205778Z remote: Enumerating objects: 1008246 2024-08-07T17:43:26.5206318Z remote: Enumerating objects: 1011035, done. 2024-08-07T17:43:26.5207537Z remote: Counting objects: 0% (1/2789) 2024-08-07T17:43:26.5208654Z remote: Counting objects: 1% (28/2789) 2024-08-07T17:43:26.5209179Z remote: Counting objects: 2% (56/2789) 2024-08-07T17:43:26.5209942Z remote: Counting objects: 3% (84/2789) 2024-08-07T17:43:26.5211216Z remote: Counting objects: 4% (112/2789) 2024-08-07T17:43:26.5213548Z remote: Counting objects: 5% (140/2789) 2024-08-07T17:43:26.5214436Z remote: Counting objects: 6% (168/2789) 2024-08-07T17:43:26.5214913Z remote: Counting objects: 7% (196/2789) 2024-08-07T17:43:26.5215370Z remote: Counting objects: 8% (224/2789) 2024-08-07T17:43:26.5215848Z remote: Counting objects: 9% (252/2789) 2024-08-07T17:43:26.5216416Z remote: Counting objects: 10% (279/2789) 2024-08-07T17:43:26.5217322Z remote: Counting objects: 11% (307/2789) 2024-08-07T17:43:26.5217803Z remote: Counting objects: 12% (335/2789) 2024-08-07T17:43:26.5218275Z remote: Counting objects: 13% (363/2789) 2024-08-07T17:43:26.5218727Z remote: Counting objects: 14% (391/2789) 2024-08-07T17:43:26.5219208Z remote: Counting objects: 15% (419/2789) 2024-08-07T17:43:26.5219705Z remote: Counting objects: 16% (447/2789) 2024-08-07T17:43:26.5220161Z remote: Counting objects: 17% (475/2789) 2024-08-07T17:43:26.5220634Z remote: Counting objects: 18% (503/2789) 2024-08-07T17:43:26.5221104Z remote: Counting objects: 19% (530/2789) 2024-08-07T17:43:26.5221556Z remote: Counting objects: 20% (558/2789) 2024-08-07T17:43:26.5222034Z remote: Counting objects: 21% (586/2789) 2024-08-07T17:43:26.5222508Z remote: Counting objects: 22% (614/2789) 2024-08-07T17:43:26.5222978Z remote: Counting objects: 23% (642/2789) 2024-08-07T17:43:26.5223438Z remote: Counting objects: 24% (670/2789) 2024-08-07T17:43:26.5223919Z remote: Counting objects: 25% (698/2789) 2024-08-07T17:43:26.5224392Z remote: Counting objects: 26% (726/2789) 2024-08-07T17:43:26.5224936Z remote: Counting objects: 27% (754/2789) 2024-08-07T17:43:26.5225579Z remote: Counting objects: 28% (781/2789) 2024-08-07T17:43:26.5226089Z remote: Counting objects: 29% (809/2789) 2024-08-07T17:43:26.5226539Z remote: Counting objects: 30% (837/2789) 2024-08-07T17:43:26.5227291Z remote: Counting objects: 31% (865/2789) 2024-08-07T17:43:26.5227780Z remote: Counting objects: 32% (893/2789) 2024-08-07T17:43:26.5228236Z remote: Counting objects: 33% (921/2789) 2024-08-07T17:43:26.5228706Z remote: Counting objects: 34% (949/2789) 2024-08-07T17:43:26.5229179Z remote: Counting objects: 35% (977/2789) 2024-08-07T17:43:26.5229638Z remote: Counting objects: 36% (1005/2789) 2024-08-07T17:43:26.5230125Z remote: Counting objects: 37% (1032/2789) 2024-08-07T17:43:26.5230602Z remote: Counting objects: 38% (1060/2789) 2024-08-07T17:43:26.5231061Z remote: Counting objects: 39% (1088/2789) 2024-08-07T17:43:26.5231542Z remote: Counting objects: 40% (1116/2789) 2024-08-07T17:43:26.5232018Z remote: Counting objects: 41% (1144/2789) 2024-08-07T17:43:26.5232473Z remote: Counting objects: 42% (1172/2789) 2024-08-07T17:43:26.5232945Z remote: Counting objects: 43% (1200/2789) 2024-08-07T17:43:26.5233417Z remote: Counting objects: 44% (1228/2789) 2024-08-07T17:43:26.5233898Z remote: Counting objects: 45% (1256/2789) 2024-08-07T17:43:26.5234383Z remote: Counting objects: 46% (1283/2789) 2024-08-07T17:43:26.5234858Z remote: Counting objects: 47% (1311/2789) 2024-08-07T17:43:26.5235337Z remote: Counting objects: 48% (1339/2789) 2024-08-07T17:43:26.5235795Z remote: Counting objects: 49% (1367/2789) 2024-08-07T17:43:26.5236282Z remote: Counting objects: 50% (1395/2789) 2024-08-07T17:43:26.5236758Z remote: Counting objects: 51% (1423/2789) 2024-08-07T17:43:26.5237217Z remote: Counting objects: 52% (1451/2789) 2024-08-07T17:43:26.5237700Z remote: Counting objects: 53% (1479/2789) 2024-08-07T17:43:26.5238173Z remote: Counting objects: 54% (1507/2789) 2024-08-07T17:43:26.5238631Z remote: Counting objects: 55% (1534/2789) 2024-08-07T17:43:26.5239114Z remote: Counting objects: 56% (1562/2789) 2024-08-07T17:43:26.5239592Z remote: Counting objects: 57% (1590/2789) 2024-08-07T17:43:26.5240049Z remote: Counting objects: 58% (1618/2789) 2024-08-07T17:43:26.5240527Z remote: Counting objects: 59% (1646/2789) 2024-08-07T17:43:26.5241004Z remote: Counting objects: 60% (1674/2789) 2024-08-07T17:43:26.5241466Z remote: Counting objects: 61% (1702/2789) 2024-08-07T17:43:26.5241940Z remote: Counting objects: 62% (1730/2789) 2024-08-07T17:43:26.5242624Z remote: Counting objects: 63% (1758/2789) 2024-08-07T17:43:26.5243113Z remote: Counting objects: 64% (1785/2789) 2024-08-07T17:43:26.5243573Z remote: Counting objects: 65% (1813/2789) 2024-08-07T17:43:26.5244057Z remote: Counting objects: 66% (1841/2789) 2024-08-07T17:43:26.5244534Z remote: Counting objects: 67% (1869/2789) 2024-08-07T17:43:26.5245024Z remote: Counting objects: 68% (1897/2789) 2024-08-07T17:43:26.5245502Z remote: Counting objects: 69% (1925/2789) 2024-08-07T17:43:26.5245978Z remote: Counting objects: 70% (1953/2789) 2024-08-07T17:43:26.5246435Z remote: Counting objects: 71% (1981/2789) 2024-08-07T17:43:26.5246907Z remote: Counting objects: 72% (2009/2789) 2024-08-07T17:43:26.5247381Z remote: Counting objects: 73% (2036/2789) 2024-08-07T17:43:26.5247852Z remote: Counting objects: 74% (2064/2789) 2024-08-07T17:43:26.5248333Z remote: Counting objects: 75% (2092/2789) 2024-08-07T17:43:26.5248807Z remote: Counting objects: 76% (2120/2789) 2024-08-07T17:43:26.5249260Z remote: Counting objects: 77% (2148/2789) 2024-08-07T17:43:26.5249731Z remote: Counting objects: 78% (2176/2789) 2024-08-07T17:43:26.5250317Z remote: Counting objects: 79% (2204/2789) 2024-08-07T17:43:26.5250793Z remote: Counting objects: 80% (2232/2789) 2024-08-07T17:43:26.5251272Z remote: Counting objects: 81% (2260/2789) 2024-08-07T17:43:26.5251750Z remote: Counting objects: 82% (2287/2789) 2024-08-07T17:43:26.5252205Z remote: Counting objects: 83% (2315/2789) 2024-08-07T17:43:26.5252678Z remote: Counting objects: 84% (2343/2789) 2024-08-07T17:43:26.5253153Z remote: Counting objects: 85% (2371/2789) 2024-08-07T17:43:26.5253610Z remote: Counting objects: 86% (2399/2789) 2024-08-07T17:43:26.5254082Z remote: Counting objects: 87% (2427/2789) 2024-08-07T17:43:26.5254563Z remote: Counting objects: 88% (2455/2789) 2024-08-07T17:43:26.5255017Z remote: Counting objects: 89% (2483/2789) 2024-08-07T17:43:26.5255486Z remote: Counting objects: 90% (2511/2789) 2024-08-07T17:43:26.5255966Z remote: Counting objects: 91% (2538/2789) 2024-08-07T17:43:26.5256428Z remote: Counting objects: 92% (2566/2789) 2024-08-07T17:43:26.5256904Z remote: Counting objects: 93% (2594/2789) 2024-08-07T17:43:26.5257470Z remote: Counting objects: 94% (2622/2789) 2024-08-07T17:43:26.5257942Z remote: Counting objects: 95% (2650/2789) 2024-08-07T17:43:26.5258397Z remote: Counting objects: 96% (2678/2789) 2024-08-07T17:43:26.5258870Z remote: Counting objects: 97% (2706/2789) 2024-08-07T17:43:26.5259341Z remote: Counting objects: 98% (2734/2789) 2024-08-07T17:43:26.5259798Z remote: Counting objects: 99% (2762/2789) 2024-08-07T17:43:26.5260287Z remote: Counting objects: 100% (2789/2789) 2024-08-07T17:43:26.5260808Z remote: Counting objects: 100% (2789/2789), done. 2024-08-07T17:43:26.5589593Z remote: Compressing objects: 0% (1/1486) 2024-08-07T17:43:26.5771718Z remote: Compressing objects: 1% (15/1486) 2024-08-07T17:43:26.6847079Z remote: Compressing objects: 2% (30/1486) 2024-08-07T17:43:26.8738914Z remote: Compressing objects: 3% (45/1486) 2024-08-07T17:43:26.9875347Z remote: Compressing objects: 4% (60/1486) 2024-08-07T17:43:27.0612624Z remote: Compressing objects: 5% (75/1486) 2024-08-07T17:43:27.1259331Z remote: Compressing objects: 6% (90/1486) 2024-08-07T17:43:27.1723520Z remote: Compressing objects: 7% (105/1486) 2024-08-07T17:43:27.2146657Z remote: Compressing objects: 8% (119/1486) 2024-08-07T17:43:27.2432862Z remote: Compressing objects: 9% (134/1486) 2024-08-07T17:43:27.2702046Z remote: Compressing objects: 10% (149/1486) 2024-08-07T17:43:27.2926159Z remote: Compressing objects: 11% (164/1486) 2024-08-07T17:43:27.3133892Z remote: Compressing objects: 12% (179/1486) 2024-08-07T17:43:27.3324375Z remote: Compressing objects: 13% (194/1486) 2024-08-07T17:43:27.3467883Z remote: Compressing objects: 14% (209/1486) 2024-08-07T17:43:27.3590022Z remote: Compressing objects: 15% (223/1486) 2024-08-07T17:43:27.3686083Z remote: Compressing objects: 16% (238/1486) 2024-08-07T17:43:27.3757794Z remote: Compressing objects: 17% (253/1486) 2024-08-07T17:43:27.3808008Z remote: Compressing objects: 18% (268/1486) 2024-08-07T17:43:27.3847081Z remote: Compressing objects: 19% (283/1486) 2024-08-07T17:43:27.3868349Z remote: Compressing objects: 20% (298/1486) 2024-08-07T17:43:27.3878585Z remote: Compressing objects: 21% (313/1486) 2024-08-07T17:43:27.3885605Z remote: Compressing objects: 22% (327/1486) 2024-08-07T17:43:27.3916789Z remote: Compressing objects: 23% (342/1486) 2024-08-07T17:43:27.3937821Z remote: Compressing objects: 24% (357/1486) 2024-08-07T17:43:27.3957924Z remote: Compressing objects: 25% (372/1486) 2024-08-07T17:43:27.3965863Z remote: Compressing objects: 26% (387/1486) 2024-08-07T17:43:27.3988043Z remote: Compressing objects: 27% (402/1486) 2024-08-07T17:43:27.4001694Z remote: Compressing objects: 28% (417/1486) 2024-08-07T17:43:27.4014314Z remote: Compressing objects: 29% (431/1486) 2024-08-07T17:43:27.4038104Z remote: Compressing objects: 30% (446/1486) 2024-08-07T17:43:27.4053393Z remote: Compressing objects: 31% (461/1486) 2024-08-07T17:43:27.4066122Z remote: Compressing objects: 32% (476/1486) 2024-08-07T17:43:27.4076682Z remote: Compressing objects: 33% (491/1486) 2024-08-07T17:43:27.4087208Z remote: Compressing objects: 34% (506/1486) 2024-08-07T17:43:27.4097612Z remote: Compressing objects: 35% (521/1486) 2024-08-07T17:43:27.4108031Z remote: Compressing objects: 36% (535/1486) 2024-08-07T17:43:27.4115901Z remote: Compressing objects: 37% (550/1486) 2024-08-07T17:43:27.4126872Z remote: Compressing objects: 38% (565/1486) 2024-08-07T17:43:27.4137344Z remote: Compressing objects: 39% (580/1486) 2024-08-07T17:43:27.4144352Z remote: Compressing objects: 40% (595/1486) 2024-08-07T17:43:27.4156339Z remote: Compressing objects: 41% (610/1486) 2024-08-07T17:43:27.4160826Z remote: Compressing objects: 42% (625/1486) 2024-08-07T17:43:27.4169588Z remote: Compressing objects: 43% (639/1486) 2024-08-07T17:43:27.4174553Z remote: Compressing objects: 44% (654/1486) 2024-08-07T17:43:27.4180962Z remote: Compressing objects: 45% (669/1486) 2024-08-07T17:43:27.4185276Z remote: Compressing objects: 46% (684/1486) 2024-08-07T17:43:27.4190818Z remote: Compressing objects: 47% (699/1486) 2024-08-07T17:43:27.4195983Z remote: Compressing objects: 48% (714/1486) 2024-08-07T17:43:27.4199979Z remote: Compressing objects: 49% (729/1486) 2024-08-07T17:43:27.4204728Z remote: Compressing objects: 50% (743/1486) 2024-08-07T17:43:27.4208418Z remote: Compressing objects: 51% (758/1486) 2024-08-07T17:43:27.4213362Z remote: Compressing objects: 52% (773/1486) 2024-08-07T17:43:27.4215624Z remote: Compressing objects: 53% (788/1486) 2024-08-07T17:43:27.4218524Z remote: Compressing objects: 54% (803/1486) 2024-08-07T17:43:27.4220862Z remote: Compressing objects: 55% (818/1486) 2024-08-07T17:43:27.4222058Z remote: Compressing objects: 56% (833/1486) 2024-08-07T17:43:27.4223235Z remote: Compressing objects: 57% (848/1486) 2024-08-07T17:43:27.4224184Z remote: Compressing objects: 58% (862/1486) 2024-08-07T17:43:27.4225079Z remote: Compressing objects: 59% (877/1486) 2024-08-07T17:43:27.4225613Z remote: Compressing objects: 60% (892/1486) 2024-08-07T17:43:27.4226293Z remote: Compressing objects: 61% (907/1486) 2024-08-07T17:43:27.4227254Z remote: Compressing objects: 62% (922/1486) 2024-08-07T17:43:27.4227763Z remote: Compressing objects: 63% (937/1486) 2024-08-07T17:43:27.4228245Z remote: Compressing objects: 64% (952/1486) 2024-08-07T17:43:27.4228757Z remote: Compressing objects: 65% (966/1486) 2024-08-07T17:43:27.4250787Z remote: Compressing objects: 66% (981/1486) 2024-08-07T17:43:27.4266482Z remote: Compressing objects: 67% (996/1486) 2024-08-07T17:43:27.4270201Z remote: Compressing objects: 68% (1011/1486) 2024-08-07T17:43:27.4278368Z remote: Compressing objects: 69% (1026/1486) 2024-08-07T17:43:27.4291112Z remote: Compressing objects: 70% (1041/1486) 2024-08-07T17:43:27.4295672Z remote: Compressing objects: 71% (1056/1486) 2024-08-07T17:43:27.4301782Z remote: Compressing objects: 72% (1070/1486) 2024-08-07T17:43:27.4307464Z remote: Compressing objects: 73% (1085/1486) 2024-08-07T17:43:27.4313044Z remote: Compressing objects: 74% (1100/1486) 2024-08-07T17:43:27.4318838Z remote: Compressing objects: 75% (1115/1486) 2024-08-07T17:43:27.4324867Z remote: Compressing objects: 76% (1130/1486) 2024-08-07T17:43:27.4325558Z remote: Compressing objects: 77% (1145/1486) 2024-08-07T17:43:27.4329862Z remote: Compressing objects: 78% (1160/1486) 2024-08-07T17:43:27.4330383Z remote: Compressing objects: 79% (1174/1486) 2024-08-07T17:43:27.4335823Z remote: Compressing objects: 80% (1189/1486) 2024-08-07T17:43:27.4341072Z remote: Compressing objects: 81% (1204/1486) 2024-08-07T17:43:27.4341573Z remote: Compressing objects: 82% (1219/1486) 2024-08-07T17:43:27.4347704Z remote: Compressing objects: 83% (1234/1486) 2024-08-07T17:43:27.4348222Z remote: Compressing objects: 84% (1249/1486) 2024-08-07T17:43:27.4352710Z remote: Compressing objects: 85% (1264/1486) 2024-08-07T17:43:27.4353239Z remote: Compressing objects: 86% (1278/1486) 2024-08-07T17:43:27.4353748Z remote: Compressing objects: 87% (1293/1486) 2024-08-07T17:43:27.4354243Z remote: Compressing objects: 88% (1308/1486) 2024-08-07T17:43:27.4354778Z remote: Compressing objects: 89% (1323/1486) 2024-08-07T17:43:27.4355592Z remote: Compressing objects: 90% (1338/1486) 2024-08-07T17:43:27.4356137Z remote: Compressing objects: 91% (1353/1486) 2024-08-07T17:43:27.4357015Z remote: Compressing objects: 92% (1368/1486) 2024-08-07T17:43:27.4359601Z remote: Compressing objects: 93% (1382/1486) 2024-08-07T17:43:27.4361699Z remote: Compressing objects: 94% (1397/1486) 2024-08-07T17:43:27.4363815Z remote: Compressing objects: 95% (1412/1486) 2024-08-07T17:43:27.4365393Z remote: Compressing objects: 96% (1427/1486) 2024-08-07T17:43:27.4367448Z remote: Compressing objects: 97% (1442/1486) 2024-08-07T17:43:27.4368836Z remote: Compressing objects: 98% (1457/1486) 2024-08-07T17:43:27.4369838Z remote: Compressing objects: 99% (1472/1486) 2024-08-07T17:43:27.4370498Z remote: Compressing objects: 100% (1486/1486) 2024-08-07T17:43:27.4371047Z remote: Compressing objects: 100% (1486/1486), done. 2024-08-07T17:43:54.1546169Z remote: Total 1011035 (delta 1772), reused 2144 (delta 1296), pack-reused 1008246 2024-08-07T17:44:16.6940515Z [command]/usr/bin/git rev-parse --verify --quiet 016588f53c6904b840aa56aa86f95460b4d9c996^{object} 2024-08-07T17:44:16.6968421Z 016588f53c6904b840aa56aa86f95460b4d9c996 2024-08-07T17:44:16.6974094Z ##[endgroup] 2024-08-07T17:44:16.6974651Z ##[group]Determining the checkout info 2024-08-07T17:44:16.6976251Z ##[endgroup] 2024-08-07T17:44:16.6976768Z ##[group]Checking out the ref 2024-08-07T17:44:16.6980684Z [command]/usr/bin/git checkout --quiet --force 016588f53c6904b840aa56aa86f95460b4d9c996 2024-08-07T17:44:18.5236909Z ##[endgroup] 2024-08-07T17:44:18.5237514Z ##[group]Setting up auth for fetching submodules 2024-08-07T17:44:18.5242899Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-07T17:44:18.5308357Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-07T17:44:18.5340144Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-07T17:44:18.5372258Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-07T17:44:18.5401807Z ##[endgroup] 2024-08-07T17:44:18.5402340Z ##[group]Fetching submodules 2024-08-07T17:44:18.5407104Z [command]/usr/bin/git submodule sync --recursive 2024-08-07T17:44:18.5727341Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-07T17:44:18.6031276Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-07T17:44:18.6033872Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-07T17:44:18.6037149Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-07T17:44:18.6040697Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-07T17:44:18.6044269Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-07T17:44:18.6047478Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-07T17:44:18.6051203Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-07T17:44:18.6055219Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-07T17:44:18.6059144Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-07T17:44:18.6063380Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-07T17:44:18.6067508Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-07T17:44:18.6072433Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-07T17:44:18.6076942Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-07T17:44:18.6081615Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-07T17:44:18.6086329Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-07T17:44:18.6091225Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-08-07T17:44:18.6096446Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-07T17:44:18.6101467Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-07T17:44:18.6106844Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-07T17:44:18.6112802Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-07T17:44:18.6118342Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-07T17:44:18.6124095Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-07T17:44:18.6130167Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-07T17:44:18.6136121Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-07T17:44:18.6142248Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-07T17:44:18.6148810Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-07T17:44:18.6155474Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-07T17:44:18.6161811Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-07T17:44:18.6168937Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-07T17:44:18.6175711Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-07T17:44:18.6182898Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-07T17:44:18.6190229Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-07T17:44:18.6197321Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-07T17:44:18.6204562Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-07T17:44:18.6212125Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-07T17:44:18.6239740Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-07T17:44:18.8967143Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-07T17:44:19.0750644Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-07T17:44:19.2504136Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-07T17:44:19.5175500Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-07T17:44:21.6824915Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-07T17:44:34.3749516Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-07T17:44:34.7883079Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-07T17:44:35.2001456Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-07T17:44:35.8736561Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-07T17:44:37.3349787Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-07T17:44:39.6735287Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-07T17:44:47.0985129Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-07T17:44:48.5385906Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-07T17:44:51.4896940Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-07T17:44:52.8090787Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-08-07T17:44:53.0007071Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-07T17:44:53.4677297Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-07T17:44:53.8616053Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-07T17:44:55.0600014Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-07T17:44:55.4005966Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-07T17:44:55.6775587Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-07T17:44:57.6273508Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-07T17:44:58.4441997Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-07T17:44:59.1129954Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-07T17:45:06.0471498Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-07T17:45:08.2183593Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-07T17:45:11.9920159Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-07T17:45:12.2283454Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-07T17:45:22.0383401Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-07T17:45:22.1857155Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-07T17:45:22.3784542Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-07T17:45:23.3325155Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-07T17:45:23.6617645Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-07T17:45:24.2795573Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-07T17:45:24.7589840Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-07T17:45:24.7715822Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-07T17:45:24.7809777Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-07T17:45:24.8088699Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-07T17:45:24.8506174Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-07T17:45:26.0259246Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-07T17:45:26.0524662Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-07T17:45:26.1065394Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-07T17:45:26.2256667Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-07T17:45:26.2648271Z Submodule path 'third_party/cudnn_frontend': checked out '98ca4e1941fe3263f128f74f10063a3ea35c7019' 2024-08-07T17:45:26.9350040Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-07T17:45:27.2436218Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-07T17:45:27.3404480Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-07T17:45:27.3422846Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-07T17:45:27.3426106Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T17:45:27.3430023Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-07T17:45:27.3433523Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-07T17:45:27.3437338Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T17:45:27.3462797Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-07T17:45:28.3143493Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-07T17:45:28.9937181Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-07T17:45:31.4388450Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-07T17:45:32.6244859Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-07T17:45:32.9857956Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-07T17:45:33.1046078Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-07T17:45:33.6165311Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-07T17:45:33.6915523Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-07T17:45:33.7047499Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-07T17:45:33.8432999Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-07T17:45:33.8910870Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-07T17:45:33.9011420Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-08-07T17:45:33.9469934Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-07T17:45:33.9756999Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-07T17:45:34.0285327Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-07T17:45:34.0424415Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-07T17:45:34.0441592Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-07T17:45:34.0468003Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-07T17:45:48.9061631Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-07T17:45:48.9267605Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-07T17:45:49.0400864Z Submodule path 'third_party/kineto': checked out 'da2f2682cabaf95d601fa2a9b7e0979f84fe7667' 2024-08-07T17:45:49.0419251Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T17:45:49.0422235Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T17:45:49.0425533Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T17:45:49.0452959Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-07T17:45:49.6248185Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-07T17:45:50.8837547Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-07T17:45:52.1043061Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-07T17:45:52.1059692Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T17:45:52.1062771Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T17:45:52.1065998Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T17:45:52.1069994Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T17:45:52.1073348Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T17:45:52.1077311Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T17:45:52.1080826Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T17:45:52.1084602Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T17:45:52.1112197Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-07T17:45:53.0529281Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-07T17:45:53.4817122Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-07T17:45:54.7240003Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-07T17:45:55.4391171Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-07T17:45:55.9920904Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-07T17:45:57.1172213Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-07T17:46:04.0011359Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-07T17:46:04.4026776Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-07T17:46:04.4240798Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-07T17:46:04.4664746Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-07T17:46:04.4807392Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-07T17:46:04.4824185Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T17:46:04.4851351Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-07T17:46:05.2544708Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-07T17:46:05.2746769Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-07T17:46:05.3220696Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-07T17:46:05.4469671Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-07T17:46:05.4654166Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-07T17:46:05.5105143Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-07T17:46:05.5780424Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-07T17:46:05.6200465Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-07T17:46:05.6471424Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-07T17:46:05.7775766Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-07T17:46:06.1985568Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-07T17:46:06.2022572Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-07T17:46:06.2025314Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-07T17:46:06.2052320Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-07T17:46:06.6845987Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-07T17:46:07.7874533Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-07T17:46:07.8267758Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-07T17:46:07.9058221Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-07T17:46:07.9078813Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T17:46:07.9082475Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T17:46:07.9086854Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T17:46:07.9091071Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T17:46:07.9095639Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T17:46:07.9100319Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T17:46:07.9104907Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T17:46:07.9109918Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T17:46:07.9137039Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-07T17:46:08.3795566Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-07T17:46:09.5528755Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-07T17:46:09.9293144Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-07T17:46:17.1943931Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-07T17:46:17.4468301Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-07T17:46:17.6407932Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-07T17:46:18.0159537Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-07T17:46:25.2013405Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-07T17:46:25.2484158Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-07T17:46:25.2656723Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-07T17:46:25.3890209Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-07T17:46:25.4029047Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-07T17:46:25.4188639Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-07T17:46:25.4353887Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-07T17:46:25.4370364Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T17:46:25.4373487Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T17:46:25.4400073Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-07T17:46:27.4873582Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-07T17:46:28.9475186Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-07T17:46:29.0014589Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-07T17:46:29.5502414Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-07T17:46:29.5627204Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-07T17:46:29.8873972Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-07T17:46:29.8896482Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-07T17:46:29.8899800Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-07T17:46:29.8926981Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-07T17:46:30.3502959Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-07T17:46:31.5156363Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-07T17:46:31.6017695Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-07T17:46:31.6111439Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-07T17:46:31.6249081Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-07T17:46:31.6681204Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-07T17:46:31.7012707Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-07T17:46:31.7505615Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-07T17:46:31.7797463Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-07T17:46:31.7814856Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-07T17:46:31.7818284Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-07T17:46:31.7821489Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-07T17:46:31.7824913Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T17:46:31.7851029Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-07T17:46:32.9799592Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-07T17:46:33.2218184Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-07T17:46:34.5457057Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-07T17:46:35.5350677Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-07T17:46:35.5516874Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-07T17:46:35.6337899Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-07T17:46:35.6665667Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-07T17:46:35.6682146Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T17:46:35.6707747Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-07T17:46:35.8844610Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-07T17:46:35.8879143Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-07T17:46:35.9195993Z Entering 'android/libs/fbjni' 2024-08-07T17:46:35.9239249Z Entering 'third_party/FP16' 2024-08-07T17:46:35.9280290Z Entering 'third_party/FXdiv' 2024-08-07T17:46:35.9321010Z Entering 'third_party/NNPACK' 2024-08-07T17:46:35.9362726Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-07T17:46:35.9404744Z Entering 'third_party/XNNPACK' 2024-08-07T17:46:35.9463486Z Entering 'third_party/benchmark' 2024-08-07T17:46:35.9505621Z Entering 'third_party/cpp-httplib' 2024-08-07T17:46:35.9547815Z Entering 'third_party/cpuinfo' 2024-08-07T17:46:35.9590826Z Entering 'third_party/cudnn_frontend' 2024-08-07T17:46:35.9633344Z Entering 'third_party/cutlass' 2024-08-07T17:46:35.9682199Z Entering 'third_party/eigen' 2024-08-07T17:46:35.9725943Z Entering 'third_party/fbgemm' 2024-08-07T17:46:35.9768198Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-07T17:46:35.9808074Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T17:46:35.9849894Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-07T17:46:35.9896823Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-07T17:46:35.9936620Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T17:46:35.9977178Z Entering 'third_party/flatbuffers' 2024-08-07T17:46:36.0023464Z Entering 'third_party/fmt' 2024-08-07T17:46:36.0064198Z Entering 'third_party/foxi' 2024-08-07T17:46:36.0105637Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-07T17:46:36.0147795Z Entering 'third_party/gloo' 2024-08-07T17:46:36.0189127Z Entering 'third_party/googletest' 2024-08-07T17:46:36.0229943Z Entering 'third_party/ideep' 2024-08-07T17:46:36.0269216Z Entering 'third_party/ideep/mkl-dnn' 2024-08-07T17:46:36.0317989Z Entering 'third_party/ittapi' 2024-08-07T17:46:36.0358802Z Entering 'third_party/kineto' 2024-08-07T17:46:36.0400987Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T17:46:36.0441539Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T17:46:36.0483331Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T17:46:36.0523614Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T17:46:36.0565357Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T17:46:36.0605431Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T17:46:36.0648101Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T17:46:36.0688673Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T17:46:36.0730288Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T17:46:36.0771475Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T17:46:36.0812912Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T17:46:36.0853412Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T17:46:36.0894913Z Entering 'third_party/mimalloc' 2024-08-07T17:46:36.0936733Z Entering 'third_party/nccl/nccl' 2024-08-07T17:46:36.0977256Z Entering 'third_party/nlohmann' 2024-08-07T17:46:36.1019488Z Entering 'third_party/onnx' 2024-08-07T17:46:36.1077994Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-07T17:46:36.1119234Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-07T17:46:36.1162680Z Entering 'third_party/opentelemetry-cpp' 2024-08-07T17:46:36.1206237Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T17:46:36.1246893Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T17:46:36.1288137Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T17:46:36.1327497Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T17:46:36.1369015Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T17:46:36.1408883Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T17:46:36.1449607Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T17:46:36.1489279Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T17:46:36.1532149Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T17:46:36.1574522Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T17:46:36.1638314Z Entering 'third_party/pocketfft' 2024-08-07T17:46:36.1679154Z Entering 'third_party/protobuf' 2024-08-07T17:46:36.1724761Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-07T17:46:36.1766417Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-07T17:46:36.1808878Z Entering 'third_party/psimd' 2024-08-07T17:46:36.1850025Z Entering 'third_party/pthreadpool' 2024-08-07T17:46:36.1891437Z Entering 'third_party/pybind11' 2024-08-07T17:46:36.1932349Z Entering 'third_party/python-peachpy' 2024-08-07T17:46:36.1973957Z Entering 'third_party/sleef' 2024-08-07T17:46:36.2014346Z Entering 'third_party/tensorpipe' 2024-08-07T17:46:36.2055092Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-07T17:46:36.2094989Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-07T17:46:36.2134339Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-07T17:46:36.2174819Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T17:46:36.2213860Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T17:46:36.2269422Z ##[endgroup] 2024-08-07T17:46:36.2271438Z ##[group]Persisting credentials for submodules 2024-08-07T17:46:36.2276857Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-07T17:46:36.2585681Z Entering 'android/libs/fbjni' 2024-08-07T17:46:36.2637572Z Entering 'third_party/FP16' 2024-08-07T17:46:36.2688671Z Entering 'third_party/FXdiv' 2024-08-07T17:46:36.2740305Z Entering 'third_party/NNPACK' 2024-08-07T17:46:36.2792606Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-07T17:46:36.2843657Z Entering 'third_party/XNNPACK' 2024-08-07T17:46:36.2912929Z Entering 'third_party/benchmark' 2024-08-07T17:46:36.2963562Z Entering 'third_party/cpp-httplib' 2024-08-07T17:46:36.3014100Z Entering 'third_party/cpuinfo' 2024-08-07T17:46:36.3066415Z Entering 'third_party/cudnn_frontend' 2024-08-07T17:46:36.3116845Z Entering 'third_party/cutlass' 2024-08-07T17:46:36.3175055Z Entering 'third_party/eigen' 2024-08-07T17:46:36.3228523Z Entering 'third_party/fbgemm' 2024-08-07T17:46:36.3278903Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-07T17:46:36.3329282Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T17:46:36.3379799Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-07T17:46:36.3436924Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-07T17:46:36.3486688Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T17:46:36.3538155Z Entering 'third_party/flatbuffers' 2024-08-07T17:46:36.3591398Z Entering 'third_party/fmt' 2024-08-07T17:46:36.3641580Z Entering 'third_party/foxi' 2024-08-07T17:46:36.3691616Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-07T17:46:36.3742443Z Entering 'third_party/gloo' 2024-08-07T17:46:36.3794393Z Entering 'third_party/googletest' 2024-08-07T17:46:36.3844073Z Entering 'third_party/ideep' 2024-08-07T17:46:36.3893259Z Entering 'third_party/ideep/mkl-dnn' 2024-08-07T17:46:36.3951005Z Entering 'third_party/ittapi' 2024-08-07T17:46:36.4001368Z Entering 'third_party/kineto' 2024-08-07T17:46:36.4051266Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T17:46:36.4102331Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T17:46:36.4154455Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T17:46:36.4204494Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T17:46:36.4255766Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T17:46:36.4304560Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T17:46:36.4356621Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T17:46:36.4406526Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T17:46:36.4457371Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T17:46:36.4509420Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T17:46:36.4561551Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T17:46:36.4612385Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T17:46:36.4665028Z Entering 'third_party/mimalloc' 2024-08-07T17:46:36.4717277Z Entering 'third_party/nccl/nccl' 2024-08-07T17:46:36.4768535Z Entering 'third_party/nlohmann' 2024-08-07T17:46:36.4820353Z Entering 'third_party/onnx' 2024-08-07T17:46:36.4889944Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-07T17:46:36.4941249Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-07T17:46:36.4995236Z Entering 'third_party/opentelemetry-cpp' 2024-08-07T17:46:36.5046868Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T17:46:36.5096339Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T17:46:36.5148589Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T17:46:36.5198618Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T17:46:36.5250967Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T17:46:36.5300938Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T17:46:36.5353256Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T17:46:36.5402270Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T17:46:36.5455085Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T17:46:36.5508524Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T17:46:36.5581744Z Entering 'third_party/pocketfft' 2024-08-07T17:46:36.5633465Z Entering 'third_party/protobuf' 2024-08-07T17:46:36.5688651Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-07T17:46:36.5738401Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-07T17:46:36.5792766Z Entering 'third_party/psimd' 2024-08-07T17:46:36.5843664Z Entering 'third_party/pthreadpool' 2024-08-07T17:46:36.5894744Z Entering 'third_party/pybind11' 2024-08-07T17:46:36.5945071Z Entering 'third_party/python-peachpy' 2024-08-07T17:46:36.5996234Z Entering 'third_party/sleef' 2024-08-07T17:46:36.6046470Z Entering 'third_party/tensorpipe' 2024-08-07T17:46:36.6097613Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-07T17:46:36.6147565Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-07T17:46:36.6198350Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-07T17:46:36.6248514Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T17:46:36.6297349Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T17:46:36.6364850Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-07T17:46:36.6674399Z Entering 'android/libs/fbjni' 2024-08-07T17:46:36.6722466Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-07T17:46:36.6741117Z Entering 'third_party/FP16' 2024-08-07T17:46:36.6790148Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-07T17:46:36.6807615Z Entering 'third_party/FXdiv' 2024-08-07T17:46:36.6856800Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-07T17:46:36.6875867Z Entering 'third_party/NNPACK' 2024-08-07T17:46:36.6923588Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-07T17:46:36.6942176Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-07T17:46:36.6991861Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-07T17:46:36.7010623Z Entering 'third_party/XNNPACK' 2024-08-07T17:46:36.7058477Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-07T17:46:36.7095834Z Entering 'third_party/benchmark' 2024-08-07T17:46:36.7143209Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-07T17:46:36.7163179Z Entering 'third_party/cpp-httplib' 2024-08-07T17:46:36.7210183Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-07T17:46:36.7228859Z Entering 'third_party/cpuinfo' 2024-08-07T17:46:36.7276680Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-07T17:46:36.7295631Z Entering 'third_party/cudnn_frontend' 2024-08-07T17:46:36.7343689Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-07T17:46:36.7362184Z Entering 'third_party/cutlass' 2024-08-07T17:46:36.7409170Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-07T17:46:36.7436013Z Entering 'third_party/eigen' 2024-08-07T17:46:36.7482781Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-07T17:46:36.7504187Z Entering 'third_party/fbgemm' 2024-08-07T17:46:36.7553235Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-07T17:46:36.7571527Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-07T17:46:36.7619032Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-07T17:46:36.7637925Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T17:46:36.7683931Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-07T17:46:36.7702141Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-07T17:46:36.7748675Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-07T17:46:36.7773430Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-07T17:46:36.7820430Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-07T17:46:36.7838730Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T17:46:36.7885130Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-07T17:46:36.7903674Z Entering 'third_party/flatbuffers' 2024-08-07T17:46:36.7951068Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-07T17:46:36.7972243Z Entering 'third_party/fmt' 2024-08-07T17:46:36.8018964Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-07T17:46:36.8037670Z Entering 'third_party/foxi' 2024-08-07T17:46:36.8085056Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-08-07T17:46:36.8103451Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-07T17:46:36.8151652Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-07T17:46:36.8170036Z Entering 'third_party/gloo' 2024-08-07T17:46:36.8217137Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-07T17:46:36.8236283Z Entering 'third_party/googletest' 2024-08-07T17:46:36.8283617Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-07T17:46:36.8301878Z Entering 'third_party/ideep' 2024-08-07T17:46:36.8349215Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-07T17:46:36.8366347Z Entering 'third_party/ideep/mkl-dnn' 2024-08-07T17:46:36.8412871Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-07T17:46:36.8439263Z Entering 'third_party/ittapi' 2024-08-07T17:46:36.8486246Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-07T17:46:36.8504371Z Entering 'third_party/kineto' 2024-08-07T17:46:36.8552240Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-07T17:46:36.8570393Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T17:46:36.8618850Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-07T17:46:36.8636799Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T17:46:36.8684916Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-07T17:46:36.8703887Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T17:46:36.8753031Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-07T17:46:36.8770876Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T17:46:36.8818618Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-07T17:46:36.8837315Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T17:46:36.8884950Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-07T17:46:36.8901701Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T17:46:36.8950644Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-07T17:46:36.8969853Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T17:46:36.9018732Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-07T17:46:36.9037178Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T17:46:36.9083754Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-07T17:46:36.9101671Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T17:46:36.9148883Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-07T17:46:36.9168347Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T17:46:36.9216311Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-07T17:46:36.9235779Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T17:46:36.9283095Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-07T17:46:36.9300792Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T17:46:36.9348398Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-07T17:46:36.9367862Z Entering 'third_party/mimalloc' 2024-08-07T17:46:36.9414551Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-07T17:46:36.9434068Z Entering 'third_party/nccl/nccl' 2024-08-07T17:46:36.9481237Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-07T17:46:36.9499770Z Entering 'third_party/nlohmann' 2024-08-07T17:46:36.9547151Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-07T17:46:36.9567381Z Entering 'third_party/onnx' 2024-08-07T17:46:36.9614035Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-07T17:46:36.9651445Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-07T17:46:36.9698147Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-07T17:46:36.9717510Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-07T17:46:36.9764763Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-07T17:46:36.9785232Z Entering 'third_party/opentelemetry-cpp' 2024-08-07T17:46:36.9833826Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-07T17:46:36.9853901Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T17:46:36.9901049Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-07T17:46:36.9919138Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T17:46:36.9965937Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-07T17:46:36.9983856Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T17:46:37.0031588Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-07T17:46:37.0049128Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T17:46:37.0095355Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-07T17:46:37.0115250Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T17:46:37.0161531Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-07T17:46:37.0178892Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T17:46:37.0225495Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-07T17:46:37.0244146Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T17:46:37.0291058Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-07T17:46:37.0307954Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T17:46:37.0355658Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-07T17:46:37.0375584Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T17:46:37.0422305Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-07T17:46:37.0442024Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T17:46:37.0488537Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-07T17:46:37.0527629Z Entering 'third_party/pocketfft' 2024-08-07T17:46:37.0573611Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-07T17:46:37.0593127Z Entering 'third_party/protobuf' 2024-08-07T17:46:37.0639207Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-07T17:46:37.0660227Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-07T17:46:37.0707988Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-07T17:46:37.0726176Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-07T17:46:37.0772527Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-07T17:46:37.0792332Z Entering 'third_party/psimd' 2024-08-07T17:46:37.0839903Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-07T17:46:37.0857808Z Entering 'third_party/pthreadpool' 2024-08-07T17:46:37.0904589Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-07T17:46:37.0923129Z Entering 'third_party/pybind11' 2024-08-07T17:46:37.0969441Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-07T17:46:37.0987880Z Entering 'third_party/python-peachpy' 2024-08-07T17:46:37.1034743Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-07T17:46:37.1053163Z Entering 'third_party/sleef' 2024-08-07T17:46:37.1100731Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-07T17:46:37.1119551Z Entering 'third_party/tensorpipe' 2024-08-07T17:46:37.1167172Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-07T17:46:37.1184786Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-07T17:46:37.1232118Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-07T17:46:37.1250048Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-07T17:46:37.1296694Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-07T17:46:37.1314290Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-07T17:46:37.1361467Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-07T17:46:37.1379411Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T17:46:37.1425993Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-07T17:46:37.1442872Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T17:46:37.1489321Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-07T17:46:37.2561546Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-07T17:46:37.2867242Z Entering 'android/libs/fbjni' 2024-08-07T17:46:37.2907984Z Entering 'third_party/FP16' 2024-08-07T17:46:37.2949248Z Entering 'third_party/FXdiv' 2024-08-07T17:46:37.2990065Z Entering 'third_party/NNPACK' 2024-08-07T17:46:37.3032371Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-07T17:46:37.3073151Z Entering 'third_party/XNNPACK' 2024-08-07T17:46:37.3131737Z Entering 'third_party/benchmark' 2024-08-07T17:46:37.3172425Z Entering 'third_party/cpp-httplib' 2024-08-07T17:46:37.3215439Z Entering 'third_party/cpuinfo' 2024-08-07T17:46:37.3257146Z Entering 'third_party/cudnn_frontend' 2024-08-07T17:46:37.3298671Z Entering 'third_party/cutlass' 2024-08-07T17:46:37.3348915Z Entering 'third_party/eigen' 2024-08-07T17:46:37.3392355Z Entering 'third_party/fbgemm' 2024-08-07T17:46:37.3433837Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-07T17:46:37.3474590Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T17:46:37.3515804Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-07T17:46:37.3563274Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-07T17:46:37.3603319Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T17:46:37.3646268Z Entering 'third_party/flatbuffers' 2024-08-07T17:46:37.3700293Z Entering 'third_party/fmt' 2024-08-07T17:46:37.3742595Z Entering 'third_party/foxi' 2024-08-07T17:46:37.3784235Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-07T17:46:37.3826763Z Entering 'third_party/gloo' 2024-08-07T17:46:37.3869187Z Entering 'third_party/googletest' 2024-08-07T17:46:37.3911264Z Entering 'third_party/ideep' 2024-08-07T17:46:37.3952118Z Entering 'third_party/ideep/mkl-dnn' 2024-08-07T17:46:37.4001366Z Entering 'third_party/ittapi' 2024-08-07T17:46:37.4043128Z Entering 'third_party/kineto' 2024-08-07T17:46:37.4084344Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T17:46:37.4126898Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T17:46:37.4170146Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T17:46:37.4211759Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T17:46:37.4255151Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T17:46:37.4295595Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T17:46:37.4339069Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T17:46:37.4380850Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T17:46:37.4421959Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T17:46:37.4465226Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T17:46:37.4508139Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T17:46:37.4549780Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T17:46:37.4592194Z Entering 'third_party/mimalloc' 2024-08-07T17:46:37.4635857Z Entering 'third_party/nccl/nccl' 2024-08-07T17:46:37.4678452Z Entering 'third_party/nlohmann' 2024-08-07T17:46:37.4722505Z Entering 'third_party/onnx' 2024-08-07T17:46:37.4784065Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-07T17:46:37.4826103Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-07T17:46:37.4870848Z Entering 'third_party/opentelemetry-cpp' 2024-08-07T17:46:37.4915554Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T17:46:37.4956285Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T17:46:37.4998143Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T17:46:37.5039344Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T17:46:37.5081436Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T17:46:37.5122017Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T17:46:37.5161780Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T17:46:37.5203470Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T17:46:37.5247766Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T17:46:37.5289674Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T17:46:37.5353998Z Entering 'third_party/pocketfft' 2024-08-07T17:46:37.5395831Z Entering 'third_party/protobuf' 2024-08-07T17:46:37.5441634Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-07T17:46:37.5483652Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-07T17:46:37.5526638Z Entering 'third_party/psimd' 2024-08-07T17:46:37.5570765Z Entering 'third_party/pthreadpool' 2024-08-07T17:46:37.5613422Z Entering 'third_party/pybind11' 2024-08-07T17:46:37.5656758Z Entering 'third_party/python-peachpy' 2024-08-07T17:46:37.5700018Z Entering 'third_party/sleef' 2024-08-07T17:46:37.5744361Z Entering 'third_party/tensorpipe' 2024-08-07T17:46:37.5786796Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-07T17:46:37.5829135Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-07T17:46:37.5872096Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-07T17:46:37.5913866Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T17:46:37.5953770Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T17:46:37.6013933Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-07T17:46:37.6340858Z Entering 'android/libs/fbjni' 2024-08-07T17:46:37.6382537Z Entering 'third_party/FP16' 2024-08-07T17:46:37.6425401Z Entering 'third_party/FXdiv' 2024-08-07T17:46:37.6468155Z Entering 'third_party/NNPACK' 2024-08-07T17:46:37.6510184Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-07T17:46:37.6552131Z Entering 'third_party/XNNPACK' 2024-08-07T17:46:37.6612449Z Entering 'third_party/benchmark' 2024-08-07T17:46:37.6654014Z Entering 'third_party/cpp-httplib' 2024-08-07T17:46:37.6696368Z Entering 'third_party/cpuinfo' 2024-08-07T17:46:37.6738602Z Entering 'third_party/cudnn_frontend' 2024-08-07T17:46:37.6780597Z Entering 'third_party/cutlass' 2024-08-07T17:46:37.6831469Z Entering 'third_party/eigen' 2024-08-07T17:46:37.6875931Z Entering 'third_party/fbgemm' 2024-08-07T17:46:37.6917874Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-07T17:46:37.6957847Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T17:46:37.6998478Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-07T17:46:37.7046866Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-07T17:46:37.7087199Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T17:46:37.7127924Z Entering 'third_party/flatbuffers' 2024-08-07T17:46:37.7172910Z Entering 'third_party/fmt' 2024-08-07T17:46:37.7213953Z Entering 'third_party/foxi' 2024-08-07T17:46:37.7255026Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-07T17:46:37.7296757Z Entering 'third_party/gloo' 2024-08-07T17:46:37.7338875Z Entering 'third_party/googletest' 2024-08-07T17:46:37.7379820Z Entering 'third_party/ideep' 2024-08-07T17:46:37.7419162Z Entering 'third_party/ideep/mkl-dnn' 2024-08-07T17:46:37.7468171Z Entering 'third_party/ittapi' 2024-08-07T17:46:37.7508236Z Entering 'third_party/kineto' 2024-08-07T17:46:37.7550487Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T17:46:37.7593573Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T17:46:37.7636369Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T17:46:37.7676806Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T17:46:37.7717614Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T17:46:37.7756908Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T17:46:37.7799145Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T17:46:37.7840996Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T17:46:37.7881358Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T17:46:37.7923413Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T17:46:37.7965253Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T17:46:37.8004805Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T17:46:37.8046944Z Entering 'third_party/mimalloc' 2024-08-07T17:46:37.8088175Z Entering 'third_party/nccl/nccl' 2024-08-07T17:46:37.8129578Z Entering 'third_party/nlohmann' 2024-08-07T17:46:37.8172618Z Entering 'third_party/onnx' 2024-08-07T17:46:37.8231540Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-07T17:46:37.8273118Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-07T17:46:37.8316346Z Entering 'third_party/opentelemetry-cpp' 2024-08-07T17:46:37.8360588Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T17:46:37.8401621Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T17:46:37.8442013Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T17:46:37.8481299Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T17:46:37.8523043Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T17:46:37.8562737Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T17:46:37.8602878Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T17:46:37.8641948Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T17:46:37.8684745Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T17:46:37.8726452Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T17:46:37.8788345Z Entering 'third_party/pocketfft' 2024-08-07T17:46:37.8830446Z Entering 'third_party/protobuf' 2024-08-07T17:46:37.8874741Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-07T17:46:37.8914967Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-07T17:46:37.8957242Z Entering 'third_party/psimd' 2024-08-07T17:46:37.8998814Z Entering 'third_party/pthreadpool' 2024-08-07T17:46:37.9041599Z Entering 'third_party/pybind11' 2024-08-07T17:46:37.9083146Z Entering 'third_party/python-peachpy' 2024-08-07T17:46:37.9124207Z Entering 'third_party/sleef' 2024-08-07T17:46:37.9165631Z Entering 'third_party/tensorpipe' 2024-08-07T17:46:37.9207493Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-07T17:46:37.9248777Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-07T17:46:37.9289296Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-07T17:46:37.9329531Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T17:46:37.9368953Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T17:46:37.9422208Z ##[endgroup] 2024-08-07T17:46:37.9465388Z [command]/usr/bin/git log -1 --format='%H' 2024-08-07T17:46:37.9492535Z '016588f53c6904b840aa56aa86f95460b4d9c996' 2024-08-07T17:46:37.9673024Z Prepare all required actions 2024-08-07T17:46:37.9673868Z Getting action download info 2024-08-07T17:46:38.1440713Z ##[group]Run ./.github/actions/setup-linux 2024-08-07T17:46:38.1441141Z env: 2024-08-07T17:46:38.1441420Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:38.1441766Z ##[endgroup] 2024-08-07T17:46:38.1522225Z ##[group]Run set -euo pipefail 2024-08-07T17:46:38.1522660Z set -euo pipefail 2024-08-07T17:46:38.1523040Z function get_ec2_metadata() { 2024-08-07T17:46:38.1523504Z  # Pulled from instance metadata endpoint for EC2 2024-08-07T17:46:38.1524275Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-07T17:46:38.1524966Z  category=$1 2024-08-07T17:46:38.1525410Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-07T17:46:38.1525962Z  runner_name_str=i-0dcb26b5a502300df 2024-08-07T17:46:38.1526456Z  if [[ -f /.inarc ]]; then 2024-08-07T17:46:38.1526891Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-07T17:46:38.1527405Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-07T17:46:38.1527988Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-07T17:46:38.1528502Z  else 2024-08-07T17:46:38.1528936Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-07T17:46:38.1529457Z  fi 2024-08-07T17:46:38.1529733Z } 2024-08-07T17:46:38.1530081Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-07T17:46:38.1530619Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-07T17:46:38.1531199Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-07T17:46:38.1531718Z echo "system info $(uname -a)" 2024-08-07T17:46:38.1539820Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:46:38.1540282Z env: 2024-08-07T17:46:38.1540589Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:38.1540947Z ##[endgroup] 2024-08-07T17:46:38.1641067Z ami-id: ami-0ce0c36d7a00b20e2 2024-08-07T17:46:38.1708155Z instance-id: i-0dcb26b5a502300df 2024-08-07T17:46:38.1772715Z instance-type: g3.8xlarge 2024-08-07T17:46:38.1782334Z system info Linux ip-10-0-1-17.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-07T17:46:38.1817888Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-07T17:46:38.1818687Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-07T17:46:38.1826677Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:46:38.1827503Z env: 2024-08-07T17:46:38.1827787Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:38.1828139Z ##[endgroup] 2024-08-07T17:46:38.1910853Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-07T17:46:38.1911429Z if systemctl is-active --quiet docker; then 2024-08-07T17:46:38.1911917Z  echo "Docker daemon is running..."; 2024-08-07T17:46:38.1912345Z else 2024-08-07T17:46:38.1912796Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-07T17:46:38.1913308Z fi 2024-08-07T17:46:38.1920367Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:46:38.1920851Z env: 2024-08-07T17:46:38.1921133Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:38.1921482Z ##[endgroup] 2024-08-07T17:46:38.1967814Z Docker daemon is running... 2024-08-07T17:46:38.2021303Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-07T17:46:38.2021880Z with: 2024-08-07T17:46:38.2022170Z shell: bash 2024-08-07T17:46:38.2022473Z timeout_minutes: 5 2024-08-07T17:46:38.2022782Z max_attempts: 3 2024-08-07T17:46:38.2023103Z retry_wait_seconds: 30 2024-08-07T17:46:38.2024542Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-08-07T17:46:38.2025981Z polling_interval_seconds: 1 2024-08-07T17:46:38.2026347Z warning_on_retry: true 2024-08-07T17:46:38.2026697Z continue_on_error: false 2024-08-07T17:46:38.2027362Z env: 2024-08-07T17:46:38.2027639Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:38.2027990Z AWS_RETRY_MODE: standard 2024-08-07T17:46:38.2028335Z AWS_MAX_ATTEMPTS: 5 2024-08-07T17:46:38.2028658Z AWS_DEFAULT_REGION: us-east-1 2024-08-07T17:46:38.2029012Z ##[endgroup] 2024-08-07T17:46:39.2500954Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-07T17:46:39.2501721Z Configure a credential helper to remove this warning. See 2024-08-07T17:46:39.2502439Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-07T17:46:39.2502914Z 2024-08-07T17:46:39.2503088Z Login Succeeded 2024-08-07T17:46:39.2745643Z Command completed after 1 attempt(s). 2024-08-07T17:46:39.2808916Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-07T17:46:39.2809584Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-07T17:46:39.2810142Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-07T17:46:39.2818514Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:46:39.2818999Z env: 2024-08-07T17:46:39.2819291Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:39.2819622Z ##[endgroup] 2024-08-07T17:46:39.2904479Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-07T17:46:39.2905195Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-07T17:46:39.2905714Z # shellcheck disable=SC2046 2024-08-07T17:46:39.2906149Z docker stop $(docker ps -q) || true 2024-08-07T17:46:39.2906611Z # Prune all of the docker images 2024-08-07T17:46:39.2907587Z docker system prune -af 2024-08-07T17:46:39.2915589Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:46:39.2916077Z env: 2024-08-07T17:46:39.2916358Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:39.2916707Z ##[endgroup] 2024-08-07T17:46:39.3254988Z "docker stop" requires at least 1 argument. 2024-08-07T17:46:39.3255459Z See 'docker stop --help'. 2024-08-07T17:46:39.3255690Z 2024-08-07T17:46:39.3255911Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-07T17:46:39.3256267Z 2024-08-07T17:46:39.3256422Z Stop one or more running containers 2024-08-07T17:46:39.3438813Z Total reclaimed space: 0B 2024-08-07T17:46:39.3477718Z ##[group]Run set +e 2024-08-07T17:46:39.3478096Z set +e 2024-08-07T17:46:39.3478415Z set -x 2024-08-07T17:46:39.3478703Z  2024-08-07T17:46:39.3479030Z PT_DOMAIN=download.pytorch.org 2024-08-07T17:46:39.3479763Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-07T17:46:39.3480677Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-07T17:46:39.3481329Z # one is returned at random 2024-08-07T17:46:39.3481835Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-07T17:46:39.3482304Z  2024-08-07T17:46:39.3482623Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-07T17:46:39.3483175Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-07T17:46:39.3483835Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-07T17:46:39.3484318Z  2024-08-07T17:46:39.3484634Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-07T17:46:39.3485129Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-07T17:46:39.3485575Z  exit 1 2024-08-07T17:46:39.3485889Z  fi 2024-08-07T17:46:39.3486161Z fi 2024-08-07T17:46:39.3486642Z  2024-08-07T17:46:39.3487126Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-07T17:46:39.3487577Z  # Clean up any old records first 2024-08-07T17:46:39.3488042Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-07T17:46:39.3488465Z fi 2024-08-07T17:46:39.3488734Z  2024-08-07T17:46:39.3489136Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-07T17:46:39.3489639Z cat /etc/hosts 2024-08-07T17:46:39.3497028Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:46:39.3497512Z env: 2024-08-07T17:46:39.3497799Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:39.3498126Z ##[endgroup] 2024-08-07T17:46:39.3521789Z + PT_DOMAIN=download.pytorch.org 2024-08-07T17:46:39.3527257Z ++ dig -4 +short download.pytorch.org 2024-08-07T17:46:39.3527811Z ++ tail -n1 2024-08-07T17:46:39.3965033Z + RESOLVED_IP=18.160.10.22 2024-08-07T17:46:39.3965408Z + '[' -z 18.160.10.22 ']' 2024-08-07T17:46:39.3965814Z + grep -r download.pytorch.org /etc/hosts 2024-08-07T17:46:39.3974299Z 18.160.10.28 download.pytorch.org 2024-08-07T17:46:39.3975133Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-08-07T17:46:39.4096841Z + echo '18.160.10.22 download.pytorch.org' 2024-08-07T17:46:39.4097373Z + sudo tee -a /etc/hosts 2024-08-07T17:46:39.4189227Z 18.160.10.22 download.pytorch.org 2024-08-07T17:46:39.4205169Z + cat /etc/hosts 2024-08-07T17:46:39.4210971Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-07T17:46:39.4216776Z ::1 localhost6 localhost6.localdomain6 2024-08-07T17:46:39.4217260Z 18.160.10.22 download.pytorch.org 2024-08-07T17:46:39.4386421Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-07T17:46:39.4387535Z with: 2024-08-07T17:46:39.4388448Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:39.4389445Z docker-build-dir: .ci/docker 2024-08-07T17:46:39.4389846Z working-directory: . 2024-08-07T17:46:39.4390283Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:39.4390796Z force-push: false 2024-08-07T17:46:39.4391113Z env: 2024-08-07T17:46:39.4391388Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:39.4391739Z ##[endgroup] 2024-08-07T17:46:39.4418725Z ##[group]Run set -ex 2024-08-07T17:46:39.4419119Z set -ex 2024-08-07T17:46:39.4419431Z  2024-08-07T17:46:39.4419930Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-07T17:46:39.4420790Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-07T17:46:39.4421474Z # job could then download the pre-built image as usual 2024-08-07T17:46:39.4422181Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-07T17:46:39.4422772Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-07T17:46:39.4423350Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-07T17:46:39.4423846Z  2024-08-07T17:46:39.4424297Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-07T17:46:39.4424847Z  exit 0 2024-08-07T17:46:39.4425131Z else 2024-08-07T17:46:39.4425480Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-07T17:46:39.4425901Z fi 2024-08-07T17:46:39.4426167Z  2024-08-07T17:46:39.4426603Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-07T17:46:39.4427959Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-07T17:46:39.4428608Z  # use it as it is, but first let's extract the tag 2024-08-07T17:46:39.4429213Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-07T17:46:39.4429850Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-07T17:46:39.4430641Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-07T17:46:39.4431158Z else 2024-08-07T17:46:39.4431570Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-07T17:46:39.4432128Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-07T17:46:39.4432908Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-07T17:46:39.4433596Z fi 2024-08-07T17:46:39.4441098Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:46:39.4441566Z env: 2024-08-07T17:46:39.4441862Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:39.4442202Z REPO_NAME: pytorch 2024-08-07T17:46:39.4443123Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:39.4444107Z DOCKER_BUILD_DIR: .ci/docker 2024-08-07T17:46:39.4444594Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:39.4445077Z ##[endgroup] 2024-08-07T17:46:39.4469176Z + [[ ! -d .ci/docker ]] 2024-08-07T17:46:39.4469554Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-07T17:46:39.4469911Z + echo skip=false 2024-08-07T17:46:39.4471104Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-07T17:46:39.4476611Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:39.4477590Z ++ awk -F '[:,]' '{print $2}' 2024-08-07T17:46:39.4487432Z + DOCKER_TAG=02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:39.4487996Z + echo docker-tag=02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:39.4489322Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:39.4526126Z ##[group]Run set +e 2024-08-07T17:46:39.4526537Z set +e 2024-08-07T17:46:39.4526830Z set -x 2024-08-07T17:46:39.4527128Z  2024-08-07T17:46:39.4527417Z login() { 2024-08-07T17:46:39.4527998Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-07T17:46:39.4528640Z } 2024-08-07T17:46:39.4528929Z  2024-08-07T17:46:39.4529232Z retry () { 2024-08-07T17:46:39.4529577Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-07T17:46:39.4529988Z } 2024-08-07T17:46:39.4530273Z  2024-08-07T17:46:39.4530569Z retry login "${DOCKER_REGISTRY}" 2024-08-07T17:46:39.4530970Z  2024-08-07T17:46:39.4531398Z # Check if image already exists, if it does then skip building it 2024-08-07T17:46:39.4532011Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-07T17:46:39.4532490Z  exit 0 2024-08-07T17:46:39.4532798Z fi 2024-08-07T17:46:39.4533065Z  2024-08-07T17:46:39.4533514Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-07T17:46:39.4534224Z # be empty. The default action would be to continue rebuild the image 2024-08-07T17:46:39.4534846Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-07T17:46:39.4535427Z  # if we're on the base branch then use the parent commit 2024-08-07T17:46:39.4535941Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-07T17:46:39.4536388Z else 2024-08-07T17:46:39.4536818Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-07T17:46:39.4537396Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-07T17:46:39.4537859Z fi 2024-08-07T17:46:39.4538329Z  2024-08-07T17:46:39.4538626Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-07T17:46:39.4539087Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-07T17:46:39.4539516Z  2024-08-07T17:46:39.4540074Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-07T17:46:39.4540744Z  exit 0 2024-08-07T17:46:39.4541045Z fi 2024-08-07T17:46:39.4541310Z  2024-08-07T17:46:39.4541706Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-07T17:46:39.4542546Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-07T17:46:39.4543240Z  exit 1 2024-08-07T17:46:39.4543541Z fi 2024-08-07T17:46:39.4543826Z  2024-08-07T17:46:39.4544268Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-07T17:46:39.4545072Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-07T17:46:39.4545794Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-07T17:46:39.4546615Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-07T17:46:39.4548054Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-07T17:46:39.4548632Z fi 2024-08-07T17:46:39.4548923Z  2024-08-07T17:46:39.4549394Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-07T17:46:39.4556636Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:46:39.4557120Z env: 2024-08-07T17:46:39.4557396Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:39.4557759Z DOCKER_BUILD_DIR: .ci/docker 2024-08-07T17:46:39.4558194Z BASE_REVISION: 6ce09a9bb33e4011761558032e2165ad7b49fb68 2024-08-07T17:46:39.4559223Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:39.4560249Z DOCKER_TAG: 02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:39.4560820Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:39.4561300Z ##[endgroup] 2024-08-07T17:46:39.4584202Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:39.4584770Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:39.4587416Z + aws ecr get-login-password --region us-east-1 2024-08-07T17:46:39.4589190Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:39.9421634Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-07T17:46:39.9422434Z Configure a credential helper to remove this warning. See 2024-08-07T17:46:39.9423124Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-07T17:46:39.9423654Z 2024-08-07T17:46:39.9423800Z Login Succeeded 2024-08-07T17:46:39.9435581Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:40.1727369Z { 2024-08-07T17:46:40.1727948Z "schemaVersion": 2, 2024-08-07T17:46:40.1728510Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-07T17:46:40.1729084Z "config": { 2024-08-07T17:46:40.1729564Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-07T17:46:40.1730126Z "size": 47454, 2024-08-07T17:46:40.1730694Z "digest": "sha256:8e4e19ae0377c273bc603b24bc49d406d6f755072d4534d040c86afbe761a129" 2024-08-07T17:46:40.1731304Z }, 2024-08-07T17:46:40.1731578Z "layers": [ 2024-08-07T17:46:40.1731875Z { 2024-08-07T17:46:40.1732430Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1733467Z "size": 28580681, 2024-08-07T17:46:40.1734255Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-07T17:46:40.1734941Z }, 2024-08-07T17:46:40.1735674Z { 2024-08-07T17:46:40.1736903Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1738015Z "size": 7943353, 2024-08-07T17:46:40.1738660Z "digest": "sha256:09d415c238d76b32a7ea4a6e6add9542db9a5641f7f183af70aae185d0709e58" 2024-08-07T17:46:40.1739639Z }, 2024-08-07T17:46:40.1740407Z { 2024-08-07T17:46:40.1740991Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1741624Z "size": 56226389, 2024-08-07T17:46:40.1742260Z "digest": "sha256:9fe6e2e61518cba6844870c03b285737daec35e62baf25ae7744629ed3a7b470" 2024-08-07T17:46:40.1742960Z }, 2024-08-07T17:46:40.1743329Z { 2024-08-07T17:46:40.1743827Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1773377Z "size": 187, 2024-08-07T17:46:40.1773943Z "digest": "sha256:41f16248e682693ff20b3032c1d5e5541cc87c5af898ae2ff9b24d2940e59100" 2024-08-07T17:46:40.1774611Z }, 2024-08-07T17:46:40.1774968Z { 2024-08-07T17:46:40.1775402Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1775919Z "size": 6885, 2024-08-07T17:46:40.1776447Z "digest": "sha256:95d7b781703928cf3c4eece39d800cccb76728c375fedf51ecd83833fb25e458" 2024-08-07T17:46:40.1777038Z }, 2024-08-07T17:46:40.1777286Z { 2024-08-07T17:46:40.1777709Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1778503Z "size": 1383662475, 2024-08-07T17:46:40.1779072Z "digest": "sha256:8f6c9048534734f4c873935293b7296225846ceb31c1a158400a67ea170dde7f" 2024-08-07T17:46:40.1779690Z }, 2024-08-07T17:46:40.1779961Z { 2024-08-07T17:46:40.1780385Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1780932Z "size": 62282, 2024-08-07T17:46:40.1781468Z "digest": "sha256:ab17245097e491b9368790714f9d90ed447bf0973bd677cfe6f2456d62b72a13" 2024-08-07T17:46:40.1782061Z }, 2024-08-07T17:46:40.1782331Z { 2024-08-07T17:46:40.1782758Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1783301Z "size": 1684, 2024-08-07T17:46:40.1783865Z "digest": "sha256:dfecd7e9912b76ed460b8edd5a85f1943666e38a973ab5458177cf2c7c3110e3" 2024-08-07T17:46:40.1784494Z }, 2024-08-07T17:46:40.1784749Z { 2024-08-07T17:46:40.1785192Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1785752Z "size": 1522, 2024-08-07T17:46:40.1786289Z "digest": "sha256:464a8f74544589bf7b57f9a4cadcb6681e5ed00758f6c35025e691df4e88e890" 2024-08-07T17:46:40.1787365Z }, 2024-08-07T17:46:40.1787729Z { 2024-08-07T17:46:40.1788162Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1788703Z "size": 2445884909, 2024-08-07T17:46:40.1789214Z "digest": "sha256:d67c111fa58886d8904335b5122039816c8e51af5844cb6fe4e14154d9a7c016" 2024-08-07T17:46:40.1789797Z }, 2024-08-07T17:46:40.1790063Z { 2024-08-07T17:46:40.1790425Z + exit 0 2024-08-07T17:46:40.1790857Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1791386Z "size": 85882, 2024-08-07T17:46:40.1791889Z "digest": "sha256:0723abde19c09431db80b5d0f0e003541f6232c2da5c4f35c635c861875937ae" 2024-08-07T17:46:40.1792469Z }, 2024-08-07T17:46:40.1792731Z { 2024-08-07T17:46:40.1793132Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1793653Z "size": 1820, 2024-08-07T17:46:40.1794180Z "digest": "sha256:ceceb076946ee95300822daccb5d3e4e45e6857d34996ef89305d9a078410f8e" 2024-08-07T17:46:40.1794748Z }, 2024-08-07T17:46:40.1795010Z { 2024-08-07T17:46:40.1795427Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1795929Z "size": 654860881, 2024-08-07T17:46:40.1796457Z "digest": "sha256:889eb8142c5187639e02cdc07adc75024843fd85a1f5c77d24c1a1123f5246bb" 2024-08-07T17:46:40.1797193Z }, 2024-08-07T17:46:40.1797443Z { 2024-08-07T17:46:40.1797866Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1798393Z "size": 542, 2024-08-07T17:46:40.1798905Z "digest": "sha256:af8081143f008c5cb1bd61f6d92fde22553cfc007b6a77c2fd2aaf6a9f0088a9" 2024-08-07T17:46:40.1799493Z }, 2024-08-07T17:46:40.1799757Z { 2024-08-07T17:46:40.1800166Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1800728Z "size": 1284, 2024-08-07T17:46:40.1801295Z "digest": "sha256:cc64a79090ad4059e2f3dbc1949b010b2733ab614bc9518ff6256ce2bed52115" 2024-08-07T17:46:40.1801900Z }, 2024-08-07T17:46:40.1802169Z { 2024-08-07T17:46:40.1802613Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1803155Z "size": 483, 2024-08-07T17:46:40.1803676Z "digest": "sha256:ad26542e0d01609840496a12691483f2c50eb8e23002a205becb5b695f9159fb" 2024-08-07T17:46:40.1804302Z }, 2024-08-07T17:46:40.1804559Z { 2024-08-07T17:46:40.1804999Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1805558Z "size": 91711921, 2024-08-07T17:46:40.1806109Z "digest": "sha256:7b42d8ae23671f21f6523a7e71cc50d5f87c4e993027a9bd924985cda6b57de9" 2024-08-07T17:46:40.1806729Z }, 2024-08-07T17:46:40.1807006Z { 2024-08-07T17:46:40.1807433Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1807992Z "size": 3229, 2024-08-07T17:46:40.1808644Z "digest": "sha256:6d46e3b97d9fd2bd0e7ceec44e4884756b26c594a1c1c51d1426ddd13f749db5" 2024-08-07T17:46:40.1809298Z }, 2024-08-07T17:46:40.1809567Z { 2024-08-07T17:46:40.1809989Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1810549Z "size": 1907, 2024-08-07T17:46:40.1811101Z "digest": "sha256:91f9dddffbee46981d9e74382704c1b53c87f1808350cf1dd518687805d3aa59" 2024-08-07T17:46:40.1811711Z }, 2024-08-07T17:46:40.1811987Z { 2024-08-07T17:46:40.1812423Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1812964Z "size": 699, 2024-08-07T17:46:40.1813524Z "digest": "sha256:03fb78f207e3cbcfedfc7619a1d60fa576d3293d508a7e97dc09a5e7a9c65ab0" 2024-08-07T17:46:40.1814161Z }, 2024-08-07T17:46:40.1814413Z { 2024-08-07T17:46:40.1814856Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1815419Z "size": 2813413845, 2024-08-07T17:46:40.1815995Z "digest": "sha256:eeedca475f8cafec096dfa0abc5c6f08938eb47c5d9d96fb001fd7f13a2531ca" 2024-08-07T17:46:40.1816649Z }, 2024-08-07T17:46:40.1816919Z { 2024-08-07T17:46:40.1817341Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1817900Z "size": 378, 2024-08-07T17:46:40.1818451Z "digest": "sha256:202c71391ade27762d5e9cafde793db67b4d5e4d6869debe2643d5092101c461" 2024-08-07T17:46:40.1819068Z }, 2024-08-07T17:46:40.1819336Z { 2024-08-07T17:46:40.1819787Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1820332Z "size": 12893, 2024-08-07T17:46:40.1820892Z "digest": "sha256:5af882ef27a2db3ee5e89c8730de161c0718b951f3f8477e63fe21517c9167f7" 2024-08-07T17:46:40.1821523Z }, 2024-08-07T17:46:40.1821782Z { 2024-08-07T17:46:40.1822265Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1822813Z "size": 802, 2024-08-07T17:46:40.1823327Z "digest": "sha256:bac001ac2fee5df3367ff9332bc1fcd706a8de3ad05a1885903f338197619520" 2024-08-07T17:46:40.1823926Z }, 2024-08-07T17:46:40.1824192Z { 2024-08-07T17:46:40.1824610Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1825138Z "size": 106, 2024-08-07T17:46:40.1825657Z "digest": "sha256:05896f1d52f7a8e2c4a53777e5575da3dc8fca8a786d14075b5bb204bcb4cf04" 2024-08-07T17:46:40.1826226Z }, 2024-08-07T17:46:40.1826489Z { 2024-08-07T17:46:40.1827312Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1828062Z "size": 502, 2024-08-07T17:46:40.1828575Z "digest": "sha256:49a0e418a175ac12f1b4b503e80e22679bc321e6163ed5b203af848204f0606a" 2024-08-07T17:46:40.1829137Z }, 2024-08-07T17:46:40.1829401Z { 2024-08-07T17:46:40.1829825Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1830337Z "size": 121477364, 2024-08-07T17:46:40.1830865Z "digest": "sha256:b5632121ff7a146647683888294bcca31252dd90ca681099dfb81d75c4710d52" 2024-08-07T17:46:40.1831440Z }, 2024-08-07T17:46:40.1831681Z { 2024-08-07T17:46:40.1832098Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1832623Z "size": 109, 2024-08-07T17:46:40.1833107Z "digest": "sha256:deceb46806b535404e025b9b532009eb211e79f85021171997af825253327d1a" 2024-08-07T17:46:40.1833671Z }, 2024-08-07T17:46:40.1833932Z { 2024-08-07T17:46:40.1834328Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1834854Z "size": 487, 2024-08-07T17:46:40.1835365Z "digest": "sha256:a2630e5fe166ae8ff146e4c14db0bdc1299ed2b6120108351c961adcd2375288" 2024-08-07T17:46:40.1835932Z }, 2024-08-07T17:46:40.1836189Z { 2024-08-07T17:46:40.1836604Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1837106Z "size": 297, 2024-08-07T17:46:40.1837615Z "digest": "sha256:603261fb379fff6cf0eb63de1787e7c98932f81e44e6af3f1645c9a977e0ca8c" 2024-08-07T17:46:40.1838194Z }, 2024-08-07T17:46:40.1838439Z { 2024-08-07T17:46:40.1838983Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1839531Z "size": 103, 2024-08-07T17:46:40.1840025Z "digest": "sha256:d8ac71337d62da7c4624f267380da27d834fcf174b873c8c53a963e6cc7193a4" 2024-08-07T17:46:40.1840599Z }, 2024-08-07T17:46:40.1840861Z { 2024-08-07T17:46:40.1841260Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1841781Z "size": 1470, 2024-08-07T17:46:40.1842310Z "digest": "sha256:2c2e35b979f5fc0ecccef6efc157a97f33db01a77958fa33855dc367544a5c46" 2024-08-07T17:46:40.1842884Z }, 2024-08-07T17:46:40.1843144Z { 2024-08-07T17:46:40.1843545Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1844075Z "size": 424146101, 2024-08-07T17:46:40.1844610Z "digest": "sha256:e559222c0ec62d839f78e4215ae553de028be354b7031e7393d1c6be6e91fd70" 2024-08-07T17:46:40.1845172Z }, 2024-08-07T17:46:40.1845434Z { 2024-08-07T17:46:40.1845852Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1846362Z "size": 159, 2024-08-07T17:46:40.1846883Z "digest": "sha256:c4dcb7e9d189eca2e414c757b5cf6d158055b29b0e71dbd082a5fb3585dba665" 2024-08-07T17:46:40.1847474Z }, 2024-08-07T17:46:40.1847722Z { 2024-08-07T17:46:40.1848143Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1848670Z "size": 563, 2024-08-07T17:46:40.1849168Z "digest": "sha256:2b9731d295d60278e361d491fc274e5d558a89ec5adeb9fdf342d7a8cc17bfec" 2024-08-07T17:46:40.1849755Z }, 2024-08-07T17:46:40.1850022Z { 2024-08-07T17:46:40.1850428Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1850953Z "size": 35807525, 2024-08-07T17:46:40.1851478Z "digest": "sha256:19da43312ebac2831028913760327e53d9812e80ab6b806cd7a920c549a3b554" 2024-08-07T17:46:40.1852032Z }, 2024-08-07T17:46:40.1852302Z { 2024-08-07T17:46:40.1852721Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1853228Z "size": 104, 2024-08-07T17:46:40.1853742Z "digest": "sha256:b846263d066405189d21552b7288e3efbc47e6cd2f236aa39f39d88ca576a33a" 2024-08-07T17:46:40.1854312Z }, 2024-08-07T17:46:40.1854554Z { 2024-08-07T17:46:40.1854982Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1855502Z "size": 424, 2024-08-07T17:46:40.1856001Z "digest": "sha256:3886e8f57dc887a87bb710fbda080ef4fd521f1ed2e560a895b000c05f6e9b0a" 2024-08-07T17:46:40.1856687Z }, 2024-08-07T17:46:40.1856951Z { 2024-08-07T17:46:40.1857347Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1857869Z "size": 20262048, 2024-08-07T17:46:40.1858392Z "digest": "sha256:58e13e7b6d7b78400d84805c843f08fb8f1d6583bccdb3373c8122b317e4d15a" 2024-08-07T17:46:40.1858948Z }, 2024-08-07T17:46:40.1859208Z { 2024-08-07T17:46:40.1859625Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1860129Z "size": 438, 2024-08-07T17:46:40.1860652Z "digest": "sha256:0eb80a0ac0006a45aa6b4581263cc1613ddf9545bd4a1d5a544310efe898ad81" 2024-08-07T17:46:40.1861236Z }, 2024-08-07T17:46:40.1861479Z { 2024-08-07T17:46:40.1861894Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1862398Z "size": 699, 2024-08-07T17:46:40.1862918Z "digest": "sha256:03fb78f207e3cbcfedfc7619a1d60fa576d3293d508a7e97dc09a5e7a9c65ab0" 2024-08-07T17:46:40.1863512Z }, 2024-08-07T17:46:40.1863758Z { 2024-08-07T17:46:40.1864174Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1864707Z "size": 141, 2024-08-07T17:46:40.1865205Z "digest": "sha256:a9f11b2b3622f5530c5cfefe973e96e028323fab1ff02e7a6337eb86fadebeb5" 2024-08-07T17:46:40.1865799Z }, 2024-08-07T17:46:40.1866063Z { 2024-08-07T17:46:40.1866462Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1867490Z "size": 135, 2024-08-07T17:46:40.1868146Z "digest": "sha256:d7e94279f8b2fe549e3f23637275d9dca8c831563ce84f1d44117eb297ddc871" 2024-08-07T17:46:40.1868727Z }, 2024-08-07T17:46:40.1868993Z { 2024-08-07T17:46:40.1869413Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1869922Z "size": 32, 2024-08-07T17:46:40.1870442Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-07T17:46:40.1871027Z }, 2024-08-07T17:46:40.1871274Z { 2024-08-07T17:46:40.1871703Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1872228Z "size": 190, 2024-08-07T17:46:40.1872731Z "digest": "sha256:0a3560fcd26c387cb3ff0638718fa9da02d3b08ad75b23c434b06d6a6d21cb55" 2024-08-07T17:46:40.1873319Z }, 2024-08-07T17:46:40.1873584Z { 2024-08-07T17:46:40.1873986Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1874515Z "size": 562, 2024-08-07T17:46:40.1875029Z "digest": "sha256:adf9d181cfbca5b0544f23797995d4ea6500cc4a635db9f7e18954c9044e9f4a" 2024-08-07T17:46:40.1875596Z }, 2024-08-07T17:46:40.1875867Z { 2024-08-07T17:46:40.1876289Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1876792Z "size": 43163050, 2024-08-07T17:46:40.1877317Z "digest": "sha256:bc18dd8315979255e24cf91e83757510baeb22f29a64d7579b02aca642215b58" 2024-08-07T17:46:40.1877889Z }, 2024-08-07T17:46:40.1878133Z { 2024-08-07T17:46:40.1878551Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1879061Z "size": 106, 2024-08-07T17:46:40.1879575Z "digest": "sha256:d62f2a6086195c833ef952a43eed18216c6cde79b07f8deeb4fb3aeea648f868" 2024-08-07T17:46:40.1880162Z }, 2024-08-07T17:46:40.1880403Z { 2024-08-07T17:46:40.1880820Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1881383Z "size": 1210, 2024-08-07T17:46:40.1881893Z "digest": "sha256:719a1c2abcfd7e4113aa2f03adeb54350a5e381c720688ea53bc0e18401ad33d" 2024-08-07T17:46:40.1882481Z }, 2024-08-07T17:46:40.1882745Z { 2024-08-07T17:46:40.1883153Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1883674Z "size": 699, 2024-08-07T17:46:40.1884193Z "digest": "sha256:03fb78f207e3cbcfedfc7619a1d60fa576d3293d508a7e97dc09a5e7a9c65ab0" 2024-08-07T17:46:40.1884766Z }, 2024-08-07T17:46:40.1885027Z { 2024-08-07T17:46:40.1885441Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1886065Z "size": 138, 2024-08-07T17:46:40.1886576Z "digest": "sha256:8e1355d516c5e0527c25402c664fc648a8a2b814090e6ee0fdd60f49a8826ac1" 2024-08-07T17:46:40.1887151Z }, 2024-08-07T17:46:40.1887394Z { 2024-08-07T17:46:40.1887815Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1888339Z "size": 117, 2024-08-07T17:46:40.1888825Z "digest": "sha256:701898e66f54fcf3b57d837ab651a876666391d9f9b51b9969ec842a23fd86f0" 2024-08-07T17:46:40.1889402Z }, 2024-08-07T17:46:40.1889664Z { 2024-08-07T17:46:40.1890070Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1890598Z "size": 1916643487, 2024-08-07T17:46:40.1891132Z "digest": "sha256:129b38cfed12360bb7c4d882141b5fb7f7337791383b9f75e4ffed521548f060" 2024-08-07T17:46:40.1891695Z }, 2024-08-07T17:46:40.1891957Z { 2024-08-07T17:46:40.1892375Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1892929Z "size": 172, 2024-08-07T17:46:40.1893470Z "digest": "sha256:1751651a072f518b34289de9dfa631c4e82c7958a5331061379684f9b9b11720" 2024-08-07T17:46:40.1894086Z }, 2024-08-07T17:46:40.1894341Z { 2024-08-07T17:46:40.1894780Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1895344Z "size": 905, 2024-08-07T17:46:40.1895890Z "digest": "sha256:fc956bf4e5b49119cab7cc7bda867c61bed76c76c1e113dfa7c7b184251f8173" 2024-08-07T17:46:40.1896528Z }, 2024-08-07T17:46:40.1896800Z { 2024-08-07T17:46:40.1897228Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1897923Z "size": 699, 2024-08-07T17:46:40.1898483Z "digest": "sha256:03fb78f207e3cbcfedfc7619a1d60fa576d3293d508a7e97dc09a5e7a9c65ab0" 2024-08-07T17:46:40.1899110Z }, 2024-08-07T17:46:40.1899382Z { 2024-08-07T17:46:40.1899805Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1900363Z "size": 134, 2024-08-07T17:46:40.1900925Z "digest": "sha256:874491333fb5b81d9cbbd71520163a6baf07e6f011dc148b1634b94e301972b8" 2024-08-07T17:46:40.1901530Z }, 2024-08-07T17:46:40.1901803Z { 2024-08-07T17:46:40.1902247Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1902788Z "size": 32, 2024-08-07T17:46:40.1903341Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-07T17:46:40.1903974Z }, 2024-08-07T17:46:40.1904227Z { 2024-08-07T17:46:40.1904669Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1905234Z "size": 156, 2024-08-07T17:46:40.1905792Z "digest": "sha256:365b4a0eec21fb9dfc58baf7b43a229ae3f8b638b17e9b4db5ee1cfaecb82de4" 2024-08-07T17:46:40.1906439Z }, 2024-08-07T17:46:40.1906712Z { 2024-08-07T17:46:40.1907597Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1908148Z "size": 1839, 2024-08-07T17:46:40.1908675Z "digest": "sha256:c1d4cac78640fc7d364011da807a363d6b248a2c46ccf99a6e1488c4974c17e3" 2024-08-07T17:46:40.1909256Z }, 2024-08-07T17:46:40.1909521Z { 2024-08-07T17:46:40.1909946Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1910456Z "size": 7529788, 2024-08-07T17:46:40.1910975Z "digest": "sha256:29afe2181ca6174db390fd0d9e8c4b8675fb99bb961c93936d4080ae92c8af57" 2024-08-07T17:46:40.1911565Z }, 2024-08-07T17:46:40.1911811Z { 2024-08-07T17:46:40.1912226Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1912756Z "size": 161, 2024-08-07T17:46:40.1913253Z "digest": "sha256:190dccfa687dc1e698f27965d8065a183168d7c66361505f368765456ff9f4bc" 2024-08-07T17:46:40.1913828Z }, 2024-08-07T17:46:40.1914094Z { 2024-08-07T17:46:40.1914495Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1915016Z "size": 7942, 2024-08-07T17:46:40.1915543Z "digest": "sha256:b01bea4b6634f47c51ede2b4a95ceade4e61356ca29551168b493ae6d4492f44" 2024-08-07T17:46:40.1916255Z }, 2024-08-07T17:46:40.1916517Z { 2024-08-07T17:46:40.1916919Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1917441Z "size": 8066, 2024-08-07T17:46:40.1917963Z "digest": "sha256:f13a9cc5968861df73c96e4ad67eef7f58576eeba8cdbee3d39e42ed3d6fe96d" 2024-08-07T17:46:40.1918536Z }, 2024-08-07T17:46:40.1918798Z { 2024-08-07T17:46:40.1919213Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1919723Z "size": 302, 2024-08-07T17:46:40.1920244Z "digest": "sha256:96d6dcd01b9f6c5557ba405d028442d808dd329f4f93e75f2c3ba0554e703a52" 2024-08-07T17:46:40.1920825Z }, 2024-08-07T17:46:40.1921066Z { 2024-08-07T17:46:40.1921485Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1922053Z "size": 7629762, 2024-08-07T17:46:40.1922562Z "digest": "sha256:fe134554f1b8a0509798c1f49e6f973f9b8b7f954eb1fda146f35a37c9c71599" 2024-08-07T17:46:40.1923149Z }, 2024-08-07T17:46:40.1923427Z { 2024-08-07T17:46:40.1923853Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1924415Z "size": 108, 2024-08-07T17:46:40.1924964Z "digest": "sha256:8068d364963fe87857fbb7c3aad42ddee962351a57eddc01b4282be12c69a074" 2024-08-07T17:46:40.1925571Z }, 2024-08-07T17:46:40.1925836Z { 2024-08-07T17:46:40.1926272Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1926814Z "size": 54145777, 2024-08-07T17:46:40.1927377Z "digest": "sha256:f307f545c1a5e086f18e23247c890ddd5b135b1cb7157807aeeaef38ceb3b7dc" 2024-08-07T17:46:40.1928109Z }, 2024-08-07T17:46:40.1928374Z { 2024-08-07T17:46:40.1928818Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1929367Z "size": 471, 2024-08-07T17:46:40.1929894Z "digest": "sha256:51c0b7eab50584e37e08f303a4bf3ba3141998800971452b2024a8b897858877" 2024-08-07T17:46:40.1930512Z }, 2024-08-07T17:46:40.1930774Z { 2024-08-07T17:46:40.1931204Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1931763Z "size": 1373229322, 2024-08-07T17:46:40.1932333Z "digest": "sha256:60aba96f31c695ec1e1433578783bab1257233794a6e8b5cc5c6365fdf68669e" 2024-08-07T17:46:40.1932939Z }, 2024-08-07T17:46:40.1933214Z { 2024-08-07T17:46:40.1933638Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1934197Z "size": 106, 2024-08-07T17:46:40.1934754Z "digest": "sha256:46db894a106ccd6c28198b9236ee84ac2973b37ae608caeb380dfaef91f13961" 2024-08-07T17:46:40.1935368Z }, 2024-08-07T17:46:40.1935647Z { 2024-08-07T17:46:40.1936092Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1936635Z "size": 557, 2024-08-07T17:46:40.1937173Z "digest": "sha256:0052583728e40920c255902250fe4a615b10ff00a5c0e7ed51a0075121d42fad" 2024-08-07T17:46:40.1937785Z }, 2024-08-07T17:46:40.1938037Z { 2024-08-07T17:46:40.1938479Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1939047Z "size": 36532090, 2024-08-07T17:46:40.1939612Z "digest": "sha256:8c6d352a3b51edbb546310cc3d97ded6aa8fd29a7d610e4d13fc54b5828fdd05" 2024-08-07T17:46:40.1940246Z }, 2024-08-07T17:46:40.1940518Z { 2024-08-07T17:46:40.1940944Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1941507Z "size": 111, 2024-08-07T17:46:40.1942073Z "digest": "sha256:380c4e16278f4e3bf0eb6aee3ff4a730e3a1adb82aeb1cfb8e4114f1759a2bcf" 2024-08-07T17:46:40.1942695Z }, 2024-08-07T17:46:40.1942966Z { 2024-08-07T17:46:40.1943417Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1943963Z "size": 32, 2024-08-07T17:46:40.1944520Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-07T17:46:40.1945155Z }, 2024-08-07T17:46:40.1945405Z { 2024-08-07T17:46:40.1945848Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1946514Z "size": 32, 2024-08-07T17:46:40.1947466Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-07T17:46:40.1948092Z }, 2024-08-07T17:46:40.1948358Z { 2024-08-07T17:46:40.1948767Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1949297Z "size": 32, 2024-08-07T17:46:40.1949820Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-07T17:46:40.1950393Z }, 2024-08-07T17:46:40.1950649Z { 2024-08-07T17:46:40.1951083Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:46:40.1951593Z "size": 32, 2024-08-07T17:46:40.1952109Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-07T17:46:40.1952694Z } 2024-08-07T17:46:40.1952942Z ] 2024-08-07T17:46:40.1953201Z } 2024-08-07T17:46:40.2036219Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-07T17:46:40.2036820Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-07T17:46:40.2037368Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-07T17:46:40.2045421Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:46:40.2045891Z env: 2024-08-07T17:46:40.2046190Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:40.2047133Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:40.2048095Z ##[endgroup] 2024-08-07T17:46:40.2074690Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9-02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:40.2132248Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-07T17:46:40.2132811Z with: 2024-08-07T17:46:40.2133680Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:40.2134761Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:40.2135234Z env: 2024-08-07T17:46:40.2135522Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:40.2135868Z ##[endgroup] 2024-08-07T17:46:40.2164691Z ##[group]Run set -x 2024-08-07T17:46:40.2165069Z set -x 2024-08-07T17:46:40.2165362Z set +e 2024-08-07T17:46:40.2165668Z  2024-08-07T17:46:40.2165958Z login() { 2024-08-07T17:46:40.2166535Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-07T17:46:40.2167176Z } 2024-08-07T17:46:40.2167464Z  2024-08-07T17:46:40.2167799Z retry () { 2024-08-07T17:46:40.2168148Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-07T17:46:40.2168563Z } 2024-08-07T17:46:40.2168847Z  2024-08-07T17:46:40.2169147Z retry login "${DOCKER_REGISTRY}" 2024-08-07T17:46:40.2169551Z  2024-08-07T17:46:40.2169832Z set -e 2024-08-07T17:46:40.2170259Z # ignore output since only exit code is used for conditional 2024-08-07T17:46:40.2170877Z # only pull docker image if it's not available locally 2024-08-07T17:46:40.2171552Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-07T17:46:40.2172169Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-07T17:46:40.2172585Z fi 2024-08-07T17:46:40.2179585Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:46:40.2180056Z env: 2024-08-07T17:46:40.2180349Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:46:40.2181284Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:40.2182352Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:40.2182827Z ##[endgroup] 2024-08-07T17:46:40.2206708Z + set +e 2024-08-07T17:46:40.2207503Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:40.2208867Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:40.2210563Z + aws ecr get-login-password --region us-east-1 2024-08-07T17:46:40.2212001Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:46:40.6954370Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-07T17:46:40.6955126Z Configure a credential helper to remove this warning. See 2024-08-07T17:46:40.6955811Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-07T17:46:40.6956436Z 2024-08-07T17:46:40.6956766Z Login Succeeded 2024-08-07T17:46:40.6967830Z + set -e 2024-08-07T17:46:40.6968765Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:40.7141882Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:40.7143436Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:46:40.9657308Z 02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9 2024-08-07T17:46:40.9658198Z 7a2c55901189: Pulling fs layer 2024-08-07T17:46:40.9658584Z 09d415c238d7: Pulling fs layer 2024-08-07T17:46:40.9658958Z 9fe6e2e61518: Pulling fs layer 2024-08-07T17:46:40.9659312Z 41f16248e682: Pulling fs layer 2024-08-07T17:46:40.9659966Z 95d7b7817039: Pulling fs layer 2024-08-07T17:46:40.9660373Z 8f6c90485347: Pulling fs layer 2024-08-07T17:46:40.9660724Z ab17245097e4: Pulling fs layer 2024-08-07T17:46:40.9661106Z dfecd7e9912b: Pulling fs layer 2024-08-07T17:46:40.9661481Z 464a8f745445: Pulling fs layer 2024-08-07T17:46:40.9661831Z d67c111fa588: Pulling fs layer 2024-08-07T17:46:40.9662218Z 0723abde19c0: Pulling fs layer 2024-08-07T17:46:40.9662585Z ceceb076946e: Pulling fs layer 2024-08-07T17:46:40.9662934Z 889eb8142c51: Pulling fs layer 2024-08-07T17:46:40.9663302Z af8081143f00: Pulling fs layer 2024-08-07T17:46:40.9663667Z cc64a79090ad: Pulling fs layer 2024-08-07T17:46:40.9664021Z ad26542e0d01: Pulling fs layer 2024-08-07T17:46:40.9664381Z 8f6c90485347: Waiting 2024-08-07T17:46:40.9664704Z ab17245097e4: Waiting 2024-08-07T17:46:40.9665022Z 7b42d8ae2367: Pulling fs layer 2024-08-07T17:46:40.9665401Z 6d46e3b97d9f: Pulling fs layer 2024-08-07T17:46:40.9665771Z 91f9dddffbee: Pulling fs layer 2024-08-07T17:46:40.9666124Z dfecd7e9912b: Waiting 2024-08-07T17:46:40.9666490Z 464a8f745445: Waiting 2024-08-07T17:46:40.9666828Z 03fb78f207e3: Pulling fs layer 2024-08-07T17:46:40.9667684Z d67c111fa588: Waiting 2024-08-07T17:46:40.9668017Z 889eb8142c51: Waiting 2024-08-07T17:46:40.9668363Z eeedca475f8c: Pulling fs layer 2024-08-07T17:46:40.9668707Z af8081143f00: Waiting 2024-08-07T17:46:40.9669052Z 202c71391ade: Pulling fs layer 2024-08-07T17:46:40.9669414Z 0723abde19c0: Waiting 2024-08-07T17:46:40.9669714Z 95d7b7817039: Waiting 2024-08-07T17:46:40.9670051Z 5af882ef27a2: Pulling fs layer 2024-08-07T17:46:40.9670423Z bac001ac2fee: Pulling fs layer 2024-08-07T17:46:40.9670765Z cc64a79090ad: Waiting 2024-08-07T17:46:40.9671092Z ceceb076946e: Waiting 2024-08-07T17:46:40.9671395Z ad26542e0d01: Waiting 2024-08-07T17:46:40.9671727Z 05896f1d52f7: Pulling fs layer 2024-08-07T17:46:40.9672385Z 49a0e418a175: Pulling fs layer 2024-08-07T17:46:40.9672773Z 7b42d8ae2367: Waiting 2024-08-07T17:46:40.9673110Z b5632121ff7a: Pulling fs layer 2024-08-07T17:46:40.9673486Z 6d46e3b97d9f: Waiting 2024-08-07T17:46:40.9673810Z deceb46806b5: Pulling fs layer 2024-08-07T17:46:40.9674172Z 03fb78f207e3: Waiting 2024-08-07T17:46:40.9674513Z a2630e5fe166: Pulling fs layer 2024-08-07T17:46:40.9674862Z 603261fb379f: Pulling fs layer 2024-08-07T17:46:40.9675233Z d8ac71337d62: Pulling fs layer 2024-08-07T17:46:40.9675602Z 2c2e35b979f5: Pulling fs layer 2024-08-07T17:46:40.9676177Z eeedca475f8c: Waiting 2024-08-07T17:46:40.9676515Z e559222c0ec6: Pulling fs layer 2024-08-07T17:46:40.9676887Z c4dcb7e9d189: Pulling fs layer 2024-08-07T17:46:40.9677244Z 2b9731d295d6: Pulling fs layer 2024-08-07T17:46:40.9677613Z 19da43312eba: Pulling fs layer 2024-08-07T17:46:40.9677972Z 49a0e418a175: Waiting 2024-08-07T17:46:40.9678275Z deceb46806b5: Waiting 2024-08-07T17:46:40.9678609Z b846263d0664: Pulling fs layer 2024-08-07T17:46:40.9678975Z 3886e8f57dc8: Pulling fs layer 2024-08-07T17:46:40.9679361Z 58e13e7b6d7b: Pulling fs layer 2024-08-07T17:46:40.9679717Z 0eb80a0ac000: Pulling fs layer 2024-08-07T17:46:40.9680089Z bac001ac2fee: Waiting 2024-08-07T17:46:40.9680418Z a2630e5fe166: Waiting 2024-08-07T17:46:40.9680735Z a9f11b2b3622: Pulling fs layer 2024-08-07T17:46:40.9681098Z 202c71391ade: Waiting 2024-08-07T17:46:40.9681400Z b846263d0664: Waiting 2024-08-07T17:46:40.9681726Z 58e13e7b6d7b: Waiting 2024-08-07T17:46:40.9682062Z d7e94279f8b2: Pulling fs layer 2024-08-07T17:46:40.9682471Z d8ac71337d62: Waiting 2024-08-07T17:46:40.9683005Z 4f4fb700ef54: Pulling fs layer 2024-08-07T17:46:40.9683386Z a9f11b2b3622: Waiting 2024-08-07T17:46:40.9683732Z 0a3560fcd26c: Pulling fs layer 2024-08-07T17:46:40.9684374Z e559222c0ec6: Waiting 2024-08-07T17:46:40.9684972Z d7e94279f8b2: Waiting 2024-08-07T17:46:40.9685584Z c4dcb7e9d189: Waiting 2024-08-07T17:46:40.9686205Z adf9d181cfbc: Pulling fs layer 2024-08-07T17:46:40.9686690Z 3886e8f57dc8: Waiting 2024-08-07T17:46:40.9687014Z 4f4fb700ef54: Waiting 2024-08-07T17:46:40.9687337Z bc18dd831597: Pulling fs layer 2024-08-07T17:46:40.9687712Z d62f2a608619: Pulling fs layer 2024-08-07T17:46:40.9688314Z bc18dd831597: Waiting 2024-08-07T17:46:40.9688664Z 719a1c2abcfd: Pulling fs layer 2024-08-07T17:46:40.9689034Z adf9d181cfbc: Waiting 2024-08-07T17:46:40.9689354Z 8e1355d516c5: Pulling fs layer 2024-08-07T17:46:40.9689719Z 701898e66f54: Pulling fs layer 2024-08-07T17:46:40.9690072Z 19da43312eba: Waiting 2024-08-07T17:46:40.9690380Z 8e1355d516c5: Waiting 2024-08-07T17:46:40.9690698Z 0eb80a0ac000: Waiting 2024-08-07T17:46:40.9691029Z 129b38cfed12: Pulling fs layer 2024-08-07T17:46:40.9691366Z 701898e66f54: Waiting 2024-08-07T17:46:40.9691691Z 1751651a072f: Pulling fs layer 2024-08-07T17:46:40.9692066Z fc956bf4e5b4: Pulling fs layer 2024-08-07T17:46:40.9692402Z 603261fb379f: Waiting 2024-08-07T17:46:40.9692722Z 129b38cfed12: Waiting 2024-08-07T17:46:40.9693039Z 1751651a072f: Waiting 2024-08-07T17:46:40.9693346Z 874491333fb5: Pulling fs layer 2024-08-07T17:46:40.9693698Z 2b9731d295d6: Waiting 2024-08-07T17:46:40.9694029Z 365b4a0eec21: Pulling fs layer 2024-08-07T17:46:40.9694479Z c1d4cac78640: Pulling fs layer 2024-08-07T17:46:40.9695028Z 29afe2181ca6: Pulling fs layer 2024-08-07T17:46:40.9695405Z 190dccfa687d: Pulling fs layer 2024-08-07T17:46:40.9695858Z 91f9dddffbee: Waiting 2024-08-07T17:46:40.9696491Z b01bea4b6634: Pulling fs layer 2024-08-07T17:46:40.9697319Z 874491333fb5: Waiting 2024-08-07T17:46:40.9697901Z f13a9cc59688: Pulling fs layer 2024-08-07T17:46:40.9698614Z c1d4cac78640: Waiting 2024-08-07T17:46:40.9699172Z 365b4a0eec21: Waiting 2024-08-07T17:46:40.9699770Z b01bea4b6634: Waiting 2024-08-07T17:46:40.9700431Z 96d6dcd01b9f: Pulling fs layer 2024-08-07T17:46:40.9701126Z fe134554f1b8: Pulling fs layer 2024-08-07T17:46:40.9701789Z 96d6dcd01b9f: Waiting 2024-08-07T17:46:40.9702383Z 8068d364963f: Pulling fs layer 2024-08-07T17:46:40.9703015Z f307f545c1a5: Pulling fs layer 2024-08-07T17:46:40.9703669Z fe134554f1b8: Waiting 2024-08-07T17:46:40.9704254Z 51c0b7eab505: Pulling fs layer 2024-08-07T17:46:40.9704865Z 8068d364963f: Waiting 2024-08-07T17:46:40.9705489Z 60aba96f31c6: Pulling fs layer 2024-08-07T17:46:40.9706139Z f307f545c1a5: Waiting 2024-08-07T17:46:40.9706701Z 46db894a106c: Pulling fs layer 2024-08-07T17:46:40.9707614Z 51c0b7eab505: Waiting 2024-08-07T17:46:40.9708201Z 0052583728e4: Pulling fs layer 2024-08-07T17:46:40.9708861Z 8c6d352a3b51: Pulling fs layer 2024-08-07T17:46:40.9709501Z fc956bf4e5b4: Waiting 2024-08-07T17:46:40.9710359Z 46db894a106c: Waiting 2024-08-07T17:46:40.9710943Z 0052583728e4: Waiting 2024-08-07T17:46:40.9711435Z 5af882ef27a2: Waiting 2024-08-07T17:46:40.9711753Z 380c4e16278f: Pulling fs layer 2024-08-07T17:46:40.9712118Z 8c6d352a3b51: Waiting 2024-08-07T17:46:40.9712442Z 29afe2181ca6: Waiting 2024-08-07T17:46:40.9712779Z 190dccfa687d: Waiting 2024-08-07T17:46:40.9713102Z d62f2a608619: Waiting 2024-08-07T17:46:40.9713419Z f13a9cc59688: Waiting 2024-08-07T17:46:40.9713717Z 60aba96f31c6: Waiting 2024-08-07T17:46:40.9714032Z 380c4e16278f: Waiting 2024-08-07T17:46:41.1120189Z 09d415c238d7: Verifying Checksum 2024-08-07T17:46:41.1120635Z 09d415c238d7: Download complete 2024-08-07T17:46:41.1918174Z 41f16248e682: Verifying Checksum 2024-08-07T17:46:41.1918636Z 41f16248e682: Download complete 2024-08-07T17:46:41.2827692Z 95d7b7817039: Verifying Checksum 2024-08-07T17:46:41.2828164Z 95d7b7817039: Download complete 2024-08-07T17:46:41.3087012Z 7a2c55901189: Download complete 2024-08-07T17:46:41.3934039Z ab17245097e4: Download complete 2024-08-07T17:46:41.4618361Z dfecd7e9912b: Download complete 2024-08-07T17:46:41.5802919Z 464a8f745445: Download complete 2024-08-07T17:46:41.6674038Z 9fe6e2e61518: Verifying Checksum 2024-08-07T17:46:41.6674469Z 9fe6e2e61518: Download complete 2024-08-07T17:46:41.7636312Z 0723abde19c0: Verifying Checksum 2024-08-07T17:46:41.7636747Z 0723abde19c0: Download complete 2024-08-07T17:46:41.8402950Z ceceb076946e: Verifying Checksum 2024-08-07T17:46:41.8403355Z ceceb076946e: Download complete 2024-08-07T17:46:42.6263914Z 7a2c55901189: Pull complete 2024-08-07T17:46:42.9901029Z 09d415c238d7: Pull complete 2024-08-07T17:46:44.1756794Z 9fe6e2e61518: Pull complete 2024-08-07T17:46:44.2378679Z 41f16248e682: Pull complete 2024-08-07T17:46:44.2894630Z 95d7b7817039: Pull complete 2024-08-07T17:46:48.4434723Z 889eb8142c51: Verifying Checksum 2024-08-07T17:46:48.4435535Z 889eb8142c51: Download complete 2024-08-07T17:46:48.5157220Z af8081143f00: Verifying Checksum 2024-08-07T17:46:48.5158001Z af8081143f00: Download complete 2024-08-07T17:46:48.6251791Z cc64a79090ad: Verifying Checksum 2024-08-07T17:46:48.6252244Z cc64a79090ad: Download complete 2024-08-07T17:46:48.7173658Z ad26542e0d01: Verifying Checksum 2024-08-07T17:46:48.7174428Z ad26542e0d01: Download complete 2024-08-07T17:46:49.7277531Z 7b42d8ae2367: Verifying Checksum 2024-08-07T17:46:49.7278024Z 7b42d8ae2367: Download complete 2024-08-07T17:46:49.8123240Z 6d46e3b97d9f: Verifying Checksum 2024-08-07T17:46:49.8123875Z 6d46e3b97d9f: Download complete 2024-08-07T17:46:49.8972204Z 91f9dddffbee: Verifying Checksum 2024-08-07T17:46:49.8972823Z 91f9dddffbee: Download complete 2024-08-07T17:46:49.9776411Z 03fb78f207e3: Verifying Checksum 2024-08-07T17:46:49.9776877Z 03fb78f207e3: Download complete 2024-08-07T17:46:55.1775008Z 8f6c90485347: Verifying Checksum 2024-08-07T17:46:55.1775806Z 8f6c90485347: Download complete 2024-08-07T17:46:55.2632204Z 202c71391ade: Verifying Checksum 2024-08-07T17:46:55.2632989Z 202c71391ade: Download complete 2024-08-07T17:46:55.3412023Z 5af882ef27a2: Download complete 2024-08-07T17:46:55.4140710Z bac001ac2fee: Verifying Checksum 2024-08-07T17:46:55.4141519Z bac001ac2fee: Download complete 2024-08-07T17:46:55.4842740Z 05896f1d52f7: Verifying Checksum 2024-08-07T17:46:55.4843501Z 05896f1d52f7: Download complete 2024-08-07T17:46:55.5616719Z 49a0e418a175: Verifying Checksum 2024-08-07T17:46:55.5617519Z 49a0e418a175: Download complete 2024-08-07T17:46:56.8325546Z b5632121ff7a: Verifying Checksum 2024-08-07T17:46:56.8326369Z b5632121ff7a: Download complete 2024-08-07T17:46:56.9073718Z deceb46806b5: Verifying Checksum 2024-08-07T17:46:56.9074508Z deceb46806b5: Download complete 2024-08-07T17:46:56.9948488Z a2630e5fe166: Download complete 2024-08-07T17:46:57.0824780Z 603261fb379f: Verifying Checksum 2024-08-07T17:46:57.0825555Z 603261fb379f: Download complete 2024-08-07T17:46:57.1571373Z d8ac71337d62: Verifying Checksum 2024-08-07T17:46:57.1572194Z d8ac71337d62: Download complete 2024-08-07T17:46:57.2280541Z 2c2e35b979f5: Download complete 2024-08-07T17:47:01.5591550Z e559222c0ec6: Verifying Checksum 2024-08-07T17:47:01.5592006Z e559222c0ec6: Download complete 2024-08-07T17:47:01.6536313Z c4dcb7e9d189: Verifying Checksum 2024-08-07T17:47:01.6536783Z c4dcb7e9d189: Download complete 2024-08-07T17:47:01.7168074Z 2b9731d295d6: Verifying Checksum 2024-08-07T17:47:01.7168815Z 2b9731d295d6: Download complete 2024-08-07T17:47:02.1593751Z 19da43312eba: Verifying Checksum 2024-08-07T17:47:02.1594232Z 19da43312eba: Download complete 2024-08-07T17:47:02.2468487Z b846263d0664: Verifying Checksum 2024-08-07T17:47:02.2469298Z b846263d0664: Download complete 2024-08-07T17:47:02.3393801Z 3886e8f57dc8: Verifying Checksum 2024-08-07T17:47:02.3394424Z 3886e8f57dc8: Download complete 2024-08-07T17:47:02.5922044Z 58e13e7b6d7b: Verifying Checksum 2024-08-07T17:47:02.5922854Z 58e13e7b6d7b: Download complete 2024-08-07T17:47:02.6723697Z 0eb80a0ac000: Verifying Checksum 2024-08-07T17:47:02.6724582Z 0eb80a0ac000: Download complete 2024-08-07T17:47:02.7923263Z a9f11b2b3622: Verifying Checksum 2024-08-07T17:47:02.7924133Z a9f11b2b3622: Download complete 2024-08-07T17:47:02.9054958Z d7e94279f8b2: Verifying Checksum 2024-08-07T17:47:02.9055434Z d7e94279f8b2: Download complete 2024-08-07T17:47:02.9137963Z 4f4fb700ef54: Download complete 2024-08-07T17:47:02.9955707Z 0a3560fcd26c: Download complete 2024-08-07T17:47:03.0691813Z adf9d181cfbc: Verifying Checksum 2024-08-07T17:47:03.0692622Z adf9d181cfbc: Download complete 2024-08-07T17:47:03.6994826Z bc18dd831597: Verifying Checksum 2024-08-07T17:47:03.6995516Z bc18dd831597: Download complete 2024-08-07T17:47:03.8041482Z d62f2a608619: Download complete 2024-08-07T17:47:03.9143963Z 719a1c2abcfd: Verifying Checksum 2024-08-07T17:47:03.9144794Z 719a1c2abcfd: Download complete 2024-08-07T17:47:03.9974587Z 8e1355d516c5: Verifying Checksum 2024-08-07T17:47:03.9975037Z 8e1355d516c5: Download complete 2024-08-07T17:47:04.0830455Z 701898e66f54: Download complete 2024-08-07T17:47:06.1110811Z d67c111fa588: Verifying Checksum 2024-08-07T17:47:06.1111355Z d67c111fa588: Download complete 2024-08-07T17:47:06.1813048Z 1751651a072f: Download complete 2024-08-07T17:47:06.2668309Z fc956bf4e5b4: Verifying Checksum 2024-08-07T17:47:06.2669117Z fc956bf4e5b4: Download complete 2024-08-07T17:47:06.3515013Z 874491333fb5: Verifying Checksum 2024-08-07T17:47:06.3515476Z 874491333fb5: Download complete 2024-08-07T17:47:06.4234395Z 365b4a0eec21: Verifying Checksum 2024-08-07T17:47:06.4234820Z 365b4a0eec21: Download complete 2024-08-07T17:47:06.5020792Z c1d4cac78640: Verifying Checksum 2024-08-07T17:47:06.5021243Z c1d4cac78640: Download complete 2024-08-07T17:47:06.6332038Z 29afe2181ca6: Verifying Checksum 2024-08-07T17:47:06.6332558Z 29afe2181ca6: Download complete 2024-08-07T17:47:06.7042982Z 190dccfa687d: Verifying Checksum 2024-08-07T17:47:06.7043411Z 190dccfa687d: Download complete 2024-08-07T17:47:06.7762112Z b01bea4b6634: Verifying Checksum 2024-08-07T17:47:06.7762567Z b01bea4b6634: Download complete 2024-08-07T17:47:06.8566702Z f13a9cc59688: Verifying Checksum 2024-08-07T17:47:06.8567174Z f13a9cc59688: Download complete 2024-08-07T17:47:06.9325766Z 96d6dcd01b9f: Download complete 2024-08-07T17:47:07.0839353Z fe134554f1b8: Verifying Checksum 2024-08-07T17:47:07.0839795Z fe134554f1b8: Download complete 2024-08-07T17:47:07.1693034Z 8068d364963f: Verifying Checksum 2024-08-07T17:47:07.1693471Z 8068d364963f: Download complete 2024-08-07T17:47:07.7523225Z f307f545c1a5: Verifying Checksum 2024-08-07T17:47:07.7524088Z f307f545c1a5: Download complete 2024-08-07T17:47:07.8396504Z 51c0b7eab505: Verifying Checksum 2024-08-07T17:47:07.8396944Z 51c0b7eab505: Download complete 2024-08-07T17:47:13.9207241Z 8f6c90485347: Pull complete 2024-08-07T17:47:13.9749079Z ab17245097e4: Pull complete 2024-08-07T17:47:14.0429428Z dfecd7e9912b: Pull complete 2024-08-07T17:47:14.1014081Z 464a8f745445: Pull complete 2024-08-07T17:47:18.1645133Z eeedca475f8c: Verifying Checksum 2024-08-07T17:47:18.1645938Z eeedca475f8c: Download complete 2024-08-07T17:47:18.2495271Z 46db894a106c: Verifying Checksum 2024-08-07T17:47:18.2496124Z 46db894a106c: Download complete 2024-08-07T17:47:18.3162158Z 0052583728e4: Download complete 2024-08-07T17:47:18.7304776Z 8c6d352a3b51: Verifying Checksum 2024-08-07T17:47:18.7305246Z 8c6d352a3b51: Download complete 2024-08-07T17:47:18.8022468Z 380c4e16278f: Download complete 2024-08-07T17:47:21.6516432Z 60aba96f31c6: Verifying Checksum 2024-08-07T17:47:21.6517001Z 60aba96f31c6: Download complete 2024-08-07T17:47:23.3161052Z 129b38cfed12: Verifying Checksum 2024-08-07T17:47:23.3161873Z 129b38cfed12: Download complete 2024-08-07T17:47:52.2335856Z d67c111fa588: Pull complete 2024-08-07T17:47:54.8098996Z 0723abde19c0: Pull complete 2024-08-07T17:47:57.3820868Z ceceb076946e: Pull complete 2024-08-07T17:48:13.9298209Z 889eb8142c51: Pull complete 2024-08-07T17:48:16.5084871Z af8081143f00: Pull complete 2024-08-07T17:48:19.0141821Z cc64a79090ad: Pull complete 2024-08-07T17:48:21.4878825Z ad26542e0d01: Pull complete 2024-08-07T17:48:27.4265754Z 7b42d8ae2367: Pull complete 2024-08-07T17:48:29.8552703Z 6d46e3b97d9f: Pull complete 2024-08-07T17:48:32.4663990Z 91f9dddffbee: Pull complete 2024-08-07T17:48:34.9413028Z 03fb78f207e3: Pull complete 2024-08-07T17:49:41.3632947Z eeedca475f8c: Pull complete 2024-08-07T17:49:45.1355492Z 202c71391ade: Pull complete 2024-08-07T17:49:48.4808780Z 5af882ef27a2: Pull complete 2024-08-07T17:49:48.5246047Z bac001ac2fee: Pull complete 2024-08-07T17:49:48.5606242Z 05896f1d52f7: Pull complete 2024-08-07T17:49:48.6065975Z 49a0e418a175: Pull complete 2024-08-07T17:49:52.6937713Z b5632121ff7a: Pull complete 2024-08-07T17:49:52.7410037Z deceb46806b5: Pull complete 2024-08-07T17:49:52.7810264Z a2630e5fe166: Pull complete 2024-08-07T17:49:52.8294303Z 603261fb379f: Pull complete 2024-08-07T17:49:52.8779351Z d8ac71337d62: Pull complete 2024-08-07T17:49:52.9207973Z 2c2e35b979f5: Pull complete 2024-08-07T17:50:04.8998691Z e559222c0ec6: Pull complete 2024-08-07T17:50:07.8622199Z c4dcb7e9d189: Pull complete 2024-08-07T17:50:10.4544170Z 2b9731d295d6: Pull complete 2024-08-07T17:50:14.0995194Z 19da43312eba: Pull complete 2024-08-07T17:50:14.1367260Z b846263d0664: Pull complete 2024-08-07T17:50:14.1848427Z 3886e8f57dc8: Pull complete 2024-08-07T17:50:14.5571495Z 58e13e7b6d7b: Pull complete 2024-08-07T17:50:14.6005673Z 0eb80a0ac000: Pull complete 2024-08-07T17:50:14.7211819Z a9f11b2b3622: Pull complete 2024-08-07T17:50:14.7727585Z d7e94279f8b2: Pull complete 2024-08-07T17:50:14.8098879Z 4f4fb700ef54: Pull complete 2024-08-07T17:50:14.8599636Z 0a3560fcd26c: Pull complete 2024-08-07T17:50:14.9087766Z adf9d181cfbc: Pull complete 2024-08-07T17:50:17.4213077Z bc18dd831597: Pull complete 2024-08-07T17:50:17.4604512Z d62f2a608619: Pull complete 2024-08-07T17:50:17.5208124Z 719a1c2abcfd: Pull complete 2024-08-07T17:50:17.6431235Z 8e1355d516c5: Pull complete 2024-08-07T17:50:17.6922583Z 701898e66f54: Pull complete 2024-08-07T17:51:02.8825284Z 129b38cfed12: Pull complete 2024-08-07T17:51:05.3536478Z 1751651a072f: Pull complete 2024-08-07T17:51:07.8252157Z fc956bf4e5b4: Pull complete 2024-08-07T17:51:12.6081816Z 874491333fb5: Pull complete 2024-08-07T17:51:17.6493142Z 365b4a0eec21: Pull complete 2024-08-07T17:51:20.5952058Z c1d4cac78640: Pull complete 2024-08-07T17:51:23.5010627Z 29afe2181ca6: Pull complete 2024-08-07T17:51:26.0698371Z 190dccfa687d: Pull complete 2024-08-07T17:51:28.7371931Z b01bea4b6634: Pull complete 2024-08-07T17:51:31.5017896Z f13a9cc59688: Pull complete 2024-08-07T17:51:31.6764516Z 96d6dcd01b9f: Pull complete 2024-08-07T17:51:32.8829750Z fe134554f1b8: Pull complete 2024-08-07T17:51:32.9178631Z 8068d364963f: Pull complete 2024-08-07T17:51:35.2447566Z f307f545c1a5: Pull complete 2024-08-07T17:51:35.3092620Z 51c0b7eab505: Pull complete 2024-08-07T17:51:52.2109985Z 60aba96f31c6: Pull complete 2024-08-07T17:51:52.2500507Z 46db894a106c: Pull complete 2024-08-07T17:51:52.3010582Z 0052583728e4: Pull complete 2024-08-07T17:51:53.0259212Z 8c6d352a3b51: Pull complete 2024-08-07T17:51:53.0940393Z 380c4e16278f: Pull complete 2024-08-07T17:51:53.4111428Z Digest: sha256:66181a134b3ebc35388dabcc46dc11e767152fd72514dad76711590f50b650f7 2024-08-07T17:51:53.4145832Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:51:53.4176631Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:51:53.4228303Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-07T17:51:53.4229210Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-07T17:51:53.4238766Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:51:53.4239256Z env: 2024-08-07T17:51:53.4239554Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:51:53.4239889Z ##[endgroup] 2024-08-07T17:51:53.4499174Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-07T17:51:53.4499749Z with: 2024-08-07T17:51:53.4500052Z driver-version: 550.54.15 2024-08-07T17:51:53.4500411Z env: 2024-08-07T17:51:53.4500702Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:51:53.4501036Z ##[endgroup] 2024-08-07T17:51:53.4554388Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-07T17:51:53.4554925Z with: 2024-08-07T17:51:53.4555226Z timeout_minutes: 10 2024-08-07T17:51:53.4555540Z max_attempts: 3 2024-08-07T17:51:53.4584482Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-07T17:51:53.4614081Z retry_wait_seconds: 10 2024-08-07T17:51:53.4614452Z polling_interval_seconds: 1 2024-08-07T17:51:53.4614815Z warning_on_retry: true 2024-08-07T17:51:53.4615163Z continue_on_error: false 2024-08-07T17:51:53.4615503Z env: 2024-08-07T17:51:53.4615774Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:51:53.4616126Z DRIVER_VERSION: 550.54.15 2024-08-07T17:51:53.4616454Z ##[endgroup] 2024-08-07T17:51:53.5371960Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-07T17:51:53.5374096Z + pre_install_nvidia_driver_amzn2 2024-08-07T17:51:53.5376302Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-07T17:51:54.0475836Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-08-07T17:51:54.1078803Z No Match for argument: nvidia-driver-latest-dkms 2024-08-07T17:51:54.1451502Z No Packages marked for removal 2024-08-07T17:51:54.1609024Z + install_nvidia_driver_common 2024-08-07T17:51:54.1612484Z + echo 'Before installing NVIDIA driver' 2024-08-07T17:51:54.1613335Z + lspci 2024-08-07T17:51:54.1615446Z Before installing NVIDIA driver 2024-08-07T17:51:54.1781442Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-08-07T17:51:54.1782396Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-07T17:51:54.1783083Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-08-07T17:51:54.1783740Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-08-07T17:51:54.1784292Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-08-07T17:51:54.1784904Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-07T17:51:54.1785637Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-07T17:51:54.1786369Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-07T17:51:54.1787513Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-08-07T17:51:54.1788227Z + lsmod 2024-08-07T17:51:54.1805510Z Module Size Used by 2024-08-07T17:51:54.1805943Z ib_core 266240 0 2024-08-07T17:51:54.1806330Z veth 16384 0 2024-08-07T17:51:54.1806692Z nvidia_modeset 1339392 0 2024-08-07T17:51:54.1807039Z nvidia_uvm 4599808 0 2024-08-07T17:51:54.1807457Z nvidia 53989376 11 nvidia_uvm,nvidia_modeset 2024-08-07T17:51:54.1807904Z drm 421888 1 nvidia 2024-08-07T17:51:54.1808305Z i2c_core 77824 2 nvidia,drm 2024-08-07T17:51:54.1808760Z backlight 16384 1 nvidia_modeset 2024-08-07T17:51:54.1809187Z xt_conntrack 16384 1 2024-08-07T17:51:54.1809537Z ipt_MASQUERADE 16384 1 2024-08-07T17:51:54.1809944Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-08-07T17:51:54.1810395Z nf_conntrack_netlink 49152 0 2024-08-07T17:51:54.1810806Z nfnetlink 16384 2 nf_conntrack_netlink 2024-08-07T17:51:54.1811259Z xfrm_user 45056 1 2024-08-07T17:51:54.1811633Z xfrm_algo 16384 1 xfrm_user 2024-08-07T17:51:54.1812019Z iptable_nat 16384 1 2024-08-07T17:51:54.1812389Z nf_conntrack_ipv4 16384 3 2024-08-07T17:51:54.1812823Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-08-07T17:51:54.1813256Z nf_nat_ipv4 16384 1 iptable_nat 2024-08-07T17:51:54.1813728Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-08-07T17:51:54.1814583Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-08-07T17:51:54.1815345Z xt_addrtype 16384 2 2024-08-07T17:51:54.1815709Z iptable_filter 16384 1 2024-08-07T17:51:54.1816075Z br_netfilter 24576 0 2024-08-07T17:51:54.1816444Z bridge 172032 1 br_netfilter 2024-08-07T17:51:54.1816865Z stp 16384 1 bridge 2024-08-07T17:51:54.1817270Z llc 16384 2 bridge,stp 2024-08-07T17:51:54.1817649Z overlay 86016 0 2024-08-07T17:51:54.1818009Z sunrpc 393216 1 2024-08-07T17:51:54.1818374Z dm_mirror 28672 0 2024-08-07T17:51:54.1818728Z dm_region_hash 20480 1 dm_mirror 2024-08-07T17:51:54.1819173Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-08-07T17:51:54.1819635Z dm_mod 143360 2 dm_log,dm_mirror 2024-08-07T17:51:54.1820037Z dax 69632 1 dm_mod 2024-08-07T17:51:54.1820413Z sb_edac 24576 0 2024-08-07T17:51:54.1820769Z crc32_pclmul 16384 0 2024-08-07T17:51:54.1821116Z ghash_clmulni_intel 16384 0 2024-08-07T17:51:54.1821482Z pcbc 16384 0 2024-08-07T17:51:54.1821841Z aesni_intel 188416 0 2024-08-07T17:51:54.1822195Z aes_x86_64 20480 1 aesni_intel 2024-08-07T17:51:54.1822598Z mousedev 24576 0 2024-08-07T17:51:54.1822977Z crypto_simd 16384 1 aesni_intel 2024-08-07T17:51:54.1823391Z glue_helper 16384 1 aesni_intel 2024-08-07T17:51:54.1823900Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-08-07T17:51:54.1824646Z pcc_cpufreq 16384 0 2024-08-07T17:51:54.1824995Z evdev 20480 3 2024-08-07T17:51:54.1825368Z psmouse 32768 0 2024-08-07T17:51:54.1825729Z button 16384 0 2024-08-07T17:51:54.1826061Z ena 139264 0 2024-08-07T17:51:54.1826414Z ptp 20480 1 ena 2024-08-07T17:51:54.1826806Z pps_core 20480 1 ptp 2024-08-07T17:51:54.1827708Z ata_piix 36864 0 2024-08-07T17:51:54.1828083Z libata 266240 1 ata_piix 2024-08-07T17:51:54.1828488Z xen_blkfront 49152 2 2024-08-07T17:51:54.1828832Z crc32c_intel 24576 0 2024-08-07T17:51:54.1829196Z scsi_mod 245760 1 libata 2024-08-07T17:51:54.1829586Z autofs4 49152 2 2024-08-07T17:51:54.1829922Z + modinfo nvidia 2024-08-07T17:51:54.1830599Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-08-07T17:51:54.1831229Z alias: char-major-195-* 2024-08-07T17:51:54.1831587Z version: 550.54.15 2024-08-07T17:51:54.1831932Z supported: external 2024-08-07T17:51:54.1832273Z license: NVIDIA 2024-08-07T17:51:54.1832627Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-07T17:51:54.1833088Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-07T17:51:54.1833524Z srcversion: 833721318DA517F0C2FEC97 2024-08-07T17:51:54.1833939Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-07T17:51:54.1834400Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-07T17:51:54.1834855Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-07T17:51:54.1835269Z depends: i2c-core,drm 2024-08-07T17:51:54.1835627Z retpoline: Y 2024-08-07T17:51:54.1835933Z name: nvidia 2024-08-07T17:51:54.1836363Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-08-07T17:51:54.1836951Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-07T17:51:54.1837552Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-07T17:51:54.1838216Z parm: NVreg_ResmanDebugLevel:int 2024-08-07T17:51:54.1838640Z parm: NVreg_RmLogonRC:int 2024-08-07T17:51:54.1839058Z parm: NVreg_ModifyDeviceFiles:int 2024-08-07T17:51:54.1839472Z parm: NVreg_DeviceFileUID:int 2024-08-07T17:51:54.1840042Z parm: NVreg_DeviceFileGID:int 2024-08-07T17:51:54.1840466Z parm: NVreg_DeviceFileMode:int 2024-08-07T17:51:54.1840935Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-07T17:51:54.1841458Z parm: NVreg_UsePageAttributeTable:int 2024-08-07T17:51:54.1841913Z parm: NVreg_EnablePCIeGen3:int 2024-08-07T17:51:54.1842309Z parm: NVreg_EnableMSI:int 2024-08-07T17:51:54.1842711Z parm: NVreg_TCEBypassMode:int 2024-08-07T17:51:54.1843156Z parm: NVreg_EnableStreamMemOPs:int 2024-08-07T17:51:54.1843631Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-07T17:51:54.1844169Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-07T17:51:54.1844679Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-07T17:51:54.1845215Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-07T17:51:54.1845772Z parm: NVreg_DynamicPowerManagement:int 2024-08-07T17:51:54.1846331Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-07T17:51:54.1846863Z parm: NVreg_EnableGpuFirmware:int 2024-08-07T17:51:54.1847320Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-07T17:51:54.1847822Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-07T17:51:54.1848315Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-07T17:51:54.1848778Z parm: NVreg_MemoryPoolSize:int 2024-08-07T17:51:54.1849226Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-07T17:51:54.1849664Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-07T17:51:54.1850108Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-07T17:51:54.1850683Z parm: NVreg_NvLinkDisable:int 2024-08-07T17:51:54.1851138Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-07T17:51:54.1851629Z parm: NVreg_RegisterPCIDriver:int 2024-08-07T17:51:54.1852079Z parm: NVreg_EnableResizableBar:int 2024-08-07T17:51:54.1852518Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-07T17:51:54.1852986Z parm: NVreg_EnableNonblockingOpen:int 2024-08-07T17:51:54.1853448Z parm: NVreg_RegistryDwords:charp 2024-08-07T17:51:54.1853897Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-07T17:51:54.1854401Z parm: NVreg_RmMsg:charp 2024-08-07T17:51:54.1854801Z parm: NVreg_GpuBlacklist:charp 2024-08-07T17:51:54.1855227Z parm: NVreg_TemporaryFilePath:charp 2024-08-07T17:51:54.1855766Z parm: NVreg_ExcludedGpus:charp 2024-08-07T17:51:54.1856211Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-07T17:51:54.1856646Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-07T17:51:54.1857107Z parm: NVreg_ImexChannelCount:int 2024-08-07T17:51:54.1857541Z parm: rm_firmware_active:charp 2024-08-07T17:51:54.1857928Z + HAS_NVIDIA_DRIVER=0 2024-08-07T17:51:54.1858271Z ++ command -v nvidia-smi 2024-08-07T17:51:54.1858612Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-07T17:51:54.1858970Z + set +e 2024-08-07T17:51:54.1859388Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-07T17:51:54.2090225Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-07T17:51:54.2090658Z + NVIDIA_SMI_STATUS=0 2024-08-07T17:51:54.2090991Z + '[' 0 -ne 0 ']' 2024-08-07T17:51:54.2091282Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-07T17:51:54.2091649Z + HAS_NVIDIA_DRIVER=1 2024-08-07T17:51:54.2092205Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-07T17:51:54.2092842Z + set -e 2024-08-07T17:51:54.2093131Z + '[' 1 -eq 0 ']' 2024-08-07T17:51:54.2093630Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-07T17:51:54.2094258Z + post_install_nvidia_driver_common 2024-08-07T17:51:54.2097624Z + sudo modprobe nvidia 2024-08-07T17:51:54.2214739Z + echo 'After installing NVIDIA driver' 2024-08-07T17:51:54.2215153Z + lspci 2024-08-07T17:51:54.2216041Z After installing NVIDIA driver 2024-08-07T17:51:54.2387728Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-08-07T17:51:54.2388418Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-07T17:51:54.2389104Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-08-07T17:51:54.2389761Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-08-07T17:51:54.2390313Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-08-07T17:51:54.2390933Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-07T17:51:54.2391668Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-07T17:51:54.2392410Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-07T17:51:54.2393136Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-08-07T17:51:54.2393675Z + lsmod 2024-08-07T17:51:54.2410013Z Module Size Used by 2024-08-07T17:51:54.2410422Z ib_core 266240 0 2024-08-07T17:51:54.2410793Z veth 16384 0 2024-08-07T17:51:54.2411154Z nvidia_modeset 1339392 0 2024-08-07T17:51:54.2411499Z nvidia_uvm 4599808 0 2024-08-07T17:51:54.2411920Z nvidia 53989376 11 nvidia_uvm,nvidia_modeset 2024-08-07T17:51:54.2412378Z drm 421888 1 nvidia 2024-08-07T17:51:54.2412765Z i2c_core 77824 2 nvidia,drm 2024-08-07T17:51:54.2413215Z backlight 16384 1 nvidia_modeset 2024-08-07T17:51:54.2413643Z xt_conntrack 16384 1 2024-08-07T17:51:54.2413994Z ipt_MASQUERADE 16384 1 2024-08-07T17:51:54.2414657Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-08-07T17:51:54.2415105Z nf_conntrack_netlink 49152 0 2024-08-07T17:51:54.2415512Z nfnetlink 16384 2 nf_conntrack_netlink 2024-08-07T17:51:54.2415954Z xfrm_user 45056 1 2024-08-07T17:51:54.2416326Z xfrm_algo 16384 1 xfrm_user 2024-08-07T17:51:54.2416714Z iptable_nat 16384 1 2024-08-07T17:51:54.2417081Z nf_conntrack_ipv4 16384 3 2024-08-07T17:51:54.2417485Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-08-07T17:51:54.2417916Z nf_nat_ipv4 16384 1 iptable_nat 2024-08-07T17:51:54.2418390Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-08-07T17:51:54.2419380Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-08-07T17:51:54.2420178Z xt_addrtype 16384 2 2024-08-07T17:51:54.2420548Z iptable_filter 16384 1 2024-08-07T17:51:54.2420923Z br_netfilter 24576 0 2024-08-07T17:51:54.2421287Z bridge 172032 1 br_netfilter 2024-08-07T17:51:54.2421698Z stp 16384 1 bridge 2024-08-07T17:51:54.2422095Z llc 16384 2 bridge,stp 2024-08-07T17:51:54.2422472Z overlay 86016 0 2024-08-07T17:51:54.2422826Z sunrpc 393216 1 2024-08-07T17:51:54.2423180Z dm_mirror 28672 0 2024-08-07T17:51:54.2423533Z dm_region_hash 20480 1 dm_mirror 2024-08-07T17:51:54.2423976Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-08-07T17:51:54.2424436Z dm_mod 143360 2 dm_log,dm_mirror 2024-08-07T17:51:54.2424840Z dax 69632 1 dm_mod 2024-08-07T17:51:54.2425217Z sb_edac 24576 0 2024-08-07T17:51:54.2425575Z crc32_pclmul 16384 0 2024-08-07T17:51:54.2425932Z ghash_clmulni_intel 16384 0 2024-08-07T17:51:54.2426297Z pcbc 16384 0 2024-08-07T17:51:54.2426661Z aesni_intel 188416 0 2024-08-07T17:51:54.2427533Z aes_x86_64 20480 1 aesni_intel 2024-08-07T17:51:54.2427950Z mousedev 24576 0 2024-08-07T17:51:54.2428330Z crypto_simd 16384 1 aesni_intel 2024-08-07T17:51:54.2428739Z glue_helper 16384 1 aesni_intel 2024-08-07T17:51:54.2429250Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-08-07T17:51:54.2429764Z pcc_cpufreq 16384 0 2024-08-07T17:51:54.2430107Z evdev 20480 3 2024-08-07T17:51:54.2430461Z psmouse 32768 0 2024-08-07T17:51:54.2430801Z button 16384 0 2024-08-07T17:51:54.2431153Z ena 139264 0 2024-08-07T17:51:54.2431509Z ptp 20480 1 ena 2024-08-07T17:51:54.2431873Z pps_core 20480 1 ptp 2024-08-07T17:51:54.2432264Z ata_piix 36864 0 2024-08-07T17:51:54.2432631Z libata 266240 1 ata_piix 2024-08-07T17:51:54.2433018Z xen_blkfront 49152 2 2024-08-07T17:51:54.2433378Z crc32c_intel 24576 0 2024-08-07T17:51:54.2433853Z scsi_mod 245760 1 libata 2024-08-07T17:51:54.2434230Z autofs4 49152 2 2024-08-07T17:51:54.2434616Z + modinfo nvidia 2024-08-07T17:51:54.2435122Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-08-07T17:51:54.2435858Z alias: char-major-195-* 2024-08-07T17:51:54.2436223Z version: 550.54.15 2024-08-07T17:51:54.2436573Z supported: external 2024-08-07T17:51:54.2436918Z license: NVIDIA 2024-08-07T17:51:54.2437268Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-07T17:51:54.2437731Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-07T17:51:54.2438145Z srcversion: 833721318DA517F0C2FEC97 2024-08-07T17:51:54.2438587Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-07T17:51:54.2439050Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-07T17:51:54.2439655Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-07T17:51:54.2440083Z depends: i2c-core,drm 2024-08-07T17:51:54.2440445Z retpoline: Y 2024-08-07T17:51:54.2440737Z name: nvidia 2024-08-07T17:51:54.2441182Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-08-07T17:51:54.2441771Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-07T17:51:54.2442351Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-07T17:51:54.2442912Z parm: NVreg_ResmanDebugLevel:int 2024-08-07T17:51:54.2443343Z parm: NVreg_RmLogonRC:int 2024-08-07T17:51:54.2443740Z parm: NVreg_ModifyDeviceFiles:int 2024-08-07T17:51:54.2444172Z parm: NVreg_DeviceFileUID:int 2024-08-07T17:51:54.2444592Z parm: NVreg_DeviceFileGID:int 2024-08-07T17:51:54.2445111Z parm: NVreg_DeviceFileMode:int 2024-08-07T17:51:54.2445616Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-07T17:51:54.2446148Z parm: NVreg_UsePageAttributeTable:int 2024-08-07T17:51:54.2446585Z parm: NVreg_EnablePCIeGen3:int 2024-08-07T17:51:54.2447002Z parm: NVreg_EnableMSI:int 2024-08-07T17:51:54.2447403Z parm: NVreg_TCEBypassMode:int 2024-08-07T17:51:54.2447822Z parm: NVreg_EnableStreamMemOPs:int 2024-08-07T17:51:54.2448316Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-07T17:51:54.2448868Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-07T17:51:54.2449364Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-07T17:51:54.2449921Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-07T17:51:54.2450470Z parm: NVreg_DynamicPowerManagement:int 2024-08-07T17:51:54.2451015Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-07T17:51:54.2451573Z parm: NVreg_EnableGpuFirmware:int 2024-08-07T17:51:54.2452037Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-07T17:51:54.2452527Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-07T17:51:54.2453034Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-07T17:51:54.2453495Z parm: NVreg_MemoryPoolSize:int 2024-08-07T17:51:54.2453920Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-07T17:51:54.2454420Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-07T17:51:54.2454868Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-07T17:51:54.2455282Z parm: NVreg_NvLinkDisable:int 2024-08-07T17:51:54.2455757Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-07T17:51:54.2456253Z parm: NVreg_RegisterPCIDriver:int 2024-08-07T17:51:54.2456687Z parm: NVreg_EnableResizableBar:int 2024-08-07T17:51:54.2457151Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-07T17:51:54.2457619Z parm: NVreg_EnableNonblockingOpen:int 2024-08-07T17:51:54.2458071Z parm: NVreg_RegistryDwords:charp 2024-08-07T17:51:54.2458542Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-07T17:51:54.2459003Z parm: NVreg_RmMsg:charp 2024-08-07T17:51:54.2459387Z parm: NVreg_GpuBlacklist:charp 2024-08-07T17:51:54.2459836Z parm: NVreg_TemporaryFilePath:charp 2024-08-07T17:51:54.2460279Z parm: NVreg_ExcludedGpus:charp 2024-08-07T17:51:54.2460692Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-07T17:51:54.2461139Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-07T17:51:54.2461585Z parm: NVreg_ImexChannelCount:int 2024-08-07T17:51:54.2461994Z parm: rm_firmware_active:charp 2024-08-07T17:51:54.2462378Z + set +e 2024-08-07T17:51:54.2462659Z + nvidia-smi 2024-08-07T17:51:54.2677286Z Wed Aug 7 17:51:54 2024 2024-08-07T17:51:54.2677787Z +-----------------------------------------------------------------------------------------+ 2024-08-07T17:51:54.2678467Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-07T17:51:54.2679124Z |-----------------------------------------+------------------------+----------------------+ 2024-08-07T17:51:54.2679963Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-07T17:51:54.2680674Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-07T17:51:54.2681269Z | | | MIG M. | 2024-08-07T17:51:54.2681728Z |=========================================+========================+======================| 2024-08-07T17:51:54.2766707Z | 0 Tesla M60 On | 00000000:00:1D.0 Off | 0 | 2024-08-07T17:51:54.2767356Z | N/A 33C P8 14W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-07T17:51:54.2768060Z | | | N/A | 2024-08-07T17:51:54.2768641Z +-----------------------------------------+------------------------+----------------------+ 2024-08-07T17:51:54.2834005Z | 1 Tesla M60 On | 00000000:00:1E.0 Off | 0 | 2024-08-07T17:51:54.2834613Z | N/A 35C P8 15W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-07T17:51:54.2835127Z | | | N/A | 2024-08-07T17:51:54.2835676Z +-----------------------------------------+------------------------+----------------------+ 2024-08-07T17:51:54.2836218Z 2024-08-07T17:51:54.2836740Z +-----------------------------------------------------------------------------------------+ 2024-08-07T17:51:54.2837337Z | Processes: | 2024-08-07T17:51:54.2837959Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-07T17:51:54.2838543Z | ID ID Usage | 2024-08-07T17:51:54.2839021Z |=========================================================================================| 2024-08-07T17:51:54.2840341Z | No running processes found | 2024-08-07T17:51:54.2840990Z +-----------------------------------------------------------------------------------------+ 2024-08-07T17:51:54.3964296Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-07T17:51:54.4154120Z Tesla M60 2024-08-07T17:51:54.4218582Z + NVIDIA_SMI_STATUS=0 2024-08-07T17:51:54.4218929Z + '[' 0 -eq 0 ']' 2024-08-07T17:51:54.4219260Z + echo 'INFO: Ignoring allowed status 0' 2024-08-07T17:51:54.4219664Z + set -e 2024-08-07T17:51:54.4220587Z INFO: Ignoring allowed status 0 2024-08-07T17:51:54.4225801Z == Installing nvidia container toolkit for amzn2 == 2024-08-07T17:51:54.4229770Z + sudo yum install -y yum-utils 2024-08-07T17:51:54.9139577Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-08-07T17:51:56.6146157Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-08-07T17:51:56.6146770Z Nothing to do 2024-08-07T17:51:56.7591447Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-08-07T17:51:56.7592110Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-08-07T17:51:56.7593028Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-08-07T17:51:57.2709854Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-08-07T17:51:57.3064333Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-08-07T17:51:57.3065345Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-08-07T17:51:57.3213345Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-08-07T17:51:57.3213839Z + sudo yum install -y nvidia-docker2 2024-08-07T17:51:57.8112021Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-08-07T17:51:59.5026392Z Package nvidia-docker2-2.14.0-1.noarch already installed and latest version 2024-08-07T17:51:59.5027369Z Nothing to do 2024-08-07T17:51:59.6451644Z + sudo systemctl restart docker 2024-08-07T17:52:18.7332983Z nvidia-persistenced failed to initialize. Check syslog for more details. 2024-08-07T17:52:18.7803214Z Wed Aug 7 17:52:18 2024 2024-08-07T17:52:18.7803775Z +-----------------------------------------------------------------------------------------+ 2024-08-07T17:52:18.7804447Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-07T17:52:18.7805110Z |-----------------------------------------+------------------------+----------------------+ 2024-08-07T17:52:18.7806065Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-07T17:52:18.7806814Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-07T17:52:18.7807442Z | | | MIG M. | 2024-08-07T17:52:18.7807901Z |=========================================+========================+======================| 2024-08-07T17:52:18.7893527Z | 0 Tesla M60 On | 00000000:00:1D.0 Off | 0 | 2024-08-07T17:52:18.7894248Z | N/A 33C P8 15W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-07T17:52:18.7894793Z | | | N/A | 2024-08-07T17:52:18.7895324Z +-----------------------------------------+------------------------+----------------------+ 2024-08-07T17:52:18.7993666Z | 1 Tesla M60 On | 00000000:00:1E.0 Off | 0 | 2024-08-07T17:52:18.7994358Z | N/A 35C P8 16W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-07T17:52:18.7995008Z | | | N/A | 2024-08-07T17:52:18.7995952Z +-----------------------------------------+------------------------+----------------------+ 2024-08-07T17:52:18.7996713Z 2024-08-07T17:52:18.7997271Z +-----------------------------------------------------------------------------------------+ 2024-08-07T17:52:18.7997847Z | Processes: | 2024-08-07T17:52:18.7998464Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-07T17:52:18.7999047Z | ID ID Usage | 2024-08-07T17:52:18.7999840Z |=========================================================================================| 2024-08-07T17:52:18.8000940Z | No running processes found | 2024-08-07T17:52:18.8001619Z +-----------------------------------------------------------------------------------------+ 2024-08-07T17:52:19.5634406Z Command completed after 1 attempt(s). 2024-08-07T17:52:19.5703267Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-07T17:52:19.5704026Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-07T17:52:19.5704640Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-07T17:52:19.5705236Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-07T17:52:19.5713775Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:52:19.5714242Z env: 2024-08-07T17:52:19.5714537Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:52:19.5714969Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:52:19.5715409Z ##[endgroup] 2024-08-07T17:52:19.8754767Z Defaulting to user installation because normal site-packages is not writeable 2024-08-07T17:52:19.8968664Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-08-07T17:52:19.9098038Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-08-07T17:52:20.0293213Z Prepare all required actions 2024-08-07T17:52:20.0294021Z Getting action download info 2024-08-07T17:52:20.1409252Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-07T17:52:20.3385162Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-07T17:52:20.4766097Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-07T17:52:20.4766564Z with: 2024-08-07T17:52:20.4766896Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-08-07T17:52:20.4767309Z s3-bucket: gha-artifacts 2024-08-07T17:52:20.4767656Z env: 2024-08-07T17:52:20.4767947Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:52:20.4768375Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:52:20.4768836Z ##[endgroup] 2024-08-07T17:52:20.4809333Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-07T17:52:20.4809773Z with: 2024-08-07T17:52:20.4810146Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-08-07T17:52:20.4810551Z s3-bucket: gha-artifacts 2024-08-07T17:52:20.4810903Z region: us-east-1 2024-08-07T17:52:20.4811211Z env: 2024-08-07T17:52:20.4811477Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:52:20.4811907Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:52:20.4812374Z ##[endgroup] 2024-08-07T17:52:21.0859482Z (node:71814) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-07T17:52:21.0860079Z 2024-08-07T17:52:21.0860344Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-07T17:52:21.0860991Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-07T17:52:21.0861714Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-07T17:52:21.1759492Z Found 1 objects with prefix pytorch/pytorch/10288745067/linux-focal-cuda11.8-py3.10-gcc9/ 2024-08-07T17:52:21.1760393Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-07T17:52:48.3544970Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-07T17:52:48.3556801Z Artifact download has finished successfully 2024-08-07T17:52:48.3773074Z ##[group]Run unzip -o artifacts.zip 2024-08-07T17:52:48.3773530Z unzip -o artifacts.zip 2024-08-07T17:52:48.3781502Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:52:48.3781998Z env: 2024-08-07T17:52:48.3782296Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:52:48.3782711Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:52:48.3783178Z ##[endgroup] 2024-08-07T17:52:48.3819754Z Archive: artifacts.zip 2024-08-07T17:52:48.3821805Z creating: dist/ 2024-08-07T17:52:50.8385619Z inflating: dist/torch-2.5.0a0+git016588f-cp310-cp310-linux_x86_64.whl 2024-08-07T17:52:50.8387819Z creating: build/custom_test_artifacts/ 2024-08-07T17:52:50.8388963Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-07T17:52:50.8389649Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-07T17:52:50.8390368Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-07T17:52:50.8395778Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-07T17:52:50.8396609Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-07T17:52:50.8397422Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-07T17:52:50.8398285Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-07T17:52:50.8399138Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-07T17:52:50.8400455Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-07T17:52:50.8402850Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-07T17:52:50.8403767Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-07T17:52:50.8404641Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-07T17:52:50.8407067Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-07T17:52:50.8408655Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-07T17:52:50.8410968Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-07T17:52:50.8411979Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-07T17:52:50.8413682Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-07T17:52:50.8414719Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-07T17:52:50.8415649Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-07T17:52:50.8416532Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-07T17:52:50.8474360Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-07T17:52:50.8475584Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-07T17:52:50.8476895Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-07T17:52:50.8478307Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-07T17:52:50.8479709Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-07T17:52:50.8481009Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-07T17:52:50.8482325Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-07T17:52:50.8483670Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-07T17:52:50.8484990Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-07T17:52:50.8531675Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-07T17:52:50.8581579Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-07T17:52:50.8583325Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-07T17:52:50.8584760Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-07T17:52:50.8585879Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-07T17:52:50.8587370Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-07T17:52:50.8588469Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-07T17:52:50.8589518Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-07T17:52:50.8591078Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-07T17:52:50.8666157Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-07T17:52:50.8742114Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-07T17:52:50.8743179Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-07T17:52:50.8744037Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-07T17:52:50.8744797Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-07T17:52:50.8745591Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-07T17:52:50.8746394Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-07T17:52:50.8747530Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-07T17:52:50.8748554Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-07T17:52:50.8749544Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-07T17:52:50.8750436Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-07T17:52:50.8751371Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-07T17:52:50.8752317Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-07T17:52:50.8753252Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-07T17:52:50.8754206Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-07T17:52:50.8755152Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-07T17:52:50.8777465Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-07T17:52:50.8957253Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-07T17:52:50.8958157Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-07T17:52:50.8959375Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-07T17:52:50.8960487Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-07T17:52:50.8961501Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-07T17:52:50.8962459Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-07T17:52:50.8963465Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-07T17:52:50.8964453Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-07T17:52:50.8965457Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-07T17:52:50.8966454Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-07T17:52:50.8967431Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-07T17:52:50.8991259Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-07T17:52:50.9095635Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-07T17:52:50.9096702Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-07T17:52:50.9098021Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-07T17:52:50.9099112Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-07T17:52:50.9099938Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-07T17:52:50.9100970Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-07T17:52:50.9101773Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-07T17:52:50.9104239Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-07T17:52:50.9104939Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-07T17:52:50.9105878Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-07T17:52:50.9255218Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-07T17:52:50.9333846Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-07T17:52:50.9334500Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-07T17:52:50.9335108Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-07T17:52:50.9335803Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-07T17:52:50.9343545Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-07T17:52:50.9344434Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-07T17:52:50.9345301Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-07T17:52:50.9346239Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-07T17:52:50.9347368Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-07T17:52:50.9348357Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-07T17:52:50.9349837Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-07T17:52:50.9350710Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-07T17:52:50.9351574Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-07T17:52:50.9353842Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-07T17:52:50.9355550Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-07T17:52:50.9357808Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-07T17:52:50.9358809Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-07T17:52:50.9360514Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-07T17:52:50.9361534Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-07T17:52:50.9362440Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-07T17:52:50.9363312Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-07T17:52:50.9420906Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-07T17:52:50.9422132Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-07T17:52:50.9423367Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-07T17:52:50.9424639Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-07T17:52:50.9426091Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-07T17:52:50.9427640Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-07T17:52:50.9428874Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-07T17:52:50.9430078Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-07T17:52:50.9431254Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-07T17:52:50.9477456Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-07T17:52:50.9527562Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-07T17:52:50.9528774Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-07T17:52:50.9529876Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-07T17:52:50.9530951Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-07T17:52:50.9532011Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-07T17:52:50.9533083Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-07T17:52:50.9534092Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-07T17:52:50.9535150Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-07T17:52:50.9611112Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-07T17:52:50.9687306Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-07T17:52:50.9688318Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-07T17:52:50.9689178Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-07T17:52:50.9689936Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-07T17:52:50.9690695Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-07T17:52:50.9691515Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-07T17:52:50.9692446Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-07T17:52:50.9693497Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-07T17:52:50.9694483Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-07T17:52:50.9695420Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-07T17:52:50.9696383Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-07T17:52:50.9697333Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-07T17:52:50.9698307Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-07T17:52:50.9699273Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-07T17:52:50.9700412Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-07T17:52:50.9723804Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-07T17:52:50.9804479Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-07T17:52:50.9805536Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-07T17:52:50.9806458Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-07T17:52:50.9807326Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-07T17:52:50.9808108Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-07T17:52:50.9808894Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-07T17:52:50.9809662Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-07T17:52:50.9812331Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-07T17:52:50.9813044Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-07T17:52:50.9814024Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-07T17:52:50.9876360Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-07T17:52:50.9877030Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-07T17:52:50.9877690Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-07T17:52:50.9878477Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-07T17:52:50.9887168Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-07T17:52:50.9888024Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-07T17:52:50.9888900Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-07T17:52:50.9889822Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-07T17:52:50.9890738Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-07T17:52:50.9891745Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-07T17:52:50.9893433Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-07T17:52:50.9894370Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-07T17:52:50.9895278Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-07T17:52:50.9897139Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-07T17:52:50.9899008Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-07T17:52:50.9901305Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-07T17:52:50.9902360Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-07T17:52:50.9904149Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-07T17:52:50.9905256Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-07T17:52:50.9906230Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-07T17:52:50.9907524Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-07T17:52:50.9964859Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-07T17:52:50.9966539Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-07T17:52:50.9968002Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-07T17:52:50.9969348Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-07T17:52:50.9970692Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-07T17:52:50.9971964Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-07T17:52:50.9973232Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-07T17:52:50.9974515Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-07T17:52:50.9975794Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-07T17:52:51.0022308Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-07T17:52:51.0072327Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-07T17:52:51.0073861Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-07T17:52:51.0075065Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-07T17:52:51.0076222Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-07T17:52:51.0077383Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-07T17:52:51.0078531Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-07T17:52:51.0079632Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-07T17:52:51.0080759Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-07T17:52:51.0156040Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-07T17:52:51.0231886Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-07T17:52:51.0232990Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-07T17:52:51.0233935Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-07T17:52:51.0234751Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-07T17:52:51.0235589Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-07T17:52:51.0236482Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-07T17:52:51.0237487Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-07T17:52:51.0238609Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-07T17:52:51.0239670Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-07T17:52:51.0240684Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-07T17:52:51.0241979Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-07T17:52:51.0243165Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-07T17:52:51.0244249Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-07T17:52:51.0245298Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-07T17:52:51.0246336Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-07T17:52:51.0247428Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-07T17:52:51.0404912Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-07T17:52:51.0405962Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-07T17:52:51.0407002Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-07T17:52:51.0408186Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-07T17:52:51.0409326Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-07T17:52:51.0410391Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-07T17:52:51.0411475Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-07T17:52:51.0412581Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-07T17:52:51.0413685Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-07T17:52:51.0414797Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-07T17:52:51.0415877Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-07T17:52:51.0438176Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-07T17:52:51.0507700Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-07T17:52:51.0509129Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-07T17:52:51.0510130Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-07T17:52:51.0511055Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-07T17:52:51.0511918Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-07T17:52:51.0512745Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-07T17:52:51.0513595Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-07T17:52:51.0516291Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-07T17:52:51.0517055Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-07T17:52:51.0518017Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-07T17:52:51.0647500Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-07T17:52:51.0701106Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-07T17:52:51.0701694Z creating: build/lib/ 2024-08-07T17:52:51.0713595Z inflating: build/lib/libpthreadpool.a 2024-08-07T17:52:51.0723438Z inflating: build/lib/libcpuinfo_internals.a 2024-08-07T17:52:51.0733595Z inflating: build/lib/libcpuinfo.a 2024-08-07T17:52:51.0734752Z inflating: build/lib/libclog.a 2024-08-07T17:52:51.0846204Z inflating: build/lib/libprotobuf-lite.a 2024-08-07T17:52:51.0848845Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-07T17:52:51.1417005Z inflating: build/lib/libprotobuf.a 2024-08-07T17:52:51.1498067Z inflating: build/lib/libgtest.a 2024-08-07T17:52:51.1591959Z inflating: build/lib/libbenchmark.a 2024-08-07T17:52:51.1614099Z inflating: build/lib/libnnpack.a 2024-08-07T17:52:51.1693489Z inflating: build/lib/libasmjit.a 2024-08-07T17:52:51.1702816Z inflating: build/lib/libittnotify.a 2024-08-07T17:52:51.1737591Z inflating: build/lib/libtensorpipe_uv.a 2024-08-07T17:52:51.1896271Z inflating: build/lib/libgloo.a 2024-08-07T17:52:51.1922443Z inflating: build/lib/libfmt.a 2024-08-07T17:52:51.2040930Z inflating: build/lib/libc10.so 2024-08-07T17:52:51.2043606Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-07T17:52:51.2044395Z inflating: build/lib/libfoxi_loader.a 2024-08-07T17:52:51.2046547Z inflating: build/lib/libtorch_global_deps.so 2024-08-07T17:52:51.2070869Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-07T17:52:51.2093270Z inflating: build/lib/libgmock.a 2024-08-07T17:52:51.2721038Z inflating: build/lib/libprotoc.a 2024-08-07T17:52:51.2721739Z inflating: build/lib/libgtest_main.a 2024-08-07T17:52:51.2722665Z inflating: build/lib/libbenchmark_main.a 2024-08-07T17:52:52.5366503Z inflating: build/lib/libdnnl.a 2024-08-07T17:52:52.6080637Z inflating: build/lib/libtensorpipe.a 2024-08-07T17:52:52.6152016Z inflating: build/lib/libc10_cuda.so 2024-08-07T17:52:52.6152750Z inflating: build/lib/libgmock_main.a 2024-08-07T17:52:52.7713244Z inflating: build/lib/libfbgemm.a 2024-08-07T17:52:52.8342780Z inflating: build/lib/libkineto.a 2024-08-07T17:52:52.8661209Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-07T17:52:52.9145664Z inflating: build/lib/libgloo_cuda.a 2024-08-07T17:52:52.9375119Z inflating: build/lib/libXNNPACK.a 2024-08-07T17:52:52.9428157Z inflating: build/lib/libonnx_proto.a 2024-08-07T17:52:53.0290462Z inflating: build/lib/libonnx.a 2024-08-07T17:52:56.0458874Z inflating: build/lib/libtorch_cpu.so 2024-08-07T17:52:56.0463450Z inflating: build/lib/libshm.so 2024-08-07T17:52:56.0470008Z inflating: build/lib/libunbox_lib.a 2024-08-07T17:52:58.5206389Z inflating: build/lib/libtorch_cuda.so 2024-08-07T17:52:58.5207604Z inflating: build/lib/libtorch.so 2024-08-07T17:52:58.5210864Z inflating: build/lib/libc10d_cuda_test.so 2024-08-07T17:52:59.6598409Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-07T17:52:59.6623977Z inflating: build/lib/libjitbackend_test.so 2024-08-07T17:52:59.6713508Z inflating: build/lib/libtorchbind_test.so 2024-08-07T17:52:59.6745782Z inflating: build/lib/libaoti_custom_ops.so 2024-08-07T17:52:59.6777675Z inflating: build/lib/libbackend_with_compiler.so 2024-08-07T17:52:59.9260651Z inflating: build/lib/libtorch_python.so 2024-08-07T17:52:59.9303745Z inflating: build/lib/libnnapi_backend.so 2024-08-07T17:52:59.9304180Z creating: build/bin/ 2024-08-07T17:52:59.9366899Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-07T17:52:59.9431308Z inflating: build/bin/c10_DeviceGuard_test 2024-08-07T17:52:59.9494079Z inflating: build/bin/c10_Device_test 2024-08-07T17:52:59.9568015Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-07T17:52:59.9635905Z inflating: build/bin/c10_Scalar_test 2024-08-07T17:52:59.9697147Z inflating: build/bin/c10_StreamGuard_test 2024-08-07T17:52:59.9761088Z inflating: build/bin/c10_SymInt_test 2024-08-07T17:52:59.9829253Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-07T17:52:59.9898950Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-07T17:52:59.9969462Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-07T17:53:00.0058074Z inflating: build/bin/c10_cow_test 2024-08-07T17:53:00.0124911Z inflating: build/bin/c10_Bitset_test 2024-08-07T17:53:00.0186307Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-07T17:53:00.0248661Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-07T17:53:00.0312461Z inflating: build/bin/c10_Half_test 2024-08-07T17:53:00.0382775Z inflating: build/bin/c10_LeftRight_test 2024-08-07T17:53:00.0451870Z inflating: build/bin/c10_Metaprogramming_test 2024-08-07T17:53:00.0514583Z inflating: build/bin/c10_Synchronized_test 2024-08-07T17:53:00.0584084Z inflating: build/bin/c10_ThreadLocal_test 2024-08-07T17:53:00.0648772Z inflating: build/bin/c10_TypeIndex_test 2024-08-07T17:53:00.0712925Z inflating: build/bin/c10_TypeList_test 2024-08-07T17:53:00.0774112Z inflating: build/bin/c10_TypeTraits_test 2024-08-07T17:53:00.0839440Z inflating: build/bin/c10_accumulate_test 2024-08-07T17:53:00.0909002Z inflating: build/bin/c10_bfloat16_test 2024-08-07T17:53:00.0972134Z inflating: build/bin/c10_bit_cast_test 2024-08-07T17:53:00.1042895Z inflating: build/bin/c10_complex_math_test 2024-08-07T17:53:00.1108945Z inflating: build/bin/c10_exception_test 2024-08-07T17:53:00.1178694Z inflating: build/bin/c10_complex_test 2024-08-07T17:53:00.1241814Z inflating: build/bin/c10_flags_test 2024-08-07T17:53:00.1304328Z inflating: build/bin/c10_generic_math_test 2024-08-07T17:53:00.1368264Z inflating: build/bin/c10_irange_test 2024-08-07T17:53:00.1574659Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-07T17:53:00.1641886Z inflating: build/bin/c10_lazy_test 2024-08-07T17:53:00.1713852Z inflating: build/bin/c10_logging_test 2024-08-07T17:53:00.1807626Z inflating: build/bin/c10_optional_test 2024-08-07T17:53:00.1885751Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-07T17:53:00.1953236Z inflating: build/bin/c10_registry_test 2024-08-07T17:53:00.2142458Z inflating: build/bin/c10_small_vector_test 2024-08-07T17:53:00.2207216Z inflating: build/bin/c10_ssize_test 2024-08-07T17:53:00.2272362Z inflating: build/bin/c10_string_util_test 2024-08-07T17:53:00.2335382Z inflating: build/bin/c10_tempfile_test 2024-08-07T17:53:00.2408907Z inflating: build/bin/c10_string_view_test 2024-08-07T17:53:00.2479412Z inflating: build/bin/c10_typeid_test 2024-08-07T17:53:00.2539996Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-07T17:53:00.3098159Z inflating: build/bin/protoc-3.13.0.0 2024-08-07T17:53:00.3656010Z inflating: build/bin/protoc 2024-08-07T17:53:00.3722422Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-07T17:53:00.3788695Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-07T17:53:00.3854522Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-07T17:53:00.3919763Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-07T17:53:00.3985743Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-07T17:53:00.4047447Z inflating: build/bin/c10_cuda_CUDATest 2024-08-07T17:53:00.4114016Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-07T17:53:00.4179829Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-07T17:53:00.4610361Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-07T17:53:00.5022990Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-07T17:53:00.5473026Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-07T17:53:00.5539304Z inflating: build/bin/FileStoreTest 2024-08-07T17:53:00.5606213Z inflating: build/bin/HashStoreTest 2024-08-07T17:53:00.5672298Z inflating: build/bin/BackoffTest 2024-08-07T17:53:00.5742046Z inflating: build/bin/TCPStoreTest 2024-08-07T17:53:00.5759169Z inflating: build/bin/ProcessGroupMPITest 2024-08-07T17:53:00.5764410Z inflating: build/bin/torch_shm_manager 2024-08-07T17:53:00.5832161Z inflating: build/bin/test_edge_op_registration 2024-08-07T17:53:00.5835982Z inflating: build/bin/example_allreduce 2024-08-07T17:53:00.5901049Z inflating: build/bin/Dimname_test 2024-08-07T17:53:00.5993466Z inflating: build/bin/Dict_test 2024-08-07T17:53:00.6074384Z inflating: build/bin/MaybeOwned_test 2024-08-07T17:53:00.6149281Z inflating: build/bin/apply_utils_test 2024-08-07T17:53:00.6220984Z inflating: build/bin/NamedTensor_test 2024-08-07T17:53:00.6294843Z inflating: build/bin/atest 2024-08-07T17:53:00.6373923Z inflating: build/bin/basic 2024-08-07T17:53:00.6442783Z inflating: build/bin/broadcast_test 2024-08-07T17:53:00.6506231Z inflating: build/bin/cpu_allocator_test 2024-08-07T17:53:00.6579206Z inflating: build/bin/cpu_generator_test 2024-08-07T17:53:00.6646062Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-07T17:53:00.6762979Z inflating: build/bin/cpu_rng_test 2024-08-07T17:53:00.6825470Z inflating: build/bin/dispatch_key_set_test 2024-08-07T17:53:00.6888504Z inflating: build/bin/dlconvertor_test 2024-08-07T17:53:00.6961758Z inflating: build/bin/extension_backend_test 2024-08-07T17:53:00.7031040Z inflating: build/bin/half_test 2024-08-07T17:53:00.7151125Z inflating: build/bin/ivalue_test 2024-08-07T17:53:00.7213261Z inflating: build/bin/lazy_tensor_test 2024-08-07T17:53:00.7281440Z inflating: build/bin/math_kernel_test 2024-08-07T17:53:00.7349249Z inflating: build/bin/memory_format_test 2024-08-07T17:53:00.7415567Z inflating: build/bin/memory_overlapping_test 2024-08-07T17:53:00.7482419Z inflating: build/bin/mobile_memory_cleanup 2024-08-07T17:53:00.7545801Z inflating: build/bin/operator_name_test 2024-08-07T17:53:00.7615949Z inflating: build/bin/native_test 2024-08-07T17:53:00.7680748Z inflating: build/bin/operators_test 2024-08-07T17:53:00.7745850Z inflating: build/bin/packedtensoraccessor_test 2024-08-07T17:53:00.7830425Z inflating: build/bin/pow_test 2024-08-07T17:53:00.7901822Z inflating: build/bin/quantized_test 2024-08-07T17:53:00.7964637Z inflating: build/bin/reduce_ops_test 2024-08-07T17:53:00.8028785Z inflating: build/bin/reportMemoryUsage_test 2024-08-07T17:53:00.8099450Z inflating: build/bin/scalar_tensor_test 2024-08-07T17:53:00.8172881Z inflating: build/bin/scalar_test 2024-08-07T17:53:00.8237839Z inflating: build/bin/StorageUtils_test 2024-08-07T17:53:00.8303795Z inflating: build/bin/stride_properties_test 2024-08-07T17:53:00.8402683Z inflating: build/bin/tensor_iterator_test 2024-08-07T17:53:00.8406348Z inflating: build/bin/thread_init_test 2024-08-07T17:53:00.8476405Z inflating: build/bin/type_ptr_test 2024-08-07T17:53:00.8544121Z inflating: build/bin/test_parallel 2024-08-07T17:53:00.8619441Z inflating: build/bin/type_test 2024-08-07T17:53:00.8684907Z inflating: build/bin/undefined_tensor_test 2024-08-07T17:53:00.8686983Z inflating: build/bin/verify_api_visibility 2024-08-07T17:53:00.8773896Z inflating: build/bin/legacy_vmap_test 2024-08-07T17:53:00.8838607Z inflating: build/bin/wrapdim_test 2024-08-07T17:53:00.8903408Z inflating: build/bin/weakref_test 2024-08-07T17:53:00.8968110Z inflating: build/bin/xla_tensor_test 2024-08-07T17:53:00.9043495Z inflating: build/bin/IListRef_test 2024-08-07T17:53:00.9176957Z inflating: build/bin/List_test 2024-08-07T17:53:00.9329668Z inflating: build/bin/kernel_function_legacy_test 2024-08-07T17:53:00.9450996Z inflating: build/bin/kernel_function_test 2024-08-07T17:53:00.9534598Z inflating: build/bin/KernelFunction_test 2024-08-07T17:53:00.9695155Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-07T17:53:00.9824901Z inflating: build/bin/kernel_lambda_test 2024-08-07T17:53:00.9901691Z inflating: build/bin/kernel_stackbased_test 2024-08-07T17:53:01.0021983Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-07T17:53:01.0085974Z inflating: build/bin/CppSignature_test 2024-08-07T17:53:01.0147095Z inflating: build/bin/op_allowlist_test 2024-08-07T17:53:01.0217376Z inflating: build/bin/backend_fallback_test 2024-08-07T17:53:01.0595458Z inflating: build/bin/op_registration_test 2024-08-07T17:53:01.0675249Z inflating: build/bin/inline_container_test 2024-08-07T17:53:01.0741738Z inflating: build/bin/cuda_apply_test 2024-08-07T17:53:01.0806587Z inflating: build/bin/cuda_allocator_test 2024-08-07T17:53:01.0875487Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-07T17:53:01.0949480Z inflating: build/bin/cuda_atomic_ops_test 2024-08-07T17:53:01.1037701Z inflating: build/bin/cuda_complex_math_test 2024-08-07T17:53:01.1111660Z inflating: build/bin/cuda_complex_test 2024-08-07T17:53:01.1173616Z inflating: build/bin/cuda_device_test 2024-08-07T17:53:01.1246230Z inflating: build/bin/cuda_cub_test 2024-08-07T17:53:01.1310604Z inflating: build/bin/cuda_dlconvertor_test 2024-08-07T17:53:01.1392612Z inflating: build/bin/cuda_distributions_test 2024-08-07T17:53:01.1456501Z inflating: build/bin/cuda_integer_divider_test 2024-08-07T17:53:01.1519055Z inflating: build/bin/cuda_half_test 2024-08-07T17:53:01.1590094Z inflating: build/bin/cuda_generator_test 2024-08-07T17:53:01.1651836Z inflating: build/bin/cuda_optional_test 2024-08-07T17:53:01.1714374Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-07T17:53:01.1790371Z inflating: build/bin/cuda_stream_test 2024-08-07T17:53:01.1856175Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-07T17:53:01.1920879Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-07T17:53:01.1982685Z inflating: build/bin/cuda_cudnn_test 2024-08-07T17:53:01.2047847Z inflating: build/bin/cuda_vectorized_test 2024-08-07T17:53:01.2130128Z inflating: build/bin/ProcessGroupGlooTest 2024-08-07T17:53:01.2148342Z inflating: build/bin/tutorial_tensorexpr 2024-08-07T17:53:01.2220421Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-07T17:53:01.2300102Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-07T17:53:01.2377851Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-07T17:53:01.2444019Z inflating: build/bin/ProcessGroupUCCTest 2024-08-07T17:53:01.2512822Z inflating: build/bin/test_dist_autograd 2024-08-07T17:53:01.2516119Z inflating: build/bin/parallel_benchmark 2024-08-07T17:53:01.2602505Z inflating: build/bin/test_cpp_rpc 2024-08-07T17:53:01.2686551Z inflating: build/bin/test_mobile_nnc 2024-08-07T17:53:01.2697687Z inflating: build/bin/aot_model_compiler_test 2024-08-07T17:53:01.3131619Z inflating: build/bin/test_lazy 2024-08-07T17:53:01.4165115Z inflating: build/bin/test_tensorexpr 2024-08-07T17:53:01.5640861Z inflating: build/bin/test_api 2024-08-07T17:53:01.6365862Z inflating: build/bin/test_jit 2024-08-07T17:53:01.6366317Z creating: .additional_ci_files/ 2024-08-07T17:53:01.6438613Z inflating: .additional_ci_files/test-times.json 2024-08-07T17:53:01.6724852Z inflating: .additional_ci_files/test-class-times.json 2024-08-07T17:53:01.6766087Z ##[group]Run rm artifacts.zip 2024-08-07T17:53:01.6766542Z rm artifacts.zip 2024-08-07T17:53:01.6774821Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:01.6775312Z env: 2024-08-07T17:53:01.6775606Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:01.6776026Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:01.6776497Z ##[endgroup] 2024-08-07T17:53:01.7923826Z ##[group]Run df -H 2024-08-07T17:53:01.7924186Z df -H 2024-08-07T17:53:01.7932134Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:01.7932624Z env: 2024-08-07T17:53:01.7932922Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:01.7933335Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:01.7934037Z ##[endgroup] 2024-08-07T17:53:01.7967316Z Filesystem Size Used Avail Use% Mounted on 2024-08-07T17:53:01.7967792Z devtmpfs 129G 0 129G 0% /dev 2024-08-07T17:53:01.7968228Z tmpfs 129G 0 129G 0% /dev/shm 2024-08-07T17:53:01.7969066Z tmpfs 129G 594k 129G 1% /run 2024-08-07T17:53:01.7969568Z tmpfs 129G 0 129G 0% /sys/fs/cgroup 2024-08-07T17:53:01.7970018Z /dev/xvda1 162G 37G 125G 23% / 2024-08-07T17:53:01.7970454Z tmpfs 26G 0 26G 0% /run/user/0 2024-08-07T17:53:01.8009120Z Prepare all required actions 2024-08-07T17:53:01.8009638Z Getting action download info 2024-08-07T17:53:01.9715712Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-07T17:53:01.9716166Z with: 2024-08-07T17:53:01.9716441Z env: 2024-08-07T17:53:01.9716711Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:01.9717135Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:01.9717591Z ##[endgroup] 2024-08-07T17:53:01.9775611Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-07T17:53:01.9776085Z with: 2024-08-07T17:53:01.9776387Z name: td_results 2024-08-07T17:53:01.9776710Z s3-bucket: gha-artifacts 2024-08-07T17:53:01.9777091Z region: us-east-1 2024-08-07T17:53:01.9777413Z env: 2024-08-07T17:53:01.9777697Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:01.9778164Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:01.9778655Z ##[endgroup] 2024-08-07T17:53:02.5905663Z (node:71838) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-07T17:53:02.5906269Z 2024-08-07T17:53:02.5906543Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-07T17:53:02.5907542Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-07T17:53:02.5908235Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-07T17:53:02.6845185Z Found 1 objects with prefix pytorch/pytorch/10288745067/td_results/ 2024-08-07T17:53:02.6845982Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-08-07T17:53:02.7484570Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-08-07T17:53:02.7495320Z Artifact download has finished successfully 2024-08-07T17:53:02.7680420Z ##[group]Run mkdir -p .additional_ci_files 2024-08-07T17:53:02.7680902Z mkdir -p .additional_ci_files 2024-08-07T17:53:02.7681428Z mv td_results.json .additional_ci_files/td_results.json 2024-08-07T17:53:02.7689649Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:02.7690133Z env: 2024-08-07T17:53:02.7690411Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:02.7690843Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:02.7691299Z ##[endgroup] 2024-08-07T17:53:02.7785681Z ##[group]Run .github/scripts/parse_ref.py 2024-08-07T17:53:02.7786319Z .github/scripts/parse_ref.py 2024-08-07T17:53:02.7794703Z shell: /usr/bin/bash -e {0} 2024-08-07T17:53:02.7795060Z env: 2024-08-07T17:53:02.7795364Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:02.7795803Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:02.7796250Z ##[endgroup] 2024-08-07T17:53:02.8092178Z Prepare all required actions 2024-08-07T17:53:02.8136484Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-07T17:53:02.8136922Z with: 2024-08-07T17:53:02.8137431Z github-token: *** 2024-08-07T17:53:02.8137755Z env: 2024-08-07T17:53:02.8138049Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:02.8138464Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:02.8138927Z ##[endgroup] 2024-08-07T17:53:02.8166743Z ##[group]Run set -eux 2024-08-07T17:53:02.8167110Z set -eux 2024-08-07T17:53:02.8167654Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-07T17:53:02.8175073Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:02.8175772Z env: 2024-08-07T17:53:02.8176058Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:02.8176500Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:02.8177149Z GITHUB_TOKEN: *** 2024-08-07T17:53:02.8177468Z ##[endgroup] 2024-08-07T17:53:02.8200695Z + python3 .github/scripts/get_workflow_job_id.py 10288745067 i-0dcb26b5a502300df 2024-08-07T17:53:06.2922259Z setting job-id=28476145276 2024-08-07T17:53:06.2923008Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:53:06.3136350Z Prepare all required actions 2024-08-07T17:53:06.3137488Z Getting action download info 2024-08-07T17:53:06.4291674Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-07T17:53:06.4292106Z with: 2024-08-07T17:53:06.4292608Z github-token: *** 2024-08-07T17:53:06.4293888Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2024-08-07T17:53:06.4295512Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:53:06.4296135Z env: 2024-08-07T17:53:06.4296425Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:06.4296854Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:06.4297292Z ##[endgroup] 2024-08-07T17:53:06.4350134Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-07T17:53:06.4350668Z with: 2024-08-07T17:53:06.4350951Z shell: bash 2024-08-07T17:53:06.4351237Z timeout_minutes: 10 2024-08-07T17:53:06.4351569Z max_attempts: 5 2024-08-07T17:53:06.4351888Z retry_wait_seconds: 30 2024-08-07T17:53:06.4352823Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-07T17:53:06.4353825Z polling_interval_seconds: 1 2024-08-07T17:53:06.4354197Z warning_on_retry: true 2024-08-07T17:53:06.4354626Z continue_on_error: false 2024-08-07T17:53:06.4355004Z env: 2024-08-07T17:53:06.4355289Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:06.4355710Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:06.4356347Z GITHUB_TOKEN: *** 2024-08-07T17:53:06.4356665Z ##[endgroup] 2024-08-07T17:53:06.5088120Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-07T17:53:06.8018731Z Defaulting to user installation because normal site-packages is not writeable 2024-08-07T17:53:06.8226565Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-08-07T17:53:06.8412501Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-08-07T17:53:06.8425025Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-08-07T17:53:06.8452313Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-08-07T17:53:06.8702907Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.7.4) 2024-08-07T17:53:06.8714788Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-08-07T17:53:07.5085279Z Command completed after 1 attempt(s). 2024-08-07T17:53:07.5143923Z ##[group]Run set -x 2024-08-07T17:53:07.5144270Z set -x 2024-08-07T17:53:07.5144558Z  2024-08-07T17:53:07.5145045Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-07T17:53:07.5145645Z # in runner workspace 2024-08-07T17:53:07.5146333Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-07T17:53:07.5154564Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:07.5155044Z env: 2024-08-07T17:53:07.5155337Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:07.5155752Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:07.5156207Z ##[endgroup] 2024-08-07T17:53:07.5180715Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-07T17:53:07.5451468Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-07T17:53:07.5452003Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-07T17:53:07.5452445Z echo "Job name: ${JOB_NAME}" 2024-08-07T17:53:07.5452830Z  2024-08-07T17:53:07.5453303Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-07T17:53:07.5453897Z # in runner workspace 2024-08-07T17:53:07.5454440Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-07T17:53:07.5455045Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-07T17:53:07.5455462Z  --job-name "${JOB_NAME}" \ 2024-08-07T17:53:07.5456843Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" \ 2024-08-07T17:53:07.5458336Z  --selected-test-configs "" \ 2024-08-07T17:53:07.5458892Z  --pr-number "${PR_NUMBER}" \ 2024-08-07T17:53:07.5459476Z  --tag "${TAG}" \ 2024-08-07T17:53:07.5459978Z  --event-name "${EVENT_NAME}" \ 2024-08-07T17:53:07.5460458Z  --schedule "${SCHEDULE}" \ 2024-08-07T17:53:07.5461022Z  --branch "${HEAD_BRANCH}" 2024-08-07T17:53:07.5485938Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:07.5486439Z env: 2024-08-07T17:53:07.5486718Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:07.5487152Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:07.5487905Z GITHUB_TOKEN: *** 2024-08-07T17:53:07.5488493Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:53:07.5489124Z PR_NUMBER: 131248 2024-08-07T17:53:07.5489438Z TAG: 2024-08-07T17:53:07.5489730Z EVENT_NAME: pull_request 2024-08-07T17:53:07.5490069Z SCHEDULE: 2024-08-07T17:53:07.5490368Z HEAD_BRANCH: 2024-08-07T17:53:07.5490673Z ##[endgroup] 2024-08-07T17:53:07.5513719Z Workflow: pull 2024-08-07T17:53:07.5514313Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:53:07.8123444Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-07T17:53:07.9773938Z ##[group]Run echo "Filtered matrix:" 2024-08-07T17:53:07.9774406Z echo "Filtered matrix:" 2024-08-07T17:53:07.9775730Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" 2024-08-07T17:53:07.9777063Z  2024-08-07T17:53:07.9777354Z echo 2024-08-07T17:53:07.9777716Z echo "Is the current job unstable? False" 2024-08-07T17:53:07.9778158Z  2024-08-07T17:53:07.9778447Z echo 2024-08-07T17:53:07.9778779Z echo "Is keep-going label set? False" 2024-08-07T17:53:07.9779200Z  2024-08-07T17:53:07.9779486Z echo 2024-08-07T17:53:07.9779789Z echo "Renabled issues? " 2024-08-07T17:53:07.9787819Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:07.9788318Z env: 2024-08-07T17:53:07.9788785Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:07.9789224Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:07.9789690Z ##[endgroup] 2024-08-07T17:53:07.9812801Z Filtered matrix: 2024-08-07T17:53:07.9814032Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}]} 2024-08-07T17:53:07.9815470Z 2024-08-07T17:53:07.9815827Z Is the current job unstable? False 2024-08-07T17:53:07.9816119Z 2024-08-07T17:53:07.9816300Z Is keep-going label set? False 2024-08-07T17:53:07.9816539Z 2024-08-07T17:53:07.9816673Z Renabled issues? 2024-08-07T17:53:07.9873048Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-07T17:53:07.9873710Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-07T17:53:07.9880899Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:07.9881397Z env: 2024-08-07T17:53:07.9881677Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:07.9882111Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:07.9882572Z JOB_TIMEOUT: 360 2024-08-07T17:53:07.9882866Z ##[endgroup] 2024-08-07T17:53:07.9960771Z ##[group]Run set -x 2024-08-07T17:53:07.9961224Z set -x 2024-08-07T17:53:07.9961521Z  2024-08-07T17:53:07.9961927Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-07T17:53:07.9962435Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-07T17:53:07.9962927Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-07T17:53:07.9963398Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-07T17:53:07.9963798Z else 2024-08-07T17:53:07.9964123Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-07T17:53:07.9964530Z fi 2024-08-07T17:53:07.9964817Z  2024-08-07T17:53:07.9965240Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-07T17:53:07.9965919Z # TODO: Stop building test binaries as part of the build phase 2024-08-07T17:53:07.9966507Z # Used for GPU_FLAG since that doesn't play nice 2024-08-07T17:53:07.9967042Z # shellcheck disable=SC2086,SC2090 2024-08-07T17:53:07.9967466Z container_name=$(docker run \ 2024-08-07T17:53:07.9967875Z  ${GPU_FLAG:-} \ 2024-08-07T17:53:07.9968244Z  -e BUILD_ENVIRONMENT \ 2024-08-07T17:53:07.9968619Z  -e PR_NUMBER \ 2024-08-07T17:53:07.9968982Z  -e GITHUB_ACTIONS \ 2024-08-07T17:53:07.9969363Z  -e GITHUB_REPOSITORY \ 2024-08-07T17:53:07.9969739Z  -e GITHUB_WORKFLOW \ 2024-08-07T17:53:07.9970122Z  -e GITHUB_JOB \ 2024-08-07T17:53:07.9970483Z  -e GITHUB_RUN_ID \ 2024-08-07T17:53:07.9970839Z  -e GITHUB_RUN_NUMBER \ 2024-08-07T17:53:07.9971240Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-07T17:53:07.9971627Z  -e JOB_ID \ 2024-08-07T17:53:07.9971962Z  -e JOB_NAME \ 2024-08-07T17:53:07.9972315Z  -e BASE_SHA \ 2024-08-07T17:53:07.9972656Z  -e BRANCH \ 2024-08-07T17:53:07.9972975Z  -e SHA1 \ 2024-08-07T17:53:07.9973329Z  -e AWS_DEFAULT_REGION \ 2024-08-07T17:53:07.9973719Z  -e IN_WHEEL_TEST \ 2024-08-07T17:53:07.9974071Z  -e SHARD_NUMBER \ 2024-08-07T17:53:07.9974443Z  -e TEST_CONFIG \ 2024-08-07T17:53:07.9974814Z  -e NUM_TEST_SHARDS \ 2024-08-07T17:53:07.9975180Z  -e REENABLED_ISSUES \ 2024-08-07T17:53:07.9975576Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-07T17:53:07.9975989Z  -e VERBOSE_TEST_LOGS \ 2024-08-07T17:53:07.9976362Z  -e TEST_SHOWLOCALS \ 2024-08-07T17:53:07.9976745Z  -e NO_TEST_TIMEOUT \ 2024-08-07T17:53:07.9977115Z  -e NO_TD \ 2024-08-07T17:53:07.9977441Z  -e TD_DISTRIBUTED \ 2024-08-07T17:53:07.9977821Z  -e PR_LABELS \ 2024-08-07T17:53:07.9978385Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-07T17:53:07.9978800Z  -e SCCACHE_BUCKET \ 2024-08-07T17:53:07.9979184Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-07T17:53:07.9979580Z  -e XLA_CUDA \ 2024-08-07T17:53:07.9979948Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-07T17:53:07.9980418Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-07T17:53:07.9980892Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-07T17:53:07.9981344Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-07T17:53:07.9981785Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-07T17:53:07.9982188Z  -e DASHBOARD_TAG \ 2024-08-07T17:53:07.9982605Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-07T17:53:07.9983113Z  --security-opt seccomp=unconfined \ 2024-08-07T17:53:07.9983557Z  --cap-add=SYS_PTRACE \ 2024-08-07T17:53:07.9983924Z  --ipc=host \ 2024-08-07T17:53:07.9984289Z  --shm-size="${SHM_SIZE}" \ 2024-08-07T17:53:07.9984677Z  --tty \ 2024-08-07T17:53:07.9984976Z  --detach \ 2024-08-07T17:53:07.9985332Z  --name="${container_name}" \ 2024-08-07T17:53:07.9985849Z  --user jenkins \ 2024-08-07T17:53:07.9986298Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-07T17:53:07.9986808Z  -w /var/lib/jenkins/workspace \ 2024-08-07T17:53:07.9987513Z  "${DOCKER_IMAGE}" 2024-08-07T17:53:07.9987849Z ) 2024-08-07T17:53:07.9988235Z # Propagate download.pytorch.org IP to container 2024-08-07T17:53:07.9989051Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-07T17:53:07.9989902Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-07T17:53:07.9990726Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-07T17:53:07.9998149Z shell: /usr/bin/bash -e {0} 2024-08-07T17:53:07.9998505Z env: 2024-08-07T17:53:07.9998804Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:07.9999238Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:07.9999797Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2024-08-07T17:53:08.0000249Z PR_NUMBER: 131248 2024-08-07T17:53:08.0000595Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-07T17:53:08.0000986Z GITHUB_WORKFLOW: pull 2024-08-07T17:53:08.0001327Z GITHUB_JOB: test 2024-08-07T17:53:08.0001653Z GITHUB_RUN_ID: 10288745067 2024-08-07T17:53:08.0002020Z GITHUB_RUN_NUMBER: 234358 2024-08-07T17:53:08.0002380Z GITHUB_RUN_ATTEMPT: 1 2024-08-07T17:53:08.0002693Z JOB_ID: 28476145276 2024-08-07T17:53:08.0003280Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:53:08.0003936Z BRANCH: pull/131248 2024-08-07T17:53:08.0004290Z SHA1: 016588f53c6904b840aa56aa86f95460b4d9c996 2024-08-07T17:53:08.0004780Z BASE_SHA: 6ce09a9bb33e4011761558032e2165ad7b49fb68 2024-08-07T17:53:08.0005228Z TEST_CONFIG: distributed 2024-08-07T17:53:08.0005564Z SHARD_NUMBER: 2 2024-08-07T17:53:08.0005878Z NUM_TEST_SHARDS: 3 2024-08-07T17:53:08.0006216Z REENABLED_ISSUES: 2024-08-07T17:53:08.0006539Z CONTINUE_THROUGH_ERROR: False 2024-08-07T17:53:08.0006916Z VERBOSE_TEST_LOGS: False 2024-08-07T17:53:08.0007274Z TEST_SHOWLOCALS: False 2024-08-07T17:53:08.0007604Z NO_TEST_TIMEOUT: False 2024-08-07T17:53:08.0007941Z NO_TD: False 2024-08-07T17:53:08.0008254Z TD_DISTRIBUTED: False 2024-08-07T17:53:08.0008643Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-07T17:53:08.0009115Z SCCACHE_S3_KEY_PREFIX: pull 2024-08-07T17:53:08.0009476Z SHM_SIZE: 2g 2024-08-07T17:53:08.0010336Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:53:08.0011435Z XLA_CUDA: 2024-08-07T17:53:08.0011894Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-07T17:53:08.0012447Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-07T17:53:08.0012864Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-07T17:53:08.0013269Z DASHBOARD_TAG: 2024-08-07T17:53:08.0013572Z HUGGING_FACE_HUB_TOKEN: 2024-08-07T17:53:08.0013916Z ##[endgroup] 2024-08-07T17:53:08.0037171Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-08-07T17:53:08.0037651Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2024-08-07T17:53:08.0038103Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-07T17:53:08.0047345Z +++ nproc --ignore=2 2024-08-07T17:53:08.0062429Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10288745067 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:53:19.1685767Z + container_name=ddfc464047ce056b8e984eda0cf5fdb3a03b3de1ff51933a15b85473696b0056 2024-08-07T17:53:19.1688273Z + grep download.pytorch.org /etc/hosts 2024-08-07T17:53:19.1689602Z + docker exec -i ddfc464047ce056b8e984eda0cf5fdb3a03b3de1ff51933a15b85473696b0056 sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-07T17:53:19.2470067Z + echo DOCKER_CONTAINER_ID=ddfc464047ce056b8e984eda0cf5fdb3a03b3de1ff51933a15b85473696b0056 2024-08-07T17:53:19.2474192Z ++ echo dist/torch-2.5.0a0+git016588f-cp310-cp310-linux_x86_64.whl 2024-08-07T17:53:19.2476418Z + docker exec -t ddfc464047ce056b8e984eda0cf5fdb3a03b3de1ff51933a15b85473696b0056 sh -c 'pip install dist/torch-2.5.0a0+git016588f-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-07T17:53:19.7970418Z Processing ./dist/torch-2.5.0a0+git016588f-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git016588f) 2024-08-07T17:53:20.9273798Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (3.13.1) 2024-08-07T17:53:20.9280503Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (4.12.2) 2024-08-07T17:53:20.9285488Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (2.8.8) 2024-08-07T17:53:20.9290594Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (3.1.4) 2024-08-07T17:53:20.9295652Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (2024.6.1) 2024-08-07T17:53:20.9310175Z Requirement already satisfied: sympy>=1.13.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (1.13.1) 2024-08-07T17:53:20.9351142Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (3.3.0) 2024-08-07T17:53:20.9432363Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (1.21.2) 2024-08-07T17:53:20.9480162Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.0->torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (1.3.0) 2024-08-07T17:53:21.0769454Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (2.1.5) 2024-08-07T17:53:22.2319871Z Installing collected packages: torch 2024-08-07T17:53:34.9965753Z Successfully installed torch-2.5.0a0+git016588f 2024-08-07T17:53:35.0672727Z ++ dirname .ci/pytorch/test.sh 2024-08-07T17:53:35.0679383Z + source .ci/pytorch/common.sh 2024-08-07T17:53:35.0683036Z +++ dirname .ci/pytorch/common.sh 2024-08-07T17:53:35.0690718Z ++ source .ci/pytorch/common_utils.sh 2024-08-07T17:53:35.0692901Z +++ declare -f -t trap_add 2024-08-07T17:53:35.0700373Z ++ set -ex 2024-08-07T17:53:35.0700746Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-08-07T17:53:35.0701222Z ++ BUILD_TEST_LIBTORCH=0 2024-08-07T17:53:35.0702532Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-08-07T17:53:35.0705571Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-07T17:53:35.0717053Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-07T17:53:35.0717458Z + trap_add cleanup_workspace EXIT 2024-08-07T17:53:35.0717863Z + trap_add_cmd=cleanup_workspace 2024-08-07T17:53:35.0718231Z + shift 2024-08-07T17:53:35.0718512Z + for trap_add_name in "$@" 2024-08-07T17:53:35.0726093Z +++ trap -p EXIT 2024-08-07T17:53:35.0729041Z ++ eval 'extract_trap_cmd ' 2024-08-07T17:53:35.0729417Z +++ extract_trap_cmd 2024-08-07T17:53:35.0730039Z +++ printf '%s\n' '' 2024-08-07T17:53:35.0730428Z ++ printf '%s\n' cleanup_workspace 2024-08-07T17:53:35.0732258Z + trap -- ' 2024-08-07T17:53:35.0732676Z cleanup_workspace' EXIT 2024-08-07T17:53:35.0733069Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-07T17:53:35.5957363Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-07T17:53:35.5975455Z + echo 'Environment variables:' 2024-08-07T17:53:35.5975897Z Environment variables: 2024-08-07T17:53:35.5976270Z + env 2024-08-07T17:53:35.5983787Z INSTALLED_DB=yes 2024-08-07T17:53:35.5984423Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-08-07T17:53:35.5984835Z NVIDIA_VISIBLE_DEVICES=all 2024-08-07T17:53:35.5985204Z NV_NVML_DEV_VERSION=11.8.86-1 2024-08-07T17:53:35.5985990Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-07T17:53:35.5986559Z CONTINUE_THROUGH_ERROR=False 2024-08-07T17:53:35.5987426Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-08-07T17:53:35.5987892Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-08-07T17:53:35.5988654Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-08-07T17:53:35.5989155Z HOSTNAME=ddfc464047ce 2024-08-07T17:53:35.5990136Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_171581d4-7fd8-41e6-b0a6-dc494026081c 2024-08-07T17:53:35.5991047Z GITHUB_ACTION=__self 2024-08-07T17:53:35.5991542Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-07T17:53:35.5993837Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-08-07T17:53:35.5996363Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-08-07T17:53:35.5997034Z NV_NVTX_VERSION=11.8.86-1 2024-08-07T17:53:35.5997398Z GITHUB_RUN_NUMBER=234358 2024-08-07T17:53:35.5997912Z TEST_CONFIG=distributed 2024-08-07T17:53:35.5998348Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-07T17:53:35.5999181Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-07T17:53:35.5999769Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-08-07T17:53:35.6000181Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-08-07T17:53:35.6000772Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-08-07T17:53:35.6001205Z GITHUB_TRIGGERING_ACTOR=zdevito 2024-08-07T17:53:35.6001641Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-07T17:53:35.6002372Z GITHUB_REF_TYPE=branch 2024-08-07T17:53:35.6002706Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-07T17:53:35.6003090Z NCCL_VERSION=2.16.2-1 2024-08-07T17:53:35.6003699Z BASE_SHA=6ce09a9bb33e4011761558032e2165ad7b49fb68 2024-08-07T17:53:35.6004113Z XLA_CUDA= 2024-08-07T17:53:35.6004420Z HUGGING_FACE_HUB_TOKEN= 2024-08-07T17:53:35.6005165Z *** 2024-08-07T17:53:35.6005448Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-07T17:53:35.6005883Z GITHUB_REPOSITORY_ID=65600975 2024-08-07T17:53:35.6006470Z GITHUB_ACTIONS=true 2024-08-07T17:53:35.6006792Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:35.6007290Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-08-07T17:53:35.6007978Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-08-07T17:53:35.6008416Z SHA1=016588f53c6904b840aa56aa86f95460b4d9c996 2024-08-07T17:53:35.6009116Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-07T17:53:35.6009751Z GITHUB_SHA=f779f6b7738020e244184bded4026b37de3f9f24 2024-08-07T17:53:35.6010682Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/131248/merge 2024-08-07T17:53:35.6011320Z UCC_HOME=/usr 2024-08-07T17:53:35.6011861Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-08-07T17:53:35.6012248Z VERBOSE_TEST_LOGS=False 2024-08-07T17:53:35.6012594Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-07T17:53:35.6013254Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-08-07T17:53:35.6013756Z GITHUB_REF=refs/pull/131248/merge 2024-08-07T17:53:35.6014262Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-08-07T17:53:35.6014762Z SHARD_NUMBER=2 2024-08-07T17:53:35.6015081Z GITHUB_REF_PROTECTED=false 2024-08-07T17:53:35.6015483Z HOME=/var/lib/jenkins 2024-08-07T17:53:35.6016041Z GITHUB_API_URL=https://api.github.com 2024-08-07T17:53:35.6016472Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-07T17:53:35.6017076Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-07T17:53:35.6017616Z SCCACHE_S3_KEY_PREFIX=pull 2024-08-07T17:53:35.6017978Z CUDA_VERSION=11.8.0 2024-08-07T17:53:35.6018562Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-08-07T17:53:35.6019015Z NUM_TEST_SHARDS=3 2024-08-07T17:53:35.6019306Z UCX_HOME=/usr 2024-08-07T17:53:35.6020003Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-08-07T17:53:35.6021066Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_171581d4-7fd8-41e6-b0a6-dc494026081c 2024-08-07T17:53:35.6022305Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:53:35.6023480Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_171581d4-7fd8-41e6-b0a6-dc494026081c 2024-08-07T17:53:35.6024726Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-07T17:53:35.6025635Z GITHUB_EVENT_NAME=pull_request 2024-08-07T17:53:35.6025998Z DASHBOARD_TAG= 2024-08-07T17:53:35.6026428Z GITHUB_RUN_ID=10288745067 2024-08-07T17:53:35.6027240Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-08-07T17:53:35.6027856Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-08-07T17:53:35.6028842Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_171581d4-7fd8-41e6-b0a6-dc494026081c 2024-08-07T17:53:35.6029937Z GITHUB_ACTOR=zdevito 2024-08-07T17:53:35.6030268Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-08-07T17:53:35.6030903Z PR_NUMBER=131248 2024-08-07T17:53:35.6031230Z GITHUB_RUN_ATTEMPT=1 2024-08-07T17:53:35.6031557Z ANACONDA_PYTHON_VERSION=3.10 2024-08-07T17:53:35.6032241Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-07T17:53:35.6032702Z TERM=xterm 2024-08-07T17:53:35.6033017Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-08-07T17:53:35.6033823Z INSTALLED_VISION=yes 2024-08-07T17:53:35.6034139Z BRANCH=pull/131248 2024-08-07T17:53:35.6034567Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-07T17:53:35.6035129Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-07T17:53:35.6035537Z CUDA_PATH=/usr/local/cuda 2024-08-07T17:53:35.6036442Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-07T17:53:35.6037250Z GITHUB_SERVER_URL=https://github.com 2024-08-07T17:53:35.6037871Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-07T17:53:35.6038323Z REENABLED_ISSUES= 2024-08-07T17:53:35.6038802Z SHLVL=1 2024-08-07T17:53:35.6039145Z MAX_JOBS=30 2024-08-07T17:53:35.6039451Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-08-07T17:53:35.6039822Z NVARCH=x86_64 2024-08-07T17:53:35.6040339Z GITHUB_ACTOR_ID=370202 2024-08-07T17:53:35.6040780Z GITHUB_WORKFLOW_SHA=f779f6b7738020e244184bded4026b37de3f9f24 2024-08-07T17:53:35.6041367Z GITHUB_REF_NAME=131248/merge 2024-08-07T17:53:35.6041914Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-08-07T17:53:35.6042481Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-07T17:53:35.6043188Z GITHUB_JOB=test 2024-08-07T17:53:35.6043609Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-08-07T17:53:35.6044275Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-07T17:53:35.6044782Z NO_TEST_TIMEOUT=False 2024-08-07T17:53:35.6045100Z TD_DISTRIBUTED=False 2024-08-07T17:53:35.6045592Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-08-07T17:53:35.6046150Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-07T17:53:35.6046532Z NV_NVPROF_VERSION=11.8.87-1 2024-08-07T17:53:35.6046896Z GITHUB_RETENTION_DAYS=90 2024-08-07T17:53:35.6047245Z OPENSSL_DIR=/opt/openssl 2024-08-07T17:53:35.6047585Z GITHUB_ACTION_REPOSITORY= 2024-08-07T17:53:35.6048515Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-07T17:53:35.6049594Z GITHUB_BASE_REF=gh/zdevito/267/base 2024-08-07T17:53:35.6050162Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-07T17:53:35.6050531Z CI=true 2024-08-07T17:53:35.6050836Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-08-07T17:53:35.6051226Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-07T17:53:35.6051605Z JOB_ID=28476145276 2024-08-07T17:53:35.6051926Z INSTALLED_PROTOBUF=yes 2024-08-07T17:53:35.6052268Z GITHUB_HEAD_REF=gh/zdevito/267/head 2024-08-07T17:53:35.6052657Z GITHUB_ACTION_REF= 2024-08-07T17:53:35.6053025Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-07T17:53:35.6053484Z TEST_SHOWLOCALS=False 2024-08-07T17:53:35.6053821Z GITHUB_WORKFLOW=pull 2024-08-07T17:53:35.6054151Z DEBIAN_FRONTEND=noninteractive 2024-08-07T17:53:35.6054922Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_171581d4-7fd8-41e6-b0a6-dc494026081c 2024-08-07T17:53:35.6055698Z NO_TD=False 2024-08-07T17:53:35.6055995Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-07T17:53:35.6056359Z _=/usr/bin/env 2024-08-07T17:53:35.6056899Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-07T17:53:35.6205802Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-07T17:53:35.6206604Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-07T17:53:35.6207617Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-07T17:53:35.6208611Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-07T17:53:35.6209175Z + BUILD_DIR=build 2024-08-07T17:53:35.6209608Z + BUILD_RENAMED_DIR=build_renamed 2024-08-07T17:53:35.6210169Z + BUILD_BIN_DIR=build/bin 2024-08-07T17:53:35.6210498Z + SHARD_NUMBER=2 2024-08-07T17:53:35.6210838Z + NUM_TEST_SHARDS=3 2024-08-07T17:53:35.6211394Z + export VALGRIND=ON 2024-08-07T17:53:35.6211711Z + VALGRIND=ON 2024-08-07T17:53:35.6212091Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2024-08-07T17:53:35.6212800Z + [[ 0 == \1 ]] 2024-08-07T17:53:35.6213270Z + [[ False == \1 ]] 2024-08-07T17:53:35.6213739Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2024-08-07T17:53:35.6214400Z ++ realpath build/custom_test_artifacts 2024-08-07T17:53:35.6222271Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-07T17:53:35.6223112Z + [[ -n '' ]] 2024-08-07T17:53:35.6223490Z + echo 'Environment variables' 2024-08-07T17:53:35.6223854Z Environment variables 2024-08-07T17:53:35.6224303Z + env 2024-08-07T17:53:35.6230701Z INSTALLED_DB=yes 2024-08-07T17:53:35.6231203Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-08-07T17:53:35.6231601Z NVIDIA_VISIBLE_DEVICES=all 2024-08-07T17:53:35.6232120Z NV_NVML_DEV_VERSION=11.8.86-1 2024-08-07T17:53:35.6232743Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-07T17:53:35.6233403Z CONTINUE_THROUGH_ERROR=False 2024-08-07T17:53:35.6234001Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-08-07T17:53:35.6234461Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-08-07T17:53:35.6235218Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-08-07T17:53:35.6235690Z HOSTNAME=ddfc464047ce 2024-08-07T17:53:35.6236659Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_171581d4-7fd8-41e6-b0a6-dc494026081c 2024-08-07T17:53:35.6237788Z GITHUB_ACTION=__self 2024-08-07T17:53:35.6238204Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-07T17:53:35.6240592Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-08-07T17:53:35.6242927Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-08-07T17:53:35.6243554Z NV_NVTX_VERSION=11.8.86-1 2024-08-07T17:53:35.6243913Z GITHUB_RUN_NUMBER=234358 2024-08-07T17:53:35.6244442Z TEST_CONFIG=distributed 2024-08-07T17:53:35.6244868Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-07T17:53:35.6245292Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-07T17:53:35.6245942Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-08-07T17:53:35.6246368Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-08-07T17:53:35.6246789Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-08-07T17:53:35.6247379Z GITHUB_TRIGGERING_ACTOR=zdevito 2024-08-07T17:53:35.6247814Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-07T17:53:35.6248406Z GITHUB_REF_TYPE=branch 2024-08-07T17:53:35.6248851Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-07T17:53:35.6249216Z NCCL_VERSION=2.16.2-1 2024-08-07T17:53:35.6249702Z BASE_SHA=6ce09a9bb33e4011761558032e2165ad7b49fb68 2024-08-07T17:53:35.6250258Z XLA_CUDA= 2024-08-07T17:53:35.6250557Z HUGGING_FACE_HUB_TOKEN= 2024-08-07T17:53:35.6251144Z *** 2024-08-07T17:53:35.6251514Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-07T17:53:35.6251909Z GITHUB_REPOSITORY_ID=65600975 2024-08-07T17:53:35.6252344Z GITHUB_ACTIONS=true 2024-08-07T17:53:35.6252854Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:35.6253279Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-08-07T17:53:35.6253911Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-08-07T17:53:35.6254434Z SHA1=016588f53c6904b840aa56aa86f95460b4d9c996 2024-08-07T17:53:35.6254892Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-07T17:53:35.6255574Z GITHUB_SHA=f779f6b7738020e244184bded4026b37de3f9f24 2024-08-07T17:53:35.6256250Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/131248/merge 2024-08-07T17:53:35.6257116Z UCC_HOME=/usr 2024-08-07T17:53:35.6257420Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-08-07T17:53:35.6258032Z VERBOSE_TEST_LOGS=False 2024-08-07T17:53:35.6258628Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-07T17:53:35.6259016Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-08-07T17:53:35.6259686Z GITHUB_REF=refs/pull/131248/merge 2024-08-07T17:53:35.6260117Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-08-07T17:53:35.6260467Z SHARD_NUMBER=2 2024-08-07T17:53:35.6261183Z GITHUB_REF_PROTECTED=false 2024-08-07T17:53:35.6261546Z HOME=/var/lib/jenkins 2024-08-07T17:53:35.6261897Z GITHUB_API_URL=https://api.github.com 2024-08-07T17:53:35.6262580Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-07T17:53:35.6263043Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-07T17:53:35.6263637Z SCCACHE_S3_KEY_PREFIX=pull 2024-08-07T17:53:35.6264115Z CUDA_VERSION=11.8.0 2024-08-07T17:53:35.6264528Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-08-07T17:53:35.6265190Z NUM_TEST_SHARDS=3 2024-08-07T17:53:35.6265509Z UCX_HOME=/usr 2024-08-07T17:53:35.6265953Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-08-07T17:53:35.6267429Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_171581d4-7fd8-41e6-b0a6-dc494026081c 2024-08-07T17:53:35.6268735Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:53:35.6270017Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_171581d4-7fd8-41e6-b0a6-dc494026081c 2024-08-07T17:53:35.6271259Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-07T17:53:35.6272200Z GITHUB_EVENT_NAME=pull_request 2024-08-07T17:53:35.6272708Z DASHBOARD_TAG= 2024-08-07T17:53:35.6273260Z GITHUB_RUN_ID=10288745067 2024-08-07T17:53:35.6273688Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-08-07T17:53:35.6274167Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-08-07T17:53:35.6275282Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_171581d4-7fd8-41e6-b0a6-dc494026081c 2024-08-07T17:53:35.6276087Z GITHUB_ACTOR=zdevito 2024-08-07T17:53:35.6276436Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-08-07T17:53:35.6276818Z PR_NUMBER=131248 2024-08-07T17:53:35.6277302Z GITHUB_RUN_ATTEMPT=1 2024-08-07T17:53:35.6277699Z VALGRIND=ON 2024-08-07T17:53:35.6278012Z ANACONDA_PYTHON_VERSION=3.10 2024-08-07T17:53:35.6278441Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-07T17:53:35.6278900Z TERM=xterm 2024-08-07T17:53:35.6279220Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-08-07T17:53:35.6279603Z INSTALLED_VISION=yes 2024-08-07T17:53:35.6279931Z BRANCH=pull/131248 2024-08-07T17:53:35.6280250Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-07T17:53:35.6280641Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-07T17:53:35.6281125Z CUDA_PATH=/usr/local/cuda 2024-08-07T17:53:35.6281946Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-07T17:53:35.6282661Z GITHUB_SERVER_URL=https://github.com 2024-08-07T17:53:35.6283121Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-07T17:53:35.6283551Z REENABLED_ISSUES= 2024-08-07T17:53:35.6283859Z SHLVL=1 2024-08-07T17:53:35.6284140Z MAX_JOBS=30 2024-08-07T17:53:35.6284425Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-08-07T17:53:35.6284776Z NVARCH=x86_64 2024-08-07T17:53:35.6285080Z GITHUB_ACTOR_ID=370202 2024-08-07T17:53:35.6285502Z GITHUB_WORKFLOW_SHA=f779f6b7738020e244184bded4026b37de3f9f24 2024-08-07T17:53:35.6285992Z GITHUB_REF_NAME=131248/merge 2024-08-07T17:53:35.6286376Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-08-07T17:53:35.6286920Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-07T17:53:35.6287466Z GITHUB_JOB=test 2024-08-07T17:53:35.6287820Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-08-07T17:53:35.6288394Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-07T17:53:35.6289095Z NO_TEST_TIMEOUT=False 2024-08-07T17:53:35.6289433Z TD_DISTRIBUTED=False 2024-08-07T17:53:35.6289771Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-08-07T17:53:35.6290195Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-07T17:53:35.6290586Z NV_NVPROF_VERSION=11.8.87-1 2024-08-07T17:53:35.6290932Z GITHUB_RETENTION_DAYS=90 2024-08-07T17:53:35.6291282Z OPENSSL_DIR=/opt/openssl 2024-08-07T17:53:35.6291643Z GITHUB_ACTION_REPOSITORY= 2024-08-07T17:53:35.6292554Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-07T17:53:35.6293670Z GITHUB_BASE_REF=gh/zdevito/267/base 2024-08-07T17:53:35.6294081Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-07T17:53:35.6294434Z CI=true 2024-08-07T17:53:35.6294743Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-08-07T17:53:35.6295152Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-07T17:53:35.6295503Z JOB_ID=28476145276 2024-08-07T17:53:35.6295824Z INSTALLED_PROTOBUF=yes 2024-08-07T17:53:35.6296179Z GITHUB_HEAD_REF=gh/zdevito/267/head 2024-08-07T17:53:35.6296555Z GITHUB_ACTION_REF= 2024-08-07T17:53:35.6296942Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-07T17:53:35.6297382Z TEST_SHOWLOCALS=False 2024-08-07T17:53:35.6297724Z GITHUB_WORKFLOW=pull 2024-08-07T17:53:35.6298075Z DEBIAN_FRONTEND=noninteractive 2024-08-07T17:53:35.6298826Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_171581d4-7fd8-41e6-b0a6-dc494026081c 2024-08-07T17:53:35.6299614Z NO_TD=False 2024-08-07T17:53:35.6299927Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-07T17:53:35.6300281Z _=/usr/bin/env 2024-08-07T17:53:35.6300599Z + echo 'Testing pytorch' 2024-08-07T17:53:35.6300944Z Testing pytorch 2024-08-07T17:53:35.6301237Z + export LANG=C.UTF-8 2024-08-07T17:53:35.6301656Z + LANG=C.UTF-8 2024-08-07T17:53:35.6301999Z + PR_NUMBER=131248 2024-08-07T17:53:35.6302549Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-08-07T17:53:35.6302977Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-07T17:53:35.6303442Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-08-07T17:53:35.6303881Z + [[ distributed == \s\l\o\w ]] 2024-08-07T17:53:35.6304344Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2024-08-07T17:53:35.6304883Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-08-07T17:53:35.6305347Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-07T17:53:35.6305797Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-07T17:53:35.6306207Z + [[ distributed == *crossref* ]] 2024-08-07T17:53:35.6306625Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-08-07T17:53:35.6307411Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-08-07T17:53:35.6307936Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-08-07T17:53:35.6308397Z + pip_install --user ninja==1.10.2 2024-08-07T17:53:35.6308862Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-07T17:53:36.1549024Z Collecting ninja==1.10.2 2024-08-07T17:53:36.1748860Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-07T17:53:36.1932634Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-07T17:53:37.4479740Z Installing collected packages: ninja 2024-08-07T17:53:37.4581475Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-07T17:53:37.4582617Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-07T17:53:37.4638635Z Successfully installed ninja-1.10.2 2024-08-07T17:53:37.5342742Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-07T17:53:37.5344585Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-07T17:53:37.5345737Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-08-07T17:53:37.5346209Z + install_tlparse 2024-08-07T17:53:37.5346560Z + pip_install --user tlparse==0.3.7 2024-08-07T17:53:37.5347306Z + pip install --progress-bar off --user tlparse==0.3.7 2024-08-07T17:53:38.0510133Z Collecting tlparse==0.3.7 2024-08-07T17:53:38.0705769Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-08-07T17:53:38.0806042Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-07T17:53:39.3282689Z Installing collected packages: tlparse 2024-08-07T17:53:39.3771990Z Successfully installed tlparse-0.3.7 2024-08-07T17:53:39.4470881Z ++ python -m site --user-base 2024-08-07T17:53:39.4697367Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-07T17:53:39.4699371Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2024-08-07T17:53:39.4699941Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2024-08-07T17:53:39.4700448Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-08-07T17:53:39.4701149Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2024-08-07T17:53:39.4702002Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2024-08-07T17:53:39.4703336Z + cd test 2024-08-07T17:53:39.4704085Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-07T17:53:41.6105072Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-07T17:53:41.6105957Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-07T17:53:41.6111689Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-07T17:53:41.6112149Z + [[ distributed == *dynamo_eager* ]] 2024-08-07T17:53:41.6112580Z + [[ distributed == *aot_eager* ]] 2024-08-07T17:53:41.6112960Z + [[ distributed == *aot_inductor* ]] 2024-08-07T17:53:41.6113376Z + [[ distributed == *inductor* ]] 2024-08-07T17:53:41.6113761Z + [[ distributed == *dynamic* ]] 2024-08-07T17:53:41.6114125Z + [[ distributed == *cpu* ]] 2024-08-07T17:53:41.6116345Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-07T17:53:41.6154092Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-08-07T17:53:41.6154623Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2024-08-07T17:53:41.6157038Z + cd test 2024-08-07T17:53:41.6157441Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-07T17:53:43.5559909Z PyTorch built with: 2024-08-07T17:53:43.5560274Z - GCC 9.4 2024-08-07T17:53:43.5560581Z - C++ Version: 201703 2024-08-07T17:53:43.5561340Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-07T17:53:43.5562202Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-07T17:53:43.5562769Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-07T17:53:43.5563211Z - LAPACK is enabled (usually provided by MKL) 2024-08-07T17:53:43.5563628Z - NNPACK is enabled 2024-08-07T17:53:43.5563978Z - CPU capability usage: AVX2 2024-08-07T17:53:43.5564344Z - CUDA Runtime 11.8 2024-08-07T17:53:43.5564774Z - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52 2024-08-07T17:53:43.5565272Z - CuDNN 90.1 2024-08-07T17:53:43.5565618Z - Magma 2.6.1 2024-08-07T17:53:43.5571641Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-07T17:53:43.5577812Z 2024-08-07T17:53:43.8115723Z + cd test 2024-08-07T17:53:43.8116535Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-07T17:53:45.7441382Z ATen/Parallel: 2024-08-07T17:53:45.7441812Z at::get_num_threads() : 16 2024-08-07T17:53:45.7442230Z at::get_num_interop_threads() : 16 2024-08-07T17:53:45.7442628Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-07T17:53:45.7443021Z omp_get_max_threads() : 16 2024-08-07T17:53:45.7443726Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-07T17:53:45.7444437Z mkl_get_max_threads() : 16 2024-08-07T17:53:45.7444945Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-07T17:53:45.7445521Z std::thread::hardware_concurrency() : 32 2024-08-07T17:53:45.7445960Z Environment variables: 2024-08-07T17:53:45.7446313Z OMP_NUM_THREADS : [not set] 2024-08-07T17:53:45.7446681Z MKL_NUM_THREADS : [not set] 2024-08-07T17:53:45.7447033Z ATen parallel backend: OpenMP 2024-08-07T17:53:45.7447638Z 2024-08-07T17:53:45.9952463Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-08-07T17:53:45.9952995Z + [[ distributed == *backward* ]] 2024-08-07T17:53:45.9953378Z + [[ distributed == *xla* ]] 2024-08-07T17:53:45.9953759Z + [[ distributed == *executorch* ]] 2024-08-07T17:53:45.9954180Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-07T17:53:45.9954637Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-08-07T17:53:45.9955117Z + [[ distributed == distributed ]] 2024-08-07T17:53:45.9955497Z + test_distributed 2024-08-07T17:53:45.9955827Z + echo 'Testing distributed python tests' 2024-08-07T17:53:45.9956260Z Testing distributed python tests 2024-08-07T17:53:45.9956762Z + python test/run_test.py --distributed-tests --shard 2 3 --verbose 2024-08-07T17:53:46.1250811Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-07T17:53:46.1251821Z import pkg_resources 2024-08-07T17:53:50.6634835Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-08-07T17:53:50.7408814Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-07T17:53:50.7983443Z Ignoring disabled issues: [''] 2024-08-07T17:53:50.8123288Z Found test times from artifacts 2024-08-07T17:53:50.8648874Z Found test times from artifacts 2024-08-07T17:53:50.8664784Z Running 25% of tests based on TD 2024-08-07T17:53:50.8788989Z Running parallel tests on 2 processes 2024-08-07T17:53:50.8791629Z Name: tests to run (est. time: 11.57min) 2024-08-07T17:53:50.8792085Z Serial tests (25): 2024-08-07T17:53:50.8792461Z distributed/_tools/test_mod_tracker 1/1 2024-08-07T17:53:50.8792925Z distributed/pipelining/test_backward 1/1 2024-08-07T17:53:50.8793396Z distributed/_shard/sharded_tensor/test_logger 1/1 2024-08-07T17:53:50.8793896Z distributed/test_symmetric_memory 1/1 2024-08-07T17:53:50.8794405Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 2024-08-07T17:53:50.8794902Z distributed/test_dynamo_distributed 1/1 2024-08-07T17:53:50.8795347Z distributed/_tools/test_mem_tracker 1/1 2024-08-07T17:53:50.8795839Z distributed/pipelining/test_schedule_multiproc 1/1 2024-08-07T17:53:50.8796332Z distributed/_tensor/test_convolution_ops 1/1 2024-08-07T17:53:50.8796797Z distributed/_tensor/test_attention 1/1 2024-08-07T17:53:50.8797228Z distributed/_tensor/test_dtensor 1/1 2024-08-07T17:53:50.8797629Z distributed/_tensor/test_api 1/1 2024-08-07T17:53:50.8798054Z distributed/_composable/test_contract 1/1 2024-08-07T17:53:50.8798801Z distributed/test_device_mesh 1/1 2024-08-07T17:53:50.8799193Z distributed/pipelining/test_pipe 1/1 2024-08-07T17:53:50.8799637Z distributed/test_c10d_functional_native 1/1 2024-08-07T17:53:50.8800096Z distributed/_tensor/test_tensor_ops 1/1 2024-08-07T17:53:50.8800516Z distributed/_tensor/test_math_ops 1/1 2024-08-07T17:53:50.8800959Z distributed/pipelining/test_unflatten 1/1 2024-08-07T17:53:50.8801419Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 2024-08-07T17:53:50.8801864Z distributed/pipelining/test_microbatch 1/1 2024-08-07T17:53:50.8802326Z distributed/optim/test_named_optimizer 1/1 2024-08-07T17:53:50.8802826Z distributed/_tensor/debug/test_comm_mode_features 1/1 2024-08-07T17:53:50.8803312Z distributed/checkpoint/test_traverse 1/1 2024-08-07T17:53:50.8803774Z distributed/test_compute_comm_reordering 1/1 2024-08-07T17:53:50.8804212Z Parallel tests (0): 2024-08-07T17:53:50.8804589Z Name: excluded (est. time: 72.59min) 2024-08-07T17:53:50.8804965Z Serial tests (55): 2024-08-07T17:53:50.8805322Z distributed/test_distributed_spawn 2/9 2024-08-07T17:53:50.8805762Z distributed/test_distributed_spawn 5/9 2024-08-07T17:53:50.8806181Z distributed/test_distributed_spawn 8/9 2024-08-07T17:53:50.8806732Z distributed/test_c10d_nccl 1/2 2024-08-07T17:53:50.8807166Z distributed/fsdp/test_fsdp_state_dict 1/2 2024-08-07T17:53:50.8807605Z distributed/fsdp/test_fsdp_optim_state 1/1 2024-08-07T17:53:50.8808106Z distributed/optim/test_zero_redundancy_optimizer 1/1 2024-08-07T17:53:50.8808670Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-08-07T17:53:50.8809164Z distributed/fsdp/test_wrap 1/1 2024-08-07T17:53:50.8809563Z distributed/test_c10d_common 1/1 2024-08-07T17:53:50.8809986Z distributed/pipelining/test_schedule 1/1 2024-08-07T17:53:50.8810449Z distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 2024-08-07T17:53:50.8810952Z distributed/fsdp/test_fsdp_ignored_modules 1/1 2024-08-07T17:53:50.8811514Z distributed/algorithms/quantization/test_quantization 1/1 2024-08-07T17:53:50.8812049Z distributed/fsdp/test_fsdp_flatten_params 1/1 2024-08-07T17:53:50.8812512Z distributed/fsdp/test_fsdp_overlap 1/1 2024-08-07T17:53:50.8812962Z distributed/fsdp/test_fsdp_grad_acc 1/1 2024-08-07T17:53:50.8813381Z distributed/algorithms/test_join 1/1 2024-08-07T17:53:50.8813830Z distributed/_tools/test_fsdp2_mem_tracker 1/1 2024-08-07T17:53:50.8814295Z distributed/fsdp/test_fsdp_exec_order 1/1 2024-08-07T17:53:50.8814740Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 2024-08-07T17:53:50.8815271Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 2024-08-07T17:53:50.8815834Z distributed/checkpoint/test_dtensor_resharding 1/1 2024-08-07T17:53:50.8816293Z distributed/test_c10d_spawn_ucc 1/1 2024-08-07T17:53:50.8816794Z distributed/_composable/fsdp/test_fully_shard_autograd 1/1 2024-08-07T17:53:50.8817460Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 1/1 2024-08-07T17:53:50.8818062Z distributed/_tensor/test_embedding_ops 1/1 2024-08-07T17:53:50.8818576Z distributed/_tensor/experimental/test_tp_transform 1/1 2024-08-07T17:53:50.8819105Z distributed/checkpoint/test_tp_checkpoint 1/1 2024-08-07T17:53:50.8819558Z distributed/fsdp/test_fsdp_apply 1/1 2024-08-07T17:53:50.8820029Z distributed/elastic/timer/local_timer_example 1/1 2024-08-07T17:53:50.8820546Z distributed/_shard/sharded_tensor/ops/test_init 1/1 2024-08-07T17:53:50.8821112Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 2024-08-07T17:53:50.8821736Z distributed/_composable/fsdp/test_fully_shard_compile 1/1 2024-08-07T17:53:50.8822252Z distributed/fsdp/test_fsdp_pure_fp16 1/1 2024-08-07T17:53:50.8822671Z distributed/fsdp/test_fsdp_input 1/1 2024-08-07T17:53:50.8823135Z distributed/checkpoint/test_fsdp_model_state 1/1 2024-08-07T17:53:50.8823714Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 1/1 2024-08-07T17:53:50.8824378Z distributed/fsdp/test_fsdp_backward_prefetch 1/1 2024-08-07T17:53:50.8824915Z distributed/_composable/fsdp/test_fully_shard_init 1/1 2024-08-07T17:53:50.8825513Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 2024-08-07T17:53:50.8826074Z distributed/fsdp/test_fsdp_multiple_forward 1/1 2024-08-07T17:53:50.8826683Z distributed/_composable/fully_shard/test_fully_shard_mixed_precision 1/1 2024-08-07T17:53:50.8827973Z distributed/checkpoint/e2e/test_pipeline 1/1 2024-08-07T17:53:50.8828761Z distributed/test_data_parallel 1/1 2024-08-07T17:53:50.8829496Z distributed/test_launcher 1/1 2024-08-07T17:53:50.8829963Z distributed/test_multi_threaded_pg 1/1 2024-08-07T17:53:50.8830405Z distributed/fsdp/test_checkpoint_wrapper 1/1 2024-08-07T17:53:50.8830870Z distributed/checkpoint/test_planner 1/1 2024-08-07T17:53:50.8831333Z distributed/checkpoint/test_compatibility 1/1 2024-08-07T17:53:50.8831790Z distributed/elastic/utils/logging_test 1/1 2024-08-07T17:53:50.8832248Z distributed/checkpoint/test_utils 1/1 2024-08-07T17:53:50.8832743Z distributed/optim/test_apply_optimizer_in_backward 1/1 2024-08-07T17:53:50.8833241Z distributed/rpc/test_tensorpipe_agent 1/1 2024-08-07T17:53:50.8833842Z distributed/pipelining/test_composability 1/1 2024-08-07T17:53:50.8834310Z Parallel tests (0): 2024-08-07T17:53:50.8834773Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-08-07T17:53:50.8863633Z Running distributed/_tools/test_mod_tracker 1/1 ... [2024-08-07 17:53:50.885972] 2024-08-07T17:53:50.8868639Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_mod_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 17:53:50.886424] 2024-08-07T17:53:54.8594926Z 2024-08-07T17:53:54.8596409Z distributed/_tools/test_mod_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_mod_tracker_1.1_49de26538bdedeb4_.log 2024-08-07T17:53:54.8598910Z Running 4 items in this shard: test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_ac, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_bw_detection, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_module_hierarchy, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_user_hooks 2024-08-07T17:53:54.8600717Z 2024-08-07T17:53:54.8601710Z Running distributed/pipelining/test_backward 1/1 ... [2024-08-07 17:53:54.859923] 2024-08-07T17:53:54.8606898Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_backward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 17:53:54.860324] 2024-08-07T17:53:58.8829365Z 2024-08-07T17:53:58.8830588Z distributed/pipelining/test_backward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_backward_1.1_7156b31aa5bd8064_.log 2024-08-07T17:53:58.8832143Z Running 1 items in this shard: test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward 2024-08-07T17:53:58.8832774Z 2024-08-07T17:53:58.8835504Z Running distributed/_shard/sharded_tensor/test_logger 1/1 ... [2024-08-07 17:53:58.883217] 2024-08-07T17:53:58.8839873Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_logger.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 17:53:58.883622] 2024-08-07T17:54:03.0565270Z 2024-08-07T17:54:03.0567858Z distributed/_shard/sharded_tensor/test_logger 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_140195a996d349db_.log 2024-08-07T17:54:03.0571385Z Running 1 items in this shard: test/distributed/_shard/sharded_tensor/test_logger.py::ShardingSpecLoggerTest::test_get_or_create_logger 2024-08-07T17:54:03.0572804Z 2024-08-07T17:54:03.0573963Z Running distributed/test_symmetric_memory 1/1 ... [2024-08-07 17:54:03.057039] 2024-08-07T17:54:03.5854671Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_symmetric_memory.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 17:54:03.584843] 2024-08-07T17:54:07.1585478Z 2024-08-07T17:54:07.1587524Z distributed/test_symmetric_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_symmetric_memory_1.1_3a51517a4eaeba00_.log 2024-08-07T17:54:07.1599047Z Running 20 items in this shard: test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_cuda_nvlink_connectivity_detection, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_empty_strided_p2p, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_empty_strided_p2p_persistent, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_gather_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_gather_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_matmul_reduce_scatter_scatter_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_matmul_reduce_scatter_scatter_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_all_gather_symm_mem_input_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_all_gather_symm_mem_input_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_avg_symm_mem_input_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_avg_symm_mem_input_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_sum_symm_mem_input_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_sum_symm_mem_input_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_optimal_layout_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_optimal_layout_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_optimal_layout_dim_2 2024-08-07T17:54:07.1609612Z 2024-08-07T17:54:07.1610050Z Running distributed/tensor/parallel/test_micro_pipeline_tp 1/1 ... [2024-08-07 17:54:07.158911] 2024-08-07T17:54:07.1611682Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_micro_pipeline_tp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 17:54:07.159333] 2024-08-07T17:54:12.0359853Z 2024-08-07T17:54:12.0361195Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_03643959fe9ee064_.log 2024-08-07T17:54:12.0390398Z Running 29 items in this shard: test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_all_gather_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_reduce_scatter_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_get_unexposed_collectives 2024-08-07T17:54:12.0427298Z 2024-08-07T17:54:12.0427980Z Running distributed/test_dynamo_distributed 1/1 ... [2024-08-07 17:54:12.036265] 2024-08-07T17:54:12.0430932Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 17:54:12.036664] 2024-08-07T17:55:29.3045599Z 2024-08-07T17:55:29.3050190Z distributed/test_dynamo_distributed 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_dynamo_distributed_1.1_b257c8b5e34819e8_.log 2024-08-07T17:55:29.3075493Z Running 46 items in this shard: test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_call_method_forward, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_issue90375, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_symbol_splitting, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_direct, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_indirect, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_no_binding, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_torture_multi, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_scalar, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_speculation_divergence, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_tensor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_graph_break_empty_graph_still_collective, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_baseline_aot_eager_multiprocess, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_setattr, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_aot_autograd, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_async_subclass_no_specialize, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_custom_layer, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_aot_eager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_empty_graph_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_diff_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_same_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_orig_params_assert, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_guards, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_register_attr_or_module, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_staticmethod, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_ctx_manager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_inference, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_training, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_transpose, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_higher_order_op, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ignored_parameters, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_no_split 2024-08-07T17:55:29.3096656Z 2024-08-07T17:55:29.3096981Z Running distributed/_tools/test_mem_tracker 1/1 ... [2024-08-07 17:55:29.304772] 2024-08-07T17:55:29.3098476Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_mem_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 17:55:29.305186] 2024-08-07T17:55:34.6805796Z 2024-08-07T17:55:34.6807320Z distributed/_tools/test_mem_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_mem_tracker_1.1_7f91e80f0268f59f_.log 2024-08-07T17:55:34.6809687Z Running 3 items in this shard: test/distributed/_tools/test_mem_tracker.py::TestMemTracker::test_cuda_tracker_equivalence, test/distributed/_tools/test_mem_tracker.py::TestMemTracker::test_tracker_attribution, test/distributed/_tools/test_mem_tracker.py::TestMemTracker::test_tracker_with_activation_checkpointing 2024-08-07T17:55:34.6811943Z 2024-08-07T17:55:34.6812670Z Running distributed/pipelining/test_schedule_multiproc 1/1 ... [2024-08-07 17:55:34.680866] 2024-08-07T17:55:34.6817004Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_schedule_multiproc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 17:55:34.681278] 2024-08-07T17:55:46.9716601Z 2024-08-07T17:55:46.9718160Z distributed/pipelining/test_schedule_multiproc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_schedule_multiproc_1.1_832b281e4740cc3e_.log 2024-08-07T17:55:46.9726851Z Running 14 items in this shard: test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_forward_only_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass0_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass1_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_weight_update_mlp_e2e_ScheduleClass0 2024-08-07T17:55:46.9742807Z Running 14 items in this shard: test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_forward_only_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass0_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass1_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_weight_update_mlp_e2e_ScheduleClass0 2024-08-07T17:55:46.9750956Z 2024-08-07T17:55:46.9751327Z Running distributed/_tensor/test_convolution_ops 1/1 ... [2024-08-07 17:55:46.971967] 2024-08-07T17:55:46.9752836Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_convolution_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 17:55:46.972387] 2024-08-07T17:56:25.5207102Z 2024-08-07T17:56:25.5208329Z distributed/_tensor/test_convolution_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_convolution_ops_1.1_3f18b13dcaf7de7a_.log 2024-08-07T17:56:25.5210385Z Running 2 items in this shard: test/distributed/_tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_depthwise_convolution, test/distributed/_tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_downsampling_convolution 2024-08-07T17:56:25.5211584Z 2024-08-07T17:56:25.5212372Z Running distributed/_tensor/test_attention 1/1 ... [2024-08-07 17:56:25.520932] 2024-08-07T17:56:25.5217029Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_attention.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 17:56:25.521339] 2024-08-07T17:56:33.3514612Z 2024-08-07T17:56:33.3515806Z distributed/_tensor/test_attention 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_attention_1.1_9e3ad3bb5afb5f33_.log 2024-08-07T17:56:33.3520697Z Running 8 items in this shard: test/distributed/_tensor/test_attention.py::RingAttentionTest::test_is_causal_behavior, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_compile_attention_fn0, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_compile_attention_fn1, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_custom_transformer, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_native_transformer_is_causal_False, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_native_transformer_is_causal_True, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_False, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_True 2024-08-07T17:56:33.3525145Z 2024-08-07T17:56:33.3525459Z Running distributed/_tensor/test_dtensor 1/1 ... [2024-08-07 17:56:33.351694] 2024-08-07T17:56:33.3526913Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 17:56:33.352095] 2024-08-07T17:58:50.4712595Z 2024-08-07T17:58:50.4715723Z distributed/_tensor/test_dtensor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_1.1_c943f9e44328900d_.log 2024-08-07T17:58:50.4730193Z Running 32 items in this shard: test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_async_output, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_constructor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_new_empty_strided, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_properties, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_save_load, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_hash, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_read_only_after_set, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_stride, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_negative_dim, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_then_to_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding_raise_error, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_grad_hint, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_sync, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_meta_dtensor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_modules_w_meta_dtensor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_to_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_to_local_grad_hint, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_auto_implicit_replication, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_default_value_sub_mesh, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_device_mesh_nd, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_2d_mesh, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_api_device_mesh_context_manager, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_device_mesh_device_conversion, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_spec_local_shard_offset, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_from_local_sub_mesh, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_implicit_replication, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_metadata_consistency_check, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_redistribute_sub_mesh, test/distributed/_tensor/test_dtensor.py::TestDTensorPlacementTypes::test_split_tensor_1D, test/distributed/_tensor/test_dtensor.py::DTensorLogTest::test_dtensor_log 2024-08-07T17:58:50.4742757Z 2024-08-07T17:58:50.4743076Z Running distributed/_tensor/test_api 1/1 ... [2024-08-07 17:58:50.471565] 2024-08-07T17:58:50.4744472Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 17:58:50.471974] 2024-08-07T17:59:25.2634787Z 2024-08-07T17:59:25.2636066Z distributed/_tensor/test_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_api_1.1_f61d5ff8f327d9f8_.log 2024-08-07T17:59:25.2640235Z Running 8 items in this shard: test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_casting, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn_warning, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_meta, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor_errors, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor_uneven_sharding 2024-08-07T17:59:25.2644247Z 2024-08-07T17:59:25.2644605Z Running distributed/_composable/test_contract 1/1 ... [2024-08-07 17:59:25.263902] 2024-08-07T17:59:25.2648235Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_contract.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 17:59:25.264372] 2024-08-07T17:59:29.1047531Z 2024-08-07T17:59:29.1049088Z distributed/_composable/test_contract 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_contract_1.1_bfd21404d3161a6f_.log 2024-08-07T17:59:29.1051921Z Running 5 items in this shard: test/distributed/_composable/test_contract.py::TestContract::test_add_hooks, test/distributed/_composable/test_contract.py::TestContract::test_modify_fqn, test/distributed/_composable/test_contract.py::TestContract::test_multi_module_api, test/distributed/_composable/test_contract.py::TestContract::test_registry, test/distributed/_composable/test_contract.py::TestContract::test_state 2024-08-07T17:59:29.1054516Z 2024-08-07T17:59:29.1054882Z Running distributed/test_device_mesh 1/1 ... [2024-08-07 17:59:29.105002] 2024-08-07T17:59:29.1058249Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_device_mesh.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 17:59:29.105414] 2024-08-07T18:02:04.2895759Z 2024-08-07T18:02:04.2897147Z distributed/test_device_mesh 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_device_mesh_1.1_5086953d88c2aa7e_.log 2024-08-07T18:02:04.2914743Z Running 39 items in this shard: test/distributed/test_device_mesh.py::DeviceMeshTest::test_assert_invalid_mesh_tensor, test/distributed/test_device_mesh.py::DeviceMeshTest::test_device_mesh_2d, test/distributed/test_device_mesh.py::DeviceMeshTest::test_device_mesh_init_backend, test/distributed/test_device_mesh.py::DeviceMeshTest::test_fake_pg_device_mesh, test/distributed/test_device_mesh.py::DeviceMeshTest::test_from_group_with_global_pg, test/distributed/test_device_mesh.py::DeviceMeshTest::test_from_group_with_invalid_mesh, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_group_and_get_all_groups, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank_raises_exception, test/distributed/test_device_mesh.py::DeviceMeshTest::test_init_process_group, test/distributed/test_device_mesh.py::DeviceMeshTest::test_raises_invalid_device_type, test/distributed/test_device_mesh.py::DeviceMeshTest::test_set_mesh_dim_group_options, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_hash, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_nd, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_parent_child_hash, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_from_group_with_mesh_shape, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_get_local_rank_3d, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_init_device_mesh, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_duplicate_mesh_dim_names, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_mesh_shape_mesh_dim_names_mismatch, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_cache_and_reuse_submesh_slice_result, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_1d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_2d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_3d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_3d_noncontinuous_slicing, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_invalid_mesh_dim_name, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_no_mesh_dim_found, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_mesh_dim_by_name, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_parent_mesh, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_parent_mesh_dim_exist, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_parent_mesh_dim_not_exist, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_gather_uneven, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_1d, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_nd, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_contiguous, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_uneven, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_1d, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_nd, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_uneven 2024-08-07T18:02:04.2930339Z 2024-08-07T18:02:04.2930693Z Running distributed/pipelining/test_pipe 1/1 ... [2024-08-07 18:02:04.289858] 2024-08-07T18:02:04.2932183Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_pipe.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:02:04.290263] 2024-08-07T18:02:09.0137304Z 2024-08-07T18:02:09.0138694Z distributed/pipelining/test_pipe 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_pipe_1.1_2d6fb07e013d5ed4_.log 2024-08-07T18:02:09.0140922Z Running 3 items in this shard: test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass0, test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass1, test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass2 2024-08-07T18:02:09.0142484Z 2024-08-07T18:02:09.0143058Z Running distributed/test_c10d_functional_native 1/1 ... [2024-08-07 18:02:09.014018] 2024-08-07T18:02:09.0148552Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_functional_native.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:02:09.014420] 2024-08-07T18:03:07.0984367Z 2024-08-07T18:03:07.0985752Z distributed/test_c10d_functional_native 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_functional_native_1.1_51914592345f15a4_.log 2024-08-07T18:03:07.0999819Z Running 25 items in this shard: test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_to_all_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_broadcast, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_inductor_dtypeview_memory_leak, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_py_work, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_threading, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_unwaited, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_to_all_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_broadcast, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_inplace_op_on_view, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reuse_buffer_after_inplace_collective, test/distributed/test_c10d_functional_native.py::CompileTest::test_ranks_and_tag 2024-08-07T18:03:07.1010891Z 2024-08-07T18:03:07.1011247Z Running distributed/_tensor/test_tensor_ops 1/1 ... [2024-08-07 18:03:07.098772] 2024-08-07T18:03:07.1012717Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_tensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:03:07.099218] 2024-08-07T18:05:05.8134455Z 2024-08-07T18:05:05.8137633Z distributed/_tensor/test_tensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_tensor_ops_1.1_e896644101ed5486_.log 2024-08-07T18:05:05.8148816Z Running 25 items in this shard: test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_aten_contiguous, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_clone, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_contiguous, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_detach, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_dtensor_dtype_conversion, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_empty_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_equal, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace_partial_sum, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_full_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_gather, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_index, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_inplace_op, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_new_empty_strided, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_new_full, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like_partial_sum, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_op_out_variant, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_scatter, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_slice, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_stack, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_where_type_promotion, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zero_inplace, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like_partial_sum 2024-08-07T18:05:05.8158848Z 2024-08-07T18:05:05.8159184Z Running distributed/_tensor/test_math_ops 1/1 ... [2024-08-07 18:05:05.813732] 2024-08-07T18:05:05.8160768Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_math_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:05:05.814162] 2024-08-07T18:06:04.5503869Z 2024-08-07T18:06:04.5505305Z distributed/_tensor/test_math_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_math_ops_1.1_d97edbb923641fc9_.log 2024-08-07T18:06:04.5511259Z Running 11 items in this shard: test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_foreach_norm, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_bwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_fwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_linear_op_reductions, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_mean, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_nll_loss_and_cross_entropy, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_shard0_svd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_shard_math_ops, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_softmax_fwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_softmax_with_bwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_topk 2024-08-07T18:06:04.5515397Z 2024-08-07T18:06:04.5515772Z Running distributed/pipelining/test_unflatten 1/1 ... [2024-08-07 18:06:04.550628] 2024-08-07T18:06:04.5517291Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_unflatten.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:06:04.551136] 2024-08-07T18:06:09.0261047Z 2024-08-07T18:06:09.0262336Z distributed/pipelining/test_unflatten 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_unflatten_1.1_13d4a2bed9d34856_.log 2024-08-07T18:06:09.0263801Z Running 1 items in this shard: test/distributed/pipelining/test_unflatten.py::UnflattenTests::test_unflatten 2024-08-07T18:06:09.0264407Z 2024-08-07T18:06:09.0267223Z Running distributed/checkpoint/e2e/test_fsdp_ep 1/1 ... [2024-08-07 18:06:09.026361] 2024-08-07T18:06:09.0271914Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/e2e/test_fsdp_ep.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:06:09.026819] 2024-08-07T18:06:16.5062599Z 2024-08-07T18:06:16.5064081Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_fsdp_ep_1.1_a2ffd958965a7e98_.log 2024-08-07T18:06:16.5065989Z Running 1 items in this shard: test/distributed/checkpoint/e2e/test_fsdp_ep.py::TestFSDPWithEP::test_e2e 2024-08-07T18:06:16.5066564Z 2024-08-07T18:06:16.5069330Z Running distributed/pipelining/test_microbatch 1/1 ... [2024-08-07 18:06:16.506579] 2024-08-07T18:06:16.5074108Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_microbatch.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:06:16.507048] 2024-08-07T18:06:20.7798966Z 2024-08-07T18:06:20.7800344Z distributed/pipelining/test_microbatch 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_microbatch_1.1_f5487e6e08c327fb_.log 2024-08-07T18:06:20.7802504Z Running 2 items in this shard: test/distributed/pipelining/test_microbatch.py::MicrobatchTests::test_chunk_spec, test/distributed/pipelining/test_microbatch.py::MicrobatchTests::test_split_and_merge 2024-08-07T18:06:20.7803661Z 2024-08-07T18:06:20.7805009Z Running distributed/optim/test_named_optimizer 1/1 ... [2024-08-07 18:06:20.780143] 2024-08-07T18:06:20.7809416Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/optim/test_named_optimizer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:06:20.780554] 2024-08-07T18:06:23.7681110Z 2024-08-07T18:06:23.7682594Z distributed/optim/test_named_optimizer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_named_optimizer_1.1_67d75294535a0ccd_.log 2024-08-07T18:06:23.7683566Z 2024-08-07T18:06:23.7686549Z Running distributed/_tensor/debug/test_comm_mode_features 1/1 ... [2024-08-07 18:06:23.768343] 2024-08-07T18:06:23.7690990Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/debug/test_comm_mode_features.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:06:23.768751] 2024-08-07T18:06:46.3815856Z 2024-08-07T18:06:46.3819213Z distributed/_tensor/debug/test_comm_mode_features 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.debug.test_comm_mode_features_1.1_cf68b9c5ef9b78d6_.log 2024-08-07T18:06:46.3822562Z Running 4 items in this shard: test/distributed/_tensor/debug/test_comm_mode_features.py::TestCommModeFeatures::test_MLPStacked_distributed_sharding_display, test/distributed/_tensor/debug/test_comm_mode_features.py::TestCommModeFeatures::test_MLP_distributed_sharding_display, test/distributed/_tensor/debug/test_comm_mode_features.py::TestCommModeFeatures::test_MLP_module_tracing, test/distributed/_tensor/debug/test_comm_mode_features.py::TestCommModeFeatures::test_transformer_module_tracing 2024-08-07T18:06:46.3824987Z 2024-08-07T18:06:46.3825326Z Running distributed/checkpoint/test_traverse 1/1 ... [2024-08-07 18:06:46.382034] 2024-08-07T18:06:46.3829235Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_traverse.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:06:46.382518] 2024-08-07T18:06:50.3051164Z 2024-08-07T18:06:50.3052554Z distributed/checkpoint/test_traverse 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_traverse_1.1_6729844a9f197e13_.log 2024-08-07T18:06:50.3056605Z Running 7 items in this shard: test/distributed/checkpoint/test_traverse.py::TestTraverse::test_get_element, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_set_element, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_doesnt_ignore_intermediate_collections, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_nested_dict, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_nested_list, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_shallow, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_with_ordered_dict 2024-08-07T18:06:50.3060097Z 2024-08-07T18:06:50.3060447Z Running distributed/test_compute_comm_reordering 1/1 ... [2024-08-07 18:06:50.305417] 2024-08-07T18:06:50.3062001Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_compute_comm_reordering.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:06:50.305863] 2024-08-07T18:06:58.2375722Z 2024-08-07T18:06:58.2377843Z distributed/test_compute_comm_reordering 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_compute_comm_reordering_1.1_3d5d05b28b9c940f_.log 2024-08-07T18:06:58.2389749Z Running 7 items in this shard: test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_grouped_scheduler_node, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_nccl_heuristics, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_raise_comms, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap_custom_runtime_estimation, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits_raise_comms 2024-08-07T18:06:58.2397922Z 2024-08-07T18:06:59.3326035Z 2024-08-07T18:06:59.3326740Z real 13m13.337s 2024-08-07T18:06:59.3327124Z user 36m24.384s 2024-08-07T18:06:59.3327419Z sys 11m52.655s 2024-08-07T18:06:59.3327727Z + assert_git_not_dirty 2024-08-07T18:06:59.3328161Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-08-07T18:06:59.3328676Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *xla* ]] 2024-08-07T18:06:59.3331952Z ++ git status --porcelain 2024-08-07T18:06:59.3332328Z ++ grep -v '?? third_party' 2024-08-07T18:07:02.2190311Z ++ true 2024-08-07T18:07:02.2191116Z + git_status= 2024-08-07T18:07:02.2191455Z + [[ -n '' ]] 2024-08-07T18:07:02.2191811Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-08-07T18:07:02.2192276Z + [[ 2 == 1 ]] 2024-08-07T18:07:02.2192570Z + [[ 2 == 1 ]] 2024-08-07T18:07:02.2193974Z + cleanup_workspace 2024-08-07T18:07:02.2194608Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-07T18:07:02.2195588Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-07T18:07:02.2196361Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-07T18:07:02.2196944Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-07T18:07:02.2197616Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-07T18:07:02.2198348Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-07T18:07:02.2198929Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-07T18:07:02.7546707Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-07T18:07:02.7547298Z cat test/**/*_toprint.log || true 2024-08-07T18:07:02.7555368Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:07:02.7555851Z env: 2024-08-07T18:07:02.7556139Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:07:02.7556566Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:07:02.7557463Z DOCKER_CONTAINER_ID: ddfc464047ce056b8e984eda0cf5fdb3a03b3de1ff51933a15b85473696b0056 2024-08-07T18:07:02.7558072Z ##[endgroup] 2024-08-07T18:07:02.7628946Z cat: test/**/*_toprint.log: No such file or directory 2024-08-07T18:07:02.7669459Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-07T18:07:02.7669936Z kill "$MONITOR_SCRIPT_PID" 2024-08-07T18:07:02.7677297Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:07:02.7677776Z env: 2024-08-07T18:07:02.7678056Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:07:02.7678492Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:07:02.7679187Z DOCKER_CONTAINER_ID: ddfc464047ce056b8e984eda0cf5fdb3a03b3de1ff51933a15b85473696b0056 2024-08-07T18:07:02.7679810Z MONITOR_SCRIPT_PID: 71798 2024-08-07T18:07:02.7680162Z ##[endgroup] 2024-08-07T18:07:02.7844719Z Prepare all required actions 2024-08-07T18:07:02.7845279Z Getting action download info 2024-08-07T18:07:02.9640398Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-07T18:07:03.1450893Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-07T18:07:03.1451351Z with: 2024-08-07T18:07:03.1451796Z file-suffix: test-distributed-2-3-linux.8xlarge.nvidia.gpu_28476145276 2024-08-07T18:07:03.1452375Z s3-bucket: gha-artifacts 2024-08-07T18:07:03.1452716Z env: 2024-08-07T18:07:03.1452985Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:07:03.1453412Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:07:03.1454104Z DOCKER_CONTAINER_ID: ddfc464047ce056b8e984eda0cf5fdb3a03b3de1ff51933a15b85473696b0056 2024-08-07T18:07:03.1454708Z ##[endgroup] 2024-08-07T18:07:03.1519513Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-07T18:07:03.1520169Z # Remove any previous test jsons if they exist 2024-08-07T18:07:03.1520639Z rm -f test-jsons-*.zip 2024-08-07T18:07:03.1521135Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-07T18:07:03.1528658Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:07:03.1529131Z env: 2024-08-07T18:07:03.1529424Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:07:03.1529859Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:07:03.1530537Z DOCKER_CONTAINER_ID: ddfc464047ce056b8e984eda0cf5fdb3a03b3de1ff51933a15b85473696b0056 2024-08-07T18:07:03.1531339Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_28476145276 2024-08-07T18:07:03.1531899Z ##[endgroup] 2024-08-07T18:07:03.1775035Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-07T18:07:03.1809439Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-07T18:07:03.1810096Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-07T18:07:03.1817632Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-07T18:07:03.1820043Z adding: test/test-reports/td_exclusions-f1a4b524c5d27291943b.json (deflated 86%) 2024-08-07T18:07:03.1826287Z adding: test/.pytorch-slow-tests.json (deflated 81%) 2024-08-07T18:07:03.1841907Z adding: test/.pytorch-disabled-tests.json (deflated 89%) 2024-08-07T18:07:03.1879083Z ##[group]Run # Remove any previous test reports if they exist 2024-08-07T18:07:03.1879672Z # Remove any previous test reports if they exist 2024-08-07T18:07:03.1880177Z rm -f test-reports-*.zip 2024-08-07T18:07:03.1880735Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-07T18:07:03.1888187Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:07:03.1888688Z env: 2024-08-07T18:07:03.1888991Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:07:03.1889417Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:07:03.1890137Z DOCKER_CONTAINER_ID: ddfc464047ce056b8e984eda0cf5fdb3a03b3de1ff51933a15b85473696b0056 2024-08-07T18:07:03.1890948Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_28476145276 2024-08-07T18:07:03.1891492Z ##[endgroup] 2024-08-07T18:07:03.2109116Z adding: test/test-reports/python-pytest/distributed._tools.test_mod_tracker/distributed._tools.test_mod_tracker-4f9fbf47ded868af.xml (deflated 65%) 2024-08-07T18:07:03.2112150Z adding: test/test-reports/python-pytest/distributed.pipelining.test_backward/distributed.pipelining.test_backward-fa83cbdbe4a44aaa.xml (deflated 37%) 2024-08-07T18:07:03.2114014Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.test_logger/distributed._shard.sharded_tensor.test_logger-16155dc327bf0b72.xml (deflated 37%) 2024-08-07T18:07:03.2117397Z adding: test/test-reports/python-pytest/distributed.test_symmetric_memory/distributed.test_symmetric_memory-2f1d141415cfad3b.xml (deflated 93%) 2024-08-07T18:07:03.2121068Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_micro_pipeline_tp/distributed.tensor.parallel.test_micro_pipeline_tp-818e9efa0fb83402.xml (deflated 95%) 2024-08-07T18:07:03.2124651Z adding: test/test-reports/python-pytest/distributed.test_dynamo_distributed/distributed.test_dynamo_distributed-7e8b2d4cb77c61fb.xml (deflated 91%) 2024-08-07T18:07:03.2127883Z adding: test/test-reports/python-pytest/distributed._tools.test_mem_tracker/distributed._tools.test_mem_tracker-5970c9d8b7bd2ee0.xml (deflated 58%) 2024-08-07T18:07:03.2130271Z adding: test/test-reports/python-pytest/distributed.pipelining.test_schedule_multiproc/distributed.pipelining.test_schedule_multiproc-5a813761d84967cf.xml (deflated 82%) 2024-08-07T18:07:03.2132274Z adding: test/test-reports/python-pytest/distributed.pipelining.test_schedule_multiproc/distributed.pipelining.test_schedule_multiproc-a7ec5a1eccfa8dd7.xml (deflated 82%) 2024-08-07T18:07:03.2133947Z adding: test/test-reports/python-pytest/distributed._tensor.test_convolution_ops/distributed._tensor.test_convolution_ops-c55c93560623fa00.xml (deflated 49%) 2024-08-07T18:07:03.2135461Z adding: test/test-reports/python-pytest/distributed._tensor.test_attention/distributed._tensor.test_attention-9c4bc12adc94e91c.xml (deflated 86%) 2024-08-07T18:07:03.2136910Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor/distributed._tensor.test_dtensor-b7a36967a61b9c8e.xml (deflated 82%) 2024-08-07T18:07:03.2138277Z adding: test/test-reports/python-pytest/distributed._tensor.test_api/distributed._tensor.test_api-7518ff2c6bfed0a2.xml (deflated 74%) 2024-08-07T18:07:03.2139691Z adding: test/test-reports/python-pytest/distributed._composable.test_contract/distributed._composable.test_contract-dd46f32e66148729.xml (deflated 69%) 2024-08-07T18:07:03.2141115Z adding: test/test-reports/python-pytest/distributed.test_device_mesh/distributed.test_device_mesh-2d723cd881c986ab.xml (deflated 85%) 2024-08-07T18:07:03.2142519Z adding: test/test-reports/python-pytest/distributed.pipelining.test_pipe/distributed.pipelining.test_pipe-4e6a7b7c0fd812ef.xml (deflated 65%) 2024-08-07T18:07:03.2144022Z adding: test/test-reports/python-pytest/distributed.test_c10d_functional_native/distributed.test_c10d_functional_native-848719076f823304.xml (deflated 90%) 2024-08-07T18:07:03.2145520Z adding: test/test-reports/python-pytest/distributed._tensor.test_tensor_ops/distributed._tensor.test_tensor_ops-3c25fe9965053abc.xml (deflated 84%) 2024-08-07T18:07:03.2147167Z adding: test/test-reports/python-pytest/distributed._tensor.test_math_ops/distributed._tensor.test_math_ops-01d3585fd071bf87.xml (deflated 76%) 2024-08-07T18:07:03.2148674Z adding: test/test-reports/python-pytest/distributed.pipelining.test_unflatten/distributed.pipelining.test_unflatten-fae695dc07147a45.xml (deflated 42%) 2024-08-07T18:07:03.2150242Z adding: test/test-reports/python-pytest/distributed.checkpoint.e2e.test_fsdp_ep/distributed.checkpoint.e2e.test_fsdp_ep-7a85711421fc1418.xml (deflated 50%) 2024-08-07T18:07:03.2151818Z adding: test/test-reports/python-pytest/distributed.pipelining.test_microbatch/distributed.pipelining.test_microbatch-98b967c353911820.xml (deflated 54%) 2024-08-07T18:07:03.2153472Z adding: test/test-reports/python-pytest/distributed._tensor.debug.test_comm_mode_features/distributed._tensor.debug.test_comm_mode_features-d8cb88c688c7f2e1.xml (deflated 63%) 2024-08-07T18:07:03.2155300Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_traverse/distributed.checkpoint.test_traverse-d33a582ff73b6a0d.xml (deflated 75%) 2024-08-07T18:07:03.2156862Z adding: test/test-reports/python-pytest/distributed.test_compute_comm_reordering/distributed.test_compute_comm_reordering-4ddff5c9ecb79344.xml (deflated 83%) 2024-08-07T18:07:03.2188167Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-07T18:07:03.2188748Z # Remove any previous usage logs if they exist 2024-08-07T18:07:03.2189224Z rm -f logs-*.zip 2024-08-07T18:07:03.2189948Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-07T18:07:03.2190649Z # so check to see if the file exists first 2024-08-07T18:07:03.2191119Z if [ -f 'usage_log.txt' ]; then 2024-08-07T18:07:03.2191591Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-07T18:07:03.2192034Z fi 2024-08-07T18:07:03.2192379Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-07T18:07:03.2192872Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-07T18:07:03.2193316Z fi 2024-08-07T18:07:03.2200658Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:07:03.2201124Z env: 2024-08-07T18:07:03.2201417Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:07:03.2201852Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:07:03.2202531Z DOCKER_CONTAINER_ID: ddfc464047ce056b8e984eda0cf5fdb3a03b3de1ff51933a15b85473696b0056 2024-08-07T18:07:03.2203328Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_28476145276 2024-08-07T18:07:03.2203890Z ##[endgroup] 2024-08-07T18:07:03.2252431Z adding: usage_log.txt (deflated 92%) 2024-08-07T18:07:03.2474210Z adding: test/test-reports/distributed._tools.test_mod_tracker_1.1_49de26538bdedeb4_.log (deflated 64%) 2024-08-07T18:07:03.2476161Z adding: test/test-reports/distributed.pipelining.test_backward_1.1_7156b31aa5bd8064_.log (deflated 54%) 2024-08-07T18:07:03.2477999Z adding: test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_140195a996d349db_.log (deflated 55%) 2024-08-07T18:07:03.2479734Z adding: test/test-reports/distributed.test_symmetric_memory_1.1_3a51517a4eaeba00_.log (deflated 86%) 2024-08-07T18:07:03.2481393Z adding: test/test-reports/distributed.checkpoint.test_traverse_1.1_6729844a9f197e13_.log (deflated 71%) 2024-08-07T18:07:03.2483511Z adding: test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_03643959fe9ee064_.log (deflated 90%) 2024-08-07T18:07:03.2485781Z adding: test/test-reports/distributed.test_dynamo_distributed_1.1_b257c8b5e34819e8_.log (deflated 85%) 2024-08-07T18:07:03.2487378Z adding: test/test-reports/distributed._tools.test_mem_tracker_1.1_7f91e80f0268f59f_.log (deflated 62%) 2024-08-07T18:07:03.2488886Z adding: test/test-reports/distributed.test_compute_comm_reordering_1.1_3d5d05b28b9c940f_.log (deflated 74%) 2024-08-07T18:07:03.2491237Z adding: test/test-reports/distributed.pipelining.test_schedule_multiproc_1.1_832b281e4740cc3e_.log (deflated 89%) 2024-08-07T18:07:03.2492368Z adding: test/test-reports/distributed._tensor.test_convolution_ops_1.1_3f18b13dcaf7de7a_.log (deflated 59%) 2024-08-07T18:07:03.2494425Z adding: test/test-reports/distributed._tensor.test_attention_1.1_9e3ad3bb5afb5f33_.log (deflated 76%) 2024-08-07T18:07:03.2495494Z adding: test/test-reports/distributed._tensor.test_dtensor_1.1_c943f9e44328900d_.log (deflated 85%) 2024-08-07T18:07:03.2497406Z adding: test/test-reports/distributed._tensor.test_api_1.1_f61d5ff8f327d9f8_.log (deflated 89%) 2024-08-07T18:07:03.2499560Z adding: test/test-reports/distributed._composable.test_contract_1.1_bfd21404d3161a6f_.log (deflated 67%) 2024-08-07T18:07:03.2501652Z adding: test/test-reports/distributed.test_device_mesh_1.1_5086953d88c2aa7e_.log (deflated 83%) 2024-08-07T18:07:03.2503778Z adding: test/test-reports/distributed.pipelining.test_pipe_1.1_2d6fb07e013d5ed4_.log (deflated 62%) 2024-08-07T18:07:03.2504799Z adding: test/test-reports/distributed.test_c10d_functional_native_1.1_51914592345f15a4_.log (deflated 88%) 2024-08-07T18:07:03.2505805Z adding: test/test-reports/distributed._tensor.test_tensor_ops_1.1_e896644101ed5486_.log (deflated 79%) 2024-08-07T18:07:03.2507276Z adding: test/test-reports/distributed._tensor.test_math_ops_1.1_d97edbb923641fc9_.log (deflated 74%) 2024-08-07T18:07:03.2508283Z adding: test/test-reports/distributed.pipelining.test_unflatten_1.1_13d4a2bed9d34856_.log (deflated 54%) 2024-08-07T18:07:03.2509497Z adding: test/test-reports/distributed.checkpoint.e2e.test_fsdp_ep_1.1_a2ffd958965a7e98_.log (deflated 62%) 2024-08-07T18:07:03.2510572Z adding: test/test-reports/distributed.pipelining.test_microbatch_1.1_f5487e6e08c327fb_.log (deflated 58%) 2024-08-07T18:07:03.2511616Z adding: test/test-reports/distributed.optim.test_named_optimizer_1.1_67d75294535a0ccd_.log (deflated 35%) 2024-08-07T18:07:03.2512680Z adding: test/test-reports/distributed._tensor.debug.test_comm_mode_features_1.1_cf68b9c5ef9b78d6_.log (deflated 73%) 2024-08-07T18:07:03.2548332Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-07T18:07:03.2549072Z # Remove any previous debugging artifacts if they exist 2024-08-07T18:07:03.2549581Z rm -f debug-*.zip 2024-08-07T18:07:03.2549963Z if [ -d 'test/debug' ]; then 2024-08-07T18:07:03.2550427Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-07T18:07:03.2550853Z fi 2024-08-07T18:07:03.2557937Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:07:03.2558490Z env: 2024-08-07T18:07:03.2558802Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:07:03.2559246Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:07:03.2559950Z DOCKER_CONTAINER_ID: ddfc464047ce056b8e984eda0cf5fdb3a03b3de1ff51933a15b85473696b0056 2024-08-07T18:07:03.2560750Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_28476145276 2024-08-07T18:07:03.2561319Z ##[endgroup] 2024-08-07T18:07:03.2653476Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-07T18:07:03.2653904Z with: 2024-08-07T18:07:03.2654209Z s3-bucket: gha-artifacts 2024-08-07T18:07:03.2654629Z s3-prefix: pytorch/pytorch/10288745067/1/artifact 2024-08-07T18:07:03.2655072Z retention-days: 14 2024-08-07T18:07:03.2655407Z if-no-files-found: warn 2024-08-07T18:07:03.2655772Z path: test-jsons-*.zip 2024-08-07T18:07:03.2656100Z name: artifact 2024-08-07T18:07:03.2656407Z region: us-east-1 2024-08-07T18:07:03.2656717Z env: 2024-08-07T18:07:03.2656990Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:07:03.2657441Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:07:03.2658151Z DOCKER_CONTAINER_ID: ddfc464047ce056b8e984eda0cf5fdb3a03b3de1ff51933a15b85473696b0056 2024-08-07T18:07:03.2658774Z ##[endgroup] 2024-08-07T18:07:03.7201608Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-07T18:07:03.7202554Z With the provided path, there will be 1 file uploaded 2024-08-07T18:07:03.7203173Z Uploading to s3 prefix: pytorch/pytorch/10288745067/1/artifact 2024-08-07T18:07:03.7252109Z Starting upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_28476145276.zip 2024-08-07T18:07:03.9428798Z Finished upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_28476145276.zip 2024-08-07T18:07:03.9669386Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-07T18:07:03.9669839Z with: 2024-08-07T18:07:03.9670131Z s3-bucket: gha-artifacts 2024-08-07T18:07:03.9670562Z s3-prefix: pytorch/pytorch/10288745067/1/artifact 2024-08-07T18:07:03.9671029Z retention-days: 14 2024-08-07T18:07:03.9671388Z if-no-files-found: error 2024-08-07T18:07:03.9671770Z path: test-reports-*.zip 2024-08-07T18:07:03.9672131Z name: artifact 2024-08-07T18:07:03.9672435Z region: us-east-1 2024-08-07T18:07:03.9672979Z env: 2024-08-07T18:07:03.9673282Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:07:03.9673719Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:07:03.9674427Z DOCKER_CONTAINER_ID: ddfc464047ce056b8e984eda0cf5fdb3a03b3de1ff51933a15b85473696b0056 2024-08-07T18:07:03.9675061Z ##[endgroup] 2024-08-07T18:07:04.4095678Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-07T18:07:04.4096840Z With the provided path, there will be 1 file uploaded 2024-08-07T18:07:04.4097427Z Uploading to s3 prefix: pytorch/pytorch/10288745067/1/artifact 2024-08-07T18:07:04.4144878Z Starting upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_28476145276.zip 2024-08-07T18:07:04.5481811Z Finished upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_28476145276.zip 2024-08-07T18:07:04.5650199Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-07T18:07:04.5650673Z with: 2024-08-07T18:07:04.5650970Z s3-bucket: gha-artifacts 2024-08-07T18:07:04.5651423Z s3-prefix: pytorch/pytorch/10288745067/1/artifact 2024-08-07T18:07:04.5651887Z retention-days: 14 2024-08-07T18:07:04.5652221Z if-no-files-found: ignore 2024-08-07T18:07:04.5652597Z path: logs-*.zip 2024-08-07T18:07:04.5652923Z name: artifact 2024-08-07T18:07:04.5653225Z region: us-east-1 2024-08-07T18:07:04.5653545Z env: 2024-08-07T18:07:04.5653845Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:07:04.5654276Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:07:04.5654988Z DOCKER_CONTAINER_ID: ddfc464047ce056b8e984eda0cf5fdb3a03b3de1ff51933a15b85473696b0056 2024-08-07T18:07:04.5655628Z ##[endgroup] 2024-08-07T18:07:05.0077726Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-07T18:07:05.0078506Z With the provided path, there will be 1 file uploaded 2024-08-07T18:07:05.0079136Z Uploading to s3 prefix: pytorch/pytorch/10288745067/1/artifact 2024-08-07T18:07:05.0127503Z Starting upload of logs-test-distributed-2-3-linux.8xlarge.nvidia.gpu_28476145276.zip 2024-08-07T18:07:05.1770346Z Finished upload of logs-test-distributed-2-3-linux.8xlarge.nvidia.gpu_28476145276.zip 2024-08-07T18:07:05.1937371Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-07T18:07:05.1937813Z with: 2024-08-07T18:07:05.1938095Z s3-bucket: gha-artifacts 2024-08-07T18:07:05.1938515Z s3-prefix: pytorch/pytorch/10288745067/1/artifact 2024-08-07T18:07:05.1938969Z retention-days: 14 2024-08-07T18:07:05.1939298Z if-no-files-found: ignore 2024-08-07T18:07:05.1939659Z path: debug-*.zip 2024-08-07T18:07:05.1939973Z name: artifact 2024-08-07T18:07:05.1940268Z region: us-east-1 2024-08-07T18:07:05.1940574Z env: 2024-08-07T18:07:05.1940865Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:07:05.1941280Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:07:05.1942000Z DOCKER_CONTAINER_ID: ddfc464047ce056b8e984eda0cf5fdb3a03b3de1ff51933a15b85473696b0056 2024-08-07T18:07:05.1942632Z ##[endgroup] 2024-08-07T18:07:05.6339283Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-07T18:07:05.6507525Z ##[group]Run # shellcheck disable=SC2156 2024-08-07T18:07:05.6508042Z # shellcheck disable=SC2156 2024-08-07T18:07:05.6508766Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-07T18:07:05.6516910Z shell: /usr/bin/bash -e {0} 2024-08-07T18:07:05.6517279Z env: 2024-08-07T18:07:05.6517576Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:07:05.6517989Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:07:05.6518691Z DOCKER_CONTAINER_ID: ddfc464047ce056b8e984eda0cf5fdb3a03b3de1ff51933a15b85473696b0056 2024-08-07T18:07:05.6519317Z ##[endgroup] 2024-08-07T18:07:05.9519813Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-07T18:07:05.9520462Z with: 2024-08-07T18:07:05.9520729Z env: 2024-08-07T18:07:05.9521022Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:07:05.9521468Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:07:05.9522394Z DOCKER_CONTAINER_ID: ddfc464047ce056b8e984eda0cf5fdb3a03b3de1ff51933a15b85473696b0056 2024-08-07T18:07:05.9523023Z ##[endgroup] 2024-08-07T18:07:05.9550593Z ##[group]Run set -eou pipefail 2024-08-07T18:07:05.9551009Z set -eou pipefail 2024-08-07T18:07:05.9551342Z  2024-08-07T18:07:05.9551813Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-07T18:07:05.9552391Z for _ in $(seq 1440); do 2024-08-07T18:07:05.9552803Z  # Break if no ssh session exists anymore 2024-08-07T18:07:05.9553253Z  if [ "$(who)" = "" ]; then 2024-08-07T18:07:05.9553642Z  break 2024-08-07T18:07:05.9554006Z  fi 2024-08-07T18:07:05.9554296Z  echo "." 2024-08-07T18:07:05.9554620Z  sleep 5 2024-08-07T18:07:05.9554937Z done 2024-08-07T18:07:05.9562395Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:07:05.9562886Z env: 2024-08-07T18:07:05.9563180Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:07:05.9563590Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:07:05.9564276Z DOCKER_CONTAINER_ID: ddfc464047ce056b8e984eda0cf5fdb3a03b3de1ff51933a15b85473696b0056 2024-08-07T18:07:05.9564903Z ##[endgroup] 2024-08-07T18:07:05.9588769Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-07T18:07:05.9645076Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-07T18:07:05.9645767Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-07T18:07:05.9646291Z # shellcheck disable=SC2046 2024-08-07T18:07:05.9646737Z docker stop $(docker ps -q) || true 2024-08-07T18:07:05.9647236Z # Prune all of the docker images 2024-08-07T18:07:05.9647641Z docker system prune -af 2024-08-07T18:07:05.9654992Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:07:05.9655475Z env: 2024-08-07T18:07:05.9655783Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:07:05.9656197Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:07:05.9656897Z DOCKER_CONTAINER_ID: ddfc464047ce056b8e984eda0cf5fdb3a03b3de1ff51933a15b85473696b0056 2024-08-07T18:07:05.9657520Z ##[endgroup] 2024-08-07T18:07:06.2665014Z ddfc464047ce 2024-08-07T18:07:06.8229049Z Deleted Containers: 2024-08-07T18:07:06.8229946Z ddfc464047ce056b8e984eda0cf5fdb3a03b3de1ff51933a15b85473696b0056 2024-08-07T18:07:06.8230401Z 2024-08-07T18:07:13.4483571Z Deleted Images: 2024-08-07T18:07:13.4484655Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T18:07:13.4487276Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9@sha256:66181a134b3ebc35388dabcc46dc11e767152fd72514dad76711590f50b650f7 2024-08-07T18:07:13.4488611Z deleted: sha256:8e4e19ae0377c273bc603b24bc49d406d6f755072d4534d040c86afbe761a129 2024-08-07T18:07:13.4489444Z deleted: sha256:bee5df81b6fe1ee0aae7e536a9bda16351d0515be5b789ad50bf0ac60fd41e33 2024-08-07T18:07:13.4490235Z deleted: sha256:8ee1ee099b65b2d73d66dc3330f82f8be2dd6cbdccf3d718b448a5e3419c214e 2024-08-07T18:07:13.4491031Z deleted: sha256:aed7424cdc4428e32e398a33e98e6e252075e6280e7068e20df7626451fa312a 2024-08-07T18:07:13.4491809Z deleted: sha256:61b7d2994ed16481c3de07839b5a3bb779dc2f3f716ad49acb91537042c3b7d1 2024-08-07T18:07:13.4492577Z deleted: sha256:79999dfec9b5df258486b014aa99c30979900c4131effd2a7bc18251cd5fc372 2024-08-07T18:07:13.4493367Z deleted: sha256:9020e3bab07b431af26c788ac2d12cbca9a76ffb1c923ea45a38e872942e1f34 2024-08-07T18:07:13.4494441Z deleted: sha256:387e2f2f359df5f69c500ae9589c224d8283ae5418520509b3c9ef0a45d116fa 2024-08-07T18:07:13.4495264Z deleted: sha256:5ddfc73c14e2e0a762c51ddecfb2710258366e1d048a8bf56d70268486907b99 2024-08-07T18:07:13.4496037Z deleted: sha256:4dffb251634d1c8105e68310ef1b2ffb923dac4c4139f494be0938ad032a1bca 2024-08-07T18:07:13.4497037Z deleted: sha256:e0df7617281f0e4396253d2f0c40d1418ad604dc0912cae13f153fe22e316fa0 2024-08-07T18:07:13.4497822Z deleted: sha256:d239b3ccf05c2850364c3f73acbc6e0017a25f8faa4c3abaa8524e63dc9f9c29 2024-08-07T18:07:13.4498592Z deleted: sha256:918d0912faf4251f3be58f9a5aaa80f59a180ace4771b9ce903cedf94a57723a 2024-08-07T18:07:13.4499362Z deleted: sha256:09d5696392736fa88c52a0656d1d831d2bcb4015630810412565a5a086eb98ff 2024-08-07T18:07:13.4500118Z deleted: sha256:67542af70388f69b345fc7f0965e1fce3c552ee987689f804c8406536f16534f 2024-08-07T18:07:13.4500880Z deleted: sha256:350337bafbf2976e8d4e0b7ce3cc66b2cb43ebb8622fa6e538b02f561f929393 2024-08-07T18:07:13.4501692Z deleted: sha256:00fe9fe12fefe10d5dbfb2f855aa385e840f360deeccbc9ee92cae51c6eea4ba 2024-08-07T18:07:13.4502495Z deleted: sha256:9895af0586ff7d94936462a92d7cec0d8ec3b01df27b664f7e4ac277ce7bbe9e 2024-08-07T18:07:13.4503263Z deleted: sha256:7dd7dbc3ec792cf9d043ff867afe169d15fe414138097ad9247a5f388d3eeccf 2024-08-07T18:07:13.4504093Z deleted: sha256:e0a3ace1b87c67d71f5db76d211740431660a7fbcd80415333c9e7b7b9a2e89b 2024-08-07T18:07:13.4504878Z deleted: sha256:b9c56b5040d5d5998961b788563b85e5be12f678bc6b3976d089a81f1c93b7e8 2024-08-07T18:07:13.4505636Z deleted: sha256:9232fe8591973938b7bdb0a7a31dcdbc79f9667d6e011a2facf4e17fb77d4922 2024-08-07T18:07:13.4506422Z deleted: sha256:52f5fbe778d735123327825c6a16dcec9bc84c2623087bb4c7c094ac99764ae7 2024-08-07T18:07:13.4508054Z deleted: sha256:4b7c4121b76b04d5c7e4a0b77add435c0caa99cccff506ea704eba27fd2b2166 2024-08-07T18:07:13.4509057Z deleted: sha256:795dd765ace5b617808fc40cd576b449ae01b8671290e6df6b49d86ea79f4145 2024-08-07T18:07:13.4509835Z deleted: sha256:51cc651f1ffbeef49ad010ab2d4cf2d5ce5f8f5767c2133346a959c773d837bb 2024-08-07T18:07:13.4510624Z deleted: sha256:4c3019549b40f7a268d344eae162d7a7cc78855880db865944da136e01c190e9 2024-08-07T18:07:13.4511414Z deleted: sha256:855b789ec056edaab8cc8dad50a82e9dbfb9b20bcf39c70d9c3bd40080b2b65f 2024-08-07T18:07:13.4512192Z deleted: sha256:cef853cb7e0f5d6351692a4b9a0413a2496ea3c7bfe5d8063528ac2d6c17696f 2024-08-07T18:07:13.4512977Z deleted: sha256:4f7b163ef5ff693ce9f7380bdf7b526131f2567489342f0630949c9341e1c63f 2024-08-07T18:07:13.4513755Z deleted: sha256:09f12e5e102232b4ec9fd5168f4aa4b20ee0dfc755b053ac57690657e72e3fe3 2024-08-07T18:07:13.4514518Z deleted: sha256:c9962be393e03353af0d6a26f5478a94cacb5e528ee3f97609fac594c9a5d387 2024-08-07T18:07:13.4515303Z deleted: sha256:ac7aa4ec25b0d9a035c438effaa309071729de3f3a0f96e8e481b9839e7460b8 2024-08-07T18:07:13.4516082Z deleted: sha256:63ff2106018988888d022ee38d6ce1805aa29d8368a18dec1ab8eb587799ab44 2024-08-07T18:07:13.4516838Z deleted: sha256:3298c1acdeb701f11351366cfe680a84b8290666fdcd76af5ac79f226422fc0d 2024-08-07T18:07:13.4517633Z deleted: sha256:f076cc29eff5dae969bc7c05003d680b6e683eaef971c0a3f67a8a4d12693ee4 2024-08-07T18:07:13.4518416Z deleted: sha256:2604655e98c557e6de2e650d21cd0306c01f12204f3538386a9f374d003a4d70 2024-08-07T18:07:13.4519183Z deleted: sha256:fb942194d6d4fc6f9c8a4c8cba17330563877c92369892edc9b005b8b491f204 2024-08-07T18:07:13.4519957Z deleted: sha256:d349cc387708a540eee839ccd78494458dc5f1ccf989db608d25c5f798ec8c76 2024-08-07T18:07:13.4520766Z deleted: sha256:269864865f9b12ee399ca9ee3e15a16cedb86fd4889e01a83567afb357a774e6 2024-08-07T18:07:13.4521575Z deleted: sha256:236a91ce743cd500427338a78915ff834ae80331fcc06afccfca82a35bffb63c 2024-08-07T18:07:13.4522371Z deleted: sha256:de968bbc1d8bc2ad420d6a1c78a9a6cf8a763bb855ca3aefe7de6c5c52201f7a 2024-08-07T18:07:13.4523160Z deleted: sha256:0f15ecc513ffdac89965bd76cd9f8c7f4e37bbe7be0bfa086353133e2dd4f086 2024-08-07T18:07:13.4523964Z deleted: sha256:754bb9b5a5af40b6ad88f0ab14be37b9e512e25edccbfc9fd5e0c5a88ca09c0f 2024-08-07T18:07:13.4524949Z deleted: sha256:427104437dfa1883feec2c441eb4e4f825481883c6d3d31759361cbc924a27fe 2024-08-07T18:07:13.4525752Z deleted: sha256:0b404434cf2c3087ef28ea3cdc5813cc555f6e261c2514a7ab5bf5d484b5b575 2024-08-07T18:07:13.4526583Z deleted: sha256:bfb91e062bab9dafb63d0d4ab366026692fbd63e642da9ab28f04c8bba9b56a3 2024-08-07T18:07:13.4527367Z deleted: sha256:54617b36a396e9e82f2092257d2f621c90de460e0e54b1cf8df065573fbaa0ba 2024-08-07T18:07:13.4528582Z deleted: sha256:d8bdc337db1750bb3ebc7c8e4e1bd48e722218f261cb35ec2056ceaa6d3c4e9c 2024-08-07T18:07:13.4529381Z deleted: sha256:25cbacf4be7b2bbcbeaf7375d0eff0eaa33e94c5bef20ff0993bf074fdaa54eb 2024-08-07T18:07:13.4530184Z deleted: sha256:43744f05dcf6dc44c2f5e06533b13e085319a78b6b5e17eba5e7bd4419f5c905 2024-08-07T18:07:13.4530965Z deleted: sha256:2ea7d74f63d16a8d6e14a87e0e916f67f99536b21ac53e80b27786dcd1ce36fd 2024-08-07T18:07:13.4531729Z deleted: sha256:dbfb6031f18e50c89ff1a268c1e80e2fa88e727c48656718b326b60e8ef0d383 2024-08-07T18:07:13.4532527Z deleted: sha256:3eb3bb1755fcc3d4aba9abe2d2270ed2a55ec6254b3e87ca7688370e67e707a3 2024-08-07T18:07:13.4533314Z deleted: sha256:04605aba2d51924a85499584edda24c28390eb777c1c88eff2f4191b48e3da8c 2024-08-07T18:07:13.4534077Z deleted: sha256:117ff9e5eddba4582a39d416380eadf1c245237612c5ae0d6781d36c36bcb87f 2024-08-07T18:07:13.4534868Z deleted: sha256:ebf86c84eafa12a81b8eb66541a56c8056f4f9bc30e4be761771e53c6e1342aa 2024-08-07T18:07:13.4535666Z deleted: sha256:23d7f7e0c9d7e60acb59e60da9b4ee4e1840c1c406bf2082822c49430975529d 2024-08-07T18:07:13.4536431Z deleted: sha256:3e7e5583645195af5eeb60fe49024a4aa2f0eb3be28ff95c9859bb6233c29c4b 2024-08-07T18:07:13.4537209Z deleted: sha256:45b8f0dd78e72f0156b1fbc4dcb14200c61954f37a75129c753275e5a45ccd9b 2024-08-07T18:07:13.4537992Z deleted: sha256:f6b7a042384e2f2bb3401da2b9f4e9b9f942adb70be7e7f89c46fa81a42774d8 2024-08-07T18:07:13.4538766Z deleted: sha256:eb547a7b8fe732bffe8cc9a66aa19f2b41f6e0616a8a27040307c022d00fbc0d 2024-08-07T18:07:13.4539550Z deleted: sha256:20f729283bb4f2063f3a8baa0b2afc17911da345bf367e1918d8cc09ddd6d964 2024-08-07T18:07:13.4540369Z deleted: sha256:8ce8da9ab0319625a66c35d24ddadf8d2688d16c5024ba2345099431e0aa7ff8 2024-08-07T18:07:13.4541155Z deleted: sha256:e7a91181c74f879360530f0bbe957b8ff96a1ec19adde8a41d9e3fa2038d0d16 2024-08-07T18:07:13.4541935Z deleted: sha256:212b7a0fad7208624a2bc56768d8c489d2690142d0db0fa20f69eb648234e8d1 2024-08-07T18:07:13.4542703Z deleted: sha256:72315e37f51ba8bcc88fc5b56f92e5bb6264456d9a755afa01fd19f9128c1df7 2024-08-07T18:07:13.4543507Z deleted: sha256:bfc5ab07cfab1698147152988eabdc7ebbdeb35b6d9aefd9360dd64104cd3888 2024-08-07T18:07:13.4544299Z deleted: sha256:89edf8c906678c217b9a88160e1561991f53909c3fd27a4a5e1f0ca0f7ff7888 2024-08-07T18:07:13.4545070Z deleted: sha256:d5ba67f8f9b0c2b6da06d2d5d9d5fedc9f17cbe1a5e13a1a4033a4619f4528ba 2024-08-07T18:07:13.4545856Z deleted: sha256:5d61311e1a293c0c9642e34871719525dfeec7028e4b45e08344a469e3fe24bf 2024-08-07T18:07:13.4546632Z deleted: sha256:b89bb70f745a94c0edd5e974aee294e160735f76fd666a1a06a0b9b1be3caf63 2024-08-07T18:07:13.4547842Z deleted: sha256:9e5477f1f21a38c3dfe21a56a7d77d1b1a42453aed3e11729afcbb9d5731770b 2024-08-07T18:07:13.4548683Z deleted: sha256:b76f4af91cffe2bb7b3f09960f5bdbb26f741419defda65c451c4407fc5b0a92 2024-08-07T18:07:13.4549478Z deleted: sha256:3f8748a00ebd394c0e9d9d481972d764dff58896841ef0150b3f7efaac0f4a4a 2024-08-07T18:07:13.4550237Z deleted: sha256:13989c771e3ba4d40f618344e55e5cb7062a35041b5efd69e2f740ba819f46ee 2024-08-07T18:07:13.4551036Z deleted: sha256:9ebb27837f6c67ad8c17e9af4e63c008faae03fa226c3641dd8bbbfeefb106c4 2024-08-07T18:07:13.4551817Z deleted: sha256:8c16907d735950f8093582bc3cf392b70e002456bf9c4224de5a5b86b78b3ea2 2024-08-07T18:07:13.4552597Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-07T18:07:13.4553060Z 2024-08-07T18:07:13.4553216Z Total reclaimed space: 26.03GB 2024-08-07T18:07:13.4622209Z Post job cleanup. 2024-08-07T18:07:13.4680486Z Post job cleanup. 2024-08-07T18:07:13.5852333Z [command]/usr/bin/git version 2024-08-07T18:07:13.5905725Z git version 2.40.1 2024-08-07T18:07:13.5968497Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/8dabaefb-7daa-482a-8531-1ef1656ec20f' before making global git config changes 2024-08-07T18:07:13.5969691Z Adding repository directory to the temporary git global config as a safe directory 2024-08-07T18:07:13.5976449Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-07T18:07:13.6018982Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-07T18:07:13.6055542Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-07T18:07:13.6372006Z Entering 'android/libs/fbjni' 2024-08-07T18:07:13.6423009Z Entering 'third_party/FP16' 2024-08-07T18:07:13.6475492Z Entering 'third_party/FXdiv' 2024-08-07T18:07:13.6526470Z Entering 'third_party/NNPACK' 2024-08-07T18:07:13.6577940Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-07T18:07:13.6630267Z Entering 'third_party/XNNPACK' 2024-08-07T18:07:13.6699397Z Entering 'third_party/benchmark' 2024-08-07T18:07:13.6752682Z Entering 'third_party/cpp-httplib' 2024-08-07T18:07:13.6801887Z Entering 'third_party/cpuinfo' 2024-08-07T18:07:13.6853337Z Entering 'third_party/cudnn_frontend' 2024-08-07T18:07:13.6903451Z Entering 'third_party/cutlass' 2024-08-07T18:07:13.6962692Z Entering 'third_party/eigen' 2024-08-07T18:07:13.7015130Z Entering 'third_party/fbgemm' 2024-08-07T18:07:13.7067103Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-07T18:07:13.7117059Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T18:07:13.7167731Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-07T18:07:13.7226987Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-07T18:07:13.7276682Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T18:07:13.7327514Z Entering 'third_party/flatbuffers' 2024-08-07T18:07:13.7382244Z Entering 'third_party/fmt' 2024-08-07T18:07:13.7434547Z Entering 'third_party/foxi' 2024-08-07T18:07:13.7485569Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-07T18:07:13.7536900Z Entering 'third_party/gloo' 2024-08-07T18:07:13.7588141Z Entering 'third_party/googletest' 2024-08-07T18:07:13.7639572Z Entering 'third_party/ideep' 2024-08-07T18:07:13.7689866Z Entering 'third_party/ideep/mkl-dnn' 2024-08-07T18:07:13.7749306Z Entering 'third_party/ittapi' 2024-08-07T18:07:13.7800275Z Entering 'third_party/kineto' 2024-08-07T18:07:13.7850792Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T18:07:13.7901604Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T18:07:13.7954082Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T18:07:13.8005751Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T18:07:13.8056304Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T18:07:13.8105635Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T18:07:13.8158354Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T18:07:13.8210647Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T18:07:13.8262212Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T18:07:13.8315478Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T18:07:13.8368804Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T18:07:13.8417960Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T18:07:13.8470175Z Entering 'third_party/mimalloc' 2024-08-07T18:07:13.8522838Z Entering 'third_party/nccl/nccl' 2024-08-07T18:07:13.8574006Z Entering 'third_party/nlohmann' 2024-08-07T18:07:13.8627403Z Entering 'third_party/onnx' 2024-08-07T18:07:13.8697012Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-07T18:07:13.8749540Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-07T18:07:13.8802748Z Entering 'third_party/opentelemetry-cpp' 2024-08-07T18:07:13.8856224Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T18:07:13.8906330Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T18:07:13.8957775Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T18:07:13.9007531Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T18:07:13.9059164Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T18:07:13.9109567Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T18:07:13.9159491Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T18:07:13.9209383Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T18:07:13.9262382Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T18:07:13.9314208Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T18:07:13.9386883Z Entering 'third_party/pocketfft' 2024-08-07T18:07:13.9438738Z Entering 'third_party/protobuf' 2024-08-07T18:07:13.9493265Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-07T18:07:13.9544187Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-07T18:07:13.9598417Z Entering 'third_party/psimd' 2024-08-07T18:07:13.9649243Z Entering 'third_party/pthreadpool' 2024-08-07T18:07:13.9700636Z Entering 'third_party/pybind11' 2024-08-07T18:07:13.9752953Z Entering 'third_party/python-peachpy' 2024-08-07T18:07:13.9803663Z Entering 'third_party/sleef' 2024-08-07T18:07:13.9854043Z Entering 'third_party/tensorpipe' 2024-08-07T18:07:13.9904510Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-07T18:07:13.9956174Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-07T18:07:14.0005537Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-07T18:07:14.0055087Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T18:07:14.0104807Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T18:07:14.0176196Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-07T18:07:14.0203116Z http.https://github.com/.extraheader 2024-08-07T18:07:14.0214786Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-07T18:07:14.0252672Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-07T18:07:14.0561896Z Entering 'android/libs/fbjni' 2024-08-07T18:07:14.0595188Z http.https://github.com/.extraheader 2024-08-07T18:07:14.0629403Z Entering 'third_party/FP16' 2024-08-07T18:07:14.0662671Z http.https://github.com/.extraheader 2024-08-07T18:07:14.0696838Z Entering 'third_party/FXdiv' 2024-08-07T18:07:14.0729865Z http.https://github.com/.extraheader 2024-08-07T18:07:14.0763752Z Entering 'third_party/NNPACK' 2024-08-07T18:07:14.0797091Z http.https://github.com/.extraheader 2024-08-07T18:07:14.0831362Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-07T18:07:14.0864101Z http.https://github.com/.extraheader 2024-08-07T18:07:14.0898752Z Entering 'third_party/XNNPACK' 2024-08-07T18:07:14.0932108Z http.https://github.com/.extraheader 2024-08-07T18:07:14.0984880Z Entering 'third_party/benchmark' 2024-08-07T18:07:14.1017633Z http.https://github.com/.extraheader 2024-08-07T18:07:14.1051376Z Entering 'third_party/cpp-httplib' 2024-08-07T18:07:14.1085131Z http.https://github.com/.extraheader 2024-08-07T18:07:14.1119645Z Entering 'third_party/cpuinfo' 2024-08-07T18:07:14.1152645Z http.https://github.com/.extraheader 2024-08-07T18:07:14.1186242Z Entering 'third_party/cudnn_frontend' 2024-08-07T18:07:14.1219452Z http.https://github.com/.extraheader 2024-08-07T18:07:14.1253799Z Entering 'third_party/cutlass' 2024-08-07T18:07:14.1287067Z http.https://github.com/.extraheader 2024-08-07T18:07:14.1329267Z Entering 'third_party/eigen' 2024-08-07T18:07:14.1362458Z http.https://github.com/.extraheader 2024-08-07T18:07:14.1398846Z Entering 'third_party/fbgemm' 2024-08-07T18:07:14.1431761Z http.https://github.com/.extraheader 2024-08-07T18:07:14.1465364Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-07T18:07:14.1498577Z http.https://github.com/.extraheader 2024-08-07T18:07:14.1531907Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T18:07:14.1564192Z http.https://github.com/.extraheader 2024-08-07T18:07:14.1598782Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-07T18:07:14.1630581Z http.https://github.com/.extraheader 2024-08-07T18:07:14.1671151Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-07T18:07:14.1703913Z http.https://github.com/.extraheader 2024-08-07T18:07:14.1737743Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T18:07:14.1769619Z http.https://github.com/.extraheader 2024-08-07T18:07:14.1803707Z Entering 'third_party/flatbuffers' 2024-08-07T18:07:14.1837410Z http.https://github.com/.extraheader 2024-08-07T18:07:14.1874454Z Entering 'third_party/fmt' 2024-08-07T18:07:14.1906503Z http.https://github.com/.extraheader 2024-08-07T18:07:14.1939800Z Entering 'third_party/foxi' 2024-08-07T18:07:14.1974161Z http.https://github.com/.extraheader 2024-08-07T18:07:14.2007666Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-07T18:07:14.2040720Z http.https://github.com/.extraheader 2024-08-07T18:07:14.2074687Z Entering 'third_party/gloo' 2024-08-07T18:07:14.2107115Z http.https://github.com/.extraheader 2024-08-07T18:07:14.2141059Z Entering 'third_party/googletest' 2024-08-07T18:07:14.2173750Z http.https://github.com/.extraheader 2024-08-07T18:07:14.2207469Z Entering 'third_party/ideep' 2024-08-07T18:07:14.2240924Z http.https://github.com/.extraheader 2024-08-07T18:07:14.2273180Z Entering 'third_party/ideep/mkl-dnn' 2024-08-07T18:07:14.2304784Z http.https://github.com/.extraheader 2024-08-07T18:07:14.2345832Z Entering 'third_party/ittapi' 2024-08-07T18:07:14.2378443Z http.https://github.com/.extraheader 2024-08-07T18:07:14.2412469Z Entering 'third_party/kineto' 2024-08-07T18:07:14.2445012Z http.https://github.com/.extraheader 2024-08-07T18:07:14.2478336Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T18:07:14.2510779Z http.https://github.com/.extraheader 2024-08-07T18:07:14.2544430Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T18:07:14.2577350Z http.https://github.com/.extraheader 2024-08-07T18:07:14.2612820Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T18:07:14.2645769Z http.https://github.com/.extraheader 2024-08-07T18:07:14.2680572Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T18:07:14.2713028Z http.https://github.com/.extraheader 2024-08-07T18:07:14.2746607Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T18:07:14.2779518Z http.https://github.com/.extraheader 2024-08-07T18:07:14.2813648Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T18:07:14.2846822Z http.https://github.com/.extraheader 2024-08-07T18:07:14.2882293Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T18:07:14.2915425Z http.https://github.com/.extraheader 2024-08-07T18:07:14.2949749Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T18:07:14.2981100Z http.https://github.com/.extraheader 2024-08-07T18:07:14.3015351Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T18:07:14.3049148Z http.https://github.com/.extraheader 2024-08-07T18:07:14.3084407Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T18:07:14.3116670Z http.https://github.com/.extraheader 2024-08-07T18:07:14.3152367Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T18:07:14.3183724Z http.https://github.com/.extraheader 2024-08-07T18:07:14.3218147Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T18:07:14.3250411Z http.https://github.com/.extraheader 2024-08-07T18:07:14.3286280Z Entering 'third_party/mimalloc' 2024-08-07T18:07:14.3319354Z http.https://github.com/.extraheader 2024-08-07T18:07:14.3353605Z Entering 'third_party/nccl/nccl' 2024-08-07T18:07:14.3385788Z http.https://github.com/.extraheader 2024-08-07T18:07:14.3419324Z Entering 'third_party/nlohmann' 2024-08-07T18:07:14.3452254Z http.https://github.com/.extraheader 2024-08-07T18:07:14.3486939Z Entering 'third_party/onnx' 2024-08-07T18:07:14.3520269Z http.https://github.com/.extraheader 2024-08-07T18:07:14.3571706Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-07T18:07:14.3605462Z http.https://github.com/.extraheader 2024-08-07T18:07:14.3639654Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-07T18:07:14.3671881Z http.https://github.com/.extraheader 2024-08-07T18:07:14.3707560Z Entering 'third_party/opentelemetry-cpp' 2024-08-07T18:07:14.3740257Z http.https://github.com/.extraheader 2024-08-07T18:07:14.3777374Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T18:07:14.3809318Z http.https://github.com/.extraheader 2024-08-07T18:07:14.3844120Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T18:07:14.3876992Z http.https://github.com/.extraheader 2024-08-07T18:07:14.3910364Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T18:07:14.3941650Z http.https://github.com/.extraheader 2024-08-07T18:07:14.3975203Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T18:07:14.4007406Z http.https://github.com/.extraheader 2024-08-07T18:07:14.4042105Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T18:07:14.4075513Z http.https://github.com/.extraheader 2024-08-07T18:07:14.4109069Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T18:07:14.4140206Z http.https://github.com/.extraheader 2024-08-07T18:07:14.4173998Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T18:07:14.4206509Z http.https://github.com/.extraheader 2024-08-07T18:07:14.4240782Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T18:07:14.4272824Z http.https://github.com/.extraheader 2024-08-07T18:07:14.4307726Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T18:07:14.4339514Z http.https://github.com/.extraheader 2024-08-07T18:07:14.4375102Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T18:07:14.4407214Z http.https://github.com/.extraheader 2024-08-07T18:07:14.4464838Z Entering 'third_party/pocketfft' 2024-08-07T18:07:14.4497969Z http.https://github.com/.extraheader 2024-08-07T18:07:14.4531532Z Entering 'third_party/protobuf' 2024-08-07T18:07:14.4565145Z http.https://github.com/.extraheader 2024-08-07T18:07:14.4602077Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-07T18:07:14.4636300Z http.https://github.com/.extraheader 2024-08-07T18:07:14.4669356Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-07T18:07:14.4700579Z http.https://github.com/.extraheader 2024-08-07T18:07:14.4735361Z Entering 'third_party/psimd' 2024-08-07T18:07:14.4768479Z http.https://github.com/.extraheader 2024-08-07T18:07:14.4802709Z Entering 'third_party/pthreadpool' 2024-08-07T18:07:14.4835462Z http.https://github.com/.extraheader 2024-08-07T18:07:14.4869363Z Entering 'third_party/pybind11' 2024-08-07T18:07:14.4901901Z http.https://github.com/.extraheader 2024-08-07T18:07:14.4936346Z Entering 'third_party/python-peachpy' 2024-08-07T18:07:14.4969265Z http.https://github.com/.extraheader 2024-08-07T18:07:14.5002896Z Entering 'third_party/sleef' 2024-08-07T18:07:14.5037039Z http.https://github.com/.extraheader 2024-08-07T18:07:14.5070637Z Entering 'third_party/tensorpipe' 2024-08-07T18:07:14.5102981Z http.https://github.com/.extraheader 2024-08-07T18:07:14.5137062Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-07T18:07:14.5168912Z http.https://github.com/.extraheader 2024-08-07T18:07:14.5202257Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-07T18:07:14.5234757Z http.https://github.com/.extraheader 2024-08-07T18:07:14.5268052Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-07T18:07:14.5300158Z http.https://github.com/.extraheader 2024-08-07T18:07:14.5333913Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T18:07:14.5365885Z http.https://github.com/.extraheader 2024-08-07T18:07:14.5398545Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T18:07:14.5431259Z http.https://github.com/.extraheader 2024-08-07T18:07:14.5647850Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-07T18:07:14.5677331Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-07T18:07:14.5684355Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:07:14.5684856Z ##[endgroup] 2024-08-07T18:07:20.5112265Z Cleaning up orphan processes