2024-05-19T18:07:05.0307498Z Current runner version: '2.316.1' 2024-05-19T18:07:05.0313866Z Runner name: 'i-0e8f077216c874026' 2024-05-19T18:07:05.0314731Z Runner group name: 'Default' 2024-05-19T18:07:05.0315667Z Machine name: 'ip-10-0-34-75' 2024-05-19T18:07:05.0320569Z ##[group]GITHUB_TOKEN Permissions 2024-05-19T18:07:05.0322586Z Actions: read 2024-05-19T18:07:05.0323192Z Attestations: read 2024-05-19T18:07:05.0323765Z Checks: read 2024-05-19T18:07:05.0324292Z Contents: read 2024-05-19T18:07:05.0324863Z Deployments: read 2024-05-19T18:07:05.0325466Z Discussions: read 2024-05-19T18:07:05.0326069Z Issues: read 2024-05-19T18:07:05.0326694Z Metadata: read 2024-05-19T18:07:05.0327224Z Packages: read 2024-05-19T18:07:05.0327767Z Pages: read 2024-05-19T18:07:05.0328332Z PullRequests: read 2024-05-19T18:07:05.0328965Z RepositoryProjects: read 2024-05-19T18:07:05.0329596Z SecurityEvents: read 2024-05-19T18:07:05.0330195Z Statuses: read 2024-05-19T18:07:05.0330740Z ##[endgroup] 2024-05-19T18:07:05.0334016Z Secret source: Actions 2024-05-19T18:07:05.0334849Z Prepare workflow directory 2024-05-19T18:07:05.3780889Z Prepare all required actions 2024-05-19T18:07:05.3965847Z Getting action download info 2024-05-19T18:07:05.5715569Z Download action repository 'pytorch/test-infra@main' (SHA:5ab7a0ff6617c37f26f7d3ce525c072902906bdf) 2024-05-19T18:07:05.8685438Z Download action repository 'pytorch/pytorch@main' (SHA:853081a8e7e84ce3f09749929bb3cb998fd95b30) 2024-05-19T18:07:08.5782370Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-05-19T18:07:08.7170195Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-05-19T18:07:08.9655665Z Getting action download info 2024-05-19T18:07:09.0840761Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-05-19T18:07:09.3108747Z Getting action download info 2024-05-19T18:07:09.4141694Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-05-19T18:07:09.5576013Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/121956 (c8c7ddfde57442cc28a394c385cab656ce9b1f5e) 2024-05-19T18:07:09.5578227Z ##[group] Inputs 2024-05-19T18:07:09.5578703Z build-environment: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:07:09.5586910Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-05-19T18:07:09.5596064Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:07:09.5597250Z sync-tag: 2024-05-19T18:07:09.5598076Z timeout-minutes: 240 2024-05-19T18:07:09.5598419Z use-gha: 2024-05-19T18:07:09.5598697Z dashboard-tag: 2024-05-19T18:07:09.5599015Z s3-bucket: gha-artifacts 2024-05-19T18:07:09.5599374Z aws-role-to-assume: 2024-05-19T18:07:09.5599706Z ##[endgroup] 2024-05-19T18:07:09.5600619Z Complete job name: cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:07:09.6411924Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-05-19T18:07:09.6412490Z with: 2024-05-19T18:07:09.6413126Z github-secret: *** 2024-05-19T18:07:09.6414048Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-05-19T18:07:09.6415040Z activate-with-label: false 2024-05-19T18:07:09.6415405Z label: with-ssh 2024-05-19T18:07:09.6415739Z remove-existing-keys: true 2024-05-19T18:07:09.6416122Z env: 2024-05-19T18:07:09.6416398Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:07:09.6416748Z ##[endgroup] 2024-05-19T18:07:09.7333107Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-05-19T18:07:09.7334123Z ciflow reference detected, attempting to extract PR number 2024-05-19T18:07:11.5875514Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-05-19T18:07:11.6455948Z No SSH keys found for user pytorch-bot[bot] 2024-05-19T18:07:11.6456739Z Grabbing public ssh keys from https://github.com/nWEIdia.keys 2024-05-19T18:07:11.7074261Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-05-19T18:07:11.7088347Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-05-19T18:07:11.7111983Z Login using: ssh ec2-user@ec2-54-174-69-89.compute-1.amazonaws.com 2024-05-19T18:07:11.7113073Z All testing is done inside the container, to start an interactive session run: 2024-05-19T18:07:11.7113890Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-05-19T18:07:11.7377491Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-05-19T18:07:11.7378147Z with: 2024-05-19T18:07:11.7378423Z submodules: recursive 2024-05-19T18:07:11.7378749Z fetch-depth: 0 2024-05-19T18:07:11.7379035Z env: 2024-05-19T18:07:11.7379303Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:07:11.7379635Z ##[endgroup] 2024-05-19T18:07:11.7592083Z ##[group]Run retry () { 2024-05-19T18:07:11.7592509Z retry () { 2024-05-19T18:07:11.7593041Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-05-19T18:07:11.7593612Z } 2024-05-19T18:07:11.7593926Z echo "${GITHUB_WORKSPACE}" 2024-05-19T18:07:11.7594348Z if [ -z "${NO_SUDO}" ]; then 2024-05-19T18:07:11.7594838Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-05-19T18:07:11.7595303Z else 2024-05-19T18:07:11.7595657Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-05-19T18:07:11.7596079Z fi 2024-05-19T18:07:11.7596428Z mkdir "${GITHUB_WORKSPACE}" 2024-05-19T18:07:11.7608040Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:07:11.7608553Z env: 2024-05-19T18:07:11.7608841Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:07:11.7609195Z NO_SUDO: 2024-05-19T18:07:11.7609476Z ##[endgroup] 2024-05-19T18:07:11.7701648Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T18:07:14.0872992Z ##[group]Run malfet/checkout@silent-checkout 2024-05-19T18:07:14.0873513Z with: 2024-05-19T18:07:14.0873925Z ref: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:07:14.0874443Z fetch-depth: 0 2024-05-19T18:07:14.0874834Z submodules: recursive 2024-05-19T18:07:14.0875252Z quiet-checkout: true 2024-05-19T18:07:14.0875672Z repository: pytorch/pytorch 2024-05-19T18:07:14.0876238Z token: *** 2024-05-19T18:07:14.0876604Z ssh-strict: true 2024-05-19T18:07:14.0877007Z persist-credentials: true 2024-05-19T18:07:14.0877442Z clean: true 2024-05-19T18:07:14.0877845Z sparse-checkout-cone-mode: true 2024-05-19T18:07:14.0878311Z lfs: false 2024-05-19T18:07:14.0878687Z set-safe-directory: true 2024-05-19T18:07:14.0879091Z env: 2024-05-19T18:07:14.0879445Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:07:14.0879868Z ##[endgroup] 2024-05-19T18:07:14.1824333Z Syncing repository: pytorch/pytorch 2024-05-19T18:07:14.1825892Z ##[group]Getting Git version info 2024-05-19T18:07:14.1826669Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-05-19T18:07:14.1827556Z [command]/usr/bin/git version 2024-05-19T18:07:14.1827923Z git version 2.40.1 2024-05-19T18:07:14.1829238Z ##[endgroup] 2024-05-19T18:07:14.1841368Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/30ba9a05-6156-413b-aedb-d0d1904e7e15' before making global git config changes 2024-05-19T18:07:14.1842726Z Adding repository directory to the temporary git global config as a safe directory 2024-05-19T18:07:14.1846256Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T18:07:14.1872935Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-05-19T18:07:14.1876658Z ##[group]Initializing the repository 2024-05-19T18:07:14.1879350Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T18:07:14.1903122Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-05-19T18:07:14.1904381Z hint: is subject to change. To configure the initial branch name to use in all 2024-05-19T18:07:14.1905283Z hint: of your new repositories, which will suppress this warning, call: 2024-05-19T18:07:14.1905917Z hint: 2024-05-19T18:07:14.1906435Z hint: git config --global init.defaultBranch 2024-05-19T18:07:14.1906981Z hint: 2024-05-19T18:07:14.1907576Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-05-19T18:07:14.1908505Z hint: 'development'. The just-created branch can be renamed via this command: 2024-05-19T18:07:14.1909418Z hint: 2024-05-19T18:07:14.1909834Z hint: git branch -m 2024-05-19T18:07:14.1911982Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-05-19T18:07:14.1913777Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-05-19T18:07:14.1937197Z ##[endgroup] 2024-05-19T18:07:14.1937899Z ##[group]Disabling automatic garbage collection 2024-05-19T18:07:14.1939735Z [command]/usr/bin/git config --local gc.auto 0 2024-05-19T18:07:14.1964288Z ##[endgroup] 2024-05-19T18:07:14.1964927Z ##[group]Setting up auth 2024-05-19T18:07:14.1971185Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-05-19T18:07:14.1998195Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-05-19T18:07:14.2187537Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-05-19T18:07:14.2210331Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-05-19T18:07:14.2406024Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-05-19T18:07:14.2448846Z ##[endgroup] 2024-05-19T18:07:14.2449911Z ##[group]Fetching the repository 2024-05-19T18:07:14.2455972Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-05-19T18:07:18.5576250Z remote: Enumerating objects: 1118700 2024-05-19T18:07:18.5577110Z remote: Enumerating objects: 1119785, done. 2024-05-19T18:07:18.5578357Z remote: Counting objects: 0% (1/1085) 2024-05-19T18:07:18.5579213Z remote: Counting objects: 1% (11/1085) 2024-05-19T18:07:18.5580134Z remote: Counting objects: 2% (22/1085) 2024-05-19T18:07:18.5580984Z remote: Counting objects: 3% (33/1085) 2024-05-19T18:07:18.5581792Z remote: Counting objects: 4% (44/1085) 2024-05-19T18:07:18.5582387Z remote: Counting objects: 5% (55/1085) 2024-05-19T18:07:18.5583121Z remote: Counting objects: 6% (66/1085) 2024-05-19T18:07:18.5584008Z remote: Counting objects: 7% (76/1085) 2024-05-19T18:07:18.5584745Z remote: Counting objects: 8% (87/1085) 2024-05-19T18:07:18.5585359Z remote: Counting objects: 9% (98/1085) 2024-05-19T18:07:18.5586139Z remote: Counting objects: 10% (109/1085) 2024-05-19T18:07:18.5586933Z remote: Counting objects: 11% (120/1085) 2024-05-19T18:07:18.5587733Z remote: Counting objects: 12% (131/1085) 2024-05-19T18:07:18.5588548Z remote: Counting objects: 13% (142/1085) 2024-05-19T18:07:18.5589148Z remote: Counting objects: 14% (152/1085) 2024-05-19T18:07:18.5589749Z remote: Counting objects: 15% (163/1085) 2024-05-19T18:07:18.5590551Z remote: Counting objects: 16% (174/1085) 2024-05-19T18:07:18.5591147Z remote: Counting objects: 17% (185/1085) 2024-05-19T18:07:18.5591725Z remote: Counting objects: 18% (196/1085) 2024-05-19T18:07:18.5592316Z remote: Counting objects: 19% (207/1085) 2024-05-19T18:07:18.5592904Z remote: Counting objects: 20% (217/1085) 2024-05-19T18:07:18.5593504Z remote: Counting objects: 21% (228/1085) 2024-05-19T18:07:18.5594086Z remote: Counting objects: 22% (239/1085) 2024-05-19T18:07:18.5594676Z remote: Counting objects: 23% (250/1085) 2024-05-19T18:07:18.5595265Z remote: Counting objects: 24% (261/1085) 2024-05-19T18:07:18.5595854Z remote: Counting objects: 25% (272/1085) 2024-05-19T18:07:18.5596432Z remote: Counting objects: 26% (283/1085) 2024-05-19T18:07:18.5597022Z remote: Counting objects: 27% (293/1085) 2024-05-19T18:07:18.5597814Z remote: Counting objects: 28% (304/1085) 2024-05-19T18:07:18.5598406Z remote: Counting objects: 29% (315/1085) 2024-05-19T18:07:18.5598980Z remote: Counting objects: 30% (326/1085) 2024-05-19T18:07:18.5599630Z remote: Counting objects: 31% (337/1085) 2024-05-19T18:07:18.5600217Z remote: Counting objects: 32% (348/1085) 2024-05-19T18:07:18.5600810Z remote: Counting objects: 33% (359/1085) 2024-05-19T18:07:18.5601395Z remote: Counting objects: 34% (369/1085) 2024-05-19T18:07:18.5601980Z remote: Counting objects: 35% (380/1085) 2024-05-19T18:07:18.5602570Z remote: Counting objects: 36% (391/1085) 2024-05-19T18:07:18.5603157Z remote: Counting objects: 37% (402/1085) 2024-05-19T18:07:18.5603738Z remote: Counting objects: 38% (413/1085) 2024-05-19T18:07:18.5604334Z remote: Counting objects: 39% (424/1085) 2024-05-19T18:07:18.5605045Z remote: Counting objects: 40% (434/1085) 2024-05-19T18:07:18.5605636Z remote: Counting objects: 41% (445/1085) 2024-05-19T18:07:18.5606208Z remote: Counting objects: 42% (456/1085) 2024-05-19T18:07:18.5606927Z remote: Counting objects: 43% (467/1085) 2024-05-19T18:07:18.5607535Z remote: Counting objects: 44% (478/1085) 2024-05-19T18:07:18.5608115Z remote: Counting objects: 45% (489/1085) 2024-05-19T18:07:18.5608840Z remote: Counting objects: 46% (500/1085) 2024-05-19T18:07:18.5609416Z remote: Counting objects: 47% (510/1085) 2024-05-19T18:07:18.5610008Z remote: Counting objects: 48% (521/1085) 2024-05-19T18:07:18.5610605Z remote: Counting objects: 49% (532/1085) 2024-05-19T18:07:18.5611195Z remote: Counting objects: 50% (543/1085) 2024-05-19T18:07:18.5611777Z remote: Counting objects: 51% (554/1085) 2024-05-19T18:07:18.5612368Z remote: Counting objects: 52% (565/1085) 2024-05-19T18:07:18.5612962Z remote: Counting objects: 53% (576/1085) 2024-05-19T18:07:18.5613564Z remote: Counting objects: 54% (586/1085) 2024-05-19T18:07:18.5614199Z remote: Counting objects: 55% (597/1085) 2024-05-19T18:07:18.5614791Z remote: Counting objects: 56% (608/1085) 2024-05-19T18:07:18.5615384Z remote: Counting objects: 57% (619/1085) 2024-05-19T18:07:18.5615976Z remote: Counting objects: 58% (630/1085) 2024-05-19T18:07:18.5616566Z remote: Counting objects: 59% (641/1085) 2024-05-19T18:07:18.5617157Z remote: Counting objects: 60% (651/1085) 2024-05-19T18:07:18.5617745Z remote: Counting objects: 61% (662/1085) 2024-05-19T18:07:18.5618335Z remote: Counting objects: 62% (673/1085) 2024-05-19T18:07:18.5618917Z remote: Counting objects: 63% (684/1085) 2024-05-19T18:07:18.5619511Z remote: Counting objects: 64% (695/1085) 2024-05-19T18:07:18.5620200Z remote: Counting objects: 65% (706/1085) 2024-05-19T18:07:18.5620786Z remote: Counting objects: 66% (717/1085) 2024-05-19T18:07:18.5621450Z remote: Counting objects: 67% (727/1085) 2024-05-19T18:07:18.5622040Z remote: Counting objects: 68% (738/1085) 2024-05-19T18:07:18.5622630Z remote: Counting objects: 69% (749/1085) 2024-05-19T18:07:18.5623272Z remote: Counting objects: 70% (760/1085) 2024-05-19T18:07:18.5623858Z remote: Counting objects: 71% (771/1085) 2024-05-19T18:07:18.5624451Z remote: Counting objects: 72% (782/1085) 2024-05-19T18:07:18.5625039Z remote: Counting objects: 73% (793/1085) 2024-05-19T18:07:18.5625615Z remote: Counting objects: 74% (803/1085) 2024-05-19T18:07:18.5626261Z remote: Counting objects: 75% (814/1085) 2024-05-19T18:07:18.5627000Z remote: Counting objects: 76% (825/1085) 2024-05-19T18:07:18.5627662Z remote: Counting objects: 77% (836/1085) 2024-05-19T18:07:18.5628317Z remote: Counting objects: 78% (847/1085) 2024-05-19T18:07:18.5628989Z remote: Counting objects: 79% (858/1085) 2024-05-19T18:07:18.5629581Z remote: Counting objects: 80% (868/1085) 2024-05-19T18:07:18.5630278Z remote: Counting objects: 81% (879/1085) 2024-05-19T18:07:18.5630858Z remote: Counting objects: 82% (890/1085) 2024-05-19T18:07:18.5631450Z remote: Counting objects: 83% (901/1085) 2024-05-19T18:07:18.5632042Z remote: Counting objects: 84% (912/1085) 2024-05-19T18:07:18.5632635Z remote: Counting objects: 85% (923/1085) 2024-05-19T18:07:18.5633216Z remote: Counting objects: 86% (934/1085) 2024-05-19T18:07:18.5633803Z remote: Counting objects: 87% (944/1085) 2024-05-19T18:07:18.5634443Z remote: Counting objects: 88% (955/1085) 2024-05-19T18:07:18.5635031Z remote: Counting objects: 89% (966/1085) 2024-05-19T18:07:18.5635609Z remote: Counting objects: 90% (977/1085) 2024-05-19T18:07:18.5636244Z remote: Counting objects: 91% (988/1085) 2024-05-19T18:07:18.5636889Z remote: Counting objects: 92% (999/1085) 2024-05-19T18:07:18.5637482Z remote: Counting objects: 93% (1010/1085) 2024-05-19T18:07:18.5638081Z remote: Counting objects: 94% (1020/1085) 2024-05-19T18:07:18.5638695Z remote: Counting objects: 95% (1031/1085) 2024-05-19T18:07:18.5639300Z remote: Counting objects: 96% (1042/1085) 2024-05-19T18:07:18.5639899Z remote: Counting objects: 97% (1053/1085) 2024-05-19T18:07:18.5640595Z remote: Counting objects: 98% (1064/1085) 2024-05-19T18:07:18.5641190Z remote: Counting objects: 99% (1075/1085) 2024-05-19T18:07:18.5641792Z remote: Counting objects: 100% (1085/1085) 2024-05-19T18:07:18.5642435Z remote: Counting objects: 100% (1085/1085), done. 2024-05-19T18:07:18.5643089Z remote: Compressing objects: 0% (1/377) 2024-05-19T18:07:18.5723573Z remote: Compressing objects: 1% (4/377) 2024-05-19T18:07:18.5959893Z remote: Compressing objects: 2% (8/377) 2024-05-19T18:07:18.6154931Z remote: Compressing objects: 3% (12/377) 2024-05-19T18:07:18.6507706Z remote: Compressing objects: 4% (16/377) 2024-05-19T18:07:18.6529269Z remote: Compressing objects: 5% (19/377) 2024-05-19T18:07:18.6529962Z remote: Compressing objects: 6% (23/377) 2024-05-19T18:07:18.6530583Z remote: Compressing objects: 7% (27/377) 2024-05-19T18:07:18.6531202Z remote: Compressing objects: 8% (31/377) 2024-05-19T18:07:18.6531831Z remote: Compressing objects: 9% (34/377) 2024-05-19T18:07:18.6532439Z remote: Compressing objects: 10% (38/377) 2024-05-19T18:07:18.6593056Z remote: Compressing objects: 11% (42/377) 2024-05-19T18:07:18.6594331Z remote: Compressing objects: 12% (46/377) 2024-05-19T18:07:18.6594966Z remote: Compressing objects: 13% (50/377) 2024-05-19T18:07:18.6595564Z remote: Compressing objects: 14% (53/377) 2024-05-19T18:07:18.6596167Z remote: Compressing objects: 15% (57/377) 2024-05-19T18:07:18.6596777Z remote: Compressing objects: 16% (61/377) 2024-05-19T18:07:18.6597387Z remote: Compressing objects: 17% (65/377) 2024-05-19T18:07:18.6605309Z remote: Compressing objects: 18% (68/377) 2024-05-19T18:07:18.6611110Z remote: Compressing objects: 19% (72/377) 2024-05-19T18:07:18.6615895Z remote: Compressing objects: 20% (76/377) 2024-05-19T18:07:18.6617097Z remote: Compressing objects: 21% (80/377) 2024-05-19T18:07:18.6623806Z remote: Compressing objects: 22% (83/377) 2024-05-19T18:07:18.6629791Z remote: Compressing objects: 23% (87/377) 2024-05-19T18:07:18.6634378Z remote: Compressing objects: 24% (91/377) 2024-05-19T18:07:18.6636860Z remote: Compressing objects: 25% (95/377) 2024-05-19T18:07:18.6637643Z remote: Compressing objects: 26% (99/377) 2024-05-19T18:07:18.6638249Z remote: Compressing objects: 27% (102/377) 2024-05-19T18:07:18.6638864Z remote: Compressing objects: 28% (106/377) 2024-05-19T18:07:18.6641589Z remote: Compressing objects: 29% (110/377) 2024-05-19T18:07:18.6643911Z remote: Compressing objects: 30% (114/377) 2024-05-19T18:07:18.6647043Z remote: Compressing objects: 31% (117/377) 2024-05-19T18:07:18.6651130Z remote: Compressing objects: 32% (121/377) 2024-05-19T18:07:18.6655646Z remote: Compressing objects: 33% (125/377) 2024-05-19T18:07:18.6658001Z remote: Compressing objects: 34% (129/377) 2024-05-19T18:07:18.6661681Z remote: Compressing objects: 35% (132/377) 2024-05-19T18:07:18.6663703Z remote: Compressing objects: 36% (136/377) 2024-05-19T18:07:18.6669268Z remote: Compressing objects: 37% (140/377) 2024-05-19T18:07:18.6670445Z remote: Compressing objects: 38% (144/377) 2024-05-19T18:07:18.6672751Z remote: Compressing objects: 39% (148/377) 2024-05-19T18:07:18.6674915Z remote: Compressing objects: 40% (151/377) 2024-05-19T18:07:18.6679731Z remote: Compressing objects: 41% (155/377) 2024-05-19T18:07:18.6681826Z remote: Compressing objects: 42% (159/377) 2024-05-19T18:07:18.6683830Z remote: Compressing objects: 43% (163/377) 2024-05-19T18:07:18.6684595Z remote: Compressing objects: 44% (166/377) 2024-05-19T18:07:18.6686261Z remote: Compressing objects: 45% (170/377) 2024-05-19T18:07:18.6687078Z remote: Compressing objects: 46% (174/377) 2024-05-19T18:07:18.6688377Z remote: Compressing objects: 47% (178/377) 2024-05-19T18:07:18.6690224Z remote: Compressing objects: 48% (181/377) 2024-05-19T18:07:18.6691055Z remote: Compressing objects: 49% (185/377) 2024-05-19T18:07:18.6691957Z remote: Compressing objects: 50% (189/377) 2024-05-19T18:07:18.6692628Z remote: Compressing objects: 51% (193/377) 2024-05-19T18:07:18.6693242Z remote: Compressing objects: 52% (197/377) 2024-05-19T18:07:18.6693868Z remote: Compressing objects: 53% (200/377) 2024-05-19T18:07:18.6694527Z remote: Compressing objects: 54% (204/377) 2024-05-19T18:07:18.6695382Z remote: Compressing objects: 55% (208/377) 2024-05-19T18:07:18.6696006Z remote: Compressing objects: 56% (212/377) 2024-05-19T18:07:18.6696716Z remote: Compressing objects: 57% (215/377) 2024-05-19T18:07:18.6697517Z remote: Compressing objects: 58% (219/377) 2024-05-19T18:07:18.6698649Z remote: Compressing objects: 59% (223/377) 2024-05-19T18:07:18.6699478Z remote: Compressing objects: 60% (227/377) 2024-05-19T18:07:18.6700612Z remote: Compressing objects: 61% (230/377) 2024-05-19T18:07:18.6701423Z remote: Compressing objects: 62% (234/377) 2024-05-19T18:07:18.6702255Z remote: Compressing objects: 63% (238/377) 2024-05-19T18:07:18.6703143Z remote: Compressing objects: 64% (242/377) 2024-05-19T18:07:18.6704028Z remote: Compressing objects: 65% (246/377) 2024-05-19T18:07:18.6704970Z remote: Compressing objects: 66% (249/377) 2024-05-19T18:07:18.6705737Z remote: Compressing objects: 67% (253/377) 2024-05-19T18:07:18.6706542Z remote: Compressing objects: 68% (257/377) 2024-05-19T18:07:18.6707224Z remote: Compressing objects: 69% (261/377) 2024-05-19T18:07:18.6707989Z remote: Compressing objects: 70% (264/377) 2024-05-19T18:07:18.6708916Z remote: Compressing objects: 71% (268/377) 2024-05-19T18:07:18.6709679Z remote: Compressing objects: 72% (272/377) 2024-05-19T18:07:18.6710704Z remote: Compressing objects: 73% (276/377) 2024-05-19T18:07:18.6711436Z remote: Compressing objects: 74% (279/377) 2024-05-19T18:07:18.6712326Z remote: Compressing objects: 75% (283/377) 2024-05-19T18:07:18.6713095Z remote: Compressing objects: 76% (287/377) 2024-05-19T18:07:18.6713961Z remote: Compressing objects: 77% (291/377) 2024-05-19T18:07:18.6714876Z remote: Compressing objects: 78% (295/377) 2024-05-19T18:07:18.6715785Z remote: Compressing objects: 79% (298/377) 2024-05-19T18:07:18.6716714Z remote: Compressing objects: 80% (302/377) 2024-05-19T18:07:18.6717619Z remote: Compressing objects: 81% (306/377) 2024-05-19T18:07:18.6718374Z remote: Compressing objects: 82% (310/377) 2024-05-19T18:07:18.6719162Z remote: Compressing objects: 83% (313/377) 2024-05-19T18:07:18.6720037Z remote: Compressing objects: 84% (317/377) 2024-05-19T18:07:18.6720684Z remote: Compressing objects: 85% (321/377) 2024-05-19T18:07:18.6721304Z remote: Compressing objects: 86% (325/377) 2024-05-19T18:07:18.6721928Z remote: Compressing objects: 87% (328/377) 2024-05-19T18:07:18.6722563Z remote: Compressing objects: 88% (332/377) 2024-05-19T18:07:18.6723186Z remote: Compressing objects: 89% (336/377) 2024-05-19T18:07:18.6723834Z remote: Compressing objects: 90% (340/377) 2024-05-19T18:07:18.6724563Z remote: Compressing objects: 91% (344/377) 2024-05-19T18:07:18.6725193Z remote: Compressing objects: 92% (347/377) 2024-05-19T18:07:18.6725821Z remote: Compressing objects: 93% (351/377) 2024-05-19T18:07:18.6726455Z remote: Compressing objects: 94% (355/377) 2024-05-19T18:07:18.6727220Z remote: Compressing objects: 95% (359/377) 2024-05-19T18:07:18.6727840Z remote: Compressing objects: 96% (362/377) 2024-05-19T18:07:18.6728469Z remote: Compressing objects: 97% (366/377) 2024-05-19T18:07:18.6729227Z remote: Compressing objects: 98% (370/377) 2024-05-19T18:07:18.6729860Z remote: Compressing objects: 99% (374/377) 2024-05-19T18:07:18.6730488Z remote: Compressing objects: 100% (377/377) 2024-05-19T18:07:18.6731162Z remote: Compressing objects: 100% (377/377), done. 2024-05-19T18:07:39.7308372Z remote: Total 1119785 (delta 762), reused 961 (delta 698), pack-reused 1118700 2024-05-19T18:08:03.6202308Z [command]/usr/bin/git rev-parse --verify --quiet c8c7ddfde57442cc28a394c385cab656ce9b1f5e^{object} 2024-05-19T18:08:03.6222486Z c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:08:03.6225919Z ##[endgroup] 2024-05-19T18:08:03.6227290Z ##[group]Determining the checkout info 2024-05-19T18:08:03.6228259Z ##[endgroup] 2024-05-19T18:08:03.6228942Z ##[group]Checking out the ref 2024-05-19T18:08:03.6229821Z [command]/usr/bin/git checkout --quiet --force c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:08:04.8396107Z ##[endgroup] 2024-05-19T18:08:04.8396724Z ##[group]Setting up auth for fetching submodules 2024-05-19T18:08:04.8401104Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-05-19T18:08:04.8447891Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-05-19T18:08:04.8469182Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-05-19T18:08:04.8488678Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-05-19T18:08:04.8504932Z ##[endgroup] 2024-05-19T18:08:04.8505742Z ##[group]Fetching submodules 2024-05-19T18:08:04.8509151Z [command]/usr/bin/git submodule sync --recursive 2024-05-19T18:08:04.8717696Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-05-19T18:08:04.8920129Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-05-19T18:08:04.8921858Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-05-19T18:08:04.8923524Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-05-19T18:08:04.8925184Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-05-19T18:08:04.8927072Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2024-05-19T18:08:04.8929006Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-05-19T18:08:04.8930902Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-05-19T18:08:04.8932348Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-05-19T18:08:04.8934060Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-05-19T18:08:04.8936083Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-05-19T18:08:04.8938729Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-05-19T18:08:04.8940737Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-05-19T18:08:04.8943204Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-05-19T18:08:04.8945680Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-05-19T18:08:04.8948295Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-05-19T18:08:04.8950676Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-05-19T18:08:04.8953751Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-05-19T18:08:04.8956669Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:08:04.8959276Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-05-19T18:08:04.8962151Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-05-19T18:08:04.8964933Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-05-19T18:08:04.8968307Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2024-05-19T18:08:04.8971242Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-05-19T18:08:04.8974372Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-05-19T18:08:04.8977590Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-05-19T18:08:04.8980945Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-05-19T18:08:04.8984058Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2024-05-19T18:08:04.8987413Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-05-19T18:08:04.8990877Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-05-19T18:08:04.8994848Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2024-05-19T18:08:04.8998501Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-05-19T18:08:04.9001940Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-05-19T18:08:04.9005788Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-05-19T18:08:04.9009585Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-05-19T18:08:04.9013434Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-05-19T18:08:04.9017157Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-05-19T18:08:04.9021176Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-05-19T18:08:04.9025127Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-05-19T18:08:04.9029237Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2024-05-19T18:08:04.9034063Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-05-19T18:08:04.9052474Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-05-19T18:08:05.1527680Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-05-19T18:08:05.3062474Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-05-19T18:08:05.4739024Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-05-19T18:08:05.6922164Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2024-05-19T18:08:05.9136665Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-05-19T18:08:07.8589046Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-05-19T18:08:17.0118363Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-05-19T18:08:17.3616167Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-05-19T18:08:17.8623619Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-05-19T18:08:18.4284329Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-05-19T18:08:19.6560420Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-05-19T18:08:21.4152136Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-05-19T18:08:27.2359068Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-05-19T18:08:28.5875664Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-05-19T18:08:30.2083429Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-05-19T18:08:31.2900842Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-05-19T18:08:31.4496243Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-05-19T18:08:31.8222218Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-05-19T18:08:32.1414324Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-05-19T18:08:33.0733245Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-05-19T18:08:33.4215436Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2024-05-19T18:08:33.5842382Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-05-19T18:08:33.8104428Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-05-19T18:08:35.2638796Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-05-19T18:08:35.9341121Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-05-19T18:08:36.5289555Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2024-05-19T18:08:36.7780989Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-05-19T18:08:42.3409510Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-05-19T18:08:44.7122605Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2024-05-19T18:08:45.1372083Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-05-19T18:08:48.9248906Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-05-19T18:08:49.1178249Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-05-19T18:10:09.7685176Z ##[error]error: RPC failed; curl 18 Transferred a partial file 2024-05-19T18:10:09.7686213Z ##[error]error: 4950 bytes of body are still expected 2024-05-19T18:10:09.7687141Z fetch-pack: unexpected disconnect while reading sideband packet 2024-05-19T18:10:09.7687844Z ##[error]fatal: early EOF 2024-05-19T18:10:09.7688498Z ##[error]fatal: fetch-pack: invalid index-pack output 2024-05-19T18:10:09.7690652Z ##[error]fatal: clone of 'https://github.com/protocolbuffers/protobuf.git' into submodule path '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf' failed 2024-05-19T18:10:09.7692126Z Failed to clone 'third_party/protobuf'. Retry scheduled 2024-05-19T18:10:09.7703157Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-05-19T18:10:09.9050515Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-05-19T18:10:10.0738466Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-05-19T18:10:10.9405493Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-05-19T18:10:11.1846649Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-05-19T18:10:11.7684769Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2024-05-19T18:10:13.7789109Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-05-19T18:10:14.1460722Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-05-19T18:10:21.6184257Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-05-19T18:10:21.6266982Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-05-19T18:10:21.6326143Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-05-19T18:10:21.6504286Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-05-19T18:10:21.6686244Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2024-05-19T18:10:21.6980328Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-05-19T18:10:22.4615507Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-05-19T18:10:22.4784111Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-05-19T18:10:22.5135363Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-05-19T18:10:22.5949761Z Submodule path 'third_party/cpuinfo': checked out 'd6860c477c99f1fce9e28eb206891af3c0e1a1d7' 2024-05-19T18:10:22.6182477Z Submodule path 'third_party/cudnn_frontend': checked out '150798fe976556078f443fdb059a1ff0361f58a2' 2024-05-19T18:10:23.0464602Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-05-19T18:10:23.2569002Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-05-19T18:10:23.3156884Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-05-19T18:10:23.3170618Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:10:23.3172541Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:10:23.3174941Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:10:23.3177197Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:10:23.3179689Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:10:23.3197223Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-05-19T18:10:24.2065695Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-05-19T18:10:24.7435554Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-05-19T18:10:26.3911898Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-05-19T18:10:27.3068236Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-05-19T18:10:27.6142131Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-05-19T18:10:27.6974018Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-05-19T18:10:28.0371279Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-05-19T18:10:28.0901677Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-05-19T18:10:28.0985438Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-05-19T18:10:28.1880878Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-05-19T18:10:28.2182876Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-05-19T18:10:28.2247467Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-05-19T18:10:28.2547441Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-05-19T18:10:28.2741702Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-05-19T18:10:28.3099449Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-05-19T18:10:28.3195299Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-05-19T18:10:28.3205367Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-05-19T18:10:28.3220828Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-05-19T18:10:38.9517367Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-05-19T18:10:38.9592226Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2024-05-19T18:10:38.9721064Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-05-19T18:10:39.0491712Z Submodule path 'third_party/kineto': checked out '3a81076cc97092666f319846f32f36b73ce2293e' 2024-05-19T18:10:39.0502993Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:10:39.0505461Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:10:39.0507197Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:10:39.0521882Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-05-19T18:10:39.5155329Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-05-19T18:10:40.6434564Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-05-19T18:10:41.6189213Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-05-19T18:10:41.6199738Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:10:41.6201810Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:10:41.6204039Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:10:41.6205902Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:10:41.6207822Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:10:41.6236210Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:10:41.6238496Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:10:41.6240466Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:10:41.6242140Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-05-19T18:10:42.5954894Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-05-19T18:10:42.9261288Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-05-19T18:10:44.0170636Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-05-19T18:10:44.2612045Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-05-19T18:10:44.5808052Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-05-19T18:10:45.5450970Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-05-19T18:10:51.2158261Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-05-19T18:10:51.5494431Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-05-19T18:10:51.5633722Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-05-19T18:10:51.5918263Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-05-19T18:10:51.6013659Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-05-19T18:10:51.6023535Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:10:51.6039715Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-05-19T18:10:51.8810929Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-05-19T18:10:51.8947190Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-05-19T18:10:51.9281407Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-05-19T18:10:52.0082724Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-05-19T18:10:52.0204037Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-05-19T18:10:52.0487062Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-05-19T18:10:52.0973269Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-05-19T18:10:52.1265013Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-05-19T18:10:52.1457186Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-05-19T18:10:52.1558781Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2024-05-19T18:10:52.2404303Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-05-19T18:10:52.5148429Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-05-19T18:10:52.5177140Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-05-19T18:10:52.5179095Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-05-19T18:10:52.5194470Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-05-19T18:10:52.8886582Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-05-19T18:11:43.4504974Z ##[error]error: RPC failed; curl 18 Transferred a partial file 2024-05-19T18:11:43.4506056Z ##[error]error: 56 bytes of body are still expected 2024-05-19T18:11:43.4511892Z fetch-pack: unexpected disconnect while reading sideband packet 2024-05-19T18:11:43.4512671Z ##[error]fatal: early EOF 2024-05-19T18:11:43.4513576Z ##[error]fatal: fetch-pack: invalid index-pack output 2024-05-19T18:11:43.4523741Z ##[error]fatal: clone of 'https://github.com/pybind/pybind11.git' into submodule path '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11' failed 2024-05-19T18:11:43.4525076Z Failed to clone 'third_party/pybind11'. Retry scheduled 2024-05-19T18:11:43.4543636Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-05-19T18:11:44.3518588Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-05-19T18:11:44.3787628Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-05-19T18:11:44.4032823Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-05-19T18:11:44.4169040Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2024-05-19T18:11:44.4181251Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:11:44.4197614Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2024-05-19T18:11:46.9833712Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2024-05-19T18:11:46.9847305Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:11:46.9849547Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:11:46.9866882Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2024-05-19T18:11:47.3376909Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2024-05-19T18:11:48.2487710Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2024-05-19T18:11:48.3138508Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2024-05-19T18:11:48.3148587Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:11:48.3166831Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2024-05-19T18:11:48.4806993Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-05-19T18:11:48.5329735Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-05-19T18:11:48.5342374Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:11:48.5344494Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:11:48.5346574Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:11:48.5348513Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:11:48.5350919Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:11:48.5353380Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:11:48.5355605Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:11:48.5357579Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:11:48.5374228Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-05-19T18:11:48.9288252Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-05-19T18:12:27.9987083Z ##[error]error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8) 2024-05-19T18:12:27.9988805Z ##[error]error: 32 bytes of body are still expected 2024-05-19T18:12:28.0003327Z fetch-pack: unexpected disconnect while reading sideband packet 2024-05-19T18:12:28.0005398Z ##[error]fatal: early EOF 2024-05-19T18:12:28.0006797Z ##[error]fatal: fetch-pack: invalid index-pack output 2024-05-19T18:12:28.0017288Z ##[error]fatal: clone of 'https://github.com/google/googletest' into submodule path '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest' failed 2024-05-19T18:12:28.0019270Z Failed to clone 'third_party/googletest'. Retry scheduled 2024-05-19T18:12:28.0033660Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-05-19T18:12:28.3024995Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-05-19T18:12:33.8086800Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-05-19T18:12:34.0645387Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-05-19T18:12:34.2370161Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-05-19T18:12:34.4978643Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-05-19T18:12:40.9478170Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-05-19T18:12:41.8878923Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-05-19T18:12:41.9208153Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-05-19T18:12:41.9332673Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-05-19T18:12:42.0162240Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-05-19T18:12:42.0258335Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-05-19T18:12:42.0363621Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-05-19T18:12:42.0479677Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-05-19T18:12:42.0491381Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:12:42.0493313Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:12:42.0508146Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-05-19T18:12:43.7815201Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-05-19T18:12:44.9294526Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-05-19T18:12:44.9656919Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-05-19T18:12:45.3228799Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-05-19T18:12:45.3555370Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-05-19T18:12:45.3620686Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-05-19T18:12:45.5777648Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-05-19T18:12:45.5791611Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:12:45.5793228Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-05-19T18:12:45.5808117Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-05-19T18:12:45.9697404Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-05-19T18:12:46.8758510Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-05-19T18:12:46.9373908Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-05-19T18:12:46.9433555Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-05-19T18:12:46.9517983Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-05-19T18:12:46.9788523Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-05-19T18:12:47.0005326Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-05-19T18:12:47.0329080Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-05-19T18:12:47.1240607Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2024-05-19T18:12:47.1440838Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-05-19T18:12:47.1452553Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:12:47.1454114Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:12:47.1455736Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:12:47.1457541Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:12:47.1472267Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-05-19T18:12:48.0966702Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-05-19T18:12:48.3003457Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-05-19T18:12:49.4916577Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-05-19T18:12:50.4340135Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-05-19T18:12:50.4455098Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-05-19T18:12:50.5029439Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-05-19T18:12:50.5255002Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-05-19T18:12:50.5264938Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:12:50.5281808Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-05-19T18:12:50.7386097Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-05-19T18:12:50.9817860Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-05-19T18:12:50.9911354Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-05-19T18:12:51.0273633Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-05-19T18:12:51.0297491Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-05-19T18:12:51.0506587Z Entering 'android/libs/fbjni' 2024-05-19T18:12:51.0535839Z Entering 'third_party/FP16' 2024-05-19T18:12:51.0561540Z Entering 'third_party/FXdiv' 2024-05-19T18:12:51.0586081Z Entering 'third_party/NNPACK' 2024-05-19T18:12:51.0614886Z Entering 'third_party/QNNPACK' 2024-05-19T18:12:51.0641784Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T18:12:51.0669175Z Entering 'third_party/XNNPACK' 2024-05-19T18:12:51.0707359Z Entering 'third_party/benchmark' 2024-05-19T18:12:51.0737322Z Entering 'third_party/cpp-httplib' 2024-05-19T18:12:51.0764127Z Entering 'third_party/cpuinfo' 2024-05-19T18:12:51.0792099Z Entering 'third_party/cudnn_frontend' 2024-05-19T18:12:51.0819370Z Entering 'third_party/cutlass' 2024-05-19T18:12:51.0852100Z Entering 'third_party/eigen' 2024-05-19T18:12:51.0883898Z Entering 'third_party/fbgemm' 2024-05-19T18:12:51.0910263Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:12:51.0938097Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:12:51.0964789Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:12:51.0993901Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:12:51.1017929Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:12:51.1047204Z Entering 'third_party/flatbuffers' 2024-05-19T18:12:51.1075283Z Entering 'third_party/fmt' 2024-05-19T18:12:51.1100268Z Entering 'third_party/foxi' 2024-05-19T18:12:51.1128955Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:12:51.1157600Z Entering 'third_party/gloo' 2024-05-19T18:12:51.1182719Z Entering 'third_party/googletest' 2024-05-19T18:12:51.1207617Z Entering 'third_party/ideep' 2024-05-19T18:12:51.1232313Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T18:12:51.1261722Z Entering 'third_party/ios-cmake' 2024-05-19T18:12:51.1287135Z Entering 'third_party/ittapi' 2024-05-19T18:12:51.1312328Z Entering 'third_party/kineto' 2024-05-19T18:12:51.1336771Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:12:51.1362117Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:12:51.1388750Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:12:51.1416878Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:12:51.1442441Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:12:51.1466021Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:12:51.1491850Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:12:51.1519303Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:12:51.1547361Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:12:51.1575247Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:12:51.1601587Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:12:51.1627027Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:12:51.1652897Z Entering 'third_party/mimalloc' 2024-05-19T18:12:51.1680200Z Entering 'third_party/nccl/nccl' 2024-05-19T18:12:51.1706141Z Entering 'third_party/neon2sse' 2024-05-19T18:12:51.1730540Z Entering 'third_party/nlohmann' 2024-05-19T18:12:51.1756333Z Entering 'third_party/onnx' 2024-05-19T18:12:51.1793438Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T18:12:51.1817859Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T18:12:51.1848406Z Entering 'third_party/onnx-tensorrt' 2024-05-19T18:12:51.1872906Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:12:51.1900177Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:12:51.1928960Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:12:51.1953805Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:12:51.1984787Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T18:12:51.2011690Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:12:51.2037085Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:12:51.2062210Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:12:51.2088993Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:12:51.2117809Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:12:51.2144043Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:12:51.2170267Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:12:51.2194519Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:12:51.2222864Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:12:51.2249185Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:12:51.2288476Z Entering 'third_party/pocketfft' 2024-05-19T18:12:51.2317205Z Entering 'third_party/protobuf' 2024-05-19T18:12:51.2346394Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:12:51.2374026Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T18:12:51.2401561Z Entering 'third_party/psimd' 2024-05-19T18:12:51.2427313Z Entering 'third_party/pthreadpool' 2024-05-19T18:12:51.2455439Z Entering 'third_party/pybind11' 2024-05-19T18:12:51.2479939Z Entering 'third_party/python-peachpy' 2024-05-19T18:12:51.2507611Z Entering 'third_party/sleef' 2024-05-19T18:12:51.2534502Z Entering 'third_party/tbb' 2024-05-19T18:12:51.2563621Z Entering 'third_party/tensorpipe' 2024-05-19T18:12:51.2592157Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:12:51.2617577Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:12:51.2644707Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:12:51.2671016Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:12:51.2697196Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:12:51.2738207Z ##[endgroup] 2024-05-19T18:12:51.2740667Z ##[group]Persisting credentials for submodules 2024-05-19T18:12:51.2743199Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-05-19T18:12:51.2953246Z Entering 'android/libs/fbjni' 2024-05-19T18:12:51.2986502Z Entering 'third_party/FP16' 2024-05-19T18:12:51.3021354Z Entering 'third_party/FXdiv' 2024-05-19T18:12:51.3054895Z Entering 'third_party/NNPACK' 2024-05-19T18:12:51.3090285Z Entering 'third_party/QNNPACK' 2024-05-19T18:12:51.3126568Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T18:12:51.3161648Z Entering 'third_party/XNNPACK' 2024-05-19T18:12:51.3206081Z Entering 'third_party/benchmark' 2024-05-19T18:12:51.3240174Z Entering 'third_party/cpp-httplib' 2024-05-19T18:12:51.3274372Z Entering 'third_party/cpuinfo' 2024-05-19T18:12:51.3308215Z Entering 'third_party/cudnn_frontend' 2024-05-19T18:12:51.3341078Z Entering 'third_party/cutlass' 2024-05-19T18:12:51.3383470Z Entering 'third_party/eigen' 2024-05-19T18:12:51.3420699Z Entering 'third_party/fbgemm' 2024-05-19T18:12:51.3456064Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:12:51.3488766Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:12:51.3520706Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:12:51.3558642Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:12:51.3592435Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:12:51.3625189Z Entering 'third_party/flatbuffers' 2024-05-19T18:12:51.3663516Z Entering 'third_party/fmt' 2024-05-19T18:12:51.3698875Z Entering 'third_party/foxi' 2024-05-19T18:12:51.3734197Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:12:51.3767456Z Entering 'third_party/gloo' 2024-05-19T18:12:51.3801560Z Entering 'third_party/googletest' 2024-05-19T18:12:51.3837542Z Entering 'third_party/ideep' 2024-05-19T18:12:51.3872145Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T18:12:51.3908898Z Entering 'third_party/ios-cmake' 2024-05-19T18:12:51.3943241Z Entering 'third_party/ittapi' 2024-05-19T18:12:51.3977339Z Entering 'third_party/kineto' 2024-05-19T18:12:51.4011476Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:12:51.4046184Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:12:51.4081755Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:12:51.4114466Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:12:51.4147170Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:12:51.4182350Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:12:51.4218716Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:12:51.4253001Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:12:51.4286017Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:12:51.4321860Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:12:51.4357215Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:12:51.4389575Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:12:51.4424402Z Entering 'third_party/mimalloc' 2024-05-19T18:12:51.4461342Z Entering 'third_party/nccl/nccl' 2024-05-19T18:12:51.4498415Z Entering 'third_party/neon2sse' 2024-05-19T18:12:51.4531194Z Entering 'third_party/nlohmann' 2024-05-19T18:12:51.4565725Z Entering 'third_party/onnx' 2024-05-19T18:12:51.4611261Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T18:12:51.4645379Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T18:12:51.4678978Z Entering 'third_party/onnx-tensorrt' 2024-05-19T18:12:51.4712996Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:12:51.4750958Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:12:51.4784298Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:12:51.4818996Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:12:51.4857377Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T18:12:51.4893522Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:12:51.4929506Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:12:51.4961069Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:12:51.4993328Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:12:51.5026621Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:12:51.5057971Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:12:51.5090155Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:12:51.5123118Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:12:51.5155990Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:12:51.5189737Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:12:51.5240402Z Entering 'third_party/pocketfft' 2024-05-19T18:12:51.5273639Z Entering 'third_party/protobuf' 2024-05-19T18:12:51.5310716Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:12:51.5342850Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T18:12:51.5378212Z Entering 'third_party/psimd' 2024-05-19T18:12:51.5413974Z Entering 'third_party/pthreadpool' 2024-05-19T18:12:51.5446678Z Entering 'third_party/pybind11' 2024-05-19T18:12:51.5482600Z Entering 'third_party/python-peachpy' 2024-05-19T18:12:51.5517440Z Entering 'third_party/sleef' 2024-05-19T18:12:51.5550831Z Entering 'third_party/tbb' 2024-05-19T18:12:51.5585996Z Entering 'third_party/tensorpipe' 2024-05-19T18:12:51.5617322Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:12:51.5651784Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:12:51.5684514Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:12:51.5718089Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:12:51.5749405Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:12:51.5796765Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-05-19T18:12:51.6002255Z Entering 'android/libs/fbjni' 2024-05-19T18:12:51.6034326Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-05-19T18:12:51.6047885Z Entering 'third_party/FP16' 2024-05-19T18:12:51.6078102Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-05-19T18:12:51.6091635Z Entering 'third_party/FXdiv' 2024-05-19T18:12:51.6123965Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-05-19T18:12:51.6136656Z Entering 'third_party/NNPACK' 2024-05-19T18:12:51.6167314Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-05-19T18:12:51.6181203Z Entering 'third_party/QNNPACK' 2024-05-19T18:12:51.6212209Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2024-05-19T18:12:51.6225910Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T18:12:51.6257287Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-05-19T18:12:51.6269238Z Entering 'third_party/XNNPACK' 2024-05-19T18:12:51.6299818Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-05-19T18:12:51.6322677Z Entering 'third_party/benchmark' 2024-05-19T18:12:51.6353464Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-05-19T18:12:51.6367134Z Entering 'third_party/cpp-httplib' 2024-05-19T18:12:51.6397420Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-05-19T18:12:51.6411452Z Entering 'third_party/cpuinfo' 2024-05-19T18:12:51.6441574Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-05-19T18:12:51.6456265Z Entering 'third_party/cudnn_frontend' 2024-05-19T18:12:51.6488551Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-05-19T18:12:51.6501483Z Entering 'third_party/cutlass' 2024-05-19T18:12:51.6531802Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-05-19T18:12:51.6550459Z Entering 'third_party/eigen' 2024-05-19T18:12:51.6581039Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-05-19T18:12:51.6595099Z Entering 'third_party/fbgemm' 2024-05-19T18:12:51.6626795Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-05-19T18:12:51.6638622Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:12:51.6669380Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-05-19T18:12:51.6682387Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:12:51.6711554Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-05-19T18:12:51.6724962Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:12:51.6756644Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-05-19T18:12:51.6773623Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:12:51.6806932Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:12:51.6819553Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:12:51.6850398Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-05-19T18:12:51.6864363Z Entering 'third_party/flatbuffers' 2024-05-19T18:12:51.6897008Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-05-19T18:12:51.6910520Z Entering 'third_party/fmt' 2024-05-19T18:12:51.6941619Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-05-19T18:12:51.6955371Z Entering 'third_party/foxi' 2024-05-19T18:12:51.6986966Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-05-19T18:12:51.7000324Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:12:51.7032391Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-05-19T18:12:51.7045019Z Entering 'third_party/gloo' 2024-05-19T18:12:51.7075687Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-05-19T18:12:51.7089303Z Entering 'third_party/googletest' 2024-05-19T18:12:51.7120473Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:12:51.7134317Z Entering 'third_party/ideep' 2024-05-19T18:12:51.7165613Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-05-19T18:12:51.7177371Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T18:12:51.7207846Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-05-19T18:12:51.7226417Z Entering 'third_party/ios-cmake' 2024-05-19T18:12:51.7257557Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2024-05-19T18:12:51.7272596Z Entering 'third_party/ittapi' 2024-05-19T18:12:51.7303874Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-05-19T18:12:51.7316478Z Entering 'third_party/kineto' 2024-05-19T18:12:51.7347200Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-05-19T18:12:51.7360295Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:12:51.7393084Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-05-19T18:12:51.7405969Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:12:51.7438454Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-05-19T18:12:51.7452757Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:12:51.7484338Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-05-19T18:12:51.7496875Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:12:51.7528294Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-05-19T18:12:51.7540756Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:12:51.7572380Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-05-19T18:12:51.7584447Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:12:51.7615984Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-05-19T18:12:51.7630525Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:12:51.7662287Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-05-19T18:12:51.7674942Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:12:51.7711202Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:12:51.7723568Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:12:51.7754386Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-05-19T18:12:51.7767052Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:12:51.7799504Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-05-19T18:12:51.7813127Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:12:51.7844841Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-05-19T18:12:51.7857836Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:12:51.7888870Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-05-19T18:12:51.7902834Z Entering 'third_party/mimalloc' 2024-05-19T18:12:51.7934238Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-05-19T18:12:51.7947224Z Entering 'third_party/nccl/nccl' 2024-05-19T18:12:51.7978686Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-05-19T18:12:51.7992718Z Entering 'third_party/neon2sse' 2024-05-19T18:12:51.8024108Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2024-05-19T18:12:51.8036483Z Entering 'third_party/nlohmann' 2024-05-19T18:12:51.8066082Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-05-19T18:12:51.8078784Z Entering 'third_party/onnx' 2024-05-19T18:12:51.8109422Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-05-19T18:12:51.8134263Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T18:12:51.8165757Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-05-19T18:12:51.8177809Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T18:12:51.8209445Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-05-19T18:12:51.8222864Z Entering 'third_party/onnx-tensorrt' 2024-05-19T18:12:51.8256171Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2024-05-19T18:12:51.8268663Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:12:51.8300251Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2024-05-19T18:12:51.8316243Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:12:51.8347522Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-05-19T18:12:51.8358992Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:12:51.8389725Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-05-19T18:12:51.8401759Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:12:51.8433624Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-05-19T18:12:51.8450351Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T18:12:51.8483375Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-05-19T18:12:51.8496873Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:12:51.8527535Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-05-19T18:12:51.8540663Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:12:51.8572744Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:12:51.8585230Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:12:51.8622298Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-05-19T18:12:51.8633481Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:12:51.8663880Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-05-19T18:12:51.8676980Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:12:51.8707515Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-05-19T18:12:51.8719749Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:12:51.8752049Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-05-19T18:12:51.8764273Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:12:51.8794002Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-05-19T18:12:51.8805152Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:12:51.8836845Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-05-19T18:12:51.8849525Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:12:51.8881411Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-05-19T18:12:51.8895406Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:12:51.8925526Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-05-19T18:12:51.8954307Z Entering 'third_party/pocketfft' 2024-05-19T18:12:51.8986368Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-05-19T18:12:51.8998188Z Entering 'third_party/protobuf' 2024-05-19T18:12:51.9029662Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-05-19T18:12:51.9044958Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:12:51.9074515Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-05-19T18:12:51.9087203Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T18:12:51.9117918Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:12:51.9131967Z Entering 'third_party/psimd' 2024-05-19T18:12:51.9163386Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-05-19T18:12:51.9174562Z Entering 'third_party/pthreadpool' 2024-05-19T18:12:51.9206726Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-05-19T18:12:51.9218316Z Entering 'third_party/pybind11' 2024-05-19T18:12:51.9250746Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-05-19T18:12:51.9263942Z Entering 'third_party/python-peachpy' 2024-05-19T18:12:51.9295402Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-05-19T18:12:51.9309515Z Entering 'third_party/sleef' 2024-05-19T18:12:51.9340555Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-05-19T18:12:51.9353595Z Entering 'third_party/tbb' 2024-05-19T18:12:51.9383790Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2024-05-19T18:12:51.9397679Z Entering 'third_party/tensorpipe' 2024-05-19T18:12:51.9427476Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-05-19T18:12:51.9438795Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:12:51.9468969Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:12:51.9480841Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:12:51.9511226Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-05-19T18:12:51.9522631Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:12:51.9552681Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-05-19T18:12:51.9565301Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:12:51.9596194Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-05-19T18:12:51.9608142Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:12:51.9639890Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-05-19T18:12:52.0647192Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-05-19T18:12:52.0852529Z Entering 'android/libs/fbjni' 2024-05-19T18:12:52.0880878Z Entering 'third_party/FP16' 2024-05-19T18:12:52.0909503Z Entering 'third_party/FXdiv' 2024-05-19T18:12:52.0934787Z Entering 'third_party/NNPACK' 2024-05-19T18:12:52.0964393Z Entering 'third_party/QNNPACK' 2024-05-19T18:12:52.0991038Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T18:12:52.1016729Z Entering 'third_party/XNNPACK' 2024-05-19T18:12:52.1056294Z Entering 'third_party/benchmark' 2024-05-19T18:12:52.1084817Z Entering 'third_party/cpp-httplib' 2024-05-19T18:12:52.1112828Z Entering 'third_party/cpuinfo' 2024-05-19T18:12:52.1141504Z Entering 'third_party/cudnn_frontend' 2024-05-19T18:12:52.1170537Z Entering 'third_party/cutlass' 2024-05-19T18:12:52.1202959Z Entering 'third_party/eigen' 2024-05-19T18:12:52.1231672Z Entering 'third_party/fbgemm' 2024-05-19T18:12:52.1257948Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:12:52.1284083Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:12:52.1309115Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:12:52.1338368Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:12:52.1365556Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:12:52.1392811Z Entering 'third_party/flatbuffers' 2024-05-19T18:12:52.1420997Z Entering 'third_party/fmt' 2024-05-19T18:12:52.1449748Z Entering 'third_party/foxi' 2024-05-19T18:12:52.1476302Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:12:52.1504781Z Entering 'third_party/gloo' 2024-05-19T18:12:52.1533686Z Entering 'third_party/googletest' 2024-05-19T18:12:52.1562267Z Entering 'third_party/ideep' 2024-05-19T18:12:52.1588580Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T18:12:52.1621842Z Entering 'third_party/ios-cmake' 2024-05-19T18:12:52.1650741Z Entering 'third_party/ittapi' 2024-05-19T18:12:52.1679920Z Entering 'third_party/kineto' 2024-05-19T18:12:52.1708161Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:12:52.1736778Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:12:52.1765041Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:12:52.1793313Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:12:52.1821267Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:12:52.1848036Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:12:52.1876306Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:12:52.1903460Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:12:52.1933396Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:12:52.1961589Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:12:52.1988884Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:12:52.2017632Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:12:52.2045526Z Entering 'third_party/mimalloc' 2024-05-19T18:12:52.2075029Z Entering 'third_party/nccl/nccl' 2024-05-19T18:12:52.2102815Z Entering 'third_party/neon2sse' 2024-05-19T18:12:52.2129616Z Entering 'third_party/nlohmann' 2024-05-19T18:12:52.2158651Z Entering 'third_party/onnx' 2024-05-19T18:12:52.2198012Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T18:12:52.2227095Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T18:12:52.2256736Z Entering 'third_party/onnx-tensorrt' 2024-05-19T18:12:52.2283586Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:12:52.2315046Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:12:52.2342899Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:12:52.2369093Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:12:52.2401565Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T18:12:52.2430851Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:12:52.2459364Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:12:52.2485169Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:12:52.2513502Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:12:52.2539902Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:12:52.2565830Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:12:52.2593702Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:12:52.2621877Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:12:52.2651298Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:12:52.2679187Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:12:52.2720993Z Entering 'third_party/pocketfft' 2024-05-19T18:12:52.2749147Z Entering 'third_party/protobuf' 2024-05-19T18:12:52.2777478Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:12:52.2803840Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T18:12:52.2833329Z Entering 'third_party/psimd' 2024-05-19T18:12:52.2861682Z Entering 'third_party/pthreadpool' 2024-05-19T18:12:52.2888599Z Entering 'third_party/pybind11' 2024-05-19T18:12:52.2916560Z Entering 'third_party/python-peachpy' 2024-05-19T18:12:52.2944171Z Entering 'third_party/sleef' 2024-05-19T18:12:52.2973118Z Entering 'third_party/tbb' 2024-05-19T18:12:52.3001650Z Entering 'third_party/tensorpipe' 2024-05-19T18:12:52.3027371Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:12:52.3054191Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:12:52.3079991Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:12:52.3107654Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:12:52.3134897Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:12:52.3175804Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-05-19T18:12:52.3382207Z Entering 'android/libs/fbjni' 2024-05-19T18:12:52.3409155Z Entering 'third_party/FP16' 2024-05-19T18:12:52.3439147Z Entering 'third_party/FXdiv' 2024-05-19T18:12:52.3464556Z Entering 'third_party/NNPACK' 2024-05-19T18:12:52.3494262Z Entering 'third_party/QNNPACK' 2024-05-19T18:12:52.3520009Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T18:12:52.3545584Z Entering 'third_party/XNNPACK' 2024-05-19T18:12:52.3584386Z Entering 'third_party/benchmark' 2024-05-19T18:12:52.3613207Z Entering 'third_party/cpp-httplib' 2024-05-19T18:12:52.3638951Z Entering 'third_party/cpuinfo' 2024-05-19T18:12:52.3667833Z Entering 'third_party/cudnn_frontend' 2024-05-19T18:12:52.3693916Z Entering 'third_party/cutlass' 2024-05-19T18:12:52.3724409Z Entering 'third_party/eigen' 2024-05-19T18:12:52.3754354Z Entering 'third_party/fbgemm' 2024-05-19T18:12:52.3781830Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:12:52.3810365Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:12:52.3837269Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:12:52.3867733Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:12:52.3897017Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:12:52.3924191Z Entering 'third_party/flatbuffers' 2024-05-19T18:12:52.3953816Z Entering 'third_party/fmt' 2024-05-19T18:12:52.3981809Z Entering 'third_party/foxi' 2024-05-19T18:12:52.4007534Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:12:52.4033504Z Entering 'third_party/gloo' 2024-05-19T18:12:52.4061568Z Entering 'third_party/googletest' 2024-05-19T18:12:52.4090272Z Entering 'third_party/ideep' 2024-05-19T18:12:52.4116613Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T18:12:52.4149633Z Entering 'third_party/ios-cmake' 2024-05-19T18:12:52.4179014Z Entering 'third_party/ittapi' 2024-05-19T18:12:52.4242384Z Entering 'third_party/kineto' 2024-05-19T18:12:52.4243215Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:12:52.4265089Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:12:52.4294827Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:12:52.4320867Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:12:52.4347400Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:12:52.4375001Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:12:52.4403195Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:12:52.4430589Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:12:52.4458375Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:12:52.4485945Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:12:52.4512506Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:12:52.4539589Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:12:52.4566532Z Entering 'third_party/mimalloc' 2024-05-19T18:12:52.4595506Z Entering 'third_party/nccl/nccl' 2024-05-19T18:12:52.4622475Z Entering 'third_party/neon2sse' 2024-05-19T18:12:52.4651349Z Entering 'third_party/nlohmann' 2024-05-19T18:12:52.4679757Z Entering 'third_party/onnx' 2024-05-19T18:12:52.4718091Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T18:12:52.4746344Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T18:12:52.4777991Z Entering 'third_party/onnx-tensorrt' 2024-05-19T18:12:52.4804354Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:12:52.4835750Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:12:52.4864328Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:12:52.4892519Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:12:52.4922275Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T18:12:52.4952051Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:12:52.4979665Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:12:52.5004061Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:12:52.5028841Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:12:52.5057766Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:12:52.5082765Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:12:52.5107529Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:12:52.5133767Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:12:52.5161405Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:12:52.5190243Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:12:52.5232858Z Entering 'third_party/pocketfft' 2024-05-19T18:12:52.5260384Z Entering 'third_party/protobuf' 2024-05-19T18:12:52.5289684Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:12:52.5317042Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T18:12:52.5344193Z Entering 'third_party/psimd' 2024-05-19T18:12:52.5373775Z Entering 'third_party/pthreadpool' 2024-05-19T18:12:52.5399005Z Entering 'third_party/pybind11' 2024-05-19T18:12:52.5424316Z Entering 'third_party/python-peachpy' 2024-05-19T18:12:52.5450225Z Entering 'third_party/sleef' 2024-05-19T18:12:52.5479308Z Entering 'third_party/tbb' 2024-05-19T18:12:52.5511011Z Entering 'third_party/tensorpipe' 2024-05-19T18:12:52.5539076Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:12:52.5565391Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:12:52.5592877Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:12:52.5619042Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:12:52.5644113Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:12:52.5682973Z ##[endgroup] 2024-05-19T18:12:52.5710721Z [command]/usr/bin/git log -1 --format='%H' 2024-05-19T18:12:52.5728277Z 'c8c7ddfde57442cc28a394c385cab656ce9b1f5e' 2024-05-19T18:12:52.5893400Z Prepare all required actions 2024-05-19T18:12:52.5893848Z Getting action download info 2024-05-19T18:12:52.7729225Z ##[group]Run ./.github/actions/setup-linux 2024-05-19T18:12:52.7729942Z env: 2024-05-19T18:12:52.7730231Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:12:52.7730578Z ##[endgroup] 2024-05-19T18:12:52.7790911Z ##[group]Run set -euo pipefail 2024-05-19T18:12:52.7791342Z set -euo pipefail 2024-05-19T18:12:52.7791742Z function get_ec2_metadata() { 2024-05-19T18:12:52.7792284Z  # Pulled from instance metadata endpoint for EC2 2024-05-19T18:12:52.7793182Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-05-19T18:12:52.7793946Z  category=$1 2024-05-19T18:12:52.7794477Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-05-19T18:12:52.7795120Z  runner_name_str=i-0e8f077216c874026 2024-05-19T18:12:52.7795635Z  if [[ -f /.inarc ]]; then 2024-05-19T18:12:52.7796143Z  echo "ARC Runner, no info on ec2 metadata" 2024-05-19T18:12:52.7796700Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-05-19T18:12:52.7797392Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-05-19T18:12:52.7797991Z  else 2024-05-19T18:12:52.7798485Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-05-19T18:12:52.7799055Z  fi 2024-05-19T18:12:52.7799341Z } 2024-05-19T18:12:52.7799703Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-05-19T18:12:52.7800284Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-05-19T18:12:52.7800928Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-05-19T18:12:52.7801495Z echo "system info $(uname -a)" 2024-05-19T18:12:52.7809250Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:12:52.7809824Z env: 2024-05-19T18:12:52.7810100Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:12:52.7810453Z ##[endgroup] 2024-05-19T18:12:52.7878778Z ami-id: ami-0ce0c36d7a00b20e2 2024-05-19T18:12:52.7918893Z instance-id: i-0e8f077216c874026 2024-05-19T18:12:52.7958272Z instance-type: g5.4xlarge 2024-05-19T18:12:52.7965212Z system info Linux ip-10-0-34-75.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-05-19T18:12:52.7985700Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-05-19T18:12:52.7986631Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-05-19T18:12:52.7995102Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:12:52.7995609Z env: 2024-05-19T18:12:52.7995884Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:12:52.7996232Z ##[endgroup] 2024-05-19T18:12:52.8059191Z ##[group]Run if systemctl is-active --quiet docker; then 2024-05-19T18:12:52.8059950Z if systemctl is-active --quiet docker; then 2024-05-19T18:12:52.8060529Z  echo "Docker daemon is running..."; 2024-05-19T18:12:52.8060988Z else 2024-05-19T18:12:52.8061475Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-05-19T18:12:52.8062065Z fi 2024-05-19T18:12:52.8068874Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:12:52.8069366Z env: 2024-05-19T18:12:52.8069655Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:12:52.8070289Z ##[endgroup] 2024-05-19T18:12:52.8105506Z Docker daemon is running... 2024-05-19T18:12:52.8150451Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-05-19T18:12:52.8151020Z with: 2024-05-19T18:12:52.8151299Z shell: bash 2024-05-19T18:12:52.8151592Z timeout_minutes: 5 2024-05-19T18:12:52.8151920Z max_attempts: 3 2024-05-19T18:12:52.8152240Z retry_wait_seconds: 30 2024-05-19T18:12:52.8153754Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-05-19T18:12:52.8155176Z polling_interval_seconds: 1 2024-05-19T18:12:52.8155548Z warning_on_retry: true 2024-05-19T18:12:52.8155901Z continue_on_error: false 2024-05-19T18:12:52.8156241Z env: 2024-05-19T18:12:52.8156508Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:12:52.8156862Z AWS_RETRY_MODE: standard 2024-05-19T18:12:52.8157216Z AWS_MAX_ATTEMPTS: 5 2024-05-19T18:12:52.8157557Z AWS_DEFAULT_REGION: us-east-1 2024-05-19T18:12:52.8157920Z ##[endgroup] 2024-05-19T18:12:53.6745788Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-05-19T18:12:53.6746621Z Configure a credential helper to remove this warning. See 2024-05-19T18:12:53.6747634Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-05-19T18:12:53.6748187Z 2024-05-19T18:12:53.6748329Z Login Succeeded 2024-05-19T18:12:53.8644693Z Command completed after 1 attempt(s). 2024-05-19T18:12:53.8685990Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-05-19T18:12:53.8686704Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-05-19T18:12:53.8687358Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-05-19T18:12:53.8695335Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:12:53.8695829Z env: 2024-05-19T18:12:53.8696113Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:12:53.8696464Z ##[endgroup] 2024-05-19T18:12:53.8771737Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-05-19T18:12:53.8772502Z # ignore expansion of "docker ps -q" since it could be empty 2024-05-19T18:12:53.8773083Z # shellcheck disable=SC2046 2024-05-19T18:12:53.8773540Z docker stop $(docker ps -q) || true 2024-05-19T18:12:53.8774033Z # Prune all of the docker images 2024-05-19T18:12:53.8774489Z docker system prune -af 2024-05-19T18:12:53.8781699Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:12:53.8782199Z env: 2024-05-19T18:12:53.8782473Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:12:53.8782818Z ##[endgroup] 2024-05-19T18:12:53.9077243Z "docker stop" requires at least 1 argument. 2024-05-19T18:12:53.9077983Z See 'docker stop --help'. 2024-05-19T18:12:53.9078228Z 2024-05-19T18:12:53.9078456Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-05-19T18:12:53.9078835Z 2024-05-19T18:12:53.9078985Z Stop one or more running containers 2024-05-19T18:12:53.9253883Z Total reclaimed space: 0B 2024-05-19T18:12:53.9286088Z ##[group]Run set +e 2024-05-19T18:12:53.9286462Z set +e 2024-05-19T18:12:53.9286774Z set -x 2024-05-19T18:12:53.9287077Z  2024-05-19T18:12:53.9287403Z PT_DOMAIN=download.pytorch.org 2024-05-19T18:12:53.9288324Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-05-19T18:12:53.9289523Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-05-19T18:12:53.9290277Z # one is returned at random 2024-05-19T18:12:53.9290829Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-05-19T18:12:53.9291339Z  2024-05-19T18:12:53.9291664Z if [ -z "${RESOLVED_IP}" ]; then 2024-05-19T18:12:53.9292290Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-05-19T18:12:53.9293053Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-05-19T18:12:53.9293606Z  2024-05-19T18:12:53.9293940Z  if [ -z "${RESOLVED_IP}" ]; then 2024-05-19T18:12:53.9294502Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-05-19T18:12:53.9295018Z  exit 1 2024-05-19T18:12:53.9295342Z  fi 2024-05-19T18:12:53.9295640Z fi 2024-05-19T18:12:53.9296031Z  2024-05-19T18:12:53.9296409Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-05-19T18:12:53.9296929Z  # Clean up any old records first 2024-05-19T18:12:53.9297448Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-05-19T18:12:53.9297914Z fi 2024-05-19T18:12:53.9298193Z  2024-05-19T18:12:53.9298636Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-05-19T18:12:53.9299321Z cat /etc/hosts 2024-05-19T18:12:53.9306948Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:12:53.9307443Z env: 2024-05-19T18:12:53.9307733Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:12:53.9308091Z ##[endgroup] 2024-05-19T18:12:53.9327780Z + PT_DOMAIN=download.pytorch.org 2024-05-19T18:12:53.9331956Z ++ dig -4 +short download.pytorch.org 2024-05-19T18:12:53.9332399Z ++ tail -n1 2024-05-19T18:12:53.9800362Z + RESOLVED_IP=18.160.10.22 2024-05-19T18:12:53.9800940Z + '[' -z 18.160.10.22 ']' 2024-05-19T18:12:53.9801395Z + grep -r download.pytorch.org /etc/hosts 2024-05-19T18:12:53.9806693Z 18.160.10.22 download.pytorch.org 2024-05-19T18:12:53.9807690Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-05-19T18:12:53.9892194Z + echo '18.160.10.22 download.pytorch.org' 2024-05-19T18:12:53.9892881Z + sudo tee -a /etc/hosts 2024-05-19T18:12:53.9953431Z 18.160.10.22 download.pytorch.org 2024-05-19T18:12:53.9966309Z + cat /etc/hosts 2024-05-19T18:12:53.9971248Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-05-19T18:12:53.9983854Z ::1 localhost6 localhost6.localdomain6 2024-05-19T18:12:53.9984357Z 18.160.10.22 download.pytorch.org 2024-05-19T18:12:54.0136369Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-05-19T18:12:54.0136979Z with: 2024-05-19T18:12:54.0138070Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:12:54.0139302Z docker-build-dir: .ci/docker 2024-05-19T18:12:54.0139685Z working-directory: . 2024-05-19T18:12:54.0140158Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:12:54.0140684Z force-push: false 2024-05-19T18:12:54.0140984Z env: 2024-05-19T18:12:54.0141258Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:12:54.0141605Z ##[endgroup] 2024-05-19T18:12:54.0159925Z ##[group]Run set -ex 2024-05-19T18:12:54.0160296Z set -ex 2024-05-19T18:12:54.0160593Z  2024-05-19T18:12:54.0161161Z # If the docker build directory or the build script doesn't exist, the action will 2024-05-19T18:12:54.0162191Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-05-19T18:12:54.0162995Z # job could then download the pre-built image as usual 2024-05-19T18:12:54.0163746Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-05-19T18:12:54.0164545Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-05-19T18:12:54.0165190Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:12:54.0165757Z  2024-05-19T18:12:54.0166270Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-05-19T18:12:54.0166904Z  exit 0 2024-05-19T18:12:54.0167212Z else 2024-05-19T18:12:54.0167587Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-05-19T18:12:54.0168045Z fi 2024-05-19T18:12:54.0168332Z  2024-05-19T18:12:54.0168813Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-05-19T18:12:54.0169757Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-05-19T18:12:54.0170537Z  # use it as it is, but first let's extract the tag 2024-05-19T18:12:54.0171244Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-05-19T18:12:54.0171979Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:12:54.0172681Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:12:54.0173238Z else 2024-05-19T18:12:54.0173680Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-05-19T18:12:54.0174343Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:12:54.0175229Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:12:54.0175969Z fi 2024-05-19T18:12:54.0183448Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:12:54.0183952Z env: 2024-05-19T18:12:54.0184231Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:12:54.0184599Z REPO_NAME: pytorch 2024-05-19T18:12:54.0185723Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:12:54.0186922Z DOCKER_BUILD_DIR: .ci/docker 2024-05-19T18:12:54.0187424Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:12:54.0187944Z ##[endgroup] 2024-05-19T18:12:54.0207701Z + [[ ! -d .ci/docker ]] 2024-05-19T18:12:54.0208136Z + [[ ! -f .ci/docker/build.sh ]] 2024-05-19T18:12:54.0208526Z + echo skip=false 2024-05-19T18:12:54.0210568Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-05-19T18:12:54.0213115Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:12:54.0214298Z ++ awk -F '[:,]' '{print $2}' 2024-05-19T18:12:54.0219907Z + DOCKER_TAG=d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:12:54.0220577Z + echo docker-tag=d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:12:54.0222217Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:12:54.0249520Z ##[group]Run set +e 2024-05-19T18:12:54.0249896Z set +e 2024-05-19T18:12:54.0250199Z set -x 2024-05-19T18:12:54.0250487Z  2024-05-19T18:12:54.0250771Z login() { 2024-05-19T18:12:54.0251443Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-05-19T18:12:54.0252162Z } 2024-05-19T18:12:54.0252438Z  2024-05-19T18:12:54.0252724Z retry () { 2024-05-19T18:12:54.0253126Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-05-19T18:12:54.0253579Z } 2024-05-19T18:12:54.0253967Z  2024-05-19T18:12:54.0254286Z retry login "${DOCKER_REGISTRY}" 2024-05-19T18:12:54.0254708Z  2024-05-19T18:12:54.0255178Z # Check if image already exists, if it does then skip building it 2024-05-19T18:12:54.0255886Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-05-19T18:12:54.0256389Z  exit 0 2024-05-19T18:12:54.0256691Z fi 2024-05-19T18:12:54.0256964Z  2024-05-19T18:12:54.0257468Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-05-19T18:12:54.0258298Z # be empty. The default action would be to continue rebuild the image 2024-05-19T18:12:54.0259049Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-05-19T18:12:54.0259723Z  # if we're on the base branch then use the parent commit 2024-05-19T18:12:54.0260330Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-05-19T18:12:54.0260767Z else 2024-05-19T18:12:54.0261249Z  # otherwise we're on a PR, so use the most recent base commit 2024-05-19T18:12:54.0261943Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-05-19T18:12:54.0262454Z fi 2024-05-19T18:12:54.0262736Z  2024-05-19T18:12:54.0263059Z if [[ -z "${MERGE_BASE}" ]]; then 2024-05-19T18:12:54.0263561Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-05-19T18:12:54.0264028Z  2024-05-19T18:12:54.0264687Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-05-19T18:12:54.0265445Z  exit 0 2024-05-19T18:12:54.0265741Z fi 2024-05-19T18:12:54.0266020Z  2024-05-19T18:12:54.0266463Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-05-19T18:12:54.0267425Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-05-19T18:12:54.0268220Z  exit 1 2024-05-19T18:12:54.0268526Z fi 2024-05-19T18:12:54.0268811Z  2024-05-19T18:12:54.0269302Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-05-19T18:12:54.0270476Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-05-19T18:12:54.0271406Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-05-19T18:12:54.0272367Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-05-19T18:12:54.0273457Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-05-19T18:12:54.0274181Z fi 2024-05-19T18:12:54.0274461Z  2024-05-19T18:12:54.0274827Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-05-19T18:12:54.0282483Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:12:54.0282992Z env: 2024-05-19T18:12:54.0283265Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:12:54.0283637Z DOCKER_BUILD_DIR: .ci/docker 2024-05-19T18:12:54.0284119Z BASE_REVISION: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:12:54.0285395Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:12:54.0286632Z DOCKER_TAG: d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:12:54.0287240Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:12:54.0287757Z ##[endgroup] 2024-05-19T18:12:54.0307430Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:12:54.0308121Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:12:54.0308997Z + aws ecr get-login-password --region us-east-1 2024-05-19T18:12:54.0309896Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:12:54.4392528Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-05-19T18:12:54.4393379Z Configure a credential helper to remove this warning. See 2024-05-19T18:12:54.4394197Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-05-19T18:12:54.4394708Z 2024-05-19T18:12:54.4394830Z Login Succeeded 2024-05-19T18:12:54.4405849Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:12:54.6364585Z { 2024-05-19T18:12:54.6364967Z "schemaVersion": 2, 2024-05-19T18:12:54.6365676Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-05-19T18:12:54.6366267Z "config": { 2024-05-19T18:12:54.6366782Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-05-19T18:12:54.6368155Z "size": 45561, 2024-05-19T18:12:54.6368979Z "digest": "sha256:a0542ac5317cc182e1aac2cb8d63a2d6287450937722bab68464513bbbc603ef" 2024-05-19T18:12:54.6369862Z }, 2024-05-19T18:12:54.6370133Z "layers": [ 2024-05-19T18:12:54.6370417Z { 2024-05-19T18:12:54.6370862Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6371421Z "size": 28584317, 2024-05-19T18:12:54.6371975Z "digest": "sha256:63e9bbe323274e77e58d77c6ab6802d247458f784222fbb07a2556d6ec74ee05" 2024-05-19T18:12:54.6372587Z }, 2024-05-19T18:12:54.6372838Z { 2024-05-19T18:12:54.6373266Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6373817Z "size": 7944771, 2024-05-19T18:12:54.6374374Z "digest": "sha256:cfb3d849840ee60cee7b02bad68c1fc3c15928ebcd88f327754766b670578ed6" 2024-05-19T18:12:54.6375018Z }, 2024-05-19T18:12:54.6375260Z { 2024-05-19T18:12:54.6375685Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6376233Z "size": 57593718, 2024-05-19T18:12:54.6376764Z "digest": "sha256:968831e596a6288f0fed9b8612ee4ee8e75511037c4305058805492c5162e481" 2024-05-19T18:12:54.6377359Z }, 2024-05-19T18:12:54.6379015Z + exit 0 2024-05-19T18:12:54.6379709Z { 2024-05-19T18:12:54.6380318Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6381049Z "size": 187, 2024-05-19T18:12:54.6381763Z "digest": "sha256:ea310eb267ca1cab61b6b16f566cd28bfd59a741395a011f5e76716e15ba57c6" 2024-05-19T18:12:54.6382610Z }, 2024-05-19T18:12:54.6382993Z { 2024-05-19T18:12:54.6383564Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6384316Z "size": 6885, 2024-05-19T18:12:54.6385091Z "digest": "sha256:3af11d09e9cd1eb9c379f0a4071231e5a5642eb728b4b33bcb76be291f3c9488" 2024-05-19T18:12:54.6386228Z }, 2024-05-19T18:12:54.6386619Z { 2024-05-19T18:12:54.6387279Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6388130Z "size": 1361380219, 2024-05-19T18:12:54.6388922Z "digest": "sha256:ebfec18059b91e56882881ac34754f917861edb5f732c395d2a1a851bbd6db46" 2024-05-19T18:12:54.6390182Z }, 2024-05-19T18:12:54.6390571Z { 2024-05-19T18:12:54.6391207Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6392054Z "size": 62686, 2024-05-19T18:12:54.6392885Z "digest": "sha256:533b4aebf16914c763b7b0de3ce657590c6f979045e9fdf1f816adaf68d8a4d3" 2024-05-19T18:12:54.6393797Z }, 2024-05-19T18:12:54.6394130Z { 2024-05-19T18:12:54.6394590Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6395147Z "size": 1685, 2024-05-19T18:12:54.6395685Z "digest": "sha256:9dd75d06a0910f28cb1e484b8808724e5a6ee570ecb8fc04631368f546b39ed9" 2024-05-19T18:12:54.6396283Z }, 2024-05-19T18:12:54.6396543Z { 2024-05-19T18:12:54.6396974Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6397520Z "size": 1523, 2024-05-19T18:12:54.6398050Z "digest": "sha256:30bfca4dd3492d60ed8035b0eeb1229897041140db117f5663465a551e25851d" 2024-05-19T18:12:54.6398820Z }, 2024-05-19T18:12:54.6399069Z { 2024-05-19T18:12:54.6399495Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6400055Z "size": 2608186849, 2024-05-19T18:12:54.6400629Z "digest": "sha256:1b57ce94cad9a5097d99b8d6f7bcd51df5a162fc3c5f5686b689b76993724bc8" 2024-05-19T18:12:54.6401250Z }, 2024-05-19T18:12:54.6401500Z { 2024-05-19T18:12:54.6401927Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6402475Z "size": 86631, 2024-05-19T18:12:54.6403021Z "digest": "sha256:9ee6bdb31195dd42fd98147a75d540efe0c5708e0ecda866a0bca060084fddab" 2024-05-19T18:12:54.6403626Z }, 2024-05-19T18:12:54.6403870Z { 2024-05-19T18:12:54.6404304Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6404850Z "size": 1822, 2024-05-19T18:12:54.6405388Z "digest": "sha256:8af83a173aa33e1330fdde4efc5b96334ef85736ec0725bfeedc72c227dcfcc5" 2024-05-19T18:12:54.6406008Z }, 2024-05-19T18:12:54.6406259Z { 2024-05-19T18:12:54.6406684Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6407240Z "size": 245735746, 2024-05-19T18:12:54.6407795Z "digest": "sha256:f8281676bd20d66f882b7edad2f27bd52787f63ff5c68768681df27d74d9d07c" 2024-05-19T18:12:54.6408405Z }, 2024-05-19T18:12:54.6408649Z { 2024-05-19T18:12:54.6409211Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6409861Z "size": 544, 2024-05-19T18:12:54.6410393Z "digest": "sha256:748e3cfdd8d6f501456917dcc4a9e8b11092aa4e784a46b00c3a7a13274e2742" 2024-05-19T18:12:54.6411002Z }, 2024-05-19T18:12:54.6411249Z { 2024-05-19T18:12:54.6411685Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6412228Z "size": 1279, 2024-05-19T18:12:54.6412760Z "digest": "sha256:5fa8d379f04e0947dc4af19169168984b6dfe62a40efbd97ee9bc60015cb2970" 2024-05-19T18:12:54.6413380Z }, 2024-05-19T18:12:54.6413627Z { 2024-05-19T18:12:54.6414054Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6414607Z "size": 484, 2024-05-19T18:12:54.6415161Z "digest": "sha256:a95b9f41fb5bbac1fd00c55419da0b54b0c1b38cd46e22b6e95293bd639615b5" 2024-05-19T18:12:54.6415784Z }, 2024-05-19T18:12:54.6416029Z { 2024-05-19T18:12:54.6416459Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6417012Z "size": 91705725, 2024-05-19T18:12:54.6417557Z "digest": "sha256:8b6d0b819d19a961789f05d3c03c73f6ee67059e9464c8bbc0a1124113a32395" 2024-05-19T18:12:54.6418154Z }, 2024-05-19T18:12:54.6418403Z { 2024-05-19T18:12:54.6418830Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6419458Z "size": 3236, 2024-05-19T18:12:54.6420001Z "digest": "sha256:c61996e733996ff4a9cf89f2a69716a6add90b5c8b1902feb91d8e0dfd0f7bc4" 2024-05-19T18:12:54.6420609Z }, 2024-05-19T18:12:54.6420856Z { 2024-05-19T18:12:54.6421275Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6421827Z "size": 1905, 2024-05-19T18:12:54.6422366Z "digest": "sha256:9e69dcd7661c1bee67677240651319ec72ee977219e8097f93be00c6555c9ab5" 2024-05-19T18:12:54.6422969Z }, 2024-05-19T18:12:54.6423211Z { 2024-05-19T18:12:54.6423644Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6424194Z "size": 699, 2024-05-19T18:12:54.6424735Z "digest": "sha256:cfcaf2eb82498ef031f7d481e635b1a37af01b9e389f30555b603d75e2c00602" 2024-05-19T18:12:54.6425347Z }, 2024-05-19T18:12:54.6425599Z { 2024-05-19T18:12:54.6426026Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6426570Z "size": 2811659560, 2024-05-19T18:12:54.6427154Z "digest": "sha256:7cb10a6ae8df3c778e529febd7eef0333c2af45e2a2ed45f45967bb7b4219131" 2024-05-19T18:12:54.6427774Z }, 2024-05-19T18:12:54.6428020Z { 2024-05-19T18:12:54.6428446Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6429050Z "size": 378, 2024-05-19T18:12:54.6429585Z "digest": "sha256:07fa87a43fb8c069109d9c97743dcc016ad605a221c1df4c7d79863d0d0a2229" 2024-05-19T18:12:54.6430413Z }, 2024-05-19T18:12:54.6430659Z { 2024-05-19T18:12:54.6431091Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6431641Z "size": 11887, 2024-05-19T18:12:54.6432241Z "digest": "sha256:f16e94b048018b4677a272dcb1cb347e9c90be713acffa5bdd2d63a972224ca9" 2024-05-19T18:12:54.6432841Z }, 2024-05-19T18:12:54.6433093Z { 2024-05-19T18:12:54.6433519Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6434057Z "size": 803, 2024-05-19T18:12:54.6434597Z "digest": "sha256:c857065196d6f476be67a20b1e26f3ef6e71ae427e4feee2412e7f65cc208493" 2024-05-19T18:12:54.6435208Z }, 2024-05-19T18:12:54.6435455Z { 2024-05-19T18:12:54.6435875Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6436428Z "size": 106, 2024-05-19T18:12:54.6436972Z "digest": "sha256:9d5c38cad0c9a12d6b9b3070caf13ae1003594cf17c75170ccadf0f8719f93ab" 2024-05-19T18:12:54.6437586Z }, 2024-05-19T18:12:54.6437828Z { 2024-05-19T18:12:54.6438255Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6438795Z "size": 503, 2024-05-19T18:12:54.6439316Z "digest": "sha256:925ac0c7ff935448f88445123dc223bb4531147eec4d0e0c34f9f05bd4e14f46" 2024-05-19T18:12:54.6439917Z }, 2024-05-19T18:12:54.6440164Z { 2024-05-19T18:12:54.6440593Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6454481Z "size": 121477245, 2024-05-19T18:12:54.6455122Z "digest": "sha256:cfbf8bf4873dc1eef0b1a35dd290348be3ceb2dafccdc4820e9460ddfd8f7c54" 2024-05-19T18:12:54.6455764Z }, 2024-05-19T18:12:54.6456016Z { 2024-05-19T18:12:54.6456453Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6457009Z "size": 109, 2024-05-19T18:12:54.6457548Z "digest": "sha256:01914a189c0b5a993f1fc3ddb461abe3964eba3ae55fab9eff5a53f77de19178" 2024-05-19T18:12:54.6458172Z }, 2024-05-19T18:12:54.6458430Z { 2024-05-19T18:12:54.6458861Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6459406Z "size": 488, 2024-05-19T18:12:54.6459943Z "digest": "sha256:3c2451fbe84c6b4678a24659b7abe1ea3cca59da338f16b7563d6875accf6eee" 2024-05-19T18:12:54.6460560Z }, 2024-05-19T18:12:54.6460812Z { 2024-05-19T18:12:54.6461233Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6461786Z "size": 298, 2024-05-19T18:12:54.6462334Z "digest": "sha256:aea8a2a73b72b6c647fafb9f4b1b3d65cb7c28172d3b7a58de3bdb6335addb00" 2024-05-19T18:12:54.6463068Z }, 2024-05-19T18:12:54.6463324Z { 2024-05-19T18:12:54.6463750Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6464305Z "size": 103, 2024-05-19T18:12:54.6464842Z "digest": "sha256:c26f348bfda1e1a6e32f77a9054fa8d26cd59cf6fe8cc3ea1ff9d1816dc74d22" 2024-05-19T18:12:54.6465473Z }, 2024-05-19T18:12:54.6465724Z { 2024-05-19T18:12:54.6466152Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6466694Z "size": 1471, 2024-05-19T18:12:54.6467234Z "digest": "sha256:347c41648f51ed569232ea48943ef3e1424197b90bfae5b89898f2273f64e76b" 2024-05-19T18:12:54.6467842Z }, 2024-05-19T18:12:54.6468084Z { 2024-05-19T18:12:54.6468511Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6469064Z "size": 444189070, 2024-05-19T18:12:54.6469613Z "digest": "sha256:00953247955f346ee0a69eb263efa2135a9fc3e9680322e5c39cf79e4c06fab5" 2024-05-19T18:12:54.6470400Z }, 2024-05-19T18:12:54.6470657Z { 2024-05-19T18:12:54.6471093Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6471705Z "size": 162, 2024-05-19T18:12:54.6472250Z "digest": "sha256:df27b4a5ce7ca0be34e1295afceb581f6ac42c1c0888e253c481bbce25efde72" 2024-05-19T18:12:54.6472972Z }, 2024-05-19T18:12:54.6473220Z { 2024-05-19T18:12:54.6473639Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6474188Z "size": 564, 2024-05-19T18:12:54.6474723Z "digest": "sha256:acb7fcd74bf2071a5d257e338f2229b9a9747f6e96de138091ed80cfa845709f" 2024-05-19T18:12:54.6475342Z }, 2024-05-19T18:12:54.6475584Z { 2024-05-19T18:12:54.6476010Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6476555Z "size": 35867331, 2024-05-19T18:12:54.6477109Z "digest": "sha256:5bd6ac51ef5d3109aba547ec49aec04535079b866d45939fa098e33b7f20d94f" 2024-05-19T18:12:54.6477711Z }, 2024-05-19T18:12:54.6477959Z { 2024-05-19T18:12:54.6478391Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6478937Z "size": 104, 2024-05-19T18:12:54.6479463Z "digest": "sha256:1795bce633b0c48054bada65aede57ae82761e922b39b354aebedd2d3fc82a01" 2024-05-19T18:12:54.6480079Z }, 2024-05-19T18:12:54.6480333Z { 2024-05-19T18:12:54.6480756Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6481306Z "size": 425, 2024-05-19T18:12:54.6481878Z "digest": "sha256:e41de6016f3c0f9309d3727db3cf335f2789260369b9c02485b5a1eda2a23949" 2024-05-19T18:12:54.6482483Z }, 2024-05-19T18:12:54.6482721Z { 2024-05-19T18:12:54.6483153Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6483698Z "size": 20262012, 2024-05-19T18:12:54.6484253Z "digest": "sha256:458f5f2af8cfff41b832b1ffa0ac89536cc75afaa2b007348475812be84a3168" 2024-05-19T18:12:54.6484858Z }, 2024-05-19T18:12:54.6485107Z { 2024-05-19T18:12:54.6485537Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6486083Z "size": 437, 2024-05-19T18:12:54.6486608Z "digest": "sha256:ef99a2a9cffb05645e67c1ff7eff9d0400a37d6277a734e0534e544e6d70abc3" 2024-05-19T18:12:54.6487221Z }, 2024-05-19T18:12:54.6487468Z { 2024-05-19T18:12:54.6487895Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6488436Z "size": 699, 2024-05-19T18:12:54.6488975Z "digest": "sha256:cfcaf2eb82498ef031f7d481e635b1a37af01b9e389f30555b603d75e2c00602" 2024-05-19T18:12:54.6489673Z }, 2024-05-19T18:12:54.6489925Z { 2024-05-19T18:12:54.6490360Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6490917Z "size": 142, 2024-05-19T18:12:54.6491472Z "digest": "sha256:1ba8d08d58765b6585b8a6430120bdf33e85a118dbe004432a9515481c728a6f" 2024-05-19T18:12:54.6492077Z }, 2024-05-19T18:12:54.6492330Z { 2024-05-19T18:12:54.6492767Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6493393Z "size": 136, 2024-05-19T18:12:54.6493938Z "digest": "sha256:0c67ae0247cf4129424322fdc4c7d64de5f02712a6851839df767491d40ac668" 2024-05-19T18:12:54.6494549Z }, 2024-05-19T18:12:54.6494805Z { 2024-05-19T18:12:54.6495236Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6495804Z "size": 4981810470, 2024-05-19T18:12:54.6496375Z "digest": "sha256:3ebf7de5956e7374b72ff809508272a8ac6ff23b2293581bef5847cd0af1b4bb" 2024-05-19T18:12:54.6496994Z }, 2024-05-19T18:12:54.6497244Z { 2024-05-19T18:12:54.6497684Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6498242Z "size": 190, 2024-05-19T18:12:54.6498794Z "digest": "sha256:a6b88127ee12fade12d3202bc71d30768c0ddc0eb46ad572bd5089b6aa1bb0e2" 2024-05-19T18:12:54.6499411Z }, 2024-05-19T18:12:54.6499668Z { 2024-05-19T18:12:54.6500105Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6500662Z "size": 565, 2024-05-19T18:12:54.6501219Z "digest": "sha256:7b0cf0b8be50e7fae7398549f658c86b1e9c348102f84ef2c230bc6af5599739" 2024-05-19T18:12:54.6501838Z }, 2024-05-19T18:12:54.6502096Z { 2024-05-19T18:12:54.6502523Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6503156Z "size": 43162755, 2024-05-19T18:12:54.6503732Z "digest": "sha256:044061f29c13bd178d3113fb9ccfc367da97f96f6bf948dfcae1ff5b1e17be2f" 2024-05-19T18:12:54.6504355Z }, 2024-05-19T18:12:54.6504604Z { 2024-05-19T18:12:54.6505045Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6505604Z "size": 106, 2024-05-19T18:12:54.6506138Z "digest": "sha256:f21869a483d43c856cf8639bdb4270d8295023741b5cddebed62bcfe3b8caa7b" 2024-05-19T18:12:54.6506755Z }, 2024-05-19T18:12:54.6507009Z { 2024-05-19T18:12:54.6507446Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6508022Z "size": 1212, 2024-05-19T18:12:54.6508586Z "digest": "sha256:a155d9e6e7e7fd2243fba6789a98b96aa2db9d5d6462b50d9c11cfc7f3ac34e7" 2024-05-19T18:12:54.6509219Z }, 2024-05-19T18:12:54.6509475Z { 2024-05-19T18:12:54.6509907Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6510645Z "size": 699, 2024-05-19T18:12:54.6511205Z "digest": "sha256:cfcaf2eb82498ef031f7d481e635b1a37af01b9e389f30555b603d75e2c00602" 2024-05-19T18:12:54.6511837Z }, 2024-05-19T18:12:54.6512091Z { 2024-05-19T18:12:54.6512533Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6513093Z "size": 137, 2024-05-19T18:12:54.6513635Z "digest": "sha256:6028f518cba8d1b51ca287446cfc0a9db237fbeea2198738ee2c114a24a8c34a" 2024-05-19T18:12:54.6514259Z }, 2024-05-19T18:12:54.6514516Z { 2024-05-19T18:12:54.6514958Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6515518Z "size": 119, 2024-05-19T18:12:54.6516076Z "digest": "sha256:9f0e16ae766e08c8bf1276b07cd6781a3f9ca54131846275ffcfc9473a6312dd" 2024-05-19T18:12:54.6516705Z }, 2024-05-19T18:12:54.6516964Z { 2024-05-19T18:12:54.6517400Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6517961Z "size": 1853517530, 2024-05-19T18:12:54.6518530Z "digest": "sha256:26d45b11322ae5593c42d7929f046ca6d236f7ec5e8a13d8930360c013d32747" 2024-05-19T18:12:54.6519139Z }, 2024-05-19T18:12:54.6519390Z { 2024-05-19T18:12:54.6519831Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6520391Z "size": 172, 2024-05-19T18:12:54.6520927Z "digest": "sha256:a78a7e4425658766ba6e70bb1367f10c352dc4a7bfb16432be3d9a3496368333" 2024-05-19T18:12:54.6521547Z }, 2024-05-19T18:12:54.6521808Z { 2024-05-19T18:12:54.6522249Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6522801Z "size": 1838, 2024-05-19T18:12:54.6523369Z "digest": "sha256:df585564bf4cc28aec4957fbec751620ae4ca50edbbdcec1c5eacc306a16e2e1" 2024-05-19T18:12:54.6524093Z }, 2024-05-19T18:12:54.6524355Z { 2024-05-19T18:12:54.6524790Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6525359Z "size": 7529792, 2024-05-19T18:12:54.6525919Z "digest": "sha256:f125619c47120e1a542ddb0e822b5ffc07f9d165a0977f8c4064fdf67865e3fd" 2024-05-19T18:12:54.6526541Z }, 2024-05-19T18:12:54.6526807Z { 2024-05-19T18:12:54.6527252Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6527808Z "size": 162, 2024-05-19T18:12:54.6528352Z "digest": "sha256:4cc671ff8870320a4e5bab71f83b755b9b96c777dba43fdbfc1a0e3d001d8936" 2024-05-19T18:12:54.6528970Z }, 2024-05-19T18:12:54.6529228Z { 2024-05-19T18:12:54.6529723Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6530271Z "size": 7939, 2024-05-19T18:12:54.6530809Z "digest": "sha256:e1d3209e04ae7e31773f49e41f7cee2949fa017569383786876f6d1ba0d7d8ac" 2024-05-19T18:12:54.6531423Z }, 2024-05-19T18:12:54.6531683Z { 2024-05-19T18:12:54.6532114Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6532671Z "size": 8072, 2024-05-19T18:12:54.6533209Z "digest": "sha256:322e57f70842c48212f87b2e4c1cc094c25470b90fec1d30ad715f4d907c1f79" 2024-05-19T18:12:54.6533882Z }, 2024-05-19T18:12:54.6534141Z { 2024-05-19T18:12:54.6534583Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6535154Z "size": 301, 2024-05-19T18:12:54.6535688Z "digest": "sha256:d58f09a95d3e18b4d653b7767e371c1e9ef320625fa4cc6a92cbd1771d7ef768" 2024-05-19T18:12:54.6536309Z }, 2024-05-19T18:12:54.6536568Z { 2024-05-19T18:12:54.6537006Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6537554Z "size": 7617584, 2024-05-19T18:12:54.6538118Z "digest": "sha256:f611e28aa59ab55b92e54aac3cc04729565ccae9d0edb75cc45a534d752a9de1" 2024-05-19T18:12:54.6538735Z }, 2024-05-19T18:12:54.6538988Z { 2024-05-19T18:12:54.6539427Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6539983Z "size": 108, 2024-05-19T18:12:54.6540522Z "digest": "sha256:ff67e6bd8e2302312131632eae7033ce84ce85503282bfd8b31c008ff533990a" 2024-05-19T18:12:54.6541131Z }, 2024-05-19T18:12:54.6541391Z { 2024-05-19T18:12:54.6541823Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6542377Z "size": 54145477, 2024-05-19T18:12:54.6542913Z "digest": "sha256:d2b06b7f79e8770329516b40d5189f7c2b2d38404132638103a594685fe1d569" 2024-05-19T18:12:54.6543506Z }, 2024-05-19T18:12:54.6543756Z { 2024-05-19T18:12:54.6544181Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6544738Z "size": 507, 2024-05-19T18:12:54.6545275Z "digest": "sha256:c27bc13ce9d41097b89fa4c7f53feb417379940ba35474fede1b3322540e85e9" 2024-05-19T18:12:54.6545887Z }, 2024-05-19T18:12:54.6546135Z { 2024-05-19T18:12:54.6546573Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6547135Z "size": 1443748723, 2024-05-19T18:12:54.6547702Z "digest": "sha256:8680ed64cc192868f4e34d4dba9121684e3bba13ac365c8e8539386fe4a5147c" 2024-05-19T18:12:54.6548304Z }, 2024-05-19T18:12:54.6548558Z { 2024-05-19T18:12:54.6549002Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6549555Z "size": 106, 2024-05-19T18:12:54.6550240Z "digest": "sha256:400e9c337687d211e1a78744e24985fbaca792e988f3caf5861040d80b088662" 2024-05-19T18:12:54.6550859Z }, 2024-05-19T18:12:54.6551115Z { 2024-05-19T18:12:54.6551544Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6552103Z "size": 558, 2024-05-19T18:12:54.6552645Z "digest": "sha256:914d4657be1b6936372f45195af96a959eab5bced80ca9deeafc3d6d9db6c03c" 2024-05-19T18:12:54.6553261Z }, 2024-05-19T18:12:54.6553510Z { 2024-05-19T18:12:54.6553944Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6554588Z "size": 46248255, 2024-05-19T18:12:54.6555159Z "digest": "sha256:ecc1d0285d28b6339b9648e2195ce576f30288563ac5de0882653a42bcc0eb5b" 2024-05-19T18:12:54.6555759Z }, 2024-05-19T18:12:54.6556014Z { 2024-05-19T18:12:54.6556450Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:12:54.6557009Z "size": 111, 2024-05-19T18:12:54.6557547Z "digest": "sha256:b882dcc190622361a09ea00b970e0fae5f81ea1f0431a45afa329fd3c97f66d9" 2024-05-19T18:12:54.6558166Z } 2024-05-19T18:12:54.6558419Z ] 2024-05-19T18:12:54.6558662Z } 2024-05-19T18:12:54.6660018Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-05-19T18:12:54.6660520Z tag=${ECR_DOCKER_IMAGE##*/} 2024-05-19T18:12:54.6661087Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-05-19T18:12:54.6668978Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:12:54.6669486Z env: 2024-05-19T18:12:54.6669867Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:12:54.6671724Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:12:54.6673516Z ##[endgroup] 2024-05-19T18:12:54.6698610Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks-d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:12:54.6747940Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-05-19T18:12:54.6748522Z with: 2024-05-19T18:12:54.6749587Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:12:54.6751125Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:12:54.6751656Z env: 2024-05-19T18:12:54.6751936Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:12:54.6752293Z ##[endgroup] 2024-05-19T18:12:54.6768235Z ##[group]Run set -x 2024-05-19T18:12:54.6768575Z set -x 2024-05-19T18:12:54.6768877Z set +e 2024-05-19T18:12:54.6769169Z  2024-05-19T18:12:54.6769529Z login() { 2024-05-19T18:12:54.6770188Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-05-19T18:12:54.6770909Z } 2024-05-19T18:12:54.6771192Z  2024-05-19T18:12:54.6771499Z retry () { 2024-05-19T18:12:54.6771906Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-05-19T18:12:54.6772362Z } 2024-05-19T18:12:54.6772642Z  2024-05-19T18:12:54.6772971Z retry login "${DOCKER_REGISTRY}" 2024-05-19T18:12:54.6773386Z  2024-05-19T18:12:54.6773667Z set -e 2024-05-19T18:12:54.6774153Z # ignore output since only exit code is used for conditional 2024-05-19T18:12:54.6774865Z # only pull docker image if it's not available locally 2024-05-19T18:12:54.6775632Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-05-19T18:12:54.6776334Z  retry docker pull "${DOCKER_IMAGE}" 2024-05-19T18:12:54.6776781Z fi 2024-05-19T18:12:54.6783536Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:12:54.6784029Z env: 2024-05-19T18:12:54.6784314Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:12:54.6785448Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:12:54.6786743Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:12:54.6787262Z ##[endgroup] 2024-05-19T18:12:54.6806913Z + set +e 2024-05-19T18:12:54.6807515Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:12:54.6808167Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:12:54.6809212Z + aws ecr get-login-password --region us-east-1 2024-05-19T18:12:54.6810301Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:12:55.0862410Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-05-19T18:12:55.0863266Z Configure a credential helper to remove this warning. See 2024-05-19T18:12:55.0864285Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-05-19T18:12:55.0864825Z 2024-05-19T18:12:55.0864957Z Login Succeeded 2024-05-19T18:12:55.0874523Z + set -e 2024-05-19T18:12:55.0876113Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:12:55.1023088Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:12:55.1025223Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:12:55.3043897Z d55d6a6f08250cd2609d528116ab6fd6e787bee0: Pulling from pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks 2024-05-19T18:12:55.3045212Z 63e9bbe32327: Pulling fs layer 2024-05-19T18:12:55.3046005Z cfb3d849840e: Pulling fs layer 2024-05-19T18:12:55.3046653Z 968831e596a6: Pulling fs layer 2024-05-19T18:12:55.3047175Z ea310eb267ca: Pulling fs layer 2024-05-19T18:12:55.3047668Z 3af11d09e9cd: Pulling fs layer 2024-05-19T18:12:55.3048174Z ebfec18059b9: Pulling fs layer 2024-05-19T18:12:55.3048685Z 533b4aebf169: Pulling fs layer 2024-05-19T18:12:55.3049197Z 9dd75d06a091: Pulling fs layer 2024-05-19T18:12:55.3049853Z 30bfca4dd349: Pulling fs layer 2024-05-19T18:12:55.3050294Z 1b57ce94cad9: Pulling fs layer 2024-05-19T18:12:55.3050678Z 9ee6bdb31195: Pulling fs layer 2024-05-19T18:12:55.3051052Z 8af83a173aa3: Pulling fs layer 2024-05-19T18:12:55.3051417Z f8281676bd20: Pulling fs layer 2024-05-19T18:12:55.3051842Z 748e3cfdd8d6: Pulling fs layer 2024-05-19T18:12:55.3052334Z 5fa8d379f04e: Pulling fs layer 2024-05-19T18:12:55.3052791Z a95b9f41fb5b: Pulling fs layer 2024-05-19T18:12:55.3053167Z 8b6d0b819d19: Pulling fs layer 2024-05-19T18:12:55.3053585Z c61996e73399: Pulling fs layer 2024-05-19T18:12:55.3054130Z 9e69dcd7661c: Pulling fs layer 2024-05-19T18:12:55.3054647Z cfcaf2eb8249: Pulling fs layer 2024-05-19T18:12:55.3055035Z 7cb10a6ae8df: Pulling fs layer 2024-05-19T18:12:55.3055410Z 07fa87a43fb8: Pulling fs layer 2024-05-19T18:12:55.3055780Z f16e94b04801: Pulling fs layer 2024-05-19T18:12:55.3056141Z c857065196d6: Pulling fs layer 2024-05-19T18:12:55.3056510Z 9d5c38cad0c9: Pulling fs layer 2024-05-19T18:12:55.3056874Z 3af11d09e9cd: Waiting 2024-05-19T18:12:55.3057189Z f8281676bd20: Waiting 2024-05-19T18:12:55.3057561Z 748e3cfdd8d6: Waiting 2024-05-19T18:12:55.3057880Z ebfec18059b9: Waiting 2024-05-19T18:12:55.3058216Z 925ac0c7ff93: Pulling fs layer 2024-05-19T18:12:55.3058584Z 30bfca4dd349: Waiting 2024-05-19T18:12:55.3058916Z cfbf8bf4873d: Pulling fs layer 2024-05-19T18:12:55.3059278Z 9e69dcd7661c: Waiting 2024-05-19T18:12:55.3059595Z 1b57ce94cad9: Waiting 2024-05-19T18:12:55.3059929Z 01914a189c0b: Pulling fs layer 2024-05-19T18:12:55.3060285Z ea310eb267ca: Waiting 2024-05-19T18:12:55.3060613Z cfcaf2eb8249: Waiting 2024-05-19T18:12:55.3060953Z 3c2451fbe84c: Pulling fs layer 2024-05-19T18:12:55.3061310Z 7cb10a6ae8df: Waiting 2024-05-19T18:12:55.3061632Z c61996e73399: Waiting 2024-05-19T18:12:55.3061962Z aea8a2a73b72: Pulling fs layer 2024-05-19T18:12:55.3062327Z 533b4aebf169: Waiting 2024-05-19T18:12:55.3062636Z 07fa87a43fb8: Waiting 2024-05-19T18:12:55.3062973Z c26f348bfda1: Pulling fs layer 2024-05-19T18:12:55.3063336Z 8b6d0b819d19: Waiting 2024-05-19T18:12:55.3063669Z 347c41648f51: Pulling fs layer 2024-05-19T18:12:55.3064019Z 9dd75d06a091: Waiting 2024-05-19T18:12:55.3064346Z 00953247955f: Pulling fs layer 2024-05-19T18:12:55.3064708Z a95b9f41fb5b: Waiting 2024-05-19T18:12:55.3065019Z 9ee6bdb31195: Waiting 2024-05-19T18:12:55.3065338Z 9d5c38cad0c9: Waiting 2024-05-19T18:12:55.3065656Z cfbf8bf4873d: Waiting 2024-05-19T18:12:55.3065979Z 925ac0c7ff93: Waiting 2024-05-19T18:12:55.3066286Z c857065196d6: Waiting 2024-05-19T18:12:55.3066601Z 01914a189c0b: Waiting 2024-05-19T18:12:55.3066921Z 3c2451fbe84c: Waiting 2024-05-19T18:12:55.3067233Z aea8a2a73b72: Waiting 2024-05-19T18:12:55.3067553Z 00953247955f: Waiting 2024-05-19T18:12:55.3067887Z df27b4a5ce7c: Pulling fs layer 2024-05-19T18:12:55.3068270Z acb7fcd74bf2: Pulling fs layer 2024-05-19T18:12:55.3068643Z 5bd6ac51ef5d: Pulling fs layer 2024-05-19T18:12:55.3069022Z 1795bce633b0: Pulling fs layer 2024-05-19T18:12:55.3069383Z df27b4a5ce7c: Waiting 2024-05-19T18:12:55.3069717Z e41de6016f3c: Pulling fs layer 2024-05-19T18:12:55.3070455Z 458f5f2af8cf: Pulling fs layer 2024-05-19T18:12:55.3070822Z acb7fcd74bf2: Waiting 2024-05-19T18:12:55.3071162Z ef99a2a9cffb: Pulling fs layer 2024-05-19T18:12:55.3071524Z 5bd6ac51ef5d: Waiting 2024-05-19T18:12:55.3071843Z 1795bce633b0: Waiting 2024-05-19T18:12:55.3072157Z e41de6016f3c: Waiting 2024-05-19T18:12:55.3072473Z 458f5f2af8cf: Waiting 2024-05-19T18:12:55.3072801Z 1ba8d08d5876: Pulling fs layer 2024-05-19T18:12:55.3073185Z 0c67ae0247cf: Pulling fs layer 2024-05-19T18:12:55.3073656Z 1ba8d08d5876: Waiting 2024-05-19T18:12:55.3073984Z 3ebf7de5956e: Pulling fs layer 2024-05-19T18:12:55.3074476Z a6b88127ee12: Pulling fs layer 2024-05-19T18:12:55.3074858Z 7b0cf0b8be50: Pulling fs layer 2024-05-19T18:12:55.3075220Z 3ebf7de5956e: Waiting 2024-05-19T18:12:55.3075548Z 044061f29c13: Pulling fs layer 2024-05-19T18:12:55.3075918Z f21869a483d4: Pulling fs layer 2024-05-19T18:12:55.3076275Z 7b0cf0b8be50: Waiting 2024-05-19T18:12:55.3076607Z a155d9e6e7e7: Pulling fs layer 2024-05-19T18:12:55.3076960Z a6b88127ee12: Waiting 2024-05-19T18:12:55.3077292Z 6028f518cba8: Pulling fs layer 2024-05-19T18:12:55.3077671Z 9f0e16ae766e: Pulling fs layer 2024-05-19T18:12:55.3078021Z f21869a483d4: Waiting 2024-05-19T18:12:55.3078354Z 26d45b11322a: Pulling fs layer 2024-05-19T18:12:55.3078727Z a78a7e442565: Pulling fs layer 2024-05-19T18:12:55.3079086Z 6028f518cba8: Waiting 2024-05-19T18:12:55.3079397Z a155d9e6e7e7: Waiting 2024-05-19T18:12:55.3079719Z 26d45b11322a: Waiting 2024-05-19T18:12:55.3080032Z a78a7e442565: Waiting 2024-05-19T18:12:55.3080376Z df585564bf4c: Pulling fs layer 2024-05-19T18:12:55.3080752Z f125619c4712: Pulling fs layer 2024-05-19T18:12:55.3081125Z 4cc671ff8870: Pulling fs layer 2024-05-19T18:12:55.3081502Z e1d3209e04ae: Pulling fs layer 2024-05-19T18:12:55.3081870Z 322e57f70842: Pulling fs layer 2024-05-19T18:12:55.3082244Z d58f09a95d3e: Pulling fs layer 2024-05-19T18:12:55.3082625Z f611e28aa59a: Pulling fs layer 2024-05-19T18:12:55.3082999Z ff67e6bd8e23: Pulling fs layer 2024-05-19T18:12:55.3083368Z d2b06b7f79e8: Pulling fs layer 2024-05-19T18:12:55.3083729Z df585564bf4c: Waiting 2024-05-19T18:12:55.3084065Z c27bc13ce9d4: Pulling fs layer 2024-05-19T18:12:55.3084424Z 4cc671ff8870: Waiting 2024-05-19T18:12:55.3084734Z e1d3209e04ae: Waiting 2024-05-19T18:12:55.3085065Z 8680ed64cc19: Pulling fs layer 2024-05-19T18:12:55.3085439Z 400e9c337687: Pulling fs layer 2024-05-19T18:12:55.3085787Z d58f09a95d3e: Waiting 2024-05-19T18:12:55.3086118Z 914d4657be1b: Pulling fs layer 2024-05-19T18:12:55.3086473Z f125619c4712: Waiting 2024-05-19T18:12:55.3086808Z ecc1d0285d28: Pulling fs layer 2024-05-19T18:12:55.3087167Z d2b06b7f79e8: Waiting 2024-05-19T18:12:55.3087486Z 322e57f70842: Waiting 2024-05-19T18:12:55.3087801Z f611e28aa59a: Waiting 2024-05-19T18:12:55.3088126Z b882dcc19062: Pulling fs layer 2024-05-19T18:12:55.3088491Z c27bc13ce9d4: Waiting 2024-05-19T18:12:55.3088813Z ff67e6bd8e23: Waiting 2024-05-19T18:12:55.3089132Z 914d4657be1b: Waiting 2024-05-19T18:12:55.3089565Z ecc1d0285d28: Waiting 2024-05-19T18:12:55.4462224Z cfb3d849840e: Verifying Checksum 2024-05-19T18:12:55.4462704Z cfb3d849840e: Download complete 2024-05-19T18:12:55.5388323Z ea310eb267ca: Download complete 2024-05-19T18:12:55.6273931Z 3af11d09e9cd: Verifying Checksum 2024-05-19T18:12:55.6274439Z 3af11d09e9cd: Download complete 2024-05-19T18:12:55.6477280Z 63e9bbe32327: Verifying Checksum 2024-05-19T18:12:55.6477866Z 63e9bbe32327: Download complete 2024-05-19T18:12:55.7379254Z 533b4aebf169: Download complete 2024-05-19T18:12:55.8208650Z 9dd75d06a091: Verifying Checksum 2024-05-19T18:12:55.8209198Z 9dd75d06a091: Download complete 2024-05-19T18:12:55.8996559Z 30bfca4dd349: Verifying Checksum 2024-05-19T18:12:55.8997036Z 30bfca4dd349: Download complete 2024-05-19T18:12:55.9325678Z 968831e596a6: Verifying Checksum 2024-05-19T18:12:55.9326215Z 968831e596a6: Download complete 2024-05-19T18:12:56.0204191Z 9ee6bdb31195: Download complete 2024-05-19T18:12:56.1081818Z 8af83a173aa3: Verifying Checksum 2024-05-19T18:12:56.1082338Z 8af83a173aa3: Download complete 2024-05-19T18:12:56.3109796Z 63e9bbe32327: Pull complete 2024-05-19T18:12:56.5503156Z cfb3d849840e: Pull complete 2024-05-19T18:12:57.4792540Z 968831e596a6: Pull complete 2024-05-19T18:12:57.5832514Z ea310eb267ca: Pull complete 2024-05-19T18:12:57.6958683Z 3af11d09e9cd: Pull complete 2024-05-19T18:12:58.6260378Z f8281676bd20: Verifying Checksum 2024-05-19T18:12:58.6260857Z f8281676bd20: Download complete 2024-05-19T18:12:58.7099590Z 748e3cfdd8d6: Verifying Checksum 2024-05-19T18:12:58.7100027Z 748e3cfdd8d6: Download complete 2024-05-19T18:12:58.8027349Z 5fa8d379f04e: Verifying Checksum 2024-05-19T18:12:58.8027890Z 5fa8d379f04e: Download complete 2024-05-19T18:12:58.8786357Z a95b9f41fb5b: Download complete 2024-05-19T18:12:59.8591198Z 8b6d0b819d19: Verifying Checksum 2024-05-19T18:12:59.8591681Z 8b6d0b819d19: Download complete 2024-05-19T18:12:59.9414899Z c61996e73399: Verifying Checksum 2024-05-19T18:12:59.9415325Z c61996e73399: Download complete 2024-05-19T18:13:00.0283718Z 9e69dcd7661c: Verifying Checksum 2024-05-19T18:13:00.0284155Z 9e69dcd7661c: Download complete 2024-05-19T18:13:00.1028222Z cfcaf2eb8249: Download complete 2024-05-19T18:13:09.3025781Z ebfec18059b9: Verifying Checksum 2024-05-19T18:13:09.3026348Z ebfec18059b9: Download complete 2024-05-19T18:13:09.3714147Z 07fa87a43fb8: Verifying Checksum 2024-05-19T18:13:09.3714622Z 07fa87a43fb8: Download complete 2024-05-19T18:13:09.4479551Z f16e94b04801: Verifying Checksum 2024-05-19T18:13:09.4480227Z f16e94b04801: Download complete 2024-05-19T18:13:09.5487405Z c857065196d6: Verifying Checksum 2024-05-19T18:13:09.5487983Z c857065196d6: Download complete 2024-05-19T18:13:09.6347939Z 9d5c38cad0c9: Download complete 2024-05-19T18:13:09.7134099Z 925ac0c7ff93: Download complete 2024-05-19T18:13:10.9956785Z cfbf8bf4873d: Verifying Checksum 2024-05-19T18:13:10.9957428Z cfbf8bf4873d: Download complete 2024-05-19T18:13:11.0714645Z 01914a189c0b: Download complete 2024-05-19T18:13:11.1541837Z 3c2451fbe84c: Download complete 2024-05-19T18:13:11.2367261Z aea8a2a73b72: Verifying Checksum 2024-05-19T18:13:11.2367835Z aea8a2a73b72: Download complete 2024-05-19T18:13:11.3165867Z c26f348bfda1: Verifying Checksum 2024-05-19T18:13:11.3166494Z c26f348bfda1: Download complete 2024-05-19T18:13:11.3919707Z 347c41648f51: Verifying Checksum 2024-05-19T18:13:11.3920193Z 347c41648f51: Download complete 2024-05-19T18:13:15.8978147Z 00953247955f: Verifying Checksum 2024-05-19T18:13:15.8991547Z 00953247955f: Download complete 2024-05-19T18:13:15.9735603Z df27b4a5ce7c: Verifying Checksum 2024-05-19T18:13:15.9736206Z df27b4a5ce7c: Download complete 2024-05-19T18:13:16.0571215Z acb7fcd74bf2: Download complete 2024-05-19T18:13:16.4793824Z 5bd6ac51ef5d: Verifying Checksum 2024-05-19T18:13:16.4794447Z 5bd6ac51ef5d: Download complete 2024-05-19T18:13:16.5593414Z 1795bce633b0: Verifying Checksum 2024-05-19T18:13:16.5593964Z 1795bce633b0: Download complete 2024-05-19T18:13:16.6347273Z e41de6016f3c: Verifying Checksum 2024-05-19T18:13:16.6347822Z e41de6016f3c: Download complete 2024-05-19T18:13:16.8890429Z 458f5f2af8cf: Verifying Checksum 2024-05-19T18:13:16.8890926Z 458f5f2af8cf: Download complete 2024-05-19T18:13:16.9685019Z ef99a2a9cffb: Verifying Checksum 2024-05-19T18:13:16.9685604Z ef99a2a9cffb: Download complete 2024-05-19T18:13:17.0448920Z 1ba8d08d5876: Verifying Checksum 2024-05-19T18:13:17.0449882Z 1ba8d08d5876: Download complete 2024-05-19T18:13:17.1226582Z 0c67ae0247cf: Download complete 2024-05-19T18:13:21.7819117Z ebfec18059b9: Pull complete 2024-05-19T18:13:21.9820227Z 533b4aebf169: Pull complete 2024-05-19T18:13:22.0304159Z 1b57ce94cad9: Verifying Checksum 2024-05-19T18:13:22.0304761Z 1b57ce94cad9: Download complete 2024-05-19T18:13:22.1170233Z a6b88127ee12: Verifying Checksum 2024-05-19T18:13:22.1170950Z a6b88127ee12: Download complete 2024-05-19T18:13:22.2112222Z 7b0cf0b8be50: Verifying Checksum 2024-05-19T18:13:22.2112922Z 7b0cf0b8be50: Download complete 2024-05-19T18:13:22.2122412Z 9dd75d06a091: Pull complete 2024-05-19T18:13:22.4408903Z 30bfca4dd349: Pull complete 2024-05-19T18:13:22.7250749Z 044061f29c13: Verifying Checksum 2024-05-19T18:13:22.7251446Z 044061f29c13: Download complete 2024-05-19T18:13:22.8050689Z f21869a483d4: Verifying Checksum 2024-05-19T18:13:22.8051571Z f21869a483d4: Download complete 2024-05-19T18:13:22.8884754Z a155d9e6e7e7: Download complete 2024-05-19T18:13:22.9636617Z 6028f518cba8: Verifying Checksum 2024-05-19T18:13:22.9637102Z 6028f518cba8: Download complete 2024-05-19T18:13:23.0698059Z 9f0e16ae766e: Verifying Checksum 2024-05-19T18:13:23.0698735Z 9f0e16ae766e: Download complete 2024-05-19T18:13:28.3759022Z 7cb10a6ae8df: Verifying Checksum 2024-05-19T18:13:28.3759849Z 7cb10a6ae8df: Download complete 2024-05-19T18:13:28.4502794Z a78a7e442565: Verifying Checksum 2024-05-19T18:13:28.4503281Z a78a7e442565: Download complete 2024-05-19T18:13:28.5321294Z df585564bf4c: Verifying Checksum 2024-05-19T18:13:28.5321937Z df585564bf4c: Download complete 2024-05-19T18:13:28.6627285Z f125619c4712: Verifying Checksum 2024-05-19T18:13:28.6627793Z f125619c4712: Download complete 2024-05-19T18:13:28.7431513Z 4cc671ff8870: Verifying Checksum 2024-05-19T18:13:28.7432189Z 4cc671ff8870: Download complete 2024-05-19T18:13:28.8279469Z e1d3209e04ae: Download complete 2024-05-19T18:13:28.9152154Z 322e57f70842: Download complete 2024-05-19T18:13:28.9967719Z d58f09a95d3e: Verifying Checksum 2024-05-19T18:13:28.9968301Z d58f09a95d3e: Download complete 2024-05-19T18:13:29.1293406Z f611e28aa59a: Download complete 2024-05-19T18:13:29.2021149Z ff67e6bd8e23: Download complete 2024-05-19T18:13:29.7958343Z d2b06b7f79e8: Verifying Checksum 2024-05-19T18:13:29.7959012Z d2b06b7f79e8: Download complete 2024-05-19T18:13:29.8768869Z c27bc13ce9d4: Verifying Checksum 2024-05-19T18:13:29.8769352Z c27bc13ce9d4: Download complete 2024-05-19T18:13:55.5999227Z 26d45b11322a: Verifying Checksum 2024-05-19T18:13:55.5999702Z 26d45b11322a: Download complete 2024-05-19T18:13:55.6835862Z 400e9c337687: Verifying Checksum 2024-05-19T18:13:55.6836376Z 400e9c337687: Download complete 2024-05-19T18:13:55.7807070Z 914d4657be1b: Verifying Checksum 2024-05-19T18:13:55.7807649Z 914d4657be1b: Download complete 2024-05-19T18:13:57.4143364Z ecc1d0285d28: Verifying Checksum 2024-05-19T18:13:57.4144209Z ecc1d0285d28: Download complete 2024-05-19T18:13:57.4923717Z b882dcc19062: Download complete 2024-05-19T18:14:01.1975589Z 8680ed64cc19: Verifying Checksum 2024-05-19T18:14:01.1976099Z 8680ed64cc19: Download complete 2024-05-19T18:14:24.6151592Z 3ebf7de5956e: Verifying Checksum 2024-05-19T18:14:40.7229748Z 1b57ce94cad9: Pull complete 2024-05-19T18:14:40.9469544Z 9ee6bdb31195: Pull complete 2024-05-19T18:14:41.1658344Z 8af83a173aa3: Pull complete 2024-05-19T18:14:46.2903199Z f8281676bd20: Pull complete 2024-05-19T18:14:46.5163217Z 748e3cfdd8d6: Pull complete 2024-05-19T18:14:46.7259724Z 5fa8d379f04e: Pull complete 2024-05-19T18:14:46.9399837Z a95b9f41fb5b: Pull complete 2024-05-19T18:14:48.7806146Z 8b6d0b819d19: Pull complete 2024-05-19T18:14:48.9998199Z c61996e73399: Pull complete 2024-05-19T18:14:49.2360331Z 9e69dcd7661c: Pull complete 2024-05-19T18:14:49.4589065Z cfcaf2eb8249: Pull complete 2024-05-19T18:15:26.8156394Z 7cb10a6ae8df: Pull complete 2024-05-19T18:15:27.0161525Z 07fa87a43fb8: Pull complete 2024-05-19T18:15:27.2375822Z f16e94b04801: Pull complete 2024-05-19T18:15:27.4569656Z c857065196d6: Pull complete 2024-05-19T18:15:27.6338344Z 9d5c38cad0c9: Pull complete 2024-05-19T18:15:27.7402717Z 925ac0c7ff93: Pull complete 2024-05-19T18:15:30.7983181Z cfbf8bf4873d: Pull complete 2024-05-19T18:15:31.0232274Z 01914a189c0b: Pull complete 2024-05-19T18:15:31.2063767Z 3c2451fbe84c: Pull complete 2024-05-19T18:15:31.3871617Z aea8a2a73b72: Pull complete 2024-05-19T18:15:31.5521733Z c26f348bfda1: Pull complete 2024-05-19T18:15:31.7302233Z 347c41648f51: Pull complete 2024-05-19T18:15:37.5744842Z 00953247955f: Pull complete 2024-05-19T18:15:37.7532046Z df27b4a5ce7c: Pull complete 2024-05-19T18:15:37.9610144Z acb7fcd74bf2: Pull complete 2024-05-19T18:15:38.8389893Z 5bd6ac51ef5d: Pull complete 2024-05-19T18:15:39.0612926Z 1795bce633b0: Pull complete 2024-05-19T18:15:39.2775281Z e41de6016f3c: Pull complete 2024-05-19T18:15:39.7249776Z 458f5f2af8cf: Pull complete 2024-05-19T18:15:39.9446260Z ef99a2a9cffb: Pull complete 2024-05-19T18:15:40.3572833Z 1ba8d08d5876: Pull complete 2024-05-19T18:15:40.5664725Z 0c67ae0247cf: Pull complete 2024-05-19T18:16:17.4162513Z 3ebf7de5956e: Pull complete 2024-05-19T18:16:17.6395392Z a6b88127ee12: Pull complete 2024-05-19T18:16:17.8584285Z 7b0cf0b8be50: Pull complete 2024-05-19T18:16:19.0893731Z 044061f29c13: Pull complete 2024-05-19T18:16:19.2960797Z f21869a483d4: Pull complete 2024-05-19T18:16:19.5523241Z a155d9e6e7e7: Pull complete 2024-05-19T18:16:20.0142688Z 6028f518cba8: Pull complete 2024-05-19T18:16:20.2490313Z 9f0e16ae766e: Pull complete 2024-05-19T18:16:49.8121921Z 26d45b11322a: Pull complete 2024-05-19T18:16:50.0384401Z a78a7e442565: Pull complete 2024-05-19T18:16:50.2497392Z df585564bf4c: Pull complete 2024-05-19T18:16:50.6093815Z f125619c4712: Pull complete 2024-05-19T18:16:50.8315058Z 4cc671ff8870: Pull complete 2024-05-19T18:16:51.0599156Z e1d3209e04ae: Pull complete 2024-05-19T18:16:51.2975698Z 322e57f70842: Pull complete 2024-05-19T18:16:51.5139391Z d58f09a95d3e: Pull complete 2024-05-19T18:16:52.1304749Z f611e28aa59a: Pull complete 2024-05-19T18:16:52.3289158Z ff67e6bd8e23: Pull complete 2024-05-19T18:16:54.0051992Z d2b06b7f79e8: Pull complete 2024-05-19T18:16:54.2203365Z c27bc13ce9d4: Pull complete 2024-05-19T18:17:08.3542711Z 8680ed64cc19: Pull complete 2024-05-19T18:17:08.5777523Z 400e9c337687: Pull complete 2024-05-19T18:17:08.7974143Z 914d4657be1b: Pull complete 2024-05-19T18:17:09.6159330Z ecc1d0285d28: Pull complete 2024-05-19T18:17:09.8475781Z b882dcc19062: Pull complete 2024-05-19T18:17:09.9578152Z Digest: sha256:66f062925f91f9b9204b1dc6d68caf8c89fa82fd4b12cb5f24c19aa4b1c87506 2024-05-19T18:17:10.0018858Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:17:10.0251254Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:17:10.0293354Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-05-19T18:17:10.0294335Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-05-19T18:17:10.0302139Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:17:10.0302643Z env: 2024-05-19T18:17:10.0302925Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:17:10.0303286Z ##[endgroup] 2024-05-19T18:17:10.0457966Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-05-19T18:17:10.0458519Z with: 2024-05-19T18:17:10.0458809Z driver-version: 550.54.15 2024-05-19T18:17:10.0459153Z env: 2024-05-19T18:17:10.0459428Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:17:10.0459781Z ##[endgroup] 2024-05-19T18:17:10.0576901Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-05-19T18:17:10.0577457Z with: 2024-05-19T18:17:10.0577729Z timeout_minutes: 10 2024-05-19T18:17:10.0578054Z max_attempts: 3 2024-05-19T18:17:10.0610003Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-05-19T18:17:10.0640821Z retry_wait_seconds: 10 2024-05-19T18:17:10.0641279Z polling_interval_seconds: 1 2024-05-19T18:17:10.0641652Z warning_on_retry: true 2024-05-19T18:17:10.0642008Z continue_on_error: false 2024-05-19T18:17:10.0642348Z env: 2024-05-19T18:17:10.0642617Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:17:10.0642964Z DRIVER_VERSION: 550.54.15 2024-05-19T18:17:10.0643312Z ##[endgroup] 2024-05-19T18:17:10.1158669Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-05-19T18:17:10.1159649Z + pre_install_nvidia_driver_amzn2 2024-05-19T18:17:10.1160233Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-05-19T18:17:10.4173709Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:17:10.4560707Z No Match for argument: nvidia-driver-latest-dkms 2024-05-19T18:17:10.4802936Z No Packages marked for removal 2024-05-19T18:17:10.4924623Z + install_nvidia_driver_common 2024-05-19T18:17:10.4926308Z + echo 'Before installing NVIDIA driver' 2024-05-19T18:17:10.4926838Z + lspci 2024-05-19T18:17:10.4927890Z Before installing NVIDIA driver 2024-05-19T18:17:10.5009786Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-05-19T18:17:10.5010554Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-05-19T18:17:10.5011474Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-05-19T18:17:10.5012292Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-05-19T18:17:10.5013054Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-05-19T18:17:10.5014075Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-05-19T18:17:10.5014839Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-05-19T18:17:10.5015630Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-05-19T18:17:10.5016221Z + lsmod 2024-05-19T18:17:10.5024509Z Module Size Used by 2024-05-19T18:17:10.5024933Z nvidia_modeset 1339392 0 2024-05-19T18:17:10.5025321Z veth 16384 0 2024-05-19T18:17:10.5025701Z nvidia_uvm 4599808 0 2024-05-19T18:17:10.5026165Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-05-19T18:17:10.5026651Z drm 421888 1 nvidia 2024-05-19T18:17:10.5027076Z i2c_core 77824 2 nvidia,drm 2024-05-19T18:17:10.5027560Z backlight 16384 1 nvidia_modeset 2024-05-19T18:17:10.5028004Z xt_conntrack 16384 1 2024-05-19T18:17:10.5028570Z ipt_MASQUERADE 16384 1 2024-05-19T18:17:10.5029025Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-05-19T18:17:10.5029521Z nf_conntrack_netlink 49152 0 2024-05-19T18:17:10.5030089Z nfnetlink 16384 2 nf_conntrack_netlink 2024-05-19T18:17:10.5030562Z xfrm_user 45056 1 2024-05-19T18:17:10.5030953Z xfrm_algo 16384 1 xfrm_user 2024-05-19T18:17:10.5031377Z iptable_nat 16384 1 2024-05-19T18:17:10.5031753Z nf_conntrack_ipv4 16384 3 2024-05-19T18:17:10.5032189Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-05-19T18:17:10.5032671Z nf_nat_ipv4 16384 1 iptable_nat 2024-05-19T18:17:10.5033228Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-05-19T18:17:10.5034165Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-05-19T18:17:10.5035023Z xt_addrtype 16384 2 2024-05-19T18:17:10.5035409Z iptable_filter 16384 1 2024-05-19T18:17:10.5035789Z br_netfilter 24576 0 2024-05-19T18:17:10.5036190Z bridge 172032 1 br_netfilter 2024-05-19T18:17:10.5036631Z stp 16384 1 bridge 2024-05-19T18:17:10.5037056Z llc 16384 2 bridge,stp 2024-05-19T18:17:10.5037472Z overlay 86016 0 2024-05-19T18:17:10.5037835Z sunrpc 393216 1 2024-05-19T18:17:10.5038261Z dm_mirror 28672 0 2024-05-19T18:17:10.5038654Z dm_region_hash 20480 1 dm_mirror 2024-05-19T18:17:10.5039138Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-05-19T18:17:10.5039643Z dm_mod 143360 2 dm_log,dm_mirror 2024-05-19T18:17:10.5040081Z dax 69632 1 dm_mod 2024-05-19T18:17:10.5040481Z crc32_pclmul 16384 0 2024-05-19T18:17:10.5040976Z ghash_clmulni_intel 16384 0 2024-05-19T18:17:10.5041351Z pcbc 16384 0 2024-05-19T18:17:10.5041722Z aesni_intel 188416 0 2024-05-19T18:17:10.5042119Z aes_x86_64 20480 1 aesni_intel 2024-05-19T18:17:10.5042571Z crypto_simd 16384 1 aesni_intel 2024-05-19T18:17:10.5042993Z mousedev 24576 0 2024-05-19T18:17:10.5043386Z glue_helper 16384 1 aesni_intel 2024-05-19T18:17:10.5043944Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-05-19T18:17:10.5044487Z evdev 20480 3 2024-05-19T18:17:10.5044935Z psmouse 32768 0 2024-05-19T18:17:10.5045308Z button 16384 0 2024-05-19T18:17:10.5045742Z ena 139264 0 2024-05-19T18:17:10.5046111Z ptp 20480 1 ena 2024-05-19T18:17:10.5046576Z pps_core 20480 1 ptp 2024-05-19T18:17:10.5046968Z crc32c_intel 24576 0 2024-05-19T18:17:10.5047336Z autofs4 49152 2 2024-05-19T18:17:10.5047699Z + modinfo nvidia 2024-05-19T18:17:10.5048342Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-05-19T18:17:10.5049142Z alias: char-major-195-* 2024-05-19T18:17:10.5049521Z version: 550.54.15 2024-05-19T18:17:10.5049859Z supported: external 2024-05-19T18:17:10.5050207Z license: NVIDIA 2024-05-19T18:17:10.5050587Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-05-19T18:17:10.5051075Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-05-19T18:17:10.5051524Z srcversion: 833721318DA517F0C2FEC97 2024-05-19T18:17:10.5051992Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-05-19T18:17:10.5052490Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-05-19T18:17:10.5052976Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-05-19T18:17:10.5053452Z depends: i2c-core,drm 2024-05-19T18:17:10.5053815Z retpoline: Y 2024-05-19T18:17:10.5054148Z name: nvidia 2024-05-19T18:17:10.5054724Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-05-19T18:17:10.5055475Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-05-19T18:17:10.5056128Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-05-19T18:17:10.5056728Z parm: NVreg_ResmanDebugLevel:int 2024-05-19T18:17:10.5057164Z parm: NVreg_RmLogonRC:int 2024-05-19T18:17:10.5057599Z parm: NVreg_ModifyDeviceFiles:int 2024-05-19T18:17:10.5058061Z parm: NVreg_DeviceFileUID:int 2024-05-19T18:17:10.5058497Z parm: NVreg_DeviceFileGID:int 2024-05-19T18:17:10.5058933Z parm: NVreg_DeviceFileMode:int 2024-05-19T18:17:10.5059462Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-05-19T18:17:10.5060028Z parm: NVreg_UsePageAttributeTable:int 2024-05-19T18:17:10.5060505Z parm: NVreg_EnablePCIeGen3:int 2024-05-19T18:17:10.5060932Z parm: NVreg_EnableMSI:int 2024-05-19T18:17:10.5061354Z parm: NVreg_TCEBypassMode:int 2024-05-19T18:17:10.5061809Z parm: NVreg_EnableStreamMemOPs:int 2024-05-19T18:17:10.5062340Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-05-19T18:17:10.5062919Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-05-19T18:17:10.5063477Z parm: NVreg_EnableS0ixPowerManagement:int 2024-05-19T18:17:10.5064081Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-05-19T18:17:10.5064678Z parm: NVreg_DynamicPowerManagement:int 2024-05-19T18:17:10.5065283Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-05-19T18:17:10.5065870Z parm: NVreg_EnableGpuFirmware:int 2024-05-19T18:17:10.5066361Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-05-19T18:17:10.5066897Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-05-19T18:17:10.5067435Z parm: NVreg_EnableUserNUMAManagement:int 2024-05-19T18:17:10.5067929Z parm: NVreg_MemoryPoolSize:int 2024-05-19T18:17:10.5068460Z parm: NVreg_KMallocHeapMaxSize:int 2024-05-19T18:17:10.5068947Z parm: NVreg_VMallocHeapMaxSize:int 2024-05-19T18:17:10.5069421Z parm: NVreg_IgnoreMMIOCheck:int 2024-05-19T18:17:10.5069871Z parm: NVreg_NvLinkDisable:int 2024-05-19T18:17:10.5070557Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-05-19T18:17:10.5071082Z parm: NVreg_RegisterPCIDriver:int 2024-05-19T18:17:10.5071564Z parm: NVreg_EnableResizableBar:int 2024-05-19T18:17:10.5072049Z parm: NVreg_EnableDbgBreakpoint:int 2024-05-19T18:17:10.5072546Z parm: NVreg_EnableNonblockingOpen:int 2024-05-19T18:17:10.5073025Z parm: NVreg_RegistryDwords:charp 2024-05-19T18:17:10.5073547Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-05-19T18:17:10.5074134Z parm: NVreg_RmMsg:charp 2024-05-19T18:17:10.5074580Z parm: NVreg_GpuBlacklist:charp 2024-05-19T18:17:10.5075041Z parm: NVreg_TemporaryFilePath:charp 2024-05-19T18:17:10.5075519Z parm: NVreg_ExcludedGpus:charp 2024-05-19T18:17:10.5075978Z parm: NVreg_DmaRemapPeerMmio:int 2024-05-19T18:17:10.5076456Z parm: NVreg_RmNvlinkBandwidth:charp 2024-05-19T18:17:10.5076923Z parm: NVreg_ImexChannelCount:int 2024-05-19T18:17:10.5077378Z parm: rm_firmware_active:charp 2024-05-19T18:17:10.5077787Z + HAS_NVIDIA_DRIVER=0 2024-05-19T18:17:10.5078154Z ++ command -v nvidia-smi 2024-05-19T18:17:10.5078545Z + '[' -x /usr/bin/nvidia-smi ']' 2024-05-19T18:17:10.5078907Z + set +e 2024-05-19T18:17:10.5079411Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-05-19T18:17:12.4914801Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-05-19T18:17:12.4915298Z + NVIDIA_SMI_STATUS=0 2024-05-19T18:17:12.4915798Z + '[' 0 -ne 0 ']' 2024-05-19T18:17:12.4916192Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-05-19T18:17:12.4916583Z + HAS_NVIDIA_DRIVER=1 2024-05-19T18:17:12.4917300Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-05-19T18:17:12.4918024Z + set -e 2024-05-19T18:17:12.4918620Z + '[' 1 -eq 0 ']' 2024-05-19T18:17:12.4919208Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-05-19T18:17:12.4919897Z + post_install_nvidia_driver_common 2024-05-19T18:17:12.4920301Z + sudo modprobe nvidia 2024-05-19T18:17:12.5004163Z + echo 'After installing NVIDIA driver' 2024-05-19T18:17:12.5004610Z + lspci 2024-05-19T18:17:12.5005197Z After installing NVIDIA driver 2024-05-19T18:17:12.5089095Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-05-19T18:17:12.5089876Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-05-19T18:17:12.5090806Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-05-19T18:17:12.5091610Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-05-19T18:17:12.5092362Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-05-19T18:17:12.5093149Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-05-19T18:17:12.5093880Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-05-19T18:17:12.5094663Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-05-19T18:17:12.5095256Z + lsmod 2024-05-19T18:17:12.5102593Z Module Size Used by 2024-05-19T18:17:12.5120194Z nvidia_modeset 1339392 0 2024-05-19T18:17:12.5120656Z veth 16384 0 2024-05-19T18:17:12.5121060Z nvidia_uvm 4599808 0 2024-05-19T18:17:12.5121546Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-05-19T18:17:12.5122077Z drm 421888 1 nvidia 2024-05-19T18:17:12.5122524Z i2c_core 77824 2 nvidia,drm 2024-05-19T18:17:12.5123024Z backlight 16384 1 nvidia_modeset 2024-05-19T18:17:12.5123490Z xt_conntrack 16384 1 2024-05-19T18:17:12.5123883Z ipt_MASQUERADE 16384 1 2024-05-19T18:17:12.5124560Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-05-19T18:17:12.5125023Z nf_conntrack_netlink 49152 0 2024-05-19T18:17:12.5125472Z nfnetlink 16384 2 nf_conntrack_netlink 2024-05-19T18:17:12.5125922Z xfrm_user 45056 1 2024-05-19T18:17:12.5126308Z xfrm_algo 16384 1 xfrm_user 2024-05-19T18:17:12.5126839Z iptable_nat 16384 1 2024-05-19T18:17:12.5127223Z nf_conntrack_ipv4 16384 3 2024-05-19T18:17:12.5127642Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-05-19T18:17:12.5128105Z nf_nat_ipv4 16384 1 iptable_nat 2024-05-19T18:17:12.5128626Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-05-19T18:17:12.5129566Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-05-19T18:17:12.5130405Z xt_addrtype 16384 2 2024-05-19T18:17:12.5130772Z iptable_filter 16384 1 2024-05-19T18:17:12.5131158Z br_netfilter 24576 0 2024-05-19T18:17:12.5131569Z bridge 172032 1 br_netfilter 2024-05-19T18:17:12.5132002Z stp 16384 1 bridge 2024-05-19T18:17:12.5132422Z llc 16384 2 bridge,stp 2024-05-19T18:17:12.5132826Z overlay 86016 0 2024-05-19T18:17:12.5133183Z sunrpc 393216 1 2024-05-19T18:17:12.5133536Z dm_mirror 28672 0 2024-05-19T18:17:12.5133930Z dm_region_hash 20480 1 dm_mirror 2024-05-19T18:17:12.5134416Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-05-19T18:17:12.5134924Z dm_mod 143360 2 dm_log,dm_mirror 2024-05-19T18:17:12.5135363Z dax 69632 1 dm_mod 2024-05-19T18:17:12.5135805Z crc32_pclmul 16384 0 2024-05-19T18:17:12.5136181Z ghash_clmulni_intel 16384 0 2024-05-19T18:17:12.5136562Z pcbc 16384 0 2024-05-19T18:17:12.5136923Z aesni_intel 188416 0 2024-05-19T18:17:12.5137323Z aes_x86_64 20480 1 aesni_intel 2024-05-19T18:17:12.5137771Z crypto_simd 16384 1 aesni_intel 2024-05-19T18:17:12.5138287Z mousedev 24576 0 2024-05-19T18:17:12.5138690Z glue_helper 16384 1 aesni_intel 2024-05-19T18:17:12.5139241Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-05-19T18:17:12.5139775Z evdev 20480 3 2024-05-19T18:17:12.5140132Z psmouse 32768 0 2024-05-19T18:17:12.5140501Z button 16384 0 2024-05-19T18:17:12.5140865Z ena 139264 0 2024-05-19T18:17:12.5141234Z ptp 20480 1 ena 2024-05-19T18:17:12.5141618Z pps_core 20480 1 ptp 2024-05-19T18:17:12.5142007Z crc32c_intel 24576 0 2024-05-19T18:17:12.5142374Z autofs4 49152 2 2024-05-19T18:17:12.5142732Z + modinfo nvidia 2024-05-19T18:17:12.5143394Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-05-19T18:17:12.5144092Z alias: char-major-195-* 2024-05-19T18:17:12.5144470Z version: 550.54.15 2024-05-19T18:17:12.5144812Z supported: external 2024-05-19T18:17:12.5145154Z license: NVIDIA 2024-05-19T18:17:12.5145532Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-05-19T18:17:12.5146014Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-05-19T18:17:12.5146461Z srcversion: 833721318DA517F0C2FEC97 2024-05-19T18:17:12.5146924Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-05-19T18:17:12.5147421Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-05-19T18:17:12.5147904Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-05-19T18:17:12.5148372Z depends: i2c-core,drm 2024-05-19T18:17:12.5148728Z retpoline: Y 2024-05-19T18:17:12.5149028Z name: nvidia 2024-05-19T18:17:12.5149573Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-05-19T18:17:12.5150603Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-05-19T18:17:12.5151402Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-05-19T18:17:12.5152047Z parm: NVreg_ResmanDebugLevel:int 2024-05-19T18:17:12.5152510Z parm: NVreg_RmLogonRC:int 2024-05-19T18:17:12.5152965Z parm: NVreg_ModifyDeviceFiles:int 2024-05-19T18:17:12.5153446Z parm: NVreg_DeviceFileUID:int 2024-05-19T18:17:12.5153902Z parm: NVreg_DeviceFileGID:int 2024-05-19T18:17:12.5154381Z parm: NVreg_DeviceFileMode:int 2024-05-19T18:17:12.5154938Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-05-19T18:17:12.5155541Z parm: NVreg_UsePageAttributeTable:int 2024-05-19T18:17:12.5156050Z parm: NVreg_EnablePCIeGen3:int 2024-05-19T18:17:12.5156500Z parm: NVreg_EnableMSI:int 2024-05-19T18:17:12.5156930Z parm: NVreg_TCEBypassMode:int 2024-05-19T18:17:12.5157415Z parm: NVreg_EnableStreamMemOPs:int 2024-05-19T18:17:12.5157975Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-05-19T18:17:12.5158606Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-05-19T18:17:12.5159207Z parm: NVreg_EnableS0ixPowerManagement:int 2024-05-19T18:17:12.5159860Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-05-19T18:17:12.5160496Z parm: NVreg_DynamicPowerManagement:int 2024-05-19T18:17:12.5161096Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-05-19T18:17:12.5161681Z parm: NVreg_EnableGpuFirmware:int 2024-05-19T18:17:12.5162170Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-05-19T18:17:12.5162706Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-05-19T18:17:12.5163243Z parm: NVreg_EnableUserNUMAManagement:int 2024-05-19T18:17:12.5163738Z parm: NVreg_MemoryPoolSize:int 2024-05-19T18:17:12.5164200Z parm: NVreg_KMallocHeapMaxSize:int 2024-05-19T18:17:12.5164686Z parm: NVreg_VMallocHeapMaxSize:int 2024-05-19T18:17:12.5165151Z parm: NVreg_IgnoreMMIOCheck:int 2024-05-19T18:17:12.5165645Z parm: NVreg_NvLinkDisable:int 2024-05-19T18:17:12.5166221Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-05-19T18:17:12.5166835Z parm: NVreg_RegisterPCIDriver:int 2024-05-19T18:17:12.5167308Z parm: NVreg_EnableResizableBar:int 2024-05-19T18:17:12.5167792Z parm: NVreg_EnableDbgBreakpoint:int 2024-05-19T18:17:12.5168291Z parm: NVreg_EnableNonblockingOpen:int 2024-05-19T18:17:12.5168771Z parm: NVreg_RegistryDwords:charp 2024-05-19T18:17:12.5169261Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-05-19T18:17:12.5169739Z parm: NVreg_RmMsg:charp 2024-05-19T18:17:12.5170146Z parm: NVreg_GpuBlacklist:charp 2024-05-19T18:17:12.5170611Z parm: NVreg_TemporaryFilePath:charp 2024-05-19T18:17:12.5171071Z parm: NVreg_ExcludedGpus:charp 2024-05-19T18:17:12.5171519Z parm: NVreg_DmaRemapPeerMmio:int 2024-05-19T18:17:12.5171998Z parm: NVreg_RmNvlinkBandwidth:charp 2024-05-19T18:17:12.5172473Z parm: NVreg_ImexChannelCount:int 2024-05-19T18:17:12.5172923Z parm: rm_firmware_active:charp 2024-05-19T18:17:12.5173319Z + set +e 2024-05-19T18:17:12.5173621Z + nvidia-smi 2024-05-19T18:17:14.0827851Z Sun May 19 18:17:14 2024 2024-05-19T18:17:14.0828760Z +-----------------------------------------------------------------------------------------+ 2024-05-19T18:17:14.0829686Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-05-19T18:17:14.0830704Z |-----------------------------------------+------------------------+----------------------+ 2024-05-19T18:17:14.0831506Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-05-19T18:17:14.0832387Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-05-19T18:17:14.0833077Z | | | MIG M. | 2024-05-19T18:17:14.0833920Z |=========================================+========================+======================| 2024-05-19T18:17:14.0956103Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-05-19T18:17:14.0957243Z | 0% 29C P0 59W / 300W | 0MiB / 23028MiB | 0% Default | 2024-05-19T18:17:14.0957996Z | | | N/A | 2024-05-19T18:17:14.0958671Z +-----------------------------------------+------------------------+----------------------+ 2024-05-19T18:17:14.0959258Z 2024-05-19T18:17:14.0960045Z +-----------------------------------------------------------------------------------------+ 2024-05-19T18:17:14.0960780Z | Processes: | 2024-05-19T18:17:14.0961509Z | GPU GI CI PID Type Process name GPU Memory | 2024-05-19T18:17:14.0962221Z | ID ID Usage | 2024-05-19T18:17:14.0962791Z |=========================================================================================| 2024-05-19T18:17:14.0963441Z | No running processes found | 2024-05-19T18:17:14.0964196Z +-----------------------------------------------------------------------------------------+ 2024-05-19T18:17:14.7223204Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-05-19T18:17:16.2867544Z NVIDIA A10G 2024-05-19T18:17:16.7262697Z + NVIDIA_SMI_STATUS=0 2024-05-19T18:17:16.7263405Z + '[' 0 -eq 0 ']' 2024-05-19T18:17:16.7263970Z + echo 'INFO: Ignoring allowed status 0' 2024-05-19T18:17:16.7264413Z + set -e 2024-05-19T18:17:16.7264717Z INFO: Ignoring allowed status 0 2024-05-19T18:17:16.7266894Z == Installing nvidia container toolkit for amzn2 == 2024-05-19T18:17:16.7269216Z + sudo yum install -y yum-utils 2024-05-19T18:17:17.0284683Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:17:18.4953324Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-05-19T18:17:18.4954034Z Nothing to do 2024-05-19T18:17:18.6439096Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-05-19T18:17:18.9495859Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:17:18.9757599Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-05-19T18:17:18.9759835Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-05-19T18:17:18.9760800Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-05-19T18:17:18.9872048Z + sudo yum install -y nvidia-docker2 2024-05-19T18:17:19.2724128Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:17:20.7179818Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-05-19T18:17:20.7180625Z Nothing to do 2024-05-19T18:17:20.8643647Z + sudo systemctl restart docker 2024-05-19T18:18:14.1806886Z Command completed after 1 attempt(s). 2024-05-19T18:18:14.1863174Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-05-19T18:18:14.1863955Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-05-19T18:18:14.1864651Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-05-19T18:18:14.1865304Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:18:14.1873449Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:18:14.1873950Z env: 2024-05-19T18:18:14.1874234Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:14.1874685Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:18:14.1875342Z ##[endgroup] 2024-05-19T18:18:14.4066889Z Defaulting to user installation because normal site-packages is not writeable 2024-05-19T18:18:14.4224168Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-05-19T18:18:14.4324031Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-05-19T18:18:14.5255824Z Prepare all required actions 2024-05-19T18:18:14.5256306Z Getting action download info 2024-05-19T18:18:14.6343393Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-05-19T18:18:14.8060422Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-05-19T18:18:14.9323283Z ##[group]Run ./.github/actions/download-build-artifacts 2024-05-19T18:18:14.9323759Z with: 2024-05-19T18:18:14.9324094Z name: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:18:14.9324562Z s3-bucket: gha-artifacts 2024-05-19T18:18:14.9324908Z env: 2024-05-19T18:18:14.9325179Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:14.9325630Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:18:14.9326113Z ##[endgroup] 2024-05-19T18:18:14.9357999Z ##[group]Run seemethere/download-artifact-s3@v4 2024-05-19T18:18:14.9358457Z with: 2024-05-19T18:18:14.9358825Z name: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:18:14.9359267Z s3-bucket: gha-artifacts 2024-05-19T18:18:14.9359622Z region: us-east-1 2024-05-19T18:18:14.9359920Z env: 2024-05-19T18:18:14.9360201Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:14.9360643Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:18:14.9361123Z ##[endgroup] 2024-05-19T18:18:15.3487032Z (node:15018) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-05-19T18:18:15.3487833Z 2024-05-19T18:18:15.3488100Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-05-19T18:18:15.3488842Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-05-19T18:18:15.3489776Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-05-19T18:18:15.4351143Z Found 1 objects with prefix pytorch/pytorch/9148947323/linux-focal-cuda12.4-py3.10-gcc9-sm86/ 2024-05-19T18:18:15.4352279Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-05-19T18:18:22.9122266Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-05-19T18:18:22.9128143Z Artifact download has finished successfully 2024-05-19T18:18:22.9283952Z ##[group]Run unzip -o artifacts.zip 2024-05-19T18:18:22.9284395Z unzip -o artifacts.zip 2024-05-19T18:18:22.9292207Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:18:22.9292718Z env: 2024-05-19T18:18:22.9293003Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:22.9293466Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:18:22.9293961Z ##[endgroup] 2024-05-19T18:18:22.9324255Z Archive: artifacts.zip 2024-05-19T18:18:22.9325823Z creating: dist/ 2024-05-19T18:18:24.8929560Z inflating: dist/torch-2.4.0a0+gitc8c7ddf-cp310-cp310-linux_x86_64.whl 2024-05-19T18:18:24.8930263Z creating: build/custom_test_artifacts/ 2024-05-19T18:18:24.8930850Z creating: build/custom_test_artifacts/custom-op-build/ 2024-05-19T18:18:24.8931581Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-05-19T18:18:24.8932405Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-05-19T18:18:24.8936349Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-05-19T18:18:24.8937295Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-05-19T18:18:24.8938224Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-05-19T18:18:24.8939188Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-05-19T18:18:24.8940480Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-05-19T18:18:24.8941581Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-05-19T18:18:24.8942791Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-05-19T18:18:24.8943782Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-05-19T18:18:24.8944773Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-05-19T18:18:24.8945916Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-05-19T18:18:24.8947062Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-05-19T18:18:24.8948184Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-05-19T18:18:24.8949304Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-05-19T18:18:24.8950832Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-05-19T18:18:24.8952011Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-05-19T18:18:24.8953043Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-05-19T18:18:24.8954046Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-05-19T18:18:24.8988924Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-05-19T18:18:24.9029054Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-05-19T18:18:24.9032227Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-05-19T18:18:24.9073312Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-05-19T18:18:24.9074727Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-05-19T18:18:24.9076145Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-05-19T18:18:24.9077606Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-05-19T18:18:24.9079028Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-05-19T18:18:24.9080405Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-05-19T18:18:24.9081800Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-05-19T18:18:24.9083181Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-05-19T18:18:24.9084525Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-05-19T18:18:24.9085804Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-05-19T18:18:24.9087155Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-05-19T18:18:24.9088388Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-05-19T18:18:24.9089626Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-05-19T18:18:24.9090935Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-05-19T18:18:24.9092159Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-05-19T18:18:24.9154287Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-05-19T18:18:24.9224194Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-05-19T18:18:24.9225433Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-05-19T18:18:24.9226419Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-05-19T18:18:24.9227288Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-05-19T18:18:24.9228185Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-05-19T18:18:24.9229125Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-05-19T18:18:24.9230333Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-05-19T18:18:24.9231499Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-05-19T18:18:24.9232605Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-05-19T18:18:24.9233639Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-05-19T18:18:24.9234714Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-05-19T18:18:24.9235789Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-05-19T18:18:24.9236864Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-05-19T18:18:24.9237941Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-05-19T18:18:24.9238995Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-05-19T18:18:24.9249074Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-05-19T18:18:24.9376729Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-05-19T18:18:24.9377734Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-05-19T18:18:24.9378826Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-05-19T18:18:24.9380044Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-05-19T18:18:24.9381204Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-05-19T18:18:24.9382306Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-05-19T18:18:24.9383442Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-05-19T18:18:24.9384625Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-05-19T18:18:24.9385782Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-05-19T18:18:24.9386908Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-05-19T18:18:24.9388032Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-05-19T18:18:24.9400310Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-05-19T18:18:24.9479194Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-05-19T18:18:24.9480506Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-05-19T18:18:24.9481577Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-05-19T18:18:24.9482614Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-05-19T18:18:24.9483537Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-05-19T18:18:24.9484449Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-05-19T18:18:24.9485412Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-05-19T18:18:24.9486265Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-05-19T18:18:24.9487166Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-05-19T18:18:24.9487992Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-05-19T18:18:24.9593031Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-05-19T18:18:24.9653144Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-05-19T18:18:24.9653892Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-05-19T18:18:24.9654600Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-05-19T18:18:24.9655421Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-05-19T18:18:24.9660566Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-05-19T18:18:24.9661484Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-05-19T18:18:24.9662429Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-05-19T18:18:24.9663417Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-05-19T18:18:24.9664391Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-05-19T18:18:24.9665542Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-05-19T18:18:24.9666647Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-05-19T18:18:24.9667658Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-05-19T18:18:24.9668660Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-05-19T18:18:24.9669814Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-05-19T18:18:24.9671105Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-05-19T18:18:24.9672243Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-05-19T18:18:24.9673377Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-05-19T18:18:24.9674533Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-05-19T18:18:24.9675688Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-05-19T18:18:24.9676714Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-05-19T18:18:24.9677712Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-05-19T18:18:24.9713067Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-05-19T18:18:24.9753129Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-05-19T18:18:24.9755228Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-05-19T18:18:24.9797475Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-05-19T18:18:24.9798868Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-05-19T18:18:24.9800384Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-05-19T18:18:24.9801850Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-05-19T18:18:24.9803251Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-05-19T18:18:24.9804637Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-05-19T18:18:24.9806040Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-05-19T18:18:24.9807536Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-05-19T18:18:24.9808878Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-05-19T18:18:24.9810147Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-05-19T18:18:24.9811373Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-05-19T18:18:24.9812572Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-05-19T18:18:24.9813794Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-05-19T18:18:24.9814978Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-05-19T18:18:24.9816187Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-05-19T18:18:24.9877083Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-05-19T18:18:24.9947882Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-05-19T18:18:24.9949064Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-05-19T18:18:24.9950236Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-05-19T18:18:24.9951104Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-05-19T18:18:24.9951997Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-05-19T18:18:24.9952945Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-05-19T18:18:24.9954022Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-05-19T18:18:24.9955257Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-05-19T18:18:24.9956408Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-05-19T18:18:24.9957481Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-05-19T18:18:24.9958580Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-05-19T18:18:24.9959691Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-05-19T18:18:24.9960801Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-05-19T18:18:24.9962023Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-05-19T18:18:24.9963107Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-05-19T18:18:24.9973551Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-05-19T18:18:25.0034989Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-05-19T18:18:25.0036174Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-05-19T18:18:25.0037239Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-05-19T18:18:25.0038209Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-05-19T18:18:25.0039113Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-05-19T18:18:25.0040008Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-05-19T18:18:25.0040901Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-05-19T18:18:25.0041741Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-05-19T18:18:25.0042510Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-05-19T18:18:25.0043317Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-05-19T18:18:25.0088297Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-05-19T18:18:25.0089060Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-05-19T18:18:25.0089817Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-05-19T18:18:25.0090703Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-05-19T18:18:25.0095631Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-05-19T18:18:25.0096629Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-05-19T18:18:25.0097629Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-05-19T18:18:25.0098678Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-05-19T18:18:25.0099724Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-05-19T18:18:25.0100894Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-05-19T18:18:25.0102073Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-05-19T18:18:25.0103136Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-05-19T18:18:25.0104182Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-05-19T18:18:25.0105402Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-05-19T18:18:25.0106612Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-05-19T18:18:25.0107797Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-05-19T18:18:25.0109011Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-05-19T18:18:25.0110433Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-05-19T18:18:25.0111661Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-05-19T18:18:25.0112753Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-05-19T18:18:25.0113817Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-05-19T18:18:25.0148004Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-05-19T18:18:25.0187138Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-05-19T18:18:25.0188720Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-05-19T18:18:25.0233449Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-05-19T18:18:25.0235502Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-05-19T18:18:25.0236994Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-05-19T18:18:25.0238517Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-05-19T18:18:25.0239983Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-05-19T18:18:25.0253328Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-05-19T18:18:25.0254943Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-05-19T18:18:25.0256379Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-05-19T18:18:25.0257765Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-05-19T18:18:25.0259107Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-05-19T18:18:25.0260406Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-05-19T18:18:25.0261666Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-05-19T18:18:25.0262939Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-05-19T18:18:25.0264169Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-05-19T18:18:25.0265479Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-05-19T18:18:25.0330462Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-05-19T18:18:25.0401419Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-05-19T18:18:25.0402681Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-05-19T18:18:25.0403721Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-05-19T18:18:25.0404632Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-05-19T18:18:25.0405588Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-05-19T18:18:25.0406676Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-05-19T18:18:25.0407807Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-05-19T18:18:25.0409041Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-05-19T18:18:25.0410240Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-05-19T18:18:25.0411542Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-05-19T18:18:25.0412704Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-05-19T18:18:25.0413941Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-05-19T18:18:25.0415113Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-05-19T18:18:25.0416280Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-05-19T18:18:25.0417443Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-05-19T18:18:25.0418686Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-05-19T18:18:25.0531626Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-05-19T18:18:25.0532807Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-05-19T18:18:25.0533991Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-05-19T18:18:25.0535338Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-05-19T18:18:25.0536586Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-05-19T18:18:25.0537771Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-05-19T18:18:25.0538986Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-05-19T18:18:25.0540215Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-05-19T18:18:25.0541452Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-05-19T18:18:25.0542676Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-05-19T18:18:25.0543892Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-05-19T18:18:25.0555366Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-05-19T18:18:25.0608119Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-05-19T18:18:25.0609400Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-05-19T18:18:25.0610534Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-05-19T18:18:25.0611554Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-05-19T18:18:25.0612494Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-05-19T18:18:25.0613447Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-05-19T18:18:25.0614392Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-05-19T18:18:25.0615293Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-05-19T18:18:25.0616119Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-05-19T18:18:25.0616959Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-05-19T18:18:25.0714001Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-05-19T18:18:25.0754930Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-05-19T18:18:25.0755686Z creating: build/lib/ 2024-05-19T18:18:25.0840530Z inflating: build/lib/libprotobuf-lite.a 2024-05-19T18:18:25.1278860Z inflating: build/lib/libprotobuf.a 2024-05-19T18:18:25.1288076Z inflating: build/lib/libpthreadpool.a 2024-05-19T18:18:25.1295737Z inflating: build/lib/libcpuinfo.a 2024-05-19T18:18:25.1303137Z inflating: build/lib/libcpuinfo_internals.a 2024-05-19T18:18:25.1303750Z inflating: build/lib/libclog.a 2024-05-19T18:18:25.1321372Z inflating: build/lib/libnnpack.a 2024-05-19T18:18:25.1323318Z inflating: build/lib/libnnpack_reference_layers.a 2024-05-19T18:18:25.1385766Z inflating: build/lib/libgtest.a 2024-05-19T18:18:25.1457229Z inflating: build/lib/libbenchmark.a 2024-05-19T18:18:25.1518755Z inflating: build/lib/libasmjit.a 2024-05-19T18:18:25.1525769Z inflating: build/lib/libittnotify.a 2024-05-19T18:18:25.1552984Z inflating: build/lib/libtensorpipe_uv.a 2024-05-19T18:18:25.1674725Z inflating: build/lib/libgloo.a 2024-05-19T18:18:25.1695399Z inflating: build/lib/libfmt.a 2024-05-19T18:18:25.1782689Z inflating: build/lib/libc10.so 2024-05-19T18:18:25.1784159Z inflating: build/lib/libcaffe2_nvrtc.so 2024-05-19T18:18:25.1784866Z inflating: build/lib/libfoxi_loader.a 2024-05-19T18:18:25.1786125Z inflating: build/lib/libtorch_global_deps.so 2024-05-19T18:18:25.1799176Z inflating: build/lib/libqnnpack.a 2024-05-19T18:18:25.2283421Z inflating: build/lib/libprotoc.a 2024-05-19T18:18:25.2302092Z inflating: build/lib/libpytorch_qnnpack.a 2024-05-19T18:18:25.2319364Z inflating: build/lib/libgmock.a 2024-05-19T18:18:25.2319825Z inflating: build/lib/libgtest_main.a 2024-05-19T18:18:25.2320646Z inflating: build/lib/libbenchmark_main.a 2024-05-19T18:18:25.2689888Z inflating: build/lib/libgloo_cuda.a 2024-05-19T18:18:26.2416591Z inflating: build/lib/libdnnl.a 2024-05-19T18:18:26.2966736Z inflating: build/lib/libtensorpipe.a 2024-05-19T18:18:26.3019099Z inflating: build/lib/libc10_cuda.so 2024-05-19T18:18:26.3019933Z inflating: build/lib/libgmock_main.a 2024-05-19T18:18:26.4222788Z inflating: build/lib/libfbgemm.a 2024-05-19T18:18:26.4467930Z inflating: build/lib/libtensorpipe_cuda.a 2024-05-19T18:18:26.4963899Z inflating: build/lib/libkineto.a 2024-05-19T18:18:26.5001100Z inflating: build/lib/libcaffe2_protos.a 2024-05-19T18:18:26.5041600Z inflating: build/lib/libonnx_proto.a 2024-05-19T18:18:26.5219260Z inflating: build/lib/libXNNPACK.a 2024-05-19T18:18:26.5892910Z inflating: build/lib/libonnx.a 2024-05-19T18:18:28.8892104Z inflating: build/lib/libtorch_cpu.so 2024-05-19T18:18:28.8895714Z inflating: build/lib/libunbox_lib.a 2024-05-19T18:18:28.8898975Z inflating: build/lib/libshm.so 2024-05-19T18:18:30.8581688Z inflating: build/lib/libtorch_cuda.so 2024-05-19T18:18:30.8582264Z inflating: build/lib/libtorch.so 2024-05-19T18:18:30.8584345Z inflating: build/lib/libc10d_cuda_test.so 2024-05-19T18:18:31.7098578Z inflating: build/lib/libtorch_cuda_linalg.so 2024-05-19T18:18:31.8946141Z inflating: build/lib/libtorch_python.so 2024-05-19T18:18:31.9013196Z inflating: build/lib/libtorchbind_test.so 2024-05-19T18:18:31.9032482Z inflating: build/lib/libjitbackend_test.so 2024-05-19T18:18:31.9057615Z inflating: build/lib/libbackend_with_compiler.so 2024-05-19T18:18:31.9090229Z inflating: build/lib/libnnapi_backend.so 2024-05-19T18:18:31.9090827Z creating: build/bin/ 2024-05-19T18:18:31.9139002Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-05-19T18:18:31.9188736Z inflating: build/bin/c10_DeviceGuard_test 2024-05-19T18:18:31.9237642Z inflating: build/bin/c10_Device_test 2024-05-19T18:18:31.9294765Z inflating: build/bin/c10_DispatchKeySet_test 2024-05-19T18:18:31.9346631Z inflating: build/bin/c10_Scalar_test 2024-05-19T18:18:31.9394634Z inflating: build/bin/c10_StreamGuard_test 2024-05-19T18:18:31.9443381Z inflating: build/bin/c10_SymInt_test 2024-05-19T18:18:31.9496240Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-05-19T18:18:31.9550218Z inflating: build/bin/c10_InlineStreamGuard_test 2024-05-19T18:18:31.9604504Z inflating: build/bin/c10_SizesAndStrides_test 2024-05-19T18:18:31.9673787Z inflating: build/bin/c10_cow_test 2024-05-19T18:18:31.9724959Z inflating: build/bin/c10_Bitset_test 2024-05-19T18:18:31.9772997Z inflating: build/bin/c10_ConstexprCrc_test 2024-05-19T18:18:31.9820415Z inflating: build/bin/c10_DeadlockDetection_test 2024-05-19T18:18:31.9869469Z inflating: build/bin/c10_Half_test 2024-05-19T18:18:31.9923855Z inflating: build/bin/c10_LeftRight_test 2024-05-19T18:18:31.9977167Z inflating: build/bin/c10_Metaprogramming_test 2024-05-19T18:18:32.0025332Z inflating: build/bin/c10_Synchronized_test 2024-05-19T18:18:32.0079550Z inflating: build/bin/c10_ThreadLocal_test 2024-05-19T18:18:32.0129620Z inflating: build/bin/c10_TypeIndex_test 2024-05-19T18:18:32.0179025Z inflating: build/bin/c10_TypeList_test 2024-05-19T18:18:32.0226266Z inflating: build/bin/c10_TypeTraits_test 2024-05-19T18:18:32.0277299Z inflating: build/bin/c10_accumulate_test 2024-05-19T18:18:32.0330270Z inflating: build/bin/c10_bfloat16_test 2024-05-19T18:18:32.0379136Z inflating: build/bin/c10_bit_cast_test 2024-05-19T18:18:32.0433778Z inflating: build/bin/c10_complex_math_test 2024-05-19T18:18:32.0487761Z inflating: build/bin/c10_complex_test 2024-05-19T18:18:32.0538759Z inflating: build/bin/c10_exception_test 2024-05-19T18:18:32.0587427Z inflating: build/bin/c10_flags_test 2024-05-19T18:18:32.0635719Z inflating: build/bin/c10_generic_math_test 2024-05-19T18:18:32.0794207Z inflating: build/bin/c10_intrusive_ptr_test 2024-05-19T18:18:32.0843444Z inflating: build/bin/c10_irange_test 2024-05-19T18:18:32.0895367Z inflating: build/bin/c10_lazy_test 2024-05-19T18:18:32.0950758Z inflating: build/bin/c10_logging_test 2024-05-19T18:18:32.1023238Z inflating: build/bin/c10_optional_test 2024-05-19T18:18:32.1083360Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-05-19T18:18:32.1135499Z inflating: build/bin/c10_registry_test 2024-05-19T18:18:32.1281166Z inflating: build/bin/c10_small_vector_test 2024-05-19T18:18:32.1330962Z inflating: build/bin/c10_ssize_test 2024-05-19T18:18:32.1381185Z inflating: build/bin/c10_string_util_test 2024-05-19T18:18:32.1438521Z inflating: build/bin/c10_string_view_test 2024-05-19T18:18:32.1486710Z inflating: build/bin/c10_tempfile_test 2024-05-19T18:18:32.1540952Z inflating: build/bin/c10_typeid_test 2024-05-19T18:18:32.1587935Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-05-19T18:18:32.2020659Z inflating: build/bin/protoc-3.13.0.0 2024-05-19T18:18:32.2453126Z inflating: build/bin/protoc 2024-05-19T18:18:32.2503927Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-05-19T18:18:32.2555912Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-05-19T18:18:32.2606186Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-05-19T18:18:32.2656426Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-05-19T18:18:32.2707482Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-05-19T18:18:32.2759560Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-05-19T18:18:32.2806097Z inflating: build/bin/c10_cuda_CUDATest 2024-05-19T18:18:32.2858152Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-05-19T18:18:32.3175651Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-05-19T18:18:32.3507240Z inflating: build/bin/vec_test_all_types_AVX512 2024-05-19T18:18:32.3853993Z inflating: build/bin/vec_test_all_types_AVX2 2024-05-19T18:18:32.3904404Z inflating: build/bin/FileStoreTest 2024-05-19T18:18:32.3959048Z inflating: build/bin/TCPStoreTest 2024-05-19T18:18:32.4010016Z inflating: build/bin/HashStoreTest 2024-05-19T18:18:32.4022956Z inflating: build/bin/ProcessGroupMPITest 2024-05-19T18:18:32.4075217Z inflating: build/bin/test_edge_op_registration 2024-05-19T18:18:32.4079092Z inflating: build/bin/torch_shm_manager 2024-05-19T18:18:32.4081758Z inflating: build/bin/example_allreduce 2024-05-19T18:18:32.4135022Z inflating: build/bin/test_dist_autograd 2024-05-19T18:18:32.4202133Z inflating: build/bin/test_cpp_rpc 2024-05-19T18:18:32.4204518Z inflating: build/bin/parallel_benchmark 2024-05-19T18:18:32.4269727Z inflating: build/bin/test_mobile_nnc 2024-05-19T18:18:32.4278925Z inflating: build/bin/aot_model_compiler_test 2024-05-19T18:18:32.4611450Z inflating: build/bin/test_lazy 2024-05-19T18:18:32.5756698Z inflating: build/bin/test_api 2024-05-19T18:18:32.5828095Z inflating: build/bin/Dict_test 2024-05-19T18:18:32.5879148Z inflating: build/bin/Dimname_test 2024-05-19T18:18:32.5941158Z inflating: build/bin/MaybeOwned_test 2024-05-19T18:18:32.5996764Z inflating: build/bin/NamedTensor_test 2024-05-19T18:18:32.6054775Z inflating: build/bin/apply_utils_test 2024-05-19T18:18:32.6111905Z inflating: build/bin/atest 2024-05-19T18:18:32.6172779Z inflating: build/bin/basic 2024-05-19T18:18:32.6225321Z inflating: build/bin/broadcast_test 2024-05-19T18:18:32.6274505Z inflating: build/bin/cpu_allocator_test 2024-05-19T18:18:32.6331528Z inflating: build/bin/cpu_generator_test 2024-05-19T18:18:32.6382402Z inflating: build/bin/cpu_profiling_allocator_test 2024-05-19T18:18:32.6473027Z inflating: build/bin/cpu_rng_test 2024-05-19T18:18:32.6522204Z inflating: build/bin/dispatch_key_set_test 2024-05-19T18:18:32.6571593Z inflating: build/bin/dlconvertor_test 2024-05-19T18:18:32.6627531Z inflating: build/bin/extension_backend_test 2024-05-19T18:18:32.6681088Z inflating: build/bin/half_test 2024-05-19T18:18:32.6774708Z inflating: build/bin/ivalue_test 2024-05-19T18:18:32.6822500Z inflating: build/bin/lazy_tensor_test 2024-05-19T18:18:32.6875427Z inflating: build/bin/math_kernel_test 2024-05-19T18:18:32.6928136Z inflating: build/bin/memory_format_test 2024-05-19T18:18:32.6979377Z inflating: build/bin/memory_overlapping_test 2024-05-19T18:18:32.7030798Z inflating: build/bin/mobile_memory_cleanup 2024-05-19T18:18:32.7084889Z inflating: build/bin/native_test 2024-05-19T18:18:32.7133831Z inflating: build/bin/operator_name_test 2024-05-19T18:18:32.7183876Z inflating: build/bin/operators_test 2024-05-19T18:18:32.7234869Z inflating: build/bin/packedtensoraccessor_test 2024-05-19T18:18:32.7299336Z inflating: build/bin/pow_test 2024-05-19T18:18:32.7355044Z inflating: build/bin/quantized_test 2024-05-19T18:18:32.7403112Z inflating: build/bin/reduce_ops_test 2024-05-19T18:18:32.7452541Z inflating: build/bin/reportMemoryUsage_test 2024-05-19T18:18:32.7507184Z inflating: build/bin/scalar_tensor_test 2024-05-19T18:18:32.7563975Z inflating: build/bin/scalar_test 2024-05-19T18:18:32.7614237Z inflating: build/bin/StorageUtils_test 2024-05-19T18:18:32.7665235Z inflating: build/bin/stride_properties_test 2024-05-19T18:18:32.7741636Z inflating: build/bin/tensor_iterator_test 2024-05-19T18:18:32.7794546Z inflating: build/bin/test_parallel 2024-05-19T18:18:32.7797225Z inflating: build/bin/thread_init_test 2024-05-19T18:18:32.7850883Z inflating: build/bin/type_ptr_test 2024-05-19T18:18:32.7908903Z inflating: build/bin/type_test 2024-05-19T18:18:32.7959557Z inflating: build/bin/undefined_tensor_test 2024-05-19T18:18:32.7960939Z inflating: build/bin/verify_api_visibility 2024-05-19T18:18:32.8028103Z inflating: build/bin/legacy_vmap_test 2024-05-19T18:18:32.8078563Z inflating: build/bin/weakref_test 2024-05-19T18:18:32.8128069Z inflating: build/bin/wrapdim_test 2024-05-19T18:18:32.8178051Z inflating: build/bin/xla_tensor_test 2024-05-19T18:18:32.8236581Z inflating: build/bin/IListRef_test 2024-05-19T18:18:32.8339234Z inflating: build/bin/List_test 2024-05-19T18:18:32.8404008Z inflating: build/bin/KernelFunction_test 2024-05-19T18:18:32.8522310Z inflating: build/bin/kernel_function_legacy_test 2024-05-19T18:18:32.8616085Z inflating: build/bin/kernel_function_test 2024-05-19T18:18:32.8740244Z inflating: build/bin/kernel_lambda_legacy_test 2024-05-19T18:18:32.8840931Z inflating: build/bin/kernel_lambda_test 2024-05-19T18:18:32.8900132Z inflating: build/bin/kernel_stackbased_test 2024-05-19T18:18:32.8993095Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-05-19T18:18:32.9042458Z inflating: build/bin/CppSignature_test 2024-05-19T18:18:32.9089893Z inflating: build/bin/op_allowlist_test 2024-05-19T18:18:32.9144000Z inflating: build/bin/backend_fallback_test 2024-05-19T18:18:32.9436650Z inflating: build/bin/op_registration_test 2024-05-19T18:18:32.9497730Z inflating: build/bin/inline_container_test 2024-05-19T18:18:32.9548942Z inflating: build/bin/cuda_apply_test 2024-05-19T18:18:32.9598846Z inflating: build/bin/cuda_allocator_test 2024-05-19T18:18:32.9651813Z inflating: build/bin/cuda_caching_host_allocator_test 2024-05-19T18:18:32.9708539Z inflating: build/bin/cuda_atomic_ops_test 2024-05-19T18:18:32.9777034Z inflating: build/bin/cuda_complex_math_test 2024-05-19T18:18:32.9834357Z inflating: build/bin/cuda_complex_test 2024-05-19T18:18:32.9882077Z inflating: build/bin/cuda_device_test 2024-05-19T18:18:32.9939196Z inflating: build/bin/cuda_cub_test 2024-05-19T18:18:32.9987802Z inflating: build/bin/cuda_dlconvertor_test 2024-05-19T18:18:33.0050597Z inflating: build/bin/cuda_distributions_test 2024-05-19T18:18:33.0105436Z inflating: build/bin/cuda_generator_test 2024-05-19T18:18:33.0154252Z inflating: build/bin/cuda_half_test 2024-05-19T18:18:33.0203315Z inflating: build/bin/cuda_integer_divider_test 2024-05-19T18:18:33.0251706Z inflating: build/bin/cuda_optional_test 2024-05-19T18:18:33.0301648Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-05-19T18:18:33.0352691Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-05-19T18:18:33.0400970Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-05-19T18:18:33.0460269Z inflating: build/bin/cuda_stream_test 2024-05-19T18:18:33.0508112Z inflating: build/bin/cuda_cudnn_test 2024-05-19T18:18:33.0558879Z inflating: build/bin/cuda_vectorized_test 2024-05-19T18:18:33.0573398Z inflating: build/bin/tutorial_tensorexpr 2024-05-19T18:18:33.0637641Z inflating: build/bin/ProcessGroupGlooTest 2024-05-19T18:18:33.0692711Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-05-19T18:18:33.0753596Z inflating: build/bin/ProcessGroupNCCLTest 2024-05-19T18:18:33.0813576Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-05-19T18:18:33.1617350Z inflating: build/bin/test_tensorexpr 2024-05-19T18:18:33.2175623Z inflating: build/bin/test_jit 2024-05-19T18:18:33.2176085Z creating: .additional_ci_files/ 2024-05-19T18:18:33.2216412Z inflating: .additional_ci_files/test-times.json 2024-05-19T18:18:33.2377814Z inflating: .additional_ci_files/test-class-times.json 2024-05-19T18:18:33.2407844Z ##[group]Run rm artifacts.zip 2024-05-19T18:18:33.2408236Z rm artifacts.zip 2024-05-19T18:18:33.2415830Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:18:33.2416330Z env: 2024-05-19T18:18:33.2416599Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:33.2417053Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:18:33.2417537Z ##[endgroup] 2024-05-19T18:18:33.3124506Z ##[group]Run df -H 2024-05-19T18:18:33.3124828Z df -H 2024-05-19T18:18:33.3132439Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:18:33.3132946Z env: 2024-05-19T18:18:33.3133220Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:33.3133842Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:18:33.3134330Z ##[endgroup] 2024-05-19T18:18:33.3159749Z Filesystem Size Used Avail Use% Mounted on 2024-05-19T18:18:33.3160382Z devtmpfs 34G 0 34G 0% /dev 2024-05-19T18:18:33.3161131Z tmpfs 34G 0 34G 0% /dev/shm 2024-05-19T18:18:33.3161763Z tmpfs 34G 512k 34G 1% /run 2024-05-19T18:18:33.3162497Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-05-19T18:18:33.3163001Z /dev/nvme0n1p1 162G 44G 118G 27% / 2024-05-19T18:18:33.3163463Z tmpfs 6.7G 0 6.7G 0% /run/user/0 2024-05-19T18:18:33.3201325Z Prepare all required actions 2024-05-19T18:18:33.3201764Z Getting action download info 2024-05-19T18:18:33.4230528Z ##[group]Run ./.github/actions/download-td-artifacts 2024-05-19T18:18:33.4231003Z with: 2024-05-19T18:18:33.4231264Z env: 2024-05-19T18:18:33.4231543Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:33.4231998Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:18:33.4232489Z ##[endgroup] 2024-05-19T18:18:33.4263971Z ##[group]Run seemethere/download-artifact-s3@v4 2024-05-19T18:18:33.4264429Z with: 2024-05-19T18:18:33.4264695Z name: td_results 2024-05-19T18:18:33.4265018Z s3-bucket: gha-artifacts 2024-05-19T18:18:33.4265375Z region: us-east-1 2024-05-19T18:18:33.4265678Z env: 2024-05-19T18:18:33.4265947Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:33.4266412Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:18:33.4266899Z ##[endgroup] 2024-05-19T18:18:33.8305425Z (node:15039) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-05-19T18:18:33.8306135Z 2024-05-19T18:18:33.8306400Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-05-19T18:18:33.8307122Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-05-19T18:18:33.8308000Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-05-19T18:18:33.9144725Z Found 0 objects with prefix pytorch/pytorch/9148947323/td_results/ 2024-05-19T18:18:33.9151360Z Artifact download has finished successfully 2024-05-19T18:18:33.9276336Z ##[group]Run mkdir -p .additional_ci_files 2024-05-19T18:18:33.9276837Z mkdir -p .additional_ci_files 2024-05-19T18:18:33.9277408Z mv td_results.json .additional_ci_files/td_results.json 2024-05-19T18:18:33.9285298Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:18:33.9285797Z env: 2024-05-19T18:18:33.9286088Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:33.9286778Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:18:33.9287281Z ##[endgroup] 2024-05-19T18:18:33.9320720Z mv: cannot stat 'td_results.json': No such file or directory 2024-05-19T18:18:33.9325756Z ##[error]Process completed with exit code 1. 2024-05-19T18:18:33.9369934Z ##[group]Run .github/scripts/parse_ref.py 2024-05-19T18:18:33.9370480Z .github/scripts/parse_ref.py 2024-05-19T18:18:33.9377298Z shell: /usr/bin/bash -e {0} 2024-05-19T18:18:33.9377652Z env: 2024-05-19T18:18:33.9377935Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:33.9378403Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:18:33.9378892Z ##[endgroup] 2024-05-19T18:18:33.9607315Z Prepare all required actions 2024-05-19T18:18:33.9644624Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-05-19T18:18:33.9645134Z with: 2024-05-19T18:18:33.9645780Z github-token: *** 2024-05-19T18:18:33.9646094Z env: 2024-05-19T18:18:33.9646372Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:33.9646932Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:18:33.9647420Z ##[endgroup] 2024-05-19T18:18:33.9663053Z ##[group]Run set -eux 2024-05-19T18:18:33.9663406Z set -eux 2024-05-19T18:18:33.9664002Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-05-19T18:18:33.9671897Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:18:33.9672522Z env: 2024-05-19T18:18:33.9672810Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:33.9673277Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:18:33.9673934Z GITHUB_TOKEN: *** 2024-05-19T18:18:33.9674251Z ##[endgroup] 2024-05-19T18:18:33.9693828Z + python3 .github/scripts/get_workflow_job_id.py 9148947323 i-0e8f077216c874026 2024-05-19T18:18:35.1723843Z setting job-id=25153197615 2024-05-19T18:18:35.1724772Z setting job-name=cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:18:35.1927060Z Prepare all required actions 2024-05-19T18:18:35.1927538Z Getting action download info 2024-05-19T18:18:35.3045323Z ##[group]Run ./.github/actions/filter-test-configs 2024-05-19T18:18:35.3045767Z with: 2024-05-19T18:18:35.3046230Z github-token: *** 2024-05-19T18:18:35.3053955Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-05-19T18:18:35.3062283Z job-name: cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:18:35.3063014Z env: 2024-05-19T18:18:35.3063285Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:35.3063732Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:18:35.3064201Z ##[endgroup] 2024-05-19T18:18:35.3105654Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-05-19T18:18:35.3106197Z with: 2024-05-19T18:18:35.3106467Z shell: bash 2024-05-19T18:18:35.3106764Z timeout_minutes: 10 2024-05-19T18:18:35.3107089Z max_attempts: 5 2024-05-19T18:18:35.3107397Z retry_wait_seconds: 30 2024-05-19T18:18:35.3108172Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore python3 -m pip install requests==2.26.0 pyyaml==6.0.1 2024-05-19T18:18:35.3109004Z polling_interval_seconds: 1 2024-05-19T18:18:35.3109379Z warning_on_retry: true 2024-05-19T18:18:35.3109721Z continue_on_error: false 2024-05-19T18:18:35.3110216Z env: 2024-05-19T18:18:35.3110492Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:35.3110943Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:18:35.3111587Z GITHUB_TOKEN: *** 2024-05-19T18:18:35.3111900Z ##[endgroup] 2024-05-19T18:18:35.3600587Z + python3 -m pip install requests==2.26.0 pyyaml==6.0.1 2024-05-19T18:18:35.5731296Z Defaulting to user installation because normal site-packages is not writeable 2024-05-19T18:18:35.5881815Z Requirement already satisfied: requests==2.26.0 in /home/ec2-user/.local/lib/python3.7/site-packages (2.26.0) 2024-05-19T18:18:35.6024436Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-05-19T18:18:35.6033786Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (1.26.18) 2024-05-19T18:18:35.6230563Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2024.2.2) 2024-05-19T18:18:35.6237636Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2.0.12) 2024-05-19T18:18:35.6258992Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (3.7) 2024-05-19T18:18:36.3603066Z Command completed after 1 attempt(s). 2024-05-19T18:18:36.3643520Z ##[group]Run set -x 2024-05-19T18:18:36.3643863Z set -x 2024-05-19T18:18:36.3644166Z  2024-05-19T18:18:36.3644714Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-05-19T18:18:36.3645392Z # in runner workspace 2024-05-19T18:18:36.3645943Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-05-19T18:18:36.3653784Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:18:36.3654289Z env: 2024-05-19T18:18:36.3654567Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:36.3655029Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:18:36.3655516Z ##[endgroup] 2024-05-19T18:18:36.3676463Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-05-19T18:18:36.3871650Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-05-19T18:18:36.3872202Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-05-19T18:18:36.3872673Z echo "Job name: ${JOB_NAME}" 2024-05-19T18:18:36.3873076Z  2024-05-19T18:18:36.3873625Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-05-19T18:18:36.3874290Z # in runner workspace 2024-05-19T18:18:36.3874882Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-05-19T18:18:36.3875523Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-05-19T18:18:36.3875990Z  --job-name "${JOB_NAME}" \ 2024-05-19T18:18:36.3883946Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-05-19T18:18:36.3892319Z  --selected-test-configs "" \ 2024-05-19T18:18:36.3892813Z  --pr-number "${PR_NUMBER}" \ 2024-05-19T18:18:36.3893225Z  --tag "${TAG}" \ 2024-05-19T18:18:36.3893622Z  --event-name "${EVENT_NAME}" \ 2024-05-19T18:18:36.3894075Z  --schedule "${SCHEDULE}" \ 2024-05-19T18:18:36.3894503Z  --branch "${HEAD_BRANCH}" 2024-05-19T18:18:36.3901768Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:18:36.3902276Z env: 2024-05-19T18:18:36.3902557Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:36.3903013Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:18:36.3903687Z GITHUB_TOKEN: *** 2024-05-19T18:18:36.3904378Z JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:18:36.3905122Z PR_NUMBER: 2024-05-19T18:18:36.3905437Z TAG: ciflow/inductor/121956 2024-05-19T18:18:36.3905798Z EVENT_NAME: push 2024-05-19T18:18:36.3906117Z SCHEDULE: 2024-05-19T18:18:36.3906412Z HEAD_BRANCH: 2024-05-19T18:18:36.3906708Z ##[endgroup] 2024-05-19T18:18:36.3925589Z Workflow: inductor 2024-05-19T18:18:36.3926460Z Job name: cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:18:36.6090285Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-05-19T18:18:36.7216575Z ##[group]Run echo "Filtered matrix:" 2024-05-19T18:18:36.7217034Z echo "Filtered matrix:" 2024-05-19T18:18:36.7225077Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-05-19T18:18:36.7233486Z  2024-05-19T18:18:36.7233757Z echo 2024-05-19T18:18:36.7234136Z echo "Is the current job unstable? False" 2024-05-19T18:18:36.7234591Z  2024-05-19T18:18:36.7234855Z echo 2024-05-19T18:18:36.7235208Z echo "Is keep-going label set? False" 2024-05-19T18:18:36.7235647Z  2024-05-19T18:18:36.7236011Z echo 2024-05-19T18:18:36.7236327Z echo "Renabled issues? " 2024-05-19T18:18:36.7243905Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:18:36.7244409Z env: 2024-05-19T18:18:36.7244688Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:36.7245142Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:18:36.7245627Z ##[endgroup] 2024-05-19T18:18:36.7265181Z Filtered matrix: 2024-05-19T18:18:36.7275566Z {include: [{config: inductor, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-05-19T18:18:36.7283354Z 2024-05-19T18:18:36.7283507Z Is the current job unstable? False 2024-05-19T18:18:36.7283794Z 2024-05-19T18:18:36.7284034Z Is keep-going label set? False 2024-05-19T18:18:36.7284298Z 2024-05-19T18:18:36.7284420Z Renabled issues? 2024-05-19T18:18:36.7334638Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-05-19T18:18:36.7335402Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-05-19T18:18:36.7343035Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:18:36.7343529Z env: 2024-05-19T18:18:36.7343813Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:36.7344272Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:18:36.7344767Z JOB_TIMEOUT: 240 2024-05-19T18:18:36.7345070Z ##[endgroup] 2024-05-19T18:18:36.7420175Z ##[group]Run set -x 2024-05-19T18:18:36.7420582Z set -x 2024-05-19T18:18:36.7420881Z  2024-05-19T18:18:36.7421241Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-05-19T18:18:36.7421782Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-05-19T18:18:36.7422353Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-05-19T18:18:36.7422863Z  TEST_COMMAND=.ci/onnx/test.sh 2024-05-19T18:18:36.7423272Z else 2024-05-19T18:18:36.7423724Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-05-19T18:18:36.7424144Z fi 2024-05-19T18:18:36.7424419Z  2024-05-19T18:18:36.7424892Z # detached container should get cleaned up by teardown_ec2_linux 2024-05-19T18:18:36.7425639Z # TODO: Stop building test binaries as part of the build phase 2024-05-19T18:18:36.7426311Z # Used for GPU_FLAG since that doesn't play nice 2024-05-19T18:18:36.7426886Z # shellcheck disable=SC2086,SC2090 2024-05-19T18:18:36.7427344Z container_name=$(docker run \ 2024-05-19T18:18:36.7427761Z  ${GPU_FLAG:-} \ 2024-05-19T18:18:36.7428134Z  -e BUILD_ENVIRONMENT \ 2024-05-19T18:18:36.7428531Z  -e PR_NUMBER \ 2024-05-19T18:18:36.7428885Z  -e GITHUB_ACTIONS \ 2024-05-19T18:18:36.7429278Z  -e GITHUB_REPOSITORY \ 2024-05-19T18:18:36.7429689Z  -e GITHUB_WORKFLOW \ 2024-05-19T18:18:36.7430310Z  -e GITHUB_JOB \ 2024-05-19T18:18:36.7430680Z  -e GITHUB_RUN_ID \ 2024-05-19T18:18:36.7431070Z  -e GITHUB_RUN_NUMBER \ 2024-05-19T18:18:36.7431478Z  -e GITHUB_RUN_ATTEMPT \ 2024-05-19T18:18:36.7431868Z  -e JOB_ID \ 2024-05-19T18:18:36.7432210Z  -e JOB_NAME \ 2024-05-19T18:18:36.7432558Z  -e BASE_SHA \ 2024-05-19T18:18:36.7432900Z  -e BRANCH \ 2024-05-19T18:18:36.7433226Z  -e SHA1 \ 2024-05-19T18:18:36.7433570Z  -e AWS_DEFAULT_REGION \ 2024-05-19T18:18:36.7433980Z  -e IN_WHEEL_TEST \ 2024-05-19T18:18:36.7434358Z  -e SHARD_NUMBER \ 2024-05-19T18:18:36.7434721Z  -e TEST_CONFIG \ 2024-05-19T18:18:36.7435088Z  -e NUM_TEST_SHARDS \ 2024-05-19T18:18:36.7435478Z  -e REENABLED_ISSUES \ 2024-05-19T18:18:36.7435888Z  -e CONTINUE_THROUGH_ERROR \ 2024-05-19T18:18:36.7436305Z  -e VERBOSE_TEST_LOGS \ 2024-05-19T18:18:36.7436708Z  -e NO_TEST_TIMEOUT \ 2024-05-19T18:18:36.7437101Z  -e NO_TD \ 2024-05-19T18:18:36.7437441Z  -e TD_DISTRIBUTED \ 2024-05-19T18:18:36.7437820Z  -e PR_LABELS \ 2024-05-19T18:18:36.7438226Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-05-19T18:18:36.7438684Z  -e SCCACHE_BUCKET \ 2024-05-19T18:18:36.7439074Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-05-19T18:18:36.7439488Z  -e XLA_CUDA \ 2024-05-19T18:18:36.7439895Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-05-19T18:18:36.7440407Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-05-19T18:18:36.7440925Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-05-19T18:18:36.7441447Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-05-19T18:18:36.7441914Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-05-19T18:18:36.7442332Z  -e DASHBOARD_TAG \ 2024-05-19T18:18:36.7442801Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-05-19T18:18:36.7443357Z  --security-opt seccomp=unconfined \ 2024-05-19T18:18:36.7443827Z  --cap-add=SYS_PTRACE \ 2024-05-19T18:18:36.7444221Z  --ipc=host \ 2024-05-19T18:18:36.7444574Z  --shm-size="${SHM_SIZE}" \ 2024-05-19T18:18:36.7444978Z  --tty \ 2024-05-19T18:18:36.7445294Z  --detach \ 2024-05-19T18:18:36.7445652Z  --name="${container_name}" \ 2024-05-19T18:18:36.7446062Z  --user jenkins \ 2024-05-19T18:18:36.7446734Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-05-19T18:18:36.7447309Z  -w /var/lib/jenkins/workspace \ 2024-05-19T18:18:36.7447749Z  "${DOCKER_IMAGE}" 2024-05-19T18:18:36.7448089Z ) 2024-05-19T18:18:36.7448501Z # Propagate download.pytorch.org IP to container 2024-05-19T18:18:36.7449419Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-05-19T18:18:36.7450381Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-05-19T18:18:36.7451352Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-05-19T18:18:36.7458698Z shell: /usr/bin/bash -e {0} 2024-05-19T18:18:36.7459054Z env: 2024-05-19T18:18:36.7459330Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:36.7459778Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:18:36.7460391Z BUILD_ENVIRONMENT: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:18:36.7460891Z PR_NUMBER: 2024-05-19T18:18:36.7461230Z GITHUB_REPOSITORY: pytorch/pytorch 2024-05-19T18:18:36.7461641Z GITHUB_WORKFLOW: inductor 2024-05-19T18:18:36.7462002Z GITHUB_JOB: test 2024-05-19T18:18:36.7462323Z GITHUB_RUN_ID: 9148947323 2024-05-19T18:18:36.7462690Z GITHUB_RUN_NUMBER: 69546 2024-05-19T18:18:36.7463035Z GITHUB_RUN_ATTEMPT: 1 2024-05-19T18:18:36.7463369Z JOB_ID: 25153197615 2024-05-19T18:18:36.7464059Z JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:18:36.7464813Z BRANCH: 2024-05-19T18:18:36.7465154Z SHA1: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:18:36.7465686Z BASE_SHA: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:18:36.7466189Z TEST_CONFIG: dynamic_inductor_torchbench 2024-05-19T18:18:36.7466622Z SHARD_NUMBER: 2 2024-05-19T18:18:36.7466927Z NUM_TEST_SHARDS: 2 2024-05-19T18:18:36.7467256Z REENABLED_ISSUES: 2024-05-19T18:18:36.7467606Z CONTINUE_THROUGH_ERROR: False 2024-05-19T18:18:36.7467992Z VERBOSE_TEST_LOGS: False 2024-05-19T18:18:36.7468350Z NO_TEST_TIMEOUT: False 2024-05-19T18:18:36.7468691Z NO_TD: False 2024-05-19T18:18:36.7468994Z TD_DISTRIBUTED: False 2024-05-19T18:18:36.7469405Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-05-19T18:18:36.7469900Z SCCACHE_S3_KEY_PREFIX: inductor 2024-05-19T18:18:36.7470502Z SHM_SIZE: 2g 2024-05-19T18:18:36.7471588Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:18:36.7472746Z XLA_CUDA: 2024-05-19T18:18:36.7473225Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-05-19T18:18:36.7473835Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-05-19T18:18:36.7474272Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-05-19T18:18:36.7474673Z DASHBOARD_TAG: 2024-05-19T18:18:36.7475186Z HUGGING_FACE_HUB_TOKEN: *** 2024-05-19T18:18:36.7475562Z ##[endgroup] 2024-05-19T18:18:36.7495208Z + [[ dynamic_inductor_torchbench == \m\u\l\t\i\g\p\u ]] 2024-05-19T18:18:36.7495957Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *onnx* ]] 2024-05-19T18:18:36.7496479Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-05-19T18:18:36.7501973Z +++ nproc --ignore=2 2024-05-19T18:18:36.7514154Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9148947323 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:18:48.0071449Z + container_name=f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T18:18:48.0072215Z + grep download.pytorch.org /etc/hosts 2024-05-19T18:18:48.0073405Z + docker exec -i f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 sudo bash -c '/bin/cat >> /etc/hosts' 2024-05-19T18:18:48.0681021Z + echo DOCKER_CONTAINER_ID=f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T18:18:48.0683253Z ++ echo dist/torch-2.4.0a0+gitc8c7ddf-cp310-cp310-linux_x86_64.whl 2024-05-19T18:18:48.0685947Z + docker exec -t f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 sh -c 'pip install dist/torch-2.4.0a0+gitc8c7ddf-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-05-19T18:18:48.4355877Z Processing ./dist/torch-2.4.0a0+gitc8c7ddf-cp310-cp310-linux_x86_64.whl (from torch==2.4.0a0+gitc8c7ddf) 2024-05-19T18:18:48.7466875Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (3.13.1) 2024-05-19T18:18:48.7471126Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (4.11.0) 2024-05-19T18:18:48.7473072Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (1.12) 2024-05-19T18:18:48.7475735Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (2.8.8) 2024-05-19T18:18:48.7478416Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (3.1.4) 2024-05-19T18:18:48.7481451Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (2024.2.0) 2024-05-19T18:18:48.7507077Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (3.3.0) 2024-05-19T18:18:48.7564071Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (1.21.2) 2024-05-19T18:18:48.7991264Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (2.1.5) 2024-05-19T18:18:48.8146675Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (1.3.0) 2024-05-19T18:18:50.1016661Z Installing collected packages: torch 2024-05-19T18:18:58.6657973Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-05-19T18:18:58.6659348Z timm 0.9.7 requires torchvision, which is not installed. 2024-05-19T18:18:58.6660017Z Successfully installed torch-2.4.0a0+gitc8c7ddf 2024-05-19T18:18:58.7488644Z ++ dirname .ci/pytorch/test.sh 2024-05-19T18:18:58.7493457Z + source .ci/pytorch/common.sh 2024-05-19T18:18:58.7495495Z +++ dirname .ci/pytorch/common.sh 2024-05-19T18:18:58.7501355Z ++ source .ci/pytorch/common_utils.sh 2024-05-19T18:18:58.7502008Z +++ declare -f -t trap_add 2024-05-19T18:18:58.7506870Z ++ set -ex 2024-05-19T18:18:58.7507531Z ++ [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]] 2024-05-19T18:18:58.7508178Z ++ BUILD_TEST_LIBTORCH=0 2024-05-19T18:18:58.7508730Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *rocm* ]] 2024-05-19T18:18:58.7510763Z ++ stat -c %u /var/lib/jenkins/workspace 2024-05-19T18:18:58.7519513Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-05-19T18:18:58.7519969Z + trap_add cleanup_workspace EXIT 2024-05-19T18:18:58.7520583Z + trap_add_cmd=cleanup_workspace 2024-05-19T18:18:58.7520968Z + shift 2024-05-19T18:18:58.7521351Z + for trap_add_name in "$@" 2024-05-19T18:18:58.7524954Z +++ trap -p EXIT 2024-05-19T18:18:58.7526433Z ++ eval 'extract_trap_cmd ' 2024-05-19T18:18:58.7526936Z +++ extract_trap_cmd 2024-05-19T18:18:58.7527323Z +++ printf '%s\n' '' 2024-05-19T18:18:58.7527720Z ++ printf '%s\n' cleanup_workspace 2024-05-19T18:18:58.7528597Z + trap -- ' 2024-05-19T18:18:58.7528956Z cleanup_workspace' EXIT 2024-05-19T18:18:58.7529447Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-05-19T18:18:59.2154385Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-05-19T18:18:59.2169164Z + echo 'Environment variables:' 2024-05-19T18:18:59.2169575Z Environment variables: 2024-05-19T18:18:59.2169904Z + env 2024-05-19T18:18:59.2174788Z INSTALLED_DB=yes 2024-05-19T18:18:59.2175309Z NV_LIBCUBLAS_VERSION=12.4.2.65-1 2024-05-19T18:18:59.2175801Z NVIDIA_VISIBLE_DEVICES=all 2024-05-19T18:18:59.2176226Z NV_NVML_DEV_VERSION=12.4.99-1 2024-05-19T18:18:59.2177017Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T18:18:59.2177641Z CONTINUE_THROUGH_ERROR=False 2024-05-19T18:18:59.2178347Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.20.5-1+cuda12.4 2024-05-19T18:18:59.2179037Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.20.5-1 2024-05-19T18:18:59.2179689Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:18:59.2180382Z HOSTNAME=f8a81ce94298 2024-05-19T18:18:59.2181560Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_407fbb83-5d98-4433-abb4-d5e94d1136b6 2024-05-19T18:18:59.2182509Z GITHUB_ACTION=__self 2024-05-19T18:18:59.2182981Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-05-19T18:18:59.2189063Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536 2024-05-19T18:18:59.2195325Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.2.65-1 2024-05-19T18:18:59.2195848Z NV_NVTX_VERSION=12.4.99-1 2024-05-19T18:18:59.2196218Z GITHUB_RUN_NUMBER=69546 2024-05-19T18:18:59.2196598Z TEST_CONFIG=dynamic_inductor_torchbench 2024-05-19T18:18:59.2197048Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-05-19T18:18:59.2197525Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-05-19T18:18:59.2198029Z NV_CUDA_CUDART_DEV_VERSION=12.4.99-1 2024-05-19T18:18:59.2198531Z NV_LIBCUSPARSE_VERSION=12.3.0.142-1 2024-05-19T18:18:59.2198971Z NV_LIBNPP_VERSION=12.2.5.2-1 2024-05-19T18:18:59.2199649Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-05-19T18:18:59.2200165Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-05-19T18:18:59.2200649Z GITHUB_REF_TYPE=tag 2024-05-19T18:18:59.2200989Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-05-19T18:18:59.2201384Z NCCL_VERSION=2.20.5-1 2024-05-19T18:18:59.2201796Z BASE_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:18:59.2202265Z XLA_CUDA= 2024-05-19T18:18:59.2202774Z HUGGING_FACE_HUB_TOKEN=*** 2024-05-19T18:18:59.2204586Z *** 2024-05-19T18:18:59.2204889Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-05-19T18:18:59.2205457Z GITHUB_REPOSITORY_ID=65600975 2024-05-19T18:18:59.2205838Z GITHUB_ACTIONS=true 2024-05-19T18:18:59.2206189Z NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:18:59.2206793Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.99-1 2024-05-19T18:18:59.2207335Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.2-1 2024-05-19T18:18:59.2207822Z SHA1=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:18:59.2208343Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-05-19T18:18:59.2208844Z GITHUB_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:18:59.2209646Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/121956 2024-05-19T18:18:59.2210367Z UCC_HOME=/usr 2024-05-19T18:18:59.2210704Z NV_LIBCUBLAS_DEV_VERSION=12.4.2.65-1 2024-05-19T18:18:59.2211123Z VERBOSE_TEST_LOGS=False 2024-05-19T18:18:59.2211528Z NVIDIA_PRODUCT_NAME=CUDA 2024-05-19T18:18:59.2212009Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4 2024-05-19T18:18:59.2212515Z GITHUB_REF=refs/tags/ciflow/inductor/121956 2024-05-19T18:18:59.2212996Z NV_CUDA_CUDART_VERSION=12.4.99-1 2024-05-19T18:18:59.2213381Z SHARD_NUMBER=2 2024-05-19T18:18:59.2213703Z GITHUB_REF_PROTECTED=false 2024-05-19T18:18:59.2214058Z HOME=/var/lib/jenkins 2024-05-19T18:18:59.2214477Z GITHUB_API_URL=https://api.github.com 2024-05-19T18:18:59.2214925Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-05-19T18:18:59.2215416Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-05-19T18:18:59.2215924Z SCCACHE_S3_KEY_PREFIX=inductor 2024-05-19T18:18:59.2216297Z CUDA_VERSION=12.4.0 2024-05-19T18:18:59.2216727Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.2.65-1 2024-05-19T18:18:59.2217177Z NUM_TEST_SHARDS=2 2024-05-19T18:18:59.2217485Z UCX_HOME=/usr 2024-05-19T18:18:59.2218003Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.0-1 2024-05-19T18:18:59.2219076Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_407fbb83-5d98-4433-abb4-d5e94d1136b6 2024-05-19T18:18:59.2220344Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:18:59.2221585Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_407fbb83-5d98-4433-abb4-d5e94d1136b6 2024-05-19T18:18:59.2222692Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-05-19T18:18:59.2223354Z GITHUB_EVENT_NAME=push 2024-05-19T18:18:59.2223698Z DASHBOARD_TAG= 2024-05-19T18:18:59.2224013Z GITHUB_RUN_ID=9148947323 2024-05-19T18:18:59.2224475Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.2-1 2024-05-19T18:18:59.2225016Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4 2024-05-19T18:18:59.2226010Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_407fbb83-5d98-4433-abb4-d5e94d1136b6 2024-05-19T18:18:59.2226926Z GITHUB_ACTOR=pytorch-bot[bot] 2024-05-19T18:18:59.2227347Z NV_LIBNPP_DEV_VERSION=12.2.5.2-1 2024-05-19T18:18:59.2227757Z PR_NUMBER= 2024-05-19T18:18:59.2228084Z GITHUB_RUN_ATTEMPT=1 2024-05-19T18:18:59.2228432Z ANACONDA_PYTHON_VERSION=3.10 2024-05-19T18:18:59.2228883Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-05-19T18:18:59.2229354Z TERM=xterm 2024-05-19T18:18:59.2229698Z NV_LIBCUSPARSE_DEV_VERSION=12.3.0.142-1 2024-05-19T18:18:59.2230344Z INSTALLED_VISION=yes 2024-05-19T18:18:59.2230672Z BRANCH= 2024-05-19T18:18:59.2230972Z OPENSSL_ROOT_DIR=/opt/openssl 2024-05-19T18:18:59.2231381Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-05-19T18:18:59.2231930Z CUDA_PATH=/usr/local/cuda 2024-05-19T18:18:59.2232709Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-05-19T18:18:59.2233468Z GITHUB_SERVER_URL=https://github.com 2024-05-19T18:18:59.2233957Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-05-19T18:18:59.2234436Z REENABLED_ISSUES= 2024-05-19T18:18:59.2234751Z SHLVL=1 2024-05-19T18:18:59.2235023Z MAX_JOBS=14 2024-05-19T18:18:59.2235341Z NV_CUDA_LIB_VERSION=12.4.0-1 2024-05-19T18:18:59.2235703Z NVARCH=x86_64 2024-05-19T18:18:59.2236104Z GITHUB_ACTOR_ID=54816060 2024-05-19T18:18:59.2236576Z GITHUB_WORKFLOW_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:18:59.2237126Z GITHUB_REF_NAME=ciflow/inductor/121956 2024-05-19T18:18:59.2237623Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4 2024-05-19T18:18:59.2238290Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-05-19T18:18:59.2238866Z GITHUB_JOB=test 2024-05-19T18:18:59.2239281Z NV_LIBNCCL_PACKAGE=libnccl2=2.20.5-1+cuda12.4 2024-05-19T18:18:59.2239850Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-05-19T18:18:59.2240374Z NO_TEST_TIMEOUT=False 2024-05-19T18:18:59.2240706Z TD_DISTRIBUTED=False 2024-05-19T18:18:59.2241100Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.0-1 2024-05-19T18:18:59.2241549Z GITHUB_REPOSITORY=pytorch/pytorch 2024-05-19T18:18:59.2241978Z NV_NVPROF_VERSION=12.4.99-1 2024-05-19T18:18:59.2242347Z GITHUB_RETENTION_DAYS=90 2024-05-19T18:18:59.2242710Z OPENSSL_DIR=/opt/openssl 2024-05-19T18:18:59.2243076Z GITHUB_ACTION_REPOSITORY= 2024-05-19T18:18:59.2244113Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-05-19T18:18:59.2245186Z GITHUB_BASE_REF= 2024-05-19T18:18:59.2245521Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-05-19T18:18:59.2245902Z CI=true 2024-05-19T18:18:59.2246216Z NV_LIBNCCL_PACKAGE_VERSION=2.20.5-1 2024-05-19T18:18:59.2246710Z GITHUB_REPOSITORY_OWNER=pytorch 2024-05-19T18:18:59.2247106Z JOB_ID=25153197615 2024-05-19T18:18:59.2247433Z INSTALLED_PROTOBUF=yes 2024-05-19T18:18:59.2247769Z GITHUB_HEAD_REF= 2024-05-19T18:18:59.2248089Z GITHUB_ACTION_REF= 2024-05-19T18:18:59.2248531Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-05-19T18:18:59.2249008Z GITHUB_WORKFLOW=inductor 2024-05-19T18:18:59.2249382Z DEBIAN_FRONTEND=noninteractive 2024-05-19T18:18:59.2250295Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_407fbb83-5d98-4433-abb4-d5e94d1136b6 2024-05-19T18:18:59.2251116Z NO_TD=False 2024-05-19T18:18:59.2251422Z SKIP_SCCACHE_INITIALIZATION=1 2024-05-19T18:18:59.2251808Z _=/usr/bin/env 2024-05-19T18:18:59.2252297Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-05-19T18:18:59.2322565Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-05-19T18:18:59.2323738Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-05-19T18:18:59.2324923Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-05-19T18:18:59.2325813Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-05-19T18:18:59.2326423Z + BUILD_DIR=build 2024-05-19T18:18:59.2326927Z + BUILD_RENAMED_DIR=build_renamed 2024-05-19T18:18:59.2327338Z + BUILD_BIN_DIR=build/bin 2024-05-19T18:18:59.2327681Z + SHARD_NUMBER=2 2024-05-19T18:18:59.2327988Z + NUM_TEST_SHARDS=2 2024-05-19T18:18:59.2328349Z + export VALGRIND=ON 2024-05-19T18:18:59.2328679Z + VALGRIND=ON 2024-05-19T18:18:59.2329169Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *clang9* ]] 2024-05-19T18:18:59.2329667Z + [[ 0 == \1 ]] 2024-05-19T18:18:59.2329961Z + [[ False == \1 ]] 2024-05-19T18:18:59.2330430Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *bazel* ]] 2024-05-19T18:18:59.2330959Z ++ realpath build/custom_test_artifacts 2024-05-19T18:18:59.2333785Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-05-19T18:18:59.2334478Z + [[ -n '' ]] 2024-05-19T18:18:59.2334952Z + echo 'Environment variables' 2024-05-19T18:18:59.2335349Z Environment variables 2024-05-19T18:18:59.2335672Z + env 2024-05-19T18:18:59.2338849Z INSTALLED_DB=yes 2024-05-19T18:18:59.2339396Z NV_LIBCUBLAS_VERSION=12.4.2.65-1 2024-05-19T18:18:59.2339945Z NVIDIA_VISIBLE_DEVICES=all 2024-05-19T18:18:59.2340468Z NV_NVML_DEV_VERSION=12.4.99-1 2024-05-19T18:18:59.2341290Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T18:18:59.2342108Z CONTINUE_THROUGH_ERROR=False 2024-05-19T18:18:59.2342814Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.20.5-1+cuda12.4 2024-05-19T18:18:59.2343415Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.20.5-1 2024-05-19T18:18:59.2344130Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:18:59.2344845Z HOSTNAME=f8a81ce94298 2024-05-19T18:18:59.2345916Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_407fbb83-5d98-4433-abb4-d5e94d1136b6 2024-05-19T18:18:59.2346820Z GITHUB_ACTION=__self 2024-05-19T18:18:59.2347310Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-05-19T18:18:59.2354449Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536 2024-05-19T18:18:59.2360194Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.2.65-1 2024-05-19T18:18:59.2360693Z NV_NVTX_VERSION=12.4.99-1 2024-05-19T18:18:59.2361041Z GITHUB_RUN_NUMBER=69546 2024-05-19T18:18:59.2361407Z TEST_CONFIG=dynamic_inductor_torchbench 2024-05-19T18:18:59.2361836Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-05-19T18:18:59.2362310Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-05-19T18:18:59.2362757Z NV_CUDA_CUDART_DEV_VERSION=12.4.99-1 2024-05-19T18:18:59.2363195Z NV_LIBCUSPARSE_VERSION=12.3.0.142-1 2024-05-19T18:18:59.2363607Z NV_LIBNPP_VERSION=12.2.5.2-1 2024-05-19T18:18:59.2364040Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-05-19T18:18:59.2364533Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-05-19T18:18:59.2364985Z GITHUB_REF_TYPE=tag 2024-05-19T18:18:59.2365316Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-05-19T18:18:59.2365691Z NCCL_VERSION=2.20.5-1 2024-05-19T18:18:59.2366090Z BASE_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:18:59.2366646Z XLA_CUDA= 2024-05-19T18:18:59.2367113Z HUGGING_FACE_HUB_TOKEN=*** 2024-05-19T18:18:59.2367507Z *** 2024-05-19T18:18:59.2367793Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-05-19T18:18:59.2368186Z GITHUB_REPOSITORY_ID=65600975 2024-05-19T18:18:59.2368554Z GITHUB_ACTIONS=true 2024-05-19T18:18:59.2368888Z NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:18:59.2369384Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.99-1 2024-05-19T18:18:59.2369909Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.2-1 2024-05-19T18:18:59.2370386Z SHA1=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:18:59.2370899Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-05-19T18:18:59.2371393Z GITHUB_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:18:59.2372190Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/121956 2024-05-19T18:18:59.2373021Z UCC_HOME=/usr 2024-05-19T18:18:59.2373371Z NV_LIBCUBLAS_DEV_VERSION=12.4.2.65-1 2024-05-19T18:18:59.2373784Z VERBOSE_TEST_LOGS=False 2024-05-19T18:18:59.2374119Z NVIDIA_PRODUCT_NAME=CUDA 2024-05-19T18:18:59.2374581Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4 2024-05-19T18:18:59.2375082Z GITHUB_REF=refs/tags/ciflow/inductor/121956 2024-05-19T18:18:59.2375557Z NV_CUDA_CUDART_VERSION=12.4.99-1 2024-05-19T18:18:59.2375929Z SHARD_NUMBER=2 2024-05-19T18:18:59.2376244Z GITHUB_REF_PROTECTED=false 2024-05-19T18:18:59.2376679Z HOME=/var/lib/jenkins 2024-05-19T18:18:59.2377040Z GITHUB_API_URL=https://api.github.com 2024-05-19T18:18:59.2377487Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-05-19T18:18:59.2378022Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-05-19T18:18:59.2378506Z SCCACHE_S3_KEY_PREFIX=inductor 2024-05-19T18:18:59.2378869Z CUDA_VERSION=12.4.0 2024-05-19T18:18:59.2379293Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.2.65-1 2024-05-19T18:18:59.2379735Z NUM_TEST_SHARDS=2 2024-05-19T18:18:59.2380037Z UCX_HOME=/usr 2024-05-19T18:18:59.2380548Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.0-1 2024-05-19T18:18:59.2381617Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_407fbb83-5d98-4433-abb4-d5e94d1136b6 2024-05-19T18:18:59.2382865Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:18:59.2384130Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_407fbb83-5d98-4433-abb4-d5e94d1136b6 2024-05-19T18:18:59.2385243Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-05-19T18:18:59.2385887Z GITHUB_EVENT_NAME=push 2024-05-19T18:18:59.2386216Z DASHBOARD_TAG= 2024-05-19T18:18:59.2386519Z GITHUB_RUN_ID=9148947323 2024-05-19T18:18:59.2386968Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.2-1 2024-05-19T18:18:59.2387489Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4 2024-05-19T18:18:59.2388475Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_407fbb83-5d98-4433-abb4-d5e94d1136b6 2024-05-19T18:18:59.2389376Z GITHUB_ACTOR=pytorch-bot[bot] 2024-05-19T18:18:59.2389782Z NV_LIBNPP_DEV_VERSION=12.2.5.2-1 2024-05-19T18:18:59.2390307Z PR_NUMBER= 2024-05-19T18:18:59.2390595Z GITHUB_RUN_ATTEMPT=1 2024-05-19T18:18:59.2390912Z VALGRIND=ON 2024-05-19T18:18:59.2391206Z ANACONDA_PYTHON_VERSION=3.10 2024-05-19T18:18:59.2391654Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-05-19T18:18:59.2392120Z TERM=xterm 2024-05-19T18:18:59.2392453Z NV_LIBCUSPARSE_DEV_VERSION=12.3.0.142-1 2024-05-19T18:18:59.2392862Z INSTALLED_VISION=yes 2024-05-19T18:18:59.2393176Z BRANCH= 2024-05-19T18:18:59.2393464Z OPENSSL_ROOT_DIR=/opt/openssl 2024-05-19T18:18:59.2393865Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-05-19T18:18:59.2394284Z CUDA_PATH=/usr/local/cuda 2024-05-19T18:18:59.2395046Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-05-19T18:18:59.2395801Z GITHUB_SERVER_URL=https://github.com 2024-05-19T18:18:59.2396284Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-05-19T18:18:59.2396757Z REENABLED_ISSUES= 2024-05-19T18:18:59.2397053Z SHLVL=1 2024-05-19T18:18:59.2397312Z MAX_JOBS=14 2024-05-19T18:18:59.2397614Z NV_CUDA_LIB_VERSION=12.4.0-1 2024-05-19T18:18:59.2397964Z NVARCH=x86_64 2024-05-19T18:18:59.2398261Z GITHUB_ACTOR_ID=54816060 2024-05-19T18:18:59.2398723Z GITHUB_WORKFLOW_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:18:59.2399260Z GITHUB_REF_NAME=ciflow/inductor/121956 2024-05-19T18:18:59.2399741Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4 2024-05-19T18:18:59.2400406Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-05-19T18:18:59.2400969Z GITHUB_JOB=test 2024-05-19T18:18:59.2401370Z NV_LIBNCCL_PACKAGE=libnccl2=2.20.5-1+cuda12.4 2024-05-19T18:18:59.2401928Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-05-19T18:18:59.2402440Z NO_TEST_TIMEOUT=False 2024-05-19T18:18:59.2402843Z TD_DISTRIBUTED=False 2024-05-19T18:18:59.2403231Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.0-1 2024-05-19T18:18:59.2403673Z GITHUB_REPOSITORY=pytorch/pytorch 2024-05-19T18:18:59.2404096Z NV_NVPROF_VERSION=12.4.99-1 2024-05-19T18:18:59.2404453Z GITHUB_RETENTION_DAYS=90 2024-05-19T18:18:59.2404804Z OPENSSL_DIR=/opt/openssl 2024-05-19T18:18:59.2405156Z GITHUB_ACTION_REPOSITORY= 2024-05-19T18:18:59.2406192Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-05-19T18:18:59.2407367Z GITHUB_BASE_REF= 2024-05-19T18:18:59.2407694Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-05-19T18:18:59.2408063Z CI=true 2024-05-19T18:18:59.2408368Z NV_LIBNCCL_PACKAGE_VERSION=2.20.5-1 2024-05-19T18:18:59.2408787Z GITHUB_REPOSITORY_OWNER=pytorch 2024-05-19T18:18:59.2409157Z JOB_ID=25153197615 2024-05-19T18:18:59.2409474Z INSTALLED_PROTOBUF=yes 2024-05-19T18:18:59.2409798Z GITHUB_HEAD_REF= 2024-05-19T18:18:59.2410108Z GITHUB_ACTION_REF= 2024-05-19T18:18:59.2410538Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-05-19T18:18:59.2411005Z GITHUB_WORKFLOW=inductor 2024-05-19T18:18:59.2411362Z DEBIAN_FRONTEND=noninteractive 2024-05-19T18:18:59.2412259Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_407fbb83-5d98-4433-abb4-d5e94d1136b6 2024-05-19T18:18:59.2413070Z NO_TD=False 2024-05-19T18:18:59.2413375Z SKIP_SCCACHE_INITIALIZATION=1 2024-05-19T18:18:59.2413731Z _=/usr/bin/env 2024-05-19T18:18:59.2414074Z + echo 'Testing pytorch' 2024-05-19T18:18:59.2414416Z Testing pytorch 2024-05-19T18:18:59.2414731Z + export LANG=C.UTF-8 2024-05-19T18:18:59.2415072Z + LANG=C.UTF-8 2024-05-19T18:18:59.2415362Z + PR_NUMBER= 2024-05-19T18:18:59.2415734Z + [[ dynamic_inductor_torchbench == \d\e\f\a\u\l\t ]] 2024-05-19T18:18:59.2416304Z + [[ dynamic_inductor_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-05-19T18:18:59.2416858Z + [[ dynamic_inductor_torchbench == \s\l\o\w ]] 2024-05-19T18:18:59.2417512Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-05-19T18:18:59.2418197Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *cuda* ]] 2024-05-19T18:18:59.2418788Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-05-19T18:18:59.2419265Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-05-19T18:18:59.2419737Z + [[ dynamic_inductor_torchbench == *crossref* ]] 2024-05-19T18:18:59.2420329Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]] 2024-05-19T18:18:59.2420959Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *xpu* ]] 2024-05-19T18:18:59.2421617Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-05-19T18:18:59.2422173Z + pip_install --user ninja==1.10.2 2024-05-19T18:18:59.2422699Z + pip install --progress-bar off --user ninja==1.10.2 2024-05-19T18:18:59.6065154Z Collecting ninja==1.10.2 2024-05-19T18:18:59.6237299Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-05-19T18:18:59.6390495Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-05-19T18:19:00.9419379Z Installing collected packages: ninja 2024-05-19T18:19:00.9485621Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-05-19T18:19:00.9486953Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-05-19T18:19:00.9532285Z Successfully installed ninja-1.10.2 2024-05-19T18:19:01.0393787Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-05-19T18:19:01.0395905Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-05-19T18:19:01.0397500Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]] 2024-05-19T18:19:01.0398024Z + install_tlparse 2024-05-19T18:19:01.0398410Z + pip_install --user tlparse==0.3.7 2024-05-19T18:19:01.0398965Z + pip install --progress-bar off --user tlparse==0.3.7 2024-05-19T18:19:01.3882150Z Collecting tlparse==0.3.7 2024-05-19T18:19:01.4032588Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-05-19T18:19:01.4148135Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-05-19T18:19:02.7226397Z Installing collected packages: tlparse 2024-05-19T18:19:02.7588155Z Successfully installed tlparse-0.3.7 2024-05-19T18:19:02.8443028Z ++ python -m site --user-base 2024-05-19T18:19:02.8598216Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-05-19T18:19:02.8600190Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *asan* ]] 2024-05-19T18:19:02.8600947Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-debug* ]] 2024-05-19T18:19:02.8601595Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-05-19T18:19:02.8602529Z + echo 'We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-05-19T18:19:02.8603654Z We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass 2024-05-19T18:19:02.8604353Z + cd test 2024-05-19T18:19:02.8604862Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-05-19T18:19:04.2044323Z + [[ dynamic_inductor_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-05-19T18:19:04.2045076Z + [[ dynamic_inductor_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-05-19T18:19:04.2047763Z + DYNAMO_BENCHMARK_FLAGS=() 2024-05-19T18:19:04.2048386Z + [[ dynamic_inductor_torchbench == *dynamo_eager* ]] 2024-05-19T18:19:04.2049107Z + [[ dynamic_inductor_torchbench == *aot_eager* ]] 2024-05-19T18:19:04.2049675Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-05-19T18:19:04.2050206Z + [[ dynamic_inductor_torchbench == *inductor* ]] 2024-05-19T18:19:04.2050706Z + [[ dynamic_inductor_torchbench != *perf* ]] 2024-05-19T18:19:04.2051327Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-05-19T18:19:04.2051800Z + [[ dynamic_inductor_torchbench == *dynamic* ]] 2024-05-19T18:19:04.2052434Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2024-05-19T18:19:04.2053054Z + [[ dynamic_inductor_torchbench == *cpu_inductor* ]] 2024-05-19T18:19:04.2053614Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-05-19T18:19:04.2061718Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *tbb* ]] 2024-05-19T18:19:04.2076299Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]] 2024-05-19T18:19:04.2077241Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-05-19T18:19:04.2078150Z + cd test 2024-05-19T18:19:04.2078784Z + python -c 'import torch; print(torch.__config__.show())' 2024-05-19T18:19:05.4419408Z PyTorch built with: 2024-05-19T18:19:05.4420079Z - GCC 9.4 2024-05-19T18:19:05.4420525Z - C++ Version: 201703 2024-05-19T18:19:05.4421561Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-05-19T18:19:05.4422642Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-05-19T18:19:05.4423314Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-05-19T18:19:05.4423832Z - LAPACK is enabled (usually provided by MKL) 2024-05-19T18:19:05.4424329Z - NNPACK is enabled 2024-05-19T18:19:05.4424712Z - CPU capability usage: AVX2 2024-05-19T18:19:05.4425122Z - CUDA Runtime 12.4 2024-05-19T18:19:05.4425644Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-05-19T18:19:05.4426262Z - CuDNN 8.9.7 (built against CUDA 12.2) 2024-05-19T18:19:05.4426708Z - Magma 2.6.1 2024-05-19T18:19:05.4434097Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.4, CUDNN_VERSION=8.9.7, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.4.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-05-19T18:19:05.4440807Z 2024-05-19T18:19:05.7211566Z + cd test 2024-05-19T18:19:05.7212300Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-05-19T18:19:06.8104480Z ATen/Parallel: 2024-05-19T18:19:06.8105050Z at::get_num_threads() : 8 2024-05-19T18:19:06.8105538Z at::get_num_interop_threads() : 16 2024-05-19T18:19:06.8105992Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-05-19T18:19:06.8106394Z omp_get_max_threads() : 8 2024-05-19T18:19:06.8107369Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-05-19T18:19:06.8108167Z mkl_get_max_threads() : 8 2024-05-19T18:19:06.8108800Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-05-19T18:19:06.8109434Z std::thread::hardware_concurrency() : 16 2024-05-19T18:19:06.8109878Z Environment variables: 2024-05-19T18:19:06.8110393Z OMP_NUM_THREADS : [not set] 2024-05-19T18:19:06.8110782Z MKL_NUM_THREADS : [not set] 2024-05-19T18:19:06.8111165Z ATen parallel backend: OpenMP 2024-05-19T18:19:06.8111421Z 2024-05-19T18:19:07.0445600Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]] 2024-05-19T18:19:07.0446429Z + [[ dynamic_inductor_torchbench == *backward* ]] 2024-05-19T18:19:07.0447241Z + [[ dynamic_inductor_torchbench == *xla* ]] 2024-05-19T18:19:07.0447904Z + [[ dynamic_inductor_torchbench == *executorch* ]] 2024-05-19T18:19:07.0448680Z + [[ dynamic_inductor_torchbench == \j\i\t\_\l\e\g\a\c\y ]] 2024-05-19T18:19:07.0449595Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]] 2024-05-19T18:19:07.0450399Z + [[ dynamic_inductor_torchbench == distributed ]] 2024-05-19T18:19:07.0451049Z + [[ dynamic_inductor_torchbench == deploy ]] 2024-05-19T18:19:07.0451621Z + [[ dynamic_inductor_torchbench == *inductor_distributed* ]] 2024-05-19T18:19:07.0452336Z + [[ dynamic_inductor_torchbench == *inductor-micro-benchmark* ]] 2024-05-19T18:19:07.0452942Z + [[ dynamic_inductor_torchbench == *huggingface* ]] 2024-05-19T18:19:07.0453499Z + [[ dynamic_inductor_torchbench == *timm* ]] 2024-05-19T18:19:07.0454001Z + [[ dynamic_inductor_torchbench == *torchbench* ]] 2024-05-19T18:19:07.0454539Z + [[ dynamic_inductor_torchbench == *cpu_inductor* ]] 2024-05-19T18:19:07.0455013Z + install_torchaudio cuda 2024-05-19T18:19:07.0455352Z + local commit 2024-05-19T18:19:07.0455663Z ++ get_pinned_commit audio 2024-05-19T18:19:07.0456050Z ++ cat .github/ci_commit_pins/audio.txt 2024-05-19T18:19:07.0460979Z + commit=1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0 2024-05-19T18:19:07.0461644Z + [[ cuda == \c\u\d\a ]] 2024-05-19T18:19:07.0462115Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-05-19T18:19:07.0463037Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0 2024-05-19T18:19:07.0464670Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0 2024-05-19T18:19:07.3544804Z Collecting git+https://github.com/pytorch/audio.git@1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0 2024-05-19T18:19:07.3548625Z Cloning https://github.com/pytorch/audio.git (to revision 1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0) to /tmp/pip-req-build-6aaf2m6x 2024-05-19T18:19:07.3563197Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-6aaf2m6x 2024-05-19T18:19:08.0638922Z Running command git rev-parse -q --verify 'sha^1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0' 2024-05-19T18:19:08.0654784Z Running command git fetch -q https://github.com/pytorch/audio.git 1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0 2024-05-19T18:19:08.4542406Z Resolved https://github.com/pytorch/audio.git to commit 1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0 2024-05-19T18:19:08.4543531Z Running command git submodule update --init --recursive -q 2024-05-19T18:19:10.4676337Z Preparing metadata (setup.py) ... [?25l- done 2024-05-19T18:19:10.4705017Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.2.0a0+1980f8a) (2.4.0a0+gitc8c7ddf) 2024-05-19T18:19:10.4770539Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (3.13.1) 2024-05-19T18:19:10.4775676Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (4.11.0) 2024-05-19T18:19:10.4777550Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (1.12) 2024-05-19T18:19:10.4780618Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (2.8.8) 2024-05-19T18:19:10.4783921Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (3.1.4) 2024-05-19T18:19:10.4787193Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (2024.2.0) 2024-05-19T18:19:10.5222953Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.2.0a0+1980f8a) (2.1.5) 2024-05-19T18:19:10.5380459Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchaudio==2.2.0a0+1980f8a) (1.3.0) 2024-05-19T18:19:10.5452409Z Building wheels for collected packages: torchaudio 2024-05-19T18:21:16.7911538Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-05-19T18:21:16.7940206Z [?25h Created wheel for torchaudio: filename=torchaudio-2.2.0a0+1980f8a-cp310-cp310-linux_x86_64.whl size=2311968 sha256=c8455f252c3f1e71883af4539fb3f2ca07e94d42f07c6e2e42ab88d819752050 2024-05-19T18:21:16.7942603Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/3c/69/04/2f8953929b90f7f98a00671a3b24cfeb70fc70e57328823e5c 2024-05-19T18:21:16.7969599Z Successfully built torchaudio 2024-05-19T18:21:18.0032295Z Installing collected packages: torchaudio 2024-05-19T18:21:18.1971577Z Successfully installed torchaudio-2.2.0a0+1980f8a 2024-05-19T18:21:18.3277032Z + install_torchtext 2024-05-19T18:21:18.3277431Z + local data_commit 2024-05-19T18:21:18.3277754Z + local text_commit 2024-05-19T18:21:18.3279379Z ++ get_pinned_commit data 2024-05-19T18:21:18.3279790Z ++ cat .github/ci_commit_pins/data.txt 2024-05-19T18:21:18.3289240Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-05-19T18:21:18.3290489Z ++ get_pinned_commit text 2024-05-19T18:21:18.3291063Z ++ cat .github/ci_commit_pins/text.txt 2024-05-19T18:21:18.3299565Z + text_commit=b0ebddc648d279826089db91775375221777a2db 2024-05-19T18:21:18.3300722Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-05-19T18:21:18.3302155Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-05-19T18:21:18.6391213Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-05-19T18:21:18.6394564Z Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-jiyzsnmm 2024-05-19T18:21:18.6409465Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-jiyzsnmm 2024-05-19T18:21:19.0816291Z Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f' 2024-05-19T18:21:19.0834135Z Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-05-19T18:21:19.3180080Z Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-05-19T18:21:19.4859227Z Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-05-19T18:21:19.4860819Z Running command git submodule update --init --recursive -q 2024-05-19T18:22:22.8551079Z Preparing metadata (setup.py) ... [?25l- done 2024-05-19T18:22:22.8601040Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.18) 2024-05-19T18:22:22.8603900Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.31.0) 2024-05-19T18:22:22.8609744Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.4.0a0+gitc8c7ddf) 2024-05-19T18:22:22.8669326Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1) 2024-05-19T18:22:22.8674884Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.11.0) 2024-05-19T18:22:22.8677809Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.12) 2024-05-19T18:22:22.8680944Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8) 2024-05-19T18:22:22.8684321Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4) 2024-05-19T18:22:22.8687578Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.2.0) 2024-05-19T18:22:22.8871011Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2) 2024-05-19T18:22:22.8874900Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7) 2024-05-19T18:22:22.8881844Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.2.2) 2024-05-19T18:22:22.9333361Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5) 2024-05-19T18:22:22.9492860Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0) 2024-05-19T18:22:22.9571929Z Building wheels for collected packages: torchdata 2024-05-19T18:22:24.5781133Z Building wheel for torchdata (setup.py) ... [?25l- \ | done 2024-05-19T18:22:24.5788671Z [?25h Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182956 sha256=44e9f2067dd764ecf270e748713e2d980e254e896759ccafe2f52ddcb3347c43 2024-05-19T18:22:24.5791773Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce 2024-05-19T18:22:24.5819049Z Successfully built torchdata 2024-05-19T18:22:25.7560070Z Installing collected packages: torchdata 2024-05-19T18:22:25.8422272Z Successfully installed torchdata-0.7.0a0+11bb5b8 2024-05-19T18:22:28.3015590Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-05-19T18:22:28.3017061Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-05-19T18:22:28.6116479Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-05-19T18:22:28.6119880Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-o88reu63 2024-05-19T18:22:28.6134654Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-o88reu63 2024-05-19T18:22:29.4231754Z Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db' 2024-05-19T18:22:29.4244679Z Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db 2024-05-19T18:22:29.8817200Z Running command git checkout -q b0ebddc648d279826089db91775375221777a2db 2024-05-19T18:22:39.3905462Z Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db 2024-05-19T18:22:39.3906499Z Running command git submodule update --init --recursive -q 2024-05-19T18:22:45.4488864Z Preparing metadata (setup.py) ... [?25l- done 2024-05-19T18:22:45.4540815Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.4) 2024-05-19T18:22:45.4543400Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.31.0) 2024-05-19T18:22:45.4547258Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.4.0a0+gitc8c7ddf) 2024-05-19T18:22:45.4549249Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.21.2) 2024-05-19T18:22:45.4552708Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8) 2024-05-19T18:22:45.4612442Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2) 2024-05-19T18:22:45.4618554Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7) 2024-05-19T18:22:45.4623478Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.18) 2024-05-19T18:22:45.4628329Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.2.2) 2024-05-19T18:22:45.4684287Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1) 2024-05-19T18:22:45.4689722Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.11.0) 2024-05-19T18:22:45.4692181Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.12) 2024-05-19T18:22:45.4695025Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8) 2024-05-19T18:22:45.4697580Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4) 2024-05-19T18:22:45.4700498Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.2.0) 2024-05-19T18:22:45.5374496Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5) 2024-05-19T18:22:45.5536403Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0) 2024-05-19T18:22:45.5616783Z Building wheels for collected packages: torchtext 2024-05-19T18:23:21.4892259Z Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-05-19T18:23:21.4919780Z [?25h Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2045362 sha256=073f6576cfd97037509e02df8f54058444e185547df8bb9069a56699765055f5 2024-05-19T18:23:21.4922280Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf 2024-05-19T18:23:21.4946181Z Successfully built torchtext 2024-05-19T18:23:22.6675111Z Installing collected packages: torchtext 2024-05-19T18:23:22.7660069Z Successfully installed torchtext-0.16.0a0+b0ebddc 2024-05-19T18:23:22.8870597Z + install_torchvision 2024-05-19T18:23:22.8871127Z + local orig_preload 2024-05-19T18:23:22.8871610Z + local commit 2024-05-19T18:23:22.8872398Z ++ get_pinned_commit vision 2024-05-19T18:23:22.8873150Z ++ cat .github/ci_commit_pins/vision.txt 2024-05-19T18:23:22.8884664Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:23:22.8885227Z + orig_preload= 2024-05-19T18:23:22.8885671Z + '[' -n '' ']' 2024-05-19T18:23:22.8886566Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:23:22.8888109Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:23:23.2008312Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:23:23.2011599Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-s1u6fqau 2024-05-19T18:23:23.2028452Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-s1u6fqau 2024-05-19T18:23:24.5600142Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-05-19T18:23:24.5618964Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:23:25.7489441Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:23:25.9463846Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:23:28.0584385Z Preparing metadata (setup.py) ... [?25l- \ done 2024-05-19T18:23:28.0634305Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-05-19T18:23:28.0637348Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.4.0a0+gitc8c7ddf) 2024-05-19T18:23:28.0644378Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-05-19T18:23:28.0848690Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-05-19T18:23:28.0857025Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.11.0) 2024-05-19T18:23:28.0859631Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12) 2024-05-19T18:23:28.0863064Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-05-19T18:23:28.0866315Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-05-19T18:23:28.0869554Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-05-19T18:23:28.1293610Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-05-19T18:23:28.1446507Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-05-19T18:23:28.1520980Z Building wheels for collected packages: torchvision 2024-05-19T18:24:37.2533365Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done 2024-05-19T18:24:37.2560212Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2025888 sha256=9c08d310c1fa2e74f54f40791e093128efef5f2047a9513111a177b35aaaf180 2024-05-19T18:24:37.2563023Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-05-19T18:24:37.2597815Z Successfully built torchvision 2024-05-19T18:24:38.4385280Z Installing collected packages: torchvision 2024-05-19T18:24:38.8138439Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-05-19T18:24:38.9244801Z + '[' -n '' ']' 2024-05-19T18:24:38.9245178Z + id=1 2024-05-19T18:24:38.9245563Z + pip_install opencv-python==4.8.0.74 2024-05-19T18:24:38.9246170Z + pip install --progress-bar off opencv-python==4.8.0.74 2024-05-19T18:24:39.3480418Z Collecting opencv-python==4.8.0.74 2024-05-19T18:24:39.3681443Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2024-05-19T18:24:39.3852773Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.21.2) 2024-05-19T18:24:39.3904680Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2024-05-19T18:24:41.2748196Z Installing collected packages: opencv-python 2024-05-19T18:24:42.0906171Z Successfully installed opencv-python-4.8.0.74 2024-05-19T18:24:42.1889990Z + [[ dynamic_inductor_torchbench == *inductor_torchbench_smoketest_perf* ]] 2024-05-19T18:24:42.1890799Z + [[ dynamic_inductor_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]] 2024-05-19T18:24:42.1891767Z + [[ dynamic_inductor_torchbench == *torchbench_gcp_smoketest* ]] 2024-05-19T18:24:42.1892503Z + checkout_install_torchbench 2024-05-19T18:24:42.1892897Z + local commit 2024-05-19T18:24:42.1893939Z ++ get_pinned_commit torchbench 2024-05-19T18:24:42.1894441Z ++ cat .github/ci_commit_pins/torchbench.txt 2024-05-19T18:24:42.1903605Z + commit=d6015d42d9a1834bc7595c4bd6852562fb80b30b 2024-05-19T18:24:42.1904274Z + git clone https://github.com/pytorch/benchmark torchbench 2024-05-19T18:24:42.1911899Z Cloning into 'torchbench'... 2024-05-19T18:24:42.3598072Z remote: Enumerating objects: 29198, done. 2024-05-19T18:24:42.3601613Z remote: Counting objects: 0% (1/4428) 2024-05-19T18:24:42.3602665Z remote: Counting objects: 1% (45/4428) 2024-05-19T18:24:42.3603640Z remote: Counting objects: 2% (89/4428) 2024-05-19T18:24:42.3604166Z remote: Counting objects: 3% (133/4428) 2024-05-19T18:24:42.3604764Z remote: Counting objects: 4% (178/4428) 2024-05-19T18:24:42.3605659Z remote: Counting objects: 5% (222/4428) 2024-05-19T18:24:42.3606213Z remote: Counting objects: 6% (266/4428) 2024-05-19T18:24:42.3607111Z remote: Counting objects: 7% (310/4428) 2024-05-19T18:24:42.3607661Z remote: Counting objects: 8% (355/4428) 2024-05-19T18:24:42.3608699Z remote: Counting objects: 9% (399/4428) 2024-05-19T18:24:42.3609381Z remote: Counting objects: 10% (443/4428) 2024-05-19T18:24:42.3609900Z remote: Counting objects: 11% (488/4428) 2024-05-19T18:24:42.3610474Z remote: Counting objects: 12% (532/4428) 2024-05-19T18:24:42.3611122Z remote: Counting objects: 13% (576/4428) 2024-05-19T18:24:42.3611696Z remote: Counting objects: 14% (620/4428) 2024-05-19T18:24:42.3612214Z remote: Counting objects: 15% (665/4428) 2024-05-19T18:24:42.3612825Z remote: Counting objects: 16% (709/4428) 2024-05-19T18:24:42.3613518Z remote: Counting objects: 17% (753/4428) 2024-05-19T18:24:42.3614173Z remote: Counting objects: 18% (798/4428) 2024-05-19T18:24:42.3614878Z remote: Counting objects: 19% (842/4428) 2024-05-19T18:24:42.3615535Z remote: Counting objects: 20% (886/4428) 2024-05-19T18:24:42.3616281Z remote: Counting objects: 21% (930/4428) 2024-05-19T18:24:42.3616927Z remote: Counting objects: 22% (975/4428) 2024-05-19T18:24:42.3617572Z remote: Counting objects: 23% (1019/4428) 2024-05-19T18:24:42.3618319Z remote: Counting objects: 24% (1063/4428) 2024-05-19T18:24:42.3618985Z remote: Counting objects: 25% (1107/4428) 2024-05-19T18:24:42.3619599Z remote: Counting objects: 26% (1152/4428) 2024-05-19T18:24:42.3620149Z remote: Counting objects: 27% (1196/4428) 2024-05-19T18:24:42.3620867Z remote: Counting objects: 28% (1240/4428) 2024-05-19T18:24:42.3621541Z remote: Counting objects: 29% (1285/4428) 2024-05-19T18:24:42.3622132Z remote: Counting objects: 30% (1329/4428) 2024-05-19T18:24:42.3622710Z remote: Counting objects: 31% (1373/4428) 2024-05-19T18:24:42.3623256Z remote: Counting objects: 32% (1417/4428) 2024-05-19T18:24:42.3623790Z remote: Counting objects: 33% (1462/4428) 2024-05-19T18:24:42.3624308Z remote: Counting objects: 34% (1506/4428) 2024-05-19T18:24:42.3624828Z remote: Counting objects: 35% (1550/4428) 2024-05-19T18:24:42.3625353Z remote: Counting objects: 36% (1595/4428) 2024-05-19T18:24:42.3625942Z remote: Counting objects: 37% (1639/4428) 2024-05-19T18:24:42.3626482Z remote: Counting objects: 38% (1683/4428) 2024-05-19T18:24:42.3627065Z remote: Counting objects: 39% (1727/4428) 2024-05-19T18:24:42.3627679Z remote: Counting objects: 40% (1772/4428) 2024-05-19T18:24:42.3628381Z remote: Counting objects: 41% (1816/4428) 2024-05-19T18:24:42.3628960Z remote: Counting objects: 42% (1860/4428) 2024-05-19T18:24:42.3629497Z remote: Counting objects: 43% (1905/4428) 2024-05-19T18:24:42.3630251Z remote: Counting objects: 44% (1949/4428) 2024-05-19T18:24:42.3630830Z remote: Counting objects: 45% (1993/4428) 2024-05-19T18:24:42.3631432Z remote: Counting objects: 46% (2037/4428) 2024-05-19T18:24:42.3631948Z remote: Counting objects: 47% (2082/4428) 2024-05-19T18:24:42.3632465Z remote: Counting objects: 48% (2126/4428) 2024-05-19T18:24:42.3633235Z remote: Counting objects: 49% (2170/4428) 2024-05-19T18:24:42.3633875Z remote: Counting objects: 50% (2214/4428) 2024-05-19T18:24:42.3634547Z remote: Counting objects: 51% (2259/4428) 2024-05-19T18:24:42.3635227Z remote: Counting objects: 52% (2303/4428) 2024-05-19T18:24:42.3635753Z remote: Counting objects: 53% (2347/4428) 2024-05-19T18:24:42.3636265Z remote: Counting objects: 54% (2392/4428) 2024-05-19T18:24:42.3636785Z remote: Counting objects: 55% (2436/4428) 2024-05-19T18:24:42.3637590Z remote: Counting objects: 56% (2480/4428) 2024-05-19T18:24:42.3638131Z remote: Counting objects: 57% (2524/4428) 2024-05-19T18:24:42.3638791Z remote: Counting objects: 58% (2569/4428) 2024-05-19T18:24:42.3639363Z remote: Counting objects: 59% (2613/4428) 2024-05-19T18:24:42.3639876Z remote: Counting objects: 60% (2657/4428) 2024-05-19T18:24:42.3640627Z remote: Counting objects: 61% (2702/4428) 2024-05-19T18:24:42.3641305Z remote: Counting objects: 62% (2746/4428) 2024-05-19T18:24:42.3642007Z remote: Counting objects: 63% (2790/4428) 2024-05-19T18:24:42.3642616Z remote: Counting objects: 64% (2834/4428) 2024-05-19T18:24:42.3643147Z remote: Counting objects: 65% (2879/4428) 2024-05-19T18:24:42.3643698Z remote: Counting objects: 66% (2923/4428) 2024-05-19T18:24:42.3644411Z remote: Counting objects: 67% (2967/4428) 2024-05-19T18:24:42.3645032Z remote: Counting objects: 68% (3012/4428) 2024-05-19T18:24:42.3645562Z remote: Counting objects: 69% (3056/4428) 2024-05-19T18:24:42.3646091Z remote: Counting objects: 70% (3100/4428) 2024-05-19T18:24:42.3646615Z remote: Counting objects: 71% (3144/4428) 2024-05-19T18:24:42.3647138Z remote: Counting objects: 72% (3189/4428) 2024-05-19T18:24:42.3647661Z remote: Counting objects: 73% (3233/4428) 2024-05-19T18:24:42.3648178Z remote: Counting objects: 74% (3277/4428) 2024-05-19T18:24:42.3648833Z remote: Counting objects: 75% (3321/4428) 2024-05-19T18:24:42.3649366Z remote: Counting objects: 76% (3366/4428) 2024-05-19T18:24:42.3649909Z remote: Counting objects: 77% (3410/4428) 2024-05-19T18:24:42.3650427Z remote: Counting objects: 78% (3454/4428) 2024-05-19T18:24:42.3650954Z remote: Counting objects: 79% (3499/4428) 2024-05-19T18:24:42.3651485Z remote: Counting objects: 80% (3543/4428) 2024-05-19T18:24:42.3652008Z remote: Counting objects: 81% (3587/4428) 2024-05-19T18:24:42.3652522Z remote: Counting objects: 82% (3631/4428) 2024-05-19T18:24:42.3653054Z remote: Counting objects: 83% (3676/4428) 2024-05-19T18:24:42.3653589Z remote: Counting objects: 84% (3720/4428) 2024-05-19T18:24:42.3654112Z remote: Counting objects: 85% (3764/4428) 2024-05-19T18:24:42.3654625Z remote: Counting objects: 86% (3809/4428) 2024-05-19T18:24:42.3655149Z remote: Counting objects: 87% (3853/4428) 2024-05-19T18:24:42.3655672Z remote: Counting objects: 88% (3897/4428) 2024-05-19T18:24:42.3656184Z remote: Counting objects: 89% (3941/4428) 2024-05-19T18:24:42.3656717Z remote: Counting objects: 90% (3986/4428) 2024-05-19T18:24:42.3657326Z remote: Counting objects: 91% (4030/4428) 2024-05-19T18:24:42.3657851Z remote: Counting objects: 92% (4074/4428) 2024-05-19T18:24:42.3658370Z remote: Counting objects: 93% (4119/4428) 2024-05-19T18:24:42.3658894Z remote: Counting objects: 94% (4163/4428) 2024-05-19T18:24:42.3659423Z remote: Counting objects: 95% (4207/4428) 2024-05-19T18:24:42.3659952Z remote: Counting objects: 96% (4251/4428) 2024-05-19T18:24:42.3660471Z remote: Counting objects: 97% (4296/4428) 2024-05-19T18:24:42.3660992Z remote: Counting objects: 98% (4340/4428) 2024-05-19T18:24:42.3661516Z remote: Counting objects: 99% (4384/4428) 2024-05-19T18:24:42.3662050Z remote: Counting objects: 100% (4428/4428) 2024-05-19T18:24:42.3662607Z remote: Counting objects: 100% (4428/4428), done. 2024-05-19T18:24:42.3663184Z remote: Compressing objects: 0% (1/839) 2024-05-19T18:24:42.3684842Z remote: Compressing objects: 1% (9/839) 2024-05-19T18:24:42.3711883Z remote: Compressing objects: 2% (17/839) 2024-05-19T18:24:42.3782645Z remote: Compressing objects: 3% (26/839) 2024-05-19T18:24:42.3788065Z remote: Compressing objects: 4% (34/839) 2024-05-19T18:24:42.3819153Z remote: Compressing objects: 5% (42/839) 2024-05-19T18:24:42.3837619Z remote: Compressing objects: 6% (51/839) 2024-05-19T18:24:42.3861221Z remote: Compressing objects: 7% (59/839) 2024-05-19T18:24:42.3875141Z remote: Compressing objects: 8% (68/839) 2024-05-19T18:24:42.3895429Z remote: Compressing objects: 9% (76/839) 2024-05-19T18:24:42.3906208Z remote: Compressing objects: 10% (84/839) 2024-05-19T18:24:42.3965339Z remote: Compressing objects: 11% (93/839) 2024-05-19T18:24:42.4049784Z remote: Compressing objects: 12% (101/839) 2024-05-19T18:24:42.4105501Z remote: Compressing objects: 13% (110/839) 2024-05-19T18:24:42.4198626Z remote: Compressing objects: 14% (118/839) 2024-05-19T18:24:42.4268082Z remote: Compressing objects: 15% (126/839) 2024-05-19T18:24:42.4331895Z remote: Compressing objects: 16% (135/839) 2024-05-19T18:24:42.4402218Z remote: Compressing objects: 17% (143/839) 2024-05-19T18:24:42.4459455Z remote: Compressing objects: 18% (152/839) 2024-05-19T18:24:42.4505121Z remote: Compressing objects: 19% (160/839) 2024-05-19T18:24:42.4565496Z remote: Compressing objects: 20% (168/839) 2024-05-19T18:24:42.4612953Z remote: Compressing objects: 21% (177/839) 2024-05-19T18:24:42.4658674Z remote: Compressing objects: 22% (185/839) 2024-05-19T18:24:42.4688980Z remote: Compressing objects: 23% (193/839) 2024-05-19T18:24:42.4713493Z remote: Compressing objects: 24% (202/839) 2024-05-19T18:24:42.4751730Z remote: Compressing objects: 25% (210/839) 2024-05-19T18:24:42.4787170Z remote: Compressing objects: 26% (219/839) 2024-05-19T18:24:42.4806019Z remote: Compressing objects: 27% (227/839) 2024-05-19T18:24:42.4835465Z remote: Compressing objects: 28% (235/839) 2024-05-19T18:24:42.4847934Z remote: Compressing objects: 29% (244/839) 2024-05-19T18:24:42.4873654Z remote: Compressing objects: 30% (252/839) 2024-05-19T18:24:42.4894315Z remote: Compressing objects: 31% (261/839) 2024-05-19T18:24:42.4906328Z remote: Compressing objects: 32% (269/839) 2024-05-19T18:24:42.4922520Z remote: Compressing objects: 33% (277/839) 2024-05-19T18:24:42.4931414Z remote: Compressing objects: 34% (286/839) 2024-05-19T18:24:42.4942944Z remote: Compressing objects: 35% (294/839) 2024-05-19T18:24:42.4949658Z remote: Compressing objects: 36% (303/839) 2024-05-19T18:24:42.4950824Z remote: Compressing objects: 37% (311/839) 2024-05-19T18:24:42.4951417Z remote: Compressing objects: 38% (319/839) 2024-05-19T18:24:42.4951965Z remote: Compressing objects: 39% (328/839) 2024-05-19T18:24:42.4952670Z remote: Compressing objects: 40% (336/839) 2024-05-19T18:24:42.4953426Z remote: Compressing objects: 41% (344/839) 2024-05-19T18:24:42.4953998Z remote: Compressing objects: 42% (353/839) 2024-05-19T18:24:42.4956164Z remote: Compressing objects: 43% (361/839) 2024-05-19T18:24:42.4956721Z remote: Compressing objects: 44% (370/839) 2024-05-19T18:24:42.4960648Z remote: Compressing objects: 45% (378/839) 2024-05-19T18:24:42.4964392Z remote: Compressing objects: 46% (386/839) 2024-05-19T18:24:42.4970367Z remote: Compressing objects: 47% (395/839) 2024-05-19T18:24:42.4975957Z remote: Compressing objects: 48% (403/839) 2024-05-19T18:24:42.4979869Z remote: Compressing objects: 49% (412/839) 2024-05-19T18:24:42.4985485Z remote: Compressing objects: 50% (420/839) 2024-05-19T18:24:42.4987284Z remote: Compressing objects: 51% (428/839) 2024-05-19T18:24:42.4987840Z remote: Compressing objects: 52% (437/839) 2024-05-19T18:24:42.4989766Z remote: Compressing objects: 53% (445/839) 2024-05-19T18:24:42.4994317Z remote: Compressing objects: 54% (454/839) 2024-05-19T18:24:42.4996882Z remote: Compressing objects: 55% (462/839) 2024-05-19T18:24:42.5001112Z remote: Compressing objects: 56% (470/839) 2024-05-19T18:24:42.5004090Z remote: Compressing objects: 57% (479/839) 2024-05-19T18:24:42.5008400Z remote: Compressing objects: 58% (487/839) 2024-05-19T18:24:42.5011975Z remote: Compressing objects: 59% (496/839) 2024-05-19T18:24:42.5013916Z remote: Compressing objects: 60% (504/839) 2024-05-19T18:24:42.5016724Z remote: Compressing objects: 61% (512/839) 2024-05-19T18:24:42.5020278Z remote: Compressing objects: 62% (521/839) 2024-05-19T18:24:42.5023460Z remote: Compressing objects: 63% (529/839) 2024-05-19T18:24:42.5026431Z remote: Compressing objects: 64% (537/839) 2024-05-19T18:24:42.5029069Z remote: Compressing objects: 65% (546/839) 2024-05-19T18:24:42.5032352Z remote: Compressing objects: 66% (554/839) 2024-05-19T18:24:42.5034677Z remote: Compressing objects: 67% (563/839) 2024-05-19T18:24:42.5037685Z remote: Compressing objects: 68% (571/839) 2024-05-19T18:24:42.5038253Z remote: Compressing objects: 69% (579/839) 2024-05-19T18:24:42.5038804Z remote: Compressing objects: 70% (588/839) 2024-05-19T18:24:42.5040687Z remote: Compressing objects: 71% (596/839) 2024-05-19T18:24:42.5041310Z remote: Compressing objects: 72% (605/839) 2024-05-19T18:24:42.5043102Z remote: Compressing objects: 73% (613/839) 2024-05-19T18:24:42.5043865Z remote: Compressing objects: 74% (621/839) 2024-05-19T18:24:42.5044584Z remote: Compressing objects: 75% (630/839) 2024-05-19T18:24:42.5045326Z remote: Compressing objects: 76% (638/839) 2024-05-19T18:24:42.5045885Z remote: Compressing objects: 77% (647/839) 2024-05-19T18:24:42.5046601Z remote: Compressing objects: 78% (655/839) 2024-05-19T18:24:42.5047202Z remote: Compressing objects: 79% (663/839) 2024-05-19T18:24:42.5047738Z remote: Compressing objects: 80% (672/839) 2024-05-19T18:24:42.5048273Z remote: Compressing objects: 81% (680/839) 2024-05-19T18:24:42.5048930Z remote: Compressing objects: 82% (688/839) 2024-05-19T18:24:42.5049469Z remote: Compressing objects: 83% (697/839) 2024-05-19T18:24:42.5050019Z remote: Compressing objects: 84% (705/839) 2024-05-19T18:24:42.5050568Z remote: Compressing objects: 85% (714/839) 2024-05-19T18:24:42.5051115Z remote: Compressing objects: 86% (722/839) 2024-05-19T18:24:42.5051841Z remote: Compressing objects: 87% (730/839) 2024-05-19T18:24:42.5052481Z remote: Compressing objects: 88% (739/839) 2024-05-19T18:24:42.5053033Z remote: Compressing objects: 89% (747/839) 2024-05-19T18:24:42.5054127Z remote: Compressing objects: 90% (756/839) 2024-05-19T18:24:42.5056667Z remote: Compressing objects: 91% (764/839) 2024-05-19T18:24:42.5057417Z remote: Compressing objects: 92% (772/839) 2024-05-19T18:24:42.5058134Z remote: Compressing objects: 93% (781/839) 2024-05-19T18:24:42.5058789Z remote: Compressing objects: 94% (789/839) 2024-05-19T18:24:42.5060972Z remote: Compressing objects: 95% (798/839) 2024-05-19T18:24:42.5067141Z remote: Compressing objects: 96% (806/839) 2024-05-19T18:24:42.5068074Z remote: Compressing objects: 97% (814/839) 2024-05-19T18:24:42.5068796Z remote: Compressing objects: 98% (823/839) 2024-05-19T18:24:42.5069335Z remote: Compressing objects: 99% (831/839) 2024-05-19T18:24:42.5069860Z remote: Compressing objects: 100% (839/839) 2024-05-19T18:24:42.5070590Z remote: Compressing objects: 100% (839/839), done. 2024-05-19T18:24:42.5141038Z Receiving objects: 0% (1/29198) 2024-05-19T18:24:42.5186610Z Receiving objects: 1% (292/29198) 2024-05-19T18:24:42.5289648Z Receiving objects: 2% (584/29198) 2024-05-19T18:24:42.5317777Z Receiving objects: 3% (876/29198) 2024-05-19T18:24:42.5416560Z Receiving objects: 4% (1168/29198) 2024-05-19T18:24:42.5455973Z Receiving objects: 5% (1460/29198) 2024-05-19T18:24:42.5541439Z Receiving objects: 6% (1752/29198) 2024-05-19T18:24:42.5566396Z Receiving objects: 7% (2044/29198) 2024-05-19T18:24:42.5595234Z Receiving objects: 8% (2336/29198) 2024-05-19T18:24:42.5628164Z Receiving objects: 9% (2628/29198) 2024-05-19T18:24:42.5656820Z Receiving objects: 10% (2920/29198) 2024-05-19T18:24:42.5686296Z Receiving objects: 11% (3212/29198) 2024-05-19T18:24:42.5716471Z Receiving objects: 12% (3504/29198) 2024-05-19T18:24:42.5753266Z Receiving objects: 13% (3796/29198) 2024-05-19T18:24:42.5784074Z Receiving objects: 14% (4088/29198) 2024-05-19T18:24:42.5813259Z Receiving objects: 15% (4380/29198) 2024-05-19T18:24:42.5840486Z Receiving objects: 16% (4672/29198) 2024-05-19T18:24:42.5879583Z Receiving objects: 17% (4964/29198) 2024-05-19T18:24:42.5930895Z Receiving objects: 18% (5256/29198) 2024-05-19T18:24:42.6040692Z Receiving objects: 19% (5548/29198) 2024-05-19T18:24:42.6173445Z Receiving objects: 20% (5840/29198) 2024-05-19T18:24:42.6214193Z Receiving objects: 21% (6132/29198) 2024-05-19T18:24:42.6240464Z Receiving objects: 22% (6424/29198) 2024-05-19T18:24:42.6253457Z Receiving objects: 23% (6716/29198) 2024-05-19T18:24:42.6276838Z Receiving objects: 24% (7008/29198) 2024-05-19T18:24:42.6305019Z Receiving objects: 25% (7300/29198) 2024-05-19T18:24:42.6316951Z Receiving objects: 26% (7592/29198) 2024-05-19T18:24:42.6333180Z Receiving objects: 27% (7884/29198) 2024-05-19T18:24:42.6350195Z Receiving objects: 28% (8176/29198) 2024-05-19T18:24:42.6966488Z Receiving objects: 29% (8468/29198) 2024-05-19T18:24:42.6982409Z Receiving objects: 30% (8760/29198) 2024-05-19T18:24:42.7517431Z Receiving objects: 31% (9052/29198) 2024-05-19T18:24:43.5085403Z Receiving objects: 32% (9344/29198) 2024-05-19T18:24:43.6320577Z Receiving objects: 32% (9566/29198), 77.07 MiB | 77.07 MiB/s 2024-05-19T18:24:44.1608166Z Receiving objects: 33% (9636/29198), 77.07 MiB | 77.07 MiB/s 2024-05-19T18:24:44.2257348Z Receiving objects: 34% (9928/29198), 115.82 MiB | 77.21 MiB/s 2024-05-19T18:24:44.2890827Z Receiving objects: 35% (10220/29198), 115.82 MiB | 77.21 MiB/s 2024-05-19T18:24:44.3524441Z Receiving objects: 36% (10512/29198), 115.82 MiB | 77.21 MiB/s 2024-05-19T18:24:44.4149596Z Receiving objects: 37% (10804/29198), 115.82 MiB | 77.21 MiB/s 2024-05-19T18:24:44.4930610Z Receiving objects: 38% (11096/29198), 115.82 MiB | 77.21 MiB/s 2024-05-19T18:24:44.5085061Z Receiving objects: 39% (11388/29198), 115.82 MiB | 77.21 MiB/s 2024-05-19T18:24:44.5584175Z Receiving objects: 39% (11450/29198), 155.59 MiB | 77.79 MiB/s 2024-05-19T18:24:44.6224951Z Receiving objects: 40% (11680/29198), 155.59 MiB | 77.79 MiB/s 2024-05-19T18:24:44.6856790Z Receiving objects: 41% (11972/29198), 155.59 MiB | 77.79 MiB/s 2024-05-19T18:24:44.8914506Z Receiving objects: 42% (12264/29198), 155.59 MiB | 77.79 MiB/s 2024-05-19T18:24:44.9485597Z Receiving objects: 43% (12556/29198), 155.59 MiB | 77.79 MiB/s 2024-05-19T18:24:44.9775419Z Receiving objects: 44% (12848/29198), 155.59 MiB | 77.79 MiB/s 2024-05-19T18:24:45.1153681Z Receiving objects: 45% (13140/29198), 155.59 MiB | 77.79 MiB/s 2024-05-19T18:24:45.3711921Z Receiving objects: 46% (13432/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4090281Z Receiving objects: 47% (13724/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4104184Z Receiving objects: 48% (14016/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4112513Z Receiving objects: 49% (14308/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4129913Z Receiving objects: 50% (14599/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4138878Z Receiving objects: 51% (14891/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4268106Z Receiving objects: 52% (15183/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4273921Z Receiving objects: 53% (15475/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4278992Z Receiving objects: 54% (15767/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4284350Z Receiving objects: 55% (16059/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4291550Z Receiving objects: 56% (16351/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4301819Z Receiving objects: 57% (16643/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4310254Z Receiving objects: 58% (16935/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4725889Z Receiving objects: 59% (17227/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4733978Z Receiving objects: 60% (17519/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4745264Z Receiving objects: 61% (17811/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4756042Z Receiving objects: 62% (18103/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4768152Z Receiving objects: 63% (18395/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4776536Z Receiving objects: 64% (18687/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4794385Z Receiving objects: 65% (18979/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4799666Z Receiving objects: 66% (19271/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4807030Z Receiving objects: 67% (19563/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.4814701Z Receiving objects: 68% (19855/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.5085484Z Receiving objects: 69% (20147/29198), 190.84 MiB | 76.34 MiB/s 2024-05-19T18:24:45.6853567Z Receiving objects: 69% (20327/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7065914Z Receiving objects: 70% (20439/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7072416Z Receiving objects: 71% (20731/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7082507Z Receiving objects: 72% (21023/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7093312Z Receiving objects: 73% (21315/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7099716Z Receiving objects: 74% (21607/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7226988Z Receiving objects: 75% (21899/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7232351Z Receiving objects: 76% (22191/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7239743Z Receiving objects: 77% (22483/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7246247Z Receiving objects: 78% (22775/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7251608Z Receiving objects: 79% (23067/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7257943Z Receiving objects: 80% (23359/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7370571Z Receiving objects: 81% (23651/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7386798Z Receiving objects: 82% (23943/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7397884Z Receiving objects: 83% (24235/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7417127Z Receiving objects: 84% (24527/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7427840Z Receiving objects: 85% (24819/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7432714Z Receiving objects: 86% (25111/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7435798Z Receiving objects: 87% (25403/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7440188Z Receiving objects: 88% (25695/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7444986Z Receiving objects: 89% (25987/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7448828Z Receiving objects: 90% (26279/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7451496Z Receiving objects: 91% (26571/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7455574Z Receiving objects: 92% (26863/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7461046Z Receiving objects: 93% (27155/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7530438Z Receiving objects: 94% (27447/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7576991Z Receiving objects: 95% (27739/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7612509Z Receiving objects: 96% (28031/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7631484Z Receiving objects: 97% (28323/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7647983Z Receiving objects: 98% (28615/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7678446Z Receiving objects: 99% (28907/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7679595Z remote: Total 29198 (delta 3940), reused 3815 (delta 3588), pack-reused 24770 2024-05-19T18:24:45.7696339Z Receiving objects: 100% (29198/29198), 229.77 MiB | 76.59 MiB/s 2024-05-19T18:24:45.7697031Z Receiving objects: 100% (29198/29198), 244.92 MiB | 75.11 MiB/s, done. 2024-05-19T18:24:45.7724105Z Resolving deltas: 0% (0/15780) 2024-05-19T18:24:45.7745247Z Resolving deltas: 1% (178/15780) 2024-05-19T18:24:45.7781312Z Resolving deltas: 2% (317/15780) 2024-05-19T18:24:45.7821673Z Resolving deltas: 3% (486/15780) 2024-05-19T18:24:45.7854579Z Resolving deltas: 4% (640/15780) 2024-05-19T18:24:45.7869180Z Resolving deltas: 5% (795/15780) 2024-05-19T18:24:45.7878558Z Resolving deltas: 6% (955/15780) 2024-05-19T18:24:45.7890287Z Resolving deltas: 7% (1107/15780) 2024-05-19T18:24:45.8003258Z Resolving deltas: 8% (1275/15780) 2024-05-19T18:24:45.8016848Z Resolving deltas: 9% (1429/15780) 2024-05-19T18:24:45.8056272Z Resolving deltas: 10% (1578/15780) 2024-05-19T18:24:45.8060484Z Resolving deltas: 11% (1870/15780) 2024-05-19T18:24:45.8070896Z Resolving deltas: 12% (1937/15780) 2024-05-19T18:24:45.8077502Z Resolving deltas: 13% (2078/15780) 2024-05-19T18:24:45.8097148Z Resolving deltas: 14% (2210/15780) 2024-05-19T18:24:45.8120525Z Resolving deltas: 15% (2370/15780) 2024-05-19T18:24:45.8134348Z Resolving deltas: 16% (2532/15780) 2024-05-19T18:24:45.8150131Z Resolving deltas: 17% (2694/15780) 2024-05-19T18:24:45.8159741Z Resolving deltas: 18% (2842/15780) 2024-05-19T18:24:45.8169353Z Resolving deltas: 19% (2999/15780) 2024-05-19T18:24:45.8179323Z Resolving deltas: 20% (3160/15780) 2024-05-19T18:24:45.8185531Z Resolving deltas: 21% (3316/15780) 2024-05-19T18:24:45.8194020Z Resolving deltas: 22% (3477/15780) 2024-05-19T18:24:45.8202344Z Resolving deltas: 23% (3630/15780) 2024-05-19T18:24:45.8211929Z Resolving deltas: 24% (3788/15780) 2024-05-19T18:24:45.8228396Z Resolving deltas: 25% (3952/15780) 2024-05-19T18:24:45.8238616Z Resolving deltas: 26% (4104/15780) 2024-05-19T18:24:45.8246578Z Resolving deltas: 27% (4263/15780) 2024-05-19T18:24:45.8252357Z Resolving deltas: 28% (4420/15780) 2024-05-19T18:24:45.8257706Z Resolving deltas: 29% (4585/15780) 2024-05-19T18:24:45.8265236Z Resolving deltas: 30% (4734/15780) 2024-05-19T18:24:45.8275831Z Resolving deltas: 31% (4944/15780) 2024-05-19T18:24:45.8283845Z Resolving deltas: 32% (5052/15780) 2024-05-19T18:24:45.8295149Z Resolving deltas: 33% (5211/15780) 2024-05-19T18:24:45.8304493Z Resolving deltas: 34% (5371/15780) 2024-05-19T18:24:45.8310708Z Resolving deltas: 35% (5603/15780) 2024-05-19T18:24:45.8316306Z Resolving deltas: 36% (5697/15780) 2024-05-19T18:24:45.8322639Z Resolving deltas: 37% (5840/15780) 2024-05-19T18:24:45.8328344Z Resolving deltas: 38% (6012/15780) 2024-05-19T18:24:45.8334542Z Resolving deltas: 39% (6177/15780) 2024-05-19T18:24:45.8341270Z Resolving deltas: 40% (6355/15780) 2024-05-19T18:24:45.8355003Z Resolving deltas: 41% (6510/15780) 2024-05-19T18:24:45.8359667Z Resolving deltas: 42% (6714/15780) 2024-05-19T18:24:45.8364388Z Resolving deltas: 43% (6799/15780) 2024-05-19T18:24:45.8371240Z Resolving deltas: 44% (6944/15780) 2024-05-19T18:24:45.8378566Z Resolving deltas: 45% (7111/15780) 2024-05-19T18:24:45.8391018Z Resolving deltas: 46% (7266/15780) 2024-05-19T18:24:45.8401723Z Resolving deltas: 47% (7419/15780) 2024-05-19T18:24:45.8409333Z Resolving deltas: 48% (7575/15780) 2024-05-19T18:24:45.8419678Z Resolving deltas: 49% (7746/15780) 2024-05-19T18:24:45.8434803Z Resolving deltas: 50% (7906/15780) 2024-05-19T18:24:45.8444109Z Resolving deltas: 51% (8053/15780) 2024-05-19T18:24:45.8450918Z Resolving deltas: 52% (8206/15780) 2024-05-19T18:24:45.8456855Z Resolving deltas: 53% (8366/15780) 2024-05-19T18:24:45.8463065Z Resolving deltas: 54% (8522/15780) 2024-05-19T18:24:45.8467632Z Resolving deltas: 55% (8685/15780) 2024-05-19T18:24:45.8473530Z Resolving deltas: 56% (8841/15780) 2024-05-19T18:24:45.8480758Z Resolving deltas: 57% (8996/15780) 2024-05-19T18:24:45.8487446Z Resolving deltas: 58% (9156/15780) 2024-05-19T18:24:45.8494418Z Resolving deltas: 59% (9315/15780) 2024-05-19T18:24:45.8500579Z Resolving deltas: 60% (9474/15780) 2024-05-19T18:24:45.8510538Z Resolving deltas: 61% (9648/15780) 2024-05-19T18:24:45.8515369Z Resolving deltas: 62% (9824/15780) 2024-05-19T18:24:45.8523918Z Resolving deltas: 63% (9944/15780) 2024-05-19T18:24:45.8530788Z Resolving deltas: 64% (10127/15780) 2024-05-19T18:24:45.8539132Z Resolving deltas: 65% (10260/15780) 2024-05-19T18:24:45.8553130Z Resolving deltas: 66% (10417/15780) 2024-05-19T18:24:45.8559601Z Resolving deltas: 67% (10584/15780) 2024-05-19T18:24:45.8567098Z Resolving deltas: 68% (10732/15780) 2024-05-19T18:24:45.8572207Z Resolving deltas: 69% (10903/15780) 2024-05-19T18:24:45.8578767Z Resolving deltas: 70% (11058/15780) 2024-05-19T18:24:45.8585195Z Resolving deltas: 71% (11205/15780) 2024-05-19T18:24:45.8591515Z Resolving deltas: 72% (11363/15780) 2024-05-19T18:24:45.8596709Z Resolving deltas: 73% (11526/15780) 2024-05-19T18:24:45.8602314Z Resolving deltas: 74% (11679/15780) 2024-05-19T18:24:45.8621283Z Resolving deltas: 75% (11836/15780) 2024-05-19T18:24:45.8624470Z Resolving deltas: 76% (12088/15780) 2024-05-19T18:24:45.8630515Z Resolving deltas: 77% (12152/15780) 2024-05-19T18:24:45.8638516Z Resolving deltas: 78% (12311/15780) 2024-05-19T18:24:45.8644566Z Resolving deltas: 79% (12498/15780) 2024-05-19T18:24:45.8667801Z Resolving deltas: 80% (12665/15780) 2024-05-19T18:24:45.8680446Z Resolving deltas: 83% (13172/15780) 2024-05-19T18:24:45.8681445Z Resolving deltas: 84% (13391/15780) 2024-05-19T18:24:45.8688755Z Resolving deltas: 85% (13414/15780) 2024-05-19T18:24:45.8696300Z Resolving deltas: 86% (13600/15780) 2024-05-19T18:24:45.8705661Z Resolving deltas: 87% (13756/15780) 2024-05-19T18:24:45.8710825Z Resolving deltas: 88% (13913/15780) 2024-05-19T18:24:45.8715717Z Resolving deltas: 89% (14052/15780) 2024-05-19T18:24:45.8724985Z Resolving deltas: 90% (14202/15780) 2024-05-19T18:24:45.8753001Z Resolving deltas: 91% (14364/15780) 2024-05-19T18:24:45.8759530Z Resolving deltas: 92% (14561/15780) 2024-05-19T18:24:45.8778052Z Resolving deltas: 93% (14676/15780) 2024-05-19T18:24:45.8796670Z Resolving deltas: 94% (14855/15780) 2024-05-19T18:24:45.8834019Z Resolving deltas: 95% (15084/15780) 2024-05-19T18:24:45.8845441Z Resolving deltas: 96% (15235/15780) 2024-05-19T18:24:45.8859414Z Resolving deltas: 97% (15330/15780) 2024-05-19T18:24:45.8873395Z Resolving deltas: 98% (15467/15780) 2024-05-19T18:24:45.8893145Z Resolving deltas: 99% (15624/15780) 2024-05-19T18:24:45.8893827Z Resolving deltas: 100% (15780/15780) 2024-05-19T18:24:45.8894327Z Resolving deltas: 100% (15780/15780), done. 2024-05-19T18:24:46.9376576Z + pushd torchbench 2024-05-19T18:24:46.9377136Z ~/workspace/torchbench ~/workspace 2024-05-19T18:24:46.9377754Z + git checkout d6015d42d9a1834bc7595c4bd6852562fb80b30b 2024-05-19T18:24:46.9889284Z Note: switching to 'd6015d42d9a1834bc7595c4bd6852562fb80b30b'. 2024-05-19T18:24:46.9889875Z 2024-05-19T18:24:46.9890392Z You are in 'detached HEAD' state. You can look around, make experimental 2024-05-19T18:24:46.9891298Z changes and commit them, and you can discard any commits you make in this 2024-05-19T18:24:46.9892064Z state without impacting any branches by switching back to a branch. 2024-05-19T18:24:46.9892505Z 2024-05-19T18:24:46.9892817Z If you want to create a new branch to retain commits you create, you may 2024-05-19T18:24:46.9893619Z do so (now or later) by using -c with the switch command. Example: 2024-05-19T18:24:46.9894055Z 2024-05-19T18:24:46.9894242Z git switch -c 2024-05-19T18:24:46.9894543Z 2024-05-19T18:24:46.9894693Z Or undo this operation with: 2024-05-19T18:24:46.9894945Z 2024-05-19T18:24:46.9895076Z git switch - 2024-05-19T18:24:46.9895262Z 2024-05-19T18:24:46.9895603Z Turn off this advice by setting config variable advice.detachedHead to false 2024-05-19T18:24:46.9896104Z 2024-05-19T18:24:46.9896329Z HEAD is now at d6015d42 Add sam_fast torchbench (#2182) 2024-05-19T18:24:46.9896847Z + '[' '' ']' 2024-05-19T18:24:46.9897408Z + python install.py --continue_on_fail 2024-05-19T18:24:48.7683664Z checking packages torch, torchvision, torchaudio are installed...OK 2024-05-19T18:25:08.1953199Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2024-05-19T18:25:10.5542183Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2024-05-19T18:25:30.4959317Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/DALLE2_pytorch...OK 2024-05-19T18:25:33.2552813Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2024-05-19T18:25:35.7840612Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2024-05-19T18:25:35.7991931Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2024-05-19T18:26:36.8699491Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...FAIL 2024-05-19T18:26:45.6697681Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2024-05-19T18:26:49.0681988Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2024-05-19T18:26:52.4866044Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2024-05-19T18:26:52.4867595Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2024-05-19T18:26:54.4893016Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2024-05-19T18:26:58.2649341Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2024-05-19T18:26:58.2802885Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2024-05-19T18:27:50.4819020Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2024-05-19T18:28:00.2214631Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2024-05-19T18:28:08.0124736Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2024-05-19T18:28:15.2149188Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2024-05-19T18:28:24.5691743Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2024-05-19T18:28:32.2343637Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2024-05-19T18:28:38.9015060Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2024-05-19T18:28:46.2873257Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2024-05-19T18:29:12.3605124Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2024-05-19T18:29:19.9237989Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2024-05-19T18:29:27.2360766Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2024-05-19T18:29:34.7912529Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2024-05-19T18:29:39.0375611Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2024-05-19T18:31:40.5096815Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...FAIL 2024-05-19T18:32:10.9817535Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2024-05-19T18:32:19.6217681Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2024-05-19T18:32:28.8450429Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2024-05-19T18:32:31.2392237Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2024-05-19T18:32:33.9020208Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2024-05-19T18:32:40.5731248Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2024-05-19T18:32:48.9523231Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2024-05-19T18:32:55.9189896Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2024-05-19T18:33:04.7881625Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2024-05-19T18:33:12.1096983Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2024-05-19T18:33:18.6594992Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2024-05-19T18:33:26.2601676Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2024-05-19T18:33:42.8426698Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2024-05-19T18:33:50.9684930Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2024-05-19T18:33:56.5554501Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2024-05-19T18:34:02.9078907Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2024-05-19T18:34:11.0549237Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2024-05-19T18:34:11.0551121Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2024-05-19T18:34:24.6355887Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2024-05-19T18:34:32.2160585Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2024-05-19T18:34:32.2162273Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2024-05-19T18:34:41.0375940Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2024-05-19T18:34:43.5143003Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2024-05-19T18:34:46.0119781Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2024-05-19T18:35:25.9092633Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2024-05-19T18:36:45.9730688Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2024-05-19T18:36:45.9888829Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2024-05-19T18:36:48.7250931Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2024-05-19T18:36:48.7410282Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2024-05-19T18:36:48.7568242Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2024-05-19T18:36:48.7723385Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2024-05-19T18:36:48.7881831Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2024-05-19T18:36:48.8044791Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2024-05-19T18:37:09.3949698Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2024-05-19T18:37:09.3952332Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2024-05-19T18:37:11.8980409Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2024-05-19T18:37:15.4673993Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2024-05-19T18:37:15.4832251Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2024-05-19T18:37:15.4981798Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2024-05-19T18:37:15.5142233Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2024-05-19T18:37:15.5296405Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2024-05-19T18:37:15.5452195Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2024-05-19T18:37:22.6761651Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2024-05-19T18:37:25.3698291Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2024-05-19T18:37:32.3071158Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2024-05-19T18:37:32.3233426Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2024-05-19T18:37:32.3395462Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2024-05-19T18:37:32.3551809Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2024-05-19T18:37:32.3708617Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2024-05-19T18:37:32.3866317Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2024-05-19T18:37:44.7725235Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2024-05-19T18:37:59.6238544Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2024-05-19T18:37:59.6400864Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2024-05-19T18:37:59.6404810Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2024-05-19T18:37:59.6406622Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2024-05-19T18:38:03.3230051Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2024-05-19T18:38:06.4188937Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2024-05-19T18:38:06.4351338Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2024-05-19T18:38:31.7195258Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2024-05-19T18:38:38.8312334Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2024-05-19T18:38:45.5300652Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2024-05-19T18:39:04.2307113Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2024-05-19T18:39:04.2470268Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2024-05-19T18:39:04.2632177Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2024-05-19T18:39:04.2783545Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2024-05-19T18:39:04.2934691Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2024-05-19T18:39:04.3084870Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2024-05-19T18:39:04.3236792Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2024-05-19T18:39:04.3387152Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2024-05-19T18:39:19.1482030Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2024-05-19T18:39:33.9178404Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2024-05-19T18:39:33.9338513Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2024-05-19T18:39:39.0954146Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2024-05-19T18:39:42.5145087Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2024-05-19T18:39:42.5146143Z Error for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn: 2024-05-19T18:39:42.5147101Z --------------------------------------------------------------------------- 2024-05-19T18:39:42.5147820Z error: subprocess-exited-with-error 2024-05-19T18:39:42.5148251Z 2024-05-19T18:39:42.5149000Z × git checkout -q cabcd4097442ba60aa1efa11e1619dd9bb8fb527 did not run successfully. 2024-05-19T18:39:42.5156115Z │ exit code: 128 2024-05-19T18:39:42.5156744Z ╰─> [4 lines of output] 2024-05-19T18:39:42.5157657Z error: RPC failed; curl 56 GnuTLS recv error (-9): Error decoding the received TLS packet. 2024-05-19T18:39:42.5158518Z fatal: the remote end hung up unexpectedly 2024-05-19T18:39:42.5158962Z fatal: early EOF 2024-05-19T18:39:42.5159350Z fatal: index-pack failed 2024-05-19T18:39:42.5159726Z [end of output] 2024-05-19T18:39:42.5160090Z 2024-05-19T18:39:42.5160614Z note: This error originates from a subprocess, and is likely not a problem with pip. 2024-05-19T18:39:42.5161332Z error: subprocess-exited-with-error 2024-05-19T18:39:42.5161625Z 2024-05-19T18:39:42.5162063Z × git checkout -q cabcd4097442ba60aa1efa11e1619dd9bb8fb527 did not run successfully. 2024-05-19T18:39:42.5162729Z │ exit code: 128 2024-05-19T18:39:42.5163072Z ╰─> See above for output. 2024-05-19T18:39:42.5163309Z 2024-05-19T18:39:42.5163671Z note: This error originates from a subprocess, and is likely not a problem with pip. 2024-05-19T18:39:42.5164338Z Traceback (most recent call last): 2024-05-19T18:39:42.5165213Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn/install.py", line 4, in 2024-05-19T18:39:42.5166069Z install_pytorch_geometric() 2024-05-19T18:39:42.5167118Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/gnn/__init__.py", line 8, in install_pytorch_geometric 2024-05-19T18:39:42.5168062Z pip_install_requirements() 2024-05-19T18:39:42.5168995Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/gnn/__init__.py", line 12, in pip_install_requirements 2024-05-19T18:39:42.5170275Z subprocess.check_call([sys.executable, '-m', 'pip', 'install', '-q', '-r', requirements_file]) 2024-05-19T18:39:42.5171188Z File "/opt/conda/envs/py_3.10/lib/python3.10/subprocess.py", line 369, in check_call 2024-05-19T18:39:42.5171873Z raise CalledProcessError(retcode, cmd) 2024-05-19T18:39:42.5173494Z subprocess.CalledProcessError: Command '['/opt/conda/envs/py_3.10/bin/python', '-m', 'pip', 'install', '-q', '-r', '/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/gnn/requirements.txt']' returned non-zero exit status 1. 2024-05-19T18:39:42.5174916Z 2024-05-19T18:39:42.5174922Z 2024-05-19T18:39:42.5174937Z 2024-05-19T18:39:42.5175207Z --------------------------------------------------------------------------- 2024-05-19T18:39:42.5175584Z 2024-05-19T18:39:42.5176284Z Error for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor: 2024-05-19T18:39:42.5177130Z --------------------------------------------------------------------------- 2024-05-19T18:39:42.5177643Z Retrieving notices: ...working... done 2024-05-19T18:39:42.5178039Z Channels: 2024-05-19T18:39:42.5178335Z - conda-forge 2024-05-19T18:39:42.5178634Z - defaults 2024-05-19T18:39:42.5178941Z Platform: linux-64 2024-05-19T18:39:42.5179398Z Collecting package metadata (repodata.json): ...working... done 2024-05-19T18:39:42.5179958Z Solving environment: ...working... done 2024-05-19T18:39:42.5180349Z 2024-05-19T18:39:42.5180467Z ## Package Plan ## 2024-05-19T18:39:42.5180667Z 2024-05-19T18:39:42.5180808Z environment location: /opt/conda 2024-05-19T18:39:42.5181087Z 2024-05-19T18:39:42.5181214Z added / updated specs: 2024-05-19T18:39:42.5181577Z - expecttest 2024-05-19T18:39:42.5181881Z - libglib 2024-05-19T18:39:42.5182182Z - pango 2024-05-19T18:39:42.5182340Z 2024-05-19T18:39:42.5182345Z 2024-05-19T18:39:42.5182527Z The following packages will be downloaded: 2024-05-19T18:39:42.5182841Z 2024-05-19T18:39:42.5182947Z ... 2024-05-19T18:39:42.5183307Z and retry with 'git restore --source=HEAD :/' 2024-05-19T18:39:42.5183753Z 2024-05-19T18:39:42.5184017Z [end of output] 2024-05-19T18:39:42.5184323Z 2024-05-19T18:39:42.5184831Z note: This error originates from a subprocess, and is likely not a problem with pip. 2024-05-19T18:39:42.5185544Z error: subprocess-exited-with-error 2024-05-19T18:39:42.5185836Z 2024-05-19T18:39:42.5186537Z × git clone --filter=blob:none --quiet https://github.com/mindee/doctr.git /tmp/pip-req-build-knmhvaeu did not run successfully. 2024-05-19T18:39:42.5187414Z │ exit code: 128 2024-05-19T18:39:42.5187763Z ╰─> See above for output. 2024-05-19T18:39:42.5187999Z 2024-05-19T18:39:42.5188361Z note: This error originates from a subprocess, and is likely not a problem with pip. 2024-05-19T18:39:42.5189014Z Traceback (most recent call last): 2024-05-19T18:39:42.5190293Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor/install.py", line 30, in 2024-05-19T18:39:42.5191199Z pip_install_requirements() 2024-05-19T18:39:42.5192146Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor/install.py", line 24, in pip_install_requirements 2024-05-19T18:39:42.5193107Z subprocess.check_call( 2024-05-19T18:39:42.5193713Z File "/opt/conda/envs/py_3.10/lib/python3.10/subprocess.py", line 369, in check_call 2024-05-19T18:39:42.5194402Z raise CalledProcessError(retcode, cmd) 2024-05-19T18:39:42.5195564Z subprocess.CalledProcessError: Command '['/opt/conda/envs/py_3.10/bin/python', '-m', 'pip', 'install', '-q', '-r', 'requirements.txt']' returned non-zero exit status 1. 2024-05-19T18:39:42.5196439Z 2024-05-19T18:39:42.5196444Z 2024-05-19T18:39:42.5196703Z --------------------------------------------------------------------------- 2024-05-19T18:39:42.5197069Z 2024-05-19T18:39:42.5197320Z Warning: some benchmarks were not installed due to failure 2024-05-19T18:39:42.9073590Z + popd 2024-05-19T18:39:42.9074011Z ~/workspace 2024-05-19T18:39:42.9074420Z + [[ dynamic_inductor_torchbench != *cpu_inductor* ]] 2024-05-19T18:39:42.9074981Z + install_torchrec_and_fbgemm 2024-05-19T18:39:42.9075418Z + local torchrec_commit 2024-05-19T18:39:42.9077034Z ++ get_pinned_commit torchrec 2024-05-19T18:39:42.9077601Z ++ cat .github/ci_commit_pins/torchrec.txt 2024-05-19T18:39:42.9096420Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-05-19T18:39:42.9097097Z + local fbgemm_commit 2024-05-19T18:39:42.9098532Z ++ get_pinned_commit fbgemm 2024-05-19T18:39:42.9099117Z ++ cat .github/ci_commit_pins/fbgemm.txt 2024-05-19T18:39:42.9107423Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd 2024-05-19T18:39:42.9108112Z + pip_uninstall torchrec-nightly 2024-05-19T18:39:42.9108575Z + pip uninstall -y torchrec-nightly 2024-05-19T18:39:43.2628022Z WARNING: Skipping torchrec-nightly as it is not installed. 2024-05-19T18:39:43.3082321Z + pip_uninstall fbgemm-gpu-nightly 2024-05-19T18:39:43.3082890Z + pip uninstall -y fbgemm-gpu-nightly 2024-05-19T18:39:43.6579322Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed. 2024-05-19T18:39:43.7047310Z + pip_install setuptools-git-versioning scikit-build pyre-extensions 2024-05-19T18:39:43.7048306Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions 2024-05-19T18:39:44.0992280Z Collecting setuptools-git-versioning 2024-05-19T18:39:44.1153278Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB) 2024-05-19T18:39:44.1436233Z Collecting scikit-build 2024-05-19T18:39:44.1472844Z Downloading scikit_build-0.17.6-py3-none-any.whl.metadata (14 kB) 2024-05-19T18:39:44.1723936Z Collecting pyre-extensions 2024-05-19T18:39:44.1763643Z Downloading pyre_extensions-0.0.30-py3-none-any.whl.metadata (4.0 kB) 2024-05-19T18:39:44.1853015Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.0) 2024-05-19T18:39:44.1855507Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (69.5.1) 2024-05-19T18:39:44.2013495Z Collecting toml>=0.10.2 (from setuptools-git-versioning) 2024-05-19T18:39:44.2041096Z Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) 2024-05-19T18:39:44.2258202Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0) 2024-05-19T18:39:44.2265342Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1) 2024-05-19T18:39:44.2270739Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.43.0) 2024-05-19T18:39:44.2446896Z Collecting typing-inspect (from pyre-extensions) 2024-05-19T18:39:44.2476486Z Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB) 2024-05-19T18:39:44.2536197Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.11.0) 2024-05-19T18:39:44.2763470Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0) 2024-05-19T18:39:44.2824817Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB) 2024-05-19T18:39:44.2915532Z Downloading scikit_build-0.17.6-py3-none-any.whl (84 kB) 2024-05-19T18:39:44.3088367Z Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB) 2024-05-19T18:39:44.3164468Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2024-05-19T18:39:44.3245431Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2024-05-19T18:39:46.6877091Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions 2024-05-19T18:39:46.8081550Z Successfully installed pyre-extensions-0.0.30 scikit-build-0.17.6 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0 2024-05-19T18:39:46.9320684Z + CUDA_PATH=/usr/local/cuda-12.1 2024-05-19T18:39:46.9321871Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-05-19T18:39:46.9323769Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-05-19T18:39:47.2742819Z Collecting fbgemm-gpu 2024-05-19T18:39:47.2745615Z Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-188fapk7/fbgemm-gpu_ab5d8622089d4d7a8f1e866b69681a5d 2024-05-19T18:39:47.2762582Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-188fapk7/fbgemm-gpu_ab5d8622089d4d7a8f1e866b69681a5d 2024-05-19T18:39:48.1673885Z Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd' 2024-05-19T18:39:48.1692308Z Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd 2024-05-19T18:39:48.5511572Z Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd 2024-05-19T18:39:48.9769388Z Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd 2024-05-19T18:39:48.9770712Z Running command git submodule update --init --recursive -q 2024-05-19T18:39:56.0620206Z Preparing metadata (setup.py) ... [?25l- done 2024-05-19T18:39:56.0641397Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.26.4) 2024-05-19T18:39:56.0659460Z Building wheels for collected packages: fbgemm-gpu 2024-05-19T19:21:08.1743438Z Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done 2024-05-19T19:21:08.3544961Z [?25h Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=253269992 sha256=31038ad1f6d7a820719d49214cb02ab4bd3219929fee67d15a5ce44cb538546a 2024-05-19T19:21:08.3546637Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39 2024-05-19T19:21:08.3565558Z Successfully built fbgemm-gpu 2024-05-19T19:21:10.6852196Z Installing collected packages: fbgemm-gpu 2024-05-19T19:21:13.8898102Z Successfully installed fbgemm-gpu-0.4.1rc0.post421 2024-05-19T19:21:15.7420004Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-05-19T19:21:15.7421540Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-05-19T19:21:16.0831021Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-05-19T19:21:16.0833912Z Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-vq_4mm3u 2024-05-19T19:21:16.0850678Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-vq_4mm3u 2024-05-19T19:21:16.7648251Z Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2' 2024-05-19T19:21:16.7666738Z Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-05-19T19:21:17.0324744Z Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-05-19T19:21:17.3856442Z Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-05-19T19:21:17.6182236Z Preparing metadata (setup.py) ... [?25l- done 2024-05-19T19:21:17.6232642Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9) 2024-05-19T19:21:17.6235918Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.2.2) 2024-05-19T19:21:17.6240104Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0) 2024-05-19T19:21:17.6243302Z Requirement already satisfied: torchmetrics in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (1.4.0.post0) 2024-05-19T19:21:17.6245860Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.4) 2024-05-19T19:21:17.6249190Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421) 2024-05-19T19:21:17.6260905Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.26.4) 2024-05-19T19:21:17.6281289Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.8.2) 2024-05-19T19:21:17.6999183Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0) 2024-05-19T19:21:17.7003428Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-05-19T19:21:17.7008417Z Requirement already satisfied: tzdata>=2022.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-05-19T19:21:17.7774912Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.0) 2024-05-19T19:21:17.7780328Z Requirement already satisfied: torch>=1.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.4.0a0+gitc8c7ddf) 2024-05-19T19:21:17.7786356Z Requirement already satisfied: lightning-utilities>=0.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (0.11.2) 2024-05-19T19:21:17.7903236Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (69.5.1) 2024-05-19T19:21:17.7908446Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.11.0) 2024-05-19T19:21:17.7946563Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0) 2024-05-19T19:21:17.8012848Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.13.1) 2024-05-19T19:21:17.8016454Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.12) 2024-05-19T19:21:17.8019562Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.8.8) 2024-05-19T19:21:17.8022216Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.1.4) 2024-05-19T19:21:17.8025337Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2024.2.0) 2024-05-19T19:21:17.8553292Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.1.5) 2024-05-19T19:21:17.8983664Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.3.0) 2024-05-19T19:21:17.9068048Z Building wheels for collected packages: torchrec 2024-05-19T19:21:18.3473796Z Building wheel for torchrec (setup.py) ... [?25l- \ | done 2024-05-19T19:21:18.3482967Z [?25h Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374488 sha256=a0cfcd8e7d3a60164f7f12f03980fff3246e51fee7e1f3f5868287325d844c0d 2024-05-19T19:21:18.3484603Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8 2024-05-19T19:21:18.3513494Z Successfully built torchrec 2024-05-19T19:21:20.4398867Z Installing collected packages: torchrec 2024-05-19T19:21:20.6542447Z Successfully installed torchrec-0.3.2 2024-05-19T19:21:20.7849393Z ++ pwd 2024-05-19T19:21:20.7851420Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2024-05-19T19:21:20.7853833Z + test_dynamo_benchmark torchbench 1 2024-05-19T19:21:20.7854308Z ++ pwd 2024-05-19T19:21:20.7856206Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-05-19T19:21:20.7856823Z + local suite=torchbench 2024-05-19T19:21:20.7859386Z + shift 2024-05-19T19:21:20.7859845Z + local shard_id=1 2024-05-19T19:21:20.7860183Z + shift 2024-05-19T19:21:20.7860559Z + [[ dynamic_inductor_torchbench == *perf_compare* ]] 2024-05-19T19:21:20.7861111Z + [[ dynamic_inductor_torchbench == *perf* ]] 2024-05-19T19:21:20.7861645Z + [[ dynamic_inductor_torchbench == *cpu_inductor* ]] 2024-05-19T19:21:20.7862194Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-05-19T19:21:20.7863068Z + test_single_dynamo_benchmark inference torchbench 1 --inference --bfloat16 2024-05-19T19:21:20.7863660Z ++ pwd 2024-05-19T19:21:20.7864118Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-05-19T19:21:20.7864778Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-05-19T19:21:20.7897394Z + local name=inference 2024-05-19T19:21:20.7897727Z + shift 2024-05-19T19:21:20.7898019Z + local suite=torchbench 2024-05-19T19:21:20.7898353Z + shift 2024-05-19T19:21:20.7898629Z + local shard_id=1 2024-05-19T19:21:20.7898946Z + shift 2024-05-19T19:21:20.7899233Z + partition_flags=() 2024-05-19T19:21:20.7899562Z + local partition_flags 2024-05-19T19:21:20.7899932Z + [[ -n 2 ]] 2024-05-19T19:21:20.7900252Z + [[ -n 1 ]] 2024-05-19T19:21:20.7900839Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-05-19T19:21:20.7901550Z + [[ dynamic_inductor_torchbench == *perf_compare* ]] 2024-05-19T19:21:20.7902074Z + [[ dynamic_inductor_torchbench == *perf* ]] 2024-05-19T19:21:20.7902598Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-05-19T19:21:20.7904417Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --inductor --dynamic-shapes --dynamic-batch-only --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2024-05-19T19:21:25.9587820Z 2024-05-19T19:21:26.3585352Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:21:26.3586250Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:21:26.3589804Z cuda eval lennard_jones 2024-05-19T19:21:30.4184240Z pass 2024-05-19T19:21:30.4197242Z TIMING: code_gen:0.8427 inductor_compile:1.06372 backend_compile:2.51066 entire_frame_compile:2.70143 2024-05-19T19:21:30.4198494Z STATS: call_* op count: 18 | FakeTensor.__torch_dispatch__:119 | FakeTensorMode.__torch_dispatch__:1241 | ProxyTorchDispatchMode.__torch_dispatch__:339 2024-05-19T19:21:30.4199814Z Dynamo produced 1 graphs covering 18 ops with 0 graph breaks (0 unique) 2024-05-19T19:21:33.8287263Z 2024-05-19T19:21:34.3235089Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:21:34.3235608Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:21:34.3268114Z cuda eval llama 2024-05-19T19:22:22.2241353Z pass 2024-05-19T19:22:22.2386926Z TIMING: code_gen:5.26711 inductor_compile:17.62699 backend_compile:39.90226 entire_frame_compile:45.63815 2024-05-19T19:22:22.2389560Z STATS: call_* op count: 1154 | FakeTensor.__torch_dispatch__:3213 | FakeTensorMode.__torch_dispatch__:45907 | attempt fast:1258 | fast is_contiguous:1258 | ProxyTorchDispatchMode.__torch_dispatch__:10250 2024-05-19T19:22:22.2391295Z Dynamo produced 1 graphs covering 1154 ops with 0 graph breaks (0 unique) 2024-05-19T19:22:27.8842088Z 2024-05-19T19:22:28.1483937Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:22:28.1484415Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:22:28.1484854Z cuda eval llama_v2_7b_16h 2024-05-19T19:22:28.1489280Z Traceback (most recent call last): 2024-05-19T19:22:28.1489968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4109, in run 2024-05-19T19:22:28.1490636Z ) = runner.load_model( 2024-05-19T19:22:28.1491300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 309, in load_model 2024-05-19T19:22:28.1492256Z benchmark = benchmark_cls( 2024-05-19T19:22:28.1492986Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-05-19T19:22:28.1493781Z obj = type.__call__(cls, *args, **kwargs) 2024-05-19T19:22:28.1494667Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h/__init__.py", line 12, in __init__ 2024-05-19T19:22:28.1495866Z super().__init__(name="llama_v2_7b_16h", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-05-19T19:22:28.1497157Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-05-19T19:22:28.1498338Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-05-19T19:22:28.1499313Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-05-19T19:22:28.1500054Z self._skip_by_device_name() 2024-05-19T19:22:28.1500859Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-05-19T19:22:28.1502094Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-05-19T19:22:28.1503233Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llama_v2_7b_16h/metadata.yaml`. 2024-05-19T19:22:28.1503833Z 2024-05-19T19:22:28.1503963Z model_fail_to_load 2024-05-19T19:22:31.1810860Z 2024-05-19T19:22:31.4459058Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:22:31.4459615Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:22:31.4460076Z cuda eval llava 2024-05-19T19:22:31.4464540Z Traceback (most recent call last): 2024-05-19T19:22:31.4465322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4109, in run 2024-05-19T19:22:31.4466095Z ) = runner.load_model( 2024-05-19T19:22:31.4466786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 309, in load_model 2024-05-19T19:22:31.4467543Z benchmark = benchmark_cls( 2024-05-19T19:22:31.4468275Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-05-19T19:22:31.4469086Z obj = type.__call__(cls, *args, **kwargs) 2024-05-19T19:22:31.4470168Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 11, in __init__ 2024-05-19T19:22:31.4471294Z super().__init__(name="llava", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-05-19T19:22:31.4472834Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-05-19T19:22:31.4474033Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-05-19T19:22:31.4475020Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-05-19T19:22:31.4475777Z self._skip_by_device_name() 2024-05-19T19:22:31.4476596Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-05-19T19:22:31.4477950Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-05-19T19:22:31.4479051Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`. 2024-05-19T19:22:31.4479627Z 2024-05-19T19:22:31.4479761Z model_fail_to_load 2024-05-19T19:22:34.4999413Z 2024-05-19T19:22:35.1299001Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:22:35.1299479Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:22:35.1305961Z cuda eval maml 2024-05-19T19:22:35.1309020Z pass_due_to_skip 2024-05-19T19:22:35.1317591Z TIMING: 2024-05-19T19:22:35.1317992Z STATS: call_* op count: 0 2024-05-19T19:22:35.1318545Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-05-19T19:22:38.2230136Z 2024-05-19T19:22:38.7204190Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:22:38.7204680Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:22:38.7211259Z cuda eval maml_omniglot 2024-05-19T19:22:42.8200209Z pass 2024-05-19T19:22:42.8212172Z TIMING: code_gen:1.19488 inductor_compile:1.42683 backend_compile:2.56363 entire_frame_compile:2.73427 2024-05-19T19:22:42.8213525Z STATS: call_* op count: 28 | FakeTensor.__torch_dispatch__:271 | FakeTensorMode.__torch_dispatch__:2358 | ProxyTorchDispatchMode.__torch_dispatch__:555 2024-05-19T19:22:42.8214744Z Dynamo produced 1 graphs covering 28 ops with 0 graph breaks (0 unique) 2024-05-19T19:22:46.2136686Z 2024-05-19T19:22:46.5228464Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/mnasnet1.0_top1_73.512-f206786ef8.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/mnasnet1.0_top1_73.512-f206786ef8.pth 2024-05-19T19:22:46.5390147Z 2024-05-19T19:22:46.5390159Z 2024-05-19T19:22:46.5801785Z 0% 0.00/16.9M [00:00 will be ignored 2024-05-19T19:23:59.1186549Z [rank0]:W0519 19:23:59.117000 139939627377280 torch/_dynamo/backends/distributed.py:88] [2/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance. 2024-05-19T19:23:59.1381160Z [rank0]:W0519 19:23:59.137000 139939627377280 torch/_dynamo/backends/distributed.py:105] [2/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes: 2024-05-19T19:23:59.1386768Z [rank0]:W0519 19:23:59.137000 139939627377280 torch/_dynamo/backends/distributed.py:105] [2/0_1] ┌─────────┬─────────────┬────────────────────────┐ 2024-05-19T19:23:59.1388424Z [rank0]:W0519 19:23:59.137000 139939627377280 torch/_dynamo/backends/distributed.py:105] [2/0_1] │ Index │ Extra Ops │ Extra Param Size (b) │ 2024-05-19T19:23:59.1390813Z [rank0]:W0519 19:23:59.137000 139939627377280 torch/_dynamo/backends/distributed.py:105] [2/0_1] ├─────────┼─────────────┼────────────────────────┤ 2024-05-19T19:23:59.1392580Z [rank0]:W0519 19:23:59.137000 139939627377280 torch/_dynamo/backends/distributed.py:105] [2/0_1] │ 0 │ 157 │ 44910720 │ 2024-05-19T19:23:59.1394100Z [rank0]:W0519 19:23:59.137000 139939627377280 torch/_dynamo/backends/distributed.py:105] [2/0_1] └─────────┴─────────────┴────────────────────────┘ 2024-05-19T19:24:16.6316886Z skipping cudagraphs due to mutated inputs (161 instances). Found from : 2024-05-19T19:24:16.6318007Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 130, in forward 2024-05-19T19:24:16.6318968Z self._momentum_update_key_encoder() # update the key encoder 2024-05-19T19:24:16.6320364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context 2024-05-19T19:24:16.6321217Z return func(*args, **kwargs) 2024-05-19T19:24:16.6322159Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 50, in _momentum_update_key_encoder 2024-05-19T19:24:16.6323237Z param_k.mul_(self.m).add_(param_q.mul(1. - self.m)) 2024-05-19T19:24:16.6323608Z 2024-05-19T19:24:17.9635541Z [rank0]:W0519 19:24:17.963000 139939627377280 torch/_inductor/utils.py:1130] [3/0_1] DeviceCopy in input program 2024-05-19T19:24:17.9733859Z skipping cudagraphs due to skipping cudagraphs due to cpu device (randperm). Found from : 2024-05-19T19:24:17.9735089Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 82, in _batch_shuffle_ddp 2024-05-19T19:24:17.9736085Z idx_shuffle = torch.randperm(batch_size_all).cuda() 2024-05-19T19:24:17.9736444Z 2024-05-19T19:24:18.8705575Z [rank0]:W0519 19:24:18.870000 139939627377280 torch/_dynamo/variables/tensor.py:700] [5/0] Graph break from `Tensor.item()`, consider setting: 2024-05-19T19:24:18.8707008Z [rank0]:W0519 19:24:18.870000 139939627377280 torch/_dynamo/variables/tensor.py:700] [5/0] torch._dynamo.config.capture_scalar_outputs = True 2024-05-19T19:24:18.8708152Z [rank0]:W0519 19:24:18.870000 139939627377280 torch/_dynamo/variables/tensor.py:700] [5/0] or: 2024-05-19T19:24:18.8709255Z [rank0]:W0519 19:24:18.870000 139939627377280 torch/_dynamo/variables/tensor.py:700] [5/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-05-19T19:24:18.8710772Z [rank0]:W0519 19:24:18.870000 139939627377280 torch/_dynamo/variables/tensor.py:700] [5/0] to include these operations in the captured graph. 2024-05-19T19:24:18.8711895Z [rank0]:W0519 19:24:18.870000 139939627377280 torch/_dynamo/variables/tensor.py:700] [5/0] 2024-05-19T19:24:30.0156318Z pass 2024-05-19T19:24:30.0157530Z TIMING: entire_frame_compile:35.75844 code_gen:8.49995 inductor_compile:15.87023 backend_compile:26.089 2024-05-19T19:24:30.0158842Z STATS: call_* op count: 877 | FakeTensor.__torch_dispatch__:10623 | FakeTensorMode.__torch_dispatch__:57626 | ProxyTorchDispatchMode.__torch_dispatch__:12396 2024-05-19T19:24:30.0159957Z Dynamo produced 7 graphs covering 877 ops with 5 graph breaks (3 unique) 2024-05-19T19:24:34.1810873Z 2024-05-19T19:24:34.4457261Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:24:34.4458181Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:24:34.4459056Z cuda eval moondream 2024-05-19T19:24:34.4464728Z Traceback (most recent call last): 2024-05-19T19:24:34.4466482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4109, in run 2024-05-19T19:24:34.4467205Z ) = runner.load_model( 2024-05-19T19:24:34.4467864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 309, in load_model 2024-05-19T19:24:34.4468595Z benchmark = benchmark_cls( 2024-05-19T19:24:34.4469317Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-05-19T19:24:34.4470285Z obj = type.__call__(cls, *args, **kwargs) 2024-05-19T19:24:34.4471156Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream/__init__.py", line 11, in __init__ 2024-05-19T19:24:34.4472312Z super().__init__(name="moondream", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-05-19T19:24:34.4473581Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-05-19T19:24:34.4474758Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-05-19T19:24:34.4475731Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-05-19T19:24:34.4476473Z self._skip_by_device_name() 2024-05-19T19:24:34.4477278Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-05-19T19:24:34.4478813Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-05-19T19:24:34.4479921Z NotImplementedError: The current device NVIDIA A10G is skipped by its `moondream/metadata.yaml`. 2024-05-19T19:24:34.4480499Z 2024-05-19T19:24:34.4480628Z model_fail_to_load 2024-05-19T19:24:37.5241347Z 2024-05-19T19:24:39.4091576Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M 2024-05-19T19:24:39.6330903Z num decayed parameter tensors: 50, with 124,354,560 parameters 2024-05-19T19:24:39.6332093Z num non-decayed parameter tensors: 98, with 121,344 parameters 2024-05-19T19:24:39.6334981Z using fused AdamW: True 2024-05-19T19:24:39.9277283Z 2024-05-19T19:24:39.9277650Z loading model: 0it [00:02, ?it/s] 2024-05-19T19:24:39.9316526Z cuda eval nanogpt 2024-05-19T19:25:14.7620250Z pass 2024-05-19T19:25:14.7629585Z TIMING: code_gen:2.88994 inductor_compile:10.02673 backend_compile:26.63372 entire_frame_compile:32.7145 2024-05-19T19:25:14.7631887Z STATS: call_* op count: 834 | FakeTensorMode.__torch_dispatch__:40021 | FakeTensor.__torch_dispatch__:2657 | attempt fast:2347 | fast is_contiguous:2347 | ProxyTorchDispatchMode.__torch_dispatch__:8284 2024-05-19T19:25:14.7633230Z Dynamo produced 1 graphs covering 834 ops with 0 graph breaks (0 unique) 2024-05-19T19:25:19.7240852Z 2024-05-19T19:25:22.0697064Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:25:22.0697625Z loading model: 0it [00:02, ?it/s] 2024-05-19T19:25:22.0700997Z cuda eval nvidia_deeprecommender 2024-05-19T19:25:26.7867770Z pass 2024-05-19T19:25:26.7877269Z TIMING: code_gen:0.70656 inductor_compile:0.94387 backend_compile:3.06889 entire_frame_compile:3.2604 2024-05-19T19:25:26.7878548Z STATS: call_* op count: 26 | FakeTensorMode.__torch_dispatch__:1763 | ProxyTorchDispatchMode.__torch_dispatch__:579 | FakeTensor.__torch_dispatch__:170 2024-05-19T19:25:26.7879674Z Dynamo produced 1 graphs covering 26 ops with 0 graph breaks (0 unique) 2024-05-19T19:25:30.1359669Z 2024-05-19T19:25:31.0907740Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:25:31.0908242Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:25:31.0927136Z cuda eval opacus_cifar10 2024-05-19T19:25:44.0436515Z pass 2024-05-19T19:25:44.0437428Z TIMING: code_gen:2.74391 inductor_compile:4.08508 backend_compile:6.39319 entire_frame_compile:11.46124 2024-05-19T19:25:44.0438850Z STATS: call_* op count: 220 | FakeTensorMode.__torch_dispatch__:7341 | ProxyTorchDispatchMode.__torch_dispatch__:2467 | FakeTensor.__torch_dispatch__:866 2024-05-19T19:25:44.0439999Z Dynamo produced 1 graphs covering 220 ops with 0 graph breaks (0 unique) 2024-05-19T19:25:47.6279234Z 2024-05-19T19:25:48.2746087Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:25:48.2746617Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:25:48.2805333Z cuda eval phlippe_densenet 2024-05-19T19:26:03.2563542Z pass 2024-05-19T19:26:03.2575926Z TIMING: code_gen:3.7599 inductor_compile:6.47926 backend_compile:10.87838 entire_frame_compile:13.27539 2024-05-19T19:26:03.2577240Z STATS: call_* op count: 372 | FakeTensor.__torch_dispatch__:3838 | FakeTensorMode.__torch_dispatch__:28599 | ProxyTorchDispatchMode.__torch_dispatch__:7588 2024-05-19T19:26:03.2578374Z Dynamo produced 1 graphs covering 372 ops with 0 graph breaks (0 unique) 2024-05-19T19:26:07.0356801Z 2024-05-19T19:26:07.5481980Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:26:07.5482479Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:26:07.5508207Z cuda eval phlippe_resnet 2024-05-19T19:26:14.3540948Z pass 2024-05-19T19:26:14.3552589Z TIMING: code_gen:1.42517 inductor_compile:2.42203 backend_compile:4.60803 entire_frame_compile:5.33334 2024-05-19T19:26:14.3556527Z STATS: call_* op count: 142 | FakeTensor.__torch_dispatch__:1447 | FakeTensorMode.__torch_dispatch__:10871 | ProxyTorchDispatchMode.__torch_dispatch__:2856 2024-05-19T19:26:14.3557693Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-05-19T19:26:17.8429887Z 2024-05-19T19:26:18.1277925Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:26:18.1278479Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:26:18.1279115Z cuda eval pyhpc_equation_of_state 2024-05-19T19:26:24.8623995Z pass 2024-05-19T19:26:24.8625485Z TIMING: code_gen:2.35872 inductor_compile:3.57661 backend_compile:5.05269 entire_frame_compile:5.44478 2024-05-19T19:26:24.8628332Z STATS: call_* op count: 732 | FakeTensorMode.__torch_dispatch__:10370 | ProxyTorchDispatchMode.__torch_dispatch__:3745 | FakeTensor.__torch_dispatch__:1061 2024-05-19T19:26:24.8631359Z Dynamo produced 1 graphs covering 732 ops with 0 graph breaks (0 unique) 2024-05-19T19:26:28.2994889Z 2024-05-19T19:26:28.6040549Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:26:28.6041032Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:26:28.6044635Z cuda eval pyhpc_isoneutral_mixing 2024-05-19T19:26:54.5177857Z skipping cudagraphs due to mutated inputs (7 instances). Found from : 2024-05-19T19:26:54.5180366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2319, in run_n_iterations 2024-05-19T19:26:54.5181919Z return self.model_iter_fn(mod, inputs, collect_outputs=True) 2024-05-19T19:26:54.5183110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 426, in forward_pass 2024-05-19T19:26:54.5183973Z return mod(*inputs) 2024-05-19T19:26:54.5185246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:26:54.5186365Z return forward_call(*args, **kwargs) 2024-05-19T19:26:54.5187303Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing/__init__.py", line 96, in forward 2024-05-19T19:26:54.5188296Z return isoneutral_pytorch.isoneutral_diffusion_pre( 2024-05-19T19:26:54.5189426Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing/isoneutral_pytorch.py", line 165, in isoneutral_diffusion_pre 2024-05-19T19:26:54.5190931Z K_11[1:-2, 2:-2, :] = sumz / (4.0 * dzt[None, None, :]) 2024-05-19T19:26:54.5191301Z 2024-05-19T19:26:54.6435487Z pass 2024-05-19T19:26:54.6436299Z TIMING: code_gen:5.57807 inductor_compile:15.33747 backend_compile:23.33419 entire_frame_compile:24.90447 2024-05-19T19:26:54.6437574Z STATS: call_* op count: 1488 | FakeTensorMode.__torch_dispatch__:55956 | FakeTensor.__torch_dispatch__:5080 | ProxyTorchDispatchMode.__torch_dispatch__:14005 2024-05-19T19:26:54.6438696Z Dynamo produced 1 graphs covering 1488 ops with 0 graph breaks (0 unique) 2024-05-19T19:26:59.0350330Z 2024-05-19T19:26:59.7655907Z loading model: 0it [00:00, ?it/s]WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-05-19T19:26:59.8462188Z 2024-05-19T19:26:59.8462779Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:26:59.8463601Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-05-19T19:26:59.8464423Z cuda eval pyhpc_turbulent_kinetic_energy 2024-05-19T19:26:59.8913939Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-05-19T19:27:27.7922483Z pass 2024-05-19T19:27:27.8025567Z TIMING: code_gen:9.48299 inductor_compile:15.88531 backend_compile:24.62308 entire_frame_compile:26.33917 2024-05-19T19:27:27.8026861Z STATS: call_* op count: 1992 | FakeTensorMode.__torch_dispatch__:49776 | FakeTensor.__torch_dispatch__:2561 | ProxyTorchDispatchMode.__torch_dispatch__:14635 2024-05-19T19:27:27.8028001Z Dynamo produced 1 graphs covering 1992 ops with 0 graph breaks (0 unique) 2024-05-19T19:27:32.0227151Z 2024-05-19T19:27:32.8614925Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:27:32.8615394Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:27:32.8626201Z cuda eval pytorch_CycleGAN_and_pix2pix 2024-05-19T19:27:51.0884970Z pass 2024-05-19T19:27:51.0916991Z TIMING: code_gen:3.63496 inductor_compile:5.53451 backend_compile:7.48313 entire_frame_compile:8.15694 2024-05-19T19:27:51.0918528Z STATS: call_* op count: 182 | FakeTensorMode.__torch_dispatch__:11273 | FakeTensor.__torch_dispatch__:1335 | ProxyTorchDispatchMode.__torch_dispatch__:3743 2024-05-19T19:27:51.0919682Z Dynamo produced 1 graphs covering 182 ops with 0 graph breaks (0 unique) 2024-05-19T19:27:54.7181192Z 2024-05-19T19:27:55.9018684Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:27:55.9019241Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:27:55.9039244Z cuda eval pytorch_stargan 2024-05-19T19:28:24.8960382Z pass 2024-05-19T19:28:24.9033429Z TIMING: code_gen:2.42541 inductor_compile:4.69269 backend_compile:7.13818 entire_frame_compile:7.80358 2024-05-19T19:28:24.9034721Z STATS: call_* op count: 112 | FakeTensorMode.__torch_dispatch__:17579 | FakeTensor.__torch_dispatch__:1566 | ProxyTorchDispatchMode.__torch_dispatch__:4659 2024-05-19T19:28:24.9035836Z Dynamo produced 1 graphs covering 112 ops with 0 graph breaks (0 unique) 2024-05-19T19:28:28.5177018Z 2024-05-19T19:28:29.1588690Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:28:29.1589589Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:28:29.1609205Z cuda eval pytorch_unet 2024-05-19T19:28:50.7389569Z pass 2024-05-19T19:28:50.7523247Z TIMING: code_gen:4.31244 inductor_compile:6.27375 backend_compile:8.34204 entire_frame_compile:9.30511 2024-05-19T19:28:50.7524547Z STATS: call_* op count: 142 | FakeTensor.__torch_dispatch__:1807 | FakeTensorMode.__torch_dispatch__:14679 | ProxyTorchDispatchMode.__torch_dispatch__:4409 2024-05-19T19:28:50.7525705Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-05-19T19:28:54.3124448Z 2024-05-19T19:28:55.1911091Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnet152-394f9c45.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnet152-394f9c45.pth 2024-05-19T19:28:55.2071966Z 2024-05-19T19:28:55.2072062Z 2024-05-19T19:28:55.3074169Z 0% 0.00/230M [00:00 2024-05-19T19:38:01.5145174Z torchbench_main() 2024-05-19T19:38:01.5145984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 445, in torchbench_main 2024-05-19T19:38:01.5146893Z main(TorchBenchmarkRunner(), original_dir) 2024-05-19T19:38:01.5147967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3650, in main 2024-05-19T19:38:01.5152342Z process_entry(0, runner, original_dir, args) 2024-05-19T19:38:01.5153590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3582, in process_entry 2024-05-19T19:38:01.5157574Z return run(runner, args, original_dir) 2024-05-19T19:38:01.5158657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4163, in run 2024-05-19T19:38:01.5163499Z assert marked, f"nothing in example_inputs had a dim with {batch_size}" 2024-05-19T19:38:01.5164517Z AssertionError: nothing in example_inputs had a dim with 4 2024-05-19T19:38:04.5687063Z Run failed with return code: 1 2024-05-19T19:38:04.5687547Z Output: None 2024-05-19T19:38:04.5688957Z Error: None 2024-05-19T19:38:07.0118454Z 2024-05-19T19:38:07.2872738Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/shufflenetv2_x1-5666bf0f80.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/shufflenetv2_x1-5666bf0f80.pth 2024-05-19T19:38:07.3035896Z 2024-05-19T19:38:07.3035908Z 2024-05-19T19:38:07.3266218Z 0% 0.00/8.79M [00:00 2024-05-19T19:43:33.6062380Z self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors] 2024-05-19T19:43:33.6062972Z 2024-05-19T19:43:34.6567724Z W0519 19:43:34.656000 140522457690752 torch/_dynamo/variables/tensor.py:700] [13/0] Graph break from `Tensor.item()`, consider setting: 2024-05-19T19:43:34.6569407Z W0519 19:43:34.656000 140522457690752 torch/_dynamo/variables/tensor.py:700] [13/0] torch._dynamo.config.capture_scalar_outputs = True 2024-05-19T19:43:34.6570499Z W0519 19:43:34.656000 140522457690752 torch/_dynamo/variables/tensor.py:700] [13/0] or: 2024-05-19T19:43:34.6571778Z W0519 19:43:34.656000 140522457690752 torch/_dynamo/variables/tensor.py:700] [13/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-05-19T19:43:34.6573054Z W0519 19:43:34.656000 140522457690752 torch/_dynamo/variables/tensor.py:700] [13/0] to include these operations in the captured graph. 2024-05-19T19:43:34.6574113Z W0519 19:43:34.656000 140522457690752 torch/_dynamo/variables/tensor.py:700] [13/0] 2024-05-19T19:44:17.6010780Z pass 2024-05-19T19:44:17.6011590Z TIMING: entire_frame_compile:65.41904 code_gen:8.87937 inductor_compile:21.04738 backend_compile:34.94426 2024-05-19T19:44:17.6013242Z STATS: call_* op count: 1358 | FakeTensorMode.__torch_dispatch__:86788 | FakeTensor.__torch_dispatch__:7891 | ProxyTorchDispatchMode.__torch_dispatch__:18316 | attempt fast:428 | fast is_contiguous:284 | slow both tensors nontrivially broadcast:144 2024-05-19T19:44:17.6014775Z Dynamo produced 25 graphs covering 1358 ops with 17 graph breaks (2 unique) 2024-05-19T19:44:22.1644230Z 2024-05-19T19:44:24.1775745Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:44:24.1776358Z loading model: 0it [00:02, ?it/s] 2024-05-19T19:44:24.1864312Z cuda eval yolov3 2024-05-19T19:44:47.8819247Z W0519 19:44:47.881000 140040831513216 torch/_inductor/utils.py:1130] [7/0] DeviceCopy in input program 2024-05-19T19:44:48.0587153Z W0519 19:44:48.058000 140040831513216 torch/_inductor/utils.py:1130] [7/0] DeviceCopy in input program 2024-05-19T19:44:49.3798975Z skipping cudagraphs due to skipping cudagraphs due to cpu device (arg1_1). Found from : 2024-05-19T19:44:49.3800163Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward 2024-05-19T19:44:49.3801886Z self.create_grids((nx, ny), p.device) 2024-05-19T19:44:49.3802858Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids 2024-05-19T19:44:49.3803786Z self.anchor_vec = self.anchor_vec.to(device) 2024-05-19T19:44:49.3804256Z 2024-05-19T19:44:51.4099493Z W0519 19:44:51.409000 140040831513216 torch/_inductor/utils.py:1130] [7/1] DeviceCopy in input program 2024-05-19T19:44:51.5618501Z W0519 19:44:51.561000 140040831513216 torch/_inductor/utils.py:1130] [7/1] DeviceCopy in input program 2024-05-19T19:44:52.9805850Z skipping cudagraphs due to skipping cudagraphs due to cpu device (arg3_1). Found from : 2024-05-19T19:44:52.9807241Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward 2024-05-19T19:44:52.9808120Z self.create_grids((nx, ny), p.device) 2024-05-19T19:44:52.9809020Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids 2024-05-19T19:44:52.9809943Z self.anchor_vec = self.anchor_vec.to(device) 2024-05-19T19:44:52.9810274Z 2024-05-19T19:45:31.8007890Z pass 2024-05-19T19:45:31.8085716Z TIMING: entire_frame_compile:53.63132 code_gen:11.74306 inductor_compile:13.91879 backend_compile:16.62162 2024-05-19T19:45:31.8088489Z STATS: call_* op count: 120 | FakeTensor.__torch_dispatch__:3116 | FakeTensorMode.__torch_dispatch__:18519 | ProxyTorchDispatchMode.__torch_dispatch__:895 | attempt fast:51 | fast is_contiguous:39 | slow no contiguity match:12 2024-05-19T19:45:31.8089942Z Dynamo produced 27 graphs covering 120 ops with 2 graph breaks (1 unique) 2024-05-19T19:45:33.2605797Z accuracy pass_rate=82.61% 2024-05-19T19:45:33.2610541Z calls_captured gmean=0.00x mean=508.174x 2024-05-19T19:45:33.2614111Z unique_graphs gmean=0.00x mean=2.217x 2024-05-19T19:45:33.2618117Z graph_breaks gmean=0.00x mean=0.783x 2024-05-19T19:45:33.2621467Z unique_graph_breaks gmean=0.00x mean=0.196x 2024-05-19T19:45:33.2624616Z autograd_captures gmean=0.00x mean=0.000x 2024-05-19T19:45:33.2628212Z autograd_compiles gmean=0.00x mean=0.000x 2024-05-19T19:45:33.2631477Z cudagraph_skips gmean=0.00x mean=0.196x 2024-05-19T19:45:34.1351984Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_inductor_torchbench_inference.csv 2024-05-19T19:45:34.4515396Z lennard_jones PASS 2024-05-19T19:45:34.4518936Z llama PASS 2024-05-19T19:45:34.4522602Z llama_v2_7b_16h XFAIL 2024-05-19T19:45:34.4527012Z llava XFAIL 2024-05-19T19:45:34.4531672Z maml XFAIL 2024-05-19T19:45:34.4535861Z maml_omniglot PASS 2024-05-19T19:45:34.4540194Z mnasnet1_0 PASS 2024-05-19T19:45:34.4544548Z mobilenet_v2 PASS 2024-05-19T19:45:34.4548894Z mobilenet_v2_quantized_qat XFAIL 2024-05-19T19:45:34.4553464Z mobilenet_v3_large PASS 2024-05-19T19:45:34.4557784Z moco PASS 2024-05-19T19:45:34.4562019Z moondream XFAIL 2024-05-19T19:45:34.4566326Z nanogpt PASS 2024-05-19T19:45:34.4570834Z nvidia_deeprecommender PASS 2024-05-19T19:45:34.4575079Z opacus_cifar10 PASS 2024-05-19T19:45:34.4579473Z phlippe_densenet PASS 2024-05-19T19:45:34.4583649Z phlippe_resnet PASS 2024-05-19T19:45:34.4587943Z pyhpc_equation_of_state PASS 2024-05-19T19:45:34.4592484Z pyhpc_isoneutral_mixing PASS 2024-05-19T19:45:34.4596915Z pyhpc_turbulent_kinetic_energy PASS 2024-05-19T19:45:34.4601292Z pytorch_CycleGAN_and_pix2pix PASS 2024-05-19T19:45:34.4605528Z pytorch_stargan PASS 2024-05-19T19:45:34.4610066Z pytorch_unet PASS 2024-05-19T19:45:34.4614312Z resnet152 PASS 2024-05-19T19:45:34.4618547Z resnet18 PASS 2024-05-19T19:45:34.4622806Z resnet50 PASS 2024-05-19T19:45:34.4627124Z resnet50_quantized_qat XFAIL 2024-05-19T19:45:34.4631459Z resnext50_32x4d PASS 2024-05-19T19:45:34.4635942Z sam PASS 2024-05-19T19:45:34.4640266Z shufflenet_v2_x1_0 PASS 2024-05-19T19:45:34.4644458Z soft_actor_critic PASS 2024-05-19T19:45:34.4649100Z speech_transformer PASS 2024-05-19T19:45:34.4653294Z squeezenet1_1 PASS 2024-05-19T19:45:34.4657518Z stable_diffusion_text_encoder PASS 2024-05-19T19:45:34.4661943Z stable_diffusion_unet XFAIL 2024-05-19T19:45:34.4666134Z timm_efficientnet PASS 2024-05-19T19:45:34.4670578Z timm_regnet PASS 2024-05-19T19:45:34.4675131Z timm_resnest PASS 2024-05-19T19:45:34.4679399Z timm_vision_transformer PASS 2024-05-19T19:45:34.4683636Z timm_vision_transformer_large XFAIL 2024-05-19T19:45:34.4688245Z timm_vovnet PASS 2024-05-19T19:45:34.4692357Z torch_multimodal_clip PASS 2024-05-19T19:45:34.4696636Z tts_angular PASS 2024-05-19T19:45:34.4700946Z vgg16 PASS 2024-05-19T19:45:34.4705229Z vision_maskrcnn PASS 2024-05-19T19:45:34.4709538Z yolov3 PASS 2024-05-19T19:45:34.5171991Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_inductor_torchbench_inference.csv 2024-05-19T19:45:34.8216654Z lennard_jones PASS 2024-05-19T19:45:34.8219875Z llama PASS 2024-05-19T19:45:34.8224997Z llama_v2_7b_16h PASS 2024-05-19T19:45:34.8230204Z llava PASS 2024-05-19T19:45:34.8235145Z maml PASS 2024-05-19T19:45:34.8240030Z maml_omniglot PASS 2024-05-19T19:45:34.8245186Z mnasnet1_0 PASS 2024-05-19T19:45:34.8250337Z mobilenet_v2 PASS 2024-05-19T19:45:34.8255186Z mobilenet_v2_quantized_qat PASS 2024-05-19T19:45:34.8260105Z mobilenet_v3_large PASS 2024-05-19T19:45:34.8264994Z moco PASS 2024-05-19T19:45:34.8269891Z moondream PASS 2024-05-19T19:45:34.8275234Z nanogpt PASS 2024-05-19T19:45:34.8279876Z nvidia_deeprecommender PASS 2024-05-19T19:45:34.8284931Z opacus_cifar10 PASS 2024-05-19T19:45:34.8290098Z phlippe_densenet PASS 2024-05-19T19:45:34.8295024Z phlippe_resnet PASS 2024-05-19T19:45:34.8300075Z pyhpc_equation_of_state PASS 2024-05-19T19:45:34.8304981Z pyhpc_isoneutral_mixing PASS 2024-05-19T19:45:34.8310035Z pyhpc_turbulent_kinetic_energy PASS 2024-05-19T19:45:34.8315397Z pytorch_CycleGAN_and_pix2pix PASS 2024-05-19T19:45:34.8320281Z pytorch_stargan PASS 2024-05-19T19:45:34.8325144Z pytorch_unet PASS 2024-05-19T19:45:34.8330458Z resnet152 PASS 2024-05-19T19:45:34.8335357Z resnet18 PASS 2024-05-19T19:45:34.8340266Z resnet50 PASS 2024-05-19T19:45:34.8345165Z resnet50_quantized_qat PASS 2024-05-19T19:45:34.8350252Z resnext50_32x4d PASS 2024-05-19T19:45:34.8355509Z sam PASS 2024-05-19T19:45:34.8360456Z shufflenet_v2_x1_0 PASS 2024-05-19T19:45:34.8365451Z soft_actor_critic PASS 2024-05-19T19:45:34.8370644Z speech_transformer PASS 2024-05-19T19:45:34.8375490Z squeezenet1_1 PASS 2024-05-19T19:45:34.8380494Z stable_diffusion_text_encoder PASS 2024-05-19T19:45:34.8385398Z stable_diffusion_unet PASS 2024-05-19T19:45:34.8390397Z timm_efficientnet PASS 2024-05-19T19:45:34.8395756Z timm_regnet PASS 2024-05-19T19:45:34.8400792Z timm_resnest PASS 2024-05-19T19:45:34.8405761Z timm_vision_transformer PASS 2024-05-19T19:45:34.8410884Z timm_vision_transformer_large PASS 2024-05-19T19:45:34.8415778Z timm_vovnet PASS 2024-05-19T19:45:34.8420811Z torch_multimodal_clip PASS 2024-05-19T19:45:34.8425686Z tts_angular PASS 2024-05-19T19:45:34.8430780Z vgg16 PASS 2024-05-19T19:45:34.8435863Z vision_maskrcnn PASS 2024-05-19T19:45:34.8440616Z yolov3 PASS 2024-05-19T19:45:34.8900866Z + test_single_dynamo_benchmark training torchbench 1 --training --amp 2024-05-19T19:45:34.8901574Z ++ pwd 2024-05-19T19:45:34.8903804Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-05-19T19:45:34.8904718Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-05-19T19:45:34.8915873Z + local name=training 2024-05-19T19:45:34.8916340Z + shift 2024-05-19T19:45:34.8916690Z + local suite=torchbench 2024-05-19T19:45:34.8917077Z + shift 2024-05-19T19:45:34.8917957Z + local shard_id=1 2024-05-19T19:45:34.8918350Z + shift 2024-05-19T19:45:34.8918754Z + partition_flags=() 2024-05-19T19:45:34.8919137Z + local partition_flags 2024-05-19T19:45:34.8919546Z + [[ -n 2 ]] 2024-05-19T19:45:34.8921540Z + [[ -n 1 ]] 2024-05-19T19:45:34.8922704Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-05-19T19:45:34.8923695Z + [[ dynamic_inductor_torchbench == *perf_compare* ]] 2024-05-19T19:45:34.8924217Z + [[ dynamic_inductor_torchbench == *perf* ]] 2024-05-19T19:45:34.8924732Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-05-19T19:45:34.8926638Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --inductor --dynamic-shapes --dynamic-batch-only --device cuda --training --amp --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv 2024-05-19T19:45:39.9034820Z 2024-05-19T19:45:40.3453364Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:45:40.3453961Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:45:40.3454426Z cuda train lennard_jones 2024-05-19T19:45:47.6424565Z W0519 19:45:47.641000 140035372544640 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:45:52.1508837Z pass 2024-05-19T19:45:52.1509642Z TIMING: entire_frame_compile:9.88072 code_gen:4.44376 inductor_compile:5.61284 backend_compile:9.05709 2024-05-19T19:45:52.1511335Z STATS: call_* op count: 61 | FakeTensor.__torch_dispatch__:835 | FakeTensorMode.__torch_dispatch__:6023 | attempt fast:86 | fast is_contiguous:86 | ProxyTorchDispatchMode.__torch_dispatch__:1264 2024-05-19T19:45:52.1512602Z Dynamo produced 3 graphs covering 61 ops with 7 graph breaks (5 unique) 2024-05-19T19:45:55.6648669Z 2024-05-19T19:45:55.9283107Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:45:55.9283720Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:45:55.9284236Z cuda train llava 2024-05-19T19:45:55.9288143Z Traceback (most recent call last): 2024-05-19T19:45:55.9289003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4109, in run 2024-05-19T19:45:55.9289677Z ) = runner.load_model( 2024-05-19T19:45:55.9290371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 302, in load_model 2024-05-19T19:45:55.9291183Z benchmark = benchmark_cls( 2024-05-19T19:45:55.9291965Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-05-19T19:45:55.9293031Z obj = type.__call__(cls, *args, **kwargs) 2024-05-19T19:45:55.9294056Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 11, in __init__ 2024-05-19T19:45:55.9295475Z super().__init__(name="llava", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-05-19T19:45:55.9296717Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-05-19T19:45:55.9297896Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-05-19T19:45:55.9298877Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-05-19T19:45:55.9299618Z self._skip_by_device_name() 2024-05-19T19:45:55.9300418Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-05-19T19:45:55.9301639Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-05-19T19:45:55.9302793Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`. 2024-05-19T19:45:55.9303359Z 2024-05-19T19:45:55.9303480Z model_fail_to_load 2024-05-19T19:45:58.9664691Z 2024-05-19T19:45:59.5621605Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:45:59.5622135Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:45:59.5622597Z cuda train maml_omniglot 2024-05-19T19:46:09.7979276Z W0519 19:46:09.796000 140690470568576 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:46:15.8034127Z pass 2024-05-19T19:46:15.8034936Z TIMING: entire_frame_compile:14.13529 code_gen:7.95382 inductor_compile:9.8752 backend_compile:13.06906 2024-05-19T19:46:15.8036363Z STATS: call_* op count: 78 | FakeTensor.__torch_dispatch__:1199 | FakeTensorMode.__torch_dispatch__:7774 | attempt fast:91 | fast is_contiguous:91 | ProxyTorchDispatchMode.__torch_dispatch__:1538 2024-05-19T19:46:15.8037629Z Dynamo produced 3 graphs covering 78 ops with 7 graph breaks (5 unique) 2024-05-19T19:46:19.4612572Z 2024-05-19T19:46:22.1418951Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:46:22.1419587Z loading model: 0it [00:02, ?it/s] 2024-05-19T19:46:22.1420057Z cuda train mnasnet1_0 2024-05-19T19:47:05.0748310Z W0519 19:47:05.074000 140717090427520 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:47:48.8575472Z pass 2024-05-19T19:47:48.8603087Z TIMING: entire_frame_compile:80.03904 code_gen:25.44411 inductor_compile:45.60707 backend_compile:68.30831 2024-05-19T19:47:48.8605430Z STATS: call_* op count: 648 | FakeTensor.__torch_dispatch__:14137 | FakeTensorMode.__torch_dispatch__:85937 | attempt fast:1489 | fast is_contiguous:1489 | ProxyTorchDispatchMode.__torch_dispatch__:17678 2024-05-19T19:47:48.8606783Z Dynamo produced 3 graphs covering 648 ops with 7 graph breaks (5 unique) 2024-05-19T19:47:55.3209587Z 2024-05-19T19:47:57.5203421Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:47:57.5204058Z loading model: 0it [00:02, ?it/s] 2024-05-19T19:47:57.5204779Z cuda train mobilenet_v2 2024-05-19T19:48:49.9473481Z pass 2024-05-19T19:48:49.9521720Z TIMING: entire_frame_compile:47.14145 code_gen:9.12571 inductor_compile:25.75333 backend_compile:41.24046 2024-05-19T19:48:49.9524873Z STATS: call_* op count: 157 | FakeTensor.__torch_dispatch__:8000 | FakeTensorMode.__torch_dispatch__:52035 | attempt fast:1661 | fast is_contiguous:1661 | ProxyTorchDispatchMode.__torch_dispatch__:10584 2024-05-19T19:48:49.9526235Z Dynamo produced 2 graphs covering 157 ops with 6 graph breaks (5 unique) 2024-05-19T19:48:55.4543449Z 2024-05-19T19:48:57.3917359Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:48:57.3917915Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:48:57.3918533Z cuda train mobilenet_v2_quantized_qat 2024-05-19T19:48:57.3919043Z Traceback (most recent call last): 2024-05-19T19:48:57.3920431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2280, in validate_model 2024-05-19T19:48:57.3921261Z self.model_iter_fn(model, example_inputs) 2024-05-19T19:48:57.3922122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 432, in forward_and_backward_pass 2024-05-19T19:48:57.3922942Z pred = mod(*cloned_inputs) 2024-05-19T19:48:57.3923939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 736, in call_wrapped 2024-05-19T19:48:57.3924991Z return self._wrapped_call(self, *args, **kwargs) 2024-05-19T19:48:57.3925933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 315, in __call__ 2024-05-19T19:48:57.3926940Z raise e 2024-05-19T19:48:57.3927778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 302, in __call__ 2024-05-19T19:48:57.3928750Z return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] 2024-05-19T19:48:57.3929882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T19:48:57.3930789Z return self._call_impl(*args, **kwargs) 2024-05-19T19:48:57.3931819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:48:57.3932664Z return forward_call(*args, **kwargs) 2024-05-19T19:48:57.3933139Z File ".3", line 207, in forward 2024-05-19T19:48:57.3934090Z activation_post_process_101 = self.activation_post_process_101(classifier_1); classifier_1 = None 2024-05-19T19:48:57.3935334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T19:48:57.3936226Z return self._call_impl(*args, **kwargs) 2024-05-19T19:48:57.3937141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:48:57.3937957Z return forward_call(*args, **kwargs) 2024-05-19T19:48:57.3938924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 342, in forward 2024-05-19T19:48:57.3939947Z return torch.fused_moving_avg_obs_fake_quant( 2024-05-19T19:48:57.3940500Z RuntimeError: expected scalar type Float but found Half 2024-05-19T19:48:57.3940889Z 2024-05-19T19:48:57.3941186Z The above exception was the direct cause of the following exception: 2024-05-19T19:48:57.3941641Z 2024-05-19T19:48:57.3941786Z Traceback (most recent call last): 2024-05-19T19:48:57.3942446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4109, in run 2024-05-19T19:48:57.3943102Z ) = runner.load_model( 2024-05-19T19:48:57.3943760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-05-19T19:48:57.3944509Z self.validate_model(model, example_inputs) 2024-05-19T19:48:57.3945263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2282, in validate_model 2024-05-19T19:48:57.3946024Z raise RuntimeError("Eager run failed") from e 2024-05-19T19:48:57.3946502Z RuntimeError: Eager run failed 2024-05-19T19:48:57.3946755Z 2024-05-19T19:48:57.3946872Z eager_fail_to_run 2024-05-19T19:49:00.4962169Z 2024-05-19T19:49:03.1790247Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:49:03.1790793Z loading model: 0it [00:02, ?it/s] 2024-05-19T19:49:03.1791302Z cuda train mobilenet_v3_large 2024-05-19T19:50:02.0512991Z W0519 19:50:02.050000 139678284296832 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:50:49.6293926Z E0519 19:50:49.628000 139678284296832 torch/_dynamo/utils.py:1392] RMSE (res-fp64): 0.00201, (ref-fp64): 0.00024 and shape=torch.Size([168]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-05-19T19:50:49.6295705Z E0519 19:50:49.628000 139678284296832 torch/_dynamo/utils.py:1306] Accuracy failed for key name features.13.block.2.fc1.bias.grad 2024-05-19T19:50:49.6323898Z pass 2024-05-19T19:50:49.6361852Z TIMING: entire_frame_compile:98.63974 code_gen:27.00031 inductor_compile:55.02158 backend_compile:84.28827 2024-05-19T19:50:49.6363332Z STATS: call_* op count: 731 | FakeTensor.__torch_dispatch__:16968 | FakeTensorMode.__torch_dispatch__:101992 | attempt fast:2123 | fast is_contiguous:2123 | ProxyTorchDispatchMode.__torch_dispatch__:19585 2024-05-19T19:50:49.6364647Z Dynamo produced 3 graphs covering 731 ops with 7 graph breaks (5 unique) 2024-05-19T19:50:57.0373613Z 2024-05-19T19:50:59.3673552Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:50:59.3674134Z loading model: 0it [00:02, ?it/s] 2024-05-19T19:50:59.3674585Z cuda train moco 2024-05-19T19:51:05.4097214Z [rank0]:W0519 19:51:05.408000 140243047334528 torch/_logging/_internal.py:1024] [3/0] Profiler function will be ignored 2024-05-19T19:51:12.7029774Z [rank0]:W0519 19:51:12.702000 140243047334528 torch/_dynamo/backends/distributed.py:88] [4/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance. 2024-05-19T19:51:12.7227525Z [rank0]:W0519 19:51:12.722000 140243047334528 torch/_dynamo/backends/distributed.py:105] [4/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes: 2024-05-19T19:51:12.7229306Z [rank0]:W0519 19:51:12.722000 140243047334528 torch/_dynamo/backends/distributed.py:105] [4/0_1] ┌─────────┬─────────────┬────────────────────────┐ 2024-05-19T19:51:12.7231094Z [rank0]:W0519 19:51:12.722000 140243047334528 torch/_dynamo/backends/distributed.py:105] [4/0_1] │ Index │ Extra Ops │ Extra Param Size (b) │ 2024-05-19T19:51:12.7232639Z [rank0]:W0519 19:51:12.722000 140243047334528 torch/_dynamo/backends/distributed.py:105] [4/0_1] ├─────────┼─────────────┼────────────────────────┤ 2024-05-19T19:51:12.7234437Z [rank0]:W0519 19:51:12.722000 140243047334528 torch/_dynamo/backends/distributed.py:105] [4/0_1] │ 0 │ 161 │ 94032128 │ 2024-05-19T19:51:12.7236743Z [rank0]:W0519 19:51:12.722000 140243047334528 torch/_dynamo/backends/distributed.py:105] [4/0_1] └─────────┴─────────────┴────────────────────────┘ 2024-05-19T19:52:03.2605770Z skipping cudagraphs due to mutated inputs (161 instances) 2024-05-19T19:52:05.6846785Z [rank0]:W0519 19:52:05.683000 140243047334528 torch/_inductor/utils.py:1130] [5/0_1] DeviceCopy in input program 2024-05-19T19:52:05.7025467Z skipping cudagraphs due to skipping cudagraphs due to cpu device (randperm). Found from : 2024-05-19T19:52:05.7027526Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 82, in _batch_shuffle_ddp 2024-05-19T19:52:05.7028567Z idx_shuffle = torch.randperm(batch_size_all).cuda() 2024-05-19T19:52:05.7028973Z 2024-05-19T19:52:07.6601076Z [rank0]:W0519 19:52:07.659000 140243047334528 torch/_dynamo/variables/tensor.py:700] [7/0] Graph break from `Tensor.item()`, consider setting: 2024-05-19T19:52:07.6602513Z [rank0]:W0519 19:52:07.659000 140243047334528 torch/_dynamo/variables/tensor.py:700] [7/0] torch._dynamo.config.capture_scalar_outputs = True 2024-05-19T19:52:07.6603782Z [rank0]:W0519 19:52:07.659000 140243047334528 torch/_dynamo/variables/tensor.py:700] [7/0] or: 2024-05-19T19:52:07.6605056Z [rank0]:W0519 19:52:07.659000 140243047334528 torch/_dynamo/variables/tensor.py:700] [7/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-05-19T19:52:07.6606541Z [rank0]:W0519 19:52:07.659000 140243047334528 torch/_dynamo/variables/tensor.py:700] [7/0] to include these operations in the captured graph. 2024-05-19T19:52:07.6607912Z [rank0]:W0519 19:52:07.659000 140243047334528 torch/_dynamo/variables/tensor.py:700] [7/0] 2024-05-19T19:52:34.4185502Z pass 2024-05-19T19:52:34.4520834Z TIMING: entire_frame_compile:81.9624 inductor_compile:40.84874 backend_compile:69.9418 code_gen:19.91134 2024-05-19T19:52:34.4523856Z STATS: call_* op count: 907 | FakeTensorMode.__torch_dispatch__:93368 | ProxyTorchDispatchMode.__torch_dispatch__:20700 | FakeTensor.__torch_dispatch__:13451 | attempt fast:2739 | fast is_contiguous:2723 | slow no contiguity match:16 2024-05-19T19:52:34.4525384Z Dynamo produced 10 graphs covering 907 ops with 11 graph breaks (8 unique) 2024-05-19T19:52:40.4304712Z 2024-05-19T19:52:42.3227212Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M 2024-05-19T19:52:42.5365003Z num decayed parameter tensors: 50, with 124,354,560 parameters 2024-05-19T19:52:42.5366843Z num non-decayed parameter tensors: 98, with 121,344 parameters 2024-05-19T19:52:42.5369684Z using fused AdamW: True 2024-05-19T19:52:42.9168853Z 2024-05-19T19:52:42.9169387Z loading model: 0it [00:02, ?it/s] 2024-05-19T19:52:42.9169884Z cuda train nanogpt 2024-05-19T19:53:33.6077284Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T19:53:33.6079423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 432, in torch_dynamo_resume_in_forward_and_backward_pass_at_430 2024-05-19T19:53:33.6080412Z pred = mod(*cloned_inputs) 2024-05-19T19:53:33.6081410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:53:33.6082771Z return forward_call(*args, **kwargs) 2024-05-19T19:53:33.6091336Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt/model.py", line 228, in forward 2024-05-19T19:53:33.6092420Z tok_emb = self.transformer.wte(idx) # token embeddings of shape (b, t, n_embd) 2024-05-19T19:53:33.6092919Z 2024-05-19T19:53:35.4622265Z W0519 19:53:35.461000 140711681143424 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:54:12.1455479Z pass 2024-05-19T19:54:12.2086351Z TIMING: entire_frame_compile:83.31186 code_gen:21.20851 inductor_compile:45.44688 backend_compile:71.9638 2024-05-19T19:54:12.2089355Z STATS: call_* op count: 883 | FakeTensorMode.__torch_dispatch__:90611 | FakeTensor.__torch_dispatch__:12825 | attempt fast:2577 | fast is_contiguous:2577 | ProxyTorchDispatchMode.__torch_dispatch__:19295 2024-05-19T19:54:12.2091365Z Dynamo produced 3 graphs covering 883 ops with 7 graph breaks (5 unique) 2024-05-19T19:54:18.9459324Z 2024-05-19T19:54:21.3031813Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:54:21.3032996Z loading model: 0it [00:02, ?it/s] 2024-05-19T19:54:21.3033945Z cuda train nvidia_deeprecommender 2024-05-19T19:54:30.0855416Z W0519 19:54:30.084000 140543800812160 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:54:35.2889862Z pass 2024-05-19T19:54:35.3179678Z TIMING: entire_frame_compile:11.2398 code_gen:4.48027 inductor_compile:6.31791 backend_compile:10.11795 2024-05-19T19:54:35.3183490Z STATS: call_* op count: 71 | FakeTensorMode.__torch_dispatch__:7532 | attempt fast:212 | fast is_contiguous:212 | FakeTensor.__torch_dispatch__:1123 | ProxyTorchDispatchMode.__torch_dispatch__:1490 2024-05-19T19:54:35.3184827Z Dynamo produced 3 graphs covering 71 ops with 7 graph breaks (5 unique) 2024-05-19T19:54:38.8645575Z 2024-05-19T19:54:39.8248111Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:54:39.8248613Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:54:39.8250055Z cuda train opacus_cifar10 2024-05-19T19:54:39.8281517Z Traceback (most recent call last): 2024-05-19T19:54:39.8282552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2280, in validate_model 2024-05-19T19:54:39.8283526Z self.model_iter_fn(model, example_inputs) 2024-05-19T19:54:39.8284465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_and_backward_pass 2024-05-19T19:54:39.8285327Z self.grad_scaler.scale(loss).backward() 2024-05-19T19:54:39.8286599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 523, in backward 2024-05-19T19:54:39.8287748Z torch.autograd.backward( 2024-05-19T19:54:39.8288866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 284, in backward 2024-05-19T19:54:39.8289685Z _engine_run_backward( 2024-05-19T19:54:39.8290885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 767, in _engine_run_backward 2024-05-19T19:54:39.8292506Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-05-19T19:54:39.8293848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 72, in __call__ 2024-05-19T19:54:39.8294789Z return self.hook(module, *args, **kwargs) 2024-05-19T19:54:39.8295864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 327, in capture_backprops_hook 2024-05-19T19:54:39.8296895Z activations, backprops = self.rearrange_grad_samples( 2024-05-19T19:54:39.8298018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 383, in rearrange_grad_samples 2024-05-19T19:54:39.8298932Z raise ValueError( 2024-05-19T19:54:39.8299990Z ValueError: No activations detected for , run forward after add_hooks(model) 2024-05-19T19:54:39.8300687Z 2024-05-19T19:54:39.8300980Z The above exception was the direct cause of the following exception: 2024-05-19T19:54:39.8301421Z 2024-05-19T19:54:39.8301574Z Traceback (most recent call last): 2024-05-19T19:54:39.8302221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4109, in run 2024-05-19T19:54:39.8302868Z ) = runner.load_model( 2024-05-19T19:54:39.8303530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-05-19T19:54:39.8304397Z self.validate_model(model, example_inputs) 2024-05-19T19:54:39.8305146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2282, in validate_model 2024-05-19T19:54:39.8305898Z raise RuntimeError("Eager run failed") from e 2024-05-19T19:54:39.8306419Z RuntimeError: Eager run failed 2024-05-19T19:54:39.8306676Z 2024-05-19T19:54:39.8306800Z eager_fail_to_run 2024-05-19T19:54:42.9567273Z 2024-05-19T19:54:43.8326730Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:54:43.8327471Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:54:43.8328136Z cuda train phlippe_densenet 2024-05-19T19:55:33.9288160Z pass 2024-05-19T19:55:33.9292542Z TIMING: entire_frame_compile:46.36498 code_gen:10.71392 inductor_compile:24.56046 backend_compile:40.42585 2024-05-19T19:55:33.9295094Z STATS: call_* op count: 190 | FakeTensor.__torch_dispatch__:6771 | FakeTensorMode.__torch_dispatch__:50121 | attempt fast:1467 | fast is_contiguous:1419 | ProxyTorchDispatchMode.__torch_dispatch__:11455 | slow no contiguity match:48 2024-05-19T19:55:33.9296625Z Dynamo produced 2 graphs covering 190 ops with 6 graph breaks (5 unique) 2024-05-19T19:55:39.2675131Z 2024-05-19T19:55:39.9818821Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:55:39.9819368Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:55:39.9819833Z cuda train phlippe_resnet 2024-05-19T19:55:59.0636307Z E0519 19:55:59.062000 139763991364224 torch/_dynamo/utils.py:1392] RMSE (res-fp64): 0.00102, (ref-fp64): 0.00001 and shape=torch.Size([]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-05-19T19:55:59.0647538Z fail_accuracy 2024-05-19T19:55:59.0648209Z TIMING: entire_frame_compile:16.81323 code_gen:3.27386 inductor_compile:8.07473 backend_compile:14.78056 2024-05-19T19:55:59.0649650Z STATS: call_* op count: 75 | FakeTensor.__torch_dispatch__:2555 | FakeTensorMode.__torch_dispatch__:18639 | attempt fast:586 | fast is_contiguous:586 | ProxyTorchDispatchMode.__torch_dispatch__:4304 2024-05-19T19:55:59.0650951Z Dynamo produced 2 graphs covering 75 ops with 6 graph breaks (5 unique) 2024-05-19T19:56:02.9856582Z 2024-05-19T19:56:04.2888510Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:56:04.2889034Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:56:04.2889575Z cuda train pytorch_CycleGAN_and_pix2pix 2024-05-19T19:56:39.2121806Z pass 2024-05-19T19:56:39.2209534Z TIMING: entire_frame_compile:8.78027 code_gen:9.65374 inductor_compile:13.65167 backend_compile:7.98959 2024-05-19T19:56:39.2211535Z STATS: call_* op count: 95 | FakeTensorMode.__torch_dispatch__:17677 | FakeTensor.__torch_dispatch__:3110 | ProxyTorchDispatchMode.__torch_dispatch__:5123 2024-05-19T19:56:39.2212681Z Dynamo produced 2 graphs covering 95 ops with 6 graph breaks (5 unique) 2024-05-19T19:56:43.1370515Z 2024-05-19T19:56:44.6848516Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:56:44.6849078Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:56:44.6849541Z cuda train pytorch_stargan 2024-05-19T19:57:44.8424123Z pass 2024-05-19T19:57:44.8498618Z TIMING: entire_frame_compile:26.37827 code_gen:4.1115 inductor_compile:12.70817 backend_compile:23.01777 2024-05-19T19:57:44.8500640Z STATS: call_* op count: 61 | FakeTensorMode.__torch_dispatch__:27466 | FakeTensor.__torch_dispatch__:2601 | attempt fast:864 | fast is_contiguous:864 | ProxyTorchDispatchMode.__torch_dispatch__:5427 2024-05-19T19:57:44.8502199Z Dynamo produced 2 graphs covering 61 ops with 6 graph breaks (5 unique) 2024-05-19T19:57:49.4522768Z 2024-05-19T19:57:50.3391457Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:57:50.3393764Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:57:50.3394395Z cuda train pytorch_unet 2024-05-19T19:58:51.6652431Z W0519 19:58:51.664000 140231939482240 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:59:12.9912558Z pass_due_to_skip 2024-05-19T19:59:13.0003928Z TIMING: entire_frame_compile:49.28164 code_gen:20.51729 inductor_compile:32.70503 backend_compile:43.95143 2024-05-19T19:59:13.0006397Z STATS: call_* op count: 315 | FakeTensor.__torch_dispatch__:6870 | FakeTensorMode.__torch_dispatch__:42472 | attempt fast:766 | fast is_contiguous:758 | ProxyTorchDispatchMode.__torch_dispatch__:9107 | slow no contiguity match:8 2024-05-19T19:59:13.0008052Z Dynamo produced 3 graphs covering 315 ops with 7 graph breaks (5 unique) 2024-05-19T19:59:17.8785538Z 2024-05-19T19:59:20.4093544Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:59:20.4094054Z loading model: 0it [00:02, ?it/s] 2024-05-19T19:59:20.4094578Z cuda train resnet152 2024-05-19T20:01:16.1183031Z W0519 20:01:16.117000 140690087445120 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T20:03:21.7118852Z pass 2024-05-19T20:03:21.7475930Z TIMING: entire_frame_compile:221.18554 code_gen:55.19056 inductor_compile:120.21689 backend_compile:192.03618 2024-05-19T20:03:21.7479635Z STATS: call_* op count: 1938 | FakeTensor.__torch_dispatch__:41495 | FakeTensorMode.__torch_dispatch__:258224 | attempt fast:4445 | fast is_contiguous:4445 | ProxyTorchDispatchMode.__torch_dispatch__:53818 2024-05-19T20:03:21.7481582Z Dynamo produced 3 graphs covering 1938 ops with 7 graph breaks (5 unique) 2024-05-19T20:03:34.6146580Z 2024-05-19T20:03:35.9849440Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:03:35.9850114Z loading model: 0it [00:01, ?it/s] 2024-05-19T20:03:35.9850878Z cuda train resnet18 2024-05-19T20:03:58.2798496Z pass 2024-05-19T20:03:58.2873721Z TIMING: entire_frame_compile:18.69205 code_gen:6.08694 inductor_compile:11.53916 backend_compile:17.18076 2024-05-19T20:03:58.2875792Z STATS: call_* op count: 73 | FakeTensor.__torch_dispatch__:2577 | FakeTensorMode.__torch_dispatch__:17879 | attempt fast:590 | fast is_contiguous:590 | ProxyTorchDispatchMode.__torch_dispatch__:4304 2024-05-19T20:03:58.2877113Z Dynamo produced 2 graphs covering 73 ops with 6 graph breaks (5 unique) 2024-05-19T20:04:02.3480127Z 2024-05-19T20:04:04.3256519Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:04:04.3257395Z loading model: 0it [00:01, ?it/s] 2024-05-19T20:04:04.3258590Z cuda train resnet50 2024-05-19T20:04:52.2751334Z pass 2024-05-19T20:04:52.2928579Z TIMING: entire_frame_compile:39.84241 code_gen:8.88385 inductor_compile:22.33293 backend_compile:36.42214 2024-05-19T20:04:52.2930101Z STATS: call_* op count: 179 | FakeTensor.__torch_dispatch__:6682 | FakeTensorMode.__torch_dispatch__:46028 | attempt fast:1521 | fast is_contiguous:1521 | ProxyTorchDispatchMode.__torch_dispatch__:11223 2024-05-19T20:04:52.2931429Z Dynamo produced 2 graphs covering 179 ops with 6 graph breaks (5 unique) 2024-05-19T20:04:57.3973179Z 2024-05-19T20:04:59.5288067Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:04:59.5288550Z loading model: 0it [00:02, ?it/s] 2024-05-19T20:04:59.5289010Z cuda train resnet50_quantized_qat 2024-05-19T20:04:59.5292813Z Traceback (most recent call last): 2024-05-19T20:04:59.5293855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2280, in validate_model 2024-05-19T20:04:59.5294871Z self.model_iter_fn(model, example_inputs) 2024-05-19T20:04:59.5295871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 432, in forward_and_backward_pass 2024-05-19T20:04:59.5296873Z pred = mod(*cloned_inputs) 2024-05-19T20:04:59.5298295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 736, in call_wrapped 2024-05-19T20:04:59.5299407Z return self._wrapped_call(self, *args, **kwargs) 2024-05-19T20:04:59.5300700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 315, in __call__ 2024-05-19T20:04:59.5301483Z raise e 2024-05-19T20:04:59.5302394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 302, in __call__ 2024-05-19T20:04:59.5303791Z return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] 2024-05-19T20:04:59.5304944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T20:04:59.5305827Z return self._call_impl(*args, **kwargs) 2024-05-19T20:04:59.5306757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T20:04:59.5307599Z return forward_call(*args, **kwargs) 2024-05-19T20:04:59.5308078Z File ".3", line 167, in forward 2024-05-19T20:04:59.5308721Z activation_post_process_73 = self.activation_post_process_73(fc); fc = None 2024-05-19T20:04:59.5309880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T20:04:59.5311071Z return self._call_impl(*args, **kwargs) 2024-05-19T20:04:59.5312012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T20:04:59.5312862Z return forward_call(*args, **kwargs) 2024-05-19T20:04:59.5313827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 342, in forward 2024-05-19T20:04:59.5314746Z return torch.fused_moving_avg_obs_fake_quant( 2024-05-19T20:04:59.5315308Z RuntimeError: expected scalar type Float but found Half 2024-05-19T20:04:59.5315687Z 2024-05-19T20:04:59.5315998Z The above exception was the direct cause of the following exception: 2024-05-19T20:04:59.5316448Z 2024-05-19T20:04:59.5316598Z Traceback (most recent call last): 2024-05-19T20:04:59.5317261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4109, in run 2024-05-19T20:04:59.5317929Z ) = runner.load_model( 2024-05-19T20:04:59.5318604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-05-19T20:04:59.5319370Z self.validate_model(model, example_inputs) 2024-05-19T20:04:59.5320148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2282, in validate_model 2024-05-19T20:04:59.5320924Z raise RuntimeError("Eager run failed") from e 2024-05-19T20:04:59.5321401Z RuntimeError: Eager run failed 2024-05-19T20:04:59.5321662Z 2024-05-19T20:04:59.5321782Z eager_fail_to_run 2024-05-19T20:05:02.6901008Z 2024-05-19T20:05:04.7881292Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:05:04.7882031Z loading model: 0it [00:02, ?it/s] 2024-05-19T20:05:04.7882483Z cuda train resnext50_32x4d 2024-05-19T20:05:52.9944256Z W0519 20:05:52.993000 139680943805056 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T20:06:36.5761556Z pass 2024-05-19T20:06:36.5877476Z TIMING: entire_frame_compile:79.85353 code_gen:22.24845 inductor_compile:44.06627 backend_compile:68.45622 2024-05-19T20:06:36.5879610Z STATS: call_* op count: 680 | FakeTensor.__torch_dispatch__:14353 | FakeTensorMode.__torch_dispatch__:89098 | attempt fast:1521 | fast is_contiguous:1521 | ProxyTorchDispatchMode.__torch_dispatch__:18492 2024-05-19T20:06:36.5880988Z Dynamo produced 3 graphs covering 680 ops with 7 graph breaks (5 unique) 2024-05-19T20:06:43.1020533Z 2024-05-19T20:06:49.0800531Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:06:49.0801019Z loading model: 0it [00:05, ?it/s] 2024-05-19T20:06:49.0801466Z cuda train sam 2024-05-19T20:06:49.0809457Z Traceback (most recent call last): 2024-05-19T20:06:49.0810229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2280, in validate_model 2024-05-19T20:06:49.0810993Z self.model_iter_fn(model, example_inputs) 2024-05-19T20:06:49.0811848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_and_backward_pass 2024-05-19T20:06:49.0812687Z self.grad_scaler.scale(loss).backward() 2024-05-19T20:06:49.0813727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 523, in backward 2024-05-19T20:06:49.0814793Z torch.autograd.backward( 2024-05-19T20:06:49.0815866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 284, in backward 2024-05-19T20:06:49.0817003Z _engine_run_backward( 2024-05-19T20:06:49.0817907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 767, in _engine_run_backward 2024-05-19T20:06:49.0819092Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-05-19T20:06:49.0820040Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn 2024-05-19T20:06:49.0820572Z 2024-05-19T20:06:49.0820869Z The above exception was the direct cause of the following exception: 2024-05-19T20:06:49.0821326Z 2024-05-19T20:06:49.0821471Z Traceback (most recent call last): 2024-05-19T20:06:49.0822125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4109, in run 2024-05-19T20:06:49.0822786Z ) = runner.load_model( 2024-05-19T20:06:49.0826497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-05-19T20:06:49.0827462Z self.validate_model(model, example_inputs) 2024-05-19T20:06:49.0828284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2282, in validate_model 2024-05-19T20:06:49.0829097Z raise RuntimeError("Eager run failed") from e 2024-05-19T20:06:49.0829575Z RuntimeError: Eager run failed 2024-05-19T20:06:49.0829890Z 2024-05-19T20:06:49.0830297Z eager_fail_to_run 2024-05-19T20:06:52.2339367Z 2024-05-19T20:06:53.5925734Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:06:53.5926217Z loading model: 0it [00:01, ?it/s] 2024-05-19T20:06:53.5926791Z cuda train shufflenet_v2_x1_0 2024-05-19T20:07:48.0534672Z pass 2024-05-19T20:07:48.0545640Z TIMING: entire_frame_compile:50.3363 code_gen:14.30635 inductor_compile:30.45078 backend_compile:46.27432 2024-05-19T20:07:48.0548872Z STATS: call_* op count: 307 | FakeTensor.__torch_dispatch__:7857 | FakeTensorMode.__torch_dispatch__:53319 | attempt fast:1628 | fast is_contiguous:1628 | ProxyTorchDispatchMode.__torch_dispatch__:12199 2024-05-19T20:07:48.0551220Z Dynamo produced 2 graphs covering 307 ops with 6 graph breaks (5 unique) 2024-05-19T20:07:53.5130854Z 2024-05-19T20:07:54.0245203Z loading model: 0it [00:00, ?it/s]/opt/conda/envs/py_3.10/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:233: DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`. (Deprecated NumPy 1.24) 2024-05-19T20:07:54.0247250Z if not isinstance(terminated, (bool, np.bool8)): 2024-05-19T20:07:54.3212103Z 2024-05-19T20:07:54.3212581Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:07:54.3216350Z cuda train soft_actor_critic 2024-05-19T20:08:01.4797832Z W0519 20:08:01.478000 139778343760512 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T20:08:04.5580178Z pass 2024-05-19T20:08:04.5586981Z TIMING: entire_frame_compile:7.89545 code_gen:3.18067 inductor_compile:4.14285 backend_compile:7.12041 2024-05-19T20:08:04.5588426Z STATS: call_* op count: 59 | FakeTensor.__torch_dispatch__:576 | FakeTensorMode.__torch_dispatch__:4519 | attempt fast:70 | fast is_contiguous:70 | ProxyTorchDispatchMode.__torch_dispatch__:908 2024-05-19T20:08:04.5590319Z Dynamo produced 4 graphs covering 59 ops with 6 graph breaks (5 unique) 2024-05-19T20:08:08.1197455Z 2024-05-19T20:08:09.3449965Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:08:09.3451159Z loading model: 0it [00:01, ?it/s] 2024-05-19T20:08:09.3452258Z Traceback (most recent call last): 2024-05-19T20:08:09.3454023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 449, in 2024-05-19T20:08:09.3455738Z torchbench_main() 2024-05-19T20:08:09.3456951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 445, in torchbench_main 2024-05-19T20:08:09.3458380Z main(TorchBenchmarkRunner(), original_dir) 2024-05-19T20:08:09.3459156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3650, in main 2024-05-19T20:08:09.3459883Z process_entry(0, runner, original_dir, args) 2024-05-19T20:08:09.3460779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3582, in process_entry 2024-05-19T20:08:09.3461945Z return run(runner, args, original_dir) 2024-05-19T20:08:09.3462768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4163, in run 2024-05-19T20:08:09.3467153Z assert marked, f"nothing in example_inputs had a dim with {batch_size}" 2024-05-19T20:08:09.3467897Z AssertionError: nothing in example_inputs had a dim with 32 2024-05-19T20:08:09.9732435Z Run failed with return code: 1 2024-05-19T20:08:09.9732854Z Output: None 2024-05-19T20:08:09.9733147Z Error: None 2024-05-19T20:08:12.4294399Z 2024-05-19T20:08:13.7759126Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:08:13.7759753Z loading model: 0it [00:01, ?it/s] 2024-05-19T20:08:13.7760286Z cuda train squeezenet1_1 2024-05-19T20:08:38.0685838Z pass 2024-05-19T20:08:38.0708365Z TIMING: entire_frame_compile:21.28832 code_gen:12.26543 inductor_compile:16.13281 backend_compile:20.20683 2024-05-19T20:08:38.0710759Z STATS: call_* op count: 70 | FakeTensor.__torch_dispatch__:1285 | FakeTensorMode.__torch_dispatch__:11426 | ProxyTorchDispatchMode.__torch_dispatch__:2530 | attempt fast:16 | fast is_contiguous:16 2024-05-19T20:08:38.0712064Z Dynamo produced 2 graphs covering 70 ops with 6 graph breaks (5 unique) 2024-05-19T20:08:42.0410362Z 2024-05-19T20:08:43.1540918Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:08:43.1541355Z 2024-05-19T20:08:43.4429729Z Loading pipeline components...: 0% 0/6 [00:00 will be ignored 2024-05-19T20:12:09.9245734Z pass 2024-05-19T20:12:09.9295706Z TIMING: entire_frame_compile:123.18767 code_gen:31.7879 inductor_compile:66.8993 backend_compile:103.41581 2024-05-19T20:12:09.9297375Z STATS: call_* op count: 974 | FakeTensor.__torch_dispatch__:20092 | FakeTensorMode.__torch_dispatch__:123124 | attempt fast:2535 | fast is_contiguous:2535 | ProxyTorchDispatchMode.__torch_dispatch__:24017 2024-05-19T20:12:09.9299775Z Dynamo produced 3 graphs covering 974 ops with 7 graph breaks (5 unique) 2024-05-19T20:12:18.5108655Z 2024-05-19T20:12:21.4455889Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:12:21.4456457Z loading model: 0it [00:02, ?it/s] 2024-05-19T20:12:21.4456908Z cuda train timm_regnet 2024-05-19T20:13:47.9929070Z pass 2024-05-19T20:13:48.0307028Z TIMING: entire_frame_compile:70.44895 code_gen:13.15503 inductor_compile:36.10267 backend_compile:59.22162 2024-05-19T20:13:48.0309354Z STATS: call_* op count: 463 | FakeTensor.__torch_dispatch__:9972 | FakeTensorMode.__torch_dispatch__:74238 | attempt fast:2248 | fast is_contiguous:2248 | ProxyTorchDispatchMode.__torch_dispatch__:17103 2024-05-19T20:13:48.0311049Z Dynamo produced 2 graphs covering 463 ops with 6 graph breaks (5 unique) 2024-05-19T20:13:54.6996781Z 2024-05-19T20:13:56.8297495Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:13:56.8298633Z loading model: 0it [00:02, ?it/s] 2024-05-19T20:13:56.8299600Z cuda train timm_resnest 2024-05-19T20:14:39.9786782Z W0519 20:14:39.977000 140492277531264 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T20:15:03.9743648Z pass 2024-05-19T20:15:03.9810118Z TIMING: entire_frame_compile:58.60308 code_gen:23.82636 inductor_compile:38.32841 backend_compile:51.82921 2024-05-19T20:15:03.9812312Z STATS: call_* op count: 431 | FakeTensor.__torch_dispatch__:7758 | FakeTensorMode.__torch_dispatch__:48921 | attempt fast:802 | fast is_contiguous:786 | ProxyTorchDispatchMode.__torch_dispatch__:9820 | slow no contiguity match:16 2024-05-19T20:15:03.9814086Z Dynamo produced 3 graphs covering 431 ops with 7 graph breaks (5 unique) 2024-05-19T20:15:09.4623523Z 2024-05-19T20:15:11.0669158Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:15:11.0669636Z loading model: 0it [00:01, ?it/s] 2024-05-19T20:15:11.0670235Z cuda train timm_vision_transformer 2024-05-19T20:16:01.7630624Z pass 2024-05-19T20:16:01.7798759Z TIMING: entire_frame_compile:46.18417 code_gen:9.0935 inductor_compile:24.18328 backend_compile:40.53979 2024-05-19T20:16:01.7802199Z STATS: call_* op count: 381 | FakeTensor.__torch_dispatch__:6173 | FakeTensorMode.__torch_dispatch__:48362 | attempt fast:2323 | fast is_contiguous:2323 | ProxyTorchDispatchMode.__torch_dispatch__:10943 2024-05-19T20:16:01.7803559Z Dynamo produced 2 graphs covering 381 ops with 6 graph breaks (5 unique) 2024-05-19T20:16:07.3750684Z 2024-05-19T20:16:19.8689193Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:16:19.8689743Z loading model: 0it [00:12, ?it/s] 2024-05-19T20:16:19.8690208Z cuda train timm_vision_transformer_large 2024-05-19T20:16:19.8692719Z pass_due_to_skip 2024-05-19T20:16:20.1317430Z TIMING: 2024-05-19T20:16:20.1317733Z STATS: call_* op count: 0 2024-05-19T20:16:20.1318423Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-05-19T20:16:23.3354290Z 2024-05-19T20:16:25.4642539Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:16:25.4643438Z loading model: 0it [00:02, ?it/s] 2024-05-19T20:16:25.4643909Z cuda train timm_vovnet 2024-05-19T20:17:22.1798330Z pass 2024-05-19T20:17:22.1975842Z TIMING: entire_frame_compile:47.13856 code_gen:16.84115 inductor_compile:28.83003 backend_compile:40.42538 2024-05-19T20:17:22.1978740Z STATS: call_* op count: 174 | FakeTensor.__torch_dispatch__:4772 | FakeTensorMode.__torch_dispatch__:35779 | attempt fast:1160 | fast is_contiguous:1100 | ProxyTorchDispatchMode.__torch_dispatch__:8526 | slow no contiguity match:60 2024-05-19T20:17:22.1980251Z Dynamo produced 2 graphs covering 174 ops with 6 graph breaks (5 unique) 2024-05-19T20:17:27.3388969Z 2024-05-19T20:17:29.2724157Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:17:29.2724640Z loading model: 0it [00:01, ?it/s] 2024-05-19T20:17:29.2728365Z cuda train torch_multimodal_clip 2024-05-19T20:18:29.3130471Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_302) 2024-05-19T20:18:53.7533132Z W0519 20:18:53.752000 140506528117376 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T20:20:05.1507437Z pass 2024-05-19T20:20:05.2317549Z TIMING: entire_frame_compile:146.26759 code_gen:41.58182 inductor_compile:83.65203 backend_compile:129.59442 2024-05-19T20:20:05.2322291Z STATS: call_* op count: 963 | FakeTensor.__torch_dispatch__:25505 | FakeTensorMode.__torch_dispatch__:177147 | attempt fast:2222 | fast is_contiguous:2140 | slow no contiguity match:82 | ProxyTorchDispatchMode.__torch_dispatch__:40036 2024-05-19T20:20:05.2324053Z Dynamo produced 3 graphs covering 963 ops with 7 graph breaks (5 unique) 2024-05-19T20:20:14.6321772Z 2024-05-19T20:20:15.2232331Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:20:15.2232819Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:20:15.2233266Z cuda train tts_angular 2024-05-19T20:20:19.1573626Z W0519 20:20:19.156000 140110540771968 torch/_logging/_internal.py:1024] [10/0] Profiler function will be ignored 2024-05-19T20:20:25.9286988Z pass 2024-05-19T20:20:25.9351210Z TIMING: entire_frame_compile:7.86069 inductor_compile:6.50298 backend_compile:7.07774 code_gen:5.54939 2024-05-19T20:20:25.9352507Z STATS: call_* op count: 71 | FakeTensorMode.__torch_dispatch__:4838 | ProxyTorchDispatchMode.__torch_dispatch__:871 | FakeTensor.__torch_dispatch__:860 2024-05-19T20:20:25.9353634Z Dynamo produced 5 graphs covering 71 ops with 9 graph breaks (6 unique) 2024-05-19T20:20:29.4849783Z 2024-05-19T20:20:31.9248018Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:20:31.9248933Z loading model: 0it [00:02, ?it/s] 2024-05-19T20:20:31.9251714Z cuda train vgg16 2024-05-19T20:21:10.5768353Z pass 2024-05-19T20:21:10.6010257Z TIMING: entire_frame_compile:13.0663 code_gen:7.03519 inductor_compile:9.84679 backend_compile:12.40112 2024-05-19T20:21:10.6013573Z STATS: call_* op count: 44 | FakeTensor.__torch_dispatch__:808 | FakeTensorMode.__torch_dispatch__:7531 | attempt fast:36 | fast is_contiguous:36 | ProxyTorchDispatchMode.__torch_dispatch__:1686 2024-05-19T20:21:10.6016144Z Dynamo produced 2 graphs covering 44 ops with 6 graph breaks (5 unique) 2024-05-19T20:21:14.4310223Z 2024-05-19T20:21:17.7677901Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:21:17.7680227Z loading model: 0it [00:03, ?it/s] 2024-05-19T20:21:17.7680752Z cuda train vision_maskrcnn 2024-05-19T20:21:49.6420781Z W0519 20:21:49.641000 140034694750848 torch/_inductor/utils.py:1130] [5/0_1] DeviceCopy in input program 2024-05-19T20:21:49.6429515Z W0519 20:21:49.642000 140034694750848 torch/_inductor/utils.py:1130] [5/0_1] DeviceCopy in input program 2024-05-19T20:21:49.6440985Z W0519 20:21:49.643000 140034694750848 torch/_inductor/utils.py:1130] [5/0_1] DeviceCopy in input program 2024-05-19T20:21:49.6452774Z W0519 20:21:49.644000 140034694750848 torch/_inductor/utils.py:1130] [5/0_1] DeviceCopy in input program 2024-05-19T20:21:49.6463808Z W0519 20:21:49.645000 140034694750848 torch/_inductor/utils.py:1130] [5/0_1] DeviceCopy in input program 2024-05-19T20:21:51.4501828Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_7). Found from : 2024-05-19T20:21:51.4503594Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/rpn.py", line 362, in forward 2024-05-19T20:21:51.4504754Z anchors = self.anchor_generator(images, features) 2024-05-19T20:21:51.4506075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T20:21:51.4506958Z return forward_call(*args, **kwargs) 2024-05-19T20:21:51.4508239Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 126, in forward 2024-05-19T20:21:51.4509293Z self.set_cell_anchors(dtype, device) 2024-05-19T20:21:51.4510681Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in set_cell_anchors 2024-05-19T20:21:51.4511932Z self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors] 2024-05-19T20:21:51.4513233Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in 2024-05-19T20:21:51.4514434Z self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors] 2024-05-19T20:21:51.4515018Z 2024-05-19T20:21:52.8966545Z W0519 20:21:52.896000 140034694750848 torch/_dynamo/variables/tensor.py:700] [18/0] Graph break from `Tensor.item()`, consider setting: 2024-05-19T20:21:52.8968097Z W0519 20:21:52.896000 140034694750848 torch/_dynamo/variables/tensor.py:700] [18/0] torch._dynamo.config.capture_scalar_outputs = True 2024-05-19T20:21:52.8969190Z W0519 20:21:52.896000 140034694750848 torch/_dynamo/variables/tensor.py:700] [18/0] or: 2024-05-19T20:21:52.8970255Z W0519 20:21:52.896000 140034694750848 torch/_dynamo/variables/tensor.py:700] [18/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-05-19T20:21:52.8971767Z W0519 20:21:52.896000 140034694750848 torch/_dynamo/variables/tensor.py:700] [18/0] to include these operations in the captured graph. 2024-05-19T20:21:52.8972846Z W0519 20:21:52.896000 140034694750848 torch/_dynamo/variables/tensor.py:700] [18/0] 2024-05-19T20:22:04.5707466Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T20:22:04.5708721Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-05-19T20:22:04.5710512Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-05-19T20:22:04.5711108Z 2024-05-19T20:22:10.8047761Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T20:22:10.8048999Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-05-19T20:22:10.8050149Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-05-19T20:22:10.8050749Z 2024-05-19T20:22:17.1447601Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T20:22:17.1448854Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-05-19T20:22:17.1449986Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-05-19T20:22:17.1450572Z 2024-05-19T20:22:53.5695440Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T20:22:53.5697431Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-05-19T20:22:53.5699363Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-05-19T20:22:53.5700348Z 2024-05-19T20:23:02.9866498Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T20:23:02.9868017Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-05-19T20:23:02.9870465Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-05-19T20:23:02.9871195Z 2024-05-19T20:23:16.5923373Z W0519 20:23:16.591000 140034694750848 torch/fx/experimental/symbolic_shapes.py:4211] [30/6] RecursionError in sympy.xreplace(Eq(Mod(2*s4, s3), 0), {s4: evaluate_static_shape_0 + 1, s3: evaluate_static_shape_1 + 1}) 2024-05-19T20:23:18.5786517Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T20:23:18.5787958Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-05-19T20:23:18.5789078Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-05-19T20:23:18.5789656Z 2024-05-19T20:23:24.3874916Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T20:23:24.3877487Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-05-19T20:23:24.3878649Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-05-19T20:23:24.3879224Z 2024-05-19T20:23:31.1775109Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T20:23:31.1776672Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 415, in torch_dynamo_resume_in_paste_mask_in_image_at_407 2024-05-19T20:23:31.1777979Z mask = F.interpolate(mask, size=(h, w), mode="bilinear", align_corners=False) 2024-05-19T20:23:31.1778476Z 2024-05-19T20:23:32.0840257Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T20:23:32.0841845Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 415, in torch_dynamo_resume_in_paste_mask_in_image_at_407 2024-05-19T20:23:32.0843373Z mask = F.interpolate(mask, size=(h, w), mode="bilinear", align_corners=False) 2024-05-19T20:23:32.0843867Z 2024-05-19T20:23:33.0644804Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T20:23:33.0646558Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 821, in torch_dynamo_resume_in_forward_at_804 2024-05-19T20:23:33.0647840Z masks_probs = maskrcnn_inference(mask_logits, labels) 2024-05-19T20:23:33.0649339Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 76, in maskrcnn_inference 2024-05-19T20:23:33.0650395Z mask_prob = mask_prob[index, labels][:, None] 2024-05-19T20:23:33.0650813Z 2024-05-19T20:23:34.9915621Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T20:23:34.9916880Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-05-19T20:23:34.9918033Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-05-19T20:23:34.9918617Z 2024-05-19T20:23:38.7387699Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T20:23:38.7389316Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/generalized_rcnn.py", line 101, in forward 2024-05-19T20:23:38.7390590Z features = self.backbone(images.tensors) 2024-05-19T20:23:38.7391893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T20:23:38.7393065Z return forward_call(*args, **kwargs) 2024-05-19T20:23:38.7394227Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/backbone_utils.py", line 58, in forward 2024-05-19T20:23:38.7395102Z x = self.fpn(x) 2024-05-19T20:23:38.7395932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T20:23:38.7396775Z return forward_call(*args, **kwargs) 2024-05-19T20:23:38.7397763Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/feature_pyramid_network.py", line 194, in forward 2024-05-19T20:23:38.7398821Z inner_top_down = F.interpolate(last_inner, size=feat_shape, mode="nearest") 2024-05-19T20:23:38.7399294Z 2024-05-19T20:23:43.3279591Z W0519 20:23:43.327000 140034694750848 torch/_dynamo/convert_frame.py:752] [30/8] torch._dynamo hit config.cache_size_limit (8) 2024-05-19T20:23:43.3281699Z W0519 20:23:43.327000 140034694750848 torch/_dynamo/convert_frame.py:752] [30/8] function: 'roi_align' (/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py:186) 2024-05-19T20:23:43.3283514Z W0519 20:23:43.327000 140034694750848 torch/_dynamo/convert_frame.py:752] [30/8] last reason: tensor 'L['boxes']' size mismatch at index 0. expected 648, actual 623 2024-05-19T20:23:43.3284994Z W0519 20:23:43.327000 140034694750848 torch/_dynamo/convert_frame.py:752] [30/8] To log all recompilation reasons, use TORCH_LOGS="recompiles". 2024-05-19T20:23:43.3286594Z W0519 20:23:43.327000 140034694750848 torch/_dynamo/convert_frame.py:752] [30/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html. 2024-05-19T20:23:51.2850335Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T20:23:51.2851772Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-05-19T20:23:51.2852912Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-05-19T20:23:51.2854248Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-05-19T20:23:51.2855244Z v1 = masked_index(y_low, x_low) 2024-05-19T20:23:51.2856474Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-05-19T20:23:51.2857430Z return input[ 2024-05-19T20:23:51.2857629Z 2024-05-19T20:23:58.3516581Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T20:23:58.3517962Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-05-19T20:23:58.3519078Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-05-19T20:23:58.3520606Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-05-19T20:23:58.3521498Z v1 = masked_index(y_low, x_low) 2024-05-19T20:23:58.3522400Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-05-19T20:23:58.3523207Z return input[ 2024-05-19T20:23:58.3523408Z 2024-05-19T20:24:04.1477649Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T20:24:04.1478919Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-05-19T20:24:04.1480013Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-05-19T20:24:04.1481249Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-05-19T20:24:04.1482128Z v1 = masked_index(y_low, x_low) 2024-05-19T20:24:04.1483041Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-05-19T20:24:04.1483847Z return input[ 2024-05-19T20:24:04.1484040Z 2024-05-19T20:24:18.1945103Z W0519 20:24:18.193000 140034694750848 torch/fx/experimental/symbolic_shapes.py:4211] [54/4] RecursionError in sympy.xreplace(Eq(Mod(2*s4, s3), 0), {s4: evaluate_static_shape_0 + 1, s3: evaluate_static_shape_1 + 1}) 2024-05-19T20:24:19.3739287Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T20:24:19.3740840Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-05-19T20:24:19.3741985Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-05-19T20:24:19.3743223Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-05-19T20:24:19.3744106Z v1 = masked_index(y_low, x_low) 2024-05-19T20:24:19.3745033Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-05-19T20:24:19.3745846Z return input[ 2024-05-19T20:24:19.3746041Z 2024-05-19T20:24:28.2309553Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T20:24:28.2311146Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-05-19T20:24:28.2312277Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-05-19T20:24:28.2313524Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-05-19T20:24:28.2314395Z v1 = masked_index(y_low, x_low) 2024-05-19T20:24:28.2315291Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-05-19T20:24:28.2316097Z return input[ 2024-05-19T20:24:28.2316305Z 2024-05-19T20:24:38.7326814Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T20:24:38.7328222Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-05-19T20:24:38.7329337Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-05-19T20:24:38.7330861Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-05-19T20:24:38.7331751Z v1 = masked_index(y_low, x_low) 2024-05-19T20:24:38.7332660Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-05-19T20:24:38.7333473Z return input[ 2024-05-19T20:24:38.7333666Z 2024-05-19T20:24:44.4090746Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T20:24:44.4092368Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-05-19T20:24:44.4094086Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-05-19T20:24:44.4095364Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-05-19T20:24:44.4096243Z v1 = masked_index(y_low, x_low) 2024-05-19T20:24:44.4097163Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-05-19T20:24:44.4097962Z return input[ 2024-05-19T20:24:44.4098164Z 2024-05-19T20:24:45.4734412Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T20:24:45.4735879Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-05-19T20:24:45.4737045Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-05-19T20:24:45.4738452Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-05-19T20:24:45.4739416Z v1 = masked_index(y_low, x_low) 2024-05-19T20:24:45.4740415Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-05-19T20:24:45.4741301Z return input[ 2024-05-19T20:24:45.4741527Z 2024-05-19T20:24:45.8090629Z pass 2024-05-19T20:24:45.8655241Z TIMING: entire_frame_compile:189.3069 inductor_compile:91.7387 backend_compile:149.44467 code_gen:47.69554 2024-05-19T20:24:45.8657057Z STATS: call_* op count: 2515 | FakeTensorMode.__torch_dispatch__:185304 | FakeTensor.__torch_dispatch__:14870 | ProxyTorchDispatchMode.__torch_dispatch__:45349 | attempt fast:3042 | slow no contiguity match:290 | fast is_contiguous:2100 | slow both tensors nontrivially broadcast:652 2024-05-19T20:24:45.8658792Z Dynamo produced 51 graphs covering 2515 ops with 34 graph breaks (8 unique) 2024-05-19T20:24:55.5157645Z 2024-05-19T20:24:58.5407465Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:24:58.5407940Z loading model: 0it [00:03, ?it/s] 2024-05-19T20:24:58.5408384Z cuda train yolov3 2024-05-19T20:25:41.7093622Z W0519 20:25:41.708000 139847316370048 torch/_inductor/utils.py:1130] [9/0] DeviceCopy in input program 2024-05-19T20:25:41.7097817Z W0519 20:25:41.709000 139847316370048 torch/_inductor/utils.py:1130] [9/0] DeviceCopy in input program 2024-05-19T20:25:42.6594218Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_2). Found from : 2024-05-19T20:25:42.6597038Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward 2024-05-19T20:25:42.6598222Z self.create_grids((nx, ny), p.device) 2024-05-19T20:25:42.6599133Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids 2024-05-19T20:25:42.6600357Z self.anchor_vec = self.anchor_vec.to(device) 2024-05-19T20:25:42.6600810Z 2024-05-19T20:25:42.9755399Z W0519 20:25:42.974000 139847316370048 torch/_inductor/utils.py:1130] [9/1] DeviceCopy in input program 2024-05-19T20:25:42.9759608Z W0519 20:25:42.975000 139847316370048 torch/_inductor/utils.py:1130] [9/1] DeviceCopy in input program 2024-05-19T20:25:44.1085143Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_4). Found from : 2024-05-19T20:25:44.1086742Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward 2024-05-19T20:25:44.1087645Z self.create_grids((nx, ny), p.device) 2024-05-19T20:25:44.1088542Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids 2024-05-19T20:25:44.1089467Z self.anchor_vec = self.anchor_vec.to(device) 2024-05-19T20:25:44.1089803Z 2024-05-19T20:25:44.6423302Z W0519 20:25:44.641000 139847316370048 torch/_logging/_internal.py:1024] [13/0] Profiler function will be ignored 2024-05-19T20:27:00.4947340Z pass 2024-05-19T20:27:00.5462018Z TIMING: entire_frame_compile:79.5702 inductor_compile:41.05459 backend_compile:56.39162 code_gen:27.89198 2024-05-19T20:27:00.5464383Z STATS: call_* op count: 743 | FakeTensor.__torch_dispatch__:13760 | FakeTensorMode.__torch_dispatch__:86411 | attempt fast:1294 | fast is_contiguous:1294 | ProxyTorchDispatchMode.__torch_dispatch__:10810 2024-05-19T20:27:00.5465783Z Dynamo produced 31 graphs covering 743 ops with 9 graph breaks (6 unique) 2024-05-19T20:27:04.6689477Z accuracy pass_rate=76.32% 2024-05-19T20:27:04.6692553Z calls_captured gmean=0.00x mean=399.079x 2024-05-19T20:27:04.6695869Z unique_graphs gmean=0.00x mean=4.342x 2024-05-19T20:27:04.6699415Z graph_breaks gmean=0.00x mean=6.211x 2024-05-19T20:27:04.6702786Z unique_graph_breaks gmean=0.00x mean=4.289x 2024-05-19T20:27:04.6706185Z autograd_captures gmean=0.00x mean=0.000x 2024-05-19T20:27:04.6709568Z autograd_compiles gmean=0.00x mean=0.000x 2024-05-19T20:27:04.6714417Z cudagraph_skips gmean=0.00x mean=0.737x 2024-05-19T20:27:05.8312179Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_inductor_torchbench_training.csv 2024-05-19T20:27:06.1623771Z lennard_jones PASS 2024-05-19T20:27:06.1626715Z llava XFAIL 2024-05-19T20:27:06.1632061Z maml_omniglot PASS 2024-05-19T20:27:06.1637211Z mnasnet1_0 PASS 2024-05-19T20:27:06.1641651Z mobilenet_v2 PASS 2024-05-19T20:27:06.1646434Z mobilenet_v2_quantized_qat XFAIL 2024-05-19T20:27:06.1651165Z mobilenet_v3_large PASS 2024-05-19T20:27:06.1656270Z moco PASS 2024-05-19T20:27:06.1660918Z nanogpt PASS 2024-05-19T20:27:06.1665573Z nvidia_deeprecommender PASS 2024-05-19T20:27:06.1670577Z opacus_cifar10 XFAIL 2024-05-19T20:27:06.1675238Z phlippe_densenet PASS 2024-05-19T20:27:06.1679854Z phlippe_resnet FAIL: accuracy=fail_accuracy, expected=pass 2024-05-19T20:27:06.1684506Z pytorch_CycleGAN_and_pix2pix PASS 2024-05-19T20:27:06.1689289Z pytorch_stargan PASS 2024-05-19T20:27:06.1693995Z pytorch_unet XFAIL 2024-05-19T20:27:06.1698569Z resnet152 PASS 2024-05-19T20:27:06.1703148Z resnet18 PASS 2024-05-19T20:27:06.1707693Z resnet50 PASS 2024-05-19T20:27:06.1712670Z resnet50_quantized_qat XFAIL 2024-05-19T20:27:06.1717597Z resnext50_32x4d PASS 2024-05-19T20:27:06.1722514Z sam XFAIL 2024-05-19T20:27:06.1727188Z shufflenet_v2_x1_0 PASS 2024-05-19T20:27:06.1731952Z soft_actor_critic PASS 2024-05-19T20:27:06.1736513Z squeezenet1_1 PASS 2024-05-19T20:27:06.1740958Z stable_diffusion_text_encoder PASS 2024-05-19T20:27:06.1745781Z stable_diffusion_unet XFAIL 2024-05-19T20:27:06.1750648Z timm_efficientnet PASS 2024-05-19T20:27:06.1755413Z timm_regnet PASS 2024-05-19T20:27:06.1759964Z timm_resnest PASS 2024-05-19T20:27:06.1764994Z timm_vision_transformer PASS 2024-05-19T20:27:06.1769530Z timm_vision_transformer_large XFAIL 2024-05-19T20:27:06.1774169Z timm_vovnet PASS 2024-05-19T20:27:06.1778841Z torch_multimodal_clip PASS 2024-05-19T20:27:06.1783336Z tts_angular PASS 2024-05-19T20:27:06.1788008Z vgg16 PASS 2024-05-19T20:27:06.1792892Z vision_maskrcnn PASS 2024-05-19T20:27:06.1797508Z yolov3 PASS 2024-05-19T20:27:06.1797987Z 2024-05-19T20:27:06.1801370Z Error: 1 models have accuracy status regressed: 2024-05-19T20:27:06.1801941Z phlippe_resnet 2024-05-19T20:27:06.1802224Z 2024-05-19T20:27:06.1802235Z 2024-05-19T20:27:06.1803108Z If this change is expected, you can update `benchmarks/dynamo/ci_expected_accuracy/dynamic_inductor_torchbench_training.csv` to reflect the new baseline. 2024-05-19T20:27:06.1804111Z from pytorch/pytorch root, run 2024-05-19T20:27:06.1804881Z `python benchmarks/dynamo/ci_expected_accuracy/update_expected.py c8c7ddfde57442cc28a394c385cab656ce9b1f5e` 2024-05-19T20:27:06.1805862Z and then `git add` the resulting local changes to expected CSVs to your commit. 2024-05-19T20:27:06.1806358Z 2024-05-19T20:27:06.2260269Z + cleanup_workspace 2024-05-19T20:27:06.2261418Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-05-19T20:27:06.2262859Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-05-19T20:27:06.2263828Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-05-19T20:27:06.2264469Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-05-19T20:27:06.2265274Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-05-19T20:27:06.2266144Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-05-19T20:27:06.2266848Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-05-19T20:27:06.7547036Z ##[error]Process completed with exit code 1. 2024-05-19T20:27:06.7620277Z Prepare all required actions 2024-05-19T20:27:06.7620720Z Getting action download info 2024-05-19T20:27:06.8993791Z ##[group]Run ./.github/actions/pytest-cache-upload 2024-05-19T20:27:06.8994266Z with: 2024-05-19T20:27:06.8994552Z cache_dir: .pytest_cache 2024-05-19T20:27:06.8994910Z shard: 2 2024-05-19T20:27:06.8995260Z sha: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T20:27:06.8995745Z test_config: dynamic_inductor_torchbench 2024-05-19T20:27:06.8996329Z job_identifier: inductor_linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T20:27:06.8996851Z env: 2024-05-19T20:27:06.8997132Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:27:06.8997580Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:27:06.8998315Z DOCKER_CONTAINER_ID: f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T20:27:06.8998950Z ##[endgroup] 2024-05-19T20:27:06.9038495Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-05-19T20:27:06.9039101Z with: 2024-05-19T20:27:06.9039371Z shell: bash 2024-05-19T20:27:06.9039680Z timeout_minutes: 5 2024-05-19T20:27:06.9040009Z max_attempts: 5 2024-05-19T20:27:06.9040333Z retry_wait_seconds: 30 2024-05-19T20:27:06.9040787Z command: set -eu python3 -m pip install boto3==1.19.12 2024-05-19T20:27:06.9041308Z polling_interval_seconds: 1 2024-05-19T20:27:06.9041688Z warning_on_retry: true 2024-05-19T20:27:06.9042053Z continue_on_error: false 2024-05-19T20:27:06.9042388Z env: 2024-05-19T20:27:06.9042672Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:27:06.9043137Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:27:06.9043876Z DOCKER_CONTAINER_ID: f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T20:27:06.9044515Z ##[endgroup] 2024-05-19T20:27:07.1697085Z Defaulting to user installation because normal site-packages is not writeable 2024-05-19T20:27:08.3841626Z Collecting boto3==1.19.12 2024-05-19T20:27:08.4022945Z Downloading boto3-1.19.12-py3-none-any.whl (131 kB) 2024-05-19T20:27:09.8247445Z Collecting botocore<1.23.0,>=1.22.12 2024-05-19T20:27:09.8331183Z Downloading botocore-1.22.12-py3-none-any.whl (8.1 MB) 2024-05-19T20:27:09.9928336Z Collecting s3transfer<0.6.0,>=0.5.0 2024-05-19T20:27:09.9968737Z Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 2024-05-19T20:27:10.0339023Z Collecting jmespath<1.0.0,>=0.7.1 2024-05-19T20:27:10.0385498Z Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 2024-05-19T20:27:10.0469672Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.18) 2024-05-19T20:27:10.1151486Z Collecting python-dateutil<3.0.0,>=2.1 2024-05-19T20:27:10.1193895Z Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) 2024-05-19T20:27:10.1641895Z Collecting six>=1.5 2024-05-19T20:27:10.1685800Z Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2024-05-19T20:27:10.2390272Z Installing collected packages: jmespath, six, python-dateutil, botocore, s3transfer, boto3 2024-05-19T20:27:10.9083294Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.9.0.post0 s3transfer-0.5.2 six-1.16.0 2024-05-19T20:27:11.9570556Z Command completed after 1 attempt(s). 2024-05-19T20:27:11.9613553Z ##[group]Run python3 .github/scripts/pytest_cache.py \ 2024-05-19T20:27:11.9614149Z python3 .github/scripts/pytest_cache.py \ 2024-05-19T20:27:11.9614641Z  --upload \ 2024-05-19T20:27:11.9615061Z  --cache_dir $GITHUB_WORKSPACE/$CACHE_DIR \ 2024-05-19T20:27:11.9615580Z  --pr_identifier $GITHUB_REF \ 2024-05-19T20:27:11.9616055Z  --job_identifier $JOB_IDENTIFIER \ 2024-05-19T20:27:11.9616500Z  --sha $SHA \ 2024-05-19T20:27:11.9616879Z  --test_config $TEST_CONFIG \ 2024-05-19T20:27:11.9617302Z  --shard $SHARD \ 2024-05-19T20:27:11.9617662Z  --repo $REPO \ 2024-05-19T20:27:11.9618159Z  --temp_dir $RUNNER_TEMP \ 2024-05-19T20:27:11.9626022Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T20:27:11.9626523Z env: 2024-05-19T20:27:11.9626799Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:27:11.9627258Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:27:11.9627993Z DOCKER_CONTAINER_ID: f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T20:27:11.9628655Z CACHE_DIR: .pytest_cache 2024-05-19T20:27:11.9629137Z JOB_IDENTIFIER: inductor_linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T20:27:11.9629730Z SHA: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T20:27:11.9630491Z TEST_CONFIG: dynamic_inductor_torchbench 2024-05-19T20:27:11.9630915Z SHARD: 2 2024-05-19T20:27:11.9631197Z REPO: pytorch/pytorch 2024-05-19T20:27:11.9631531Z ##[endgroup] 2024-05-19T20:27:12.0961304Z PR identifier for `refs/tags/ciflow/inductor/121956` is `5ec4bd29b9652d20d61d9e0a665b8d46` 2024-05-19T20:27:12.0964476Z Uploading cache with args Namespace(bucket=None, cache_dir='/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache', download=False, job_identifier='inductor_linux-focal-cuda12.4-py3.10-gcc9-sm86', pr_identifier='refs/tags/ciflow/inductor/121956', repo='pytorch/pytorch', sha='c8c7ddfde57442cc28a394c385cab656ce9b1f5e', shard='2', temp_dir='/home/ec2-user/actions-runner/_work/_temp', test_config='dynamic_inductor_torchbench', upload=True) 2024-05-19T20:27:12.0967513Z The pytest cache dir `/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache` does not exist. Skipping upload 2024-05-19T20:27:12.1134705Z ##[group]Run cat test/**/*_toprint.log || true 2024-05-19T20:27:12.1135225Z cat test/**/*_toprint.log || true 2024-05-19T20:27:12.1142889Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T20:27:12.1143390Z env: 2024-05-19T20:27:12.1143669Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:27:12.1144133Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:27:12.1144869Z DOCKER_CONTAINER_ID: f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T20:27:12.1145506Z ##[endgroup] 2024-05-19T20:27:12.1207822Z cat: test/**/*_toprint.log: No such file or directory 2024-05-19T20:27:12.1236608Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-05-19T20:27:12.1237076Z kill "$MONITOR_SCRIPT_PID" 2024-05-19T20:27:12.1244333Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T20:27:12.1244952Z env: 2024-05-19T20:27:12.1245234Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:27:12.1245694Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:27:12.1246434Z DOCKER_CONTAINER_ID: f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T20:27:12.1247200Z MONITOR_SCRIPT_PID: 14990 2024-05-19T20:27:12.1247553Z ##[endgroup] 2024-05-19T20:27:12.1430263Z Prepare all required actions 2024-05-19T20:27:12.1430709Z Getting action download info 2024-05-19T20:27:12.2493981Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-05-19T20:27:12.3985269Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-05-19T20:27:12.3985737Z with: 2024-05-19T20:27:12.3986303Z file-suffix: test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25153197615 2024-05-19T20:27:12.3987013Z s3-bucket: gha-artifacts 2024-05-19T20:27:12.3987357Z env: 2024-05-19T20:27:12.3987627Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:27:12.3988094Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:27:12.3988824Z DOCKER_CONTAINER_ID: f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T20:27:12.3989457Z ##[endgroup] 2024-05-19T20:27:12.4014761Z ##[group]Run # Remove any previous test jsons if they exist 2024-05-19T20:27:12.4015411Z # Remove any previous test jsons if they exist 2024-05-19T20:27:12.4015929Z rm -f test-jsons-*.zip 2024-05-19T20:27:12.4016454Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-05-19T20:27:12.4024136Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T20:27:12.4024636Z env: 2024-05-19T20:27:12.4024919Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:27:12.4025367Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:27:12.4026101Z DOCKER_CONTAINER_ID: f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T20:27:12.4027036Z FILE_SUFFIX: test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25153197615 2024-05-19T20:27:12.4027719Z ##[endgroup] 2024-05-19T20:27:12.4195600Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-05-19T20:27:12.4225186Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-05-19T20:27:12.4226535Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-05-19T20:27:12.4232010Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-05-19T20:27:12.4257830Z ##[group]Run # Remove any previous test reports if they exist 2024-05-19T20:27:12.4258479Z # Remove any previous test reports if they exist 2024-05-19T20:27:12.4259016Z rm -f test-reports-*.zip 2024-05-19T20:27:12.4259606Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-05-19T20:27:12.4267548Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T20:27:12.4268052Z env: 2024-05-19T20:27:12.4268335Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:27:12.4268809Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:27:12.4269546Z DOCKER_CONTAINER_ID: f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T20:27:12.4270847Z FILE_SUFFIX: test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25153197615 2024-05-19T20:27:12.4271527Z ##[endgroup] 2024-05-19T20:27:12.4431421Z adding: test/test-reports/inference_torchbench.csv (deflated 68%) 2024-05-19T20:27:12.4433465Z adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 93%) 2024-05-19T20:27:12.4434423Z adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 80%) 2024-05-19T20:27:12.4435291Z adding: test/test-reports/training_torchbench.csv (deflated 66%) 2024-05-19T20:27:12.4441771Z adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 96%) 2024-05-19T20:27:12.4442904Z adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 79%) 2024-05-19T20:27:12.4468904Z ##[group]Run # Remove any previous usage logs if they exist 2024-05-19T20:27:12.4469702Z # Remove any previous usage logs if they exist 2024-05-19T20:27:12.4470412Z rm -f logs-*.zip 2024-05-19T20:27:12.4471079Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-05-19T20:27:12.4471965Z # so check to see if the file exists first 2024-05-19T20:27:12.4472464Z if [ -f 'usage_log.txt' ]; then 2024-05-19T20:27:12.4472998Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-05-19T20:27:12.4473488Z fi 2024-05-19T20:27:12.4473856Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-05-19T20:27:12.4474431Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-05-19T20:27:12.4474923Z fi 2024-05-19T20:27:12.4482107Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T20:27:12.4482603Z env: 2024-05-19T20:27:12.4482884Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:27:12.4483346Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:27:12.4484085Z DOCKER_CONTAINER_ID: f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T20:27:12.4484999Z FILE_SUFFIX: test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25153197615 2024-05-19T20:27:12.4485675Z ##[endgroup] 2024-05-19T20:27:12.4715056Z adding: usage_log.txt (deflated 96%) 2024-05-19T20:27:12.4787666Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-05-19T20:27:12.4788548Z # Remove any previous debugging artifacts if they exist 2024-05-19T20:27:12.4789102Z rm -f debug-*.zip 2024-05-19T20:27:12.4789484Z if [ -d 'test/debug' ]; then 2024-05-19T20:27:12.4790160Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-05-19T20:27:12.4790640Z fi 2024-05-19T20:27:12.4797510Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T20:27:12.4798015Z env: 2024-05-19T20:27:12.4798289Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:27:12.4798757Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:27:12.4799495Z DOCKER_CONTAINER_ID: f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T20:27:12.4800417Z FILE_SUFFIX: test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25153197615 2024-05-19T20:27:12.4801101Z ##[endgroup] 2024-05-19T20:27:12.4874663Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-05-19T20:27:12.4875101Z with: 2024-05-19T20:27:12.4875395Z s3-bucket: gha-artifacts 2024-05-19T20:27:12.4875831Z s3-prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T20:27:12.4876309Z retention-days: 14 2024-05-19T20:27:12.4876649Z if-no-files-found: warn 2024-05-19T20:27:12.4877020Z path: test-jsons-*.zip 2024-05-19T20:27:12.4877368Z name: artifact 2024-05-19T20:27:12.4877671Z region: us-east-1 2024-05-19T20:27:12.4877980Z env: 2024-05-19T20:27:12.4878263Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:27:12.4878720Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:27:12.4879470Z DOCKER_CONTAINER_ID: f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T20:27:12.4880113Z ##[endgroup] 2024-05-19T20:27:12.7984886Z NOTE: s3-prefix specified, ignoring name parameter 2024-05-19T20:27:12.7985495Z With the provided path, there will be 1 file uploaded 2024-05-19T20:27:12.7986124Z Uploading to s3 prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T20:27:12.8011613Z Starting upload of test-jsons-test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25153197615.zip 2024-05-19T20:27:12.9072858Z Finished upload of test-jsons-test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25153197615.zip 2024-05-19T20:27:12.9213795Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-05-19T20:27:12.9214247Z with: 2024-05-19T20:27:12.9214528Z s3-bucket: gha-artifacts 2024-05-19T20:27:12.9214958Z s3-prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T20:27:12.9215428Z retention-days: 14 2024-05-19T20:27:12.9215889Z if-no-files-found: error 2024-05-19T20:27:12.9216256Z path: test-reports-*.zip 2024-05-19T20:27:12.9216612Z name: artifact 2024-05-19T20:27:12.9216920Z region: us-east-1 2024-05-19T20:27:12.9217213Z env: 2024-05-19T20:27:12.9217490Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:27:12.9218035Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:27:12.9218778Z DOCKER_CONTAINER_ID: f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T20:27:12.9219409Z ##[endgroup] 2024-05-19T20:27:13.2305064Z NOTE: s3-prefix specified, ignoring name parameter 2024-05-19T20:27:13.2305687Z With the provided path, there will be 1 file uploaded 2024-05-19T20:27:13.2306313Z Uploading to s3 prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T20:27:13.2331952Z Starting upload of test-reports-test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25153197615.zip 2024-05-19T20:27:13.3813458Z Finished upload of test-reports-test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25153197615.zip 2024-05-19T20:27:13.3954491Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-05-19T20:27:13.3954948Z with: 2024-05-19T20:27:13.3955222Z s3-bucket: gha-artifacts 2024-05-19T20:27:13.3955667Z s3-prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T20:27:13.3956145Z retention-days: 14 2024-05-19T20:27:13.3956494Z if-no-files-found: ignore 2024-05-19T20:27:13.3956859Z path: logs-*.zip 2024-05-19T20:27:13.3957180Z name: artifact 2024-05-19T20:27:13.3957487Z region: us-east-1 2024-05-19T20:27:13.3957785Z env: 2024-05-19T20:27:13.3958066Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:27:13.3958535Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:27:13.3959283Z DOCKER_CONTAINER_ID: f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T20:27:13.3959931Z ##[endgroup] 2024-05-19T20:27:13.7114317Z NOTE: s3-prefix specified, ignoring name parameter 2024-05-19T20:27:13.7115526Z With the provided path, there will be 1 file uploaded 2024-05-19T20:27:13.7116787Z Uploading to s3 prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T20:27:13.7139724Z Starting upload of logs-test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25153197615.zip 2024-05-19T20:27:13.9384295Z Finished upload of logs-test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_25153197615.zip 2024-05-19T20:27:13.9522654Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-05-19T20:27:13.9523096Z with: 2024-05-19T20:27:13.9523431Z s3-bucket: gha-artifacts 2024-05-19T20:27:13.9523867Z s3-prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T20:27:13.9524339Z retention-days: 14 2024-05-19T20:27:13.9524673Z if-no-files-found: ignore 2024-05-19T20:27:13.9525041Z path: debug-*.zip 2024-05-19T20:27:13.9525363Z name: artifact 2024-05-19T20:27:13.9525668Z region: us-east-1 2024-05-19T20:27:13.9525973Z env: 2024-05-19T20:27:13.9526255Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:27:13.9526713Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:27:13.9527579Z DOCKER_CONTAINER_ID: f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T20:27:13.9528220Z ##[endgroup] 2024-05-19T20:27:14.2610840Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-05-19T20:27:14.2760368Z ##[group]Run # shellcheck disable=SC2156 2024-05-19T20:27:14.2760888Z # shellcheck disable=SC2156 2024-05-19T20:27:14.2761696Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-05-19T20:27:14.2769806Z shell: /usr/bin/bash -e {0} 2024-05-19T20:27:14.2770168Z env: 2024-05-19T20:27:14.2770454Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:27:14.2770921Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:27:14.2771661Z DOCKER_CONTAINER_ID: f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T20:27:14.2772303Z ##[endgroup] 2024-05-19T20:27:14.4575437Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-05-19T20:27:14.4575893Z with: 2024-05-19T20:27:14.4576388Z name: coredumps-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu 2024-05-19T20:27:14.4577006Z retention-days: 14 2024-05-19T20:27:14.4577339Z if-no-files-found: ignore 2024-05-19T20:27:14.4577697Z path: ./**/core.[1-9]* 2024-05-19T20:27:14.4578041Z s3-bucket: gha-artifacts 2024-05-19T20:27:14.4578386Z region: us-east-1 2024-05-19T20:27:14.4578672Z env: 2024-05-19T20:27:14.4578940Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:27:14.4579384Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:27:14.4580114Z DOCKER_CONTAINER_ID: f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T20:27:14.4580737Z ##[endgroup] 2024-05-19T20:27:24.2478968Z No files were found with the provided path: ./**/core.[1-9]*. No artifacts will be uploaded. 2024-05-19T20:27:24.2708084Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-05-19T20:27:24.2708713Z with: 2024-05-19T20:27:24.2708978Z env: 2024-05-19T20:27:24.2709258Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:27:24.2709713Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:27:24.2710717Z DOCKER_CONTAINER_ID: f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T20:27:24.2711351Z ##[endgroup] 2024-05-19T20:27:24.2731213Z ##[group]Run set -eou pipefail 2024-05-19T20:27:24.2731628Z set -eou pipefail 2024-05-19T20:27:24.2731986Z  2024-05-19T20:27:24.2732498Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-05-19T20:27:24.2733132Z for _ in $(seq 1440); do 2024-05-19T20:27:24.2733609Z  # Break if no ssh session exists anymore 2024-05-19T20:27:24.2734106Z  if [ "$(who)" = "" ]; then 2024-05-19T20:27:24.2734524Z  break 2024-05-19T20:27:24.2734864Z  fi 2024-05-19T20:27:24.2735158Z  echo "." 2024-05-19T20:27:24.2735491Z  sleep 5 2024-05-19T20:27:24.2735808Z done 2024-05-19T20:27:24.2743365Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T20:27:24.2743861Z env: 2024-05-19T20:27:24.2744139Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:27:24.2744596Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:27:24.2745332Z DOCKER_CONTAINER_ID: f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T20:27:24.2745960Z ##[endgroup] 2024-05-19T20:27:24.2770947Z Holding runner for 2 hours until all ssh sessions have logged out 2024-05-19T20:27:24.2806261Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-05-19T20:27:24.2807169Z # ignore expansion of "docker ps -q" since it could be empty 2024-05-19T20:27:24.2807781Z # shellcheck disable=SC2046 2024-05-19T20:27:24.2808268Z docker stop $(docker ps -q) || true 2024-05-19T20:27:24.2808755Z # Prune all of the docker images 2024-05-19T20:27:24.2809242Z docker system prune -af 2024-05-19T20:27:24.2816638Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T20:27:24.2817150Z env: 2024-05-19T20:27:24.2817433Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:27:24.2817901Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:27:24.2818650Z DOCKER_CONTAINER_ID: f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T20:27:24.2819303Z ##[endgroup] 2024-05-19T20:27:24.8489240Z f8a81ce94298 2024-05-19T20:27:27.2117063Z Deleted Containers: 2024-05-19T20:27:27.2117722Z f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T20:27:27.2118171Z 2024-05-19T20:27:31.2031733Z Deleted Images: 2024-05-19T20:27:31.2034803Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T20:27:31.2037410Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks@sha256:66f062925f91f9b9204b1dc6d68caf8c89fa82fd4b12cb5f24c19aa4b1c87506 2024-05-19T20:27:31.2039159Z deleted: sha256:a0542ac5317cc182e1aac2cb8d63a2d6287450937722bab68464513bbbc603ef 2024-05-19T20:27:31.2040007Z deleted: sha256:1a656f83b702d1cdc805b9509e7194f6ca9a24882785be5289f8bc566c1c1a0b 2024-05-19T20:27:31.2040862Z deleted: sha256:5b7ca0efbb8f2f6ed9ac2e375c4a52f07e2bd16e65d1914a58dcfb56ab8b1623 2024-05-19T20:27:31.2041750Z deleted: sha256:6d2255f7483ad8cfa448dc385ff2880171aacd4db919fbf38e2b3c4580f80e1e 2024-05-19T20:27:31.2042591Z deleted: sha256:a39c366330a4b21f0585d02ed65f5beb66bd03e36471ebb255c5f7766f29c118 2024-05-19T20:27:31.2043576Z deleted: sha256:5b2e96a220babb6e3a9b843ce53d3f2015dd0e505fed00a447cbe551b22d329e 2024-05-19T20:27:31.2044402Z deleted: sha256:d7247668eca42c13d2c6440886a9b43b633e5cb9304e3e3f85ac8803de7dd873 2024-05-19T20:27:31.2045413Z deleted: sha256:989398e2b08b3b7c9bf03cdbdaa903261d145b7d45bc7c1fa3a1a4a54a7ced68 2024-05-19T20:27:31.2046259Z deleted: sha256:5c2558f875db144549ed0efa5deb36d738647f1c172a9628a34677fddb6a96ee 2024-05-19T20:27:31.2047274Z deleted: sha256:b56f2c08b00650c89ac0f4d07686ba8ea22d64f666d27e8be716aeae4e042f87 2024-05-19T20:27:31.2048101Z deleted: sha256:35689f96fd20d72b6c5fd24a65e3c52c46359609c897b0552d8b36e4d62f31d9 2024-05-19T20:27:31.2048934Z deleted: sha256:8a251e66682a1ad41addcf4890ffb62aa8b85f5d9f17d433e828f170339a767f 2024-05-19T20:27:31.2049937Z deleted: sha256:9482b91acd78f655bcd9a0a6ade0f1958caa1a8274b13f7366f830fa7b4ca16e 2024-05-19T20:27:31.2050994Z deleted: sha256:1f1bf62bcfe25406d5e8f3a48198af80782c6a996bcafaae53575bcc843bd50a 2024-05-19T20:27:31.2051978Z deleted: sha256:6b680df6a8beceb613eb411958551bf0368a6d4a932eead77896d41ff535946a 2024-05-19T20:27:31.2053115Z deleted: sha256:7aa1322c73a827485b091076ffa2dc207e0a900941ca2eb2704442ca7c23ed7d 2024-05-19T20:27:31.2053995Z deleted: sha256:f0d92b27a23be6ff6a30b0974841e39e2ca9cb761797d5cb3ee5676708fbfddc 2024-05-19T20:27:31.2054845Z deleted: sha256:15e17f6dcd9e56d48cc4d19f06cf6d42e155d9dcb5d6d3289f432b1e1adb78c2 2024-05-19T20:27:31.2055682Z deleted: sha256:a9363d4f861657188bb086df9f77430971cf00513d7f3c151d954bba04092dd4 2024-05-19T20:27:31.2056527Z deleted: sha256:afad6ce30f8cd2b2fc987fb31626f14465a034dd85cea22a30e012f7b50e831c 2024-05-19T20:27:31.2057386Z deleted: sha256:a17532a675e4ec56db1fc54e2f285986b4333c89eaed6520edb8c85209ef76ea 2024-05-19T20:27:31.2058228Z deleted: sha256:55ff1a7a2aa478a7a6c00675e825d39cea2c5955f3f318832fa06cf293ddb137 2024-05-19T20:27:31.2059101Z deleted: sha256:21b7839e8103541d09ad0af189e5ecbf9df69cc798831eada41d8ce869555e94 2024-05-19T20:27:31.2059943Z deleted: sha256:3f93640e4b85ecbb5cb4239715bd521fe05dac28e2e25e53b71b808bbc193590 2024-05-19T20:27:31.2060783Z deleted: sha256:887d227bae679856dd096dad895dbcf90b807f5c82c15ff17ec5a58bf6dac184 2024-05-19T20:27:31.2061635Z deleted: sha256:d2beb8f0ce120b86322d41f423b7f42e64fdb9e4ff5d5d912252849b8146b293 2024-05-19T20:27:31.2062456Z deleted: sha256:e41d38911c997f96258de8880c0251dee4bf259158eb081eafc17f32f8948763 2024-05-19T20:27:31.2063298Z deleted: sha256:b85b0e100a5a15d221dbbccb1e864e367d917d7a79fa199c15db2c1d9a52c92a 2024-05-19T20:27:31.2064142Z deleted: sha256:75258ce71d03c6b0525b6f842fa5efd1738dbe82b883613717fcb707a60621a3 2024-05-19T20:27:31.2064972Z deleted: sha256:a12c28acec35eb76da2791220f40353777e633b6cd7d4fd187c3954e03406c46 2024-05-19T20:27:31.2065894Z deleted: sha256:eaf53ac56a05ff328d594fda97812e9db215cf74b87965669e54d4c6c1669cb3 2024-05-19T20:27:31.2066735Z deleted: sha256:795e72492c71c7cce01ac9e0408bc22fa8f7da1e41614874b812afc49c93fabf 2024-05-19T20:27:31.2067857Z deleted: sha256:1d1e16daa70a0b1c04b023ebbbe56366e3b0cbb9147851a544b43581d66ab50c 2024-05-19T20:27:31.2068883Z deleted: sha256:7752b0e90516dde8d6b7499bd8261515a2eb4063ca5d381313ea697291402d42 2024-05-19T20:27:31.2069717Z deleted: sha256:13082c1d2da03c75509cd19b1e7d211d98b359b41d53223e861fea8ec6be2366 2024-05-19T20:27:31.2070854Z deleted: sha256:fef51b8c2cf2579b04f0d82fba989e43cdd93a484741e14588a942c728328c80 2024-05-19T20:27:31.2071842Z deleted: sha256:cfd02989b5663a4fc04e8468da08dbb6202dd256e851a6e30e7370818fa65830 2024-05-19T20:27:31.2072695Z deleted: sha256:7b3853f547cd41caf732c9bf113d7f11db7fb026f979a50cf7a56ab673d8a2d9 2024-05-19T20:27:31.2073536Z deleted: sha256:8eb59cbbd7ea753bff3586c088340d20ca32887792f3f68c5b2da7d57d11447c 2024-05-19T20:27:31.2074388Z deleted: sha256:d4eb9793eb3f33e1fc18c62f8a4765625d5e2accaffffeb52543799bfe45315a 2024-05-19T20:27:31.2075224Z deleted: sha256:9b388bd2595f78854fce5ceb1a6c673a7bf3f50f84f6350351171599fcad68c4 2024-05-19T20:27:31.2076045Z deleted: sha256:1443f9602fff883313532012c5506b56993ec95031e611cace0b38b3e215413b 2024-05-19T20:27:31.2077037Z deleted: sha256:7d5157e7800e8039c647490628f0b8f64c7ed0017070e69ed1200ae786c318d2 2024-05-19T20:27:31.2077888Z deleted: sha256:3b8e4334d22e46371d21e1d80991345f300244c4932c1687b2723408f8fe213c 2024-05-19T20:27:31.2078817Z deleted: sha256:bc9b902883a611c9653e47dfcdce6f88287f0ce083995cd17612e1ee52d02f45 2024-05-19T20:27:31.2079678Z deleted: sha256:87241399dcaeb6b49174c5033c7aff0f1e914591bd2a727a80fcd0e453b1bdbc 2024-05-19T20:27:31.2080526Z deleted: sha256:9b60d0bfab86b98267697851c8871dae33fc146608b8af9764f5e3d3e459c75b 2024-05-19T20:27:31.2081362Z deleted: sha256:ad76dc292ef5f88364d52bdd2e83680e697b28b2f6df337cd9c829ab007217e1 2024-05-19T20:27:31.2082204Z deleted: sha256:7a4baeb5479a531e458c9fa438d16143efc760acf3c997b1fed3129f2d8d2fc7 2024-05-19T20:27:31.2083043Z deleted: sha256:e682379b3ae2d005fd368b974d3314ece2598d1a6928ae7341bb89f3c79e8e03 2024-05-19T20:27:31.2083899Z deleted: sha256:d8d802f39c7d9d7ddd8f5acef34ee7e07fac7f84fa106255d5228d659daf1073 2024-05-19T20:27:31.2084756Z deleted: sha256:9f893b39e0b2a09fdee21eec466f95f637ccd2e77ab254c00eeca60d4f1a4671 2024-05-19T20:27:31.2085654Z deleted: sha256:01d8e759bdd0bbea54aea263352ffd962e77eeab5d109a7fa8e279f2efdb6719 2024-05-19T20:27:31.2086511Z deleted: sha256:cd0863a43a1691ccbba624e9d63b85a341adb8bc16335dec1291e5a04147b040 2024-05-19T20:27:31.2087465Z deleted: sha256:5fec222edb2dc032d9a5203274fbd944dfb9494597fbb76b063b89ac7e84273f 2024-05-19T20:27:31.2088445Z deleted: sha256:742f8005153dd61c5f7718e2b2a8b3f934780866ef56f7ca11a48088dc638b97 2024-05-19T20:27:31.2089271Z deleted: sha256:e670dd8df73c884bf8257489df70a65828a55222a493f717eca2b9df1f3e41fa 2024-05-19T20:27:31.2090108Z deleted: sha256:e09fa8af11b0b73b35ad8f1f6ec0f0f0bf8a05ed64aa9d530377ddd2e1f94b56 2024-05-19T20:27:31.2090953Z deleted: sha256:02c8d457f61de51676900b87bd530f776b761f2b8dad7f345152b256d7bb7864 2024-05-19T20:27:31.2091788Z deleted: sha256:433544e4353c9e0269dc71fa16c00ddcecdcd5676ed0df618c7b51ac78e23053 2024-05-19T20:27:31.2092640Z deleted: sha256:fc24af9fd5121adfbc7b75df0abb1ef8c2d3868f33b7b6b1e8bf4e214fac4bd6 2024-05-19T20:27:31.2093485Z deleted: sha256:ce1492ad94b71204f8b3fb8903d0ad48a96652635e7fc9347e430639ca15a737 2024-05-19T20:27:31.2094311Z deleted: sha256:96f2ae9435e6e12f68a88dc404c8079b2796639589970c34894a86ccff3f3732 2024-05-19T20:27:31.2095158Z deleted: sha256:5310d2bc0952e7c6d8cbade7ecafd88ce3a04d9da303fa6419583e13a02b58a9 2024-05-19T20:27:31.2096001Z deleted: sha256:099076d608a55a027d60f5b572a3991a8d690a01500acf2c696933a7e6d38650 2024-05-19T20:27:31.2096818Z deleted: sha256:862b05d0f8f7e3dff2d04bd49784a9901f4e22c5dd6544ae72eb3be05e23fffe 2024-05-19T20:27:31.2097663Z deleted: sha256:f7a888e5f7904827f6d71cebcba5169dbc7a78b20f228b9b9b625f38e0c52f24 2024-05-19T20:27:31.2098512Z deleted: sha256:ab9f75a4a7a636aae480f630eaddc05133dda64f1c42445b617c97d3d209991c 2024-05-19T20:27:31.2099369Z deleted: sha256:56dda0c9cee64db5c10d5dd5f08c6f6707263f5ee98e8a075d635e5be996e855 2024-05-19T20:27:31.2100215Z deleted: sha256:5faf9c0a9efe4675ecd21a4ec417d51077d5e75da9e673161a94e7d6cd43f92c 2024-05-19T20:27:31.2100731Z 2024-05-19T20:27:31.2100883Z Total reclaimed space: 42.28GB 2024-05-19T20:27:31.2135105Z ##[group]Run set +e 2024-05-19T20:27:31.2135529Z set +e 2024-05-19T20:27:31.2135830Z set -x 2024-05-19T20:27:31.2136122Z  2024-05-19T20:27:31.2136404Z nvidia-smi 2024-05-19T20:27:31.2137131Z # NB: Surprisingly, nvidia-smi command returns successfully with return code 0 even in 2024-05-19T20:27:31.2138133Z # the case where the driver has already crashed as it still can get the driver version 2024-05-19T20:27:31.2139118Z # and some basic information like the bus ID. However, the rest of the information 2024-05-19T20:27:31.2139865Z # would be missing (ERR!), for example: 2024-05-19T20:27:31.2140326Z # 2024-05-19T20:27:31.2140749Z # +-----------------------------------------------------------------------------+ 2024-05-19T20:27:31.2141531Z # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | 2024-05-19T20:27:31.2142327Z # |-------------------------------+----------------------+----------------------+ 2024-05-19T20:27:31.2143090Z # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-05-19T20:27:31.2143959Z # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | 2024-05-19T20:27:31.2144742Z # | | | MIG M. | 2024-05-19T20:27:31.2145395Z # |===============================+======================+======================| 2024-05-19T20:27:31.2146047Z # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | 2024-05-19T20:27:31.2146804Z # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | 2024-05-19T20:27:31.2147512Z # | | | ERR! | 2024-05-19T20:27:31.2148155Z # +-------------------------------+----------------------+----------------------+ 2024-05-19T20:27:31.2148676Z # 2024-05-19T20:27:31.2149094Z # +-----------------------------------------------------------------------------+ 2024-05-19T20:27:31.2149748Z # | Processes: | 2024-05-19T20:27:31.2150711Z # | GPU GI CI PID Type Process name GPU Memory | 2024-05-19T20:27:31.2151439Z # | ID ID Usage | 2024-05-19T20:27:31.2152065Z # |=============================================================================| 2024-05-19T20:27:31.2152683Z # +-----------------------------------------------------------------------------+ 2024-05-19T20:27:31.2153202Z # 2024-05-19T20:27:31.2153764Z # This should be reported as a failure instead as it will guarantee to fail when 2024-05-19T20:27:31.2154493Z # Docker tries to run with --gpus all 2024-05-19T20:27:31.2154930Z # 2024-05-19T20:27:31.2155471Z # So, the correct check here is to query one of the missing piece of info like 2024-05-19T20:27:31.2156264Z # GPU name, so that the command can fail accordingly 2024-05-19T20:27:31.2156974Z nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-05-19T20:27:31.2157559Z NVIDIA_SMI_STATUS=$? 2024-05-19T20:27:31.2157921Z  2024-05-19T20:27:31.2158542Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2024-05-19T20:27:31.2159479Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2024-05-19T20:27:31.2160324Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2024-05-19T20:27:31.2161038Z  .github/scripts/stop_runner_service.sh 2024-05-19T20:27:31.2161509Z fi 2024-05-19T20:27:31.2161789Z  2024-05-19T20:27:31.2162481Z # For runner with multiple GPUs, we also want to confirm that the number of GPUs are the 2024-05-19T20:27:31.2163469Z # power of 2, i.e. 1, 2, 4, or 8. This is to avoid flaky test issue when one GPU fails 2024-05-19T20:27:31.2164352Z # https://github.com/pytorch/test-infra/issues/4000 2024-05-19T20:27:31.2164956Z GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) 2024-05-19T20:27:31.2165443Z NVIDIA_SMI_STATUS=$? 2024-05-19T20:27:31.2165799Z  2024-05-19T20:27:31.2166413Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2024-05-19T20:27:31.2167419Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2024-05-19T20:27:31.2168257Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2024-05-19T20:27:31.2168970Z  .github/scripts/stop_runner_service.sh 2024-05-19T20:27:31.2169419Z fi 2024-05-19T20:27:31.2169705Z  2024-05-19T20:27:31.2170058Z # Check the GPU count to be a power of 2 2024-05-19T20:27:31.2170910Z if [ "$GPU_COUNT" -le 8 ] && [ "$GPU_COUNT" -ne 1 ] && [ "$GPU_COUNT" -ne 2 ] && [ "$GPU_COUNT" -ne 4 ] && [ "$GPU_COUNT" -ne 8 ]; then 2024-05-19T20:27:31.2172023Z  echo "NVIDIA driver detects $GPU_COUNT GPUs. The runner has a broken GPU, shutting it down..." 2024-05-19T20:27:31.2172829Z  .github/scripts/stop_runner_service.sh 2024-05-19T20:27:31.2173292Z fi 2024-05-19T20:27:31.2180885Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T20:27:31.2181387Z env: 2024-05-19T20:27:31.2181674Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:27:31.2182139Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:27:31.2182892Z DOCKER_CONTAINER_ID: f8a81ce94298a1ec584fbd787799106a3a668b41bfd17e30515ec5886a4fbbf7 2024-05-19T20:27:31.2183671Z RUNNER_WORKSPACE: /home/ec2-user/actions-runner/_work/pytorch 2024-05-19T20:27:31.2184197Z ##[endgroup] 2024-05-19T20:27:31.2203802Z + nvidia-smi 2024-05-19T20:27:32.7930309Z Sun May 19 20:27:32 2024 2024-05-19T20:27:32.7931207Z +-----------------------------------------------------------------------------------------+ 2024-05-19T20:27:32.7932245Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-05-19T20:27:32.7933052Z |-----------------------------------------+------------------------+----------------------+ 2024-05-19T20:27:32.7933851Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-05-19T20:27:32.7934743Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-05-19T20:27:32.7935439Z | | | MIG M. | 2024-05-19T20:27:32.7935989Z |=========================================+========================+======================| 2024-05-19T20:27:32.8058294Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-05-19T20:27:32.8059026Z | 0% 31C P0 58W / 300W | 0MiB / 23028MiB | 0% Default | 2024-05-19T20:27:32.8059663Z | | | N/A | 2024-05-19T20:27:32.8060334Z +-----------------------------------------+------------------------+----------------------+ 2024-05-19T20:27:32.8060919Z 2024-05-19T20:27:32.8061556Z +-----------------------------------------------------------------------------------------+ 2024-05-19T20:27:32.8062185Z | Processes: | 2024-05-19T20:27:32.8062887Z | GPU GI CI PID Type Process name GPU Memory | 2024-05-19T20:27:32.8063875Z | ID ID Usage | 2024-05-19T20:27:32.8064443Z |=========================================================================================| 2024-05-19T20:27:32.8065090Z | No running processes found | 2024-05-19T20:27:32.8065969Z +-----------------------------------------------------------------------------------------+ 2024-05-19T20:27:33.4250984Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-05-19T20:27:34.9980411Z NVIDIA A10G 2024-05-19T20:27:35.4349872Z + NVIDIA_SMI_STATUS=0 2024-05-19T20:27:35.4350678Z + '[' 0 -ne 0 ']' 2024-05-19T20:27:35.4355587Z ++ nvidia-smi --list-gpus 2024-05-19T20:27:37.4512069Z ++ wc -l 2024-05-19T20:27:37.4512527Z + GPU_COUNT=1 2024-05-19T20:27:37.4512973Z + NVIDIA_SMI_STATUS=0 2024-05-19T20:27:37.4513484Z + '[' 0 -ne 0 ']' 2024-05-19T20:27:37.4513841Z + '[' 1 -le 8 ']' 2024-05-19T20:27:37.4514170Z + '[' 1 -ne 1 ']' 2024-05-19T20:27:37.4642025Z Post job cleanup. 2024-05-19T20:27:37.4692596Z Post job cleanup. 2024-05-19T20:27:37.5503898Z [command]/usr/bin/git version 2024-05-19T20:27:37.5545435Z git version 2.40.1 2024-05-19T20:27:37.5585309Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/a1c32b9b-051d-4736-b6ba-341adda517cf' before making global git config changes 2024-05-19T20:27:37.5586576Z Adding repository directory to the temporary git global config as a safe directory 2024-05-19T20:27:37.5591207Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T20:27:37.5623302Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-05-19T20:27:37.5650920Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-05-19T20:27:37.5875784Z Entering 'android/libs/fbjni' 2024-05-19T20:27:37.5912672Z Entering 'third_party/FP16' 2024-05-19T20:27:37.5948274Z Entering 'third_party/FXdiv' 2024-05-19T20:27:37.5984783Z Entering 'third_party/NNPACK' 2024-05-19T20:27:37.6020207Z Entering 'third_party/QNNPACK' 2024-05-19T20:27:37.6057167Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T20:27:37.6094645Z Entering 'third_party/XNNPACK' 2024-05-19T20:27:37.6143192Z Entering 'third_party/benchmark' 2024-05-19T20:27:37.6175727Z Entering 'third_party/cpp-httplib' 2024-05-19T20:27:37.6211506Z Entering 'third_party/cpuinfo' 2024-05-19T20:27:37.6249515Z Entering 'third_party/cudnn_frontend' 2024-05-19T20:27:37.6287674Z Entering 'third_party/cutlass' 2024-05-19T20:27:37.6331499Z Entering 'third_party/eigen' 2024-05-19T20:27:37.6370730Z Entering 'third_party/fbgemm' 2024-05-19T20:27:37.6406906Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T20:27:37.6443057Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T20:27:37.6478824Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T20:27:37.6520564Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T20:27:37.6555552Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T20:27:37.6592190Z Entering 'third_party/flatbuffers' 2024-05-19T20:27:37.6630517Z Entering 'third_party/fmt' 2024-05-19T20:27:37.6665564Z Entering 'third_party/foxi' 2024-05-19T20:27:37.6699955Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T20:27:37.6735569Z Entering 'third_party/gloo' 2024-05-19T20:27:37.6772433Z Entering 'third_party/googletest' 2024-05-19T20:27:37.6807533Z Entering 'third_party/ideep' 2024-05-19T20:27:37.6843694Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T20:27:37.6882424Z Entering 'third_party/ios-cmake' 2024-05-19T20:27:37.6916237Z Entering 'third_party/ittapi' 2024-05-19T20:27:37.6954097Z Entering 'third_party/kineto' 2024-05-19T20:27:37.6988634Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T20:27:37.7023109Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T20:27:37.7059008Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T20:27:37.7092464Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T20:27:37.7127001Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T20:27:37.7158625Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T20:27:37.7194048Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T20:27:37.7227501Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T20:27:37.7262685Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T20:27:37.7300882Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T20:27:37.7338275Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T20:27:37.7373930Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T20:27:37.7408404Z Entering 'third_party/mimalloc' 2024-05-19T20:27:37.7441543Z Entering 'third_party/nccl/nccl' 2024-05-19T20:27:37.7478425Z Entering 'third_party/neon2sse' 2024-05-19T20:27:37.7511096Z Entering 'third_party/nlohmann' 2024-05-19T20:27:37.7546223Z Entering 'third_party/onnx' 2024-05-19T20:27:37.7590568Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T20:27:37.7624417Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T20:27:37.7661731Z Entering 'third_party/onnx-tensorrt' 2024-05-19T20:27:37.7696711Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T20:27:37.7732049Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T20:27:37.7767366Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T20:27:37.7800772Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T20:27:37.7839429Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T20:27:37.7876245Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T20:27:37.7909157Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T20:27:37.7942500Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T20:27:37.7977199Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T20:27:37.8011126Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T20:27:37.8044894Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T20:27:37.8078848Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T20:27:37.8112145Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T20:27:37.8147902Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T20:27:37.8184095Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T20:27:37.8232753Z Entering 'third_party/pocketfft' 2024-05-19T20:27:37.8271615Z Entering 'third_party/protobuf' 2024-05-19T20:27:37.8309349Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T20:27:37.8343894Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T20:27:37.8380625Z Entering 'third_party/psimd' 2024-05-19T20:27:37.8416609Z Entering 'third_party/pthreadpool' 2024-05-19T20:27:37.8452170Z Entering 'third_party/pybind11' 2024-05-19T20:27:37.8485814Z Entering 'third_party/python-peachpy' 2024-05-19T20:27:37.8520407Z Entering 'third_party/sleef' 2024-05-19T20:27:37.8553484Z Entering 'third_party/tbb' 2024-05-19T20:27:37.8585882Z Entering 'third_party/tensorpipe' 2024-05-19T20:27:37.8619815Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T20:27:37.8654411Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T20:27:37.8686871Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T20:27:37.8717848Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T20:27:37.8752569Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T20:27:37.8800518Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-05-19T20:27:37.8822229Z http.https://github.com/.extraheader 2024-05-19T20:27:37.8831358Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-05-19T20:27:37.8859134Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-05-19T20:27:37.9067772Z Entering 'android/libs/fbjni' 2024-05-19T20:27:37.9088932Z http.https://github.com/.extraheader 2024-05-19T20:27:37.9113467Z Entering 'third_party/FP16' 2024-05-19T20:27:37.9135226Z http.https://github.com/.extraheader 2024-05-19T20:27:37.9158295Z Entering 'third_party/FXdiv' 2024-05-19T20:27:37.9180104Z http.https://github.com/.extraheader 2024-05-19T20:27:37.9203896Z Entering 'third_party/NNPACK' 2024-05-19T20:27:37.9224848Z http.https://github.com/.extraheader 2024-05-19T20:27:37.9248043Z Entering 'third_party/QNNPACK' 2024-05-19T20:27:37.9268082Z http.https://github.com/.extraheader 2024-05-19T20:27:37.9290212Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T20:27:37.9312114Z http.https://github.com/.extraheader 2024-05-19T20:27:37.9336417Z Entering 'third_party/XNNPACK' 2024-05-19T20:27:37.9356887Z http.https://github.com/.extraheader 2024-05-19T20:27:37.9388871Z Entering 'third_party/benchmark' 2024-05-19T20:27:37.9410698Z http.https://github.com/.extraheader 2024-05-19T20:27:37.9433798Z Entering 'third_party/cpp-httplib' 2024-05-19T20:27:37.9455274Z http.https://github.com/.extraheader 2024-05-19T20:27:37.9477065Z Entering 'third_party/cpuinfo' 2024-05-19T20:27:37.9499425Z http.https://github.com/.extraheader 2024-05-19T20:27:37.9522955Z Entering 'third_party/cudnn_frontend' 2024-05-19T20:27:37.9544567Z http.https://github.com/.extraheader 2024-05-19T20:27:37.9568464Z Entering 'third_party/cutlass' 2024-05-19T20:27:37.9589790Z http.https://github.com/.extraheader 2024-05-19T20:27:37.9619076Z Entering 'third_party/eigen' 2024-05-19T20:27:37.9640728Z http.https://github.com/.extraheader 2024-05-19T20:27:37.9664829Z Entering 'third_party/fbgemm' 2024-05-19T20:27:37.9687325Z http.https://github.com/.extraheader 2024-05-19T20:27:37.9710203Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T20:27:37.9731190Z http.https://github.com/.extraheader 2024-05-19T20:27:37.9754268Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T20:27:37.9774985Z http.https://github.com/.extraheader 2024-05-19T20:27:37.9795845Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T20:27:37.9817201Z http.https://github.com/.extraheader 2024-05-19T20:27:37.9844471Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T20:27:37.9865143Z http.https://github.com/.extraheader 2024-05-19T20:27:37.9887995Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T20:27:37.9908843Z http.https://github.com/.extraheader 2024-05-19T20:27:37.9932217Z Entering 'third_party/flatbuffers' 2024-05-19T20:27:37.9954889Z http.https://github.com/.extraheader 2024-05-19T20:27:37.9981161Z Entering 'third_party/fmt' 2024-05-19T20:27:38.0002595Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0026189Z Entering 'third_party/foxi' 2024-05-19T20:27:38.0047536Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0072220Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T20:27:38.0093594Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0117803Z Entering 'third_party/gloo' 2024-05-19T20:27:38.0139803Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0164239Z Entering 'third_party/googletest' 2024-05-19T20:27:38.0184829Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0207509Z Entering 'third_party/ideep' 2024-05-19T20:27:38.0229039Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0253229Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T20:27:38.0274144Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0301858Z Entering 'third_party/ios-cmake' 2024-05-19T20:27:38.0324032Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0347642Z Entering 'third_party/ittapi' 2024-05-19T20:27:38.0369736Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0395011Z Entering 'third_party/kineto' 2024-05-19T20:27:38.0416823Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0440370Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T20:27:38.0462296Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0486466Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T20:27:38.0507636Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0532708Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T20:27:38.0554581Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0577458Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T20:27:38.0599635Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0622965Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T20:27:38.0643919Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0666491Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T20:27:38.0687602Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0713026Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T20:27:38.0733857Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0757969Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T20:27:38.0778684Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0800274Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T20:27:38.0821245Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0845589Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T20:27:38.0866740Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0893061Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T20:27:38.0914966Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0937984Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T20:27:38.0959670Z http.https://github.com/.extraheader 2024-05-19T20:27:38.0983215Z Entering 'third_party/mimalloc' 2024-05-19T20:27:38.1005475Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1029532Z Entering 'third_party/nccl/nccl' 2024-05-19T20:27:38.1050987Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1074034Z Entering 'third_party/neon2sse' 2024-05-19T20:27:38.1095412Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1117419Z Entering 'third_party/nlohmann' 2024-05-19T20:27:38.1138155Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1163041Z Entering 'third_party/onnx' 2024-05-19T20:27:38.1184939Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1220501Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T20:27:38.1241962Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1265821Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T20:27:38.1287156Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1310327Z Entering 'third_party/onnx-tensorrt' 2024-05-19T20:27:38.1332787Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1354356Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T20:27:38.1376222Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1402892Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T20:27:38.1423704Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1446501Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T20:27:38.1467477Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1489567Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T20:27:38.1509875Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1538369Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T20:27:38.1559665Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1585325Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T20:27:38.1607535Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1631536Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T20:27:38.1652566Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1675597Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T20:27:38.1695692Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1717229Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T20:27:38.1738131Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1762153Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T20:27:38.1783199Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1806452Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T20:27:38.1827880Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1850883Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T20:27:38.1870345Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1893595Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T20:27:38.1915317Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1951979Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T20:27:38.1961008Z http.https://github.com/.extraheader 2024-05-19T20:27:38.1986262Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T20:27:38.2008454Z http.https://github.com/.extraheader 2024-05-19T20:27:38.2047114Z Entering 'third_party/pocketfft' 2024-05-19T20:27:38.2070922Z http.https://github.com/.extraheader 2024-05-19T20:27:38.2095460Z Entering 'third_party/protobuf' 2024-05-19T20:27:38.2116877Z http.https://github.com/.extraheader 2024-05-19T20:27:38.2140227Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T20:27:38.2161087Z http.https://github.com/.extraheader 2024-05-19T20:27:38.2184024Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T20:27:38.2206150Z http.https://github.com/.extraheader 2024-05-19T20:27:38.2231531Z Entering 'third_party/psimd' 2024-05-19T20:27:38.2253065Z http.https://github.com/.extraheader 2024-05-19T20:27:38.2275906Z Entering 'third_party/pthreadpool' 2024-05-19T20:27:38.2297259Z http.https://github.com/.extraheader 2024-05-19T20:27:38.2321035Z Entering 'third_party/pybind11' 2024-05-19T20:27:38.2342489Z http.https://github.com/.extraheader 2024-05-19T20:27:38.2365855Z Entering 'third_party/python-peachpy' 2024-05-19T20:27:38.2387081Z http.https://github.com/.extraheader 2024-05-19T20:27:38.2410107Z Entering 'third_party/sleef' 2024-05-19T20:27:38.2431512Z http.https://github.com/.extraheader 2024-05-19T20:27:38.2455641Z Entering 'third_party/tbb' 2024-05-19T20:27:38.2477443Z http.https://github.com/.extraheader 2024-05-19T20:27:38.2501686Z Entering 'third_party/tensorpipe' 2024-05-19T20:27:38.2523778Z http.https://github.com/.extraheader 2024-05-19T20:27:38.2547257Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T20:27:38.2568220Z http.https://github.com/.extraheader 2024-05-19T20:27:38.2593091Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T20:27:38.2614150Z http.https://github.com/.extraheader 2024-05-19T20:27:38.2636697Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T20:27:38.2656528Z http.https://github.com/.extraheader 2024-05-19T20:27:38.2677278Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T20:27:38.2699176Z http.https://github.com/.extraheader 2024-05-19T20:27:38.2722330Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T20:27:38.2743902Z http.https://github.com/.extraheader 2024-05-19T20:27:38.3012490Z Cleaning up orphan processes