2024-06-26T07:03:56.8243276Z Current runner version: '2.317.0' 2024-06-26T07:03:56.8249885Z Runner name: 'i-06740481be1ad1e06' 2024-06-26T07:03:56.8250760Z Runner group name: 'Default' 2024-06-26T07:03:56.8251709Z Machine name: 'ip-10-0-46-137' 2024-06-26T07:03:56.8256826Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T07:03:56.8258890Z Actions: read 2024-06-26T07:03:56.8259441Z Attestations: read 2024-06-26T07:03:56.8260053Z Checks: read 2024-06-26T07:03:56.8260591Z Contents: read 2024-06-26T07:03:56.8261136Z Deployments: read 2024-06-26T07:03:56.8261681Z Discussions: read 2024-06-26T07:03:56.8262253Z Issues: read 2024-06-26T07:03:56.8262782Z Metadata: read 2024-06-26T07:03:56.8263358Z Packages: read 2024-06-26T07:03:56.8263890Z Pages: read 2024-06-26T07:03:56.8264505Z PullRequests: read 2024-06-26T07:03:56.8265115Z RepositoryProjects: read 2024-06-26T07:03:56.8265755Z SecurityEvents: read 2024-06-26T07:03:56.8266331Z Statuses: read 2024-06-26T07:03:56.8266908Z ##[endgroup] 2024-06-26T07:03:56.8270411Z Secret source: Actions 2024-06-26T07:03:56.8271228Z Prepare workflow directory 2024-06-26T07:03:57.1760960Z Prepare all required actions 2024-06-26T07:03:57.1943451Z Getting action download info 2024-06-26T07:03:57.3690136Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T07:03:57.7082779Z Download action repository 'pytorch/pytorch@main' (SHA:c422a9549d2b320b47a9391f6cbcf592b2876c88) 2024-06-26T07:04:00.5227668Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-26T07:04:00.6560165Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T07:04:00.9156493Z Getting action download info 2024-06-26T07:04:01.0122474Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T07:04:01.1896096Z Getting action download info 2024-06-26T07:04:01.2896972Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T07:04:01.4216677Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/129470 (b8c4c54d347aa776934c60784e35936878ef18dc) 2024-06-26T07:04:01.4218894Z ##[group] Inputs 2024-06-26T07:04:01.4219388Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:04:01.4226843Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:04:01.4235095Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:01.4236343Z sync-tag: 2024-06-26T07:04:01.4237188Z timeout-minutes: 240 2024-06-26T07:04:01.4237540Z use-gha: 2024-06-26T07:04:01.4237843Z dashboard-tag: 2024-06-26T07:04:01.4238178Z s3-bucket: gha-artifacts 2024-06-26T07:04:01.4238544Z aws-role-to-assume: 2024-06-26T07:04:01.4238899Z ##[endgroup] 2024-06-26T07:04:01.4240019Z Complete job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:04:01.4808276Z A job started hook has been configured by the self-hosted runner administrator 2024-06-26T07:04:01.4945893Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T07:04:01.4956358Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:01.4956907Z ##[endgroup] 2024-06-26T07:04:02.2581049Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-26T07:04:02.2581634Z with: 2024-06-26T07:04:02.2582360Z github-secret: *** 2024-06-26T07:04:02.2583368Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-26T07:04:02.2584518Z activate-with-label: false 2024-06-26T07:04:02.2584916Z label: with-ssh 2024-06-26T07:04:02.2585269Z remove-existing-keys: true 2024-06-26T07:04:02.2585658Z fail-silently: true 2024-06-26T07:04:02.2586007Z env: 2024-06-26T07:04:02.2586311Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:02.2586679Z ##[endgroup] 2024-06-26T07:04:02.3450353Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-26T07:04:02.3451460Z ciflow reference detected, attempting to extract PR number 2024-06-26T07:04:02.6324010Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-06-26T07:04:02.6962423Z No SSH keys found for user pytorch-bot[bot] 2024-06-26T07:04:02.6963355Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T07:04:02.7639186Z No SSH keys found for user leslie-fang-intel 2024-06-26T07:04:02.7747602Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T07:04:02.7748199Z with: 2024-06-26T07:04:02.7748503Z submodules: recursive 2024-06-26T07:04:02.7748856Z fetch-depth: 0 2024-06-26T07:04:02.7749171Z env: 2024-06-26T07:04:02.7749462Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:02.7749821Z ##[endgroup] 2024-06-26T07:04:02.7942748Z ##[group]Run retry () { 2024-06-26T07:04:02.7943227Z retry () { 2024-06-26T07:04:02.7943787Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T07:04:02.7944516Z } 2024-06-26T07:04:02.7944898Z echo "${GITHUB_WORKSPACE}" 2024-06-26T07:04:02.7945403Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T07:04:02.7945975Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:04:02.7946518Z else 2024-06-26T07:04:02.7946940Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:04:02.7947411Z fi 2024-06-26T07:04:02.7947919Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T07:04:02.7956017Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:02.7956578Z env: 2024-06-26T07:04:02.7956932Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:02.7957396Z NO_SUDO: 2024-06-26T07:04:02.7957742Z ##[endgroup] 2024-06-26T07:04:02.7980897Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:04:05.0030254Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T07:04:05.0030720Z with: 2024-06-26T07:04:05.0031063Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:04:05.0031513Z fetch-depth: 0 2024-06-26T07:04:05.0031845Z submodules: recursive 2024-06-26T07:04:05.0032209Z quiet-checkout: true 2024-06-26T07:04:05.0032712Z repository: pytorch/pytorch 2024-06-26T07:04:05.0033204Z token: *** 2024-06-26T07:04:05.0033512Z ssh-strict: true 2024-06-26T07:04:05.0033855Z persist-credentials: true 2024-06-26T07:04:05.0034230Z clean: true 2024-06-26T07:04:05.0034575Z sparse-checkout-cone-mode: true 2024-06-26T07:04:05.0034978Z lfs: false 2024-06-26T07:04:05.0035289Z set-safe-directory: true 2024-06-26T07:04:05.0035641Z env: 2024-06-26T07:04:05.0035915Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:05.0036285Z ##[endgroup] 2024-06-26T07:04:05.0944901Z Syncing repository: pytorch/pytorch 2024-06-26T07:04:05.0946445Z ##[group]Getting Git version info 2024-06-26T07:04:05.0947180Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:04:05.0948098Z [command]/usr/bin/git version 2024-06-26T07:04:05.0948484Z git version 2.40.1 2024-06-26T07:04:05.0949916Z ##[endgroup] 2024-06-26T07:04:05.0960890Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/78e4771b-e5bd-4326-81dc-995476c2bd6a' before making global git config changes 2024-06-26T07:04:05.0962279Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T07:04:05.0963499Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:04:05.0988414Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:04:05.0992199Z ##[group]Initializing the repository 2024-06-26T07:04:05.0994852Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:04:05.1019306Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T07:04:05.1020255Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T07:04:05.1021534Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T07:04:05.1022745Z hint: 2024-06-26T07:04:05.1023928Z hint: git config --global init.defaultBranch 2024-06-26T07:04:05.1025132Z hint: 2024-06-26T07:04:05.1026213Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T07:04:05.1028019Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T07:04:05.1029552Z hint: 2024-06-26T07:04:05.1030224Z hint: git branch -m 2024-06-26T07:04:05.1031242Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T07:04:05.1032812Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T07:04:05.1054110Z ##[endgroup] 2024-06-26T07:04:05.1054995Z ##[group]Disabling automatic garbage collection 2024-06-26T07:04:05.1057699Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T07:04:05.1082572Z ##[endgroup] 2024-06-26T07:04:05.1083124Z ##[group]Setting up auth 2024-06-26T07:04:05.1089004Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T07:04:05.1109428Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T07:04:05.1303612Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T07:04:05.1327247Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T07:04:05.1522387Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:04:05.1559245Z ##[endgroup] 2024-06-26T07:04:05.1560077Z ##[group]Fetching the repository 2024-06-26T07:04:05.1566036Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T07:04:07.5138962Z remote: Enumerating objects: 987619 2024-06-26T07:04:07.5140552Z remote: Enumerating objects: 990209, done. 2024-06-26T07:04:07.5142460Z remote: Counting objects: 0% (1/2590) 2024-06-26T07:04:07.5143251Z remote: Counting objects: 1% (26/2590) 2024-06-26T07:04:07.5145035Z remote: Counting objects: 2% (52/2590) 2024-06-26T07:04:07.5145823Z remote: Counting objects: 3% (78/2590) 2024-06-26T07:04:07.5146473Z remote: Counting objects: 4% (104/2590) 2024-06-26T07:04:07.5147267Z remote: Counting objects: 5% (130/2590) 2024-06-26T07:04:07.5147903Z remote: Counting objects: 6% (156/2590) 2024-06-26T07:04:07.5148463Z remote: Counting objects: 7% (182/2590) 2024-06-26T07:04:07.5149003Z remote: Counting objects: 8% (208/2590) 2024-06-26T07:04:07.5149531Z remote: Counting objects: 9% (234/2590) 2024-06-26T07:04:07.5150066Z remote: Counting objects: 10% (259/2590) 2024-06-26T07:04:07.5150601Z remote: Counting objects: 11% (285/2590) 2024-06-26T07:04:07.5151147Z remote: Counting objects: 12% (311/2590) 2024-06-26T07:04:07.5152330Z remote: Counting objects: 13% (337/2590) 2024-06-26T07:04:07.5152945Z remote: Counting objects: 14% (363/2590) 2024-06-26T07:04:07.5153485Z remote: Counting objects: 15% (389/2590) 2024-06-26T07:04:07.5156809Z remote: Counting objects: 16% (415/2590) 2024-06-26T07:04:07.5157459Z remote: Counting objects: 17% (441/2590) 2024-06-26T07:04:07.5158007Z remote: Counting objects: 18% (467/2590) 2024-06-26T07:04:07.5158551Z remote: Counting objects: 19% (493/2590) 2024-06-26T07:04:07.5159108Z remote: Counting objects: 20% (518/2590) 2024-06-26T07:04:07.5159814Z remote: Counting objects: 21% (544/2590) 2024-06-26T07:04:07.5160453Z remote: Counting objects: 22% (570/2590) 2024-06-26T07:04:07.5160990Z remote: Counting objects: 23% (596/2590) 2024-06-26T07:04:07.5161520Z remote: Counting objects: 24% (622/2590) 2024-06-26T07:04:07.5162114Z remote: Counting objects: 25% (648/2590) 2024-06-26T07:04:07.5162665Z remote: Counting objects: 26% (674/2590) 2024-06-26T07:04:07.5163205Z remote: Counting objects: 27% (700/2590) 2024-06-26T07:04:07.5163957Z remote: Counting objects: 28% (726/2590) 2024-06-26T07:04:07.5164503Z remote: Counting objects: 29% (752/2590) 2024-06-26T07:04:07.5165040Z remote: Counting objects: 30% (777/2590) 2024-06-26T07:04:07.5165593Z remote: Counting objects: 31% (803/2590) 2024-06-26T07:04:07.5166131Z remote: Counting objects: 32% (829/2590) 2024-06-26T07:04:07.5166671Z remote: Counting objects: 33% (855/2590) 2024-06-26T07:04:07.5167634Z remote: Counting objects: 34% (881/2590) 2024-06-26T07:04:07.5168164Z remote: Counting objects: 35% (907/2590) 2024-06-26T07:04:07.5168698Z remote: Counting objects: 36% (933/2590) 2024-06-26T07:04:07.5169306Z remote: Counting objects: 37% (959/2590) 2024-06-26T07:04:07.5169838Z remote: Counting objects: 38% (985/2590) 2024-06-26T07:04:07.5170387Z remote: Counting objects: 39% (1011/2590) 2024-06-26T07:04:07.5170961Z remote: Counting objects: 40% (1036/2590) 2024-06-26T07:04:07.5171532Z remote: Counting objects: 41% (1062/2590) 2024-06-26T07:04:07.5172085Z remote: Counting objects: 42% (1088/2590) 2024-06-26T07:04:07.5172649Z remote: Counting objects: 43% (1114/2590) 2024-06-26T07:04:07.5173204Z remote: Counting objects: 44% (1140/2590) 2024-06-26T07:04:07.5173762Z remote: Counting objects: 45% (1166/2590) 2024-06-26T07:04:07.5174388Z remote: Counting objects: 46% (1192/2590) 2024-06-26T07:04:07.5174946Z remote: Counting objects: 47% (1218/2590) 2024-06-26T07:04:07.5175492Z remote: Counting objects: 48% (1244/2590) 2024-06-26T07:04:07.5176054Z remote: Counting objects: 49% (1270/2590) 2024-06-26T07:04:07.5176707Z remote: Counting objects: 50% (1295/2590) 2024-06-26T07:04:07.5177359Z remote: Counting objects: 51% (1321/2590) 2024-06-26T07:04:07.5178003Z remote: Counting objects: 52% (1347/2590) 2024-06-26T07:04:07.5178570Z remote: Counting objects: 53% (1373/2590) 2024-06-26T07:04:07.5179197Z remote: Counting objects: 54% (1399/2590) 2024-06-26T07:04:07.5179918Z remote: Counting objects: 55% (1425/2590) 2024-06-26T07:04:07.5180541Z remote: Counting objects: 56% (1451/2590) 2024-06-26T07:04:07.5181111Z remote: Counting objects: 57% (1477/2590) 2024-06-26T07:04:07.5181655Z remote: Counting objects: 58% (1503/2590) 2024-06-26T07:04:07.5182301Z remote: Counting objects: 59% (1529/2590) 2024-06-26T07:04:07.5182969Z remote: Counting objects: 60% (1554/2590) 2024-06-26T07:04:07.5183535Z remote: Counting objects: 61% (1580/2590) 2024-06-26T07:04:07.5184086Z remote: Counting objects: 62% (1606/2590) 2024-06-26T07:04:07.5184868Z remote: Counting objects: 63% (1632/2590) 2024-06-26T07:04:07.5185482Z remote: Counting objects: 64% (1658/2590) 2024-06-26T07:04:07.5186040Z remote: Counting objects: 65% (1684/2590) 2024-06-26T07:04:07.5186598Z remote: Counting objects: 66% (1710/2590) 2024-06-26T07:04:07.5187221Z remote: Counting objects: 67% (1736/2590) 2024-06-26T07:04:07.5187904Z remote: Counting objects: 68% (1762/2590) 2024-06-26T07:04:07.5188452Z remote: Counting objects: 69% (1788/2590) 2024-06-26T07:04:07.5188997Z remote: Counting objects: 70% (1813/2590) 2024-06-26T07:04:07.5189539Z remote: Counting objects: 71% (1839/2590) 2024-06-26T07:04:07.5190108Z remote: Counting objects: 72% (1865/2590) 2024-06-26T07:04:07.5190663Z remote: Counting objects: 73% (1891/2590) 2024-06-26T07:04:07.5191212Z remote: Counting objects: 74% (1917/2590) 2024-06-26T07:04:07.5191750Z remote: Counting objects: 75% (1943/2590) 2024-06-26T07:04:07.5192310Z remote: Counting objects: 76% (1969/2590) 2024-06-26T07:04:07.5192857Z remote: Counting objects: 77% (1995/2590) 2024-06-26T07:04:07.5193404Z remote: Counting objects: 78% (2021/2590) 2024-06-26T07:04:07.5194081Z remote: Counting objects: 79% (2047/2590) 2024-06-26T07:04:07.5194630Z remote: Counting objects: 80% (2072/2590) 2024-06-26T07:04:07.5195171Z remote: Counting objects: 81% (2098/2590) 2024-06-26T07:04:07.5195708Z remote: Counting objects: 82% (2124/2590) 2024-06-26T07:04:07.5196245Z remote: Counting objects: 83% (2150/2590) 2024-06-26T07:04:07.5196784Z remote: Counting objects: 84% (2176/2590) 2024-06-26T07:04:07.5197321Z remote: Counting objects: 85% (2202/2590) 2024-06-26T07:04:07.5197865Z remote: Counting objects: 86% (2228/2590) 2024-06-26T07:04:07.5198403Z remote: Counting objects: 87% (2254/2590) 2024-06-26T07:04:07.5198952Z remote: Counting objects: 88% (2280/2590) 2024-06-26T07:04:07.5199498Z remote: Counting objects: 89% (2306/2590) 2024-06-26T07:04:07.5200042Z remote: Counting objects: 90% (2331/2590) 2024-06-26T07:04:07.5200584Z remote: Counting objects: 91% (2357/2590) 2024-06-26T07:04:07.5201133Z remote: Counting objects: 92% (2383/2590) 2024-06-26T07:04:07.5201735Z remote: Counting objects: 93% (2409/2590) 2024-06-26T07:04:07.5202277Z remote: Counting objects: 94% (2435/2590) 2024-06-26T07:04:07.5202809Z remote: Counting objects: 95% (2461/2590) 2024-06-26T07:04:07.5203351Z remote: Counting objects: 96% (2487/2590) 2024-06-26T07:04:07.5203899Z remote: Counting objects: 97% (2513/2590) 2024-06-26T07:04:07.5204435Z remote: Counting objects: 98% (2539/2590) 2024-06-26T07:04:07.5204974Z remote: Counting objects: 99% (2565/2590) 2024-06-26T07:04:07.5205516Z remote: Counting objects: 100% (2590/2590) 2024-06-26T07:04:07.5206111Z remote: Counting objects: 100% (2590/2590), done. 2024-06-26T07:04:07.5321764Z remote: Compressing objects: 0% (1/1068) 2024-06-26T07:04:07.5748115Z remote: Compressing objects: 1% (11/1068) 2024-06-26T07:04:07.6177269Z remote: Compressing objects: 2% (22/1068) 2024-06-26T07:04:07.6558446Z remote: Compressing objects: 3% (33/1068) 2024-06-26T07:04:07.6796627Z remote: Compressing objects: 4% (43/1068) 2024-06-26T07:04:07.7075719Z remote: Compressing objects: 5% (54/1068) 2024-06-26T07:04:07.7292028Z remote: Compressing objects: 6% (65/1068) 2024-06-26T07:04:07.7710380Z remote: Compressing objects: 7% (75/1068) 2024-06-26T07:04:07.8560864Z remote: Compressing objects: 8% (86/1068) 2024-06-26T07:04:07.9396949Z remote: Compressing objects: 9% (97/1068) 2024-06-26T07:04:07.9644891Z remote: Compressing objects: 10% (107/1068) 2024-06-26T07:04:07.9838074Z remote: Compressing objects: 11% (118/1068) 2024-06-26T07:04:07.9955777Z remote: Compressing objects: 12% (129/1068) 2024-06-26T07:04:07.9956370Z remote: Compressing objects: 13% (139/1068) 2024-06-26T07:04:08.0086951Z remote: Compressing objects: 14% (150/1068) 2024-06-26T07:04:08.0087568Z remote: Compressing objects: 15% (161/1068) 2024-06-26T07:04:08.0155910Z remote: Compressing objects: 16% (171/1068) 2024-06-26T07:04:08.0156496Z remote: Compressing objects: 17% (182/1068) 2024-06-26T07:04:08.0158426Z remote: Compressing objects: 18% (193/1068) 2024-06-26T07:04:08.0176834Z remote: Compressing objects: 19% (203/1068) 2024-06-26T07:04:08.0208148Z remote: Compressing objects: 20% (214/1068) 2024-06-26T07:04:08.0214309Z remote: Compressing objects: 21% (225/1068) 2024-06-26T07:04:08.0233605Z remote: Compressing objects: 22% (235/1068) 2024-06-26T07:04:08.0260995Z remote: Compressing objects: 23% (246/1068) 2024-06-26T07:04:08.0273076Z remote: Compressing objects: 24% (257/1068) 2024-06-26T07:04:08.0282216Z remote: Compressing objects: 25% (267/1068) 2024-06-26T07:04:08.0291334Z remote: Compressing objects: 26% (278/1068) 2024-06-26T07:04:08.0297186Z remote: Compressing objects: 27% (289/1068) 2024-06-26T07:04:08.0304161Z remote: Compressing objects: 28% (300/1068) 2024-06-26T07:04:08.0312220Z remote: Compressing objects: 29% (310/1068) 2024-06-26T07:04:08.0321301Z remote: Compressing objects: 30% (321/1068) 2024-06-26T07:04:08.0326365Z remote: Compressing objects: 31% (332/1068) 2024-06-26T07:04:08.0341762Z remote: Compressing objects: 32% (342/1068) 2024-06-26T07:04:08.0347548Z remote: Compressing objects: 33% (353/1068) 2024-06-26T07:04:08.0366392Z remote: Compressing objects: 34% (364/1068) 2024-06-26T07:04:08.0379023Z remote: Compressing objects: 35% (374/1068) 2024-06-26T07:04:08.0404103Z remote: Compressing objects: 36% (385/1068) 2024-06-26T07:04:08.0411672Z remote: Compressing objects: 37% (396/1068) 2024-06-26T07:04:08.0429232Z remote: Compressing objects: 38% (406/1068) 2024-06-26T07:04:08.0443156Z remote: Compressing objects: 39% (417/1068) 2024-06-26T07:04:08.0460622Z remote: Compressing objects: 40% (428/1068) 2024-06-26T07:04:08.0483111Z remote: Compressing objects: 41% (438/1068) 2024-06-26T07:04:08.0495425Z remote: Compressing objects: 42% (449/1068) 2024-06-26T07:04:08.0508632Z remote: Compressing objects: 43% (460/1068) 2024-06-26T07:04:08.0511246Z remote: Compressing objects: 44% (470/1068) 2024-06-26T07:04:08.0518611Z remote: Compressing objects: 45% (481/1068) 2024-06-26T07:04:08.0524575Z remote: Compressing objects: 46% (492/1068) 2024-06-26T07:04:08.0543060Z remote: Compressing objects: 47% (502/1068) 2024-06-26T07:04:08.0554963Z remote: Compressing objects: 48% (513/1068) 2024-06-26T07:04:08.0557928Z remote: Compressing objects: 49% (524/1068) 2024-06-26T07:04:08.0568554Z remote: Compressing objects: 50% (534/1068) 2024-06-26T07:04:08.0574806Z remote: Compressing objects: 51% (545/1068) 2024-06-26T07:04:08.0588520Z remote: Compressing objects: 52% (556/1068) 2024-06-26T07:04:08.0611215Z remote: Compressing objects: 53% (567/1068) 2024-06-26T07:04:08.0622613Z remote: Compressing objects: 54% (577/1068) 2024-06-26T07:04:08.0632450Z remote: Compressing objects: 55% (588/1068) 2024-06-26T07:04:08.0642037Z remote: Compressing objects: 56% (599/1068) 2024-06-26T07:04:08.0657588Z remote: Compressing objects: 57% (609/1068) 2024-06-26T07:04:08.0668682Z remote: Compressing objects: 58% (620/1068) 2024-06-26T07:04:08.0677389Z remote: Compressing objects: 59% (631/1068) 2024-06-26T07:04:08.0683036Z remote: Compressing objects: 60% (641/1068) 2024-06-26T07:04:08.0700576Z remote: Compressing objects: 61% (652/1068) 2024-06-26T07:04:08.0708932Z remote: Compressing objects: 62% (663/1068) 2024-06-26T07:04:08.0720035Z remote: Compressing objects: 63% (673/1068) 2024-06-26T07:04:08.0725078Z remote: Compressing objects: 64% (684/1068) 2024-06-26T07:04:08.0729742Z remote: Compressing objects: 65% (695/1068) 2024-06-26T07:04:08.0732693Z remote: Compressing objects: 66% (705/1068) 2024-06-26T07:04:08.0744260Z remote: Compressing objects: 67% (716/1068) 2024-06-26T07:04:08.0748382Z remote: Compressing objects: 68% (727/1068) 2024-06-26T07:04:08.0750949Z remote: Compressing objects: 69% (737/1068) 2024-06-26T07:04:08.0755151Z remote: Compressing objects: 70% (748/1068) 2024-06-26T07:04:08.0762419Z remote: Compressing objects: 71% (759/1068) 2024-06-26T07:04:08.0765119Z remote: Compressing objects: 72% (769/1068) 2024-06-26T07:04:08.0765911Z remote: Compressing objects: 73% (780/1068) 2024-06-26T07:04:08.0766572Z remote: Compressing objects: 74% (791/1068) 2024-06-26T07:04:08.0767327Z remote: Compressing objects: 75% (801/1068) 2024-06-26T07:04:08.0767918Z remote: Compressing objects: 76% (812/1068) 2024-06-26T07:04:08.0768639Z remote: Compressing objects: 77% (823/1068) 2024-06-26T07:04:08.0769227Z remote: Compressing objects: 78% (834/1068) 2024-06-26T07:04:08.0773579Z remote: Compressing objects: 79% (844/1068) 2024-06-26T07:04:08.0774363Z remote: Compressing objects: 80% (855/1068) 2024-06-26T07:04:08.0774971Z remote: Compressing objects: 81% (866/1068) 2024-06-26T07:04:08.0778314Z remote: Compressing objects: 82% (876/1068) 2024-06-26T07:04:08.0779060Z remote: Compressing objects: 83% (887/1068) 2024-06-26T07:04:08.0780889Z remote: Compressing objects: 84% (898/1068) 2024-06-26T07:04:08.0781654Z remote: Compressing objects: 85% (908/1068) 2024-06-26T07:04:08.0782261Z remote: Compressing objects: 86% (919/1068) 2024-06-26T07:04:08.0787112Z remote: Compressing objects: 87% (930/1068) 2024-06-26T07:04:08.0787878Z remote: Compressing objects: 88% (940/1068) 2024-06-26T07:04:08.0788526Z remote: Compressing objects: 89% (951/1068) 2024-06-26T07:04:08.0789113Z remote: Compressing objects: 90% (962/1068) 2024-06-26T07:04:08.0789821Z remote: Compressing objects: 91% (972/1068) 2024-06-26T07:04:08.0790544Z remote: Compressing objects: 92% (983/1068) 2024-06-26T07:04:08.0791124Z remote: Compressing objects: 93% (994/1068) 2024-06-26T07:04:08.0794846Z remote: Compressing objects: 94% (1004/1068) 2024-06-26T07:04:08.0797428Z remote: Compressing objects: 95% (1015/1068) 2024-06-26T07:04:08.0798218Z remote: Compressing objects: 96% (1026/1068) 2024-06-26T07:04:08.0798827Z remote: Compressing objects: 97% (1036/1068) 2024-06-26T07:04:08.0799424Z remote: Compressing objects: 98% (1047/1068) 2024-06-26T07:04:08.0800190Z remote: Compressing objects: 99% (1058/1068) 2024-06-26T07:04:08.0801006Z remote: Compressing objects: 100% (1068/1068) 2024-06-26T07:04:08.0801626Z remote: Compressing objects: 100% (1068/1068), done. 2024-06-26T07:04:28.3711522Z remote: Total 990209 (delta 2055), reused 1861 (delta 1521), pack-reused 987619 2024-06-26T07:04:53.2036465Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T07:04:53.2057656Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:04:53.2062091Z ##[endgroup] 2024-06-26T07:04:53.2062685Z ##[group]Determining the checkout info 2024-06-26T07:04:53.2065235Z ##[endgroup] 2024-06-26T07:04:53.2065963Z ##[group]Checking out the ref 2024-06-26T07:04:53.2068103Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:04:54.4188492Z ##[endgroup] 2024-06-26T07:04:54.4189709Z ##[group]Setting up auth for fetching submodules 2024-06-26T07:04:54.4192256Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:04:54.4236461Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T07:04:54.4261854Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T07:04:54.4281447Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T07:04:54.4300240Z ##[endgroup] 2024-06-26T07:04:54.4300797Z ##[group]Fetching submodules 2024-06-26T07:04:54.4303843Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T07:04:54.4511870Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T07:04:54.4714945Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T07:04:54.4716731Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T07:04:54.4718530Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T07:04:54.4720530Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T07:04:54.4722649Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:54.4724648Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T07:04:54.4726264Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T07:04:54.4728047Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T07:04:54.4729769Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T07:04:54.4731819Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T07:04:54.4733933Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T07:04:54.4736536Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T07:04:54.4738972Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T07:04:54.4742720Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T07:04:54.4745367Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T07:04:54.4748480Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T07:04:54.4751728Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:54.4754605Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T07:04:54.4757807Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T07:04:54.4760885Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T07:04:54.4764262Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T07:04:54.4767910Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T07:04:54.4771529Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T07:04:54.4774834Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T07:04:54.4778480Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T07:04:54.4782028Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T07:04:54.4786129Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T07:04:54.4789685Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T07:04:54.4793495Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T07:04:54.4797348Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T07:04:54.4801449Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T07:04:54.4805536Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T07:04:54.4810175Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T07:04:54.4814158Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T07:04:54.4818424Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T07:04:54.4839402Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T07:04:54.7183384Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T07:04:54.8869810Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T07:04:55.0603104Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T07:04:55.2828158Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T07:04:57.3330343Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T07:05:07.1315632Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T07:05:07.5258604Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T07:05:08.0813222Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T07:05:08.6506482Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T07:05:09.7597938Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T07:05:11.5236621Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T07:05:17.3664733Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T07:05:18.6562740Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T07:05:20.3150805Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T07:05:21.5030380Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T07:05:21.7472674Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T07:05:22.0997836Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T07:05:22.4021263Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T07:05:23.3140511Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T07:05:23.6480109Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T07:05:23.8799824Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T07:05:25.3866871Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T07:05:26.0976139Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T07:05:26.8588322Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T07:05:32.6016168Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T07:05:34.9718265Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T07:05:40.5308841Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T07:05:40.7193672Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T07:05:48.7955503Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T07:05:48.9609968Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T07:05:49.1345609Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T07:05:50.1321318Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T07:05:50.3998911Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T07:05:51.0065781Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T07:05:51.4434089Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T07:05:51.4515203Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T07:05:51.4575647Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T07:05:51.4754732Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T07:05:51.5060938Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T07:05:52.2735000Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T07:05:52.2902972Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T07:05:52.3252411Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T07:05:52.4089240Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T07:05:52.4362698Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T07:05:52.8679388Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T07:05:53.0801797Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T07:05:53.1437936Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T07:05:53.1448921Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:05:53.1450546Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:05:53.1452387Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:05:53.1454426Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:05:53.1456452Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:05:53.1474269Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T07:05:54.0111309Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T07:05:54.5507258Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T07:05:56.3072348Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T07:05:57.2720227Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T07:05:57.6293509Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T07:05:57.7139459Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T07:05:58.0558538Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T07:05:58.1092492Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T07:05:58.1177692Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T07:05:58.2084977Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T07:05:58.2359765Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T07:05:58.2421398Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T07:05:58.2724708Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T07:05:58.2917004Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T07:05:58.3275246Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:05:58.3380312Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T07:05:58.3391541Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T07:05:58.3406402Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T07:06:10.0949357Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T07:06:10.1087930Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T07:06:10.1869630Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T07:06:10.1880564Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:06:10.1882450Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:06:10.1884328Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:06:10.1902014Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T07:06:10.7077855Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T07:06:11.9082301Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T07:06:12.9578962Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T07:06:12.9588987Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:06:12.9590916Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:06:12.9592967Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:06:12.9594835Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:06:12.9596923Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:06:12.9599389Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:06:12.9601599Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:06:12.9603859Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:06:12.9621834Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T07:06:14.0358183Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T07:06:14.3559790Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T07:06:15.5609126Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T07:06:15.8538388Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T07:06:16.3327584Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T07:06:17.2585407Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T07:06:22.8700842Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T07:06:23.2162501Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T07:06:23.2299363Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T07:06:23.2596802Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T07:06:23.2695128Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T07:06:23.2706172Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:06:23.2721845Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T07:06:23.5584664Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T07:06:23.5724313Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T07:06:23.6060876Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T07:06:23.6848267Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T07:06:23.6971624Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T07:06:23.7269667Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T07:06:23.7758330Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T07:06:23.8066594Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T07:06:23.8261341Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T07:06:23.9117530Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T07:06:24.1928866Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T07:06:24.1957460Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T07:06:24.1959330Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T07:06:24.1975446Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T07:06:24.5556415Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T07:06:25.9536587Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T07:06:25.9811941Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T07:06:26.0348478Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T07:06:26.0360839Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:06:26.0380738Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:06:26.0382905Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:06:26.0385231Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:06:26.0387427Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:06:26.0389769Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:06:26.0391953Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:06:26.0393720Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:06:26.0395266Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T07:06:26.4937047Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T07:06:27.5771211Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T07:06:27.9206663Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T07:06:33.5350204Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T07:06:33.8044028Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T07:06:34.0052183Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T07:06:34.3023051Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T07:06:41.7568755Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T07:06:41.7901762Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T07:06:41.8029084Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T07:06:41.8865713Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T07:06:41.8958436Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T07:06:41.9064148Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T07:06:41.9176476Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T07:06:41.9187018Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:06:41.9189188Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:06:41.9205529Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T07:06:43.6677141Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T07:06:45.8320092Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T07:06:45.8691340Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:06:46.2284091Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T07:06:46.2375403Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T07:06:46.4557772Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T07:06:46.4571925Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:06:46.4573691Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T07:06:46.4591713Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T07:06:46.8860738Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T07:06:47.8307059Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T07:06:47.8927400Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T07:06:47.8989180Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T07:06:47.9076613Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T07:06:47.9360415Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T07:06:47.9586267Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T07:06:47.9915766Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T07:06:48.0127412Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T07:06:48.0138879Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:06:48.0140745Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:06:48.0142387Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:06:48.0144113Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:06:48.0162802Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T07:06:48.9783063Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T07:06:49.1937533Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T07:06:50.2252952Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T07:06:51.2771215Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T07:06:51.2885898Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T07:06:51.3461056Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T07:06:51.3686233Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T07:06:51.3695083Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:06:51.3710330Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T07:06:51.5885578Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T07:06:51.5909260Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T07:06:51.6114823Z Entering 'android/libs/fbjni' 2024-06-26T07:06:51.6143667Z Entering 'third_party/FP16' 2024-06-26T07:06:51.6171694Z Entering 'third_party/FXdiv' 2024-06-26T07:06:51.6201692Z Entering 'third_party/NNPACK' 2024-06-26T07:06:51.6228905Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:06:51.6253863Z Entering 'third_party/XNNPACK' 2024-06-26T07:06:51.6294171Z Entering 'third_party/benchmark' 2024-06-26T07:06:51.6320085Z Entering 'third_party/cpp-httplib' 2024-06-26T07:06:51.6346370Z Entering 'third_party/cpuinfo' 2024-06-26T07:06:51.6377181Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:06:51.6403223Z Entering 'third_party/cutlass' 2024-06-26T07:06:51.6438307Z Entering 'third_party/eigen' 2024-06-26T07:06:51.6466971Z Entering 'third_party/fbgemm' 2024-06-26T07:06:51.6495559Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:06:51.6523601Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:06:51.6552340Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:06:51.6583244Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:06:51.6610865Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:06:51.6640024Z Entering 'third_party/flatbuffers' 2024-06-26T07:06:51.6669658Z Entering 'third_party/fmt' 2024-06-26T07:06:51.6697999Z Entering 'third_party/foxi' 2024-06-26T07:06:51.6724527Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:06:51.6750247Z Entering 'third_party/gloo' 2024-06-26T07:06:51.6777262Z Entering 'third_party/googletest' 2024-06-26T07:06:51.6804180Z Entering 'third_party/ideep' 2024-06-26T07:06:51.6832798Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:06:51.6867675Z Entering 'third_party/ittapi' 2024-06-26T07:06:51.6895015Z Entering 'third_party/kineto' 2024-06-26T07:06:51.6924895Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:06:51.6953138Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:06:51.6981979Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:06:51.7009452Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:06:51.7037218Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:06:51.7063174Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:06:51.7093560Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:06:51.7119551Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:06:51.7145748Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:06:51.7173431Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:06:51.7202118Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:06:51.7230507Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:06:51.7258232Z Entering 'third_party/mimalloc' 2024-06-26T07:06:51.7288451Z Entering 'third_party/nccl/nccl' 2024-06-26T07:06:51.7317973Z Entering 'third_party/nlohmann' 2024-06-26T07:06:51.7346470Z Entering 'third_party/onnx' 2024-06-26T07:06:51.7386174Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:06:51.7414804Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:06:51.7441726Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:06:51.7472398Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:06:51.7499190Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:06:51.7525452Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:06:51.7554300Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:06:51.7581000Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:06:51.7608052Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:06:51.7635991Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:06:51.7661815Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:06:51.7690705Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:06:51.7718133Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:06:51.7761649Z Entering 'third_party/pocketfft' 2024-06-26T07:06:51.7792629Z Entering 'third_party/protobuf' 2024-06-26T07:06:51.7821970Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:06:51.7848303Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:06:51.7875425Z Entering 'third_party/psimd' 2024-06-26T07:06:51.7902594Z Entering 'third_party/pthreadpool' 2024-06-26T07:06:51.7933219Z Entering 'third_party/pybind11' 2024-06-26T07:06:51.7959113Z Entering 'third_party/python-peachpy' 2024-06-26T07:06:51.7985748Z Entering 'third_party/sleef' 2024-06-26T07:06:51.8015964Z Entering 'third_party/tensorpipe' 2024-06-26T07:06:51.8045168Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:06:51.8073028Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:06:51.8101708Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:06:51.8129737Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:06:51.8156137Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:06:51.8197344Z ##[endgroup] 2024-06-26T07:06:51.8199612Z ##[group]Persisting credentials for submodules 2024-06-26T07:06:51.8201727Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T07:06:51.8417533Z Entering 'android/libs/fbjni' 2024-06-26T07:06:51.8453353Z Entering 'third_party/FP16' 2024-06-26T07:06:51.8488643Z Entering 'third_party/FXdiv' 2024-06-26T07:06:51.8524698Z Entering 'third_party/NNPACK' 2024-06-26T07:06:51.8560984Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:06:51.8597276Z Entering 'third_party/XNNPACK' 2024-06-26T07:06:51.8644129Z Entering 'third_party/benchmark' 2024-06-26T07:06:51.8678190Z Entering 'third_party/cpp-httplib' 2024-06-26T07:06:51.8713620Z Entering 'third_party/cpuinfo' 2024-06-26T07:06:51.8749192Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:06:51.8784740Z Entering 'third_party/cutlass' 2024-06-26T07:06:51.8826449Z Entering 'third_party/eigen' 2024-06-26T07:06:51.8863141Z Entering 'third_party/fbgemm' 2024-06-26T07:06:51.8899831Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:06:51.8934030Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:06:51.8967911Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:06:51.9007212Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:06:51.9040769Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:06:51.9077107Z Entering 'third_party/flatbuffers' 2024-06-26T07:06:51.9115570Z Entering 'third_party/fmt' 2024-06-26T07:06:51.9152320Z Entering 'third_party/foxi' 2024-06-26T07:06:51.9187688Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:06:51.9224755Z Entering 'third_party/gloo' 2024-06-26T07:06:51.9258065Z Entering 'third_party/googletest' 2024-06-26T07:06:51.9293626Z Entering 'third_party/ideep' 2024-06-26T07:06:51.9330786Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:06:51.9372056Z Entering 'third_party/ittapi' 2024-06-26T07:06:51.9406390Z Entering 'third_party/kineto' 2024-06-26T07:06:51.9441180Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:06:51.9476683Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:06:51.9513523Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:06:51.9549389Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:06:51.9583986Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:06:51.9616257Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:06:51.9657631Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:06:51.9690171Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:06:51.9724390Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:06:51.9759981Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:06:51.9796863Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:06:51.9833433Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:06:51.9867967Z Entering 'third_party/mimalloc' 2024-06-26T07:06:51.9904107Z Entering 'third_party/nccl/nccl' 2024-06-26T07:06:51.9940579Z Entering 'third_party/nlohmann' 2024-06-26T07:06:51.9977362Z Entering 'third_party/onnx' 2024-06-26T07:06:52.0026071Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:06:52.0062285Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:06:52.0100233Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:06:52.0138653Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:06:52.0174883Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:06:52.0209665Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:06:52.0244064Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:06:52.0279447Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:06:52.0314404Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:06:52.0347486Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:06:52.0381058Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:06:52.0417273Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:06:52.0450709Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:06:52.0500711Z Entering 'third_party/pocketfft' 2024-06-26T07:06:52.0536569Z Entering 'third_party/protobuf' 2024-06-26T07:06:52.0576340Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:06:52.0611694Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:06:52.0645706Z Entering 'third_party/psimd' 2024-06-26T07:06:52.0681190Z Entering 'third_party/pthreadpool' 2024-06-26T07:06:52.0716924Z Entering 'third_party/pybind11' 2024-06-26T07:06:52.0752149Z Entering 'third_party/python-peachpy' 2024-06-26T07:06:52.0786960Z Entering 'third_party/sleef' 2024-06-26T07:06:52.0822849Z Entering 'third_party/tensorpipe' 2024-06-26T07:06:52.0859457Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:06:52.0895275Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:06:52.0930409Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:06:52.0965792Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:06:52.1000465Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:06:52.1049593Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T07:06:52.1261750Z Entering 'android/libs/fbjni' 2024-06-26T07:06:52.1293016Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T07:06:52.1307573Z Entering 'third_party/FP16' 2024-06-26T07:06:52.1339164Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T07:06:52.1352760Z Entering 'third_party/FXdiv' 2024-06-26T07:06:52.1384736Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T07:06:52.1398088Z Entering 'third_party/NNPACK' 2024-06-26T07:06:52.1430223Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T07:06:52.1444360Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:06:52.1477118Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T07:06:52.1490586Z Entering 'third_party/XNNPACK' 2024-06-26T07:06:52.1520967Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T07:06:52.1546317Z Entering 'third_party/benchmark' 2024-06-26T07:06:52.1578730Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:06:52.1592593Z Entering 'third_party/cpp-httplib' 2024-06-26T07:06:52.1624348Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T07:06:52.1638033Z Entering 'third_party/cpuinfo' 2024-06-26T07:06:52.1669076Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:06:52.1683323Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:06:52.1714079Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T07:06:52.1724374Z Entering 'third_party/cutlass' 2024-06-26T07:06:52.1757327Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:06:52.1775478Z Entering 'third_party/eigen' 2024-06-26T07:06:52.1807064Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T07:06:52.1821890Z Entering 'third_party/fbgemm' 2024-06-26T07:06:52.1853733Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T07:06:52.1866293Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:06:52.1897670Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T07:06:52.1911684Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:06:52.1942705Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:06:52.1955946Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:06:52.1989157Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:06:52.2004503Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:06:52.2036032Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:06:52.2048614Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:06:52.2079738Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T07:06:52.2092761Z Entering 'third_party/flatbuffers' 2024-06-26T07:06:52.2124521Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T07:06:52.2138428Z Entering 'third_party/fmt' 2024-06-26T07:06:52.2170621Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:06:52.2184109Z Entering 'third_party/foxi' 2024-06-26T07:06:52.2214768Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T07:06:52.2227417Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:06:52.2257002Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T07:06:52.2268166Z Entering 'third_party/gloo' 2024-06-26T07:06:52.2298850Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T07:06:52.2312629Z Entering 'third_party/googletest' 2024-06-26T07:06:52.2343422Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:06:52.2357875Z Entering 'third_party/ideep' 2024-06-26T07:06:52.2390119Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T07:06:52.2402395Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:06:52.2432636Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T07:06:52.2448751Z Entering 'third_party/ittapi' 2024-06-26T07:06:52.2480134Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T07:06:52.2492759Z Entering 'third_party/kineto' 2024-06-26T07:06:52.2523568Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T07:06:52.2537578Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:06:52.2570631Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T07:06:52.2583861Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:06:52.2615812Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T07:06:52.2628630Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:06:52.2661468Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T07:06:52.2673781Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:06:52.2705385Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:06:52.2718532Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:06:52.2752042Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T07:06:52.2762245Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:06:52.2793943Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T07:06:52.2808101Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:06:52.2839968Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T07:06:52.2853282Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:06:52.2884783Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:06:52.2897753Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:06:52.2929178Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T07:06:52.2941954Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:06:52.2972435Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T07:06:52.2985989Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:06:52.3017094Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T07:06:52.3030423Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:06:52.3061990Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T07:06:52.3079540Z Entering 'third_party/mimalloc' 2024-06-26T07:06:52.3113590Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T07:06:52.3125196Z Entering 'third_party/nccl/nccl' 2024-06-26T07:06:52.3156670Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T07:06:52.3168656Z Entering 'third_party/nlohmann' 2024-06-26T07:06:52.3200735Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T07:06:52.3213626Z Entering 'third_party/onnx' 2024-06-26T07:06:52.3246002Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T07:06:52.3272048Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:06:52.3304161Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:06:52.3316543Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:06:52.3349247Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:06:52.3364336Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:06:52.3398529Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T07:06:52.3413773Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:06:52.3445552Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:06:52.3458640Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:06:52.3490440Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:06:52.3502951Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:06:52.3534586Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T07:06:52.3545192Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:06:52.3575741Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T07:06:52.3589465Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:06:52.3622183Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T07:06:52.3634997Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:06:52.3665559Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T07:06:52.3678459Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:06:52.3710888Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T07:06:52.3722605Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:06:52.3754332Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T07:06:52.3769283Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:06:52.3800914Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T07:06:52.3815151Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:06:52.3846560Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T07:06:52.3874345Z Entering 'third_party/pocketfft' 2024-06-26T07:06:52.3906287Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T07:06:52.3920374Z Entering 'third_party/protobuf' 2024-06-26T07:06:52.3954051Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T07:06:52.3971234Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:06:52.4003697Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:06:52.4016034Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:06:52.4047974Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:06:52.4062329Z Entering 'third_party/psimd' 2024-06-26T07:06:52.4094329Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T07:06:52.4106742Z Entering 'third_party/pthreadpool' 2024-06-26T07:06:52.4139298Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T07:06:52.4152188Z Entering 'third_party/pybind11' 2024-06-26T07:06:52.4186300Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:06:52.4199499Z Entering 'third_party/python-peachpy' 2024-06-26T07:06:52.4232575Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T07:06:52.4245457Z Entering 'third_party/sleef' 2024-06-26T07:06:52.4278292Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T07:06:52.4290914Z Entering 'third_party/tensorpipe' 2024-06-26T07:06:52.4323638Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T07:06:52.4334958Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:06:52.4366222Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:06:52.4379318Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:06:52.4412081Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T07:06:52.4424122Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:06:52.4454141Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T07:06:52.4464731Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:06:52.4497115Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:06:52.4509284Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:06:52.4542729Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T07:06:52.5289799Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T07:06:52.5498823Z Entering 'android/libs/fbjni' 2024-06-26T07:06:52.5524705Z Entering 'third_party/FP16' 2024-06-26T07:06:52.5550869Z Entering 'third_party/FXdiv' 2024-06-26T07:06:52.5580373Z Entering 'third_party/NNPACK' 2024-06-26T07:06:52.5611501Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:06:52.5639719Z Entering 'third_party/XNNPACK' 2024-06-26T07:06:52.5680359Z Entering 'third_party/benchmark' 2024-06-26T07:06:52.5706833Z Entering 'third_party/cpp-httplib' 2024-06-26T07:06:52.5734212Z Entering 'third_party/cpuinfo' 2024-06-26T07:06:52.5763441Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:06:52.5792649Z Entering 'third_party/cutlass' 2024-06-26T07:06:52.5827797Z Entering 'third_party/eigen' 2024-06-26T07:06:52.5857358Z Entering 'third_party/fbgemm' 2024-06-26T07:06:52.5885675Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:06:52.5912662Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:06:52.5941807Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:06:52.5974559Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:06:52.6003105Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:06:52.6032400Z Entering 'third_party/flatbuffers' 2024-06-26T07:06:52.6062990Z Entering 'third_party/fmt' 2024-06-26T07:06:52.6093785Z Entering 'third_party/foxi' 2024-06-26T07:06:52.6123405Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:06:52.6153725Z Entering 'third_party/gloo' 2024-06-26T07:06:52.6181403Z Entering 'third_party/googletest' 2024-06-26T07:06:52.6208444Z Entering 'third_party/ideep' 2024-06-26T07:06:52.6235162Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:06:52.6269168Z Entering 'third_party/ittapi' 2024-06-26T07:06:52.6299478Z Entering 'third_party/kineto' 2024-06-26T07:06:52.6326476Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:06:52.6354321Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:06:52.6384868Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:06:52.6417026Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:06:52.6443792Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:06:52.6469906Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:06:52.6500065Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:06:52.6530164Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:06:52.6558201Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:06:52.6586804Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:06:52.6617053Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:06:52.6643727Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:06:52.6673474Z Entering 'third_party/mimalloc' 2024-06-26T07:06:52.6702478Z Entering 'third_party/nccl/nccl' 2024-06-26T07:06:52.6728307Z Entering 'third_party/nlohmann' 2024-06-26T07:06:52.6756998Z Entering 'third_party/onnx' 2024-06-26T07:06:52.6798868Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:06:52.6829098Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:06:52.6858148Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:06:52.6887270Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:06:52.6915573Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:06:52.6942900Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:06:52.6970234Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:06:52.6998845Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:06:52.7025710Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:06:52.7054084Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:06:52.7081105Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:06:52.7111407Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:06:52.7139605Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:06:52.7182066Z Entering 'third_party/pocketfft' 2024-06-26T07:06:52.7208662Z Entering 'third_party/protobuf' 2024-06-26T07:06:52.7240259Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:06:52.7266769Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:06:52.7293456Z Entering 'third_party/psimd' 2024-06-26T07:06:52.7321783Z Entering 'third_party/pthreadpool' 2024-06-26T07:06:52.7349950Z Entering 'third_party/pybind11' 2024-06-26T07:06:52.7378068Z Entering 'third_party/python-peachpy' 2024-06-26T07:06:52.7406147Z Entering 'third_party/sleef' 2024-06-26T07:06:52.7435575Z Entering 'third_party/tensorpipe' 2024-06-26T07:06:52.7462892Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:06:52.7491454Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:06:52.7518504Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:06:52.7547284Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:06:52.7574991Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:06:52.7618234Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T07:06:52.7832903Z Entering 'android/libs/fbjni' 2024-06-26T07:06:52.7861401Z Entering 'third_party/FP16' 2024-06-26T07:06:52.7891672Z Entering 'third_party/FXdiv' 2024-06-26T07:06:52.7918461Z Entering 'third_party/NNPACK' 2024-06-26T07:06:52.7945482Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:06:52.7976042Z Entering 'third_party/XNNPACK' 2024-06-26T07:06:52.8016527Z Entering 'third_party/benchmark' 2024-06-26T07:06:52.8045854Z Entering 'third_party/cpp-httplib' 2024-06-26T07:06:52.8075471Z Entering 'third_party/cpuinfo' 2024-06-26T07:06:52.8104351Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:06:52.8132683Z Entering 'third_party/cutlass' 2024-06-26T07:06:52.8167532Z Entering 'third_party/eigen' 2024-06-26T07:06:52.8199452Z Entering 'third_party/fbgemm' 2024-06-26T07:06:52.8230438Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:06:52.8255593Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:06:52.8282540Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:06:52.8313472Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:06:52.8342932Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:06:52.8370949Z Entering 'third_party/flatbuffers' 2024-06-26T07:06:52.8400432Z Entering 'third_party/fmt' 2024-06-26T07:06:52.8427508Z Entering 'third_party/foxi' 2024-06-26T07:06:52.8454329Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:06:52.8480934Z Entering 'third_party/gloo' 2024-06-26T07:06:52.8507524Z Entering 'third_party/googletest' 2024-06-26T07:06:52.8535067Z Entering 'third_party/ideep' 2024-06-26T07:06:52.8562318Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:06:52.8596917Z Entering 'third_party/ittapi' 2024-06-26T07:06:52.8622494Z Entering 'third_party/kineto' 2024-06-26T07:06:52.8650051Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:06:52.8677708Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:06:52.8707532Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:06:52.8736140Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:06:52.8765256Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:06:52.8793151Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:06:52.8821891Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:06:52.8847700Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:06:52.8875935Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:06:52.8902479Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:06:52.8931668Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:06:52.8958198Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:06:52.8986286Z Entering 'third_party/mimalloc' 2024-06-26T07:06:52.9015209Z Entering 'third_party/nccl/nccl' 2024-06-26T07:06:52.9045416Z Entering 'third_party/nlohmann' 2024-06-26T07:06:52.9074100Z Entering 'third_party/onnx' 2024-06-26T07:06:52.9115243Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:06:52.9144461Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:06:52.9174246Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:06:52.9201426Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:06:52.9230514Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:06:52.9258447Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:06:52.9285755Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:06:52.9315994Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:06:52.9342779Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:06:52.9372077Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:06:52.9399198Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:06:52.9430114Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:06:52.9459119Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:06:52.9501567Z Entering 'third_party/pocketfft' 2024-06-26T07:06:52.9530601Z Entering 'third_party/protobuf' 2024-06-26T07:06:52.9562462Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:06:52.9591730Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:06:52.9620103Z Entering 'third_party/psimd' 2024-06-26T07:06:52.9649163Z Entering 'third_party/pthreadpool' 2024-06-26T07:06:52.9677227Z Entering 'third_party/pybind11' 2024-06-26T07:06:52.9707052Z Entering 'third_party/python-peachpy' 2024-06-26T07:06:52.9734776Z Entering 'third_party/sleef' 2024-06-26T07:06:52.9762212Z Entering 'third_party/tensorpipe' 2024-06-26T07:06:52.9792373Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:06:52.9819858Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:06:52.9845774Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:06:52.9874603Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:06:52.9901087Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:06:52.9942166Z ##[endgroup] 2024-06-26T07:06:52.9970752Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T07:06:52.9992023Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T07:06:53.0166350Z Prepare all required actions 2024-06-26T07:06:53.0167055Z Getting action download info 2024-06-26T07:06:53.1569287Z ##[group]Run ./.github/actions/setup-linux 2024-06-26T07:06:53.1569723Z env: 2024-06-26T07:06:53.1569993Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:53.1570339Z ##[endgroup] 2024-06-26T07:06:53.1627763Z ##[group]Run set -euo pipefail 2024-06-26T07:06:53.1628222Z set -euo pipefail 2024-06-26T07:06:53.1628637Z function get_ec2_metadata() { 2024-06-26T07:06:53.1629206Z  # Pulled from instance metadata endpoint for EC2 2024-06-26T07:06:53.1630149Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-26T07:06:53.1630955Z  category=$1 2024-06-26T07:06:53.1631501Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-26T07:06:53.1632174Z  runner_name_str=i-06740481be1ad1e06 2024-06-26T07:06:53.1632691Z  if [[ -f /.inarc ]]; then 2024-06-26T07:06:53.1633206Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-26T07:06:53.1633807Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-26T07:06:53.1634549Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-26T07:06:53.1635175Z  else 2024-06-26T07:06:53.1635680Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-26T07:06:53.1636284Z  fi 2024-06-26T07:06:53.1636570Z } 2024-06-26T07:06:53.1636946Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-26T07:06:53.1637550Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-26T07:06:53.1638235Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-26T07:06:53.1638835Z echo "system info $(uname -a)" 2024-06-26T07:06:53.1646974Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:53.1647479Z env: 2024-06-26T07:06:53.1647770Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:53.1648125Z ##[endgroup] 2024-06-26T07:06:53.1722065Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-26T07:06:53.1767394Z instance-id: i-06740481be1ad1e06 2024-06-26T07:06:53.1812753Z instance-type: g5.4xlarge 2024-06-26T07:06:53.1819215Z system info Linux ip-10-0-46-137.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-26T07:06:53.1839486Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:06:53.1840441Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:06:53.1848737Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:53.1849242Z env: 2024-06-26T07:06:53.1849525Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:53.1849882Z ##[endgroup] 2024-06-26T07:06:53.1910699Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-26T07:06:53.1911339Z if systemctl is-active --quiet docker; then 2024-06-26T07:06:53.1911898Z  echo "Docker daemon is running..."; 2024-06-26T07:06:53.1912507Z else 2024-06-26T07:06:53.1913019Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-26T07:06:53.1913631Z fi 2024-06-26T07:06:53.1921200Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:53.1921735Z env: 2024-06-26T07:06:53.1922025Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:53.1922399Z ##[endgroup] 2024-06-26T07:06:53.1959793Z Docker daemon is running... 2024-06-26T07:06:53.2003470Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:06:53.2004041Z with: 2024-06-26T07:06:53.2004313Z shell: bash 2024-06-26T07:06:53.2004604Z timeout_minutes: 5 2024-06-26T07:06:53.2004935Z max_attempts: 3 2024-06-26T07:06:53.2005254Z retry_wait_seconds: 30 2024-06-26T07:06:53.2007047Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-26T07:06:53.2008560Z polling_interval_seconds: 1 2024-06-26T07:06:53.2008955Z warning_on_retry: true 2024-06-26T07:06:53.2009325Z continue_on_error: false 2024-06-26T07:06:53.2009674Z env: 2024-06-26T07:06:53.2009953Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:53.2010326Z AWS_RETRY_MODE: standard 2024-06-26T07:06:53.2010693Z AWS_MAX_ATTEMPTS: 5 2024-06-26T07:06:53.2011047Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:06:53.2011436Z ##[endgroup] 2024-06-26T07:06:54.0634456Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:06:54.0635331Z Configure a credential helper to remove this warning. See 2024-06-26T07:06:54.0637164Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:06:54.0637957Z 2024-06-26T07:06:54.0638159Z Login Succeeded 2024-06-26T07:06:54.2499763Z Command completed after 1 attempt(s). 2024-06-26T07:06:54.2540106Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:06:54.2540863Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:06:54.2541543Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:06:54.2549484Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:54.2549990Z env: 2024-06-26T07:06:54.2550274Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:54.2550627Z ##[endgroup] 2024-06-26T07:06:54.2612298Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:06:54.2613113Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:06:54.2613738Z # shellcheck disable=SC2046 2024-06-26T07:06:54.2614227Z docker stop $(docker ps -q) || true 2024-06-26T07:06:54.2614739Z # Prune all of the docker images 2024-06-26T07:06:54.2615218Z docker system prune -af 2024-06-26T07:06:54.2622289Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:54.2622829Z env: 2024-06-26T07:06:54.2623117Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:54.2623488Z ##[endgroup] 2024-06-26T07:06:54.2868257Z "docker stop" requires at least 1 argument. 2024-06-26T07:06:54.2868936Z See 'docker stop --help'. 2024-06-26T07:06:54.2869188Z 2024-06-26T07:06:54.2869423Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T07:06:54.2869819Z 2024-06-26T07:06:54.2869972Z Stop one or more running containers 2024-06-26T07:06:54.3013476Z Total reclaimed space: 0B 2024-06-26T07:06:54.3042637Z ##[group]Run set +e 2024-06-26T07:06:54.3043013Z set +e 2024-06-26T07:06:54.3043317Z set -x 2024-06-26T07:06:54.3043615Z  2024-06-26T07:06:54.3043949Z PT_DOMAIN=download.pytorch.org 2024-06-26T07:06:54.3044773Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-26T07:06:54.3045887Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-26T07:06:54.3047200Z # one is returned at random 2024-06-26T07:06:54.3047779Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-26T07:06:54.3048301Z  2024-06-26T07:06:54.3048622Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:06:54.3049259Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-26T07:06:54.3050042Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-26T07:06:54.3050606Z  2024-06-26T07:06:54.3050932Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:06:54.3051494Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-26T07:06:54.3052013Z  exit 1 2024-06-26T07:06:54.3052321Z  fi 2024-06-26T07:06:54.3052608Z fi 2024-06-26T07:06:54.3053013Z  2024-06-26T07:06:54.3053398Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-26T07:06:54.3053937Z  # Clean up any old records first 2024-06-26T07:06:54.3054466Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-26T07:06:54.3054936Z fi 2024-06-26T07:06:54.3055217Z  2024-06-26T07:06:54.3055664Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-26T07:06:54.3056231Z cat /etc/hosts 2024-06-26T07:06:54.3063889Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:54.3064507Z env: 2024-06-26T07:06:54.3064796Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:54.3065148Z ##[endgroup] 2024-06-26T07:06:54.3084948Z + PT_DOMAIN=download.pytorch.org 2024-06-26T07:06:54.3089433Z ++ dig -4 +short download.pytorch.org 2024-06-26T07:06:54.3090046Z ++ tail -n1 2024-06-26T07:06:54.3179194Z + RESOLVED_IP=18.67.76.121 2024-06-26T07:06:54.3179765Z + '[' -z 18.67.76.121 ']' 2024-06-26T07:06:54.3180279Z + grep -r download.pytorch.org /etc/hosts 2024-06-26T07:06:54.3185361Z 18.160.10.76 download.pytorch.org 2024-06-26T07:06:54.3186373Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-06-26T07:06:54.3285520Z + echo '18.67.76.121 download.pytorch.org' 2024-06-26T07:06:54.3286158Z + sudo tee -a /etc/hosts 2024-06-26T07:06:54.3642817Z 18.67.76.121 download.pytorch.org 2024-06-26T07:06:54.3656411Z + cat /etc/hosts 2024-06-26T07:06:54.3661853Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-26T07:06:54.3673593Z ::1 localhost6 localhost6.localdomain6 2024-06-26T07:06:54.3674122Z 18.67.76.121 download.pytorch.org 2024-06-26T07:06:54.3827634Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T07:06:54.3828266Z with: 2024-06-26T07:06:54.3829431Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:54.3830727Z docker-build-dir: .ci/docker 2024-06-26T07:06:54.3831145Z working-directory: . 2024-06-26T07:06:54.3831659Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:54.3832238Z force-push: false 2024-06-26T07:06:54.3848140Z env: 2024-06-26T07:06:54.3848477Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:54.3848852Z ##[endgroup] 2024-06-26T07:06:54.3867388Z ##[group]Run set -ex 2024-06-26T07:06:54.3867769Z set -ex 2024-06-26T07:06:54.3868075Z  2024-06-26T07:06:54.3868657Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T07:06:54.3869670Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T07:06:54.3870507Z # job could then download the pre-built image as usual 2024-06-26T07:06:54.3871280Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T07:06:54.3871978Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:54.3872644Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:54.3873353Z  2024-06-26T07:06:54.3873895Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T07:06:54.3874546Z  exit 0 2024-06-26T07:06:54.3874873Z else 2024-06-26T07:06:54.3875257Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:54.3875735Z fi 2024-06-26T07:06:54.3876025Z  2024-06-26T07:06:54.3876531Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T07:06:54.3877425Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T07:06:54.3878245Z  # use it as it is, but first let's extract the tag 2024-06-26T07:06:54.3878986Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T07:06:54.3879751Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:54.3880497Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:54.3881100Z else 2024-06-26T07:06:54.3881568Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T07:06:54.3882274Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:54.3883212Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:54.3884000Z fi 2024-06-26T07:06:54.3892016Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:54.3892592Z env: 2024-06-26T07:06:54.3892891Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:54.3893251Z REPO_NAME: pytorch 2024-06-26T07:06:54.3894430Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:54.3895688Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:06:54.3896210Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:54.3896737Z ##[endgroup] 2024-06-26T07:06:54.3917263Z + [[ ! -d .ci/docker ]] 2024-06-26T07:06:54.3917686Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T07:06:54.3918067Z + echo skip=false 2024-06-26T07:06:54.3920068Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T07:06:54.3923567Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:54.3925310Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T07:06:54.3930059Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:54.3930998Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:54.3932689Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:54.3958218Z ##[group]Run set +e 2024-06-26T07:06:54.3958596Z set +e 2024-06-26T07:06:54.3958911Z set -x 2024-06-26T07:06:54.3959208Z  2024-06-26T07:06:54.3959506Z login() { 2024-06-26T07:06:54.3960201Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:06:54.3960941Z } 2024-06-26T07:06:54.3961217Z  2024-06-26T07:06:54.3961504Z retry () { 2024-06-26T07:06:54.3961912Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:06:54.3962417Z } 2024-06-26T07:06:54.3962696Z  2024-06-26T07:06:54.3963025Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:06:54.3963458Z  2024-06-26T07:06:54.3963948Z # Check if image already exists, if it does then skip building it 2024-06-26T07:06:54.3964793Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T07:06:54.3965324Z  exit 0 2024-06-26T07:06:54.3965643Z fi 2024-06-26T07:06:54.3965927Z  2024-06-26T07:06:54.3966471Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T07:06:54.3967641Z # be empty. The default action would be to continue rebuild the image 2024-06-26T07:06:54.3968419Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T07:06:54.3969111Z  # if we're on the base branch then use the parent commit 2024-06-26T07:06:54.3969738Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T07:06:54.3970185Z else 2024-06-26T07:06:54.3970684Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T07:06:54.3971401Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T07:06:54.3971925Z fi 2024-06-26T07:06:54.3972214Z  2024-06-26T07:06:54.3972543Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T07:06:54.3973057Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:54.3973538Z  2024-06-26T07:06:54.3974221Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T07:06:54.3975003Z  exit 0 2024-06-26T07:06:54.3975307Z fi 2024-06-26T07:06:54.3975596Z  2024-06-26T07:06:54.3976053Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T07:06:54.3977063Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T07:06:54.3977893Z  exit 1 2024-06-26T07:06:54.3978203Z fi 2024-06-26T07:06:54.3978485Z  2024-06-26T07:06:54.3978995Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T07:06:54.3979984Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T07:06:54.3980887Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T07:06:54.3981886Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T07:06:54.3983023Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T07:06:54.3983803Z fi 2024-06-26T07:06:54.3984098Z  2024-06-26T07:06:54.3984561Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:54.3991586Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:54.3992101Z env: 2024-06-26T07:06:54.3992420Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:54.3992803Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:06:54.3993289Z BASE_REVISION: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:06:54.3994602Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:54.3995880Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:54.3996490Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:54.3997017Z ##[endgroup] 2024-06-26T07:06:54.4015946Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:54.4016611Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:54.4017595Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:06:54.4018808Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:54.8152347Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:06:54.8153193Z Configure a credential helper to remove this warning. See 2024-06-26T07:06:54.8154399Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:06:54.8154997Z 2024-06-26T07:06:54.8155131Z Login Succeeded 2024-06-26T07:06:54.8165366Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:55.0335406Z { 2024-06-26T07:06:55.0335777Z "schemaVersion": 2, 2024-06-26T07:06:55.0337679Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T07:06:55.0338738Z "config": { 2024-06-26T07:06:55.0339605Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T07:06:55.0340600Z "size": 48681, 2024-06-26T07:06:55.0341552Z "digest": "sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd" 2024-06-26T07:06:55.0342640Z }, 2024-06-26T07:06:55.0343187Z "layers": [ 2024-06-26T07:06:55.0343738Z { 2024-06-26T07:06:55.0344587Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0345621Z "size": 28580681, 2024-06-26T07:06:55.0346654Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-06-26T07:06:55.0347775Z }, 2024-06-26T07:06:55.0348330Z { 2024-06-26T07:06:55.0349163Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0350102Z "size": 7943451, 2024-06-26T07:06:55.0350837Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-06-26T07:06:55.0351476Z + exit 0 2024-06-26T07:06:55.0351956Z }, 2024-06-26T07:06:55.0352327Z { 2024-06-26T07:06:55.0352850Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0353639Z "size": 55728572, 2024-06-26T07:06:55.0354307Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-06-26T07:06:55.0354950Z }, 2024-06-26T07:06:55.0355257Z { 2024-06-26T07:06:55.0355874Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0356618Z "size": 186, 2024-06-26T07:06:55.0357183Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-06-26T07:06:55.0357819Z }, 2024-06-26T07:06:55.0358060Z { 2024-06-26T07:06:55.0358513Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0359084Z "size": 6886, 2024-06-26T07:06:55.0359636Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-06-26T07:06:55.0360552Z }, 2024-06-26T07:06:55.0360869Z { 2024-06-26T07:06:55.0361331Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0361979Z "size": 1329001756, 2024-06-26T07:06:55.0362803Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-06-26T07:06:55.0363635Z }, 2024-06-26T07:06:55.0363977Z { 2024-06-26T07:06:55.0364514Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0365100Z "size": 62501, 2024-06-26T07:06:55.0365675Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-06-26T07:06:55.0366311Z }, 2024-06-26T07:06:55.0366570Z { 2024-06-26T07:06:55.0367283Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0367875Z "size": 1684, 2024-06-26T07:06:55.0368490Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-06-26T07:06:55.0369239Z }, 2024-06-26T07:06:55.0369596Z { 2024-06-26T07:06:55.0370253Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0370981Z "size": 1523, 2024-06-26T07:06:55.0371638Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-06-26T07:06:55.0372474Z }, 2024-06-26T07:06:55.0372796Z { 2024-06-26T07:06:55.0373367Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0374088Z "size": 2528295403, 2024-06-26T07:06:55.0375042Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-06-26T07:06:55.0375861Z }, 2024-06-26T07:06:55.0376126Z { 2024-06-26T07:06:55.0376575Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0377155Z "size": 86016, 2024-06-26T07:06:55.0377742Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-06-26T07:06:55.0378381Z }, 2024-06-26T07:06:55.0378634Z { 2024-06-26T07:06:55.0379097Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0379682Z "size": 1823, 2024-06-26T07:06:55.0380244Z "digest": "sha256:309b872da805fcdf74137a887464876fa0615dc8b24199307119eb65fc2bfb49" 2024-06-26T07:06:55.0380873Z }, 2024-06-26T07:06:55.0381137Z { 2024-06-26T07:06:55.0381593Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0382171Z "size": 246737542, 2024-06-26T07:06:55.0382754Z "digest": "sha256:e82932cb002c06a782077b8e5965f8733eb8563406a23cab260d9d8c4a994f7d" 2024-06-26T07:06:55.0383375Z }, 2024-06-26T07:06:55.0383642Z { 2024-06-26T07:06:55.0384090Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0384789Z "size": 545, 2024-06-26T07:06:55.0385359Z "digest": "sha256:ca9dd73e3e9378225ebdcfd12199ef44649e34fe55003cfbc8afe2e8a4527c5b" 2024-06-26T07:06:55.0386005Z }, 2024-06-26T07:06:55.0386262Z { 2024-06-26T07:06:55.0386717Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0387299Z "size": 1283, 2024-06-26T07:06:55.0387852Z "digest": "sha256:16d76a79d5d6616e228537785ec83a869f68a402520bbc6e826201621ca6cb98" 2024-06-26T07:06:55.0388472Z }, 2024-06-26T07:06:55.0388744Z { 2024-06-26T07:06:55.0389193Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0389767Z "size": 485, 2024-06-26T07:06:55.0390341Z "digest": "sha256:4bdf41c9ecbe1651e0986ccbea4663a0556af27b7145a679cfa45c35dc96b891" 2024-06-26T07:06:55.0391005Z }, 2024-06-26T07:06:55.0391271Z { 2024-06-26T07:06:55.0391731Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0392311Z "size": 91710937, 2024-06-26T07:06:55.0392904Z "digest": "sha256:f4863801e30cdbcd1d4e6cb1cbbcde7dbcaf6ae317ca80429295741ce60ac31d" 2024-06-26T07:06:55.0393555Z }, 2024-06-26T07:06:55.0393809Z { 2024-06-26T07:06:55.0394259Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0394938Z "size": 3233, 2024-06-26T07:06:55.0395512Z "digest": "sha256:d37dd9ed8948a0d02e1198bbedadbca36477a0980dd553196385aab0c025581d" 2024-06-26T07:06:55.0396163Z }, 2024-06-26T07:06:55.0396423Z { 2024-06-26T07:06:55.0396879Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0397450Z "size": 1905, 2024-06-26T07:06:55.0398027Z "digest": "sha256:cf2f34f225dbed17c1a9a8b9ea4ad030aba797774a716dc0a82fe212324aa751" 2024-06-26T07:06:55.0398679Z }, 2024-06-26T07:06:55.0399023Z { 2024-06-26T07:06:55.0399469Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0400041Z "size": 702, 2024-06-26T07:06:55.0400602Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:06:55.0401240Z }, 2024-06-26T07:06:55.0401495Z { 2024-06-26T07:06:55.0401948Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0402528Z "size": 2784789002, 2024-06-26T07:06:55.0403120Z "digest": "sha256:3d914950f8addfc4d657a2365c76e1143eb80b710cd3f6e9ae05640217fc6347" 2024-06-26T07:06:55.0403748Z }, 2024-06-26T07:06:55.0404011Z { 2024-06-26T07:06:55.0404471Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0405047Z "size": 381, 2024-06-26T07:06:55.0405615Z "digest": "sha256:96d0e2032184c57e62a652aaffcf0241f0d0c7a23e894ca0ad192c4a64a36583" 2024-06-26T07:06:55.0406324Z }, 2024-06-26T07:06:55.0406593Z { 2024-06-26T07:06:55.0407270Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0407853Z "size": 11879, 2024-06-26T07:06:55.0408419Z "digest": "sha256:06e12ce59790cb8b091e5d619c2a62a28329008ee184a8b9a562e3db12471f30" 2024-06-26T07:06:55.0409046Z }, 2024-06-26T07:06:55.0409300Z { 2024-06-26T07:06:55.0409763Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0410328Z "size": 803, 2024-06-26T07:06:55.0410884Z "digest": "sha256:ed712f63cde7070bdec2a6179c4c35e752c6991ddbc69fe9f2724a9903250ceb" 2024-06-26T07:06:55.0411520Z }, 2024-06-26T07:06:55.0411774Z { 2024-06-26T07:06:55.0412225Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0412789Z "size": 106, 2024-06-26T07:06:55.0413345Z "digest": "sha256:a97a71ef8a4e66234dfc2098834f8395ac5630240e60164d1a251fdb0115ba85" 2024-06-26T07:06:55.0413969Z }, 2024-06-26T07:06:55.0414227Z { 2024-06-26T07:06:55.0414673Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0415251Z "size": 503, 2024-06-26T07:06:55.0415801Z "digest": "sha256:25cd0c482ad612f936b35b4f220c580e16be982226aac8801a6a849a6bfdc221" 2024-06-26T07:06:55.0416425Z }, 2024-06-26T07:06:55.0416674Z { 2024-06-26T07:06:55.0417120Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0417697Z "size": 121477177, 2024-06-26T07:06:55.0418265Z "digest": "sha256:9bf94bc456584365fda21d0ed83b7cc690b08a71727b71c4c1e667e15028a386" 2024-06-26T07:06:55.0418895Z }, 2024-06-26T07:06:55.0419151Z { 2024-06-26T07:06:55.0419592Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0420151Z "size": 109, 2024-06-26T07:06:55.0420701Z "digest": "sha256:de45a14c9b227e4b0c9d45d42223b50a71c03522ddd9e2ca0d9517835206c823" 2024-06-26T07:06:55.0421325Z }, 2024-06-26T07:06:55.0421577Z { 2024-06-26T07:06:55.0422012Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0422581Z "size": 491, 2024-06-26T07:06:55.0423129Z "digest": "sha256:e99241e77209b0b35a0fb59a61c0821fc161190178af7e4af45e6eeaa043375e" 2024-06-26T07:06:55.0423744Z }, 2024-06-26T07:06:55.0423994Z { 2024-06-26T07:06:55.0424504Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0425258Z "size": 291, 2024-06-26T07:06:55.0425821Z "digest": "sha256:35e8f4e2a42f9df0c969dedd2199cf71719cf5ecfa676dfa014bbdeb4fdb295f" 2024-06-26T07:06:55.0426574Z }, 2024-06-26T07:06:55.0426837Z { 2024-06-26T07:06:55.0427288Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0427852Z "size": 101, 2024-06-26T07:06:55.0428405Z "digest": "sha256:a823ffa4c541394b52953391dc0a6867e26c37fe4a71fa022a2cc3f07f0ba70d" 2024-06-26T07:06:55.0429039Z }, 2024-06-26T07:06:55.0429308Z { 2024-06-26T07:06:55.0429748Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0430324Z "size": 1473, 2024-06-26T07:06:55.0430873Z "digest": "sha256:44ba272eae2064603605a6238b3f35cc6578859d660068e702a047a9cd8ab2fd" 2024-06-26T07:06:55.0431482Z }, 2024-06-26T07:06:55.0431739Z { 2024-06-26T07:06:55.0432188Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0432761Z "size": 446710781, 2024-06-26T07:06:55.0433331Z "digest": "sha256:da02d278e5240c285bc855ce4e42cd6121d5b903787f5c14f03e575ad1268650" 2024-06-26T07:06:55.0433960Z }, 2024-06-26T07:06:55.0434225Z { 2024-06-26T07:06:55.0434674Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0435244Z "size": 161, 2024-06-26T07:06:55.0435789Z "digest": "sha256:73a0ce681c295135456e190578733273276b38b4dd5babbb39298f384d43b1c6" 2024-06-26T07:06:55.0436415Z }, 2024-06-26T07:06:55.0436667Z { 2024-06-26T07:06:55.0437117Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0437691Z "size": 564, 2024-06-26T07:06:55.0438343Z "digest": "sha256:49737bd5ae709dbf7956e21fc8b05ba009e9e6c9b130b0126a73bd3d3f5efc78" 2024-06-26T07:06:55.0438969Z }, 2024-06-26T07:06:55.0439226Z { 2024-06-26T07:06:55.0439673Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0440251Z "size": 35874600, 2024-06-26T07:06:55.0440823Z "digest": "sha256:9cd4c745f8e4f9d5c9bb8050923575afad21f5b7852975a4a1981bc9eff282c5" 2024-06-26T07:06:55.0441458Z }, 2024-06-26T07:06:55.0441715Z { 2024-06-26T07:06:55.0442163Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0442758Z "size": 104, 2024-06-26T07:06:55.0443330Z "digest": "sha256:b2950a0365345d4c17a48627c905fd8930b3b3154c727f0526eb9ca547102bb7" 2024-06-26T07:06:55.0443953Z }, 2024-06-26T07:06:55.0444208Z { 2024-06-26T07:06:55.0444660Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0445232Z "size": 425, 2024-06-26T07:06:55.0445785Z "digest": "sha256:25e19d81d9fc081962ff4cbc09e098f4035419822712c19a410f5282300dfb9e" 2024-06-26T07:06:55.0446397Z }, 2024-06-26T07:06:55.0446660Z { 2024-06-26T07:06:55.0447285Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0447861Z "size": 20262084, 2024-06-26T07:06:55.0448438Z "digest": "sha256:60156b40fcbec2c25f0170b0a82fa8cd9303e0daae8bb1811fd0fc6562e88e8e" 2024-06-26T07:06:55.0449077Z }, 2024-06-26T07:06:55.0449338Z { 2024-06-26T07:06:55.0449785Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0450366Z "size": 439, 2024-06-26T07:06:55.0450905Z "digest": "sha256:7a868239227fd19b30a19927bbc8f058d247403fe5b343c55f4b4b8987918911" 2024-06-26T07:06:55.0451524Z }, 2024-06-26T07:06:55.0451775Z { 2024-06-26T07:06:55.0452222Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0452790Z "size": 702, 2024-06-26T07:06:55.0453362Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:06:55.0453997Z }, 2024-06-26T07:06:55.0454255Z { 2024-06-26T07:06:55.0454717Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0455284Z "size": 143, 2024-06-26T07:06:55.0455827Z "digest": "sha256:83911d7c7ecf11b5932bb7449975c52e7f737a5b867691935e1ccb85ceb25b52" 2024-06-26T07:06:55.0456456Z }, 2024-06-26T07:06:55.0456725Z { 2024-06-26T07:06:55.0457170Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0457832Z "size": 135, 2024-06-26T07:06:55.0458411Z "digest": "sha256:b3c5d66c679f58c71febdd502889bd8cd3d36e5001754cdf1e30eabe21135cce" 2024-06-26T07:06:55.0459053Z }, 2024-06-26T07:06:55.0459303Z { 2024-06-26T07:06:55.0459758Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0460336Z "size": 4911495247, 2024-06-26T07:06:55.0460921Z "digest": "sha256:84b2c7a899820f6acdb22148c5e8346d8b26c33269fd01842c9573515251e1e1" 2024-06-26T07:06:55.0461539Z }, 2024-06-26T07:06:55.0461797Z { 2024-06-26T07:06:55.0462284Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0462864Z "size": 190, 2024-06-26T07:06:55.0463438Z "digest": "sha256:c1068d03a811d8dead5da9e82fdc8869205fc4ee9cb92d4f7a4acfbb5ba83950" 2024-06-26T07:06:55.0464089Z }, 2024-06-26T07:06:55.0464354Z { 2024-06-26T07:06:55.0464880Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0465451Z "size": 565, 2024-06-26T07:06:55.0466006Z "digest": "sha256:671b4a301d81074fca17d36851f81801552c4ecb2f90ff3412aa9ab6a1a91371" 2024-06-26T07:06:55.0466634Z }, 2024-06-26T07:06:55.0466885Z { 2024-06-26T07:06:55.0467331Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0467902Z "size": 43163095, 2024-06-26T07:06:55.0468481Z "digest": "sha256:67760cf5ec46ab58d3ca98637fd9b26cb47a67446bfd0db176fe5e023606aa0d" 2024-06-26T07:06:55.0469112Z }, 2024-06-26T07:06:55.0469447Z { 2024-06-26T07:06:55.0469898Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0470464Z "size": 106, 2024-06-26T07:06:55.0471023Z "digest": "sha256:ddc9740bcea2b301e99b50525059aea8d9c7121a1c8eab566561a8968438a8c2" 2024-06-26T07:06:55.0471658Z }, 2024-06-26T07:06:55.0471919Z { 2024-06-26T07:06:55.0472358Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0472965Z "size": 1213, 2024-06-26T07:06:55.0473538Z "digest": "sha256:fb2e977685609fcc01de170ba7f50a573e73e6520e47253717e92a28952c5ee8" 2024-06-26T07:06:55.0474165Z }, 2024-06-26T07:06:55.0474418Z { 2024-06-26T07:06:55.0474871Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0475457Z "size": 702, 2024-06-26T07:06:55.0476017Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:06:55.0476646Z }, 2024-06-26T07:06:55.0476903Z { 2024-06-26T07:06:55.0477356Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0477922Z "size": 138, 2024-06-26T07:06:55.0478488Z "digest": "sha256:b599c5a8c7cb461cbd7b3c98b99960b8eb04506e05e79db1cc0cfadb14a47e91" 2024-06-26T07:06:55.0479128Z }, 2024-06-26T07:06:55.0479384Z { 2024-06-26T07:06:55.0479824Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0480396Z "size": 120, 2024-06-26T07:06:55.0480980Z "digest": "sha256:e08e88c1366827455ee7ddbf92b6ae615ceb3cbd3fb11cd03ddd31bd3fa94a9c" 2024-06-26T07:06:55.0481626Z }, 2024-06-26T07:06:55.0481880Z { 2024-06-26T07:06:55.0482329Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0482904Z "size": 1904157495, 2024-06-26T07:06:55.0483496Z "digest": "sha256:12ba7431ff38de8cf6af3052a2a52e79f4e712b8c99f5bafde3c20e63b1f570f" 2024-06-26T07:06:55.0484137Z }, 2024-06-26T07:06:55.0484399Z { 2024-06-26T07:06:55.0484847Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0485412Z "size": 172, 2024-06-26T07:06:55.0485974Z "digest": "sha256:cf7c2cb896cba6a594028cc0fd246e127a2f4481d48905a3aa12c5f5872332d4" 2024-06-26T07:06:55.0486611Z }, 2024-06-26T07:06:55.0487037Z { 2024-06-26T07:06:55.0487474Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0488043Z "size": 908, 2024-06-26T07:06:55.0488608Z "digest": "sha256:3a778ddfac2af5b3a11f49ddf2f3f1ad1731f74ba691f434ab69966d1e2a189f" 2024-06-26T07:06:55.0489364Z }, 2024-06-26T07:06:55.0489616Z { 2024-06-26T07:06:55.0490059Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0490623Z "size": 702, 2024-06-26T07:06:55.0491164Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:06:55.0491787Z }, 2024-06-26T07:06:55.0492035Z { 2024-06-26T07:06:55.0492476Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0493038Z "size": 134, 2024-06-26T07:06:55.0493589Z "digest": "sha256:236c8f0245f4b40616a4322eaf1e1b3e178d5990db1fcaba39622ea4e79c06a1" 2024-06-26T07:06:55.0494209Z }, 2024-06-26T07:06:55.0494458Z { 2024-06-26T07:06:55.0494889Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0495447Z "size": 32, 2024-06-26T07:06:55.0496014Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:06:55.0496639Z }, 2024-06-26T07:06:55.0496907Z { 2024-06-26T07:06:55.0497345Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0497904Z "size": 156, 2024-06-26T07:06:55.0498456Z "digest": "sha256:dda1ce302b9a9df0c8f1145d2e3eaa03c96909bb45a792451f1fcb555b06738d" 2024-06-26T07:06:55.0499091Z }, 2024-06-26T07:06:55.0499352Z { 2024-06-26T07:06:55.0499804Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0500357Z "size": 1841, 2024-06-26T07:06:55.0500987Z "digest": "sha256:e797fe86caef19319e03f8dca92cf96838ab3bb3644f4447c6dfc04230118afa" 2024-06-26T07:06:55.0501613Z }, 2024-06-26T07:06:55.0501858Z { 2024-06-26T07:06:55.0502327Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0502909Z "size": 7529784, 2024-06-26T07:06:55.0503463Z "digest": "sha256:967190076fc0ab2bb2fc0d74e58edd3d662a537d702ff3f8b6e876230154a16f" 2024-06-26T07:06:55.0504079Z }, 2024-06-26T07:06:55.0504331Z { 2024-06-26T07:06:55.0504858Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0505416Z "size": 165, 2024-06-26T07:06:55.0505956Z "digest": "sha256:4fb0a83f06855534ef82b80dcf7c685303edb2bcbd446196066dc2ad2dd71281" 2024-06-26T07:06:55.0506576Z }, 2024-06-26T07:06:55.0506832Z { 2024-06-26T07:06:55.0507270Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0507823Z "size": 7944, 2024-06-26T07:06:55.0508377Z "digest": "sha256:68af2ec4e30a33f26947ece9f3a31b7d3b04c2bca5865d4b48618457832a1da2" 2024-06-26T07:06:55.0509004Z }, 2024-06-26T07:06:55.0509249Z { 2024-06-26T07:06:55.0509686Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0510245Z "size": 8070, 2024-06-26T07:06:55.0510786Z "digest": "sha256:cc91933e985a044c6df53caa15488d2285896f5e991f38ec068c253ceec53f14" 2024-06-26T07:06:55.0511392Z }, 2024-06-26T07:06:55.0511637Z { 2024-06-26T07:06:55.0512078Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0512646Z "size": 303, 2024-06-26T07:06:55.0513187Z "digest": "sha256:9f3187916e7dcad33cc572f8f0e949d2f507b887f99cfd6acefd14826b44960d" 2024-06-26T07:06:55.0513810Z }, 2024-06-26T07:06:55.0514059Z { 2024-06-26T07:06:55.0514490Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0515057Z "size": 7629071, 2024-06-26T07:06:55.0515593Z "digest": "sha256:52608399485738d10d779254946c9e1d536010dd45593c69bf1eb9f62f85dc24" 2024-06-26T07:06:55.0516195Z }, 2024-06-26T07:06:55.0516439Z { 2024-06-26T07:06:55.0516887Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0517455Z "size": 108, 2024-06-26T07:06:55.0518004Z "digest": "sha256:05a6f6dcc1c00bac6660923a22adc7d7c34416718f8c89daf9b3351d9365f694" 2024-06-26T07:06:55.0518623Z }, 2024-06-26T07:06:55.0518876Z { 2024-06-26T07:06:55.0519328Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0519957Z "size": 54145775, 2024-06-26T07:06:55.0520523Z "digest": "sha256:6bc9dc8315b09b27c976bfba9700767ba412d72fd0528b18bbe8f8e62005ec22" 2024-06-26T07:06:55.0521155Z }, 2024-06-26T07:06:55.0521411Z { 2024-06-26T07:06:55.0521850Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0522428Z "size": 474, 2024-06-26T07:06:55.0522978Z "digest": "sha256:1b0e27c35f07b3fbf0aeb548d1949bf1d60a420136da21789d94d196c891d981" 2024-06-26T07:06:55.0523599Z }, 2024-06-26T07:06:55.0523847Z { 2024-06-26T07:06:55.0524282Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0524847Z "size": 1374858747, 2024-06-26T07:06:55.0525410Z "digest": "sha256:95237e7311a1c5ce6937db7196c866632f44e8788514a5c0eb4a3f04da4017e6" 2024-06-26T07:06:55.0526003Z }, 2024-06-26T07:06:55.0526251Z { 2024-06-26T07:06:55.0526693Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0527459Z "size": 106, 2024-06-26T07:06:55.0528031Z "digest": "sha256:789eff6c51aada70bf75eb1acafed63c0d839b768638001500616ff6e53b854a" 2024-06-26T07:06:55.0528681Z }, 2024-06-26T07:06:55.0528939Z { 2024-06-26T07:06:55.0529380Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0529952Z "size": 558, 2024-06-26T07:06:55.0530522Z "digest": "sha256:de0f8a8c45c6077b3c5ba3b84d7f07de98353fd618dad9a1e2f368ffe00493c7" 2024-06-26T07:06:55.0531166Z }, 2024-06-26T07:06:55.0531506Z { 2024-06-26T07:06:55.0531953Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0532534Z "size": 46248513, 2024-06-26T07:06:55.0533100Z "digest": "sha256:2060028f74888d218bebdd10d723f66c33ee1c708935c4ffd4654d34912ac4dc" 2024-06-26T07:06:55.0533716Z }, 2024-06-26T07:06:55.0533975Z { 2024-06-26T07:06:55.0534424Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0534992Z "size": 111, 2024-06-26T07:06:55.0535560Z "digest": "sha256:e412c7f3b2ded440992955bd92512345cbece8a8894cd3592f714ebc758ad1a0" 2024-06-26T07:06:55.0536192Z }, 2024-06-26T07:06:55.0536456Z { 2024-06-26T07:06:55.0536899Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0537473Z "size": 32, 2024-06-26T07:06:55.0538029Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:06:55.0538677Z }, 2024-06-26T07:06:55.0538931Z { 2024-06-26T07:06:55.0539391Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0539963Z "size": 32, 2024-06-26T07:06:55.0540521Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:06:55.0541153Z }, 2024-06-26T07:06:55.0541415Z { 2024-06-26T07:06:55.0541867Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0542436Z "size": 32, 2024-06-26T07:06:55.0542997Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:06:55.0543630Z }, 2024-06-26T07:06:55.0543889Z { 2024-06-26T07:06:55.0544332Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:55.0544969Z "size": 32, 2024-06-26T07:06:55.0545521Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:06:55.0546152Z } 2024-06-26T07:06:55.0546408Z ] 2024-06-26T07:06:55.0546668Z } 2024-06-26T07:06:55.0648364Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:06:55.0648881Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:06:55.0649445Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-26T07:06:55.0657509Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:55.0658022Z env: 2024-06-26T07:06:55.0658305Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:55.0674156Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:55.0675442Z ##[endgroup] 2024-06-26T07:06:55.0699011Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:55.0750691Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T07:06:55.0751298Z with: 2024-06-26T07:06:55.0752410Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:55.0753797Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:55.0754323Z env: 2024-06-26T07:06:55.0754618Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:55.0754984Z ##[endgroup] 2024-06-26T07:06:55.0770659Z ##[group]Run set -x 2024-06-26T07:06:55.0771016Z set -x 2024-06-26T07:06:55.0771322Z set +e 2024-06-26T07:06:55.0771621Z  2024-06-26T07:06:55.0771901Z login() { 2024-06-26T07:06:55.0772583Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:06:55.0773321Z } 2024-06-26T07:06:55.0773601Z  2024-06-26T07:06:55.0773906Z retry () { 2024-06-26T07:06:55.0774310Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:06:55.0774758Z } 2024-06-26T07:06:55.0775038Z  2024-06-26T07:06:55.0775482Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:06:55.0775914Z  2024-06-26T07:06:55.0776197Z set -e 2024-06-26T07:06:55.0776699Z # ignore output since only exit code is used for conditional 2024-06-26T07:06:55.0777447Z # only pull docker image if it's not available locally 2024-06-26T07:06:55.0778255Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T07:06:55.0778986Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T07:06:55.0779457Z fi 2024-06-26T07:06:55.0787041Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:55.0787552Z env: 2024-06-26T07:06:55.0787840Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:55.0789026Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:55.0790361Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:55.0790896Z ##[endgroup] 2024-06-26T07:06:55.0810982Z + set +e 2024-06-26T07:06:55.0811617Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:55.0812277Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:55.0813858Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:06:55.0814653Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:55.5119381Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:06:55.5120470Z Configure a credential helper to remove this warning. See 2024-06-26T07:06:55.5121591Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:06:55.5122321Z 2024-06-26T07:06:55.5122486Z Login Succeeded 2024-06-26T07:06:55.5131056Z + set -e 2024-06-26T07:06:55.5132435Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:55.5250546Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:55.5252817Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:55.7397372Z 91382da70d5719cd7007b6b80b71d2f48398f6b7: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-06-26T07:06:55.7399032Z 7a2c55901189: Pulling fs layer 2024-06-26T07:06:55.7400202Z 224fe954d725: Pulling fs layer 2024-06-26T07:06:55.7401134Z 75722010b82e: Pulling fs layer 2024-06-26T07:06:55.7401809Z d527cbbb87e3: Pulling fs layer 2024-06-26T07:06:55.7402344Z b57676e46aee: Pulling fs layer 2024-06-26T07:06:55.7402885Z a8c1e85b5e14: Pulling fs layer 2024-06-26T07:06:55.7403394Z a41a8d1c11c8: Pulling fs layer 2024-06-26T07:06:55.7403910Z 0c1227890755: Pulling fs layer 2024-06-26T07:06:55.7404417Z d8d1234baab3: Pulling fs layer 2024-06-26T07:06:55.7404932Z 7ed32bc8e469: Pulling fs layer 2024-06-26T07:06:55.7405456Z ec1e7978c1fe: Pulling fs layer 2024-06-26T07:06:55.7405965Z 309b872da805: Pulling fs layer 2024-06-26T07:06:55.7406483Z e82932cb002c: Pulling fs layer 2024-06-26T07:06:55.7407091Z ca9dd73e3e93: Pulling fs layer 2024-06-26T07:06:55.7407486Z 16d76a79d5d6: Pulling fs layer 2024-06-26T07:06:55.7407873Z 4bdf41c9ecbe: Pulling fs layer 2024-06-26T07:06:55.7408270Z f4863801e30c: Pulling fs layer 2024-06-26T07:06:55.7408647Z d527cbbb87e3: Waiting 2024-06-26T07:06:55.7409002Z d37dd9ed8948: Pulling fs layer 2024-06-26T07:06:55.7409462Z cf2f34f225db: Pulling fs layer 2024-06-26T07:06:55.7410063Z df546d7b3e34: Pulling fs layer 2024-06-26T07:06:55.7410589Z 3d914950f8ad: Pulling fs layer 2024-06-26T07:06:55.7411137Z 96d0e2032184: Pulling fs layer 2024-06-26T07:06:55.7411958Z 06e12ce59790: Pulling fs layer 2024-06-26T07:06:55.7412566Z ed712f63cde7: Pulling fs layer 2024-06-26T07:06:55.7413034Z a97a71ef8a4e: Pulling fs layer 2024-06-26T07:06:55.7413448Z 25cd0c482ad6: Pulling fs layer 2024-06-26T07:06:55.7413868Z a41a8d1c11c8: Waiting 2024-06-26T07:06:55.7414215Z 9bf94bc45658: Pulling fs layer 2024-06-26T07:06:55.7414590Z a8c1e85b5e14: Waiting 2024-06-26T07:06:55.7414916Z b57676e46aee: Waiting 2024-06-26T07:06:55.7415262Z de45a14c9b22: Pulling fs layer 2024-06-26T07:06:55.7415633Z 0c1227890755: Waiting 2024-06-26T07:06:55.7415952Z e82932cb002c: Waiting 2024-06-26T07:06:55.7416314Z e99241e77209: Pulling fs layer 2024-06-26T07:06:55.7416747Z 35e8f4e2a42f: Pulling fs layer 2024-06-26T07:06:55.7417117Z cf2f34f225db: Waiting 2024-06-26T07:06:55.7417444Z 309b872da805: Waiting 2024-06-26T07:06:55.7417918Z ca9dd73e3e93: Waiting 2024-06-26T07:06:55.7418453Z a823ffa4c541: Pulling fs layer 2024-06-26T07:06:55.7418971Z ec1e7978c1fe: Waiting 2024-06-26T07:06:55.7419405Z 44ba272eae20: Pulling fs layer 2024-06-26T07:06:55.7419793Z d37dd9ed8948: Waiting 2024-06-26T07:06:55.7420119Z f4863801e30c: Waiting 2024-06-26T07:06:55.7420442Z df546d7b3e34: Waiting 2024-06-26T07:06:55.7420769Z 16d76a79d5d6: Waiting 2024-06-26T07:06:55.7421093Z d8d1234baab3: Waiting 2024-06-26T07:06:55.7421435Z da02d278e524: Pulling fs layer 2024-06-26T07:06:55.7421849Z 73a0ce681c29: Pulling fs layer 2024-06-26T07:06:55.7422284Z 4bdf41c9ecbe: Waiting 2024-06-26T07:06:55.7422619Z 7ed32bc8e469: Waiting 2024-06-26T07:06:55.7422939Z 3d914950f8ad: Waiting 2024-06-26T07:06:55.7423329Z 49737bd5ae70: Pulling fs layer 2024-06-26T07:06:55.7423711Z 9cd4c745f8e4: Pulling fs layer 2024-06-26T07:06:55.7424076Z ed712f63cde7: Waiting 2024-06-26T07:06:55.7424511Z de45a14c9b22: Waiting 2024-06-26T07:06:55.7424981Z b2950a036534: Pulling fs layer 2024-06-26T07:06:55.7425508Z e99241e77209: Waiting 2024-06-26T07:06:55.7425947Z 25cd0c482ad6: Waiting 2024-06-26T07:06:55.7426409Z 25e19d81d9fc: Pulling fs layer 2024-06-26T07:06:55.7426958Z 60156b40fcbe: Pulling fs layer 2024-06-26T07:06:55.7427331Z 9bf94bc45658: Waiting 2024-06-26T07:06:55.7427712Z 7a868239227f: Pulling fs layer 2024-06-26T07:06:55.7428273Z 35e8f4e2a42f: Waiting 2024-06-26T07:06:55.7428704Z 73a0ce681c29: Waiting 2024-06-26T07:06:55.7429140Z a97a71ef8a4e: Waiting 2024-06-26T07:06:55.7429570Z 44ba272eae20: Waiting 2024-06-26T07:06:55.7430010Z 9cd4c745f8e4: Waiting 2024-06-26T07:06:55.7430450Z a823ffa4c541: Waiting 2024-06-26T07:06:55.7430903Z 83911d7c7ecf: Pulling fs layer 2024-06-26T07:06:55.7431405Z 06e12ce59790: Waiting 2024-06-26T07:06:55.7431841Z 49737bd5ae70: Waiting 2024-06-26T07:06:55.7432342Z b3c5d66c679f: Pulling fs layer 2024-06-26T07:06:55.7432885Z 60156b40fcbe: Waiting 2024-06-26T07:06:55.7433354Z 84b2c7a89982: Pulling fs layer 2024-06-26T07:06:55.7434038Z da02d278e524: Waiting 2024-06-26T07:06:55.7434482Z 7a868239227f: Waiting 2024-06-26T07:06:55.7434900Z 83911d7c7ecf: Waiting 2024-06-26T07:06:55.7435340Z b3c5d66c679f: Waiting 2024-06-26T07:06:55.7435801Z c1068d03a811: Pulling fs layer 2024-06-26T07:06:55.7436337Z 671b4a301d81: Pulling fs layer 2024-06-26T07:06:55.7436856Z 25e19d81d9fc: Waiting 2024-06-26T07:06:55.7437325Z 67760cf5ec46: Pulling fs layer 2024-06-26T07:06:55.7437846Z b2950a036534: Waiting 2024-06-26T07:06:55.7438301Z ddc9740bcea2: Pulling fs layer 2024-06-26T07:06:55.7438861Z fb2e97768560: Pulling fs layer 2024-06-26T07:06:55.7439390Z b599c5a8c7cb: Pulling fs layer 2024-06-26T07:06:55.7439921Z e08e88c13668: Pulling fs layer 2024-06-26T07:06:55.7440438Z 12ba7431ff38: Pulling fs layer 2024-06-26T07:06:55.7440990Z cf7c2cb896cb: Pulling fs layer 2024-06-26T07:06:55.7441540Z 3a778ddfac2a: Pulling fs layer 2024-06-26T07:06:55.7442069Z c1068d03a811: Waiting 2024-06-26T07:06:55.7442522Z 236c8f0245f4: Pulling fs layer 2024-06-26T07:06:55.7443065Z 4f4fb700ef54: Pulling fs layer 2024-06-26T07:06:55.7443575Z 671b4a301d81: Waiting 2024-06-26T07:06:55.7444007Z b599c5a8c7cb: Waiting 2024-06-26T07:06:55.7444478Z dda1ce302b9a: Pulling fs layer 2024-06-26T07:06:55.7444995Z 67760cf5ec46: Waiting 2024-06-26T07:06:55.7445528Z 12ba7431ff38: Waiting 2024-06-26T07:06:55.7445993Z e797fe86caef: Pulling fs layer 2024-06-26T07:06:55.7446524Z ddc9740bcea2: Waiting 2024-06-26T07:06:55.7447125Z cf7c2cb896cb: Waiting 2024-06-26T07:06:55.7447594Z 967190076fc0: Pulling fs layer 2024-06-26T07:06:55.7448088Z e08e88c13668: Waiting 2024-06-26T07:06:55.7448533Z 4f4fb700ef54: Waiting 2024-06-26T07:06:55.7448973Z 3a778ddfac2a: Waiting 2024-06-26T07:06:55.7449441Z 4fb0a83f0685: Pulling fs layer 2024-06-26T07:06:55.7449998Z 68af2ec4e30a: Pulling fs layer 2024-06-26T07:06:55.7450545Z cc91933e985a: Pulling fs layer 2024-06-26T07:06:55.7451086Z 9f3187916e7d: Pulling fs layer 2024-06-26T07:06:55.7451588Z 68af2ec4e30a: Waiting 2024-06-26T07:06:55.7452053Z 526083994857: Pulling fs layer 2024-06-26T07:06:55.7452634Z 05a6f6dcc1c0: Pulling fs layer 2024-06-26T07:06:55.7453139Z 9f3187916e7d: Waiting 2024-06-26T07:06:55.7453563Z 967190076fc0: Waiting 2024-06-26T07:06:55.7454027Z 6bc9dc8315b0: Pulling fs layer 2024-06-26T07:06:55.7454582Z 1b0e27c35f07: Pulling fs layer 2024-06-26T07:06:55.7455187Z 95237e7311a1: Pulling fs layer 2024-06-26T07:06:55.7455716Z 789eff6c51aa: Pulling fs layer 2024-06-26T07:06:55.7456243Z de0f8a8c45c6: Pulling fs layer 2024-06-26T07:06:55.7456767Z 2060028f7488: Pulling fs layer 2024-06-26T07:06:55.7457284Z e412c7f3b2de: Pulling fs layer 2024-06-26T07:06:55.7457809Z 05a6f6dcc1c0: Waiting 2024-06-26T07:06:55.7458258Z 95237e7311a1: Waiting 2024-06-26T07:06:55.7458695Z 789eff6c51aa: Waiting 2024-06-26T07:06:55.7459147Z 6bc9dc8315b0: Waiting 2024-06-26T07:06:55.7459596Z 2060028f7488: Waiting 2024-06-26T07:06:55.7460090Z 1b0e27c35f07: Waiting 2024-06-26T07:06:55.7460514Z e412c7f3b2de: Waiting 2024-06-26T07:06:55.7461092Z de0f8a8c45c6: Waiting 2024-06-26T07:06:55.8782078Z 224fe954d725: Verifying Checksum 2024-06-26T07:06:55.8782637Z 224fe954d725: Download complete 2024-06-26T07:06:55.9679107Z d527cbbb87e3: Download complete 2024-06-26T07:06:56.0528313Z b57676e46aee: Verifying Checksum 2024-06-26T07:06:56.0528987Z b57676e46aee: Download complete 2024-06-26T07:06:56.0811133Z 7a2c55901189: Verifying Checksum 2024-06-26T07:06:56.0811710Z 7a2c55901189: Download complete 2024-06-26T07:06:56.1572692Z a41a8d1c11c8: Download complete 2024-06-26T07:06:56.2479968Z 0c1227890755: Verifying Checksum 2024-06-26T07:06:56.2480888Z 0c1227890755: Download complete 2024-06-26T07:06:56.3558919Z d8d1234baab3: Download complete 2024-06-26T07:06:56.3618035Z 75722010b82e: Verifying Checksum 2024-06-26T07:06:56.3618566Z 75722010b82e: Download complete 2024-06-26T07:06:56.4678089Z ec1e7978c1fe: Verifying Checksum 2024-06-26T07:06:56.4678687Z ec1e7978c1fe: Download complete 2024-06-26T07:06:56.5517454Z 309b872da805: Download complete 2024-06-26T07:06:57.1173623Z 7a2c55901189: Pull complete 2024-06-26T07:06:57.4142715Z 224fe954d725: Pull complete 2024-06-26T07:06:58.3335519Z 75722010b82e: Pull complete 2024-06-26T07:06:58.3727582Z d527cbbb87e3: Pull complete 2024-06-26T07:06:58.4119048Z b57676e46aee: Pull complete 2024-06-26T07:06:59.0802114Z e82932cb002c: Verifying Checksum 2024-06-26T07:06:59.0802758Z e82932cb002c: Download complete 2024-06-26T07:06:59.1645834Z ca9dd73e3e93: Verifying Checksum 2024-06-26T07:06:59.1646311Z ca9dd73e3e93: Download complete 2024-06-26T07:06:59.2381838Z 16d76a79d5d6: Verifying Checksum 2024-06-26T07:06:59.2382311Z 16d76a79d5d6: Download complete 2024-06-26T07:06:59.3119462Z 4bdf41c9ecbe: Verifying Checksum 2024-06-26T07:06:59.3120191Z 4bdf41c9ecbe: Download complete 2024-06-26T07:07:00.2852445Z f4863801e30c: Verifying Checksum 2024-06-26T07:07:00.2852952Z f4863801e30c: Download complete 2024-06-26T07:07:00.3596542Z d37dd9ed8948: Download complete 2024-06-26T07:07:00.4541846Z cf2f34f225db: Verifying Checksum 2024-06-26T07:07:00.4542442Z cf2f34f225db: Download complete 2024-06-26T07:07:00.5396643Z df546d7b3e34: Verifying Checksum 2024-06-26T07:07:00.5397118Z df546d7b3e34: Download complete 2024-06-26T07:07:09.4299459Z a8c1e85b5e14: Verifying Checksum 2024-06-26T07:07:09.4299961Z a8c1e85b5e14: Download complete 2024-06-26T07:07:09.5018681Z 96d0e2032184: Verifying Checksum 2024-06-26T07:07:09.5019216Z 96d0e2032184: Download complete 2024-06-26T07:07:09.5844595Z 06e12ce59790: Download complete 2024-06-26T07:07:09.6645168Z ed712f63cde7: Verifying Checksum 2024-06-26T07:07:09.6645678Z ed712f63cde7: Download complete 2024-06-26T07:07:09.7416196Z a97a71ef8a4e: Verifying Checksum 2024-06-26T07:07:09.7416689Z a97a71ef8a4e: Download complete 2024-06-26T07:07:09.8536173Z 25cd0c482ad6: Verifying Checksum 2024-06-26T07:07:09.8536757Z 25cd0c482ad6: Download complete 2024-06-26T07:07:11.1329002Z 9bf94bc45658: Verifying Checksum 2024-06-26T07:07:11.1329498Z 9bf94bc45658: Download complete 2024-06-26T07:07:11.2131652Z de45a14c9b22: Verifying Checksum 2024-06-26T07:07:11.2132258Z de45a14c9b22: Download complete 2024-06-26T07:07:11.2918945Z e99241e77209: Download complete 2024-06-26T07:07:11.3682004Z 35e8f4e2a42f: Verifying Checksum 2024-06-26T07:07:11.3682690Z 35e8f4e2a42f: Download complete 2024-06-26T07:07:11.4582233Z a823ffa4c541: Download complete 2024-06-26T07:07:11.5287660Z 44ba272eae20: Verifying Checksum 2024-06-26T07:07:11.5288096Z 44ba272eae20: Download complete 2024-06-26T07:07:16.0611668Z da02d278e524: Verifying Checksum 2024-06-26T07:07:16.0612248Z da02d278e524: Download complete 2024-06-26T07:07:16.1331217Z 73a0ce681c29: Verifying Checksum 2024-06-26T07:07:16.1331827Z 73a0ce681c29: Download complete 2024-06-26T07:07:16.2045598Z 49737bd5ae70: Verifying Checksum 2024-06-26T07:07:16.2046145Z 49737bd5ae70: Download complete 2024-06-26T07:07:16.6422951Z 9cd4c745f8e4: Verifying Checksum 2024-06-26T07:07:16.6423534Z 9cd4c745f8e4: Download complete 2024-06-26T07:07:16.7033463Z b2950a036534: Download complete 2024-06-26T07:07:16.7735868Z 25e19d81d9fc: Verifying Checksum 2024-06-26T07:07:16.7736521Z 25e19d81d9fc: Download complete 2024-06-26T07:07:17.0182263Z 60156b40fcbe: Verifying Checksum 2024-06-26T07:07:17.0182916Z 60156b40fcbe: Download complete 2024-06-26T07:07:17.1018962Z 7a868239227f: Verifying Checksum 2024-06-26T07:07:17.1019604Z 7a868239227f: Download complete 2024-06-26T07:07:17.1755642Z 83911d7c7ecf: Download complete 2024-06-26T07:07:17.2481381Z b3c5d66c679f: Verifying Checksum 2024-06-26T07:07:17.2481927Z b3c5d66c679f: Download complete 2024-06-26T07:07:21.7054302Z 7ed32bc8e469: Verifying Checksum 2024-06-26T07:07:21.7054946Z 7ed32bc8e469: Download complete 2024-06-26T07:07:21.8379735Z c1068d03a811: Verifying Checksum 2024-06-26T07:07:21.8380231Z c1068d03a811: Download complete 2024-06-26T07:07:21.9339325Z 671b4a301d81: Verifying Checksum 2024-06-26T07:07:21.9339798Z 671b4a301d81: Download complete 2024-06-26T07:07:22.1366509Z a8c1e85b5e14: Pull complete 2024-06-26T07:07:22.3515397Z a41a8d1c11c8: Pull complete 2024-06-26T07:07:22.4356014Z 67760cf5ec46: Verifying Checksum 2024-06-26T07:07:22.4356710Z 67760cf5ec46: Download complete 2024-06-26T07:07:22.4990167Z ddc9740bcea2: Verifying Checksum 2024-06-26T07:07:22.4990799Z ddc9740bcea2: Download complete 2024-06-26T07:07:22.5867702Z fb2e97768560: Verifying Checksum 2024-06-26T07:07:22.5868298Z fb2e97768560: Download complete 2024-06-26T07:07:22.6099097Z 0c1227890755: Pull complete 2024-06-26T07:07:22.6605179Z b599c5a8c7cb: Verifying Checksum 2024-06-26T07:07:22.6605802Z b599c5a8c7cb: Download complete 2024-06-26T07:07:22.7261363Z e08e88c13668: Verifying Checksum 2024-06-26T07:07:22.7261858Z e08e88c13668: Download complete 2024-06-26T07:07:22.8504951Z d8d1234baab3: Pull complete 2024-06-26T07:07:28.4440868Z 3d914950f8ad: Verifying Checksum 2024-06-26T07:07:28.4441466Z 3d914950f8ad: Download complete 2024-06-26T07:07:28.5155310Z cf7c2cb896cb: Verifying Checksum 2024-06-26T07:07:28.5155795Z cf7c2cb896cb: Download complete 2024-06-26T07:07:28.5772098Z 3a778ddfac2a: Download complete 2024-06-26T07:07:28.6375981Z 236c8f0245f4: Verifying Checksum 2024-06-26T07:07:28.6376543Z 236c8f0245f4: Download complete 2024-06-26T07:07:28.6463724Z 4f4fb700ef54: Verifying Checksum 2024-06-26T07:07:28.6464176Z 4f4fb700ef54: Download complete 2024-06-26T07:07:28.7201046Z dda1ce302b9a: Verifying Checksum 2024-06-26T07:07:28.7201849Z dda1ce302b9a: Download complete 2024-06-26T07:07:28.7884280Z e797fe86caef: Verifying Checksum 2024-06-26T07:07:28.7884765Z e797fe86caef: Download complete 2024-06-26T07:07:28.9151253Z 967190076fc0: Verifying Checksum 2024-06-26T07:07:28.9151816Z 967190076fc0: Download complete 2024-06-26T07:07:28.9835731Z 4fb0a83f0685: Verifying Checksum 2024-06-26T07:07:28.9836325Z 4fb0a83f0685: Download complete 2024-06-26T07:07:29.0641345Z 68af2ec4e30a: Verifying Checksum 2024-06-26T07:07:29.0641899Z 68af2ec4e30a: Download complete 2024-06-26T07:07:29.1463770Z cc91933e985a: Download complete 2024-06-26T07:07:29.2108097Z 9f3187916e7d: Verifying Checksum 2024-06-26T07:07:29.2108657Z 9f3187916e7d: Download complete 2024-06-26T07:07:29.3320759Z 526083994857: Verifying Checksum 2024-06-26T07:07:29.4169702Z 526083994857: Download complete 2024-06-26T07:07:29.4170167Z 05a6f6dcc1c0: Download complete 2024-06-26T07:07:30.0082648Z 6bc9dc8315b0: Verifying Checksum 2024-06-26T07:07:30.0083118Z 6bc9dc8315b0: Download complete 2024-06-26T07:07:30.0723162Z 1b0e27c35f07: Verifying Checksum 2024-06-26T07:07:30.0723909Z 1b0e27c35f07: Download complete 2024-06-26T07:07:52.8177933Z 12ba7431ff38: Verifying Checksum 2024-06-26T07:07:52.8179449Z 12ba7431ff38: Download complete 2024-06-26T07:07:52.8978747Z 789eff6c51aa: Verifying Checksum 2024-06-26T07:07:52.8980273Z 789eff6c51aa: Download complete 2024-06-26T07:07:52.9680503Z de0f8a8c45c6: Verifying Checksum 2024-06-26T07:07:52.9680948Z de0f8a8c45c6: Download complete 2024-06-26T07:07:54.3114977Z 2060028f7488: Verifying Checksum 2024-06-26T07:07:54.3115463Z 2060028f7488: Download complete 2024-06-26T07:07:54.3820122Z e412c7f3b2de: Download complete 2024-06-26T07:07:57.9938014Z 95237e7311a1: Verifying Checksum 2024-06-26T07:07:57.9938568Z 95237e7311a1: Download complete 2024-06-26T07:08:20.7980483Z 84b2c7a89982: Verifying Checksum 2024-06-26T07:08:38.2516416Z 7ed32bc8e469: Pull complete 2024-06-26T07:08:38.4881449Z ec1e7978c1fe: Pull complete 2024-06-26T07:08:38.7123854Z 309b872da805: Pull complete 2024-06-26T07:08:46.0528939Z e82932cb002c: Pull complete 2024-06-26T07:08:46.2632091Z ca9dd73e3e93: Pull complete 2024-06-26T07:08:46.4838461Z 16d76a79d5d6: Pull complete 2024-06-26T07:08:46.7041411Z 4bdf41c9ecbe: Pull complete 2024-06-26T07:08:49.1227693Z f4863801e30c: Pull complete 2024-06-26T07:08:49.3290218Z d37dd9ed8948: Pull complete 2024-06-26T07:08:49.5153760Z cf2f34f225db: Pull complete 2024-06-26T07:08:49.7237212Z df546d7b3e34: Pull complete 2024-06-26T07:09:39.0421644Z 3d914950f8ad: Pull complete 2024-06-26T07:09:39.2203657Z 96d0e2032184: Pull complete 2024-06-26T07:09:39.3651971Z 06e12ce59790: Pull complete 2024-06-26T07:09:39.4413165Z ed712f63cde7: Pull complete 2024-06-26T07:09:39.5398772Z a97a71ef8a4e: Pull complete 2024-06-26T07:09:39.6455446Z 25cd0c482ad6: Pull complete 2024-06-26T07:09:43.1337325Z 9bf94bc45658: Pull complete 2024-06-26T07:09:43.2724393Z de45a14c9b22: Pull complete 2024-06-26T07:09:43.3986659Z e99241e77209: Pull complete 2024-06-26T07:09:43.6127640Z 35e8f4e2a42f: Pull complete 2024-06-26T07:09:43.8174283Z a823ffa4c541: Pull complete 2024-06-26T07:09:44.0218349Z 44ba272eae20: Pull complete 2024-06-26T07:09:52.3546657Z da02d278e524: Pull complete 2024-06-26T07:09:52.5457239Z 73a0ce681c29: Pull complete 2024-06-26T07:09:52.7500280Z 49737bd5ae70: Pull complete 2024-06-26T07:09:53.6617590Z 9cd4c745f8e4: Pull complete 2024-06-26T07:09:53.8184535Z b2950a036534: Pull complete 2024-06-26T07:09:53.9878474Z 25e19d81d9fc: Pull complete 2024-06-26T07:09:54.4413828Z 60156b40fcbe: Pull complete 2024-06-26T07:09:54.6086335Z 7a868239227f: Pull complete 2024-06-26T07:09:54.9731380Z 83911d7c7ecf: Pull complete 2024-06-26T07:09:55.1332804Z b3c5d66c679f: Pull complete 2024-06-26T07:10:33.3975453Z 84b2c7a89982: Pull complete 2024-06-26T07:10:33.6119349Z c1068d03a811: Pull complete 2024-06-26T07:10:33.8304133Z 671b4a301d81: Pull complete 2024-06-26T07:10:35.8526683Z 67760cf5ec46: Pull complete 2024-06-26T07:10:35.8836161Z ddc9740bcea2: Pull complete 2024-06-26T07:10:35.9248407Z fb2e97768560: Pull complete 2024-06-26T07:10:36.0010092Z b599c5a8c7cb: Pull complete 2024-06-26T07:10:36.0361639Z e08e88c13668: Pull complete 2024-06-26T07:11:10.6720354Z 12ba7431ff38: Pull complete 2024-06-26T07:11:10.8326460Z cf7c2cb896cb: Pull complete 2024-06-26T07:11:10.9601003Z 3a778ddfac2a: Pull complete 2024-06-26T07:11:11.3268176Z 236c8f0245f4: Pull complete 2024-06-26T07:11:11.5110669Z 4f4fb700ef54: Pull complete 2024-06-26T07:11:11.7438265Z dda1ce302b9a: Pull complete 2024-06-26T07:11:11.9651075Z e797fe86caef: Pull complete 2024-06-26T07:11:12.3413257Z 967190076fc0: Pull complete 2024-06-26T07:11:12.5376817Z 4fb0a83f0685: Pull complete 2024-06-26T07:11:12.7403049Z 68af2ec4e30a: Pull complete 2024-06-26T07:11:12.9472415Z cc91933e985a: Pull complete 2024-06-26T07:11:13.1437159Z 9f3187916e7d: Pull complete 2024-06-26T07:11:14.1775437Z 526083994857: Pull complete 2024-06-26T07:11:14.3544320Z 05a6f6dcc1c0: Pull complete 2024-06-26T07:11:16.4035876Z 6bc9dc8315b0: Pull complete 2024-06-26T07:11:16.5957138Z 1b0e27c35f07: Pull complete 2024-06-26T07:11:30.2507771Z 95237e7311a1: Pull complete 2024-06-26T07:11:30.3010092Z 789eff6c51aa: Pull complete 2024-06-26T07:11:30.3378011Z de0f8a8c45c6: Pull complete 2024-06-26T07:11:31.0473235Z 2060028f7488: Pull complete 2024-06-26T07:11:31.1243428Z e412c7f3b2de: Pull complete 2024-06-26T07:11:31.5286339Z Digest: sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:11:31.5470320Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:11:31.5649703Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:11:31.5706779Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:11:31.5707750Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:11:31.5715856Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:31.5716369Z env: 2024-06-26T07:11:31.5716649Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:31.5717003Z ##[endgroup] 2024-06-26T07:11:31.5899208Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-26T07:11:31.5899769Z with: 2024-06-26T07:11:31.5900097Z driver-version: 550.54.15 2024-06-26T07:11:31.5900466Z env: 2024-06-26T07:11:31.5900741Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:31.5901091Z ##[endgroup] 2024-06-26T07:11:31.6060111Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:11:31.6060688Z with: 2024-06-26T07:11:31.6060975Z timeout_minutes: 10 2024-06-26T07:11:31.6061314Z max_attempts: 3 2024-06-26T07:11:31.6094905Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-26T07:11:31.6127997Z retry_wait_seconds: 10 2024-06-26T07:11:31.6128488Z polling_interval_seconds: 1 2024-06-26T07:11:31.6128880Z warning_on_retry: true 2024-06-26T07:11:31.6129253Z continue_on_error: false 2024-06-26T07:11:31.6129606Z env: 2024-06-26T07:11:31.6129894Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:31.6130265Z DRIVER_VERSION: 550.54.15 2024-06-26T07:11:31.6130637Z ##[endgroup] 2024-06-26T07:11:31.6653442Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-26T07:11:31.6654400Z + pre_install_nvidia_driver_amzn2 2024-06-26T07:11:31.6654930Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-26T07:11:31.9713504Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:11:32.0106449Z No Match for argument: nvidia-driver-latest-dkms 2024-06-26T07:11:32.0343729Z No Packages marked for removal 2024-06-26T07:11:32.0460969Z + install_nvidia_driver_common 2024-06-26T07:11:32.0463596Z + echo 'Before installing NVIDIA driver' 2024-06-26T07:11:32.0464211Z + lspci 2024-06-26T07:11:32.0464778Z Before installing NVIDIA driver 2024-06-26T07:11:32.0545449Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:11:32.0546265Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:11:32.0547257Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:11:32.0548100Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:11:32.0548899Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:11:32.0549711Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:11:32.0550517Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:11:32.0551426Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:11:32.0552054Z + lsmod 2024-06-26T07:11:32.0559538Z Module Size Used by 2024-06-26T07:11:32.0560056Z xt_nat 16384 0 2024-06-26T07:11:32.0560532Z ib_core 266240 0 2024-06-26T07:11:32.0561013Z nvidia_modeset 1339392 0 2024-06-26T07:11:32.0561489Z veth 16384 0 2024-06-26T07:11:32.0561861Z nvidia_uvm 4599808 0 2024-06-26T07:11:32.0562504Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:11:32.0563018Z drm 421888 1 nvidia 2024-06-26T07:11:32.0563489Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:11:32.0564006Z backlight 16384 1 nvidia_modeset 2024-06-26T07:11:32.0564476Z xt_conntrack 16384 1 2024-06-26T07:11:32.0564880Z ipt_MASQUERADE 16384 1 2024-06-26T07:11:32.0565341Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:11:32.0565839Z nf_conntrack_netlink 49152 0 2024-06-26T07:11:32.0566316Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:11:32.0567008Z xfrm_user 45056 1 2024-06-26T07:11:32.0567455Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:11:32.0567884Z iptable_nat 16384 1 2024-06-26T07:11:32.0568274Z nf_conntrack_ipv4 16384 3 2024-06-26T07:11:32.0568718Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:11:32.0569215Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:11:32.0569780Z nf_nat 32768 3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat 2024-06-26T07:11:32.0570976Z nf_conntrack 155648 8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink 2024-06-26T07:11:32.0572032Z xt_addrtype 16384 2 2024-06-26T07:11:32.0572433Z iptable_filter 16384 1 2024-06-26T07:11:32.0572837Z br_netfilter 24576 0 2024-06-26T07:11:32.0573273Z bridge 172032 1 br_netfilter 2024-06-26T07:11:32.0573742Z stp 16384 1 bridge 2024-06-26T07:11:32.0574184Z llc 16384 2 bridge,stp 2024-06-26T07:11:32.0574628Z overlay 86016 0 2024-06-26T07:11:32.0575190Z sunrpc 393216 1 2024-06-26T07:11:32.0575582Z dm_mirror 28672 0 2024-06-26T07:11:32.0575992Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:11:32.0576514Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:11:32.0577055Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:11:32.0577528Z dax 69632 1 dm_mod 2024-06-26T07:11:32.0577952Z crc32_pclmul 16384 0 2024-06-26T07:11:32.0578361Z ghash_clmulni_intel 16384 0 2024-06-26T07:11:32.0578766Z pcbc 16384 0 2024-06-26T07:11:32.0579154Z aesni_intel 188416 0 2024-06-26T07:11:32.0579567Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:11:32.0580043Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:11:32.0580522Z glue_helper 16384 1 aesni_intel 2024-06-26T07:11:32.0581194Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:11:32.0581767Z mousedev 24576 0 2024-06-26T07:11:32.0582197Z evdev 20480 3 2024-06-26T07:11:32.0582575Z psmouse 32768 0 2024-06-26T07:11:32.0582942Z button 16384 0 2024-06-26T07:11:32.0583315Z ena 139264 0 2024-06-26T07:11:32.0583700Z ptp 20480 1 ena 2024-06-26T07:11:32.0584106Z pps_core 20480 1 ptp 2024-06-26T07:11:32.0584631Z crc32c_intel 24576 0 2024-06-26T07:11:32.0585020Z autofs4 49152 2 2024-06-26T07:11:32.0585398Z + modinfo nvidia 2024-06-26T07:11:32.0586115Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:11:32.0586840Z alias: char-major-195-* 2024-06-26T07:11:32.0587245Z version: 550.54.15 2024-06-26T07:11:32.0587606Z supported: external 2024-06-26T07:11:32.0587964Z license: NVIDIA 2024-06-26T07:11:32.0588356Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:11:32.0588867Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:11:32.0589355Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:11:32.0589838Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:11:32.0590377Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:11:32.0591015Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:11:32.0591532Z depends: i2c-core,drm 2024-06-26T07:11:32.0591905Z retpoline: Y 2024-06-26T07:11:32.0592233Z name: nvidia 2024-06-26T07:11:32.0592815Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:11:32.0593506Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:11:32.0594173Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:11:32.0594789Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:11:32.0595246Z parm: NVreg_RmLogonRC:int 2024-06-26T07:11:32.0595697Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:11:32.0596168Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:11:32.0596621Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:11:32.0597081Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:11:32.0597635Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:11:32.0598222Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:11:32.0598724Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:11:32.0599173Z parm: NVreg_EnableMSI:int 2024-06-26T07:11:32.0599601Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:11:32.0600074Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:11:32.0600628Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:11:32.0601237Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:11:32.0601823Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:11:32.0602454Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:11:32.0603080Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:11:32.0603807Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:11:32.0604425Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:11:32.0604928Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:11:32.0605495Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:11:32.0606067Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:11:32.0606578Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:11:32.0607385Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:11:32.0607895Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:11:32.0608396Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:11:32.0608861Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:11:32.0609393Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:11:32.0609949Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:11:32.0610505Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:11:32.0611021Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:11:32.0611552Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:11:32.0612063Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:11:32.0612599Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:11:32.0613102Z parm: NVreg_RmMsg:charp 2024-06-26T07:11:32.0613534Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:11:32.0614027Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:11:32.0614520Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:11:32.0614996Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:11:32.0615502Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:11:32.0616005Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:11:32.0616476Z parm: rm_firmware_active:charp 2024-06-26T07:11:32.0616918Z + HAS_NVIDIA_DRIVER=0 2024-06-26T07:11:32.0617333Z ++ command -v nvidia-smi 2024-06-26T07:11:32.0617741Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-26T07:11:32.0618121Z + set +e 2024-06-26T07:11:32.0618650Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-26T07:11:34.0861553Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-26T07:11:34.0862122Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:11:34.0862954Z + '[' 0 -ne 0 ']' 2024-06-26T07:11:34.0863357Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-26T07:11:34.0863752Z + HAS_NVIDIA_DRIVER=1 2024-06-26T07:11:34.0864593Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-26T07:11:34.0865341Z + set -e 2024-06-26T07:11:34.0865658Z + '[' 1 -eq 0 ']' 2024-06-26T07:11:34.0866268Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-26T07:11:34.0866992Z + post_install_nvidia_driver_common 2024-06-26T07:11:34.0867424Z + sudo modprobe nvidia 2024-06-26T07:11:34.0962321Z + echo 'After installing NVIDIA driver' 2024-06-26T07:11:34.0962804Z + lspci 2024-06-26T07:11:34.0963137Z After installing NVIDIA driver 2024-06-26T07:11:34.1040250Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:11:34.1041315Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:11:34.1042323Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:11:34.1043173Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:11:34.1043980Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:11:34.1044790Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:11:34.1045572Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:11:34.1046403Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:11:34.1047394Z + lsmod 2024-06-26T07:11:34.1053951Z Module Size Used by 2024-06-26T07:11:34.1054554Z xt_nat 16384 0 2024-06-26T07:11:34.1055538Z ib_core 266240 0 2024-06-26T07:11:34.1056063Z nvidia_modeset 1339392 0 2024-06-26T07:11:34.1056542Z veth 16384 0 2024-06-26T07:11:34.1057023Z nvidia_uvm 4599808 0 2024-06-26T07:11:34.1057617Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:11:34.1058139Z drm 421888 1 nvidia 2024-06-26T07:11:34.1058587Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:11:34.1059082Z backlight 16384 1 nvidia_modeset 2024-06-26T07:11:34.1059546Z xt_conntrack 16384 1 2024-06-26T07:11:34.1059971Z ipt_MASQUERADE 16384 1 2024-06-26T07:11:34.1060440Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:11:34.1060938Z nf_conntrack_netlink 49152 0 2024-06-26T07:11:34.1061404Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:11:34.1061902Z xfrm_user 45056 1 2024-06-26T07:11:34.1062320Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:11:34.1062770Z iptable_nat 16384 1 2024-06-26T07:11:34.1063253Z nf_conntrack_ipv4 16384 3 2024-06-26T07:11:34.1063779Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:11:34.1064390Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:11:34.1066204Z nf_nat 32768 3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat 2024-06-26T07:11:34.1067246Z nf_conntrack 155648 8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink 2024-06-26T07:11:34.1068167Z xt_addrtype 16384 2 2024-06-26T07:11:34.1068562Z iptable_filter 16384 1 2024-06-26T07:11:34.1068952Z br_netfilter 24576 0 2024-06-26T07:11:34.1069365Z bridge 172032 1 br_netfilter 2024-06-26T07:11:34.1069819Z stp 16384 1 bridge 2024-06-26T07:11:34.1070279Z llc 16384 2 bridge,stp 2024-06-26T07:11:34.1070821Z overlay 86016 0 2024-06-26T07:11:34.1071200Z sunrpc 393216 1 2024-06-26T07:11:34.1071598Z dm_mirror 28672 0 2024-06-26T07:11:34.1072053Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:11:34.1072570Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:11:34.1073258Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:11:34.1073734Z dax 69632 1 dm_mod 2024-06-26T07:11:34.1074157Z crc32_pclmul 16384 0 2024-06-26T07:11:34.1074551Z ghash_clmulni_intel 16384 0 2024-06-26T07:11:34.1074951Z pcbc 16384 0 2024-06-26T07:11:34.1075341Z aesni_intel 188416 0 2024-06-26T07:11:34.1075752Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:11:34.1076216Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:11:34.1076687Z glue_helper 16384 1 aesni_intel 2024-06-26T07:11:34.1077269Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:11:34.1077836Z mousedev 24576 0 2024-06-26T07:11:34.1078222Z evdev 20480 3 2024-06-26T07:11:34.1078610Z psmouse 32768 0 2024-06-26T07:11:34.1078995Z button 16384 0 2024-06-26T07:11:34.1079375Z ena 139264 0 2024-06-26T07:11:34.1079757Z ptp 20480 1 ena 2024-06-26T07:11:34.1080180Z pps_core 20480 1 ptp 2024-06-26T07:11:34.1080637Z crc32c_intel 24576 0 2024-06-26T07:11:34.1081018Z autofs4 49152 2 2024-06-26T07:11:34.1081396Z + modinfo nvidia 2024-06-26T07:11:34.1082125Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:11:34.1082855Z alias: char-major-195-* 2024-06-26T07:11:34.1083257Z version: 550.54.15 2024-06-26T07:11:34.1083616Z supported: external 2024-06-26T07:11:34.1083975Z license: NVIDIA 2024-06-26T07:11:34.1084373Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:11:34.1084874Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:11:34.1085432Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:11:34.1085918Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:11:34.1086441Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:11:34.1087151Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:11:34.1087671Z depends: i2c-core,drm 2024-06-26T07:11:34.1088053Z retpoline: Y 2024-06-26T07:11:34.1088383Z name: nvidia 2024-06-26T07:11:34.1088948Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:11:34.1089637Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:11:34.1090319Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:11:34.1090946Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:11:34.1091411Z parm: NVreg_RmLogonRC:int 2024-06-26T07:11:34.1091870Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:11:34.1092355Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:11:34.1092825Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:11:34.1093281Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:11:34.1093838Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:11:34.1094434Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:11:34.1094938Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:11:34.1095397Z parm: NVreg_EnableMSI:int 2024-06-26T07:11:34.1095842Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:11:34.1096327Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:11:34.1096879Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:11:34.1097500Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:11:34.1098091Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:11:34.1098734Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:11:34.1099356Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:11:34.1100009Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:11:34.1100686Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:11:34.1101214Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:11:34.1101855Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:11:34.1102443Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:11:34.1102977Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:11:34.1103472Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:11:34.1103978Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:11:34.1104538Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:11:34.1105016Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:11:34.1105555Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:11:34.1106107Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:11:34.1106621Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:11:34.1107151Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:11:34.1107676Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:11:34.1108193Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:11:34.1108736Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:11:34.1109254Z parm: NVreg_RmMsg:charp 2024-06-26T07:11:34.1109685Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:11:34.1110179Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:11:34.1110690Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:11:34.1111182Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:11:34.1111681Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:11:34.1112184Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:11:34.1112667Z parm: rm_firmware_active:charp 2024-06-26T07:11:34.1113088Z + set +e 2024-06-26T07:11:34.1113393Z + nvidia-smi 2024-06-26T07:11:35.6732033Z Wed Jun 26 07:11:35 2024 2024-06-26T07:11:35.6733259Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:11:35.6734236Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-26T07:11:35.6735073Z |-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:11:35.6735889Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-26T07:11:35.6736824Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-26T07:11:35.6737545Z | | | MIG M. | 2024-06-26T07:11:35.6738120Z |=========================================+========================+======================| 2024-06-26T07:11:35.6865060Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-06-26T07:11:35.6865810Z | 0% 32C P0 56W / 300W | 0MiB / 23028MiB | 0% Default | 2024-06-26T07:11:35.6866487Z | | | N/A | 2024-06-26T07:11:35.6867233Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:11:35.6867944Z 2024-06-26T07:11:35.6868615Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:11:35.6869269Z | Processes: | 2024-06-26T07:11:35.6870006Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-26T07:11:35.6870731Z | ID ID Usage | 2024-06-26T07:11:35.6871323Z |=========================================================================================| 2024-06-26T07:11:35.6871990Z | No running processes found | 2024-06-26T07:11:35.6872768Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:11:36.2893732Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-26T07:11:37.8570976Z NVIDIA A10G 2024-06-26T07:11:38.3000503Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:11:38.3001130Z + '[' 0 -eq 0 ']' 2024-06-26T07:11:38.3001537Z + echo 'INFO: Ignoring allowed status 0' 2024-06-26T07:11:38.3001982Z + set -e 2024-06-26T07:11:38.3002287Z INFO: Ignoring allowed status 0 2024-06-26T07:11:38.3004861Z == Installing nvidia container toolkit for amzn2 == 2024-06-26T07:11:38.3007302Z + sudo yum install -y yum-utils 2024-06-26T07:11:38.6031391Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:11:40.2529053Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-26T07:11:40.2529940Z Nothing to do 2024-06-26T07:11:40.4064010Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-06-26T07:11:40.4065046Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:11:40.4066161Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:11:40.7306431Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:11:40.7564446Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:11:40.7567220Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:11:40.7569216Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:11:40.7682551Z + sudo yum install -y nvidia-docker2 2024-06-26T07:11:41.0696059Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:11:42.5687407Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-26T07:11:42.5708302Z Nothing to do 2024-06-26T07:11:42.7171254Z + sudo systemctl restart docker 2024-06-26T07:12:19.7135778Z Command completed after 1 attempt(s). 2024-06-26T07:12:19.7196643Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:12:19.7197453Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:12:19.7198172Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T07:12:19.7198852Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:12:19.7207020Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:19.7207570Z env: 2024-06-26T07:12:19.7207866Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:19.7208353Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:19.7208858Z ##[endgroup] 2024-06-26T07:12:19.9412304Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:12:19.9562886Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-06-26T07:12:19.9661985Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-06-26T07:12:20.0580893Z Prepare all required actions 2024-06-26T07:12:20.0581408Z Getting action download info 2024-06-26T07:12:20.1695969Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T07:12:20.3189041Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T07:12:20.4503486Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T07:12:20.4503988Z with: 2024-06-26T07:12:20.4504341Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:12:20.4504901Z s3-bucket: gha-artifacts 2024-06-26T07:12:20.4505242Z env: 2024-06-26T07:12:20.4505525Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:20.4505998Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:20.4506507Z ##[endgroup] 2024-06-26T07:12:20.4539146Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:12:20.4539601Z with: 2024-06-26T07:12:20.4539965Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:12:20.4540438Z s3-bucket: gha-artifacts 2024-06-26T07:12:20.4540801Z region: us-east-1 2024-06-26T07:12:20.4541106Z env: 2024-06-26T07:12:20.4541381Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:20.4541849Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:20.4542345Z ##[endgroup] 2024-06-26T07:12:20.8691375Z (node:21319) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:12:20.8692173Z 2024-06-26T07:12:20.8692459Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:12:20.8693228Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:12:20.8694342Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:12:20.9707969Z Found 1 objects with prefix pytorch/pytorch/9673646806/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-06-26T07:12:20.9710216Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:12:28.4564931Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:12:28.4571289Z Artifact download has finished successfully 2024-06-26T07:12:28.4720005Z ##[group]Run unzip -o artifacts.zip 2024-06-26T07:12:28.4720472Z unzip -o artifacts.zip 2024-06-26T07:12:28.4728428Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:28.4728948Z env: 2024-06-26T07:12:28.4729231Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:28.4729703Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:28.4730199Z ##[endgroup] 2024-06-26T07:12:28.4756832Z Archive: artifacts.zip 2024-06-26T07:12:28.4759007Z creating: dist/ 2024-06-26T07:12:30.4493889Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:12:30.4494890Z creating: build/custom_test_artifacts/ 2024-06-26T07:12:30.4495522Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T07:12:30.4496278Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T07:12:30.4497155Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:12:30.4501510Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:12:30.4502484Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T07:12:30.4503472Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:12:30.4504611Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:12:30.4505623Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:12:30.4506773Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:12:30.4508220Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:12:30.4509281Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:12:30.4510382Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:12:30.4511641Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:12:30.4512894Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:12:30.4514202Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:12:30.4515399Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:12:30.4516673Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:12:30.4518023Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:12:30.4519092Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:12:30.4520143Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:12:30.4557347Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:12:30.4595184Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:12:30.4596709Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:12:30.4639612Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:12:30.4641229Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:12:30.4642877Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:12:30.4644555Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:12:30.4646118Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:12:30.4648066Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:12:30.4649648Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:12:30.4651344Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:12:30.4652766Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:12:30.4654116Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:12:30.4655413Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:12:30.4656685Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:12:30.4657972Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:12:30.4659221Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:12:30.4660501Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:12:30.4709719Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:12:30.4768206Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:12:30.4769544Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:12:30.4770779Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:12:30.4771769Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:12:30.4772754Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T07:12:30.4773850Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T07:12:30.4775041Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T07:12:30.4776386Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T07:12:30.4777563Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T07:12:30.4778653Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T07:12:30.4779767Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T07:12:30.4780890Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T07:12:30.4782024Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T07:12:30.4783147Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T07:12:30.4784247Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T07:12:30.4796953Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T07:12:30.4924505Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T07:12:30.4925636Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T07:12:30.4927144Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T07:12:30.4928565Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T07:12:30.4929923Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T07:12:30.4931132Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T07:12:30.4932318Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T07:12:30.4933669Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T07:12:30.4934883Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T07:12:30.4936075Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T07:12:30.4937245Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T07:12:30.4951003Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T07:12:30.5030327Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T07:12:30.5031696Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:12:30.5032915Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:12:30.5034102Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T07:12:30.5035056Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T07:12:30.5035989Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T07:12:30.5036984Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-26T07:12:30.5037924Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T07:12:30.5038803Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T07:12:30.5039647Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T07:12:30.5146267Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T07:12:30.5207183Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T07:12:30.5208016Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T07:12:30.5208752Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T07:12:30.5209604Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:12:30.5214798Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:12:30.5215859Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T07:12:30.5216829Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:12:30.5217846Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:12:30.5218844Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:12:30.5219970Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:12:30.5221113Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:12:30.5222143Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:12:30.5223166Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:12:30.5224475Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:12:30.5225672Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:12:30.5226834Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:12:30.5227998Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:12:30.5229185Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:12:30.5230527Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:12:30.5231576Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:12:30.5232601Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:12:30.5269399Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:12:30.5307259Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:12:30.5308756Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:12:30.5352552Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:12:30.5354586Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:12:30.5356165Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:12:30.5357698Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:12:30.5359174Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:12:30.5360599Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:12:30.5362056Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:12:30.5363494Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:12:30.5364959Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:12:30.5366285Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:12:30.5368769Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:12:30.5370104Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:12:30.5371399Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:12:30.5372649Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:12:30.5374005Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:12:30.5421958Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:12:30.5479903Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:12:30.5481156Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:12:30.5482189Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:12:30.5483082Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:12:30.5484024Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T07:12:30.5485017Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T07:12:30.5486131Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T07:12:30.5487864Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T07:12:30.5489073Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T07:12:30.5490190Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T07:12:30.5491357Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T07:12:30.5492524Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T07:12:30.5493694Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T07:12:30.5494898Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T07:12:30.5496047Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T07:12:30.5508323Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T07:12:30.5571036Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T07:12:30.5573323Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:12:30.5574989Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:12:30.5575997Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T07:12:30.5576936Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T07:12:30.5577865Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T07:12:30.5578809Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-26T07:12:30.5579686Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T07:12:30.5580488Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T07:12:30.5581322Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T07:12:30.5625462Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T07:12:30.5626246Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T07:12:30.5627205Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T07:12:30.5628128Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:12:30.5633204Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:12:30.5634246Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T07:12:30.5635273Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:12:30.5636371Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:12:30.5637451Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:12:30.5638655Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:12:30.5639879Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:12:30.5640970Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:12:30.5642056Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:12:30.5643312Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:12:30.5644629Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:12:30.5646001Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:12:30.5647377Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:12:30.5648650Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:12:30.5649917Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:12:30.5651049Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:12:30.5652154Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:12:30.5688762Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:12:30.5727020Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:12:30.5730366Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:12:30.5770284Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:12:30.5771854Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:12:30.5773438Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:12:30.5775054Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:12:30.5776602Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:12:30.5778133Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:12:30.5779640Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:12:30.5781136Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:12:30.5782594Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:12:30.5783994Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:12:30.5800414Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:12:30.5801859Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:12:30.5803234Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:12:30.5804549Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:12:30.5805893Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:12:30.5840186Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:12:30.5898240Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:12:30.5899558Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:12:30.5900819Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:12:30.5901794Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:12:30.5902815Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T07:12:30.5903880Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T07:12:30.5905117Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T07:12:30.5906429Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T07:12:30.5907691Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T07:12:30.5908871Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T07:12:30.5910113Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T07:12:30.5911452Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T07:12:30.5912695Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T07:12:30.5913936Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T07:12:30.5915154Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T07:12:30.5916453Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T07:12:30.6030996Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T07:12:30.6032231Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T07:12:30.6033500Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T07:12:30.6034931Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T07:12:30.6036261Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T07:12:30.6037539Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T07:12:30.6038836Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T07:12:30.6040145Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T07:12:30.6041444Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T07:12:30.6042752Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T07:12:30.6044044Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T07:12:30.6057105Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T07:12:30.6110143Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T07:12:30.6111503Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:12:30.6112711Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:12:30.6113804Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T07:12:30.6114961Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T07:12:30.6115977Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T07:12:30.6116975Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-26T07:12:30.6117930Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T07:12:30.6118803Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T07:12:30.6119699Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T07:12:30.6217628Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T07:12:30.6258767Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T07:12:30.6259431Z creating: build/lib/ 2024-06-26T07:12:30.6344457Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T07:12:30.6783431Z inflating: build/lib/libprotobuf.a 2024-06-26T07:12:30.6793440Z inflating: build/lib/libpthreadpool.a 2024-06-26T07:12:30.6801192Z inflating: build/lib/libcpuinfo.a 2024-06-26T07:12:30.6809398Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T07:12:30.6810047Z inflating: build/lib/libclog.a 2024-06-26T07:12:30.6828013Z inflating: build/lib/libnnpack.a 2024-06-26T07:12:30.6830087Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T07:12:30.6892435Z inflating: build/lib/libgtest.a 2024-06-26T07:12:30.6965130Z inflating: build/lib/libbenchmark.a 2024-06-26T07:12:30.7026799Z inflating: build/lib/libasmjit.a 2024-06-26T07:12:30.7034523Z inflating: build/lib/libittnotify.a 2024-06-26T07:12:30.7061043Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T07:12:30.7183345Z inflating: build/lib/libgloo.a 2024-06-26T07:12:30.7204722Z inflating: build/lib/libfmt.a 2024-06-26T07:12:30.7293568Z inflating: build/lib/libc10.so 2024-06-26T07:12:30.7295357Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T07:12:30.7296362Z inflating: build/lib/libfoxi_loader.a 2024-06-26T07:12:30.7297716Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T07:12:30.7316456Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T07:12:30.7333953Z inflating: build/lib/libgmock.a 2024-06-26T07:12:30.7334518Z inflating: build/lib/libgtest_main.a 2024-06-26T07:12:30.7335352Z inflating: build/lib/libbenchmark_main.a 2024-06-26T07:12:30.7820023Z inflating: build/lib/libprotoc.a 2024-06-26T07:12:30.8190420Z inflating: build/lib/libgloo_cuda.a 2024-06-26T07:12:31.7918563Z inflating: build/lib/libdnnl.a 2024-06-26T07:12:31.8467481Z inflating: build/lib/libtensorpipe.a 2024-06-26T07:12:31.8520085Z inflating: build/lib/libc10_cuda.so 2024-06-26T07:12:31.8520821Z inflating: build/lib/libgmock_main.a 2024-06-26T07:12:31.9725176Z inflating: build/lib/libfbgemm.a 2024-06-26T07:12:31.9969673Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-26T07:12:32.0469722Z inflating: build/lib/libkineto.a 2024-06-26T07:12:32.0648188Z inflating: build/lib/libXNNPACK.a 2024-06-26T07:12:32.0688973Z inflating: build/lib/libonnx_proto.a 2024-06-26T07:12:32.1353127Z inflating: build/lib/libonnx.a 2024-06-26T07:12:34.4633759Z inflating: build/lib/libtorch_cpu.so 2024-06-26T07:12:34.4638116Z inflating: build/lib/libunbox_lib.a 2024-06-26T07:12:34.4642166Z inflating: build/lib/libshm.so 2024-06-26T07:12:36.4496619Z inflating: build/lib/libtorch_cuda.so 2024-06-26T07:12:36.4497442Z inflating: build/lib/libtorch.so 2024-06-26T07:12:37.2438348Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-26T07:12:37.2440404Z inflating: build/lib/libc10d_cuda_test.so 2024-06-26T07:12:37.4338501Z inflating: build/lib/libtorch_python.so 2024-06-26T07:12:37.4405052Z inflating: build/lib/libtorchbind_test.so 2024-06-26T07:12:37.4425417Z inflating: build/lib/libjitbackend_test.so 2024-06-26T07:12:37.4450264Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T07:12:37.4483726Z inflating: build/lib/libnnapi_backend.so 2024-06-26T07:12:37.4484383Z creating: build/bin/ 2024-06-26T07:12:37.4533407Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T07:12:37.4583432Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T07:12:37.4633120Z inflating: build/bin/c10_Device_test 2024-06-26T07:12:37.4690910Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T07:12:37.4742473Z inflating: build/bin/c10_Scalar_test 2024-06-26T07:12:37.4790877Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T07:12:37.4843268Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T07:12:37.4893054Z inflating: build/bin/c10_SymInt_test 2024-06-26T07:12:37.4947470Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T07:12:37.5002078Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T07:12:37.5053667Z inflating: build/bin/c10_Bitset_test 2024-06-26T07:12:37.5122757Z inflating: build/bin/c10_cow_test 2024-06-26T07:12:37.5170528Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T07:12:37.5219151Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T07:12:37.5268333Z inflating: build/bin/c10_Half_test 2024-06-26T07:12:37.5322978Z inflating: build/bin/c10_LeftRight_test 2024-06-26T07:12:37.5376695Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T07:12:37.5425508Z inflating: build/bin/c10_Synchronized_test 2024-06-26T07:12:37.5479629Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T07:12:37.5530097Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T07:12:37.5579727Z inflating: build/bin/c10_TypeList_test 2024-06-26T07:12:37.5627391Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T07:12:37.5677720Z inflating: build/bin/c10_accumulate_test 2024-06-26T07:12:37.5731725Z inflating: build/bin/c10_bfloat16_test 2024-06-26T07:12:37.5780923Z inflating: build/bin/c10_bit_cast_test 2024-06-26T07:12:37.5836077Z inflating: build/bin/c10_complex_math_test 2024-06-26T07:12:37.5890285Z inflating: build/bin/c10_complex_test 2024-06-26T07:12:37.5941488Z inflating: build/bin/c10_exception_test 2024-06-26T07:12:37.5991060Z inflating: build/bin/c10_flags_test 2024-06-26T07:12:37.6039314Z inflating: build/bin/c10_generic_math_test 2024-06-26T07:12:37.6199489Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T07:12:37.6248924Z inflating: build/bin/c10_irange_test 2024-06-26T07:12:37.6301088Z inflating: build/bin/c10_lazy_test 2024-06-26T07:12:37.6356884Z inflating: build/bin/c10_logging_test 2024-06-26T07:12:37.6431057Z inflating: build/bin/c10_optional_test 2024-06-26T07:12:37.6491676Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T07:12:37.6544056Z inflating: build/bin/c10_registry_test 2024-06-26T07:12:37.6690042Z inflating: build/bin/c10_small_vector_test 2024-06-26T07:12:37.6740327Z inflating: build/bin/c10_ssize_test 2024-06-26T07:12:37.6791037Z inflating: build/bin/c10_string_util_test 2024-06-26T07:12:37.6850111Z inflating: build/bin/c10_string_view_test 2024-06-26T07:12:37.6899425Z inflating: build/bin/c10_tempfile_test 2024-06-26T07:12:37.6946833Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T07:12:37.7001300Z inflating: build/bin/c10_typeid_test 2024-06-26T07:12:37.7435568Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T07:12:37.7867604Z inflating: build/bin/protoc 2024-06-26T07:12:37.7919302Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-26T07:12:37.7970980Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-26T07:12:37.8022294Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-26T07:12:37.8072915Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-26T07:12:37.8124378Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T07:12:37.8176161Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T07:12:37.8223501Z inflating: build/bin/c10_cuda_CUDATest 2024-06-26T07:12:37.8275556Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-26T07:12:37.8594312Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T07:12:37.8927893Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T07:12:37.9274663Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T07:12:37.9325968Z inflating: build/bin/FileStoreTest 2024-06-26T07:12:37.9380202Z inflating: build/bin/TCPStoreTest 2024-06-26T07:12:37.9432508Z inflating: build/bin/HashStoreTest 2024-06-26T07:12:37.9445243Z inflating: build/bin/ProcessGroupMPITest 2024-06-26T07:12:37.9497509Z inflating: build/bin/test_edge_op_registration 2024-06-26T07:12:37.9501683Z inflating: build/bin/torch_shm_manager 2024-06-26T07:12:37.9504574Z inflating: build/bin/example_allreduce 2024-06-26T07:12:37.9558052Z inflating: build/bin/test_dist_autograd 2024-06-26T07:12:37.9625744Z inflating: build/bin/test_cpp_rpc 2024-06-26T07:12:37.9628139Z inflating: build/bin/parallel_benchmark 2024-06-26T07:12:37.9694093Z inflating: build/bin/test_mobile_nnc 2024-06-26T07:12:37.9702611Z inflating: build/bin/aot_model_compiler_test 2024-06-26T07:12:38.0035188Z inflating: build/bin/test_lazy 2024-06-26T07:12:38.1178273Z inflating: build/bin/test_api 2024-06-26T07:12:38.1249792Z inflating: build/bin/Dict_test 2024-06-26T07:12:38.1300434Z inflating: build/bin/Dimname_test 2024-06-26T07:12:38.1363117Z inflating: build/bin/MaybeOwned_test 2024-06-26T07:12:38.1419351Z inflating: build/bin/NamedTensor_test 2024-06-26T07:12:38.1476795Z inflating: build/bin/apply_utils_test 2024-06-26T07:12:38.1534463Z inflating: build/bin/atest 2024-06-26T07:12:38.1595758Z inflating: build/bin/basic 2024-06-26T07:12:38.1648904Z inflating: build/bin/broadcast_test 2024-06-26T07:12:38.1698130Z inflating: build/bin/cpu_allocator_test 2024-06-26T07:12:38.1755123Z inflating: build/bin/cpu_generator_test 2024-06-26T07:12:38.1806210Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T07:12:38.1897217Z inflating: build/bin/cpu_rng_test 2024-06-26T07:12:38.1945601Z inflating: build/bin/dispatch_key_set_test 2024-06-26T07:12:38.1994434Z inflating: build/bin/dlconvertor_test 2024-06-26T07:12:38.2051093Z inflating: build/bin/extension_backend_test 2024-06-26T07:12:38.2104548Z inflating: build/bin/half_test 2024-06-26T07:12:38.2197655Z inflating: build/bin/ivalue_test 2024-06-26T07:12:38.2245913Z inflating: build/bin/lazy_tensor_test 2024-06-26T07:12:38.2299024Z inflating: build/bin/math_kernel_test 2024-06-26T07:12:38.2351694Z inflating: build/bin/memory_format_test 2024-06-26T07:12:38.2403160Z inflating: build/bin/memory_overlapping_test 2024-06-26T07:12:38.2455282Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T07:12:38.2509329Z inflating: build/bin/native_test 2024-06-26T07:12:38.2558780Z inflating: build/bin/operator_name_test 2024-06-26T07:12:38.2608686Z inflating: build/bin/operators_test 2024-06-26T07:12:38.2658903Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T07:12:38.2724100Z inflating: build/bin/pow_test 2024-06-26T07:12:38.2780711Z inflating: build/bin/quantized_test 2024-06-26T07:12:38.2829551Z inflating: build/bin/reduce_ops_test 2024-06-26T07:12:38.2878950Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T07:12:38.2933848Z inflating: build/bin/scalar_tensor_test 2024-06-26T07:12:38.2990941Z inflating: build/bin/scalar_test 2024-06-26T07:12:38.3041302Z inflating: build/bin/StorageUtils_test 2024-06-26T07:12:38.3092922Z inflating: build/bin/stride_properties_test 2024-06-26T07:12:38.3169558Z inflating: build/bin/tensor_iterator_test 2024-06-26T07:12:38.3222062Z inflating: build/bin/test_parallel 2024-06-26T07:12:38.3225048Z inflating: build/bin/thread_init_test 2024-06-26T07:12:38.3279370Z inflating: build/bin/type_ptr_test 2024-06-26T07:12:38.3337676Z inflating: build/bin/type_test 2024-06-26T07:12:38.3388361Z inflating: build/bin/undefined_tensor_test 2024-06-26T07:12:38.3389960Z inflating: build/bin/verify_api_visibility 2024-06-26T07:12:38.3457304Z inflating: build/bin/legacy_vmap_test 2024-06-26T07:12:38.3507774Z inflating: build/bin/weakref_test 2024-06-26T07:12:38.3557922Z inflating: build/bin/wrapdim_test 2024-06-26T07:12:38.3608149Z inflating: build/bin/xla_tensor_test 2024-06-26T07:12:38.3666431Z inflating: build/bin/IListRef_test 2024-06-26T07:12:38.3769751Z inflating: build/bin/List_test 2024-06-26T07:12:38.3834118Z inflating: build/bin/KernelFunction_test 2024-06-26T07:12:38.3952210Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T07:12:38.4046552Z inflating: build/bin/kernel_function_test 2024-06-26T07:12:38.4170303Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T07:12:38.4272026Z inflating: build/bin/kernel_lambda_test 2024-06-26T07:12:38.4331303Z inflating: build/bin/kernel_stackbased_test 2024-06-26T07:12:38.4424605Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T07:12:38.4474138Z inflating: build/bin/CppSignature_test 2024-06-26T07:12:38.4528445Z inflating: build/bin/backend_fallback_test 2024-06-26T07:12:38.4576325Z inflating: build/bin/op_allowlist_test 2024-06-26T07:12:38.4868442Z inflating: build/bin/op_registration_test 2024-06-26T07:12:38.4930052Z inflating: build/bin/inline_container_test 2024-06-26T07:12:38.4981004Z inflating: build/bin/cuda_apply_test 2024-06-26T07:12:38.5031064Z inflating: build/bin/cuda_allocator_test 2024-06-26T07:12:38.5088180Z inflating: build/bin/cuda_atomic_ops_test 2024-06-26T07:12:38.5141265Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-26T07:12:38.5209906Z inflating: build/bin/cuda_complex_math_test 2024-06-26T07:12:38.5266981Z inflating: build/bin/cuda_complex_test 2024-06-26T07:12:38.5315411Z inflating: build/bin/cuda_device_test 2024-06-26T07:12:38.5371180Z inflating: build/bin/cuda_cub_test 2024-06-26T07:12:38.5420580Z inflating: build/bin/cuda_dlconvertor_test 2024-06-26T07:12:38.5483528Z inflating: build/bin/cuda_distributions_test 2024-06-26T07:12:38.5540657Z inflating: build/bin/cuda_generator_test 2024-06-26T07:12:38.5588908Z inflating: build/bin/cuda_half_test 2024-06-26T07:12:38.5638831Z inflating: build/bin/cuda_integer_divider_test 2024-06-26T07:12:38.5686710Z inflating: build/bin/cuda_optional_test 2024-06-26T07:12:38.5737089Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-26T07:12:38.5788542Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-26T07:12:38.5837577Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-26T07:12:38.5897070Z inflating: build/bin/cuda_stream_test 2024-06-26T07:12:38.5945120Z inflating: build/bin/cuda_cudnn_test 2024-06-26T07:12:38.5995950Z inflating: build/bin/cuda_vectorized_test 2024-06-26T07:12:38.6010737Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T07:12:38.6075049Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T07:12:38.6130647Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-26T07:12:38.6192046Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-26T07:12:38.6252685Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-26T07:12:38.7057842Z inflating: build/bin/test_tensorexpr 2024-06-26T07:12:38.7616911Z inflating: build/bin/test_jit 2024-06-26T07:12:38.7617383Z creating: .additional_ci_files/ 2024-06-26T07:12:38.7670930Z inflating: .additional_ci_files/test-times.json 2024-06-26T07:12:38.7882364Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T07:12:38.7908933Z ##[group]Run rm artifacts.zip 2024-06-26T07:12:38.7909455Z rm artifacts.zip 2024-06-26T07:12:38.7918995Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:38.7919506Z env: 2024-06-26T07:12:38.7919804Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:38.7920278Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:38.7920783Z ##[endgroup] 2024-06-26T07:12:38.8629177Z ##[group]Run df -H 2024-06-26T07:12:38.8629513Z df -H 2024-06-26T07:12:38.8637169Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:38.8637684Z env: 2024-06-26T07:12:38.8637967Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:38.8638438Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:38.8638939Z ##[endgroup] 2024-06-26T07:12:38.8665155Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T07:12:38.8665679Z devtmpfs 34G 0 34G 0% /dev 2024-06-26T07:12:38.8666160Z tmpfs 34G 2.8M 34G 1% /dev/shm 2024-06-26T07:12:38.8666700Z tmpfs 34G 455k 34G 1% /run 2024-06-26T07:12:38.8667448Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-06-26T07:12:38.8667946Z /dev/nvme0n1p1 162G 44G 118G 27% / 2024-06-26T07:12:38.8706336Z Prepare all required actions 2024-06-26T07:12:38.8706790Z Getting action download info 2024-06-26T07:12:38.9889698Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T07:12:38.9890174Z with: 2024-06-26T07:12:38.9890433Z env: 2024-06-26T07:12:38.9890719Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:38.9891200Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:38.9891702Z ##[endgroup] 2024-06-26T07:12:38.9923419Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:12:38.9923878Z with: 2024-06-26T07:12:38.9924150Z name: td_results 2024-06-26T07:12:38.9924482Z s3-bucket: gha-artifacts 2024-06-26T07:12:38.9924847Z region: us-east-1 2024-06-26T07:12:38.9925156Z env: 2024-06-26T07:12:38.9925444Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:38.9925930Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:38.9926426Z ##[endgroup] 2024-06-26T07:12:39.3974332Z (node:21363) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:12:39.3975081Z 2024-06-26T07:12:39.3975360Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:12:39.3976121Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:12:39.3977002Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:12:39.4677825Z Found 0 objects with prefix pytorch/pytorch/9673646806/td_results/ 2024-06-26T07:12:39.4684618Z Artifact download has finished successfully 2024-06-26T07:12:39.4813788Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T07:12:39.4814285Z mkdir -p .additional_ci_files 2024-06-26T07:12:39.4814862Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T07:12:39.4822646Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:39.4823166Z env: 2024-06-26T07:12:39.4823456Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:39.4823943Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:39.4824442Z ##[endgroup] 2024-06-26T07:12:39.4858951Z mv: cannot stat 'td_results.json': No such file or directory 2024-06-26T07:12:39.4871991Z ##[error]Process completed with exit code 1. 2024-06-26T07:12:39.4903714Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T07:12:39.4904216Z .github/scripts/parse_ref.py 2024-06-26T07:12:39.4911219Z shell: /usr/bin/bash -e {0} 2024-06-26T07:12:39.4911587Z env: 2024-06-26T07:12:39.4911881Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:39.4912372Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:39.4912873Z ##[endgroup] 2024-06-26T07:12:39.5155218Z Prepare all required actions 2024-06-26T07:12:39.5193706Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T07:12:39.5194169Z with: 2024-06-26T07:12:39.5194868Z github-token: *** 2024-06-26T07:12:39.5195182Z env: 2024-06-26T07:12:39.5195460Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:39.5195933Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:39.5196429Z ##[endgroup] 2024-06-26T07:12:39.5212094Z ##[group]Run set -eux 2024-06-26T07:12:39.5212440Z set -eux 2024-06-26T07:12:39.5213041Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T07:12:39.5220740Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:39.5221247Z env: 2024-06-26T07:12:39.5221539Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:39.5222020Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:39.5222691Z GITHUB_TOKEN: *** 2024-06-26T07:12:39.5223006Z ##[endgroup] 2024-06-26T07:12:39.5242895Z + python3 .github/scripts/get_workflow_job_id.py 9673646806 i-06740481be1ad1e06 2024-06-26T07:12:40.2840007Z setting job-id=26688916888 2024-06-26T07:12:40.2841265Z setting job-name=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:40.3047800Z Prepare all required actions 2024-06-26T07:12:40.3048263Z Getting action download info 2024-06-26T07:12:40.4105494Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T07:12:40.4105950Z with: 2024-06-26T07:12:40.4106414Z github-token: *** 2024-06-26T07:12:40.4113246Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:12:40.4121030Z job-name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:40.4121978Z env: 2024-06-26T07:12:40.4122257Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:40.4122728Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:40.4123224Z ##[endgroup] 2024-06-26T07:12:40.4165182Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:12:40.4165749Z with: 2024-06-26T07:12:40.4166043Z shell: bash 2024-06-26T07:12:40.4166382Z timeout_minutes: 10 2024-06-26T07:12:40.4166721Z max_attempts: 5 2024-06-26T07:12:40.4167181Z retry_wait_seconds: 30 2024-06-26T07:12:40.4168347Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:12:40.4169565Z polling_interval_seconds: 1 2024-06-26T07:12:40.4170102Z warning_on_retry: true 2024-06-26T07:12:40.4170464Z continue_on_error: false 2024-06-26T07:12:40.4170815Z env: 2024-06-26T07:12:40.4171097Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:40.4171574Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:40.4172233Z GITHUB_TOKEN: *** 2024-06-26T07:12:40.4172553Z ##[endgroup] 2024-06-26T07:12:40.4667930Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:12:40.6812889Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:12:40.6967058Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-06-26T07:12:40.7111170Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-06-26T07:12:40.7119831Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-06-26T07:12:40.7318203Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2) 2024-06-26T07:12:40.7325682Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-06-26T07:12:40.7339402Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-06-26T07:12:41.4662246Z Command completed after 1 attempt(s). 2024-06-26T07:12:41.4702103Z ##[group]Run set -x 2024-06-26T07:12:41.4702450Z set -x 2024-06-26T07:12:41.4702753Z  2024-06-26T07:12:41.4703322Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:12:41.4704017Z # in runner workspace 2024-06-26T07:12:41.4704565Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T07:12:41.4712479Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:41.4713001Z env: 2024-06-26T07:12:41.4713291Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:41.4713768Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:41.4714272Z ##[endgroup] 2024-06-26T07:12:41.4739219Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T07:12:41.4936938Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:12:41.4937505Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:12:41.4937988Z echo "Job name: ${JOB_NAME}" 2024-06-26T07:12:41.4938395Z  2024-06-26T07:12:41.4938964Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:12:41.4939655Z # in runner workspace 2024-06-26T07:12:41.4940259Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T07:12:41.4940932Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T07:12:41.4941415Z  --job-name "${JOB_NAME}" \ 2024-06-26T07:12:41.4948446Z  --test-matrix "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-06-26T07:12:41.4955858Z  --selected-test-configs "" \ 2024-06-26T07:12:41.4956323Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T07:12:41.4956758Z  --tag "${TAG}" \ 2024-06-26T07:12:41.4957171Z  --event-name "${EVENT_NAME}" \ 2024-06-26T07:12:41.4957631Z  --schedule "${SCHEDULE}" \ 2024-06-26T07:12:41.4958159Z  --branch "${HEAD_BRANCH}" 2024-06-26T07:12:41.4965553Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:41.4966069Z env: 2024-06-26T07:12:41.4966363Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:41.4966973Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:41.4967665Z GITHUB_TOKEN: *** 2024-06-26T07:12:41.4968551Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:41.4969506Z PR_NUMBER: 2024-06-26T07:12:41.4969816Z TAG: ciflow/inductor/129470 2024-06-26T07:12:41.4970195Z EVENT_NAME: push 2024-06-26T07:12:41.4970509Z SCHEDULE: 2024-06-26T07:12:41.4970800Z HEAD_BRANCH: 2024-06-26T07:12:41.4971095Z ##[endgroup] 2024-06-26T07:12:41.5004545Z Workflow: inductor-periodic 2024-06-26T07:12:41.5005687Z Job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:41.7050173Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T07:12:41.8355213Z ##[group]Run echo "Filtered matrix:" 2024-06-26T07:12:41.8355680Z echo "Filtered matrix:" 2024-06-26T07:12:41.8362679Z echo "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-06-26T07:12:41.8371186Z  2024-06-26T07:12:41.8371473Z echo 2024-06-26T07:12:41.8371875Z echo "Is the current job unstable? False" 2024-06-26T07:12:41.8372356Z  2024-06-26T07:12:41.8372624Z echo 2024-06-26T07:12:41.8372992Z echo "Is keep-going label set? False" 2024-06-26T07:12:41.8373452Z  2024-06-26T07:12:41.8373726Z echo 2024-06-26T07:12:41.8374055Z echo "Renabled issues? " 2024-06-26T07:12:41.8381875Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:41.8382399Z env: 2024-06-26T07:12:41.8382680Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:41.8383153Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:41.8383652Z ##[endgroup] 2024-06-26T07:12:41.8403414Z Filtered matrix: 2024-06-26T07:12:41.8412830Z {include: [{config: dynamo_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-06-26T07:12:41.8420051Z 2024-06-26T07:12:41.8420208Z Is the current job unstable? False 2024-06-26T07:12:41.8420514Z 2024-06-26T07:12:41.8420801Z Is keep-going label set? False 2024-06-26T07:12:41.8421071Z 2024-06-26T07:12:41.8421204Z Renabled issues? 2024-06-26T07:12:41.8460460Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:12:41.8461198Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:12:41.8468368Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:41.8468878Z env: 2024-06-26T07:12:41.8469155Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:41.8469624Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:41.8470134Z JOB_TIMEOUT: 240 2024-06-26T07:12:41.8470447Z ##[endgroup] 2024-06-26T07:12:41.8544513Z ##[group]Run set -x 2024-06-26T07:12:41.8545013Z set -x 2024-06-26T07:12:41.8545330Z  2024-06-26T07:12:41.8545699Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T07:12:41.8546264Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T07:12:41.8546858Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T07:12:41.8547395Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-26T07:12:41.8547832Z else 2024-06-26T07:12:41.8548189Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:12:41.8548632Z fi 2024-06-26T07:12:41.8548927Z  2024-06-26T07:12:41.8549417Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T07:12:41.8550203Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T07:12:41.8550911Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T07:12:41.8551662Z # shellcheck disable=SC2086,SC2090 2024-06-26T07:12:41.8552144Z container_name=$(docker run \ 2024-06-26T07:12:41.8552588Z  ${GPU_FLAG:-} \ 2024-06-26T07:12:41.8552979Z  -e BUILD_ENVIRONMENT \ 2024-06-26T07:12:41.8553398Z  -e PR_NUMBER \ 2024-06-26T07:12:41.8553775Z  -e GITHUB_ACTIONS \ 2024-06-26T07:12:41.8554189Z  -e GITHUB_REPOSITORY \ 2024-06-26T07:12:41.8554618Z  -e GITHUB_WORKFLOW \ 2024-06-26T07:12:41.8555022Z  -e GITHUB_JOB \ 2024-06-26T07:12:41.8555398Z  -e GITHUB_RUN_ID \ 2024-06-26T07:12:41.8555804Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T07:12:41.8556233Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T07:12:41.8556692Z  -e JOB_ID \ 2024-06-26T07:12:41.8557055Z  -e JOB_NAME \ 2024-06-26T07:12:41.8557426Z  -e BASE_SHA \ 2024-06-26T07:12:41.8557789Z  -e BRANCH \ 2024-06-26T07:12:41.8558138Z  -e SHA1 \ 2024-06-26T07:12:41.8558504Z  -e AWS_DEFAULT_REGION \ 2024-06-26T07:12:41.8558931Z  -e IN_WHEEL_TEST \ 2024-06-26T07:12:41.8559328Z  -e SHARD_NUMBER \ 2024-06-26T07:12:41.8559711Z  -e TEST_CONFIG \ 2024-06-26T07:12:41.8560103Z  -e NUM_TEST_SHARDS \ 2024-06-26T07:12:41.8560515Z  -e REENABLED_ISSUES \ 2024-06-26T07:12:41.8560955Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T07:12:41.8561394Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T07:12:41.8561814Z  -e NO_TEST_TIMEOUT \ 2024-06-26T07:12:41.8562208Z  -e NO_TD \ 2024-06-26T07:12:41.8562569Z  -e TD_DISTRIBUTED \ 2024-06-26T07:12:41.8562962Z  -e PR_LABELS \ 2024-06-26T07:12:41.8563390Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T07:12:41.8563870Z  -e SCCACHE_BUCKET \ 2024-06-26T07:12:41.8564279Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-26T07:12:41.8564709Z  -e XLA_CUDA \ 2024-06-26T07:12:41.8565141Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T07:12:41.8565678Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T07:12:41.8566221Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T07:12:41.8566942Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-26T07:12:41.8567446Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-26T07:12:41.8567890Z  -e DASHBOARD_TAG \ 2024-06-26T07:12:41.8568393Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T07:12:41.8568988Z  --security-opt seccomp=unconfined \ 2024-06-26T07:12:41.8569483Z  --cap-add=SYS_PTRACE \ 2024-06-26T07:12:41.8569896Z  --ipc=host \ 2024-06-26T07:12:41.8570274Z  --shm-size="${SHM_SIZE}" \ 2024-06-26T07:12:41.8570694Z  --tty \ 2024-06-26T07:12:41.8571027Z  --detach \ 2024-06-26T07:12:41.8571409Z  --name="${container_name}" \ 2024-06-26T07:12:41.8571843Z  --user jenkins \ 2024-06-26T07:12:41.8572445Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T07:12:41.8573048Z  -w /var/lib/jenkins/workspace \ 2024-06-26T07:12:41.8573508Z  "${DOCKER_IMAGE}" 2024-06-26T07:12:41.8573867Z ) 2024-06-26T07:12:41.8574292Z # Propagate download.pytorch.org IP to container 2024-06-26T07:12:41.8575262Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-26T07:12:41.8576281Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-26T07:12:41.8577285Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-26T07:12:41.8585262Z shell: /usr/bin/bash -e {0} 2024-06-26T07:12:41.8585631Z env: 2024-06-26T07:12:41.8585924Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:41.8586400Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:41.8587163Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:12:41.8587689Z PR_NUMBER: 2024-06-26T07:12:41.8588032Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T07:12:41.8588487Z GITHUB_WORKFLOW: inductor-periodic 2024-06-26T07:12:41.8588904Z GITHUB_JOB: test 2024-06-26T07:12:41.8589240Z GITHUB_RUN_ID: 9673646806 2024-06-26T07:12:41.8589618Z GITHUB_RUN_NUMBER: 45567 2024-06-26T07:12:41.8589983Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T07:12:41.8590336Z JOB_ID: 26688916888 2024-06-26T07:12:41.8591246Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:41.8592212Z BRANCH: 2024-06-26T07:12:41.8592561Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:41.8593089Z BASE_SHA: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:41.8593612Z TEST_CONFIG: dynamic_aot_eager_torchbench 2024-06-26T07:12:41.8594061Z SHARD_NUMBER: 1 2024-06-26T07:12:41.8594389Z NUM_TEST_SHARDS: 2 2024-06-26T07:12:41.8594732Z REENABLED_ISSUES: 2024-06-26T07:12:41.8595094Z CONTINUE_THROUGH_ERROR: False 2024-06-26T07:12:41.8595503Z VERBOSE_TEST_LOGS: False 2024-06-26T07:12:41.8595875Z NO_TEST_TIMEOUT: False 2024-06-26T07:12:41.8596227Z NO_TD: False 2024-06-26T07:12:41.8596579Z TD_DISTRIBUTED: False 2024-06-26T07:12:41.8597026Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T07:12:41.8597555Z SCCACHE_S3_KEY_PREFIX: inductor-periodic 2024-06-26T07:12:41.8598004Z SHM_SIZE: 2g 2024-06-26T07:12:41.8599140Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:12:41.8600358Z XLA_CUDA: 2024-06-26T07:12:41.8600849Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:12:41.8601488Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T07:12:41.8601950Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T07:12:41.8602395Z DASHBOARD_TAG: 2024-06-26T07:12:41.8602918Z HUGGING_FACE_HUB_TOKEN: *** 2024-06-26T07:12:41.8603301Z ##[endgroup] 2024-06-26T07:12:41.8622440Z + [[ dynamic_aot_eager_torchbench == \m\u\l\t\i\g\p\u ]] 2024-06-26T07:12:41.8623199Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-06-26T07:12:41.8623734Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:12:41.8629598Z +++ nproc --ignore=2 2024-06-26T07:12:41.8642749Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9673646806 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:12:53.0699741Z + container_name=57400afaa3f29946832de03690d2b6347f2f66970ba9ac0b0161152c7a47ef91 2024-06-26T07:12:53.0701201Z + grep download.pytorch.org /etc/hosts 2024-06-26T07:12:53.0703655Z + docker exec -i 57400afaa3f29946832de03690d2b6347f2f66970ba9ac0b0161152c7a47ef91 sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-26T07:12:53.1301765Z + echo DOCKER_CONTAINER_ID=57400afaa3f29946832de03690d2b6347f2f66970ba9ac0b0161152c7a47ef91 2024-06-26T07:12:53.1304075Z ++ echo dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:12:53.1305923Z + docker exec -t 57400afaa3f29946832de03690d2b6347f2f66970ba9ac0b0161152c7a47ef91 sh -c 'pip install dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-26T07:12:53.5020643Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb8c4c54) 2024-06-26T07:12:53.8158979Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T07:12:53.8164526Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T07:12:53.8166677Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T07:12:53.8169060Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T07:12:53.8172232Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T07:12:53.8175337Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T07:12:53.8201119Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.3.0) 2024-06-26T07:12:53.8260672Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.21.2) 2024-06-26T07:12:53.8699752Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T07:12:53.8858786Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T07:12:55.2239444Z Installing collected packages: torch 2024-06-26T07:13:03.9289589Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-06-26T07:13:03.9291013Z timm 0.9.7 requires torchvision, which is not installed. 2024-06-26T07:13:03.9291708Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T07:13:04.0235354Z ++ dirname .ci/pytorch/test.sh 2024-06-26T07:13:04.0238681Z + source .ci/pytorch/common.sh 2024-06-26T07:13:04.0240518Z +++ dirname .ci/pytorch/common.sh 2024-06-26T07:13:04.0245389Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T07:13:04.0246317Z +++ declare -f -t trap_add 2024-06-26T07:13:04.0252293Z ++ set -ex 2024-06-26T07:13:04.0252988Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:13:04.0253548Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T07:13:04.0254075Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-06-26T07:13:04.0256635Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-26T07:13:04.0265677Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-26T07:13:04.0266117Z + trap_add cleanup_workspace EXIT 2024-06-26T07:13:04.0266536Z + trap_add_cmd=cleanup_workspace 2024-06-26T07:13:04.0266914Z + shift 2024-06-26T07:13:04.0270050Z + for trap_add_name in "$@" 2024-06-26T07:13:04.0271676Z +++ trap -p EXIT 2024-06-26T07:13:04.0273761Z ++ eval 'extract_trap_cmd ' 2024-06-26T07:13:04.0274290Z +++ extract_trap_cmd 2024-06-26T07:13:04.0274800Z +++ printf '%s\n' '' 2024-06-26T07:13:04.0275657Z ++ printf '%s\n' cleanup_workspace 2024-06-26T07:13:04.0276105Z + trap -- ' 2024-06-26T07:13:04.0276457Z cleanup_workspace' EXIT 2024-06-26T07:13:04.0276935Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-26T07:13:04.4305967Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-26T07:13:04.4319048Z + echo 'Environment variables:' 2024-06-26T07:13:04.4319482Z Environment variables: 2024-06-26T07:13:04.4319824Z + env 2024-06-26T07:13:04.4327398Z INSTALLED_DB=yes 2024-06-26T07:13:04.4327951Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:13:04.4328501Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:13:04.4329017Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:13:04.4329805Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:13:04.4330582Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:13:04.4331257Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:13:04.4332009Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:13:04.4332896Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:13:04.4333612Z HOSTNAME=57400afaa3f2 2024-06-26T07:13:04.4334810Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_ec0756d7-5868-4fd1-ab1b-7006d79cb6b6 2024-06-26T07:13:04.4335891Z GITHUB_ACTION=__self 2024-06-26T07:13:04.4336355Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:13:04.4340273Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:13:04.4344356Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:13:04.4345020Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:13:04.4345404Z GITHUB_RUN_NUMBER=45567 2024-06-26T07:13:04.4345789Z TEST_CONFIG=dynamic_aot_eager_torchbench 2024-06-26T07:13:04.4346259Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:13:04.4346756Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:13:04.4347244Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:13:04.4347706Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:13:04.4348150Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:13:04.4350351Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:13:04.4351070Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:13:04.4351725Z GITHUB_REF_TYPE=tag 2024-06-26T07:13:04.4352064Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:13:04.4352478Z NCCL_VERSION=2.17.1-1 2024-06-26T07:13:04.4353150Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:04.4353608Z XLA_CUDA= 2024-06-26T07:13:04.4354114Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:13:04.4355897Z *** 2024-06-26T07:13:04.4356216Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:13:04.4356640Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:13:04.4357024Z GITHUB_ACTIONS=true 2024-06-26T07:13:04.4357374Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:04.4357904Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:13:04.4358461Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:13:04.4358940Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:04.4359462Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:13:04.4359967Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:04.4360926Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:13:04.4361723Z UCC_HOME=/usr 2024-06-26T07:13:04.4362269Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:13:04.4362699Z VERBOSE_TEST_LOGS=False 2024-06-26T07:13:04.4363057Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:13:04.4363541Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:13:04.4364066Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:13:04.4364555Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:13:04.4364947Z SHARD_NUMBER=1 2024-06-26T07:13:04.4365276Z GITHUB_REF_PROTECTED=false 2024-06-26T07:13:04.4365648Z HOME=/var/lib/jenkins 2024-06-26T07:13:04.4366031Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:13:04.4366483Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:13:04.4367266Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:13:04.4367847Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-06-26T07:13:04.4368286Z CUDA_VERSION=12.1.1 2024-06-26T07:13:04.4368715Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:13:04.4369177Z NUM_TEST_SHARDS=2 2024-06-26T07:13:04.4369492Z UCX_HOME=/usr 2024-06-26T07:13:04.4370028Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:13:04.4371144Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_ec0756d7-5868-4fd1-ab1b-7006d79cb6b6 2024-06-26T07:13:04.4372717Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:13:04.4374188Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_ec0756d7-5868-4fd1-ab1b-7006d79cb6b6 2024-06-26T07:13:04.4375337Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:13:04.4376013Z GITHUB_EVENT_NAME=push 2024-06-26T07:13:04.4376365Z DASHBOARD_TAG= 2024-06-26T07:13:04.4376686Z GITHUB_RUN_ID=9673646806 2024-06-26T07:13:04.4377152Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:13:04.4377709Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:13:04.4378740Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_ec0756d7-5868-4fd1-ab1b-7006d79cb6b6 2024-06-26T07:13:04.4379684Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:13:04.4380103Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:13:04.4380503Z PR_NUMBER= 2024-06-26T07:13:04.4380807Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:13:04.4381196Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:13:04.4381665Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:13:04.4382138Z TERM=xterm 2024-06-26T07:13:04.4382487Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:13:04.4382918Z INSTALLED_VISION=yes 2024-06-26T07:13:04.4383241Z BRANCH= 2024-06-26T07:13:04.4383539Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:13:04.4383960Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:13:04.4384405Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:13:04.4385289Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:13:04.4386077Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:13:04.4386736Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:13:04.4387232Z REENABLED_ISSUES= 2024-06-26T07:13:04.4387538Z SHLVL=1 2024-06-26T07:13:04.4387812Z MAX_JOBS=14 2024-06-26T07:13:04.4388146Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:13:04.4388507Z NVARCH=x86_64 2024-06-26T07:13:04.4388820Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:13:04.4389291Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:04.4389839Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:13:04.4390337Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:13:04.4391012Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:13:04.4391600Z GITHUB_JOB=test 2024-06-26T07:13:04.4392061Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:13:04.4392650Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:13:04.4393185Z NO_TEST_TIMEOUT=False 2024-06-26T07:13:04.4393533Z TD_DISTRIBUTED=False 2024-06-26T07:13:04.4394049Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:13:04.4394502Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:13:04.4394939Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:13:04.4395322Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:13:04.4395681Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:13:04.4396046Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:13:04.4397140Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:13:04.4398254Z GITHUB_BASE_REF= 2024-06-26T07:13:04.4398593Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:13:04.4398978Z CI=true 2024-06-26T07:13:04.4399310Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:13:04.4399745Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:13:04.4400129Z JOB_ID=26688916888 2024-06-26T07:13:04.4400461Z INSTALLED_PROTOBUF=yes 2024-06-26T07:13:04.4400803Z GITHUB_HEAD_REF= 2024-06-26T07:13:04.4401113Z GITHUB_ACTION_REF= 2024-06-26T07:13:04.4401563Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:13:04.4402107Z GITHUB_WORKFLOW=inductor-periodic 2024-06-26T07:13:04.4402582Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:13:04.4403503Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_ec0756d7-5868-4fd1-ab1b-7006d79cb6b6 2024-06-26T07:13:04.4404353Z NO_TD=False 2024-06-26T07:13:04.4404679Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:13:04.4405068Z _=/usr/bin/env 2024-06-26T07:13:04.4405554Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T07:13:04.4480673Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-26T07:13:04.4481987Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T07:13:04.4483179Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-26T07:13:04.4484416Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-26T07:13:04.4485261Z + BUILD_DIR=build 2024-06-26T07:13:04.4485733Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T07:13:04.4486299Z + BUILD_BIN_DIR=build/bin 2024-06-26T07:13:04.4486712Z + SHARD_NUMBER=1 2024-06-26T07:13:04.4487302Z + NUM_TEST_SHARDS=2 2024-06-26T07:13:04.4487635Z + export VALGRIND=ON 2024-06-26T07:13:04.4488006Z + VALGRIND=ON 2024-06-26T07:13:04.4488501Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-06-26T07:13:04.4489019Z + [[ 0 == \1 ]] 2024-06-26T07:13:04.4489331Z + [[ False == \1 ]] 2024-06-26T07:13:04.4489827Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-06-26T07:13:04.4490371Z ++ realpath build/custom_test_artifacts 2024-06-26T07:13:04.4494783Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-26T07:13:04.4495655Z + [[ -n '' ]] 2024-06-26T07:13:04.4496018Z + echo 'Environment variables' 2024-06-26T07:13:04.4496410Z Environment variables 2024-06-26T07:13:04.4496736Z + env 2024-06-26T07:13:04.4501284Z INSTALLED_DB=yes 2024-06-26T07:13:04.4501742Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:13:04.4502553Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:13:04.4503133Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:13:04.4504005Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:13:04.4505066Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:13:04.4505774Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:13:04.4506539Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:13:04.4507380Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:13:04.4508109Z HOSTNAME=57400afaa3f2 2024-06-26T07:13:04.4509293Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_ec0756d7-5868-4fd1-ab1b-7006d79cb6b6 2024-06-26T07:13:04.4510382Z GITHUB_ACTION=__self 2024-06-26T07:13:04.4510742Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:13:04.4514866Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:13:04.4519138Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:13:04.4519674Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:13:04.4520052Z GITHUB_RUN_NUMBER=45567 2024-06-26T07:13:04.4520443Z TEST_CONFIG=dynamic_aot_eager_torchbench 2024-06-26T07:13:04.4520913Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:13:04.4521417Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:13:04.4521915Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:13:04.4522396Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:13:04.4522868Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:13:04.4523335Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:13:04.4523860Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:13:04.4524341Z GITHUB_REF_TYPE=tag 2024-06-26T07:13:04.4524689Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:13:04.4525084Z NCCL_VERSION=2.17.1-1 2024-06-26T07:13:04.4525491Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:04.4525953Z XLA_CUDA= 2024-06-26T07:13:04.4526471Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:13:04.4527155Z *** 2024-06-26T07:13:04.4527463Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:13:04.4527879Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:13:04.4528265Z GITHUB_ACTIONS=true 2024-06-26T07:13:04.4528617Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:04.4529126Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:13:04.4529696Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:13:04.4530183Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:04.4530706Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:13:04.4531206Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:04.4532159Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:13:04.4533016Z UCC_HOME=/usr 2024-06-26T07:13:04.4533368Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:13:04.4533785Z VERBOSE_TEST_LOGS=False 2024-06-26T07:13:04.4534146Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:13:04.4534621Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:13:04.4535141Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:13:04.4535622Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:13:04.4536019Z SHARD_NUMBER=1 2024-06-26T07:13:04.4536344Z GITHUB_REF_PROTECTED=false 2024-06-26T07:13:04.4536715Z HOME=/var/lib/jenkins 2024-06-26T07:13:04.4537208Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:13:04.4537676Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:13:04.4538175Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:13:04.4538735Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-06-26T07:13:04.4539168Z CUDA_VERSION=12.1.1 2024-06-26T07:13:04.4539603Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:13:04.4540063Z NUM_TEST_SHARDS=2 2024-06-26T07:13:04.4540370Z UCX_HOME=/usr 2024-06-26T07:13:04.4540899Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:13:04.4542005Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_ec0756d7-5868-4fd1-ab1b-7006d79cb6b6 2024-06-26T07:13:04.4543504Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:13:04.4545055Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_ec0756d7-5868-4fd1-ab1b-7006d79cb6b6 2024-06-26T07:13:04.4546320Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:13:04.4547002Z GITHUB_EVENT_NAME=push 2024-06-26T07:13:04.4547345Z DASHBOARD_TAG= 2024-06-26T07:13:04.4547654Z GITHUB_RUN_ID=9673646806 2024-06-26T07:13:04.4548130Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:13:04.4548689Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:13:04.4549717Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_ec0756d7-5868-4fd1-ab1b-7006d79cb6b6 2024-06-26T07:13:04.4550660Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:13:04.4551090Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:13:04.4551486Z PR_NUMBER= 2024-06-26T07:13:04.4551784Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:13:04.4552161Z VALGRIND=ON 2024-06-26T07:13:04.4552480Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:13:04.4552941Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:13:04.4553423Z TERM=xterm 2024-06-26T07:13:04.4553780Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:13:04.4554219Z INSTALLED_VISION=yes 2024-06-26T07:13:04.4554552Z BRANCH= 2024-06-26T07:13:04.4554847Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:13:04.4555272Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:13:04.4555716Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:13:04.4556503Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:13:04.4557287Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:13:04.4557799Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:13:04.4558285Z REENABLED_ISSUES= 2024-06-26T07:13:04.4558589Z SHLVL=1 2024-06-26T07:13:04.4558867Z MAX_JOBS=14 2024-06-26T07:13:04.4559195Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:13:04.4559564Z NVARCH=x86_64 2024-06-26T07:13:04.4559870Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:13:04.4560353Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:04.4560926Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:13:04.4561437Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:13:04.4562114Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:13:04.4562713Z GITHUB_JOB=test 2024-06-26T07:13:04.4563129Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:13:04.4563708Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:13:04.4564247Z NO_TEST_TIMEOUT=False 2024-06-26T07:13:04.4564600Z TD_DISTRIBUTED=False 2024-06-26T07:13:04.4565007Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:13:04.4565460Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:13:04.4565901Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:13:04.4566292Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:13:04.4566658Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:13:04.4567274Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:13:04.4568473Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:13:04.4569611Z GITHUB_BASE_REF= 2024-06-26T07:13:04.4569977Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:13:04.4570355Z CI=true 2024-06-26T07:13:04.4570697Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:13:04.4571131Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:13:04.4584004Z JOB_ID=26688916888 2024-06-26T07:13:04.4584374Z INSTALLED_PROTOBUF=yes 2024-06-26T07:13:04.4584731Z GITHUB_HEAD_REF= 2024-06-26T07:13:04.4585163Z GITHUB_ACTION_REF= 2024-06-26T07:13:04.4585640Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:13:04.4586188Z GITHUB_WORKFLOW=inductor-periodic 2024-06-26T07:13:04.4586613Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:13:04.4587550Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_ec0756d7-5868-4fd1-ab1b-7006d79cb6b6 2024-06-26T07:13:04.4588403Z NO_TD=False 2024-06-26T07:13:04.4588721Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:13:04.4589246Z _=/usr/bin/env 2024-06-26T07:13:04.4589603Z + echo 'Testing pytorch' 2024-06-26T07:13:04.4589965Z Testing pytorch 2024-06-26T07:13:04.4590300Z + export LANG=C.UTF-8 2024-06-26T07:13:04.4590645Z + LANG=C.UTF-8 2024-06-26T07:13:04.4590948Z + PR_NUMBER= 2024-06-26T07:13:04.4591341Z + [[ dynamic_aot_eager_torchbench == \d\e\f\a\u\l\t ]] 2024-06-26T07:13:04.4591952Z + [[ dynamic_aot_eager_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T07:13:04.4592533Z + [[ dynamic_aot_eager_torchbench == \s\l\o\w ]] 2024-06-26T07:13:04.4593220Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-06-26T07:13:04.4593939Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-06-26T07:13:04.4594510Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:13:04.4595001Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:13:04.4595507Z + [[ dynamic_aot_eager_torchbench == *crossref* ]] 2024-06-26T07:13:04.4596145Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:13:04.4596809Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-06-26T07:13:04.4597476Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:13:04.4598066Z + pip_install --user ninja==1.10.2 2024-06-26T07:13:04.4598625Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T07:13:04.8187543Z Collecting ninja==1.10.2 2024-06-26T07:13:04.8354540Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T07:13:04.8490493Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T07:13:06.1582361Z Installing collected packages: ninja 2024-06-26T07:13:06.1650241Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T07:13:06.1651567Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T07:13:06.1689207Z Successfully installed ninja-1.10.2 2024-06-26T07:13:06.2667404Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:13:06.2669678Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:13:06.2671258Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:13:06.2671776Z + install_tlparse 2024-06-26T07:13:06.2672162Z + pip_install --user tlparse==0.3.7 2024-06-26T07:13:06.2672724Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T07:13:06.6174514Z Collecting tlparse==0.3.7 2024-06-26T07:13:06.6309645Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T07:13:06.6405909Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T07:13:07.9714842Z Installing collected packages: tlparse 2024-06-26T07:13:08.0075874Z Successfully installed tlparse-0.3.7 2024-06-26T07:13:08.1036219Z ++ python -m site --user-base 2024-06-26T07:13:08.1189379Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:13:08.1191282Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-06-26T07:13:08.1192072Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-06-26T07:13:08.1192836Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:13:08.1194084Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-06-26T07:13:08.1195367Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-06-26T07:13:08.1196109Z + cd test 2024-06-26T07:13:08.1196878Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T07:13:09.5734850Z + [[ dynamic_aot_eager_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T07:13:09.5735786Z + [[ dynamic_aot_eager_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T07:13:09.5736648Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:13:09.5737168Z + pushd test 2024-06-26T07:13:09.5737491Z ~/workspace/test ~/workspace 2024-06-26T07:13:09.5738027Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T07:13:11.0383657Z + CUDA_VERSION=12.1 2024-06-26T07:13:11.0384189Z + '[' 12.1 == 12.4 ']' 2024-06-26T07:13:11.0384548Z + ISCUDA124= 2024-06-26T07:13:11.0384948Z + popd 2024-06-26T07:13:11.0385232Z ~/workspace 2024-06-26T07:13:11.0388546Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T07:13:11.0389059Z + [[ dynamic_aot_eager_torchbench == *dynamo_eager* ]] 2024-06-26T07:13:11.0389762Z + [[ dynamic_aot_eager_torchbench == *aot_eager* ]] 2024-06-26T07:13:11.0390431Z + DYNAMO_BENCHMARK_FLAGS+=(--backend aot_eager) 2024-06-26T07:13:11.0391006Z + [[ dynamic_aot_eager_torchbench == *dynamic* ]] 2024-06-26T07:13:11.0391686Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2024-06-26T07:13:11.0392334Z + [[ dynamic_aot_eager_torchbench == *cpu_inductor* ]] 2024-06-26T07:13:11.0392921Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T07:13:11.0416482Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:13:11.0417470Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-06-26T07:13:11.0418153Z + cd test 2024-06-26T07:13:11.0418777Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T07:13:12.4041434Z PyTorch built with: 2024-06-26T07:13:12.4042345Z - GCC 9.4 2024-06-26T07:13:12.4042714Z - C++ Version: 201703 2024-06-26T07:13:12.4043756Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:13:12.4045076Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:13:12.4045867Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:13:12.4046467Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T07:13:12.4047299Z - NNPACK is enabled 2024-06-26T07:13:12.4047714Z - CPU capability usage: AVX2 2024-06-26T07:13:12.4048198Z - CUDA Runtime 12.1 2024-06-26T07:13:12.4048781Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-06-26T07:13:12.4049444Z - CuDNN 90.1 (built against CUDA 12.4) 2024-06-26T07:13:12.4049976Z - Magma 2.6.1 2024-06-26T07:13:12.4058113Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T07:13:12.4065555Z 2024-06-26T07:13:12.6781772Z + cd test 2024-06-26T07:13:12.6782536Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T07:13:13.8999012Z ATen/Parallel: 2024-06-26T07:13:13.8999629Z at::get_num_threads() : 8 2024-06-26T07:13:13.9000157Z at::get_num_interop_threads() : 16 2024-06-26T07:13:13.9000740Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:13:13.9001267Z omp_get_max_threads() : 8 2024-06-26T07:13:13.9002280Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:13:13.9003125Z mkl_get_max_threads() : 8 2024-06-26T07:13:13.9003771Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:13:13.9004442Z std::thread::hardware_concurrency() : 16 2024-06-26T07:13:13.9004900Z Environment variables: 2024-06-26T07:13:13.9005262Z OMP_NUM_THREADS : [not set] 2024-06-26T07:13:13.9005661Z MKL_NUM_THREADS : [not set] 2024-06-26T07:13:13.9006063Z ATen parallel backend: OpenMP 2024-06-26T07:13:13.9006328Z 2024-06-26T07:13:14.1444749Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:13:14.1445547Z + [[ dynamic_aot_eager_torchbench == *backward* ]] 2024-06-26T07:13:14.1446097Z + [[ dynamic_aot_eager_torchbench == *xla* ]] 2024-06-26T07:13:14.1446638Z + [[ dynamic_aot_eager_torchbench == *executorch* ]] 2024-06-26T07:13:14.1447512Z + [[ dynamic_aot_eager_torchbench == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T07:13:14.1448242Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:13:14.1448998Z + [[ dynamic_aot_eager_torchbench == distributed ]] 2024-06-26T07:13:14.1449605Z + [[ dynamic_aot_eager_torchbench == *inductor_distributed* ]] 2024-06-26T07:13:14.1450275Z + [[ dynamic_aot_eager_torchbench == *inductor-halide* ]] 2024-06-26T07:13:14.1450979Z + [[ dynamic_aot_eager_torchbench == *inductor-micro-benchmark* ]] 2024-06-26T07:13:14.1451619Z + [[ dynamic_aot_eager_torchbench == *huggingface* ]] 2024-06-26T07:13:14.1452159Z + [[ dynamic_aot_eager_torchbench == *timm* ]] 2024-06-26T07:13:14.1452683Z + [[ dynamic_aot_eager_torchbench == *torchbench* ]] 2024-06-26T07:13:14.1453263Z + [[ dynamic_aot_eager_torchbench == *cpu_inductor* ]] 2024-06-26T07:13:14.1453768Z + install_torchaudio cuda 2024-06-26T07:13:14.1454119Z + local commit 2024-06-26T07:13:14.1454463Z ++ get_pinned_commit audio 2024-06-26T07:13:14.1454889Z ++ cat .github/ci_commit_pins/audio.txt 2024-06-26T07:13:14.1459256Z + commit=b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:14.1459902Z + [[ cuda == \c\u\d\a ]] 2024-06-26T07:13:14.1460341Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-06-26T07:13:14.1461275Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:14.1462749Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:14.4599599Z Collecting git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:14.4603189Z Cloning https://github.com/pytorch/audio.git (to revision b829e936f7cc61b48149f5f957a451a38bf2a178) to /tmp/pip-req-build-iaw3uh9s 2024-06-26T07:13:14.4621299Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-iaw3uh9s 2024-06-26T07:13:15.1473531Z Running command git rev-parse -q --verify 'sha^b829e936f7cc61b48149f5f957a451a38bf2a178' 2024-06-26T07:13:15.1491626Z Running command git fetch -q https://github.com/pytorch/audio.git b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:15.5268790Z Running command git checkout -q b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:15.6396796Z Resolved https://github.com/pytorch/audio.git to commit b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:15.6397872Z Running command git submodule update --init --recursive -q 2024-06-26T07:13:17.7890777Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:13:17.7925903Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.4.0a0+b829e93) (2.5.0a0+gitb8c4c54) 2024-06-26T07:13:17.7993714Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (3.13.1) 2024-06-26T07:13:17.7997821Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (4.12.2) 2024-06-26T07:13:17.8000118Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (1.12.1) 2024-06-26T07:13:17.8003546Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (2.8.8) 2024-06-26T07:13:17.8007174Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (3.1.4) 2024-06-26T07:13:17.8010359Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (2024.2.0) 2024-06-26T07:13:17.8442734Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.4.0a0+b829e93) (2.1.5) 2024-06-26T07:13:17.8600068Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchaudio==2.4.0a0+b829e93) (1.3.0) 2024-06-26T07:13:17.8674075Z Building wheels for collected packages: torchaudio 2024-06-26T07:15:42.9074151Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-06-26T07:15:42.9102166Z [?25h Created wheel for torchaudio: filename=torchaudio-2.4.0a0+b829e93-cp310-cp310-linux_x86_64.whl size=2308812 sha256=575de8964251da5877da850740fb7f6605079c9aae2249367c89c484e7c65cc6 2024-06-26T07:15:42.9105995Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/30/b3/9c/882f183f00af2e582825654a92a04400924519922db53ff522 2024-06-26T07:15:42.9132905Z Successfully built torchaudio 2024-06-26T07:15:44.1546443Z Installing collected packages: torchaudio 2024-06-26T07:15:44.3465106Z Successfully installed torchaudio-2.4.0a0+b829e93 2024-06-26T07:15:44.4811544Z + install_torchtext 2024-06-26T07:15:44.4819207Z + local data_commit 2024-06-26T07:15:44.4819916Z + local text_commit 2024-06-26T07:15:44.4820303Z ++ get_pinned_commit data 2024-06-26T07:15:44.4820708Z ++ cat .github/ci_commit_pins/data.txt 2024-06-26T07:15:44.4823847Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:44.4825644Z ++ get_pinned_commit text 2024-06-26T07:15:44.4826150Z ++ cat .github/ci_commit_pins/text.txt 2024-06-26T07:15:44.4834046Z + text_commit=b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:15:44.4835264Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:44.4836848Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:44.8015925Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:44.8021754Z Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-wgna6m8n 2024-06-26T07:15:44.8039631Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-wgna6m8n 2024-06-26T07:15:45.1977679Z Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f' 2024-06-26T07:15:45.1997152Z Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:45.4253658Z Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:45.5503197Z Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:45.5504287Z Running command git submodule update --init --recursive -q 2024-06-26T07:16:57.9373636Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:16:57.9434248Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.19) 2024-06-26T07:16:57.9436410Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.32.3) 2024-06-26T07:16:57.9442477Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.5.0a0+gitb8c4c54) 2024-06-26T07:16:57.9503213Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1) 2024-06-26T07:16:57.9508477Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.12.2) 2024-06-26T07:16:57.9511389Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.12.1) 2024-06-26T07:16:57.9514715Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8) 2024-06-26T07:16:57.9517508Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4) 2024-06-26T07:16:57.9520556Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.2.0) 2024-06-26T07:16:57.9707138Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2) 2024-06-26T07:16:57.9712954Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7) 2024-06-26T07:16:57.9718824Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.6.2) 2024-06-26T07:16:58.0178041Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5) 2024-06-26T07:16:58.0340530Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0) 2024-06-26T07:16:58.0420998Z Building wheels for collected packages: torchdata 2024-06-26T07:16:59.7889473Z Building wheel for torchdata (setup.py) ... [?25l- \ | done 2024-06-26T07:16:59.7895799Z [?25h Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182956 sha256=84c581bb0365ff5561bde01ed6b0885f16baffd0d15d30be5ec2c548d3fe59ba 2024-06-26T07:16:59.7898011Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce 2024-06-26T07:16:59.7923412Z Successfully built torchdata 2024-06-26T07:17:01.0102461Z Installing collected packages: torchdata 2024-06-26T07:17:01.0941948Z Successfully installed torchdata-0.7.0a0+11bb5b8 2024-06-26T07:17:03.5988348Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:17:03.5989894Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:17:03.9118990Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:17:03.9123147Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-mxnbnko1 2024-06-26T07:17:03.9140873Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-mxnbnko1 2024-06-26T07:17:04.6714680Z Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db' 2024-06-26T07:17:04.6732689Z Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:17:05.1279637Z Running command git checkout -q b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:17:05.2794272Z Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:17:05.2795833Z Running command git submodule update --init --recursive -q 2024-06-26T07:17:12.3906537Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:17:12.3963302Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.4) 2024-06-26T07:17:12.3965673Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.32.3) 2024-06-26T07:17:12.3969394Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.5.0a0+gitb8c4c54) 2024-06-26T07:17:12.3972633Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.21.2) 2024-06-26T07:17:12.3975611Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8) 2024-06-26T07:17:12.4036806Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2) 2024-06-26T07:17:12.4041954Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7) 2024-06-26T07:17:12.4048927Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.19) 2024-06-26T07:17:12.4054561Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.6.2) 2024-06-26T07:17:12.4111066Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1) 2024-06-26T07:17:12.4117580Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.12.2) 2024-06-26T07:17:12.4120526Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.12.1) 2024-06-26T07:17:12.4123587Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8) 2024-06-26T07:17:12.4126979Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4) 2024-06-26T07:17:12.4129706Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.2.0) 2024-06-26T07:17:12.4909362Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5) 2024-06-26T07:17:12.5071625Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0) 2024-06-26T07:17:12.5154464Z Building wheels for collected packages: torchtext 2024-06-26T07:17:48.8835446Z Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-06-26T07:17:48.8863104Z [?25h Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2045303 sha256=a56d2883aa852293cd609fee7da7c113d6e4e1b673fdc5c4af08b4f1ca5747d4 2024-06-26T07:17:48.8865642Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf 2024-06-26T07:17:48.8889780Z Successfully built torchtext 2024-06-26T07:17:50.0911508Z Installing collected packages: torchtext 2024-06-26T07:17:50.1890258Z Successfully installed torchtext-0.16.0a0+b0ebddc 2024-06-26T07:17:50.3140557Z + install_torchvision 2024-06-26T07:17:50.3141033Z + local orig_preload 2024-06-26T07:17:50.3141435Z + local commit 2024-06-26T07:17:50.3141767Z ++ get_pinned_commit vision 2024-06-26T07:17:50.3142169Z ++ cat .github/ci_commit_pins/vision.txt 2024-06-26T07:17:50.3151882Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:17:50.3152512Z + orig_preload= 2024-06-26T07:17:50.3152965Z + '[' -n '' ']' 2024-06-26T07:17:50.3153849Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:17:50.3155328Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:17:50.6305158Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:17:50.6310827Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-067xqowh 2024-06-26T07:17:50.6327139Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-067xqowh 2024-06-26T07:17:52.0363827Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-06-26T07:17:52.0383065Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:17:53.2724126Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:17:53.4462119Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:17:55.7905188Z Preparing metadata (setup.py) ... [?25l- \ done 2024-06-26T07:17:55.7954852Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-06-26T07:17:55.7958781Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+gitb8c4c54) 2024-06-26T07:17:55.7964078Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-06-26T07:17:55.8175995Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-06-26T07:17:55.8181880Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-06-26T07:17:55.8185018Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1) 2024-06-26T07:17:55.8187993Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-06-26T07:17:55.8191201Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-06-26T07:17:55.8194244Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-06-26T07:17:55.8635983Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-06-26T07:17:55.8796974Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-06-26T07:17:55.8873091Z Building wheels for collected packages: torchvision 2024-06-26T07:19:08.6582873Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-06-26T07:19:08.6610201Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2112909 sha256=059a17b95d34e7564691b681cd827bf01167bc9cb28342d0d9af77709c8b5072 2024-06-26T07:19:08.6612251Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-06-26T07:19:08.6646194Z Successfully built torchvision 2024-06-26T07:19:09.8874849Z Installing collected packages: torchvision 2024-06-26T07:19:10.2682837Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-06-26T07:19:10.3791763Z + '[' -n '' ']' 2024-06-26T07:19:10.3792116Z + id=0 2024-06-26T07:19:10.3792546Z + pip_install opencv-python==4.8.0.74 2024-06-26T07:19:10.3796055Z + pip install --progress-bar off opencv-python==4.8.0.74 2024-06-26T07:19:10.8084484Z Collecting opencv-python==4.8.0.74 2024-06-26T07:19:10.8237218Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2024-06-26T07:19:10.8411180Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.21.2) 2024-06-26T07:19:10.8478436Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2024-06-26T07:19:12.6291415Z Installing collected packages: opencv-python 2024-06-26T07:19:13.4474913Z Successfully installed opencv-python-4.8.0.74 2024-06-26T07:19:13.5429204Z + [[ dynamic_aot_eager_torchbench == *inductor_torchbench_smoketest_perf* ]] 2024-06-26T07:19:13.5430294Z + [[ dynamic_aot_eager_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]] 2024-06-26T07:19:13.5431264Z + [[ dynamic_aot_eager_torchbench == *torchbench_gcp_smoketest* ]] 2024-06-26T07:19:13.5431970Z + checkout_install_torchbench 2024-06-26T07:19:13.5432408Z + local commit 2024-06-26T07:19:13.5433412Z ++ get_pinned_commit torchbench 2024-06-26T07:19:13.5434033Z ++ cat .github/ci_commit_pins/torchbench.txt 2024-06-26T07:19:13.5444150Z + commit=23512dbebd44a11eb84afbf53c3c071dd105297e 2024-06-26T07:19:13.5444842Z + git clone https://github.com/pytorch/benchmark torchbench 2024-06-26T07:19:13.5453513Z Cloning into 'torchbench'... 2024-06-26T07:19:13.6797539Z remote: Enumerating objects: 30187, done. 2024-06-26T07:19:13.6798248Z remote: Counting objects: 0% (1/546) 2024-06-26T07:19:13.6798929Z remote: Counting objects: 1% (6/546) 2024-06-26T07:19:13.6799450Z remote: Counting objects: 2% (11/546) 2024-06-26T07:19:13.6800090Z remote: Counting objects: 3% (17/546) 2024-06-26T07:19:13.6800730Z remote: Counting objects: 4% (22/546) 2024-06-26T07:19:13.6801373Z remote: Counting objects: 5% (28/546) 2024-06-26T07:19:13.6802092Z remote: Counting objects: 6% (33/546) 2024-06-26T07:19:13.6802785Z remote: Counting objects: 7% (39/546) 2024-06-26T07:19:13.6803332Z remote: Counting objects: 8% (44/546) 2024-06-26T07:19:13.6804066Z remote: Counting objects: 9% (50/546) 2024-06-26T07:19:13.6804966Z remote: Counting objects: 10% (55/546) 2024-06-26T07:19:13.6805705Z remote: Counting objects: 11% (61/546) 2024-06-26T07:19:13.6806434Z remote: Counting objects: 12% (66/546) 2024-06-26T07:19:13.6807454Z remote: Counting objects: 13% (71/546) 2024-06-26T07:19:13.6808165Z remote: Counting objects: 14% (77/546) 2024-06-26T07:19:13.6808754Z remote: Counting objects: 15% (82/546) 2024-06-26T07:19:13.6809258Z remote: Counting objects: 16% (88/546) 2024-06-26T07:19:13.6809761Z remote: Counting objects: 17% (93/546) 2024-06-26T07:19:13.6810267Z remote: Counting objects: 18% (99/546) 2024-06-26T07:19:13.6810789Z remote: Counting objects: 19% (104/546) 2024-06-26T07:19:13.6811496Z remote: Counting objects: 20% (110/546) 2024-06-26T07:19:13.6812219Z remote: Counting objects: 21% (115/546) 2024-06-26T07:19:13.6812897Z remote: Counting objects: 22% (121/546) 2024-06-26T07:19:13.6813598Z remote: Counting objects: 23% (126/546) 2024-06-26T07:19:13.6814131Z remote: Counting objects: 24% (132/546) 2024-06-26T07:19:13.6814645Z remote: Counting objects: 25% (137/546) 2024-06-26T07:19:13.6815159Z remote: Counting objects: 26% (142/546) 2024-06-26T07:19:13.6815675Z remote: Counting objects: 27% (148/546) 2024-06-26T07:19:13.6816200Z remote: Counting objects: 28% (153/546) 2024-06-26T07:19:13.6816707Z remote: Counting objects: 29% (159/546) 2024-06-26T07:19:13.6817215Z remote: Counting objects: 30% (164/546) 2024-06-26T07:19:13.6817727Z remote: Counting objects: 31% (170/546) 2024-06-26T07:19:13.6818241Z remote: Counting objects: 32% (175/546) 2024-06-26T07:19:13.6818759Z remote: Counting objects: 33% (181/546) 2024-06-26T07:19:13.6819272Z remote: Counting objects: 34% (186/546) 2024-06-26T07:19:13.6819785Z remote: Counting objects: 35% (192/546) 2024-06-26T07:19:13.6820293Z remote: Counting objects: 36% (197/546) 2024-06-26T07:19:13.6820804Z remote: Counting objects: 37% (203/546) 2024-06-26T07:19:13.6821325Z remote: Counting objects: 38% (208/546) 2024-06-26T07:19:13.6821841Z remote: Counting objects: 39% (213/546) 2024-06-26T07:19:13.6822347Z remote: Counting objects: 40% (219/546) 2024-06-26T07:19:13.6822882Z remote: Counting objects: 41% (224/546) 2024-06-26T07:19:13.6823390Z remote: Counting objects: 42% (230/546) 2024-06-26T07:19:13.6823905Z remote: Counting objects: 43% (235/546) 2024-06-26T07:19:13.6824423Z remote: Counting objects: 44% (241/546) 2024-06-26T07:19:13.6825033Z remote: Counting objects: 45% (246/546) 2024-06-26T07:19:13.6825559Z remote: Counting objects: 46% (252/546) 2024-06-26T07:19:13.6826083Z remote: Counting objects: 47% (257/546) 2024-06-26T07:19:13.6826598Z remote: Counting objects: 48% (263/546) 2024-06-26T07:19:13.6827109Z remote: Counting objects: 49% (268/546) 2024-06-26T07:19:13.6827636Z remote: Counting objects: 50% (273/546) 2024-06-26T07:19:13.6828154Z remote: Counting objects: 51% (279/546) 2024-06-26T07:19:13.6828675Z remote: Counting objects: 52% (284/546) 2024-06-26T07:19:13.6829355Z remote: Counting objects: 53% (290/546) 2024-06-26T07:19:13.6829891Z remote: Counting objects: 54% (295/546) 2024-06-26T07:19:13.6830423Z remote: Counting objects: 55% (301/546) 2024-06-26T07:19:13.6830941Z remote: Counting objects: 56% (306/546) 2024-06-26T07:19:13.6831477Z remote: Counting objects: 57% (312/546) 2024-06-26T07:19:13.6832008Z remote: Counting objects: 58% (317/546) 2024-06-26T07:19:13.6832532Z remote: Counting objects: 59% (323/546) 2024-06-26T07:19:13.6833047Z remote: Counting objects: 60% (328/546) 2024-06-26T07:19:13.6833567Z remote: Counting objects: 61% (334/546) 2024-06-26T07:19:13.6834087Z remote: Counting objects: 62% (339/546) 2024-06-26T07:19:13.6834609Z remote: Counting objects: 63% (344/546) 2024-06-26T07:19:13.6835125Z remote: Counting objects: 64% (350/546) 2024-06-26T07:19:13.6835653Z remote: Counting objects: 65% (355/546) 2024-06-26T07:19:13.6836256Z remote: Counting objects: 66% (361/546) 2024-06-26T07:19:13.6836791Z remote: Counting objects: 67% (366/546) 2024-06-26T07:19:13.6837303Z remote: Counting objects: 68% (372/546) 2024-06-26T07:19:13.6837827Z remote: Counting objects: 69% (377/546) 2024-06-26T07:19:13.6838357Z remote: Counting objects: 70% (383/546) 2024-06-26T07:19:13.6838878Z remote: Counting objects: 71% (388/546) 2024-06-26T07:19:13.6839412Z remote: Counting objects: 72% (394/546) 2024-06-26T07:19:13.6839936Z remote: Counting objects: 73% (399/546) 2024-06-26T07:19:13.6840455Z remote: Counting objects: 74% (405/546) 2024-06-26T07:19:13.6840974Z remote: Counting objects: 75% (410/546) 2024-06-26T07:19:13.6841504Z remote: Counting objects: 76% (415/546) 2024-06-26T07:19:13.6842040Z remote: Counting objects: 77% (421/546) 2024-06-26T07:19:13.6842572Z remote: Counting objects: 78% (426/546) 2024-06-26T07:19:13.6843093Z remote: Counting objects: 79% (432/546) 2024-06-26T07:19:13.6843632Z remote: Counting objects: 80% (437/546) 2024-06-26T07:19:13.6844162Z remote: Counting objects: 81% (443/546) 2024-06-26T07:19:13.6844685Z remote: Counting objects: 82% (448/546) 2024-06-26T07:19:13.6845204Z remote: Counting objects: 83% (454/546) 2024-06-26T07:19:13.6845732Z remote: Counting objects: 84% (459/546) 2024-06-26T07:19:13.6846261Z remote: Counting objects: 85% (465/546) 2024-06-26T07:19:13.6846930Z remote: Counting objects: 86% (470/546) 2024-06-26T07:19:13.6847448Z remote: Counting objects: 87% (476/546) 2024-06-26T07:19:13.6847980Z remote: Counting objects: 88% (481/546) 2024-06-26T07:19:13.6848508Z remote: Counting objects: 89% (486/546) 2024-06-26T07:19:13.6849027Z remote: Counting objects: 90% (492/546) 2024-06-26T07:19:13.6849555Z remote: Counting objects: 91% (497/546) 2024-06-26T07:19:13.6850080Z remote: Counting objects: 92% (503/546) 2024-06-26T07:19:13.6850602Z remote: Counting objects: 93% (508/546) 2024-06-26T07:19:13.6851123Z remote: Counting objects: 94% (514/546) 2024-06-26T07:19:13.6851646Z remote: Counting objects: 95% (519/546) 2024-06-26T07:19:13.6852169Z remote: Counting objects: 96% (525/546) 2024-06-26T07:19:13.6852699Z remote: Counting objects: 97% (530/546) 2024-06-26T07:19:13.6853222Z remote: Counting objects: 98% (536/546) 2024-06-26T07:19:13.6853755Z remote: Counting objects: 99% (541/546) 2024-06-26T07:19:13.6854291Z remote: Counting objects: 100% (546/546) 2024-06-26T07:19:13.6854860Z remote: Counting objects: 100% (546/546), done. 2024-06-26T07:19:13.6855448Z remote: Compressing objects: 0% (1/318) 2024-06-26T07:19:13.6856022Z remote: Compressing objects: 1% (4/318) 2024-06-26T07:19:13.6856594Z remote: Compressing objects: 2% (7/318) 2024-06-26T07:19:13.6857181Z remote: Compressing objects: 3% (10/318) 2024-06-26T07:19:13.6857745Z remote: Compressing objects: 4% (13/318) 2024-06-26T07:19:13.6858336Z remote: Compressing objects: 5% (16/318) 2024-06-26T07:19:13.6858897Z remote: Compressing objects: 6% (20/318) 2024-06-26T07:19:13.6859526Z remote: Compressing objects: 7% (23/318) 2024-06-26T07:19:13.6860078Z remote: Compressing objects: 8% (26/318) 2024-06-26T07:19:13.6860628Z remote: Compressing objects: 9% (29/318) 2024-06-26T07:19:13.6861181Z remote: Compressing objects: 10% (32/318) 2024-06-26T07:19:13.6861742Z remote: Compressing objects: 11% (35/318) 2024-06-26T07:19:13.6863878Z remote: Compressing objects: 12% (39/318) 2024-06-26T07:19:13.6866042Z remote: Compressing objects: 13% (42/318) 2024-06-26T07:19:13.6867670Z remote: Compressing objects: 14% (45/318) 2024-06-26T07:19:13.6868234Z remote: Compressing objects: 15% (48/318) 2024-06-26T07:19:13.6870951Z remote: Compressing objects: 16% (51/318) 2024-06-26T07:19:13.6871964Z remote: Compressing objects: 17% (55/318) 2024-06-26T07:19:13.6872706Z remote: Compressing objects: 18% (58/318) 2024-06-26T07:19:13.6873478Z remote: Compressing objects: 19% (61/318) 2024-06-26T07:19:13.6874332Z remote: Compressing objects: 20% (64/318) 2024-06-26T07:19:13.6874882Z remote: Compressing objects: 21% (67/318) 2024-06-26T07:19:13.6875423Z remote: Compressing objects: 22% (70/318) 2024-06-26T07:19:13.6875970Z remote: Compressing objects: 23% (74/318) 2024-06-26T07:19:13.6876503Z remote: Compressing objects: 24% (77/318) 2024-06-26T07:19:13.6877043Z remote: Compressing objects: 25% (80/318) 2024-06-26T07:19:13.6877624Z remote: Compressing objects: 26% (83/318) 2024-06-26T07:19:13.6878172Z remote: Compressing objects: 27% (86/318) 2024-06-26T07:19:13.6878709Z remote: Compressing objects: 28% (90/318) 2024-06-26T07:19:13.6879389Z remote: Compressing objects: 29% (93/318) 2024-06-26T07:19:13.6879994Z remote: Compressing objects: 30% (96/318) 2024-06-26T07:19:13.6880533Z remote: Compressing objects: 31% (99/318) 2024-06-26T07:19:13.6881086Z remote: Compressing objects: 32% (102/318) 2024-06-26T07:19:13.6881653Z remote: Compressing objects: 33% (105/318) 2024-06-26T07:19:13.6882337Z remote: Compressing objects: 34% (109/318) 2024-06-26T07:19:13.6883114Z remote: Compressing objects: 35% (112/318) 2024-06-26T07:19:13.6883851Z remote: Compressing objects: 36% (115/318) 2024-06-26T07:19:13.6884563Z remote: Compressing objects: 37% (118/318) 2024-06-26T07:19:13.6885111Z remote: Compressing objects: 38% (121/318) 2024-06-26T07:19:13.6885647Z remote: Compressing objects: 39% (125/318) 2024-06-26T07:19:13.6886191Z remote: Compressing objects: 40% (128/318) 2024-06-26T07:19:13.6886742Z remote: Compressing objects: 41% (131/318) 2024-06-26T07:19:13.6887532Z remote: Compressing objects: 42% (134/318) 2024-06-26T07:19:13.6888084Z remote: Compressing objects: 43% (137/318) 2024-06-26T07:19:13.6888645Z remote: Compressing objects: 44% (140/318) 2024-06-26T07:19:13.6889196Z remote: Compressing objects: 45% (144/318) 2024-06-26T07:19:13.6889751Z remote: Compressing objects: 46% (147/318) 2024-06-26T07:19:13.6890315Z remote: Compressing objects: 47% (150/318) 2024-06-26T07:19:13.6890869Z remote: Compressing objects: 48% (153/318) 2024-06-26T07:19:13.6891424Z remote: Compressing objects: 49% (156/318) 2024-06-26T07:19:13.6891974Z remote: Compressing objects: 50% (159/318) 2024-06-26T07:19:13.6892519Z remote: Compressing objects: 51% (163/318) 2024-06-26T07:19:13.6893070Z remote: Compressing objects: 52% (166/318) 2024-06-26T07:19:13.6893617Z remote: Compressing objects: 53% (169/318) 2024-06-26T07:19:13.6894168Z remote: Compressing objects: 54% (172/318) 2024-06-26T07:19:13.6894721Z remote: Compressing objects: 55% (175/318) 2024-06-26T07:19:13.6895268Z remote: Compressing objects: 56% (179/318) 2024-06-26T07:19:13.6895818Z remote: Compressing objects: 57% (182/318) 2024-06-26T07:19:13.6896367Z remote: Compressing objects: 58% (185/318) 2024-06-26T07:19:13.6896911Z remote: Compressing objects: 59% (188/318) 2024-06-26T07:19:13.6897467Z remote: Compressing objects: 60% (191/318) 2024-06-26T07:19:13.6898198Z remote: Compressing objects: 61% (194/318) 2024-06-26T07:19:13.6898759Z remote: Compressing objects: 62% (198/318) 2024-06-26T07:19:13.6899308Z remote: Compressing objects: 63% (201/318) 2024-06-26T07:19:13.6899867Z remote: Compressing objects: 64% (204/318) 2024-06-26T07:19:13.6900414Z remote: Compressing objects: 65% (207/318) 2024-06-26T07:19:13.6900962Z remote: Compressing objects: 66% (210/318) 2024-06-26T07:19:13.6901503Z remote: Compressing objects: 67% (214/318) 2024-06-26T07:19:13.6902051Z remote: Compressing objects: 68% (217/318) 2024-06-26T07:19:13.6902607Z remote: Compressing objects: 69% (220/318) 2024-06-26T07:19:13.6903167Z remote: Compressing objects: 70% (223/318) 2024-06-26T07:19:13.6903892Z remote: Compressing objects: 71% (226/318) 2024-06-26T07:19:13.6904607Z remote: Compressing objects: 72% (229/318) 2024-06-26T07:19:13.6905348Z remote: Compressing objects: 73% (233/318) 2024-06-26T07:19:13.6906194Z remote: Compressing objects: 74% (236/318) 2024-06-26T07:19:13.6906888Z remote: Compressing objects: 75% (239/318) 2024-06-26T07:19:13.6907621Z remote: Compressing objects: 76% (242/318) 2024-06-26T07:19:13.6908357Z remote: Compressing objects: 77% (245/318) 2024-06-26T07:19:13.6909097Z remote: Compressing objects: 78% (249/318) 2024-06-26T07:19:13.6909859Z remote: Compressing objects: 79% (252/318) 2024-06-26T07:19:13.6910617Z remote: Compressing objects: 80% (255/318) 2024-06-26T07:19:13.6911364Z remote: Compressing objects: 81% (258/318) 2024-06-26T07:19:13.6912125Z remote: Compressing objects: 82% (261/318) 2024-06-26T07:19:13.6912856Z remote: Compressing objects: 83% (264/318) 2024-06-26T07:19:13.6913575Z remote: Compressing objects: 84% (268/318) 2024-06-26T07:19:13.6914298Z remote: Compressing objects: 85% (271/318) 2024-06-26T07:19:13.6915045Z remote: Compressing objects: 86% (274/318) 2024-06-26T07:19:13.6915807Z remote: Compressing objects: 87% (277/318) 2024-06-26T07:19:13.6916600Z remote: Compressing objects: 88% (280/318) 2024-06-26T07:19:13.6917334Z remote: Compressing objects: 89% (284/318) 2024-06-26T07:19:13.6918096Z remote: Compressing objects: 90% (287/318) 2024-06-26T07:19:13.6918861Z remote: Compressing objects: 91% (290/318) 2024-06-26T07:19:13.6919685Z remote: Compressing objects: 92% (293/318) 2024-06-26T07:19:13.6920482Z remote: Compressing objects: 93% (296/318) 2024-06-26T07:19:13.6921268Z remote: Compressing objects: 94% (299/318) 2024-06-26T07:19:13.6922100Z remote: Compressing objects: 95% (303/318) 2024-06-26T07:19:13.6922927Z remote: Compressing objects: 96% (306/318) 2024-06-26T07:19:13.6923739Z remote: Compressing objects: 97% (309/318) 2024-06-26T07:19:13.6924529Z remote: Compressing objects: 98% (312/318) 2024-06-26T07:19:13.6925323Z remote: Compressing objects: 99% (315/318) 2024-06-26T07:19:13.6926128Z remote: Compressing objects: 100% (318/318) 2024-06-26T07:19:13.6927178Z remote: Compressing objects: 100% (318/318), done. 2024-06-26T07:19:13.6970597Z Receiving objects: 0% (1/30187) 2024-06-26T07:19:13.7008434Z Receiving objects: 1% (302/30187) 2024-06-26T07:19:13.7046609Z Receiving objects: 2% (604/30187) 2024-06-26T07:19:13.7080110Z Receiving objects: 3% (906/30187) 2024-06-26T07:19:13.7109014Z Receiving objects: 4% (1208/30187) 2024-06-26T07:19:13.7141799Z Receiving objects: 5% (1510/30187) 2024-06-26T07:19:13.7174007Z Receiving objects: 6% (1812/30187) 2024-06-26T07:19:13.7212878Z Receiving objects: 7% (2114/30187) 2024-06-26T07:19:13.7245512Z Receiving objects: 8% (2415/30187) 2024-06-26T07:19:13.7276089Z Receiving objects: 9% (2717/30187) 2024-06-26T07:19:13.7312944Z Receiving objects: 10% (3019/30187) 2024-06-26T07:19:13.7375292Z Receiving objects: 11% (3321/30187) 2024-06-26T07:19:13.7439919Z Receiving objects: 12% (3623/30187) 2024-06-26T07:19:13.7496852Z Receiving objects: 13% (3925/30187) 2024-06-26T07:19:13.7670229Z Receiving objects: 14% (4227/30187) 2024-06-26T07:19:13.7774292Z Receiving objects: 15% (4529/30187) 2024-06-26T07:19:13.7862236Z Receiving objects: 16% (4830/30187) 2024-06-26T07:19:13.7916856Z Receiving objects: 17% (5132/30187) 2024-06-26T07:19:13.7954529Z Receiving objects: 18% (5434/30187) 2024-06-26T07:19:13.7982632Z Receiving objects: 19% (5736/30187) 2024-06-26T07:19:13.8052466Z Receiving objects: 20% (6038/30187) 2024-06-26T07:19:13.8071480Z Receiving objects: 21% (6340/30187) 2024-06-26T07:19:13.8090260Z Receiving objects: 22% (6642/30187) 2024-06-26T07:19:13.8912333Z Receiving objects: 23% (6944/30187) 2024-06-26T07:19:14.6923734Z Receiving objects: 24% (7245/30187) 2024-06-26T07:19:14.8348304Z Receiving objects: 24% (7445/30187), 76.73 MiB | 76.73 MiB/s 2024-06-26T07:19:15.3392263Z Receiving objects: 25% (7547/30187), 76.73 MiB | 76.73 MiB/s 2024-06-26T07:19:15.4072655Z Receiving objects: 26% (7849/30187), 114.68 MiB | 76.45 MiB/s 2024-06-26T07:19:15.4746366Z Receiving objects: 27% (8151/30187), 114.68 MiB | 76.45 MiB/s 2024-06-26T07:19:15.5588372Z Receiving objects: 28% (8453/30187), 114.68 MiB | 76.45 MiB/s 2024-06-26T07:19:15.6356934Z Receiving objects: 29% (8755/30187), 114.68 MiB | 76.45 MiB/s 2024-06-26T07:19:15.6922978Z Receiving objects: 30% (9057/30187), 114.68 MiB | 76.45 MiB/s 2024-06-26T07:19:15.7186540Z Receiving objects: 30% (9262/30187), 151.80 MiB | 75.90 MiB/s 2024-06-26T07:19:15.7982460Z Receiving objects: 31% (9358/30187), 151.80 MiB | 75.90 MiB/s 2024-06-26T07:19:15.8762527Z Receiving objects: 32% (9660/30187), 151.80 MiB | 75.90 MiB/s 2024-06-26T07:19:16.1077935Z Receiving objects: 33% (9962/30187), 151.80 MiB | 75.90 MiB/s 2024-06-26T07:19:16.1814874Z Receiving objects: 34% (10264/30187), 151.80 MiB | 75.90 MiB/s 2024-06-26T07:19:16.1942654Z Receiving objects: 35% (10566/30187), 151.80 MiB | 75.90 MiB/s 2024-06-26T07:19:16.3942876Z Receiving objects: 36% (10868/30187), 184.64 MiB | 73.86 MiB/s 2024-06-26T07:19:16.6724023Z Receiving objects: 37% (11170/30187), 184.64 MiB | 73.86 MiB/s 2024-06-26T07:19:16.6922352Z Receiving objects: 38% (11472/30187), 184.64 MiB | 73.86 MiB/s 2024-06-26T07:19:16.6932466Z Receiving objects: 38% (11586/30187), 184.64 MiB | 73.86 MiB/s 2024-06-26T07:19:16.6946772Z Receiving objects: 39% (11773/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.6972584Z Receiving objects: 40% (12075/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.6985789Z Receiving objects: 41% (12377/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.6999706Z Receiving objects: 42% (12679/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7011694Z Receiving objects: 43% (12981/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7022815Z Receiving objects: 44% (13283/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7045211Z Receiving objects: 45% (13585/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7057365Z Receiving objects: 46% (13887/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7062391Z Receiving objects: 47% (14188/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7069295Z Receiving objects: 48% (14490/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7090899Z Receiving objects: 49% (14792/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7107348Z Receiving objects: 50% (15094/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7117687Z Receiving objects: 51% (15396/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7332148Z Receiving objects: 52% (15698/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7337247Z Receiving objects: 53% (16000/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7345229Z Receiving objects: 54% (16301/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7351165Z Receiving objects: 55% (16603/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7357442Z Receiving objects: 56% (16905/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7374114Z Receiving objects: 57% (17207/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7384436Z Receiving objects: 58% (17509/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7394120Z Receiving objects: 59% (17811/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7407526Z Receiving objects: 60% (18113/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7416074Z Receiving objects: 61% (18415/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7426246Z Receiving objects: 62% (18716/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7560440Z Receiving objects: 63% (19018/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7566721Z Receiving objects: 64% (19320/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7573360Z Receiving objects: 65% (19622/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7585777Z Receiving objects: 66% (19924/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7591017Z Receiving objects: 67% (20226/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7598680Z Receiving objects: 68% (20528/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.7847214Z Receiving objects: 69% (20830/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.8241494Z Receiving objects: 70% (21131/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.8248793Z Receiving objects: 71% (21433/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.8261560Z Receiving objects: 72% (21735/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.8274967Z Receiving objects: 73% (22037/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.8288060Z Receiving objects: 74% (22339/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.8298831Z Receiving objects: 75% (22641/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.8307617Z Receiving objects: 76% (22943/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.8325952Z Receiving objects: 77% (23244/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.8332400Z Receiving objects: 78% (23546/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.8340694Z Receiving objects: 79% (23848/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:16.8363528Z Receiving objects: 80% (24150/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:17.0339603Z Receiving objects: 81% (24452/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:17.0553675Z Receiving objects: 82% (24754/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:17.0586715Z Receiving objects: 83% (25056/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:17.0596841Z Receiving objects: 84% (25358/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:17.0609686Z Receiving objects: 85% (25659/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:17.0642611Z Receiving objects: 86% (25961/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:17.0675082Z Receiving objects: 87% (26263/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:17.0843230Z Receiving objects: 88% (26565/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:17.0849772Z Receiving objects: 89% (26867/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:17.0857617Z Receiving objects: 90% (27169/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:17.0864034Z Receiving objects: 91% (27471/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:17.0871976Z Receiving objects: 92% (27773/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:17.0878689Z Receiving objects: 93% (28074/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:17.0994075Z Receiving objects: 94% (28376/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:17.1009899Z Receiving objects: 95% (28678/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:17.1020469Z Receiving objects: 96% (28980/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:17.1040048Z Receiving objects: 97% (29282/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:17.1055283Z Receiving objects: 98% (29584/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:17.1098717Z Receiving objects: 99% (29886/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:17.1099674Z remote: Total 30187 (delta 256), reused 486 (delta 216), pack-reused 29641 2024-06-26T07:19:17.1117160Z Receiving objects: 100% (30187/30187), 221.86 MiB | 73.95 MiB/s 2024-06-26T07:19:17.1117868Z Receiving objects: 100% (30187/30187), 245.25 MiB | 71.73 MiB/s, done. 2024-06-26T07:19:17.1146891Z Resolving deltas: 0% (0/16205) 2024-06-26T07:19:17.1155091Z Resolving deltas: 1% (163/16205) 2024-06-26T07:19:17.1170384Z Resolving deltas: 2% (328/16205) 2024-06-26T07:19:17.1185063Z Resolving deltas: 3% (515/16205) 2024-06-26T07:19:17.1206163Z Resolving deltas: 4% (650/16205) 2024-06-26T07:19:17.1216789Z Resolving deltas: 5% (821/16205) 2024-06-26T07:19:17.1225406Z Resolving deltas: 6% (973/16205) 2024-06-26T07:19:17.1236303Z Resolving deltas: 7% (1140/16205) 2024-06-26T07:19:17.1245195Z Resolving deltas: 8% (1303/16205) 2024-06-26T07:19:17.1254564Z Resolving deltas: 9% (1473/16205) 2024-06-26T07:19:17.1265719Z Resolving deltas: 10% (1633/16205) 2024-06-26T07:19:17.1276792Z Resolving deltas: 11% (1785/16205) 2024-06-26T07:19:17.1285471Z Resolving deltas: 12% (1948/16205) 2024-06-26T07:19:17.1298761Z Resolving deltas: 13% (2110/16205) 2024-06-26T07:19:17.1308107Z Resolving deltas: 14% (2269/16205) 2024-06-26T07:19:17.1326040Z Resolving deltas: 15% (2438/16205) 2024-06-26T07:19:17.1342464Z Resolving deltas: 16% (2620/16205) 2024-06-26T07:19:17.1359127Z Resolving deltas: 17% (2762/16205) 2024-06-26T07:19:17.1361401Z Resolving deltas: 18% (3046/16205) 2024-06-26T07:19:17.1369141Z Resolving deltas: 19% (3094/16205) 2024-06-26T07:19:17.1388418Z Resolving deltas: 20% (3268/16205) 2024-06-26T07:19:17.1415780Z Resolving deltas: 21% (3416/16205) 2024-06-26T07:19:17.1442131Z Resolving deltas: 22% (3566/16205) 2024-06-26T07:19:17.1450982Z Resolving deltas: 24% (3890/16205) 2024-06-26T07:19:17.1459836Z Resolving deltas: 25% (4062/16205) 2024-06-26T07:19:17.1470305Z Resolving deltas: 26% (4230/16205) 2024-06-26T07:19:17.1478770Z Resolving deltas: 27% (4410/16205) 2024-06-26T07:19:17.1520757Z Resolving deltas: 28% (4545/16205) 2024-06-26T07:19:17.1539822Z Resolving deltas: 29% (4721/16205) 2024-06-26T07:19:17.1553534Z Resolving deltas: 30% (4863/16205) 2024-06-26T07:19:17.1567506Z Resolving deltas: 31% (5028/16205) 2024-06-26T07:19:17.1582626Z Resolving deltas: 32% (5200/16205) 2024-06-26T07:19:17.1597712Z Resolving deltas: 33% (5350/16205) 2024-06-26T07:19:17.1607044Z Resolving deltas: 34% (5533/16205) 2024-06-26T07:19:17.1616157Z Resolving deltas: 35% (5705/16205) 2024-06-26T07:19:17.1627339Z Resolving deltas: 36% (5842/16205) 2024-06-26T07:19:17.1647855Z Resolving deltas: 37% (6004/16205) 2024-06-26T07:19:17.1661274Z Resolving deltas: 38% (6170/16205) 2024-06-26T07:19:17.1680270Z Resolving deltas: 39% (6321/16205) 2024-06-26T07:19:17.1687502Z Resolving deltas: 40% (6490/16205) 2024-06-26T07:19:17.1708554Z Resolving deltas: 41% (6659/16205) 2024-06-26T07:19:17.1718759Z Resolving deltas: 42% (6822/16205) 2024-06-26T07:19:17.1725577Z Resolving deltas: 43% (7012/16205) 2024-06-26T07:19:17.1749153Z Resolving deltas: 44% (7161/16205) 2024-06-26T07:19:17.1754492Z Resolving deltas: 45% (7360/16205) 2024-06-26T07:19:17.1762348Z Resolving deltas: 46% (7493/16205) 2024-06-26T07:19:17.1773497Z Resolving deltas: 47% (7643/16205) 2024-06-26T07:19:17.1794445Z Resolving deltas: 48% (7779/16205) 2024-06-26T07:19:17.1814229Z Resolving deltas: 49% (7943/16205) 2024-06-26T07:19:17.1826452Z Resolving deltas: 50% (8106/16205) 2024-06-26T07:19:17.1851224Z Resolving deltas: 51% (8265/16205) 2024-06-26T07:19:17.1864334Z Resolving deltas: 52% (8446/16205) 2024-06-26T07:19:17.1871578Z Resolving deltas: 53% (8594/16205) 2024-06-26T07:19:17.1881613Z Resolving deltas: 54% (8752/16205) 2024-06-26T07:19:17.1895066Z Resolving deltas: 55% (8944/16205) 2024-06-26T07:19:17.1903749Z Resolving deltas: 56% (9089/16205) 2024-06-26T07:19:17.1919598Z Resolving deltas: 57% (9239/16205) 2024-06-26T07:19:17.1927998Z Resolving deltas: 58% (9411/16205) 2024-06-26T07:19:17.1938055Z Resolving deltas: 59% (9613/16205) 2024-06-26T07:19:17.1939955Z Resolving deltas: 60% (9871/16205) 2024-06-26T07:19:17.1951185Z Resolving deltas: 61% (9894/16205) 2024-06-26T07:19:17.1966151Z Resolving deltas: 62% (10049/16205) 2024-06-26T07:19:17.1978063Z Resolving deltas: 63% (10235/16205) 2024-06-26T07:19:17.1981368Z Resolving deltas: 64% (10466/16205) 2024-06-26T07:19:17.1996855Z Resolving deltas: 65% (10536/16205) 2024-06-26T07:19:17.2014548Z Resolving deltas: 66% (10741/16205) 2024-06-26T07:19:17.2037778Z Resolving deltas: 67% (10957/16205) 2024-06-26T07:19:17.2057114Z Resolving deltas: 69% (11248/16205) 2024-06-26T07:19:17.2065924Z Resolving deltas: 70% (11381/16205) 2024-06-26T07:19:17.2074041Z Resolving deltas: 71% (11509/16205) 2024-06-26T07:19:17.2087995Z Resolving deltas: 72% (11681/16205) 2024-06-26T07:19:17.2100410Z Resolving deltas: 73% (11866/16205) 2024-06-26T07:19:17.2111947Z Resolving deltas: 74% (11992/16205) 2024-06-26T07:19:17.2139335Z Resolving deltas: 75% (12154/16205) 2024-06-26T07:19:17.2142285Z Resolving deltas: 77% (12598/16205) 2024-06-26T07:19:17.2148891Z Resolving deltas: 78% (12674/16205) 2024-06-26T07:19:17.2156508Z Resolving deltas: 79% (12840/16205) 2024-06-26T07:19:17.2163948Z Resolving deltas: 80% (12983/16205) 2024-06-26T07:19:17.2173950Z Resolving deltas: 81% (13135/16205) 2024-06-26T07:19:17.2182350Z Resolving deltas: 82% (13294/16205) 2024-06-26T07:19:17.2188791Z Resolving deltas: 83% (13451/16205) 2024-06-26T07:19:17.2199106Z Resolving deltas: 84% (13618/16205) 2024-06-26T07:19:17.2220178Z Resolving deltas: 85% (13776/16205) 2024-06-26T07:19:17.2228911Z Resolving deltas: 86% (13937/16205) 2024-06-26T07:19:17.2236851Z Resolving deltas: 87% (14101/16205) 2024-06-26T07:19:17.2243950Z Resolving deltas: 88% (14279/16205) 2024-06-26T07:19:17.2251679Z Resolving deltas: 89% (14449/16205) 2024-06-26T07:19:17.2260507Z Resolving deltas: 90% (14609/16205) 2024-06-26T07:19:17.2267454Z Resolving deltas: 91% (14765/16205) 2024-06-26T07:19:17.2277117Z Resolving deltas: 92% (14917/16205) 2024-06-26T07:19:17.2280289Z Resolving deltas: 93% (15139/16205) 2024-06-26T07:19:17.2288642Z Resolving deltas: 94% (15252/16205) 2024-06-26T07:19:17.2295409Z Resolving deltas: 95% (15428/16205) 2024-06-26T07:19:17.2302311Z Resolving deltas: 96% (15560/16205) 2024-06-26T07:19:17.2309369Z Resolving deltas: 97% (15719/16205) 2024-06-26T07:19:17.2317401Z Resolving deltas: 98% (15887/16205) 2024-06-26T07:19:17.2359643Z Resolving deltas: 99% (16052/16205) 2024-06-26T07:19:17.2360208Z Resolving deltas: 100% (16205/16205) 2024-06-26T07:19:17.2360662Z Resolving deltas: 100% (16205/16205), done. 2024-06-26T07:19:18.2921048Z + pushd torchbench 2024-06-26T07:19:18.2921516Z ~/workspace/torchbench ~/workspace 2024-06-26T07:19:18.2922054Z + git checkout 23512dbebd44a11eb84afbf53c3c071dd105297e 2024-06-26T07:19:18.3492253Z Note: switching to '23512dbebd44a11eb84afbf53c3c071dd105297e'. 2024-06-26T07:19:18.3492749Z 2024-06-26T07:19:18.3493251Z You are in 'detached HEAD' state. You can look around, make experimental 2024-06-26T07:19:18.3494366Z changes and commit them, and you can discard any commits you make in this 2024-06-26T07:19:18.3495193Z state without impacting any branches by switching back to a branch. 2024-06-26T07:19:18.3495697Z 2024-06-26T07:19:18.3496046Z If you want to create a new branch to retain commits you create, you may 2024-06-26T07:19:18.3496896Z do so (now or later) by using -c with the switch command. Example: 2024-06-26T07:19:18.3497347Z 2024-06-26T07:19:18.3497541Z git switch -c 2024-06-26T07:19:18.3497828Z 2024-06-26T07:19:18.3497974Z Or undo this operation with: 2024-06-26T07:19:18.3498240Z 2024-06-26T07:19:18.3498361Z git switch - 2024-06-26T07:19:18.3498560Z 2024-06-26T07:19:18.3498914Z Turn off this advice by setting config variable advice.detachedHead to false 2024-06-26T07:19:18.3499424Z 2024-06-26T07:19:18.3499606Z HEAD is now at 23512dbe Pin numpy version 2024-06-26T07:19:18.3500073Z + '[' '' ']' 2024-06-26T07:19:18.3500443Z + python install.py --continue_on_fail 2024-06-26T07:19:20.3084135Z checking packages torch, torchvision, torchaudio are installed...OK 2024-06-26T07:19:35.7778038Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2024-06-26T07:19:38.1529329Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2024-06-26T07:19:40.8174202Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2024-06-26T07:19:43.1311535Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2024-06-26T07:19:43.1467838Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2024-06-26T07:19:51.9094353Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2024-06-26T07:19:55.2863517Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2024-06-26T07:19:58.7071011Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2024-06-26T07:20:02.1160069Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2024-06-26T07:20:02.1162834Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2024-06-26T07:20:03.9749815Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2024-06-26T07:20:07.5170920Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2024-06-26T07:20:07.5317259Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2024-06-26T07:21:02.0740451Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2024-06-26T07:21:10.5996049Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2024-06-26T07:21:18.1799544Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2024-06-26T07:21:25.7018612Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2024-06-26T07:21:34.2676208Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2024-06-26T07:21:42.1193140Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2024-06-26T07:21:49.0068215Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2024-06-26T07:21:56.5043244Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2024-06-26T07:22:04.0243380Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2024-06-26T07:22:11.9895129Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2024-06-26T07:22:19.4074795Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2024-06-26T07:22:26.9091213Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2024-06-26T07:22:30.9085599Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2024-06-26T07:23:08.7239232Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2024-06-26T07:23:14.8415777Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2024-06-26T07:23:21.5016939Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2024-06-26T07:23:32.5507507Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2024-06-26T07:23:34.8121879Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2024-06-26T07:23:37.3206637Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2024-06-26T07:23:45.2378660Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2024-06-26T07:23:53.3270277Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2024-06-26T07:24:00.0402860Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2024-06-26T07:24:09.4390018Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2024-06-26T07:24:16.7512565Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2024-06-26T07:24:23.0514419Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2024-06-26T07:24:30.4004992Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2024-06-26T07:24:46.6486506Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2024-06-26T07:24:53.9739914Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2024-06-26T07:24:59.3090801Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2024-06-26T07:25:05.3939123Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2024-06-26T07:25:13.2380109Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2024-06-26T07:25:13.2382748Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2024-06-26T07:25:26.5659875Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2024-06-26T07:25:31.8828854Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2024-06-26T07:25:31.8830528Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2024-06-26T07:25:40.4976839Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2024-06-26T07:25:42.8023182Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2024-06-26T07:25:45.1479263Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2024-06-26T07:26:24.9351470Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2024-06-26T07:27:44.9236130Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2024-06-26T07:27:44.9390869Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2024-06-26T07:27:47.5100182Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2024-06-26T07:27:47.5257906Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2024-06-26T07:27:47.5414148Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2024-06-26T07:27:47.5568633Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2024-06-26T07:27:47.5719579Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2024-06-26T07:27:47.5872846Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2024-06-26T07:28:08.2020440Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2024-06-26T07:28:08.2023281Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2024-06-26T07:28:10.5497992Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2024-06-26T07:28:13.9343169Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2024-06-26T07:28:13.9499362Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2024-06-26T07:28:13.9655517Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2024-06-26T07:28:13.9808152Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2024-06-26T07:28:13.9954659Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2024-06-26T07:28:14.0107935Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2024-06-26T07:28:21.2321877Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2024-06-26T07:28:23.7962977Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2024-06-26T07:28:28.3802926Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2024-06-26T07:28:28.3959992Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2024-06-26T07:28:28.4118293Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2024-06-26T07:28:28.4269248Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2024-06-26T07:28:28.4423390Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2024-06-26T07:28:28.4576331Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2024-06-26T07:28:41.1194240Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2024-06-26T07:28:52.2373846Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2024-06-26T07:28:52.2531963Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2024-06-26T07:28:52.2534031Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2024-06-26T07:28:52.2537034Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2024-06-26T07:28:55.9925813Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2024-06-26T07:28:58.8754157Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2024-06-26T07:28:58.8910020Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2024-06-26T07:29:21.2724963Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2024-06-26T07:29:28.1764093Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2024-06-26T07:29:33.8460196Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2024-06-26T07:29:41.0952783Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2024-06-26T07:29:41.1114433Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2024-06-26T07:29:41.1275935Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2024-06-26T07:29:41.1431857Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2024-06-26T07:29:41.1586710Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2024-06-26T07:29:41.1743271Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2024-06-26T07:29:41.1896056Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2024-06-26T07:29:41.2051293Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2024-06-26T07:29:52.8913883Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2024-06-26T07:30:05.7573575Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2024-06-26T07:30:05.7731146Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2024-06-26T07:30:08.7754892Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2024-06-26T07:30:12.0146119Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2024-06-26T07:30:12.4343294Z + popd 2024-06-26T07:30:12.4343704Z ~/workspace 2024-06-26T07:30:12.4344387Z + [[ dynamic_aot_eager_torchbench != *cpu_inductor* ]] 2024-06-26T07:30:12.4344964Z + install_torchrec_and_fbgemm 2024-06-26T07:30:12.4345413Z + local torchrec_commit 2024-06-26T07:30:12.4347242Z ++ get_pinned_commit torchrec 2024-06-26T07:30:12.4347984Z ++ cat .github/ci_commit_pins/torchrec.txt 2024-06-26T07:30:12.4399135Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T07:30:12.4399672Z + local fbgemm_commit 2024-06-26T07:30:12.4401346Z ++ get_pinned_commit fbgemm 2024-06-26T07:30:12.4401833Z ++ cat .github/ci_commit_pins/fbgemm.txt 2024-06-26T07:30:12.4411545Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:30:12.4412352Z + pip_uninstall torchrec-nightly 2024-06-26T07:30:12.4412833Z + pip uninstall -y torchrec-nightly 2024-06-26T07:30:12.7991739Z WARNING: Skipping torchrec-nightly as it is not installed. 2024-06-26T07:30:12.8405864Z + pip_uninstall fbgemm-gpu-nightly 2024-06-26T07:30:12.8406404Z + pip uninstall -y fbgemm-gpu-nightly 2024-06-26T07:30:13.1872861Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed. 2024-06-26T07:30:13.2290524Z + pip_install setuptools-git-versioning scikit-build pyre-extensions 2024-06-26T07:30:13.2292493Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions 2024-06-26T07:30:13.6678118Z Collecting setuptools-git-versioning 2024-06-26T07:30:13.6845430Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB) 2024-06-26T07:30:13.7784756Z Collecting scikit-build 2024-06-26T07:30:13.7815618Z Downloading scikit_build-0.18.0-py3-none-any.whl.metadata (17 kB) 2024-06-26T07:30:13.8837257Z Collecting pyre-extensions 2024-06-26T07:30:13.8867708Z Downloading pyre_extensions-0.0.30-py3-none-any.whl.metadata (4.0 kB) 2024-06-26T07:30:13.9306951Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.1) 2024-06-26T07:30:13.9309309Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (69.5.1) 2024-06-26T07:30:13.9791129Z Collecting toml>=0.10.2 (from setuptools-git-versioning) 2024-06-26T07:30:13.9821687Z Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) 2024-06-26T07:30:14.0403662Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0) 2024-06-26T07:30:14.0414483Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1) 2024-06-26T07:30:14.0419169Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.43.0) 2024-06-26T07:30:14.0934593Z Collecting typing-inspect (from pyre-extensions) 2024-06-26T07:30:14.0966520Z Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB) 2024-06-26T07:30:14.1397139Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.2) 2024-06-26T07:30:14.1628460Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0) 2024-06-26T07:30:14.1674024Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB) 2024-06-26T07:30:14.2231141Z Downloading scikit_build-0.18.0-py3-none-any.whl (85 kB) 2024-06-26T07:30:14.2859717Z Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB) 2024-06-26T07:30:14.3420049Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2024-06-26T07:30:14.4004249Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2024-06-26T07:30:16.6101990Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions 2024-06-26T07:30:16.9468258Z Successfully installed pyre-extensions-0.0.30 scikit-build-0.18.0 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0 2024-06-26T07:30:17.0647581Z + CUDA_PATH=/usr/local/cuda-12.1 2024-06-26T07:30:17.0648853Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-06-26T07:30:17.0651043Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-06-26T07:30:17.4021033Z Collecting fbgemm-gpu 2024-06-26T07:30:17.4023219Z Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-3r0c8hwl/fbgemm-gpu_3561a83626fd4463a6591ae35543106c 2024-06-26T07:30:17.4039021Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-3r0c8hwl/fbgemm-gpu_3561a83626fd4463a6591ae35543106c 2024-06-26T07:30:18.3319877Z Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd' 2024-06-26T07:30:18.3334386Z Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:30:18.7319213Z Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:30:19.1888371Z Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:30:19.1890382Z Running command git submodule update --init --recursive -q 2024-06-26T07:30:26.1314035Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:30:26.1337968Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.21.2) 2024-06-26T07:30:26.1345622Z Building wheels for collected packages: fbgemm-gpu 2024-06-26T08:12:38.1673999Z Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done 2024-06-26T08:12:38.3474020Z [?25h Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=252628182 sha256=b161b2525e83a797c0d8b2e79880f7ba2eb3958331fac3bccdc92b4f6b936fbf 2024-06-26T08:12:38.3476096Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39 2024-06-26T08:12:38.3492541Z Successfully built fbgemm-gpu 2024-06-26T08:12:40.5547158Z Installing collected packages: fbgemm-gpu 2024-06-26T08:12:43.6977287Z Successfully installed fbgemm-gpu-0.4.1rc0.post421 2024-06-26T08:12:44.3176392Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:44.3177973Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:44.6566203Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:44.6570807Z Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-coordy0c 2024-06-26T08:12:44.6591232Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-coordy0c 2024-06-26T08:12:45.2675057Z Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2' 2024-06-26T08:12:45.2693512Z Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:45.5573349Z Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:45.8621489Z Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:46.0966319Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T08:12:46.1018848Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9) 2024-06-26T08:12:46.1021991Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3) 2024-06-26T08:12:46.1026602Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0) 2024-06-26T08:12:46.1556087Z Collecting torchmetrics (from torchrec==0.3.2) 2024-06-26T08:12:46.1708081Z Downloading torchmetrics-1.4.0.post0-py3-none-any.whl.metadata (19 kB) 2024-06-26T08:12:46.1763311Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.4) 2024-06-26T08:12:46.1767251Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421) 2024-06-26T08:12:46.1780883Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.21.2) 2024-06-26T08:12:46.1802284Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.10.0) 2024-06-26T08:12:46.2490265Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0) 2024-06-26T08:12:46.2494912Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-06-26T08:12:46.2498762Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-06-26T08:12:46.3284910Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.1) 2024-06-26T08:12:46.3290040Z Requirement already satisfied: torch>=1.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.5.0a0+gitb8c4c54) 2024-06-26T08:12:46.3471114Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2) 2024-06-26T08:12:46.3496347Z Downloading lightning_utilities-0.11.2-py3-none-any.whl.metadata (4.7 kB) 2024-06-26T08:12:46.3660515Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (69.5.1) 2024-06-26T08:12:46.3663231Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.2) 2024-06-26T08:12:46.3697075Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0) 2024-06-26T08:12:46.3768247Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.13.1) 2024-06-26T08:12:46.3772248Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.12.1) 2024-06-26T08:12:46.3775568Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.8.8) 2024-06-26T08:12:46.3778682Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.1.4) 2024-06-26T08:12:46.3781881Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2024.2.0) 2024-06-26T08:12:46.4453246Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.1.5) 2024-06-26T08:12:46.4770596Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.3.0) 2024-06-26T08:12:46.4911191Z Downloading torchmetrics-1.4.0.post0-py3-none-any.whl (868 kB) 2024-06-26T08:12:46.5073221Z Downloading lightning_utilities-0.11.2-py3-none-any.whl (26 kB) 2024-06-26T08:12:46.5163663Z Building wheels for collected packages: torchrec 2024-06-26T08:12:46.9513452Z Building wheel for torchrec (setup.py) ... [?25l- \ | done 2024-06-26T08:12:46.9524347Z [?25h Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374488 sha256=b7c907332d36083e0b3949866d5c89cc065c45a1cfc2c8d4300793217944f94d 2024-06-26T08:12:46.9526158Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8 2024-06-26T08:12:46.9547314Z Successfully built torchrec 2024-06-26T08:12:49.0598510Z Installing collected packages: lightning-utilities, torchmetrics, torchrec 2024-06-26T08:12:49.6296724Z Successfully installed lightning-utilities-0.11.2 torchmetrics-1.4.0.post0 torchrec-0.3.2 2024-06-26T08:12:49.7643877Z ++ pwd 2024-06-26T08:12:49.7644327Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2024-06-26T08:12:49.7644871Z + test_dynamo_benchmark torchbench 0 2024-06-26T08:12:49.7648373Z ++ pwd 2024-06-26T08:12:49.7649306Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:12:49.7650298Z + local suite=torchbench 2024-06-26T08:12:49.7650646Z + shift 2024-06-26T08:12:49.7650924Z + local shard_id=0 2024-06-26T08:12:49.7651231Z + shift 2024-06-26T08:12:49.7651616Z + [[ dynamic_aot_eager_torchbench == *perf_compare* ]] 2024-06-26T08:12:49.7652521Z + [[ dynamic_aot_eager_torchbench == *perf* ]] 2024-06-26T08:12:49.7653085Z + [[ dynamic_aot_eager_torchbench == *cpu_inductor* ]] 2024-06-26T08:12:49.7653817Z + [[ dynamic_aot_eager_torchbench == *aot_inductor* ]] 2024-06-26T08:12:49.7654882Z + test_single_dynamo_benchmark inference torchbench 0 --inference --bfloat16 2024-06-26T08:12:49.7655563Z ++ pwd 2024-06-26T08:12:49.7656680Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:12:49.7657841Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:12:49.7686094Z + local name=inference 2024-06-26T08:12:49.7686976Z + shift 2024-06-26T08:12:49.7687471Z + local suite=torchbench 2024-06-26T08:12:49.7687876Z + shift 2024-06-26T08:12:49.7688287Z + local shard_id=0 2024-06-26T08:12:49.7688635Z + shift 2024-06-26T08:12:49.7690421Z + partition_flags=() 2024-06-26T08:12:49.7690832Z + local partition_flags 2024-06-26T08:12:49.7691339Z + [[ -n 2 ]] 2024-06-26T08:12:49.7691640Z + [[ -n 0 ]] 2024-06-26T08:12:49.7692234Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-06-26T08:12:49.7692970Z + [[ dynamic_aot_eager_torchbench == *perf_compare* ]] 2024-06-26T08:12:49.7693510Z + [[ dynamic_aot_eager_torchbench == *perf* ]] 2024-06-26T08:12:49.7694043Z + [[ dynamic_aot_eager_torchbench == *aot_inductor* ]] 2024-06-26T08:12:49.7696203Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --backend aot_eager --dynamic-shapes --dynamic-batch-only --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2024-06-26T08:12:55.3395049Z 2024-06-26T08:12:55.9011871Z loading model: 0it [00:00, ?it/s]/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/optim/apply_optimizer_in_backward.py:47: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead. 2024-06-26T08:12:55.9013997Z from torch.distributed.optim import _apply_optimizer_in_backward 2024-06-26T08:12:56.0914520Z 2024-06-26T08:12:56.0915052Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:12:56.0915566Z cuda eval torchrec_dlrm 2024-06-26T08:12:56.0925634Z Traceback (most recent call last): 2024-06-26T08:12:56.0927167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:12:56.0928668Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:12:56.0929741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 447, in forward_pass 2024-06-26T08:12:56.0931103Z return mod(*inputs) 2024-06-26T08:12:56.0932204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:12:56.0933149Z return self._call_impl(*args, **kwargs) 2024-06-26T08:12:56.0934123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:12:56.0935008Z return forward_call(*args, **kwargs) 2024-06-26T08:12:56.0935914Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward 2024-06-26T08:12:56.0936877Z logits = self.model(batch.dense_features, batch.sparse_features) 2024-06-26T08:12:56.0938057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:12:56.0938993Z return self._call_impl(*args, **kwargs) 2024-06-26T08:12:56.0939960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:12:56.0940841Z return forward_call(*args, **kwargs) 2024-06-26T08:12:56.0941740Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 572, in forward 2024-06-26T08:12:56.0942562Z concatenated_dense = self.inter_arch( 2024-06-26T08:12:56.0943572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:12:56.0944530Z return self._call_impl(*args, **kwargs) 2024-06-26T08:12:56.0945661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:12:56.0946716Z return forward_call(*args, **kwargs) 2024-06-26T08:12:56.0947630Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 284, in forward 2024-06-26T08:12:56.0948601Z return self.crossnet(combined_values.reshape([B, -1])) 2024-06-26T08:12:56.0949705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:12:56.0950636Z return self._call_impl(*args, **kwargs) 2024-06-26T08:12:56.0951609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:12:56.0952491Z return forward_call(*args, **kwargs) 2024-06-26T08:12:56.0953425Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/crossnet.py", line 175, in forward 2024-06-26T08:12:56.0954409Z x_l_v = torch.nn.functional.linear(x_l, self.V_kernels[layer]) 2024-06-26T08:12:56.0955404Z RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::BFloat16 2024-06-26T08:12:56.0955994Z 2024-06-26T08:12:56.0956308Z The above exception was the direct cause of the following exception: 2024-06-26T08:12:56.0956782Z 2024-06-26T08:12:56.0956939Z Traceback (most recent call last): 2024-06-26T08:12:56.0957633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:12:56.0958322Z ) = runner.load_model( 2024-06-26T08:12:56.0959024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:12:56.0959831Z self.validate_model(model, example_inputs) 2024-06-26T08:12:56.0960637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:12:56.0961444Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:12:56.0961946Z RuntimeError: Eager run failed 2024-06-26T08:12:56.0962223Z 2024-06-26T08:12:56.0962344Z eager_fail_to_run 2024-06-26T08:12:59.4442952Z 2024-06-26T08:13:01.4363168Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:01.4363939Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:13:01.4428534Z cuda eval BERT_pytorch 2024-06-26T08:13:37.3356041Z pass 2024-06-26T08:13:37.3360874Z TIMING: backend_compile:25.85455 entire_frame_compile:35.08832 2024-06-26T08:13:37.3362213Z STATS: call_* op count: 1100 | FakeTensor.__torch_dispatch__:897 | FakeTensorMode.__torch_dispatch__:46623 | attempt fast:2246 | fast is_contiguous:2246 | ProxyTorchDispatchMode.__torch_dispatch__:9196 2024-06-26T08:13:37.3366705Z Dynamo produced 1 graphs covering 1100 ops with 0 graph breaks (0 unique) 2024-06-26T08:13:42.4987743Z 2024-06-26T08:13:45.0576312Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:45.0576824Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:13:45.0620425Z cuda eval Background_Matting 2024-06-26T08:13:45.0623026Z pass_due_to_skip 2024-06-26T08:13:45.1162691Z TIMING: 2024-06-26T08:13:45.1163140Z STATS: call_* op count: 0 2024-06-26T08:13:45.1163918Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T08:13:48.3677516Z 2024-06-26T08:13:50.9962483Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:50.9962980Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:13:50.9988414Z cuda eval LearningToPaint 2024-06-26T08:13:53.3123713Z pass 2024-06-26T08:13:53.3124416Z TIMING: backend_compile:1.18551 entire_frame_compile:1.82704 2024-06-26T08:13:53.3125564Z STATS: call_* op count: 142 | FakeTensor.__torch_dispatch__:408 | FakeTensorMode.__torch_dispatch__:8715 | ProxyTorchDispatchMode.__torch_dispatch__:1795 2024-06-26T08:13:53.3127245Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-06-26T08:13:56.7886545Z 2024-06-26T08:13:58.0129396Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/vgg16-397923af.pth 2024-06-26T08:13:58.0300598Z 2024-06-26T08:13:58.0300676Z 2024-06-26T08:13:58.1306362Z 0% 0.00/528M [00:00 will be ignored 2024-06-26T08:40:21.7339802Z pass 2024-06-26T08:40:21.7770400Z TIMING: entire_frame_compile:124.87676 backend_compile:26.23548 2024-06-26T08:40:21.7772013Z STATS: call_* op count: 672 | FakeTensorMode.__torch_dispatch__:54004 | FakeTensor.__torch_dispatch__:2683 | attempt fast:1715 | fast is_contiguous:1553 | slow no contiguity match:162 | ProxyTorchDispatchMode.__torch_dispatch__:13583 2024-06-26T08:40:21.7773552Z Dynamo produced 3 graphs covering 672 ops with 7 graph breaks (5 unique) 2024-06-26T08:40:26.6556909Z 2024-06-26T08:40:27.8522569Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:40:27.8523071Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:40:27.8523522Z cuda train alexnet 2024-06-26T08:40:31.0813571Z pass 2024-06-26T08:40:31.1022901Z TIMING: entire_frame_compile:1.90359 backend_compile:1.41174 2024-06-26T08:40:31.1024420Z STATS: call_* op count: 26 | FakeTensor.__torch_dispatch__:131 | FakeTensorMode.__torch_dispatch__:2426 | attempt fast:24 | fast is_contiguous:24 | ProxyTorchDispatchMode.__torch_dispatch__:674 2024-06-26T08:40:31.1025837Z Dynamo produced 2 graphs covering 26 ops with 6 graph breaks (5 unique) 2024-06-26T08:40:34.4729124Z 2024-06-26T08:40:36.1988059Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:40:36.1988555Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:40:36.1990104Z cuda train basic_gnn_edgecnn 2024-06-26T08:40:38.8497769Z W0626 08:40:38.848000 140051788116608 torch/_logging/_internal.py:1040] [24/0] Profiler function will be ignored 2024-06-26T08:40:40.0187117Z pass 2024-06-26T08:40:40.0366332Z TIMING: entire_frame_compile:2.89727 backend_compile:1.18808 2024-06-26T08:40:40.0368413Z STATS: call_* op count: 103 | FakeTensorMode.__torch_dispatch__:4597 | FakeTensor.__torch_dispatch__:406 | ProxyTorchDispatchMode.__torch_dispatch__:1462 2024-06-26T08:40:40.0369852Z Dynamo produced 8 graphs covering 103 ops with 20 graph breaks (6 unique) 2024-06-26T08:40:43.5676114Z 2024-06-26T08:40:45.3152438Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:40:45.3153427Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:40:45.3154304Z cuda train basic_gnn_gcn 2024-06-26T08:40:46.9229996Z W0626 08:40:46.922000 140647267050112 torch/_logging/_internal.py:1040] [18/0] Profiler function will be ignored 2024-06-26T08:40:47.5944211Z pass 2024-06-26T08:40:47.6056030Z TIMING: entire_frame_compile:1.69064 backend_compile:0.77502 2024-06-26T08:40:47.6057137Z STATS: call_* op count: 99 | FakeTensorMode.__torch_dispatch__:2705 | FakeTensor.__torch_dispatch__:195 | ProxyTorchDispatchMode.__torch_dispatch__:866 2024-06-26T08:40:47.6058287Z Dynamo produced 10 graphs covering 99 ops with 13 graph breaks (6 unique) 2024-06-26T08:40:51.0921547Z 2024-06-26T08:40:52.8132619Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:40:52.8133650Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:40:52.8134334Z cuda train basic_gnn_gin 2024-06-26T08:40:54.3921333Z W0626 08:40:54.391000 140464441885312 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:40:55.5672611Z pass 2024-06-26T08:40:55.5758653Z TIMING: entire_frame_compile:2.20503 backend_compile:1.09996 2024-06-26T08:40:55.5759956Z STATS: call_* op count: 113 | FakeTensorMode.__torch_dispatch__:3951 | FakeTensor.__torch_dispatch__:259 | ProxyTorchDispatchMode.__torch_dispatch__:1395 2024-06-26T08:40:55.5761197Z Dynamo produced 3 graphs covering 113 ops with 7 graph breaks (5 unique) 2024-06-26T08:40:59.0331933Z 2024-06-26T08:41:00.7410985Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:41:00.7411497Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:41:00.7411962Z cuda train basic_gnn_sage 2024-06-26T08:41:02.1690515Z W0626 08:41:02.168000 140434345616000 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:41:03.1019999Z pass 2024-06-26T08:41:03.1094830Z TIMING: entire_frame_compile:1.81457 backend_compile:0.88019 2024-06-26T08:41:03.1096282Z STATS: call_* op count: 107 | FakeTensorMode.__torch_dispatch__:3189 | FakeTensor.__torch_dispatch__:202 | ProxyTorchDispatchMode.__torch_dispatch__:1102 2024-06-26T08:41:03.1097576Z Dynamo produced 3 graphs covering 107 ops with 7 graph breaks (5 unique) 2024-06-26T08:41:06.5434525Z 2024-06-26T08:41:07.0316105Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:41:07.0316754Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:41:07.0317436Z cuda train dcgan 2024-06-26T08:41:09.2684380Z pass 2024-06-26T08:41:09.2700992Z TIMING: entire_frame_compile:1.97215 backend_compile:1.34513 2024-06-26T08:41:09.2702446Z STATS: call_* op count: 17 | FakeTensor.__torch_dispatch__:111 | FakeTensorMode.__torch_dispatch__:2295 | attempt fast:80 | fast is_contiguous:80 | ProxyTorchDispatchMode.__torch_dispatch__:439 2024-06-26T08:41:09.2703787Z Dynamo produced 2 graphs covering 17 ops with 6 graph breaks (5 unique) 2024-06-26T08:41:12.6310199Z 2024-06-26T08:41:16.0392622Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:41:16.0393245Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:41:16.0393730Z cuda train demucs 2024-06-26T08:42:13.8197346Z pass 2024-06-26T08:42:13.8415642Z TIMING: entire_frame_compile:3.13993 backend_compile:1.56148 2024-06-26T08:42:13.8419209Z STATS: call_* op count: 97 | FakeTensorMode.__torch_dispatch__:6095 | attempt fast:25 | fast is_contiguous:10 | slow no contiguity match:10 | slow both tensors nontrivially broadcast:5 | FakeTensor.__torch_dispatch__:441 | ProxyTorchDispatchMode.__torch_dispatch__:1789 2024-06-26T08:42:13.8420949Z Dynamo produced 6 graphs covering 97 ops with 9 graph breaks (6 unique) 2024-06-26T08:42:17.2413510Z 2024-06-26T08:42:19.3169845Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:42:19.3170336Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:42:19.3170789Z cuda train densenet121 2024-06-26T08:43:18.2835685Z pass 2024-06-26T08:43:18.2952062Z TIMING: entire_frame_compile:53.17619 backend_compile:42.97389 2024-06-26T08:43:18.2955500Z STATS: call_* op count: 435 | FakeTensor.__torch_dispatch__:2272 | FakeTensorMode.__torch_dispatch__:81471 | attempt fast:2842 | fast is_contiguous:2311 | ProxyTorchDispatchMode.__torch_dispatch__:18924 | slow no contiguity match:531 2024-06-26T08:43:18.2957065Z Dynamo produced 2 graphs covering 435 ops with 6 graph breaks (5 unique) 2024-06-26T08:43:24.6955340Z 2024-06-26T08:43:29.2585045Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:43:29.2585745Z loading model: 0it [00:04, ?it/s] 2024-06-26T08:43:29.2586500Z cuda train detectron2_maskrcnn_r_50_c4 2024-06-26T08:43:29.2598535Z Traceback (most recent call last): 2024-06-26T08:43:29.2599773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:43:29.2600683Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:43:29.2601840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in forward_and_backward_pass 2024-06-26T08:43:29.2602931Z pred = mod(*cloned_inputs) 2024-06-26T08:43:29.2607098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:43:29.2608242Z return self._call_impl(*args, **kwargs) 2024-06-26T08:43:29.2609320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:43:29.2610256Z return forward_call(*args, **kwargs) 2024-06-26T08:43:29.2611456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-06-26T08:43:29.2612493Z return self.inference(batched_inputs) 2024-06-26T08:43:29.2613836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference 2024-06-26T08:43:29.2615133Z results, _ = self.roi_heads(images, features, proposals, None) 2024-06-26T08:43:29.2616709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:43:29.2617824Z return self._call_impl(*args, **kwargs) 2024-06-26T08:43:29.2618808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:43:29.2619685Z return forward_call(*args, **kwargs) 2024-06-26T08:43:29.2620712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward 2024-06-26T08:43:29.2621667Z box_features = self._shared_roi_transform( 2024-06-26T08:43:29.2622808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform 2024-06-26T08:43:29.2623822Z x = self.pooler(features, boxes) 2024-06-26T08:43:29.2624931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:43:29.2625862Z return self._call_impl(*args, **kwargs) 2024-06-26T08:43:29.2626852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:43:29.2627723Z return forward_call(*args, **kwargs) 2024-06-26T08:43:29.2628674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward 2024-06-26T08:43:29.2629612Z return self.level_poolers[0](x[0], pooler_fmt_boxes) 2024-06-26T08:43:29.2630694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:43:29.2631627Z return self._call_impl(*args, **kwargs) 2024-06-26T08:43:29.2632605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:43:29.2633480Z return forward_call(*args, **kwargs) 2024-06-26T08:43:29.2634410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward 2024-06-26T08:43:29.2635241Z return roi_align( 2024-06-26T08:43:29.2636093Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-06-26T08:43:29.2637256Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-06-26T08:43:29.2638487Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-06-26T08:43:29.2639614Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-06-26T08:43:29.2641026Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-06-26T08:43:29.2641959Z v1 = masked_index(y_low, x_low) 2024-06-26T08:43:29.2642899Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-06-26T08:43:29.2643742Z return input[ 2024-06-26T08:43:29.2647342Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 21890.87 GiB. GPU 0 has a total capacity of 21.98 GiB of which 16.54 GiB is free. Process 29638 has 5.42 GiB memory in use. Of the allocated memory 4.58 GiB is allocated by PyTorch, and 558.89 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T08:43:29.2650222Z 2024-06-26T08:43:29.2650534Z The above exception was the direct cause of the following exception: 2024-06-26T08:43:29.2651131Z 2024-06-26T08:43:29.2651292Z Traceback (most recent call last): 2024-06-26T08:43:29.2651984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:43:29.2652682Z ) = runner.load_model( 2024-06-26T08:43:29.2653372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:43:29.2654176Z self.validate_model(model, example_inputs) 2024-06-26T08:43:29.2654976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:43:29.2655784Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:43:29.2656314Z RuntimeError: Eager run failed 2024-06-26T08:43:29.2656594Z 2024-06-26T08:43:29.2656714Z eager_fail_to_run 2024-06-26T08:43:32.5916993Z 2024-06-26T08:43:40.7741276Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:43:40.7741851Z loading model: 0it [00:08, ?it/s] 2024-06-26T08:43:40.7742344Z cuda train dlrm 2024-06-26T08:43:42.5447936Z pass 2024-06-26T08:43:42.5849528Z TIMING: entire_frame_compile:1.12533 backend_compile:0.73008 2024-06-26T08:43:42.5850896Z STATS: call_* op count: 40 | FakeTensor.__torch_dispatch__:151 | FakeTensorMode.__torch_dispatch__:3115 | ProxyTorchDispatchMode.__torch_dispatch__:1110 2024-06-26T08:43:42.5852070Z Dynamo produced 2 graphs covering 40 ops with 6 graph breaks (5 unique) 2024-06-26T08:43:45.8886475Z 2024-06-26T08:43:47.2579268Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:43:47.2579927Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:43:47.2580598Z cuda train drq 2024-06-26T08:43:48.7779922Z W0626 08:43:48.777000 140165311832704 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:43:51.4938080Z pass 2024-06-26T08:43:51.4967329Z TIMING: entire_frame_compile:3.94491 backend_compile:2.06921 2024-06-26T08:43:51.4969964Z STATS: call_* op count: 137 | FakeTensorMode.__torch_dispatch__:7382 | FakeTensor.__torch_dispatch__:419 | ProxyTorchDispatchMode.__torch_dispatch__:2562 2024-06-26T08:43:51.4972215Z Dynamo produced 5 graphs covering 137 ops with 7 graph breaks (5 unique) 2024-06-26T08:43:54.9834982Z 2024-06-26T08:43:56.5721578Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:43:56.5722072Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:43:56.5722532Z cuda train fastNLP_Bert 2024-06-26T08:44:02.3818137Z W0626 08:44:02.381000 140254245745280 torch/_dynamo/variables/tensor.py:714] [3/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T08:44:02.3819572Z W0626 08:44:02.381000 140254245745280 torch/_dynamo/variables/tensor.py:714] [3/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T08:44:02.3820701Z W0626 08:44:02.381000 140254245745280 torch/_dynamo/variables/tensor.py:714] [3/0] or: 2024-06-26T08:44:02.3822125Z W0626 08:44:02.381000 140254245745280 torch/_dynamo/variables/tensor.py:714] [3/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T08:44:02.3823578Z W0626 08:44:02.381000 140254245745280 torch/_dynamo/variables/tensor.py:714] [3/0] to include these operations in the captured graph. 2024-06-26T08:44:02.3824905Z W0626 08:44:02.381000 140254245745280 torch/_dynamo/variables/tensor.py:714] [3/0] 2024-06-26T08:44:12.9565381Z pass 2024-06-26T08:44:13.0437546Z TIMING: entire_frame_compile:10.24086 backend_compile:6.97266 2024-06-26T08:44:13.0438726Z STATS: call_* op count: 450 | FakeTensorMode.__torch_dispatch__:29752 | ProxyTorchDispatchMode.__torch_dispatch__:13734 | FakeTensor.__torch_dispatch__:648 2024-06-26T08:44:13.0439901Z Dynamo produced 7 graphs covering 450 ops with 10 graph breaks (6 unique) 2024-06-26T08:44:16.6726474Z 2024-06-26T08:44:17.4502789Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:44:17.4503272Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:44:17.4503745Z cuda train functorch_dp_cifar10 2024-06-26T08:44:29.1918576Z W0626 08:44:29.190000 139854800732800 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:44:36.2788156Z pass 2024-06-26T08:44:36.2892427Z TIMING: entire_frame_compile:18.23986 backend_compile:12.83001 2024-06-26T08:44:36.2894040Z STATS: call_* op count: 277 | FakeTensor.__torch_dispatch__:1658 | FakeTensorMode.__torch_dispatch__:23851 | attempt fast:852 | fast is_contiguous:832 | ProxyTorchDispatchMode.__torch_dispatch__:5405 | slow both tensors nontrivially broadcast:20 2024-06-26T08:44:36.2895637Z Dynamo produced 3 graphs covering 277 ops with 7 graph breaks (5 unique) 2024-06-26T08:44:40.1888345Z 2024-06-26T08:44:40.7166550Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:44:40.7167412Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:44:40.7167885Z cuda train functorch_maml_omniglot 2024-06-26T08:44:41.5896215Z W0626 08:44:41.588000 140347785581184 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:44:43.1020439Z pass 2024-06-26T08:44:43.1021088Z TIMING: entire_frame_compile:2.09597 backend_compile:1.24849 2024-06-26T08:44:43.1022220Z STATS: call_* op count: 78 | FakeTensor.__torch_dispatch__:398 | FakeTensorMode.__torch_dispatch__:3886 | ProxyTorchDispatchMode.__torch_dispatch__:1104 2024-06-26T08:44:43.1023381Z Dynamo produced 3 graphs covering 78 ops with 7 graph breaks (5 unique) 2024-06-26T08:44:46.4618452Z 2024-06-26T08:44:47.6821852Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:44:47.6822370Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:44:47.6822822Z cuda train hf_Albert 2024-06-26T08:45:28.2213522Z W0626 08:45:28.220000 140135694459520 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:45:30.7764809Z pass 2024-06-26T08:45:30.8292295Z TIMING: entire_frame_compile:36.27053 backend_compile:26.49534 2024-06-26T08:45:30.8294729Z STATS: call_* op count: 616 | FakeTensorMode.__torch_dispatch__:51879 | attempt fast:2272 | fast is_contiguous:2272 | FakeTensor.__torch_dispatch__:1841 | ProxyTorchDispatchMode.__torch_dispatch__:15170 2024-06-26T08:45:30.8296161Z Dynamo produced 3 graphs covering 616 ops with 6 graph breaks (5 unique) 2024-06-26T08:45:35.8769435Z 2024-06-26T08:45:39.5710961Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:45:39.5711557Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:45:39.5712142Z cuda train hf_Bart 2024-06-26T08:46:24.3960296Z W0626 08:46:24.395000 140452749931136 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:46:50.5591051Z pass 2024-06-26T08:46:50.6143138Z TIMING: entire_frame_compile:60.90062 backend_compile:41.07065 2024-06-26T08:46:50.6145873Z STATS: call_* op count: 1290 | FakeTensorMode.__torch_dispatch__:92883 | FakeTensor.__torch_dispatch__:6773 | attempt fast:2300 | fast is_contiguous:2292 | slow no contiguity match:8 | ProxyTorchDispatchMode.__torch_dispatch__:26116 2024-06-26T08:46:50.6147437Z Dynamo produced 3 graphs covering 1290 ops with 6 graph breaks (5 unique) 2024-06-26T08:46:56.2537548Z 2024-06-26T08:46:58.7124916Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:46:58.7125966Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:46:58.7127299Z cuda train hf_Bert 2024-06-26T08:47:40.6186986Z W0626 08:47:40.617000 140541186417280 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:48:01.1965315Z pass 2024-06-26T08:48:01.2443137Z TIMING: entire_frame_compile:54.4866 backend_compile:35.54131 2024-06-26T08:48:01.2445370Z STATS: call_* op count: 1093 | FakeTensorMode.__torch_dispatch__:76417 | attempt fast:2121 | fast is_contiguous:2121 | FakeTensor.__torch_dispatch__:5120 | ProxyTorchDispatchMode.__torch_dispatch__:21388 2024-06-26T08:48:01.2447460Z Dynamo produced 3 graphs covering 1093 ops with 6 graph breaks (5 unique) 2024-06-26T08:48:06.6397407Z 2024-06-26T08:48:11.1467019Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:48:11.1478301Z loading model: 0it [00:04, ?it/s] 2024-06-26T08:48:11.1479054Z cuda train hf_Bert_large 2024-06-26T08:49:39.6082236Z W0626 08:49:39.607000 139677998371456 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:50:19.1691732Z pass 2024-06-26T08:50:19.3376897Z TIMING: entire_frame_compile:107.88649 backend_compile:69.85177 2024-06-26T08:50:19.3379518Z STATS: call_* op count: 2113 | FakeTensorMode.__torch_dispatch__:149341 | attempt fast:4125 | fast is_contiguous:4125 | FakeTensor.__torch_dispatch__:9920 | ProxyTorchDispatchMode.__torch_dispatch__:41896 2024-06-26T08:50:19.3380948Z Dynamo produced 3 graphs covering 2113 ops with 6 graph breaks (5 unique) 2024-06-26T08:50:26.9838150Z 2024-06-26T08:50:30.2881093Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:50:30.2881680Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:50:30.2882143Z cuda train hf_BigBird 2024-06-26T08:50:41.5564859Z WARNING:common:fp64 golden ref were not generated for hf_BigBird. Setting accuracy check to cosine 2024-06-26T08:51:26.1546419Z E0626 08:51:26.153000 140143402967680 torch/_guards.py:261] [3/0_1] Error while creating guard: 2024-06-26T08:51:26.1547832Z E0626 08:51:26.153000 140143402967680 torch/_guards.py:261] [3/0_1] Name: '' 2024-06-26T08:51:26.1548833Z E0626 08:51:26.153000 140143402967680 torch/_guards.py:261] [3/0_1] Source: shape_env 2024-06-26T08:51:26.1553152Z E0626 08:51:26.153000 140143402967680 torch/_guards.py:261] [3/0_1] Create Function: SHAPE_ENV 2024-06-26T08:51:26.1554151Z E0626 08:51:26.153000 140143402967680 torch/_guards.py:261] [3/0_1] Guard Types: None 2024-06-26T08:51:26.1555046Z E0626 08:51:26.153000 140143402967680 torch/_guards.py:261] [3/0_1] Code List: None 2024-06-26T08:51:26.1555989Z E0626 08:51:26.153000 140143402967680 torch/_guards.py:261] [3/0_1] Object Weakref: None 2024-06-26T08:51:26.1556962Z E0626 08:51:26.153000 140143402967680 torch/_guards.py:261] [3/0_1] Guarded Class Weakref: None 2024-06-26T08:51:26.1557983Z E0626 08:51:26.153000 140143402967680 torch/_guards.py:261] [3/0_1] Traceback (most recent call last): 2024-06-26T08:51:26.1559983Z E0626 08:51:26.153000 140143402967680 torch/_guards.py:261] [3/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_guards.py", line 259, in create 2024-06-26T08:51:26.1561680Z E0626 08:51:26.153000 140143402967680 torch/_guards.py:261] [3/0_1] return self.create_fn(builder, self) 2024-06-26T08:51:26.1563819Z E0626 08:51:26.153000 140143402967680 torch/_guards.py:261] [3/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 1680, in SHAPE_ENV 2024-06-26T08:51:26.1565684Z E0626 08:51:26.153000 140143402967680 torch/_guards.py:261] [3/0_1] guards = output_graph.shape_env.produce_guards( 2024-06-26T08:51:26.1568155Z E0626 08:51:26.153000 140143402967680 torch/_guards.py:261] [3/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 4180, in produce_guards 2024-06-26T08:51:26.1569735Z E0626 08:51:26.153000 140143402967680 torch/_guards.py:261] [3/0_1] raise ConstraintViolationError( 2024-06-26T08:51:26.1571624Z E0626 08:51:26.153000 140143402967680 torch/_guards.py:261] [3/0_1] torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['cloned_inputs'][0].size()[0])! For more information, run with TORCH_LOGS="+dynamic". 2024-06-26T08:51:26.1574050Z E0626 08:51:26.153000 140143402967680 torch/_guards.py:261] [3/0_1] - Not all values of RelaxedUnspecConstraint(L['cloned_inputs'][0].size()[0]) are valid because L['cloned_inputs'][0].size()[0] was inferred to be a constant (4). 2024-06-26T08:51:26.1575527Z E0626 08:51:26.154000 140143402967680 torch/_guards.py:263] [3/0_1] Created at: 2024-06-26T08:51:26.1577127Z E0626 08:51:26.154000 140143402967680 torch/_guards.py:263] [3/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 564, in transform 2024-06-26T08:51:26.1578585Z E0626 08:51:26.154000 140143402967680 torch/_guards.py:263] [3/0_1] tracer = InstructionTranslator( 2024-06-26T08:51:26.1580190Z E0626 08:51:26.154000 140143402967680 torch/_guards.py:263] [3/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2385, in __init__ 2024-06-26T08:51:26.1581576Z E0626 08:51:26.154000 140143402967680 torch/_guards.py:263] [3/0_1] output=OutputGraph( 2024-06-26T08:51:26.1583079Z E0626 08:51:26.154000 140143402967680 torch/_guards.py:263] [3/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 313, in __init__ 2024-06-26T08:51:26.1584476Z E0626 08:51:26.154000 140143402967680 torch/_guards.py:263] [3/0_1] self.init_ambient_guards() 2024-06-26T08:51:26.1586220Z E0626 08:51:26.154000 140143402967680 torch/_guards.py:263] [3/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 452, in init_ambient_guards 2024-06-26T08:51:26.1587907Z E0626 08:51:26.154000 140143402967680 torch/_guards.py:263] [3/0_1] self.guards.add(ShapeEnvSource().make_guard(GuardBuilder.SHAPE_ENV)) 2024-06-26T08:51:26.1588794Z ERROR:common: 2024-06-26T08:51:26.1589153Z Traceback (most recent call last): 2024-06-26T08:51:26.1589919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2627, in check_accuracy 2024-06-26T08:51:26.1590833Z new_result = optimized_model_iter_fn(model_copy, example_inputs) 2024-06-26T08:51:26.1591875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 434, in _fn 2024-06-26T08:51:26.1592683Z return fn(*args, **kwargs) 2024-06-26T08:51:26.1593418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2359, in run_n_iterations 2024-06-26T08:51:26.1594291Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-06-26T08:51:26.1595254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 450, in forward_and_backward_pass 2024-06-26T08:51:26.1596187Z cloned_inputs = clone_inputs(inputs) 2024-06-26T08:51:26.1597228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 451, in torch_dynamo_resume_in_forward_and_backward_pass_at_450 2024-06-26T08:51:26.1598251Z self.optimizer_zero_grad(mod) 2024-06-26T08:51:26.1599185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1121, in __call__ 2024-06-26T08:51:26.1600068Z return self._torchdynamo_orig_callable( 2024-06-26T08:51:26.1601036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 948, in __call__ 2024-06-26T08:51:26.1601899Z result = self._inner_convert( 2024-06-26T08:51:26.1602896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 472, in __call__ 2024-06-26T08:51:26.1603732Z return _compile( 2024-06-26T08:51:26.1604601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 85, in wrapper_function 2024-06-26T08:51:26.1605617Z return StrobelightCompileTimeProfiler.profile_compile_time( 2024-06-26T08:51:26.1607084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_strobelight/compile_time_profiler.py", line 129, in profile_compile_time 2024-06-26T08:51:26.1608088Z return func(*args, **kwargs) 2024-06-26T08:51:26.1608725Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2024-06-26T08:51:26.1609401Z return func(*args, **kwds) 2024-06-26T08:51:26.1610569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 817, in _compile 2024-06-26T08:51:26.1611666Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-06-26T08:51:26.1612841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 233, in time_wrapper 2024-06-26T08:51:26.1613667Z r = func(*args, **kwargs) 2024-06-26T08:51:26.1614592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 726, in compile_inner 2024-06-26T08:51:26.1615482Z check_fn = CheckFunctionManager( 2024-06-26T08:51:26.1616378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 2094, in __init__ 2024-06-26T08:51:26.1617188Z guard.create(builder) 2024-06-26T08:51:26.1617966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_guards.py", line 259, in create 2024-06-26T08:51:26.1618758Z return self.create_fn(builder, self) 2024-06-26T08:51:26.1619668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 1680, in SHAPE_ENV 2024-06-26T08:51:26.1620554Z guards = output_graph.shape_env.produce_guards( 2024-06-26T08:51:26.1621699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 4180, in produce_guards 2024-06-26T08:51:26.1622678Z raise ConstraintViolationError( 2024-06-26T08:51:26.1623904Z torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['cloned_inputs'][0].size()[0])! For more information, run with TORCH_LOGS="+dynamic". 2024-06-26T08:51:26.1625787Z - Not all values of RelaxedUnspecConstraint(L['cloned_inputs'][0].size()[0]) are valid because L['cloned_inputs'][0].size()[0] was inferred to be a constant (4). 2024-06-26T08:51:26.1626685Z 2024-06-26T08:51:26.1626691Z 2024-06-26T08:51:26.1626989Z You can suppress this exception and fall back to eager by setting: 2024-06-26T08:51:26.1627579Z import torch._dynamo 2024-06-26T08:51:26.1628004Z torch._dynamo.config.suppress_errors = True 2024-06-26T08:51:26.1628349Z 2024-06-26T08:51:26.1628653Z TorchDynamo optimized model failed to run because of following error 2024-06-26T08:51:26.1629247Z fail_to_run 2024-06-26T08:51:26.2203308Z TIMING: entire_frame_compile:0.02182 backend_compile:29.67922 2024-06-26T08:51:26.2206079Z STATS: call_* op count: 3361 | FakeTensorMode.__torch_dispatch__:142368 | attempt fast:102 | slow both tensors nontrivially broadcast:2 | fast is_contiguous:100 | FakeTensor.__torch_dispatch__:3 | ProxyTorchDispatchMode.__torch_dispatch__:58466 2024-06-26T08:51:26.2207822Z Dynamo produced 1 graphs covering 3361 ops with 3 graph breaks (3 unique) 2024-06-26T08:51:30.7567529Z 2024-06-26T08:51:32.7213360Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:51:32.7213920Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:51:32.7214375Z cuda train hf_DistilBert 2024-06-26T08:51:54.2085466Z W0626 08:51:54.207000 140015832060544 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:52:04.6442940Z pass 2024-06-26T08:52:04.6695919Z TIMING: entire_frame_compile:27.14556 backend_compile:17.99259 2024-06-26T08:52:04.6698968Z STATS: call_* op count: 553 | FakeTensor.__torch_dispatch__:2690 | FakeTensorMode.__torch_dispatch__:40365 | attempt fast:1085 | fast is_contiguous:1061 | slow no contiguity match:24 | ProxyTorchDispatchMode.__torch_dispatch__:11112 2024-06-26T08:52:04.6700532Z Dynamo produced 3 graphs covering 553 ops with 6 graph breaks (5 unique) 2024-06-26T08:52:08.9846088Z 2024-06-26T08:52:11.9652926Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:52:11.9653415Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:52:11.9656458Z cuda train hf_GPT2 2024-06-26T08:52:49.0450260Z W0626 08:52:49.043000 140064082465408 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:53:03.5452528Z pass 2024-06-26T08:53:03.6382248Z TIMING: entire_frame_compile:46.56294 backend_compile:32.00756 2024-06-26T08:53:03.6385459Z STATS: call_* op count: 1280 | FakeTensorMode.__torch_dispatch__:63343 | FakeTensor.__torch_dispatch__:3212 | attempt fast:1931 | fast is_contiguous:1919 | ProxyTorchDispatchMode.__torch_dispatch__:17951 | slow no contiguity match:12 2024-06-26T08:53:03.6387071Z Dynamo produced 3 graphs covering 1280 ops with 6 graph breaks (5 unique) 2024-06-26T08:53:08.8390316Z 2024-06-26T08:53:21.1921812Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:53:21.1922381Z loading model: 0it [00:12, ?it/s] 2024-06-26T08:53:21.1925586Z cuda train hf_GPT2_large 2024-06-26T08:53:21.1928238Z pass_due_to_skip 2024-06-26T08:53:21.6875450Z TIMING: 2024-06-26T08:53:21.6875832Z STATS: call_* op count: 0 2024-06-26T08:53:21.6876414Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T08:53:24.9361072Z 2024-06-26T08:53:26.3869036Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:53:26.3869518Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:53:26.3869988Z cuda train hf_Reformer 2024-06-26T08:53:40.0148955Z W0626 08:53:40.014000 140703804785280 torch/_logging/_internal.py:1040] [32/0] Profiler function will be ignored 2024-06-26T08:53:48.0147271Z pass 2024-06-26T08:53:48.0817138Z TIMING: entire_frame_compile:13.59179 backend_compile:7.59442 2024-06-26T08:53:48.0820692Z STATS: call_* op count: 693 | FakeTensorMode.__torch_dispatch__:24832 | ProxyTorchDispatchMode.__torch_dispatch__:7567 | attempt fast:20 | fast is_contiguous:14 | slow no contiguity match:6 | FakeTensor.__torch_dispatch__:1592 2024-06-26T08:53:48.0822334Z Dynamo produced 17 graphs covering 693 ops with 25 graph breaks (9 unique) 2024-06-26T08:53:51.7687854Z 2024-06-26T08:53:56.7562411Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:53:56.7562898Z loading model: 0it [00:04, ?it/s] 2024-06-26T08:53:56.7563356Z cuda train hf_T5_base 2024-06-26T08:53:59.8930461Z WARNING:common:fp64 golden ref were not generated for hf_T5_base. Setting accuracy check to cosine 2024-06-26T08:54:02.0008707Z ERROR:common: 2024-06-26T08:54:02.0009140Z Traceback (most recent call last): 2024-06-26T08:54:02.0009936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2564, in check_accuracy 2024-06-26T08:54:02.0010759Z correct_rerun_result = self.run_n_iterations( 2024-06-26T08:54:02.0014429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2359, in run_n_iterations 2024-06-26T08:54:02.0015490Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-06-26T08:54:02.0016556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 458, in forward_and_backward_pass 2024-06-26T08:54:02.0017499Z self.grad_scaler.scale(loss).backward() 2024-06-26T08:54:02.0018692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-06-26T08:54:02.0019646Z torch.autograd.backward( 2024-06-26T08:54:02.0020719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 288, in backward 2024-06-26T08:54:02.0021753Z _engine_run_backward( 2024-06-26T08:54:02.0022700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 768, in _engine_run_backward 2024-06-26T08:54:02.0023924Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-06-26T08:54:02.0027682Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 192.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 88.44 MiB is free. Process 31381 has 21.88 GiB memory in use. Of the allocated memory 20.95 GiB is allocated by PyTorch, and 595.70 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T08:54:02.0877959Z eager_2nd_run_OOM 2024-06-26T08:54:02.0910274Z TIMING: 2024-06-26T08:54:02.0910892Z STATS: call_* op count: 0 2024-06-26T08:54:02.0912026Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T08:54:05.4098975Z 2024-06-26T08:54:15.0745299Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:54:15.0745797Z loading model: 0it [00:09, ?it/s] 2024-06-26T08:54:15.0746260Z cuda train hf_T5_large 2024-06-26T08:54:15.0750263Z pass_due_to_skip 2024-06-26T08:54:15.5416053Z TIMING: 2024-06-26T08:54:15.5416426Z STATS: call_* op count: 0 2024-06-26T08:54:15.5417011Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T08:54:18.8290748Z 2024-06-26T08:54:19.8774616Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:54:19.8775129Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:54:19.8775591Z cuda train hf_Whisper 2024-06-26T08:54:32.0727892Z W0626 08:54:32.071000 140338194272896 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:54:39.1270618Z pass 2024-06-26T08:54:39.1448857Z TIMING: entire_frame_compile:16.37869 backend_compile:10.71945 2024-06-26T08:54:39.1450442Z STATS: call_* op count: 355 | FakeTensor.__torch_dispatch__:1872 | FakeTensorMode.__torch_dispatch__:25069 | attempt fast:657 | fast is_contiguous:620 | slow no contiguity match:37 | ProxyTorchDispatchMode.__torch_dispatch__:6891 2024-06-26T08:54:39.1453856Z Dynamo produced 3 graphs covering 355 ops with 6 graph breaks (5 unique) 2024-06-26T08:54:42.9663084Z 2024-06-26T08:54:43.2332300Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:54:43.2333427Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:54:43.2334347Z cuda train hf_distil_whisper 2024-06-26T08:54:43.2336184Z Traceback (most recent call last): 2024-06-26T08:54:43.2337340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:54:43.2338231Z ) = runner.load_model( 2024-06-26T08:54:43.2339178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 320, in load_model 2024-06-26T08:54:43.2340155Z benchmark = benchmark_cls( 2024-06-26T08:54:43.2340915Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-26T08:54:43.2341749Z obj = type.__call__(cls, *args, **kwargs) 2024-06-26T08:54:43.2342709Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper/__init__.py", line 12, in __init__ 2024-06-26T08:54:43.2343768Z raise NotImplementedError("Training is not implemented.") 2024-06-26T08:54:43.2344442Z NotImplementedError: Training is not implemented. 2024-06-26T08:54:43.2344979Z 2024-06-26T08:54:43.2345101Z model_fail_to_load 2024-06-26T08:54:46.4360881Z 2024-06-26T08:54:46.8198755Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:54:46.8199342Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:54:46.8199804Z cuda train lennard_jones 2024-06-26T08:54:48.1128632Z W0626 08:54:48.112000 139697720599168 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:54:49.1876118Z pass 2024-06-26T08:54:49.1876760Z TIMING: entire_frame_compile:2.14269 backend_compile:1.26261 2024-06-26T08:54:49.1878552Z STATS: call_* op count: 61 | FakeTensor.__torch_dispatch__:298 | FakeTensorMode.__torch_dispatch__:3345 | attempt fast:52 | fast is_contiguous:52 | ProxyTorchDispatchMode.__torch_dispatch__:921 2024-06-26T08:54:49.1880459Z Dynamo produced 3 graphs covering 61 ops with 7 graph breaks (5 unique) 2024-06-26T08:54:49.9356771Z accuracy pass_rate=75.76% 2024-06-26T08:54:49.9360457Z calls_captured gmean=0.00x mean=449.061x 2024-06-26T08:54:49.9364394Z unique_graphs gmean=0.00x mean=3.212x 2024-06-26T08:54:49.9368505Z graph_breaks gmean=0.00x mean=6.364x 2024-06-26T08:54:49.9372950Z unique_graph_breaks gmean=0.00x mean=4.212x 2024-06-26T08:54:49.9376888Z autograd_captures gmean=0.00x mean=0.000x 2024-06-26T08:54:49.9381131Z autograd_compiles gmean=0.00x mean=0.000x 2024-06-26T08:54:49.9384630Z cudagraph_skips gmean=0.00x mean=0.000x 2024-06-26T08:54:50.4747230Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamic_aot_eager_torchbench_training.csv 2024-06-26T08:54:50.7871907Z torchrec_dlrm XFAIL 2024-06-26T08:54:50.7876531Z BERT_pytorch PASS 2024-06-26T08:54:50.7880936Z Background_Matting XFAIL 2024-06-26T08:54:50.7885203Z LearningToPaint PASS 2024-06-26T08:54:50.7890073Z Super_SloMo PASS 2024-06-26T08:54:50.7894540Z alexnet PASS 2024-06-26T08:54:50.7898950Z basic_gnn_edgecnn PASS 2024-06-26T08:54:50.7903538Z basic_gnn_gcn PASS 2024-06-26T08:54:50.7908552Z basic_gnn_gin PASS 2024-06-26T08:54:50.7912861Z basic_gnn_sage PASS 2024-06-26T08:54:50.7917195Z dcgan PASS 2024-06-26T08:54:50.7921588Z demucs PASS 2024-06-26T08:54:50.7926162Z densenet121 PASS 2024-06-26T08:54:50.7930886Z detectron2_maskrcnn_r_50_c4 XFAIL 2024-06-26T08:54:50.7935264Z dlrm PASS 2024-06-26T08:54:50.7939535Z drq PASS 2024-06-26T08:54:50.7944024Z fastNLP_Bert PASS 2024-06-26T08:54:50.7948670Z functorch_dp_cifar10 PASS 2024-06-26T08:54:50.7953151Z functorch_maml_omniglot PASS 2024-06-26T08:54:50.7957460Z hf_Albert PASS 2024-06-26T08:54:50.7961865Z hf_Bart PASS 2024-06-26T08:54:50.7966385Z hf_Bert PASS 2024-06-26T08:54:50.7970934Z hf_Bert_large PASS 2024-06-26T08:54:50.7975586Z hf_BigBird XFAIL 2024-06-26T08:54:50.7980043Z hf_DistilBert PASS 2024-06-26T08:54:50.7984453Z hf_GPT2 PASS 2024-06-26T08:54:50.7989227Z hf_GPT2_large XFAIL 2024-06-26T08:54:50.7993586Z hf_Reformer PASS 2024-06-26T08:54:50.7997895Z hf_T5_base XFAIL 2024-06-26T08:54:50.8002310Z hf_T5_large XFAIL 2024-06-26T08:54:50.8006704Z hf_Whisper PASS 2024-06-26T08:54:50.8011440Z hf_distil_whisper XFAIL 2024-06-26T08:54:50.8015783Z lennard_jones PASS 2024-06-26T08:54:50.8467342Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamic_aot_eager_torchbench_training.csv 2024-06-26T08:54:51.1577432Z torchrec_dlrm PASS 2024-06-26T08:54:51.1580920Z BERT_pytorch PASS 2024-06-26T08:54:51.1586492Z Background_Matting PASS 2024-06-26T08:54:51.1591396Z LearningToPaint PASS 2024-06-26T08:54:51.1596435Z Super_SloMo PASS 2024-06-26T08:54:51.1601461Z alexnet PASS 2024-06-26T08:54:51.1606708Z basic_gnn_edgecnn PASS 2024-06-26T08:54:51.1612283Z basic_gnn_gcn PASS 2024-06-26T08:54:51.1617282Z basic_gnn_gin PASS 2024-06-26T08:54:51.1622161Z basic_gnn_sage PASS 2024-06-26T08:54:51.1627558Z dcgan PASS 2024-06-26T08:54:51.1632414Z demucs PASS 2024-06-26T08:54:51.1637287Z densenet121 PASS 2024-06-26T08:54:51.1642428Z detectron2_maskrcnn_r_50_c4 PASS 2024-06-26T08:54:51.1647499Z dlrm PASS 2024-06-26T08:54:51.1652688Z drq PASS 2024-06-26T08:54:51.1657619Z fastNLP_Bert PASS 2024-06-26T08:54:51.1662633Z functorch_dp_cifar10 PASS 2024-06-26T08:54:51.1667986Z functorch_maml_omniglot PASS 2024-06-26T08:54:51.1672738Z hf_Albert PASS 2024-06-26T08:54:51.1677961Z hf_Bart PASS 2024-06-26T08:54:51.1682848Z hf_Bert PASS 2024-06-26T08:54:51.1687953Z hf_Bert_large PASS 2024-06-26T08:54:51.1693009Z hf_BigBird PASS 2024-06-26T08:54:51.1697922Z hf_DistilBert PASS 2024-06-26T08:54:51.1702957Z hf_GPT2 PASS 2024-06-26T08:54:51.1708369Z hf_GPT2_large PASS 2024-06-26T08:54:51.1713254Z hf_Reformer PASS 2024-06-26T08:54:51.1718263Z hf_T5_base PASS 2024-06-26T08:54:51.1723234Z hf_T5_large PASS 2024-06-26T08:54:51.1728342Z hf_Whisper PASS 2024-06-26T08:54:51.1733404Z hf_distil_whisper PASS 2024-06-26T08:54:51.1738266Z lennard_jones PASS 2024-06-26T08:54:51.2199987Z + cleanup_workspace 2024-06-26T08:54:51.2201069Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-26T08:54:51.2202222Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-26T08:54:51.2203184Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-26T08:54:51.2203858Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T08:54:51.2204693Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-26T08:54:51.2205604Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-26T08:54:51.2206324Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-26T08:54:51.7726968Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T08:54:51.7727527Z cat test/**/*_toprint.log || true 2024-06-26T08:54:51.7735168Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:54:51.7735688Z env: 2024-06-26T08:54:51.7735973Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:54:51.7736456Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:54:51.7737218Z DOCKER_CONTAINER_ID: 57400afaa3f29946832de03690d2b6347f2f66970ba9ac0b0161152c7a47ef91 2024-06-26T08:54:51.7737863Z ##[endgroup] 2024-06-26T08:54:51.7800804Z cat: test/**/*_toprint.log: No such file or directory 2024-06-26T08:54:51.7830280Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:54:51.7830778Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:54:51.7838369Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:54:51.7838887Z env: 2024-06-26T08:54:51.7839174Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:54:51.7839645Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:54:51.7840415Z DOCKER_CONTAINER_ID: 57400afaa3f29946832de03690d2b6347f2f66970ba9ac0b0161152c7a47ef91 2024-06-26T08:54:51.7841081Z MONITOR_SCRIPT_PID: 21291 2024-06-26T08:54:51.7841447Z ##[endgroup] 2024-06-26T08:54:51.8035615Z Prepare all required actions 2024-06-26T08:54:51.8036084Z Getting action download info 2024-06-26T08:54:51.9630778Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T08:54:52.1101335Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T08:54:52.1101817Z with: 2024-06-26T08:54:52.1102416Z file-suffix: test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916888 2024-06-26T08:54:52.1103145Z s3-bucket: gha-artifacts 2024-06-26T08:54:52.1103498Z env: 2024-06-26T08:54:52.1103775Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:54:52.1104257Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:54:52.1105087Z DOCKER_CONTAINER_ID: 57400afaa3f29946832de03690d2b6347f2f66970ba9ac0b0161152c7a47ef91 2024-06-26T08:54:52.1105968Z ##[endgroup] 2024-06-26T08:54:52.1138118Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-26T08:54:52.1138792Z # Remove any previous test jsons if they exist 2024-06-26T08:54:52.1139311Z rm -f test-jsons-*.zip 2024-06-26T08:54:52.1139840Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-26T08:54:52.1147685Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:54:52.1148193Z env: 2024-06-26T08:54:52.1148482Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:54:52.1148946Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:54:52.1149686Z DOCKER_CONTAINER_ID: 57400afaa3f29946832de03690d2b6347f2f66970ba9ac0b0161152c7a47ef91 2024-06-26T08:54:52.1150632Z FILE_SUFFIX: test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916888 2024-06-26T08:54:52.1151329Z ##[endgroup] 2024-06-26T08:54:52.1321220Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-26T08:54:52.1351171Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-26T08:54:52.1352627Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-26T08:54:52.1357306Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-26T08:54:52.1382817Z ##[group]Run # Remove any previous test reports if they exist 2024-06-26T08:54:52.1383489Z # Remove any previous test reports if they exist 2024-06-26T08:54:52.1384042Z rm -f test-reports-*.zip 2024-06-26T08:54:52.1384769Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-26T08:54:52.1392399Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:54:52.1392912Z env: 2024-06-26T08:54:52.1393205Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:54:52.1393679Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:54:52.1394438Z DOCKER_CONTAINER_ID: 57400afaa3f29946832de03690d2b6347f2f66970ba9ac0b0161152c7a47ef91 2024-06-26T08:54:52.1395420Z FILE_SUFFIX: test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916888 2024-06-26T08:54:52.1396130Z ##[endgroup] 2024-06-26T08:54:52.1549256Z adding: test/test-reports/inference_torchbench.csv (deflated 73%) 2024-06-26T08:54:52.1553640Z adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 92%) 2024-06-26T08:54:52.1555356Z adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 79%) 2024-06-26T08:54:52.1556407Z adding: test/test-reports/training_torchbench.csv (deflated 64%) 2024-06-26T08:54:52.1562141Z adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 96%) 2024-06-26T08:54:52.1563322Z adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 77%) 2024-06-26T08:54:52.1589404Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-26T08:54:52.1590049Z # Remove any previous usage logs if they exist 2024-06-26T08:54:52.1590565Z rm -f logs-*.zip 2024-06-26T08:54:52.1591265Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-26T08:54:52.1592197Z # so check to see if the file exists first 2024-06-26T08:54:52.1592718Z if [ -f 'usage_log.txt' ]; then 2024-06-26T08:54:52.1593261Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-26T08:54:52.1593763Z fi 2024-06-26T08:54:52.1594136Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-26T08:54:52.1594725Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-26T08:54:52.1595233Z fi 2024-06-26T08:54:52.1602552Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:54:52.1603064Z env: 2024-06-26T08:54:52.1603340Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:54:52.1603815Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:54:52.1604566Z DOCKER_CONTAINER_ID: 57400afaa3f29946832de03690d2b6347f2f66970ba9ac0b0161152c7a47ef91 2024-06-26T08:54:52.1605670Z FILE_SUFFIX: test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916888 2024-06-26T08:54:52.1606384Z ##[endgroup] 2024-06-26T08:54:52.1687031Z adding: usage_log.txt (deflated 92%) 2024-06-26T08:54:52.1752731Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-26T08:54:52.1753480Z # Remove any previous debugging artifacts if they exist 2024-06-26T08:54:52.1754042Z rm -f debug-*.zip 2024-06-26T08:54:52.1754430Z if [ -d 'test/debug' ]; then 2024-06-26T08:54:52.1754940Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-26T08:54:52.1755425Z fi 2024-06-26T08:54:52.1762071Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:54:52.1762586Z env: 2024-06-26T08:54:52.1762865Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:54:52.1763334Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:54:52.1764079Z DOCKER_CONTAINER_ID: 57400afaa3f29946832de03690d2b6347f2f66970ba9ac0b0161152c7a47ef91 2024-06-26T08:54:52.1765041Z FILE_SUFFIX: test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916888 2024-06-26T08:54:52.1765744Z ##[endgroup] 2024-06-26T08:54:52.1856244Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:54:52.1856709Z with: 2024-06-26T08:54:52.1856995Z s3-bucket: gha-artifacts 2024-06-26T08:54:52.1857431Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:54:52.1857917Z retention-days: 14 2024-06-26T08:54:52.1858267Z if-no-files-found: warn 2024-06-26T08:54:52.1858629Z path: test-jsons-*.zip 2024-06-26T08:54:52.1858977Z name: artifact 2024-06-26T08:54:52.1859287Z region: us-east-1 2024-06-26T08:54:52.1859583Z env: 2024-06-26T08:54:52.1859861Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:54:52.1860332Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:54:52.1861082Z DOCKER_CONTAINER_ID: 57400afaa3f29946832de03690d2b6347f2f66970ba9ac0b0161152c7a47ef91 2024-06-26T08:54:52.1861735Z ##[endgroup] 2024-06-26T08:54:52.4968385Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:54:52.4969018Z With the provided path, there will be 1 file uploaded 2024-06-26T08:54:52.4969689Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:54:52.4995339Z Starting upload of test-jsons-test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916888.zip 2024-06-26T08:54:52.6199530Z Finished upload of test-jsons-test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916888.zip 2024-06-26T08:54:52.6341157Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:54:52.6341612Z with: 2024-06-26T08:54:52.6341915Z s3-bucket: gha-artifacts 2024-06-26T08:54:52.6342364Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:54:52.6342859Z retention-days: 14 2024-06-26T08:54:52.6343208Z if-no-files-found: error 2024-06-26T08:54:52.6343594Z path: test-reports-*.zip 2024-06-26T08:54:52.6343961Z name: artifact 2024-06-26T08:54:52.6344290Z region: us-east-1 2024-06-26T08:54:52.6344743Z env: 2024-06-26T08:54:52.6345037Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:54:52.6345634Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:54:52.6346386Z DOCKER_CONTAINER_ID: 57400afaa3f29946832de03690d2b6347f2f66970ba9ac0b0161152c7a47ef91 2024-06-26T08:54:52.6347039Z ##[endgroup] 2024-06-26T08:54:52.9503465Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:54:52.9504101Z With the provided path, there will be 1 file uploaded 2024-06-26T08:54:52.9504822Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:54:52.9530384Z Starting upload of test-reports-test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916888.zip 2024-06-26T08:54:53.2703870Z Finished upload of test-reports-test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916888.zip 2024-06-26T08:54:53.2845320Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:54:53.2845768Z with: 2024-06-26T08:54:53.2846220Z s3-bucket: gha-artifacts 2024-06-26T08:54:53.2846666Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:54:53.2847343Z retention-days: 14 2024-06-26T08:54:53.2847705Z if-no-files-found: ignore 2024-06-26T08:54:53.2848083Z path: logs-*.zip 2024-06-26T08:54:53.2848410Z name: artifact 2024-06-26T08:54:53.2848721Z region: us-east-1 2024-06-26T08:54:53.2849032Z env: 2024-06-26T08:54:53.2849319Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:54:53.2849793Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:54:53.2850538Z DOCKER_CONTAINER_ID: 57400afaa3f29946832de03690d2b6347f2f66970ba9ac0b0161152c7a47ef91 2024-06-26T08:54:53.2851184Z ##[endgroup] 2024-06-26T08:54:53.5895545Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:54:53.5896674Z With the provided path, there will be 1 file uploaded 2024-06-26T08:54:53.5897316Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:54:53.5920854Z Starting upload of logs-test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916888.zip 2024-06-26T08:54:53.6988349Z Finished upload of logs-test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916888.zip 2024-06-26T08:54:53.7127303Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:54:53.7127758Z with: 2024-06-26T08:54:53.7128052Z s3-bucket: gha-artifacts 2024-06-26T08:54:53.7128496Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:54:53.7128986Z retention-days: 14 2024-06-26T08:54:53.7129331Z if-no-files-found: ignore 2024-06-26T08:54:53.7129705Z path: debug-*.zip 2024-06-26T08:54:53.7130032Z name: artifact 2024-06-26T08:54:53.7130340Z region: us-east-1 2024-06-26T08:54:53.7130651Z env: 2024-06-26T08:54:53.7130939Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:54:53.7131413Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:54:53.7132156Z DOCKER_CONTAINER_ID: 57400afaa3f29946832de03690d2b6347f2f66970ba9ac0b0161152c7a47ef91 2024-06-26T08:54:53.7132821Z ##[endgroup] 2024-06-26T08:54:54.0135188Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-26T08:54:54.0281385Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T08:54:54.0281884Z # shellcheck disable=SC2156 2024-06-26T08:54:54.0282720Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T08:54:54.0290844Z shell: /usr/bin/bash -e {0} 2024-06-26T08:54:54.0291204Z env: 2024-06-26T08:54:54.0291493Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:54:54.0291968Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:54:54.0292719Z DOCKER_CONTAINER_ID: 57400afaa3f29946832de03690d2b6347f2f66970ba9ac0b0161152c7a47ef91 2024-06-26T08:54:54.0293380Z ##[endgroup] 2024-06-26T08:54:54.1956716Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-26T08:54:54.1957364Z with: 2024-06-26T08:54:54.1957638Z env: 2024-06-26T08:54:54.1957935Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:54:54.1958426Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:54:54.1959189Z DOCKER_CONTAINER_ID: 57400afaa3f29946832de03690d2b6347f2f66970ba9ac0b0161152c7a47ef91 2024-06-26T08:54:54.1959849Z ##[endgroup] 2024-06-26T08:54:54.1977776Z ##[group]Run set -eou pipefail 2024-06-26T08:54:54.1978205Z set -eou pipefail 2024-06-26T08:54:54.1978563Z  2024-06-26T08:54:54.1979091Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-26T08:54:54.1979745Z for _ in $(seq 1440); do 2024-06-26T08:54:54.1980233Z  # Break if no ssh session exists anymore 2024-06-26T08:54:54.1980747Z  if [ "$(who)" = "" ]; then 2024-06-26T08:54:54.1981164Z  break 2024-06-26T08:54:54.1981510Z  fi 2024-06-26T08:54:54.1981815Z  echo "." 2024-06-26T08:54:54.1982152Z  sleep 5 2024-06-26T08:54:54.1982478Z done 2024-06-26T08:54:54.1990122Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:54:54.1990627Z env: 2024-06-26T08:54:54.1990915Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:54:54.1991388Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:54:54.1992147Z DOCKER_CONTAINER_ID: 57400afaa3f29946832de03690d2b6347f2f66970ba9ac0b0161152c7a47ef91 2024-06-26T08:54:54.1992788Z ##[endgroup] 2024-06-26T08:54:54.2012803Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-26T08:54:54.2063678Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:54:54.2064472Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:54:54.2065168Z # shellcheck disable=SC2046 2024-06-26T08:54:54.2065658Z docker stop $(docker ps -q) || true 2024-06-26T08:54:54.2066163Z # Prune all of the docker images 2024-06-26T08:54:54.2066629Z docker system prune -af 2024-06-26T08:54:54.2074323Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:54:54.2074859Z env: 2024-06-26T08:54:54.2075155Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:54:54.2075627Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:54:54.2076379Z DOCKER_CONTAINER_ID: 57400afaa3f29946832de03690d2b6347f2f66970ba9ac0b0161152c7a47ef91 2024-06-26T08:54:54.2077033Z ##[endgroup] 2024-06-26T08:54:54.5402179Z 57400afaa3f2 2024-06-26T08:54:56.8296055Z Deleted Containers: 2024-06-26T08:54:56.8296650Z 57400afaa3f29946832de03690d2b6347f2f66970ba9ac0b0161152c7a47ef91 2024-06-26T08:54:56.8297106Z 2024-06-26T08:55:00.6580309Z Deleted Images: 2024-06-26T08:55:00.6582183Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T08:55:00.6584777Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T08:55:00.6586391Z deleted: sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd 2024-06-26T08:55:00.6587261Z deleted: sha256:754ead7012f0b7691885a9660f02e80be6cee8c5665486fb995c1fb06adb6e92 2024-06-26T08:55:00.6588114Z deleted: sha256:fbe4496d8830e11be86d8f222522e7956f32f85bfc06b05ea53961f34a58af74 2024-06-26T08:55:00.6589009Z deleted: sha256:1c83814aa1a1a7e0cc998e3479bdab8e8766b975560cbf00690acf39ed899a80 2024-06-26T08:55:00.6589917Z deleted: sha256:e9ea8e61942c7f8ae4aab97b8d102b01d6bb4b6cf2a605808cd34130a17d8a0f 2024-06-26T08:55:00.6590801Z deleted: sha256:acc5fc32588cd95e082a6ce35e48d0f303ae8f96e3ab65c4e375b85e1f3c4a92 2024-06-26T08:55:00.6591740Z deleted: sha256:a93e53400904d29e4185d3be942ea34f19572ea438962788ae4e9b3d9bf4c498 2024-06-26T08:55:00.6592927Z deleted: sha256:c97092b88b5e4c77c68bbc4b51afab64e38719af1ff58013ec49a696cdd40a6e 2024-06-26T08:55:00.6593812Z deleted: sha256:b132889c90d67ffa53655846603e3fcc2372c23f04dbdbc54690a39d304eb89b 2024-06-26T08:55:00.6594655Z deleted: sha256:915d54081ed607710624abe3f2490114c745858831f4fc13396bd3b4b9a121bc 2024-06-26T08:55:00.6595493Z deleted: sha256:bfa8d446c498c87f535aef102695564de1b22c755e6c63944ae2f5073a2b31ec 2024-06-26T08:55:00.6596336Z deleted: sha256:e71460fb363c29e70bbe357d680aaa7ae694f094ef2f2713609158ad90611744 2024-06-26T08:55:00.6597194Z deleted: sha256:84079bb1b4ee1d0119ae3d3e701de460a1324912297c675425c9a36d30fcdb0f 2024-06-26T08:55:00.6598084Z deleted: sha256:d86ac3ca148d4758cb03b493fdd49109d8f1a2c7c85c4b755493c3c4529553e5 2024-06-26T08:55:00.6598939Z deleted: sha256:532b3ab97c44a673a0d5f62ee46fa6a83248a84de8877754a612355be56ed0a3 2024-06-26T08:55:00.6599789Z deleted: sha256:617e2a35f25ae24500c88c0c9b8475f0687fbcbed3d200013b4c99c6d8941aff 2024-06-26T08:55:00.6600644Z deleted: sha256:b88120bcd5a26a660c09c5df13176e0ef098fb2c8cdb57d74ab0796a3d1194d6 2024-06-26T08:55:00.6601495Z deleted: sha256:c016b2e7764735243205fb76fa9a273bb3dd4bf05532288e3a613ce022131ac3 2024-06-26T08:55:00.6602492Z deleted: sha256:bcaca462c390deddd2995de2d245a7495e7d2e1e94f0edad30a573c83d0588fd 2024-06-26T08:55:00.6603355Z deleted: sha256:9e260eba003d5c25d7b410e5c4c3093845d9e46611896840901c689d7fbf8ee6 2024-06-26T08:55:00.6604192Z deleted: sha256:3e62b86f8b911a9b59c62d55f063e71683ae6a01c22418db5d27930712adaad6 2024-06-26T08:55:00.6605043Z deleted: sha256:905920ccb1d88449b8dde686c34a4b2a8b6f59e78ac44916b189ea3593b9fbbb 2024-06-26T08:55:00.6605898Z deleted: sha256:961201237e03acd5061406238dd7a3c1343ac888d7583212cd0ab5ae1b4f0540 2024-06-26T08:55:00.6607053Z deleted: sha256:70047a2aac62159fa28fb4dbd4567ec2ffbab68fe046967e09acb0eb3e5cb9cb 2024-06-26T08:55:00.6607923Z deleted: sha256:1ec680ad781436ae65fa987f5fa1ed032ecd5c82a6e35eea6707de501f62c0f4 2024-06-26T08:55:00.6608814Z deleted: sha256:ef859623a50e85abf6e6c1dbd6188aa765a66cc7b2bd618a1cbe4825b7b2bd91 2024-06-26T08:55:00.6609697Z deleted: sha256:404ade5cacff59773b93ab438bfa64cd3a171d07f6f168bdc4a5d4bfa0b43526 2024-06-26T08:55:00.6610574Z deleted: sha256:5b27de6ee3d9f2f18217bf18dec725f7b4954c0e48c85d35752634c22d62209b 2024-06-26T08:55:00.6611441Z deleted: sha256:498e39c575b3fa069acaac7e46b91b4e54ae3a05f2ada5c4347091f346f25d4c 2024-06-26T08:55:00.6612277Z deleted: sha256:1405919d2694d65669b1f4c842282021afe89208ce86570e96464903733da2d0 2024-06-26T08:55:00.6613115Z deleted: sha256:d5b41e5d70896c0472fc3e6da0516a17d89cd6635dc75010ecddc31ce9690fad 2024-06-26T08:55:00.6613966Z deleted: sha256:18e9379b62b9d0bb40293d28a652eafaee66a499f66f0776bf14e37a513101b7 2024-06-26T08:55:00.6614820Z deleted: sha256:a781f80ed00ff2926a699fd01a309bb0b4fb7d5655eb652c205bb9c7f811035f 2024-06-26T08:55:00.6615661Z deleted: sha256:c0430973b7cf759075330ee29c79bfd1312c91e6033d9dd883b4b1c29743a171 2024-06-26T08:55:00.6616516Z deleted: sha256:e733b3d66efb9342e5a283376c0e1ddffa47a42fecc26e61ecaf0a3472885d97 2024-06-26T08:55:00.6617558Z deleted: sha256:8929c0b2047f74b84c82c39eff9077a92ae1a828cde00c9ab78fb3e0ed5e0283 2024-06-26T08:55:00.6618406Z deleted: sha256:f5b56155fb58a0a60fe6391d3581cf91610ed69d4f0a321257c19944da296a52 2024-06-26T08:55:00.6619278Z deleted: sha256:b54d2cb355238d1aa91e58df93064447b80f53706d1ca62823acc834f5ac54fe 2024-06-26T08:55:00.6620135Z deleted: sha256:e7311285af9eeddb51aa5629aa98386e89f1456ff32843b6aee6443247edf66c 2024-06-26T08:55:00.6620978Z deleted: sha256:67657ceec347da8dc8085d5731633efeac115ba677fa4b7c5966c4260f7abab0 2024-06-26T08:55:00.6621827Z deleted: sha256:b496e50409dbf414930fc9b4437aa0aa4ce27bc7840e050ae459aaffcf9aca86 2024-06-26T08:55:00.6622692Z deleted: sha256:8f5276170cc4a60d36ad7b964a502306b29546254cedcbcb101ea9123497d2c0 2024-06-26T08:55:00.6623523Z deleted: sha256:53d524d1564745e96a7e515462171b73184d9f1ff5454b32fcfbbca39d361467 2024-06-26T08:55:00.6624378Z deleted: sha256:2acbadb00d69b6aca575f04baba4ea26ae2b5c2d85f09945004ad93a78a8eb75 2024-06-26T08:55:00.6625457Z deleted: sha256:02665efeff878aeb603c1edb2e21cb700c171c025d64f5aae1b2b20b12f3964a 2024-06-26T08:55:00.6626334Z deleted: sha256:19bd23e9a51dfe883a8b8d2ff6d66ff1b3476b8912856406bb79b81762aa6d4b 2024-06-26T08:55:00.6627191Z deleted: sha256:73aa8637bf9299d6298dabd858480c274a62117388f2ca1d5e9eb15c6d1e4155 2024-06-26T08:55:00.6628035Z deleted: sha256:5f8968406305f348b1caebc0e035187d6e9965ace7b9d5f2f21114d37c9fe2e1 2024-06-26T08:55:00.6628956Z deleted: sha256:d827104914db0c503cf5c81f13e2456bf4efce42d4cf1f8d5d70838207278bee 2024-06-26T08:55:00.6629811Z deleted: sha256:867b219724f2f1aaf56f42e3e9f03a5d64bdca8a6a72407efd10328aeea5d510 2024-06-26T08:55:00.6630682Z deleted: sha256:b9ff2a7cceb9237b26bcb17728edf8a558eec5a76f63626e84f80541dfc0f1d4 2024-06-26T08:55:00.6631552Z deleted: sha256:be73f509106819007ce2b0f5a1301ac56ae583cb6c1fb69bee93efa0840368f1 2024-06-26T08:55:00.6632413Z deleted: sha256:2a11c07c915c3fdbb4e73096dea73ea950c970d50fe1a5764060bd62b7b7cf2c 2024-06-26T08:55:00.6633277Z deleted: sha256:5e9989622e2abc1c72f8184343507b45d75e4ff971edb68b96cbcd75ae484ff5 2024-06-26T08:55:00.6634124Z deleted: sha256:05ce642bd9cf5a9205bff107df28b1d6cef7d5f6ab3ea944d28c032ec61270df 2024-06-26T08:55:00.6634969Z deleted: sha256:017702617f6c07b73f2088678f4710b794cfe897a213a5cf122a0e2519eae0ac 2024-06-26T08:55:00.6635902Z deleted: sha256:44b3b37bb1ca39dc5cf356de41a23a5c12f605a1992813e408087fb2b0ad6d1c 2024-06-26T08:55:00.6636748Z deleted: sha256:8c387982e9f27db357a1d149dc149cabc0bd04668ad6956387a5072682dada67 2024-06-26T08:55:00.6637583Z deleted: sha256:10d415dca4e3fbb88a747b40317f3c6333c041ba8f73855f67fb9fe981fae5ee 2024-06-26T08:55:00.6638436Z deleted: sha256:6c65469bf0f53dcd625faefe363b8f3136c08388a6cf64d9158e8576e4ee89b1 2024-06-26T08:55:00.6639340Z deleted: sha256:027db5d8b76deeefbd8c34b485ab7ff14ed2a3e6800425c1ce0d77c2e9fdd4ab 2024-06-26T08:55:00.6640196Z deleted: sha256:24941234ad28cf5c8174d7cf18326c10b4e17f2c2137672ea0322df461597dd7 2024-06-26T08:55:00.6641041Z deleted: sha256:240ea2eebfa8519d6c2d88f4c99e66ce4e038d000f93ce8d03726c01d81180ef 2024-06-26T08:55:00.6641890Z deleted: sha256:09e823cb391a69f6a195e3ca8a41624b5f8b34eee14c6eb36277de814a91966e 2024-06-26T08:55:00.6642718Z deleted: sha256:730479d695169a6593bf0ac16b377491dce65175a6d158e012422367d04577fd 2024-06-26T08:55:00.6643553Z deleted: sha256:b0a30f9f7bd5459ff662453b9346a7ba8ea8db1191528345c1953eab4aa2f54e 2024-06-26T08:55:00.6644396Z deleted: sha256:75cefa87effaa3ad361c74642839923101f0f9982f814f3c9e2ed4ad52f2a008 2024-06-26T08:55:00.6645223Z deleted: sha256:6f97db87ba9512e615e871c6982000602982e1e5a5be334a56e7292251e84e00 2024-06-26T08:55:00.6646057Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-06-26T08:55:00.6647661Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-06-26T08:55:00.6648519Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-06-26T08:55:00.6649357Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-06-26T08:55:00.6650218Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-06-26T08:55:00.6651071Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-06-26T08:55:00.6651942Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-06-26T08:55:00.6652803Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-06-26T08:55:00.6653674Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-06-26T08:55:00.6654539Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-06-26T08:55:00.6655385Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-06-26T08:55:00.6655896Z 2024-06-26T08:55:00.6656061Z Total reclaimed space: 40.92GB 2024-06-26T08:55:00.6721634Z Post job cleanup. 2024-06-26T08:55:00.6771844Z Post job cleanup. 2024-06-26T08:55:00.7584102Z [command]/usr/bin/git version 2024-06-26T08:55:00.7625444Z git version 2.40.1 2024-06-26T08:55:00.7665307Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/85e5d610-20df-4108-adb9-caf905650383' before making global git config changes 2024-06-26T08:55:00.7666624Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T08:55:00.7670937Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T08:55:00.7703821Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T08:55:00.7733454Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T08:55:00.7954876Z Entering 'android/libs/fbjni' 2024-06-26T08:55:00.7994228Z Entering 'third_party/FP16' 2024-06-26T08:55:00.8031858Z Entering 'third_party/FXdiv' 2024-06-26T08:55:00.8071150Z Entering 'third_party/NNPACK' 2024-06-26T08:55:00.8107159Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:55:00.8145001Z Entering 'third_party/XNNPACK' 2024-06-26T08:55:00.8193659Z Entering 'third_party/benchmark' 2024-06-26T08:55:00.8231601Z Entering 'third_party/cpp-httplib' 2024-06-26T08:55:00.8267311Z Entering 'third_party/cpuinfo' 2024-06-26T08:55:00.8304675Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:55:00.8343395Z Entering 'third_party/cutlass' 2024-06-26T08:55:00.8385750Z Entering 'third_party/eigen' 2024-06-26T08:55:00.8420699Z Entering 'third_party/fbgemm' 2024-06-26T08:55:00.8458896Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:55:00.8493938Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:55:00.8528817Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:55:00.8568732Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:55:00.8604401Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:55:00.8639668Z Entering 'third_party/flatbuffers' 2024-06-26T08:55:00.8679551Z Entering 'third_party/fmt' 2024-06-26T08:55:00.8714498Z Entering 'third_party/foxi' 2024-06-26T08:55:00.8750419Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:55:00.8785019Z Entering 'third_party/gloo' 2024-06-26T08:55:00.8820037Z Entering 'third_party/googletest' 2024-06-26T08:55:00.8857460Z Entering 'third_party/ideep' 2024-06-26T08:55:00.8893508Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:55:00.8934452Z Entering 'third_party/ittapi' 2024-06-26T08:55:00.8970277Z Entering 'third_party/kineto' 2024-06-26T08:55:00.9001463Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:55:00.9036931Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:55:00.9072820Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:55:00.9108754Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:55:00.9143002Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:55:00.9175787Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:55:00.9213389Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:55:00.9248988Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:55:00.9283841Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:55:00.9320737Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:55:00.9357575Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:55:00.9393267Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:55:00.9428273Z Entering 'third_party/mimalloc' 2024-06-26T08:55:00.9466536Z Entering 'third_party/nccl/nccl' 2024-06-26T08:55:00.9503746Z Entering 'third_party/nlohmann' 2024-06-26T08:55:00.9538586Z Entering 'third_party/onnx' 2024-06-26T08:55:00.9586398Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:55:00.9622874Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:55:00.9658379Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:55:00.9696370Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:55:00.9732091Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:55:00.9767455Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:55:00.9801602Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:55:00.9834609Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:55:00.9869348Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:55:00.9904270Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:55:00.9939013Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:55:00.9974581Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:55:01.0010111Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:55:01.0061147Z Entering 'third_party/pocketfft' 2024-06-26T08:55:01.0094819Z Entering 'third_party/protobuf' 2024-06-26T08:55:01.0133792Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:55:01.0166995Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:55:01.0202769Z Entering 'third_party/psimd' 2024-06-26T08:55:01.0238613Z Entering 'third_party/pthreadpool' 2024-06-26T08:55:01.0276246Z Entering 'third_party/pybind11' 2024-06-26T08:55:01.0310560Z Entering 'third_party/python-peachpy' 2024-06-26T08:55:01.0345782Z Entering 'third_party/sleef' 2024-06-26T08:55:01.0381048Z Entering 'third_party/tensorpipe' 2024-06-26T08:55:01.0416461Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:55:01.0451508Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:55:01.0485935Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:55:01.0518816Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:55:01.0552482Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:55:01.0601225Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T08:55:01.0621688Z http.https://github.com/.extraheader 2024-06-26T08:55:01.0629762Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T08:55:01.0658380Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T08:55:01.0863499Z Entering 'android/libs/fbjni' 2024-06-26T08:55:01.0884744Z http.https://github.com/.extraheader 2024-06-26T08:55:01.0911009Z Entering 'third_party/FP16' 2024-06-26T08:55:01.0932163Z http.https://github.com/.extraheader 2024-06-26T08:55:01.0956039Z Entering 'third_party/FXdiv' 2024-06-26T08:55:01.0978019Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1000773Z Entering 'third_party/NNPACK' 2024-06-26T08:55:01.1023829Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1048892Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:55:01.1071885Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1094574Z Entering 'third_party/XNNPACK' 2024-06-26T08:55:01.1115953Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1152429Z Entering 'third_party/benchmark' 2024-06-26T08:55:01.1175191Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1199180Z Entering 'third_party/cpp-httplib' 2024-06-26T08:55:01.1221348Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1245324Z Entering 'third_party/cpuinfo' 2024-06-26T08:55:01.1269053Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1293176Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:55:01.1315508Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1339507Z Entering 'third_party/cutlass' 2024-06-26T08:55:01.1361350Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1389877Z Entering 'third_party/eigen' 2024-06-26T08:55:01.1412821Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1437941Z Entering 'third_party/fbgemm' 2024-06-26T08:55:01.1460797Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1485201Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:55:01.1508202Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1531379Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:55:01.1553194Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1577732Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:55:01.1599383Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1627016Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:55:01.1646672Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1671075Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:55:01.1692791Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1716823Z Entering 'third_party/flatbuffers' 2024-06-26T08:55:01.1739886Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1765795Z Entering 'third_party/fmt' 2024-06-26T08:55:01.1789315Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1813822Z Entering 'third_party/foxi' 2024-06-26T08:55:01.1837276Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1861102Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:55:01.1881401Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1904649Z Entering 'third_party/gloo' 2024-06-26T08:55:01.1925296Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1947234Z Entering 'third_party/googletest' 2024-06-26T08:55:01.1967888Z http.https://github.com/.extraheader 2024-06-26T08:55:01.1992897Z Entering 'third_party/ideep' 2024-06-26T08:55:01.2014787Z http.https://github.com/.extraheader 2024-06-26T08:55:01.2037684Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:55:01.2059629Z http.https://github.com/.extraheader 2024-06-26T08:55:01.2089040Z Entering 'third_party/ittapi' 2024-06-26T08:55:01.2111117Z http.https://github.com/.extraheader 2024-06-26T08:55:01.2134207Z Entering 'third_party/kineto' 2024-06-26T08:55:01.2156908Z http.https://github.com/.extraheader 2024-06-26T08:55:01.2181403Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:55:01.2201592Z http.https://github.com/.extraheader 2024-06-26T08:55:01.2226321Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:55:01.2247412Z http.https://github.com/.extraheader 2024-06-26T08:55:01.2273931Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:55:01.2296038Z http.https://github.com/.extraheader 2024-06-26T08:55:01.2320374Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:55:01.2341983Z http.https://github.com/.extraheader 2024-06-26T08:55:01.2365560Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:55:01.2387951Z http.https://github.com/.extraheader 2024-06-26T08:55:01.2411169Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:55:01.2433075Z http.https://github.com/.extraheader 2024-06-26T08:55:01.2456845Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:55:01.2478536Z http.https://github.com/.extraheader 2024-06-26T08:55:01.2502919Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:55:01.2524736Z http.https://github.com/.extraheader 2024-06-26T08:55:01.2548489Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:55:01.2569725Z http.https://github.com/.extraheader 2024-06-26T08:55:01.2593919Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:55:01.2615816Z http.https://github.com/.extraheader 2024-06-26T08:55:01.2640067Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:55:01.2661945Z http.https://github.com/.extraheader 2024-06-26T08:55:01.2685073Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:55:01.2707384Z http.https://github.com/.extraheader 2024-06-26T08:55:01.2733608Z Entering 'third_party/mimalloc' 2024-06-26T08:55:01.2755119Z http.https://github.com/.extraheader 2024-06-26T08:55:01.2779882Z Entering 'third_party/nccl/nccl' 2024-06-26T08:55:01.2804155Z http.https://github.com/.extraheader 2024-06-26T08:55:01.2828536Z Entering 'third_party/nlohmann' 2024-06-26T08:55:01.2849345Z http.https://github.com/.extraheader 2024-06-26T08:55:01.2873524Z Entering 'third_party/onnx' 2024-06-26T08:55:01.2895776Z http.https://github.com/.extraheader 2024-06-26T08:55:01.2931855Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:55:01.2957104Z http.https://github.com/.extraheader 2024-06-26T08:55:01.2980599Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:55:01.3001295Z http.https://github.com/.extraheader 2024-06-26T08:55:01.3029102Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:55:01.3051571Z http.https://github.com/.extraheader 2024-06-26T08:55:01.3076134Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:55:01.3098289Z http.https://github.com/.extraheader 2024-06-26T08:55:01.3122957Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:55:01.3145517Z http.https://github.com/.extraheader 2024-06-26T08:55:01.3170358Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:55:01.3191484Z http.https://github.com/.extraheader 2024-06-26T08:55:01.3215523Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:55:01.3236675Z http.https://github.com/.extraheader 2024-06-26T08:55:01.3260417Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:55:01.3281514Z http.https://github.com/.extraheader 2024-06-26T08:55:01.3305583Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:55:01.3326628Z http.https://github.com/.extraheader 2024-06-26T08:55:01.3350067Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:55:01.3370364Z http.https://github.com/.extraheader 2024-06-26T08:55:01.3393858Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:55:01.3416721Z http.https://github.com/.extraheader 2024-06-26T08:55:01.3442280Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:55:01.3464746Z http.https://github.com/.extraheader 2024-06-26T08:55:01.3490498Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:55:01.3511811Z http.https://github.com/.extraheader 2024-06-26T08:55:01.3552139Z Entering 'third_party/pocketfft' 2024-06-26T08:55:01.3575363Z http.https://github.com/.extraheader 2024-06-26T08:55:01.3599261Z Entering 'third_party/protobuf' 2024-06-26T08:55:01.3621560Z http.https://github.com/.extraheader 2024-06-26T08:55:01.3646465Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:55:01.3668654Z http.https://github.com/.extraheader 2024-06-26T08:55:01.3691848Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:55:01.3712700Z http.https://github.com/.extraheader 2024-06-26T08:55:01.3738249Z Entering 'third_party/psimd' 2024-06-26T08:55:01.3760643Z http.https://github.com/.extraheader 2024-06-26T08:55:01.3785839Z Entering 'third_party/pthreadpool' 2024-06-26T08:55:01.3807982Z http.https://github.com/.extraheader 2024-06-26T08:55:01.3832043Z Entering 'third_party/pybind11' 2024-06-26T08:55:01.3853011Z http.https://github.com/.extraheader 2024-06-26T08:55:01.3876137Z Entering 'third_party/python-peachpy' 2024-06-26T08:55:01.3898890Z http.https://github.com/.extraheader 2024-06-26T08:55:01.3922367Z Entering 'third_party/sleef' 2024-06-26T08:55:01.3944370Z http.https://github.com/.extraheader 2024-06-26T08:55:01.3968280Z Entering 'third_party/tensorpipe' 2024-06-26T08:55:01.3989747Z http.https://github.com/.extraheader 2024-06-26T08:55:01.4013393Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:55:01.4034496Z http.https://github.com/.extraheader 2024-06-26T08:55:01.4057291Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:55:01.4079222Z http.https://github.com/.extraheader 2024-06-26T08:55:01.4102771Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:55:01.4124261Z http.https://github.com/.extraheader 2024-06-26T08:55:01.4148820Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:55:01.4170991Z http.https://github.com/.extraheader 2024-06-26T08:55:01.4192371Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:55:01.4214265Z http.https://github.com/.extraheader 2024-06-26T08:55:01.4310813Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-26T08:55:01.4328615Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T08:55:01.4335325Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:55:01.4335870Z ##[endgroup] 2024-06-26T08:55:02.2311173Z Cleaning up orphan processes