2024-05-19T18:07:26.2048528Z Current runner version: '2.316.1' 2024-05-19T18:07:26.2054734Z Runner name: 'i-01c138b15d87f731e' 2024-05-19T18:07:26.2055621Z Runner group name: 'Default' 2024-05-19T18:07:26.2056563Z Machine name: 'ip-10-0-40-27' 2024-05-19T18:07:26.2061636Z ##[group]GITHUB_TOKEN Permissions 2024-05-19T18:07:26.2063617Z Actions: read 2024-05-19T18:07:26.2064237Z Attestations: read 2024-05-19T18:07:26.2064779Z Checks: read 2024-05-19T18:07:26.2065282Z Contents: read 2024-05-19T18:07:26.2065821Z Deployments: read 2024-05-19T18:07:26.2066366Z Discussions: read 2024-05-19T18:07:26.2066876Z Issues: read 2024-05-19T18:07:26.2067422Z Metadata: read 2024-05-19T18:07:26.2067944Z Packages: read 2024-05-19T18:07:26.2068470Z Pages: read 2024-05-19T18:07:26.2068990Z PullRequests: read 2024-05-19T18:07:26.2069567Z RepositoryProjects: read 2024-05-19T18:07:26.2070211Z SecurityEvents: read 2024-05-19T18:07:26.2070769Z Statuses: read 2024-05-19T18:07:26.2071334Z ##[endgroup] 2024-05-19T18:07:26.2074590Z Secret source: Actions 2024-05-19T18:07:26.2075369Z Prepare workflow directory 2024-05-19T18:07:26.3004406Z Prepare all required actions 2024-05-19T18:07:26.3171284Z Getting action download info 2024-05-19T18:07:26.4579189Z Download action repository 'pytorch/test-infra@main' (SHA:5ab7a0ff6617c37f26f7d3ce525c072902906bdf) 2024-05-19T18:07:26.7438866Z Download action repository 'pytorch/pytorch@main' (SHA:853081a8e7e84ce3f09749929bb3cb998fd95b30) 2024-05-19T18:07:29.4829673Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-05-19T18:07:29.5904984Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-05-19T18:07:29.8688009Z Getting action download info 2024-05-19T18:07:29.9598690Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-05-19T18:08:11.2259668Z Getting action download info 2024-05-19T18:08:11.3319472Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-05-19T18:08:11.4558809Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/121956 (c8c7ddfde57442cc28a394c385cab656ce9b1f5e) 2024-05-19T18:08:11.4561021Z ##[group] Inputs 2024-05-19T18:08:11.4561500Z build-environment: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:08:11.4569606Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-05-19T18:08:11.4578671Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:08:11.4579849Z sync-tag: 2024-05-19T18:08:11.4580731Z timeout-minutes: 240 2024-05-19T18:08:11.4581070Z use-gha: 2024-05-19T18:08:11.4581363Z dashboard-tag: 2024-05-19T18:08:11.4581688Z s3-bucket: gha-artifacts 2024-05-19T18:08:11.4582045Z aws-role-to-assume: 2024-05-19T18:08:11.4582386Z ##[endgroup] 2024-05-19T18:08:11.4583254Z Complete job name: cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:08:11.5459256Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-05-19T18:08:11.5459824Z with: 2024-05-19T18:08:11.5460461Z github-secret: *** 2024-05-19T18:08:11.5461377Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-05-19T18:08:11.5462357Z activate-with-label: false 2024-05-19T18:08:11.5462729Z label: with-ssh 2024-05-19T18:08:11.5463056Z remove-existing-keys: true 2024-05-19T18:08:11.5463413Z env: 2024-05-19T18:08:11.5463695Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:08:11.5464040Z ##[endgroup] 2024-05-19T18:08:11.6365104Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-05-19T18:08:11.6366154Z ciflow reference detected, attempting to extract PR number 2024-05-19T18:08:13.3239870Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-05-19T18:08:13.3800247Z No SSH keys found for user pytorch-bot[bot] 2024-05-19T18:08:13.3801050Z Grabbing public ssh keys from https://github.com/nWEIdia.keys 2024-05-19T18:08:13.4784031Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-05-19T18:08:13.4797497Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-05-19T18:08:13.4825416Z Login using: ssh ec2-user@ec2-44-202-49-198.compute-1.amazonaws.com 2024-05-19T18:08:13.4826395Z All testing is done inside the container, to start an interactive session run: 2024-05-19T18:08:13.4827194Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-05-19T18:08:13.5072438Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-05-19T18:08:13.5072983Z with: 2024-05-19T18:08:13.5073272Z submodules: recursive 2024-05-19T18:08:13.5073604Z fetch-depth: 0 2024-05-19T18:08:13.5073890Z env: 2024-05-19T18:08:13.5074158Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:08:13.5074493Z ##[endgroup] 2024-05-19T18:08:13.5287986Z ##[group]Run retry () { 2024-05-19T18:08:13.5288393Z retry () { 2024-05-19T18:08:13.5288907Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-05-19T18:08:13.5289450Z } 2024-05-19T18:08:13.5289762Z echo "${GITHUB_WORKSPACE}" 2024-05-19T18:08:13.5290176Z if [ -z "${NO_SUDO}" ]; then 2024-05-19T18:08:13.5290647Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-05-19T18:08:13.5291092Z else 2024-05-19T18:08:13.5291431Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-05-19T18:08:13.5291842Z fi 2024-05-19T18:08:13.5292187Z mkdir "${GITHUB_WORKSPACE}" 2024-05-19T18:08:13.5303811Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:08:13.5304308Z env: 2024-05-19T18:08:13.5304586Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:08:13.5304934Z NO_SUDO: 2024-05-19T18:08:13.5305216Z ##[endgroup] 2024-05-19T18:08:13.5395723Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T18:08:13.5593236Z ##[group]Run malfet/checkout@silent-checkout 2024-05-19T18:08:13.5593661Z with: 2024-05-19T18:08:13.5593990Z ref: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:08:13.5594428Z fetch-depth: 0 2024-05-19T18:08:13.5594737Z submodules: recursive 2024-05-19T18:08:13.5595064Z quiet-checkout: true 2024-05-19T18:08:13.5595407Z repository: pytorch/pytorch 2024-05-19T18:08:13.5595882Z token: *** 2024-05-19T18:08:13.5596170Z ssh-strict: true 2024-05-19T18:08:13.5596485Z persist-credentials: true 2024-05-19T18:08:13.5596836Z clean: true 2024-05-19T18:08:13.5597162Z sparse-checkout-cone-mode: true 2024-05-19T18:08:13.5597541Z lfs: false 2024-05-19T18:08:13.5597832Z set-safe-directory: true 2024-05-19T18:08:13.5598162Z env: 2024-05-19T18:08:13.5598431Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:08:13.5598866Z ##[endgroup] 2024-05-19T18:08:13.6507930Z Syncing repository: pytorch/pytorch 2024-05-19T18:08:13.6509441Z ##[group]Getting Git version info 2024-05-19T18:08:13.6510205Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-05-19T18:08:13.6511064Z [command]/usr/bin/git version 2024-05-19T18:08:13.6511425Z git version 2.40.1 2024-05-19T18:08:13.6512739Z ##[endgroup] 2024-05-19T18:08:13.6523736Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/a5d3b4d3-6e8c-402d-a868-dae5eda5b3ff' before making global git config changes 2024-05-19T18:08:13.6525051Z Adding repository directory to the temporary git global config as a safe directory 2024-05-19T18:08:13.6526166Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T18:08:13.6551607Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-05-19T18:08:13.6555243Z ##[group]Initializing the repository 2024-05-19T18:08:13.6558032Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T18:08:13.6581420Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-05-19T18:08:13.6582355Z hint: is subject to change. To configure the initial branch name to use in all 2024-05-19T18:08:13.6583351Z hint: of your new repositories, which will suppress this warning, call: 2024-05-19T18:08:13.6583912Z hint: 2024-05-19T18:08:13.6584350Z hint: git config --global init.defaultBranch 2024-05-19T18:08:13.6584819Z hint: 2024-05-19T18:08:13.6585327Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-05-19T18:08:13.6586184Z hint: 'development'. The just-created branch can be renamed via this command: 2024-05-19T18:08:13.6587017Z hint: 2024-05-19T18:08:13.6587352Z hint: git branch -m 2024-05-19T18:08:13.6588086Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-05-19T18:08:13.6590753Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-05-19T18:08:13.6613719Z ##[endgroup] 2024-05-19T18:08:13.6614327Z ##[group]Disabling automatic garbage collection 2024-05-19T18:08:13.6616382Z [command]/usr/bin/git config --local gc.auto 0 2024-05-19T18:08:13.6634722Z ##[endgroup] 2024-05-19T18:08:13.6635265Z ##[group]Setting up auth 2024-05-19T18:08:13.6640588Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-05-19T18:08:13.6660527Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-05-19T18:08:13.6871992Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-05-19T18:08:13.6891325Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-05-19T18:08:13.7086654Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-05-19T18:08:13.7121630Z ##[endgroup] 2024-05-19T18:08:13.7122187Z ##[group]Fetching the repository 2024-05-19T18:08:13.7127967Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-05-19T18:08:23.6786113Z remote: Enumerating objects: 1118700 2024-05-19T18:08:23.6789792Z remote: Enumerating objects: 1119785, done. 2024-05-19T18:08:23.6791022Z remote: Counting objects: 0% (1/1085) 2024-05-19T18:08:23.6791989Z remote: Counting objects: 1% (11/1085) 2024-05-19T18:08:23.6792625Z remote: Counting objects: 2% (22/1085) 2024-05-19T18:08:23.6793562Z remote: Counting objects: 3% (33/1085) 2024-05-19T18:08:23.6794470Z remote: Counting objects: 4% (44/1085) 2024-05-19T18:08:23.6795303Z remote: Counting objects: 5% (55/1085) 2024-05-19T18:08:23.6795976Z remote: Counting objects: 6% (66/1085) 2024-05-19T18:08:23.6796665Z remote: Counting objects: 7% (76/1085) 2024-05-19T18:08:23.6797189Z remote: Counting objects: 8% (87/1085) 2024-05-19T18:08:23.6797730Z remote: Counting objects: 9% (98/1085) 2024-05-19T18:08:23.6798255Z remote: Counting objects: 10% (109/1085) 2024-05-19T18:08:23.6798759Z remote: Counting objects: 11% (120/1085) 2024-05-19T18:08:23.6799255Z remote: Counting objects: 12% (131/1085) 2024-05-19T18:08:23.6799742Z remote: Counting objects: 13% (142/1085) 2024-05-19T18:08:23.6800245Z remote: Counting objects: 14% (152/1085) 2024-05-19T18:08:23.6800749Z remote: Counting objects: 15% (163/1085) 2024-05-19T18:08:23.6801240Z remote: Counting objects: 16% (174/1085) 2024-05-19T18:08:23.6801723Z remote: Counting objects: 17% (185/1085) 2024-05-19T18:08:23.6802220Z remote: Counting objects: 18% (196/1085) 2024-05-19T18:08:23.6802712Z remote: Counting objects: 19% (207/1085) 2024-05-19T18:08:23.6803203Z remote: Counting objects: 20% (217/1085) 2024-05-19T18:08:23.6803692Z remote: Counting objects: 21% (228/1085) 2024-05-19T18:08:23.6804194Z remote: Counting objects: 22% (239/1085) 2024-05-19T18:08:23.6804798Z remote: Counting objects: 23% (250/1085) 2024-05-19T18:08:23.6805288Z remote: Counting objects: 24% (261/1085) 2024-05-19T18:08:23.6805767Z remote: Counting objects: 25% (272/1085) 2024-05-19T18:08:23.6806260Z remote: Counting objects: 26% (283/1085) 2024-05-19T18:08:23.6806751Z remote: Counting objects: 27% (293/1085) 2024-05-19T18:08:23.6807564Z remote: Counting objects: 28% (304/1085) 2024-05-19T18:08:23.6808055Z remote: Counting objects: 29% (315/1085) 2024-05-19T18:08:23.6808543Z remote: Counting objects: 30% (326/1085) 2024-05-19T18:08:23.6809035Z remote: Counting objects: 31% (337/1085) 2024-05-19T18:08:23.6809521Z remote: Counting objects: 32% (348/1085) 2024-05-19T18:08:23.6810001Z remote: Counting objects: 33% (359/1085) 2024-05-19T18:08:23.6810498Z remote: Counting objects: 34% (369/1085) 2024-05-19T18:08:23.6810985Z remote: Counting objects: 35% (380/1085) 2024-05-19T18:08:23.6811474Z remote: Counting objects: 36% (391/1085) 2024-05-19T18:08:23.6811954Z remote: Counting objects: 37% (402/1085) 2024-05-19T18:08:23.6812439Z remote: Counting objects: 38% (413/1085) 2024-05-19T18:08:23.6812928Z remote: Counting objects: 39% (424/1085) 2024-05-19T18:08:23.6813416Z remote: Counting objects: 40% (434/1085) 2024-05-19T18:08:23.6813902Z remote: Counting objects: 41% (445/1085) 2024-05-19T18:08:23.6814392Z remote: Counting objects: 42% (456/1085) 2024-05-19T18:08:23.6814879Z remote: Counting objects: 43% (467/1085) 2024-05-19T18:08:23.6815370Z remote: Counting objects: 44% (478/1085) 2024-05-19T18:08:23.6815853Z remote: Counting objects: 45% (489/1085) 2024-05-19T18:08:23.6816343Z remote: Counting objects: 46% (500/1085) 2024-05-19T18:08:23.6816930Z remote: Counting objects: 47% (510/1085) 2024-05-19T18:08:23.6817413Z remote: Counting objects: 48% (521/1085) 2024-05-19T18:08:23.6818156Z remote: Counting objects: 49% (532/1085) 2024-05-19T18:08:23.6818647Z remote: Counting objects: 50% (543/1085) 2024-05-19T18:08:23.6819135Z remote: Counting objects: 51% (554/1085) 2024-05-19T18:08:23.6819615Z remote: Counting objects: 52% (565/1085) 2024-05-19T18:08:23.6820107Z remote: Counting objects: 53% (576/1085) 2024-05-19T18:08:23.6820606Z remote: Counting objects: 54% (586/1085) 2024-05-19T18:08:23.6821096Z remote: Counting objects: 55% (597/1085) 2024-05-19T18:08:23.6821578Z remote: Counting objects: 56% (608/1085) 2024-05-19T18:08:23.6822068Z remote: Counting objects: 57% (619/1085) 2024-05-19T18:08:23.6822558Z remote: Counting objects: 58% (630/1085) 2024-05-19T18:08:23.6823058Z remote: Counting objects: 59% (641/1085) 2024-05-19T18:08:23.6823544Z remote: Counting objects: 60% (651/1085) 2024-05-19T18:08:23.6824055Z remote: Counting objects: 61% (662/1085) 2024-05-19T18:08:23.6824544Z remote: Counting objects: 62% (673/1085) 2024-05-19T18:08:23.6825043Z remote: Counting objects: 63% (684/1085) 2024-05-19T18:08:23.6825535Z remote: Counting objects: 64% (695/1085) 2024-05-19T18:08:23.6826106Z remote: Counting objects: 65% (706/1085) 2024-05-19T18:08:23.6826847Z remote: Counting objects: 66% (717/1085) 2024-05-19T18:08:23.6827453Z remote: Counting objects: 67% (727/1085) 2024-05-19T18:08:23.6827946Z remote: Counting objects: 68% (738/1085) 2024-05-19T18:08:23.6828436Z remote: Counting objects: 69% (749/1085) 2024-05-19T18:08:23.6828920Z remote: Counting objects: 70% (760/1085) 2024-05-19T18:08:23.6829408Z remote: Counting objects: 71% (771/1085) 2024-05-19T18:08:23.6829906Z remote: Counting objects: 72% (782/1085) 2024-05-19T18:08:23.6830391Z remote: Counting objects: 73% (793/1085) 2024-05-19T18:08:23.6830874Z remote: Counting objects: 74% (803/1085) 2024-05-19T18:08:23.6831371Z remote: Counting objects: 75% (814/1085) 2024-05-19T18:08:23.6831865Z remote: Counting objects: 76% (825/1085) 2024-05-19T18:08:23.6832355Z remote: Counting objects: 77% (836/1085) 2024-05-19T18:08:23.6832840Z remote: Counting objects: 78% (847/1085) 2024-05-19T18:08:23.6833443Z remote: Counting objects: 79% (858/1085) 2024-05-19T18:08:23.6833942Z remote: Counting objects: 80% (868/1085) 2024-05-19T18:08:23.6834439Z remote: Counting objects: 81% (879/1085) 2024-05-19T18:08:23.6834929Z remote: Counting objects: 82% (890/1085) 2024-05-19T18:08:23.6835419Z remote: Counting objects: 83% (901/1085) 2024-05-19T18:08:23.6835907Z remote: Counting objects: 84% (912/1085) 2024-05-19T18:08:23.6836406Z remote: Counting objects: 85% (923/1085) 2024-05-19T18:08:23.6836885Z remote: Counting objects: 86% (934/1085) 2024-05-19T18:08:23.6837392Z remote: Counting objects: 87% (944/1085) 2024-05-19T18:08:23.6837921Z remote: Counting objects: 88% (955/1085) 2024-05-19T18:08:23.6838409Z remote: Counting objects: 89% (966/1085) 2024-05-19T18:08:23.6838904Z remote: Counting objects: 90% (977/1085) 2024-05-19T18:08:23.6839405Z remote: Counting objects: 91% (988/1085) 2024-05-19T18:08:23.6839909Z remote: Counting objects: 92% (999/1085) 2024-05-19T18:08:23.6840412Z remote: Counting objects: 93% (1010/1085) 2024-05-19T18:08:23.6840938Z remote: Counting objects: 94% (1020/1085) 2024-05-19T18:08:23.6841459Z remote: Counting objects: 95% (1031/1085) 2024-05-19T18:08:23.6841983Z remote: Counting objects: 96% (1042/1085) 2024-05-19T18:08:23.6842491Z remote: Counting objects: 97% (1053/1085) 2024-05-19T18:08:23.6843193Z remote: Counting objects: 98% (1064/1085) 2024-05-19T18:08:23.6843767Z remote: Counting objects: 99% (1075/1085) 2024-05-19T18:08:23.6844314Z remote: Counting objects: 100% (1085/1085) 2024-05-19T18:08:23.6845047Z remote: Counting objects: 100% (1085/1085), done. 2024-05-19T18:08:23.6845639Z remote: Compressing objects: 0% (1/372) 2024-05-19T18:08:23.6933590Z remote: Compressing objects: 1% (4/372) 2024-05-19T18:08:23.7071502Z remote: Compressing objects: 2% (8/372) 2024-05-19T18:08:23.7252333Z remote: Compressing objects: 3% (12/372) 2024-05-19T18:08:23.7637423Z remote: Compressing objects: 4% (15/372) 2024-05-19T18:08:23.7835789Z remote: Compressing objects: 5% (19/372) 2024-05-19T18:08:23.7837094Z remote: Compressing objects: 6% (23/372) 2024-05-19T18:08:23.7837794Z remote: Compressing objects: 7% (27/372) 2024-05-19T18:08:23.7838387Z remote: Compressing objects: 8% (30/372) 2024-05-19T18:08:23.7838943Z remote: Compressing objects: 9% (34/372) 2024-05-19T18:08:23.7839494Z remote: Compressing objects: 10% (38/372) 2024-05-19T18:08:23.7840060Z remote: Compressing objects: 11% (41/372) 2024-05-19T18:08:23.7896434Z remote: Compressing objects: 12% (45/372) 2024-05-19T18:08:23.7896985Z remote: Compressing objects: 13% (49/372) 2024-05-19T18:08:23.7897943Z remote: Compressing objects: 14% (53/372) 2024-05-19T18:08:23.7898493Z remote: Compressing objects: 15% (56/372) 2024-05-19T18:08:23.7899076Z remote: Compressing objects: 16% (60/372) 2024-05-19T18:08:23.7899616Z remote: Compressing objects: 17% (64/372) 2024-05-19T18:08:23.7900170Z remote: Compressing objects: 18% (67/372) 2024-05-19T18:08:23.7909104Z remote: Compressing objects: 19% (71/372) 2024-05-19T18:08:23.7912660Z remote: Compressing objects: 20% (75/372) 2024-05-19T18:08:23.7915141Z remote: Compressing objects: 21% (79/372) 2024-05-19T18:08:23.7915780Z remote: Compressing objects: 22% (82/372) 2024-05-19T18:08:23.7922325Z remote: Compressing objects: 23% (86/372) 2024-05-19T18:08:23.7956976Z remote: Compressing objects: 24% (90/372) 2024-05-19T18:08:23.7957710Z remote: Compressing objects: 25% (93/372) 2024-05-19T18:08:23.7958377Z remote: Compressing objects: 26% (97/372) 2024-05-19T18:08:23.7958902Z remote: Compressing objects: 27% (101/372) 2024-05-19T18:08:23.7959424Z remote: Compressing objects: 28% (105/372) 2024-05-19T18:08:23.7960395Z remote: Compressing objects: 29% (108/372) 2024-05-19T18:08:23.7961003Z remote: Compressing objects: 30% (112/372) 2024-05-19T18:08:23.7961526Z remote: Compressing objects: 31% (116/372) 2024-05-19T18:08:23.7962043Z remote: Compressing objects: 32% (120/372) 2024-05-19T18:08:23.7962576Z remote: Compressing objects: 33% (123/372) 2024-05-19T18:08:23.7964420Z remote: Compressing objects: 34% (127/372) 2024-05-19T18:08:23.7965078Z remote: Compressing objects: 35% (131/372) 2024-05-19T18:08:23.7965592Z remote: Compressing objects: 36% (134/372) 2024-05-19T18:08:23.7969625Z remote: Compressing objects: 37% (138/372) 2024-05-19T18:08:23.7970655Z remote: Compressing objects: 38% (142/372) 2024-05-19T18:08:23.7972239Z remote: Compressing objects: 39% (146/372) 2024-05-19T18:08:23.7973894Z remote: Compressing objects: 40% (149/372) 2024-05-19T18:08:23.7978220Z remote: Compressing objects: 41% (153/372) 2024-05-19T18:08:23.7980186Z remote: Compressing objects: 42% (157/372) 2024-05-19T18:08:23.7982822Z remote: Compressing objects: 43% (160/372) 2024-05-19T18:08:23.7983439Z remote: Compressing objects: 44% (164/372) 2024-05-19T18:08:23.7984637Z remote: Compressing objects: 45% (168/372) 2024-05-19T18:08:23.7985359Z remote: Compressing objects: 46% (172/372) 2024-05-19T18:08:23.7986183Z remote: Compressing objects: 47% (175/372) 2024-05-19T18:08:23.7986805Z remote: Compressing objects: 48% (179/372) 2024-05-19T18:08:23.7987439Z remote: Compressing objects: 49% (183/372) 2024-05-19T18:08:23.7988324Z remote: Compressing objects: 50% (186/372) 2024-05-19T18:08:23.7989005Z remote: Compressing objects: 51% (190/372) 2024-05-19T18:08:23.7989629Z remote: Compressing objects: 52% (194/372) 2024-05-19T18:08:23.7990205Z remote: Compressing objects: 53% (198/372) 2024-05-19T18:08:23.7990914Z remote: Compressing objects: 54% (201/372) 2024-05-19T18:08:23.7991519Z remote: Compressing objects: 55% (205/372) 2024-05-19T18:08:23.7992180Z remote: Compressing objects: 56% (209/372) 2024-05-19T18:08:23.7992823Z remote: Compressing objects: 57% (213/372) 2024-05-19T18:08:23.7993456Z remote: Compressing objects: 58% (216/372) 2024-05-19T18:08:23.7995173Z remote: Compressing objects: 59% (220/372) 2024-05-19T18:08:23.7998498Z remote: Compressing objects: 60% (224/372) 2024-05-19T18:08:23.8000493Z remote: Compressing objects: 61% (227/372) 2024-05-19T18:08:23.8001538Z remote: Compressing objects: 62% (231/372) 2024-05-19T18:08:23.8002459Z remote: Compressing objects: 63% (235/372) 2024-05-19T18:08:23.8004459Z remote: Compressing objects: 64% (239/372) 2024-05-19T18:08:23.8005518Z remote: Compressing objects: 65% (242/372) 2024-05-19T18:08:23.8006194Z remote: Compressing objects: 66% (246/372) 2024-05-19T18:08:23.8008424Z remote: Compressing objects: 67% (250/372) 2024-05-19T18:08:23.8009411Z remote: Compressing objects: 68% (253/372) 2024-05-19T18:08:23.8011571Z remote: Compressing objects: 69% (257/372) 2024-05-19T18:08:23.8012295Z remote: Compressing objects: 70% (261/372) 2024-05-19T18:08:23.8012990Z remote: Compressing objects: 71% (265/372) 2024-05-19T18:08:23.8013691Z remote: Compressing objects: 72% (268/372) 2024-05-19T18:08:23.8014391Z remote: Compressing objects: 73% (272/372) 2024-05-19T18:08:23.8014997Z remote: Compressing objects: 74% (276/372) 2024-05-19T18:08:23.8015521Z remote: Compressing objects: 75% (279/372) 2024-05-19T18:08:23.8016040Z remote: Compressing objects: 76% (283/372) 2024-05-19T18:08:23.8016559Z remote: Compressing objects: 77% (287/372) 2024-05-19T18:08:23.8017070Z remote: Compressing objects: 78% (291/372) 2024-05-19T18:08:23.8018153Z remote: Compressing objects: 79% (294/372) 2024-05-19T18:08:23.8018700Z remote: Compressing objects: 80% (298/372) 2024-05-19T18:08:23.8019362Z remote: Compressing objects: 81% (302/372) 2024-05-19T18:08:23.8020025Z remote: Compressing objects: 82% (306/372) 2024-05-19T18:08:23.8020618Z remote: Compressing objects: 83% (309/372) 2024-05-19T18:08:23.8021248Z remote: Compressing objects: 84% (313/372) 2024-05-19T18:08:23.8021774Z remote: Compressing objects: 85% (317/372) 2024-05-19T18:08:23.8022297Z remote: Compressing objects: 86% (320/372) 2024-05-19T18:08:23.8022992Z remote: Compressing objects: 87% (324/372) 2024-05-19T18:08:23.8023675Z remote: Compressing objects: 88% (328/372) 2024-05-19T18:08:23.8024369Z remote: Compressing objects: 89% (332/372) 2024-05-19T18:08:23.8025037Z remote: Compressing objects: 90% (335/372) 2024-05-19T18:08:23.8025556Z remote: Compressing objects: 91% (339/372) 2024-05-19T18:08:23.8026063Z remote: Compressing objects: 92% (343/372) 2024-05-19T18:08:23.8026574Z remote: Compressing objects: 93% (346/372) 2024-05-19T18:08:23.8027087Z remote: Compressing objects: 94% (350/372) 2024-05-19T18:08:23.8027598Z remote: Compressing objects: 95% (354/372) 2024-05-19T18:08:23.8028110Z remote: Compressing objects: 96% (358/372) 2024-05-19T18:08:23.8028618Z remote: Compressing objects: 97% (361/372) 2024-05-19T18:08:23.8029254Z remote: Compressing objects: 98% (365/372) 2024-05-19T18:08:23.8029770Z remote: Compressing objects: 99% (369/372) 2024-05-19T18:08:23.8030274Z remote: Compressing objects: 100% (372/372) 2024-05-19T18:08:23.8030824Z remote: Compressing objects: 100% (372/372), done. 2024-05-19T18:08:44.6066692Z remote: Total 1119785 (delta 765), reused 958 (delta 703), pack-reused 1118700 2024-05-19T18:09:07.9911386Z [command]/usr/bin/git rev-parse --verify --quiet c8c7ddfde57442cc28a394c385cab656ce9b1f5e^{object} 2024-05-19T18:09:07.9929899Z c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:09:07.9933261Z ##[endgroup] 2024-05-19T18:09:07.9933812Z ##[group]Determining the checkout info 2024-05-19T18:09:07.9934609Z ##[endgroup] 2024-05-19T18:09:07.9936635Z ##[group]Checking out the ref 2024-05-19T18:09:07.9939421Z [command]/usr/bin/git checkout --quiet --force c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:09:09.2063162Z ##[endgroup] 2024-05-19T18:09:09.2063783Z ##[group]Setting up auth for fetching submodules 2024-05-19T18:09:09.2068031Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-05-19T18:09:09.2109010Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-05-19T18:09:09.2128451Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-05-19T18:09:09.2147278Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-05-19T18:09:09.2162477Z ##[endgroup] 2024-05-19T18:09:09.2163001Z ##[group]Fetching submodules 2024-05-19T18:09:09.2166613Z [command]/usr/bin/git submodule sync --recursive 2024-05-19T18:09:09.2361306Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-05-19T18:09:09.2558194Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-05-19T18:09:09.2559769Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-05-19T18:09:09.2561157Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-05-19T18:09:09.2562632Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-05-19T18:09:09.2564448Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2024-05-19T18:09:09.2567141Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-05-19T18:09:09.2568864Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-05-19T18:09:09.2571090Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-05-19T18:09:09.2573337Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-05-19T18:09:09.2575362Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-05-19T18:09:09.2578218Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-05-19T18:09:09.2581027Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-05-19T18:09:09.2583367Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-05-19T18:09:09.2585904Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-05-19T18:09:09.2588603Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-05-19T18:09:09.2591202Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-05-19T18:09:09.2593967Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-05-19T18:09:09.2596925Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:09:09.2599658Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-05-19T18:09:09.2602665Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-05-19T18:09:09.2605750Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-05-19T18:09:09.2609044Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2024-05-19T18:09:09.2612124Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-05-19T18:09:09.2615394Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-05-19T18:09:09.2619005Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-05-19T18:09:09.2622672Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-05-19T18:09:09.2626124Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2024-05-19T18:09:09.2629568Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-05-19T18:09:09.2633102Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-05-19T18:09:09.2638398Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2024-05-19T18:09:09.2642193Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-05-19T18:09:09.2645824Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-05-19T18:09:09.2649910Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-05-19T18:09:09.2653674Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-05-19T18:09:09.2657968Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-05-19T18:09:09.2662112Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-05-19T18:09:09.2666293Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-05-19T18:09:09.2670326Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-05-19T18:09:09.2674623Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2024-05-19T18:09:09.2679138Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-05-19T18:09:09.2699224Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-05-19T18:09:09.4935358Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-05-19T18:09:09.6271245Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-05-19T18:09:09.7844082Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-05-19T18:09:09.9978890Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2024-05-19T18:09:10.1919403Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-05-19T18:10:03.9037182Z ##[error]error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8) 2024-05-19T18:10:03.9038332Z ##[error]error: 2983 bytes of body are still expected 2024-05-19T18:10:03.9039174Z fetch-pack: unexpected disconnect while reading sideband packet 2024-05-19T18:10:03.9039844Z ##[error]fatal: early EOF 2024-05-19T18:10:03.9040482Z ##[error]fatal: fetch-pack: invalid index-pack output 2024-05-19T18:10:03.9042944Z ##[error]fatal: clone of 'https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git' into submodule path '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator' failed 2024-05-19T18:10:03.9044440Z Failed to clone 'third_party/VulkanMemoryAllocator'. Retry scheduled 2024-05-19T18:10:03.9049972Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-05-19T18:10:48.0455527Z ##[error]error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8) 2024-05-19T18:10:48.0457405Z ##[error]error: 5257 bytes of body are still expected 2024-05-19T18:10:48.0467446Z fetch-pack: unexpected disconnect while reading sideband packet 2024-05-19T18:10:48.0468193Z ##[error]fatal: early EOF 2024-05-19T18:10:48.0469057Z ##[error]fatal: fetch-pack: invalid index-pack output 2024-05-19T18:10:48.0480465Z ##[error]fatal: clone of 'https://github.com/google/XNNPACK.git' into submodule path '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK' failed 2024-05-19T18:10:48.0481691Z Failed to clone 'third_party/XNNPACK'. Retry scheduled 2024-05-19T18:10:48.0499157Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-05-19T18:10:48.3999403Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-05-19T18:10:48.8745186Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-05-19T18:10:49.4350954Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-05-19T18:10:50.5640839Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-05-19T18:10:52.3524252Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-05-19T18:10:58.0915402Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-05-19T18:10:59.4037649Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-05-19T18:11:00.9750381Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-05-19T18:11:02.0747196Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-05-19T18:11:02.2345182Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-05-19T18:11:02.6112494Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-05-19T18:11:34.1362691Z ##[error]error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8) 2024-05-19T18:11:34.1363760Z ##[error]error: 5251 bytes of body are still expected 2024-05-19T18:11:34.1378365Z fetch-pack: unexpected disconnect while reading sideband packet 2024-05-19T18:11:34.1380717Z ##[error]fatal: early EOF 2024-05-19T18:11:34.1381500Z ##[error]fatal: fetch-pack: invalid index-pack output 2024-05-19T18:11:34.1390518Z ##[error]fatal: clone of 'https://github.com/facebookincubator/gloo' into submodule path '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo' failed 2024-05-19T18:11:34.1391895Z Failed to clone 'third_party/gloo'. Retry scheduled 2024-05-19T18:11:34.1410522Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-05-19T18:11:35.0157098Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-05-19T18:11:35.3483341Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2024-05-19T18:11:35.5428649Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-05-19T18:11:35.7509987Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-05-19T18:11:37.1360938Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-05-19T18:11:37.7836826Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-05-19T18:11:38.3648073Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2024-05-19T18:11:38.6082794Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-05-19T18:11:44.2891769Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-05-19T18:11:46.6648109Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2024-05-19T18:12:36.1610604Z ##[error]error: RPC failed; curl 18 Transferred a partial file 2024-05-19T18:12:36.1611406Z ##[error]error: 357 bytes of body are still expected 2024-05-19T18:12:36.1617955Z fetch-pack: unexpected disconnect while reading sideband packet 2024-05-19T18:12:36.1618738Z ##[error]fatal: early EOF 2024-05-19T18:12:36.1619747Z ##[error]fatal: fetch-pack: invalid index-pack output 2024-05-19T18:12:36.1629398Z ##[error]fatal: clone of 'https://github.com/onnx/onnx-tensorrt' into submodule path '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt' failed 2024-05-19T18:12:36.1630773Z Failed to clone 'third_party/onnx-tensorrt'. Retry scheduled 2024-05-19T18:12:36.1650125Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-05-19T18:12:39.8557153Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-05-19T18:12:40.0202716Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-05-19T18:12:47.5052751Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-05-19T18:12:47.6408661Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-05-19T18:12:47.8011828Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-05-19T18:12:48.6762645Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-05-19T18:12:48.9275126Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-05-19T18:12:49.4700557Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2024-05-19T18:12:51.5272309Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-05-19T18:12:51.8928728Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-05-19T18:12:53.8921247Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-05-19T18:13:02.6282724Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-05-19T18:13:02.9164264Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2024-05-19T18:13:03.3425991Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-05-19T18:13:03.3505181Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-05-19T18:13:03.3562873Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-05-19T18:13:03.3748747Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-05-19T18:13:03.3929620Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2024-05-19T18:13:03.4242781Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-05-19T18:13:04.2024597Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-05-19T18:13:04.2195525Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-05-19T18:13:04.2544524Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-05-19T18:13:04.3363446Z Submodule path 'third_party/cpuinfo': checked out 'd6860c477c99f1fce9e28eb206891af3c0e1a1d7' 2024-05-19T18:13:04.3597877Z Submodule path 'third_party/cudnn_frontend': checked out '150798fe976556078f443fdb059a1ff0361f58a2' 2024-05-19T18:13:04.7845920Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-05-19T18:13:04.9955136Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-05-19T18:13:05.0536709Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-05-19T18:13:05.0547084Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:13:05.0548785Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:13:05.0550594Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:13:05.0552675Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:13:05.0554846Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:13:05.0573164Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-05-19T18:13:05.9722052Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-05-19T18:13:06.5304846Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-05-19T18:13:08.1275010Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-05-19T18:13:09.0637046Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-05-19T18:13:09.3563546Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-05-19T18:13:09.4398607Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-05-19T18:13:09.7783948Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-05-19T18:13:09.8315087Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-05-19T18:13:09.8401222Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-05-19T18:13:09.9325072Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-05-19T18:13:09.9632017Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-05-19T18:13:09.9695491Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-05-19T18:13:09.9997061Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-05-19T18:13:10.0192895Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-05-19T18:13:10.0546671Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-05-19T18:13:10.0645300Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-05-19T18:13:10.0654932Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-05-19T18:13:10.0671365Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-05-19T18:13:20.5613984Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-05-19T18:13:20.5691690Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2024-05-19T18:13:20.5818691Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-05-19T18:13:20.6591988Z Submodule path 'third_party/kineto': checked out '3a81076cc97092666f319846f32f36b73ce2293e' 2024-05-19T18:13:20.6603133Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:13:20.6605251Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:13:20.6607192Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:13:20.6622628Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-05-19T18:13:21.1372322Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-05-19T18:13:22.1811261Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-05-19T18:13:23.1769834Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-05-19T18:13:23.1779679Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:13:23.1781528Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:13:23.1783442Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:13:23.1785246Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:13:23.1787065Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:13:23.1789111Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:13:23.1791056Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:13:23.1793568Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:13:23.1811660Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-05-19T18:13:24.3169854Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-05-19T18:13:24.6402680Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-05-19T18:13:25.6905335Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-05-19T18:13:25.9718547Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-05-19T18:13:26.3303875Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-05-19T18:13:27.2403553Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-05-19T18:13:32.7293774Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-05-19T18:13:33.0628847Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-05-19T18:13:33.0762128Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-05-19T18:13:33.1043660Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-05-19T18:13:33.1143394Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-05-19T18:13:33.1154762Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:13:33.1170317Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-05-19T18:13:33.3755365Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-05-19T18:13:33.3891409Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-05-19T18:13:33.4220098Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-05-19T18:13:33.5021335Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-05-19T18:13:33.5139796Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-05-19T18:13:33.5424813Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-05-19T18:13:33.5918232Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-05-19T18:13:33.6209181Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-05-19T18:13:33.6404204Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-05-19T18:13:33.6505568Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2024-05-19T18:13:33.7355640Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-05-19T18:13:34.0096350Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-05-19T18:13:34.0118543Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-05-19T18:13:34.0120403Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-05-19T18:13:34.0136715Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-05-19T18:13:34.3736205Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-05-19T18:13:35.3330069Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-05-19T18:13:35.3606871Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-05-19T18:13:35.3747536Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2024-05-19T18:13:35.3758480Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:13:35.3773796Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2024-05-19T18:13:38.2051103Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2024-05-19T18:13:38.2065706Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:13:38.2067720Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:13:38.2086933Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2024-05-19T18:13:38.6221861Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2024-05-19T18:13:39.5747621Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2024-05-19T18:13:39.6390851Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2024-05-19T18:13:39.6401934Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:13:39.6417957Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2024-05-19T18:13:39.8669041Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-05-19T18:13:39.9195069Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-05-19T18:13:39.9206699Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:13:39.9208357Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:13:39.9210173Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:13:39.9211894Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:13:39.9213959Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:13:39.9216110Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:13:39.9218323Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:13:39.9220209Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:13:39.9238426Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-05-19T18:13:40.2600731Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-05-19T18:13:41.1816018Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-05-19T18:13:41.4557185Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-05-19T18:13:47.0265791Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-05-19T18:13:47.2868889Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-05-19T18:13:47.4611934Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-05-19T18:13:47.7418440Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-05-19T18:13:53.9143488Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-05-19T18:13:53.9472772Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-05-19T18:13:53.9595186Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-05-19T18:13:54.0439837Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-05-19T18:13:54.0532122Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-05-19T18:13:54.0637318Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-05-19T18:13:54.0747410Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-05-19T18:13:54.0756846Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:13:54.0758872Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:13:54.0776391Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-05-19T18:13:55.8961854Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-05-19T18:13:56.9929800Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-05-19T18:13:57.0294819Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-05-19T18:13:57.3917025Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-05-19T18:13:57.3993614Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-05-19T18:13:57.6153671Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-05-19T18:13:57.6168010Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:13:57.6169903Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-05-19T18:13:57.6185270Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-05-19T18:13:57.9695808Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-05-19T18:13:58.9113031Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-05-19T18:13:58.9733999Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-05-19T18:13:58.9791960Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-05-19T18:13:58.9877463Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-05-19T18:13:59.0150497Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-05-19T18:13:59.0375010Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-05-19T18:13:59.0712817Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-05-19T18:13:59.1620840Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2024-05-19T18:13:59.1821111Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-05-19T18:13:59.1831106Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:13:59.1833028Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:13:59.1834667Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:13:59.1836315Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:13:59.1853721Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-05-19T18:14:00.1075154Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-05-19T18:14:00.3252574Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-05-19T18:14:01.5088679Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-05-19T18:14:02.4578157Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-05-19T18:14:02.4689702Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-05-19T18:14:02.5262763Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-05-19T18:14:02.5487527Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-05-19T18:14:02.5498346Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:14:02.5514396Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-05-19T18:14:02.7599902Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-05-19T18:14:02.7894882Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-05-19T18:14:03.2665545Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-05-19T18:14:03.2837402Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-05-19T18:14:03.2923908Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2024-05-19T18:14:03.3792375Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2024-05-19T18:14:03.3896850Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2024-05-19T18:14:03.4086451Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2024-05-19T18:14:03.4143587Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-05-19T18:14:03.4171298Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-05-19T18:14:03.4368343Z Entering 'android/libs/fbjni' 2024-05-19T18:14:03.4395460Z Entering 'third_party/FP16' 2024-05-19T18:14:03.4421032Z Entering 'third_party/FXdiv' 2024-05-19T18:14:03.4448675Z Entering 'third_party/NNPACK' 2024-05-19T18:14:03.4473699Z Entering 'third_party/QNNPACK' 2024-05-19T18:14:03.4502278Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T18:14:03.4530693Z Entering 'third_party/XNNPACK' 2024-05-19T18:14:03.4569518Z Entering 'third_party/benchmark' 2024-05-19T18:14:03.4597182Z Entering 'third_party/cpp-httplib' 2024-05-19T18:14:03.4624743Z Entering 'third_party/cpuinfo' 2024-05-19T18:14:03.4652956Z Entering 'third_party/cudnn_frontend' 2024-05-19T18:14:03.4679474Z Entering 'third_party/cutlass' 2024-05-19T18:14:03.4712154Z Entering 'third_party/eigen' 2024-05-19T18:14:03.4741326Z Entering 'third_party/fbgemm' 2024-05-19T18:14:03.4768698Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:14:03.4794808Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:14:03.4821141Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:14:03.4853851Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:14:03.4881365Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:14:03.4908741Z Entering 'third_party/flatbuffers' 2024-05-19T18:14:03.4936286Z Entering 'third_party/fmt' 2024-05-19T18:14:03.4965186Z Entering 'third_party/foxi' 2024-05-19T18:14:03.4992235Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:14:03.5021813Z Entering 'third_party/gloo' 2024-05-19T18:14:03.5050268Z Entering 'third_party/googletest' 2024-05-19T18:14:03.5075644Z Entering 'third_party/ideep' 2024-05-19T18:14:03.5099480Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T18:14:03.5131874Z Entering 'third_party/ios-cmake' 2024-05-19T18:14:03.5161148Z Entering 'third_party/ittapi' 2024-05-19T18:14:03.5186954Z Entering 'third_party/kineto' 2024-05-19T18:14:03.5213988Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:14:03.5241677Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:14:03.5268288Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:14:03.5295153Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:14:03.5323656Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:14:03.5349723Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:14:03.5378903Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:14:03.5406298Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:14:03.5433839Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:14:03.5462390Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:14:03.5491247Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:14:03.5517995Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:14:03.5545647Z Entering 'third_party/mimalloc' 2024-05-19T18:14:03.5574982Z Entering 'third_party/nccl/nccl' 2024-05-19T18:14:03.5604179Z Entering 'third_party/neon2sse' 2024-05-19T18:14:03.5632457Z Entering 'third_party/nlohmann' 2024-05-19T18:14:03.5660547Z Entering 'third_party/onnx' 2024-05-19T18:14:03.5700778Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T18:14:03.5727932Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T18:14:03.5756919Z Entering 'third_party/onnx-tensorrt' 2024-05-19T18:14:03.5783125Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:14:03.5813303Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:14:03.5842901Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:14:03.5868460Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:14:03.5898454Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T18:14:03.5927477Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:14:03.5953490Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:14:03.5980575Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:14:03.6005675Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:14:03.6034428Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:14:03.6061282Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:14:03.6086437Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:14:03.6110831Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:14:03.6137332Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:14:03.6164030Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:14:03.6205049Z Entering 'third_party/pocketfft' 2024-05-19T18:14:03.6233689Z Entering 'third_party/protobuf' 2024-05-19T18:14:03.6263021Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:14:03.6287313Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T18:14:03.6314219Z Entering 'third_party/psimd' 2024-05-19T18:14:03.6342056Z Entering 'third_party/pthreadpool' 2024-05-19T18:14:03.6368431Z Entering 'third_party/pybind11' 2024-05-19T18:14:03.6394284Z Entering 'third_party/python-peachpy' 2024-05-19T18:14:03.6422748Z Entering 'third_party/sleef' 2024-05-19T18:14:03.6451177Z Entering 'third_party/tbb' 2024-05-19T18:14:03.6482165Z Entering 'third_party/tensorpipe' 2024-05-19T18:14:03.6507617Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:14:03.6535386Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:14:03.6561714Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:14:03.6589195Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:14:03.6614289Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:14:03.6653600Z ##[endgroup] 2024-05-19T18:14:03.6655287Z ##[group]Persisting credentials for submodules 2024-05-19T18:14:03.6660161Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-05-19T18:14:03.6866341Z Entering 'android/libs/fbjni' 2024-05-19T18:14:03.6901945Z Entering 'third_party/FP16' 2024-05-19T18:14:03.6935323Z Entering 'third_party/FXdiv' 2024-05-19T18:14:03.6971425Z Entering 'third_party/NNPACK' 2024-05-19T18:14:03.7005273Z Entering 'third_party/QNNPACK' 2024-05-19T18:14:03.7038354Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T18:14:03.7074176Z Entering 'third_party/XNNPACK' 2024-05-19T18:14:03.7118680Z Entering 'third_party/benchmark' 2024-05-19T18:14:03.7151185Z Entering 'third_party/cpp-httplib' 2024-05-19T18:14:03.7183860Z Entering 'third_party/cpuinfo' 2024-05-19T18:14:03.7217204Z Entering 'third_party/cudnn_frontend' 2024-05-19T18:14:03.7252039Z Entering 'third_party/cutlass' 2024-05-19T18:14:03.7290648Z Entering 'third_party/eigen' 2024-05-19T18:14:03.7327167Z Entering 'third_party/fbgemm' 2024-05-19T18:14:03.7361503Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:14:03.7395044Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:14:03.7427881Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:14:03.7465169Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:14:03.7499338Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:14:03.7531714Z Entering 'third_party/flatbuffers' 2024-05-19T18:14:03.7567848Z Entering 'third_party/fmt' 2024-05-19T18:14:03.7602760Z Entering 'third_party/foxi' 2024-05-19T18:14:03.7639616Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:14:03.7674672Z Entering 'third_party/gloo' 2024-05-19T18:14:03.7707408Z Entering 'third_party/googletest' 2024-05-19T18:14:03.7742277Z Entering 'third_party/ideep' 2024-05-19T18:14:03.7773607Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T18:14:03.7812060Z Entering 'third_party/ios-cmake' 2024-05-19T18:14:03.7844500Z Entering 'third_party/ittapi' 2024-05-19T18:14:03.7878944Z Entering 'third_party/kineto' 2024-05-19T18:14:03.7914616Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:14:03.7946063Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:14:03.7979218Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:14:03.8013478Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:14:03.8046538Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:14:03.8082411Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:14:03.8118285Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:14:03.8153551Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:14:03.8187138Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:14:03.8221755Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:14:03.8255747Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:14:03.8287733Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:14:03.8321855Z Entering 'third_party/mimalloc' 2024-05-19T18:14:03.8357588Z Entering 'third_party/nccl/nccl' 2024-05-19T18:14:03.8393711Z Entering 'third_party/neon2sse' 2024-05-19T18:14:03.8425562Z Entering 'third_party/nlohmann' 2024-05-19T18:14:03.8460988Z Entering 'third_party/onnx' 2024-05-19T18:14:03.8507697Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T18:14:03.8543351Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T18:14:03.8580578Z Entering 'third_party/onnx-tensorrt' 2024-05-19T18:14:03.8614889Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:14:03.8690307Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:14:03.8691875Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:14:03.8721910Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:14:03.8759207Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T18:14:03.8793340Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:14:03.8826123Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:14:03.8859679Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:14:03.8892913Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:14:03.8926915Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:14:03.8960083Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:14:03.8993021Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:14:03.9025215Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:14:03.9060109Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:14:03.9094581Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:14:03.9144455Z Entering 'third_party/pocketfft' 2024-05-19T18:14:03.9176847Z Entering 'third_party/protobuf' 2024-05-19T18:14:03.9213075Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:14:03.9246123Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T18:14:03.9282824Z Entering 'third_party/psimd' 2024-05-19T18:14:03.9316456Z Entering 'third_party/pthreadpool' 2024-05-19T18:14:03.9351455Z Entering 'third_party/pybind11' 2024-05-19T18:14:03.9385817Z Entering 'third_party/python-peachpy' 2024-05-19T18:14:03.9415707Z Entering 'third_party/sleef' 2024-05-19T18:14:03.9451452Z Entering 'third_party/tbb' 2024-05-19T18:14:03.9487565Z Entering 'third_party/tensorpipe' 2024-05-19T18:14:03.9522532Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:14:03.9556922Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:14:03.9587620Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:14:03.9620792Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:14:03.9653140Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:14:03.9699903Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-05-19T18:14:03.9902382Z Entering 'android/libs/fbjni' 2024-05-19T18:14:03.9932588Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-05-19T18:14:03.9946557Z Entering 'third_party/FP16' 2024-05-19T18:14:03.9977284Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-05-19T18:14:03.9989463Z Entering 'third_party/FXdiv' 2024-05-19T18:14:04.0021284Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-05-19T18:14:04.0033558Z Entering 'third_party/NNPACK' 2024-05-19T18:14:04.0063242Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-05-19T18:14:04.0075966Z Entering 'third_party/QNNPACK' 2024-05-19T18:14:04.0106501Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2024-05-19T18:14:04.0120360Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T18:14:04.0151062Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-05-19T18:14:04.0164964Z Entering 'third_party/XNNPACK' 2024-05-19T18:14:04.0195582Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-05-19T18:14:04.0220323Z Entering 'third_party/benchmark' 2024-05-19T18:14:04.0251024Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-05-19T18:14:04.0264096Z Entering 'third_party/cpp-httplib' 2024-05-19T18:14:04.0296484Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-05-19T18:14:04.0310597Z Entering 'third_party/cpuinfo' 2024-05-19T18:14:04.0341801Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-05-19T18:14:04.0355474Z Entering 'third_party/cudnn_frontend' 2024-05-19T18:14:04.0387116Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-05-19T18:14:04.0401557Z Entering 'third_party/cutlass' 2024-05-19T18:14:04.0432312Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-05-19T18:14:04.0450788Z Entering 'third_party/eigen' 2024-05-19T18:14:04.0480835Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-05-19T18:14:04.0495934Z Entering 'third_party/fbgemm' 2024-05-19T18:14:04.0528292Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-05-19T18:14:04.0542908Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:14:04.0572399Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-05-19T18:14:04.0585413Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:14:04.0615571Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-05-19T18:14:04.0627681Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:14:04.0658813Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-05-19T18:14:04.0674944Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:14:04.0704994Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:14:04.0717965Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:14:04.0749934Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-05-19T18:14:04.0762653Z Entering 'third_party/flatbuffers' 2024-05-19T18:14:04.0793258Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-05-19T18:14:04.0808472Z Entering 'third_party/fmt' 2024-05-19T18:14:04.0840368Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-05-19T18:14:04.0852349Z Entering 'third_party/foxi' 2024-05-19T18:14:04.0882794Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-05-19T18:14:04.0894040Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:14:04.0925397Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-05-19T18:14:04.0939787Z Entering 'third_party/gloo' 2024-05-19T18:14:04.0969725Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-05-19T18:14:04.0981950Z Entering 'third_party/googletest' 2024-05-19T18:14:04.1012390Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:14:04.1025757Z Entering 'third_party/ideep' 2024-05-19T18:14:04.1056424Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-05-19T18:14:04.1067569Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T18:14:04.1097180Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-05-19T18:14:04.1114144Z Entering 'third_party/ios-cmake' 2024-05-19T18:14:04.1146281Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2024-05-19T18:14:04.1159355Z Entering 'third_party/ittapi' 2024-05-19T18:14:04.1190534Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-05-19T18:14:04.1202731Z Entering 'third_party/kineto' 2024-05-19T18:14:04.1234534Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-05-19T18:14:04.1245682Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:14:04.1276526Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-05-19T18:14:04.1288826Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:14:04.1319822Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-05-19T18:14:04.1334115Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:14:04.1365566Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-05-19T18:14:04.1378322Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:14:04.1411409Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-05-19T18:14:04.1424691Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:14:04.1455630Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-05-19T18:14:04.1469586Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:14:04.1501336Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-05-19T18:14:04.1514392Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:14:04.1546099Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-05-19T18:14:04.1558580Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:14:04.1590251Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:14:04.1603101Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:14:04.1635504Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-05-19T18:14:04.1648403Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:14:04.1678955Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-05-19T18:14:04.1693515Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:14:04.1726246Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-05-19T18:14:04.1738373Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:14:04.1768546Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-05-19T18:14:04.1782825Z Entering 'third_party/mimalloc' 2024-05-19T18:14:04.1814354Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-05-19T18:14:04.1826262Z Entering 'third_party/nccl/nccl' 2024-05-19T18:14:04.1856917Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-05-19T18:14:04.1870795Z Entering 'third_party/neon2sse' 2024-05-19T18:14:04.1899516Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2024-05-19T18:14:04.1911663Z Entering 'third_party/nlohmann' 2024-05-19T18:14:04.1941697Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-05-19T18:14:04.1954395Z Entering 'third_party/onnx' 2024-05-19T18:14:04.1985119Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-05-19T18:14:04.2009573Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T18:14:04.2040643Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-05-19T18:14:04.2052876Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T18:14:04.2084035Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-05-19T18:14:04.2099616Z Entering 'third_party/onnx-tensorrt' 2024-05-19T18:14:04.2130886Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2024-05-19T18:14:04.2142929Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:14:04.2173305Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2024-05-19T18:14:04.2188472Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:14:04.2219193Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-05-19T18:14:04.2231435Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:14:04.2262244Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-05-19T18:14:04.2274212Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:14:04.2305124Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-05-19T18:14:04.2321701Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T18:14:04.2352654Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-05-19T18:14:04.2367227Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:14:04.2398528Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-05-19T18:14:04.2410006Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:14:04.2441433Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:14:04.2453898Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:14:04.2487934Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-05-19T18:14:04.2501164Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:14:04.2532133Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-05-19T18:14:04.2547079Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:14:04.2578020Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-05-19T18:14:04.2589940Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:14:04.2620906Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-05-19T18:14:04.2633716Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:14:04.2664252Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-05-19T18:14:04.2675849Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:14:04.2707005Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-05-19T18:14:04.2721003Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:14:04.2751359Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-05-19T18:14:04.2765126Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:14:04.2794642Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-05-19T18:14:04.2822347Z Entering 'third_party/pocketfft' 2024-05-19T18:14:04.2853240Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-05-19T18:14:04.2865026Z Entering 'third_party/protobuf' 2024-05-19T18:14:04.2894884Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-05-19T18:14:04.2908936Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:14:04.2939128Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-05-19T18:14:04.2952067Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T18:14:04.2981970Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:14:04.2994459Z Entering 'third_party/psimd' 2024-05-19T18:14:04.3024543Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-05-19T18:14:04.3037982Z Entering 'third_party/pthreadpool' 2024-05-19T18:14:04.3067611Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-05-19T18:14:04.3078941Z Entering 'third_party/pybind11' 2024-05-19T18:14:04.3110610Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-05-19T18:14:04.3123562Z Entering 'third_party/python-peachpy' 2024-05-19T18:14:04.3155159Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-05-19T18:14:04.3169637Z Entering 'third_party/sleef' 2024-05-19T18:14:04.3201660Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-05-19T18:14:04.3213548Z Entering 'third_party/tbb' 2024-05-19T18:14:04.3243632Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2024-05-19T18:14:04.3257143Z Entering 'third_party/tensorpipe' 2024-05-19T18:14:04.3288755Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-05-19T18:14:04.3300951Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:14:04.3331867Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:14:04.3343549Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:14:04.3374324Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-05-19T18:14:04.3387487Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:14:04.3418046Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-05-19T18:14:04.3430225Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:14:04.3460585Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-05-19T18:14:04.3471520Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:14:04.3502879Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-05-19T18:14:04.4523821Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-05-19T18:14:04.4726220Z Entering 'android/libs/fbjni' 2024-05-19T18:14:04.4786196Z Entering 'third_party/FP16' 2024-05-19T18:14:04.4814150Z Entering 'third_party/FXdiv' 2024-05-19T18:14:04.4841180Z Entering 'third_party/NNPACK' 2024-05-19T18:14:04.4868961Z Entering 'third_party/QNNPACK' 2024-05-19T18:14:04.4896581Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T18:14:04.4925552Z Entering 'third_party/XNNPACK' 2024-05-19T18:14:04.4963785Z Entering 'third_party/benchmark' 2024-05-19T18:14:04.4991297Z Entering 'third_party/cpp-httplib' 2024-05-19T18:14:04.5017107Z Entering 'third_party/cpuinfo' 2024-05-19T18:14:04.5045136Z Entering 'third_party/cudnn_frontend' 2024-05-19T18:14:04.5074078Z Entering 'third_party/cutlass' 2024-05-19T18:14:04.5108405Z Entering 'third_party/eigen' 2024-05-19T18:14:04.5136537Z Entering 'third_party/fbgemm' 2024-05-19T18:14:04.5162158Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:14:04.5187218Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:14:04.5213212Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:14:04.5244530Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:14:04.5271376Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:14:04.5300214Z Entering 'third_party/flatbuffers' 2024-05-19T18:14:04.5330570Z Entering 'third_party/fmt' 2024-05-19T18:14:04.5357523Z Entering 'third_party/foxi' 2024-05-19T18:14:04.5384878Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:14:04.5412115Z Entering 'third_party/gloo' 2024-05-19T18:14:04.5439825Z Entering 'third_party/googletest' 2024-05-19T18:14:04.5467673Z Entering 'third_party/ideep' 2024-05-19T18:14:04.5493938Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T18:14:04.5526545Z Entering 'third_party/ios-cmake' 2024-05-19T18:14:04.5552894Z Entering 'third_party/ittapi' 2024-05-19T18:14:04.5581618Z Entering 'third_party/kineto' 2024-05-19T18:14:04.5608763Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:14:04.5637072Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:14:04.5665372Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:14:04.5691702Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:14:04.5719747Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:14:04.5744278Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:14:04.5773378Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:14:04.5799255Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:14:04.5825866Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:14:04.5852464Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:14:04.5879059Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:14:04.5906484Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:14:04.5935458Z Entering 'third_party/mimalloc' 2024-05-19T18:14:04.5965299Z Entering 'third_party/nccl/nccl' 2024-05-19T18:14:04.5992004Z Entering 'third_party/neon2sse' 2024-05-19T18:14:04.6020774Z Entering 'third_party/nlohmann' 2024-05-19T18:14:04.6049181Z Entering 'third_party/onnx' 2024-05-19T18:14:04.6090162Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T18:14:04.6119668Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T18:14:04.6147091Z Entering 'third_party/onnx-tensorrt' 2024-05-19T18:14:04.6175047Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:14:04.6206730Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:14:04.6233411Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:14:04.6259752Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:14:04.6291426Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T18:14:04.6321396Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:14:04.6348412Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:14:04.6374271Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:14:04.6399212Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:14:04.6427663Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:14:04.6453209Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:14:04.6480299Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:14:04.6507114Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:14:04.6534257Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:14:04.6562869Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:14:04.6603621Z Entering 'third_party/pocketfft' 2024-05-19T18:14:04.6633551Z Entering 'third_party/protobuf' 2024-05-19T18:14:04.6660995Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:14:04.6685829Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T18:14:04.6717206Z Entering 'third_party/psimd' 2024-05-19T18:14:04.6745589Z Entering 'third_party/pthreadpool' 2024-05-19T18:14:04.6775824Z Entering 'third_party/pybind11' 2024-05-19T18:14:04.6802653Z Entering 'third_party/python-peachpy' 2024-05-19T18:14:04.6827722Z Entering 'third_party/sleef' 2024-05-19T18:14:04.6855354Z Entering 'third_party/tbb' 2024-05-19T18:14:04.6884852Z Entering 'third_party/tensorpipe' 2024-05-19T18:14:04.6913437Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:14:04.6937506Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:14:04.6965353Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:14:04.6993421Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:14:04.7018174Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:14:04.7060175Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-05-19T18:14:04.7266527Z Entering 'android/libs/fbjni' 2024-05-19T18:14:04.7295332Z Entering 'third_party/FP16' 2024-05-19T18:14:04.7323568Z Entering 'third_party/FXdiv' 2024-05-19T18:14:04.7351035Z Entering 'third_party/NNPACK' 2024-05-19T18:14:04.7380090Z Entering 'third_party/QNNPACK' 2024-05-19T18:14:04.7405814Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T18:14:04.7434822Z Entering 'third_party/XNNPACK' 2024-05-19T18:14:04.7472505Z Entering 'third_party/benchmark' 2024-05-19T18:14:04.7499247Z Entering 'third_party/cpp-httplib' 2024-05-19T18:14:04.7527271Z Entering 'third_party/cpuinfo' 2024-05-19T18:14:04.7552613Z Entering 'third_party/cudnn_frontend' 2024-05-19T18:14:04.7580213Z Entering 'third_party/cutlass' 2024-05-19T18:14:04.7613210Z Entering 'third_party/eigen' 2024-05-19T18:14:04.7641952Z Entering 'third_party/fbgemm' 2024-05-19T18:14:04.7670412Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:14:04.7697130Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:14:04.7724294Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:14:04.7756153Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:14:04.7785790Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:14:04.7814355Z Entering 'third_party/flatbuffers' 2024-05-19T18:14:04.7845246Z Entering 'third_party/fmt' 2024-05-19T18:14:04.7871821Z Entering 'third_party/foxi' 2024-05-19T18:14:04.7900640Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:14:04.7927651Z Entering 'third_party/gloo' 2024-05-19T18:14:04.7954155Z Entering 'third_party/googletest' 2024-05-19T18:14:04.7983334Z Entering 'third_party/ideep' 2024-05-19T18:14:04.8008469Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T18:14:04.8039570Z Entering 'third_party/ios-cmake' 2024-05-19T18:14:04.8068668Z Entering 'third_party/ittapi' 2024-05-19T18:14:04.8094904Z Entering 'third_party/kineto' 2024-05-19T18:14:04.8124000Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:14:04.8150278Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:14:04.8176695Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:14:04.8204320Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:14:04.8230094Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:14:04.8259287Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:14:04.8285954Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:14:04.8311257Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:14:04.8337164Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:14:04.8363453Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:14:04.8389511Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:14:04.8414324Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:14:04.8440411Z Entering 'third_party/mimalloc' 2024-05-19T18:14:04.8466104Z Entering 'third_party/nccl/nccl' 2024-05-19T18:14:04.8490971Z Entering 'third_party/neon2sse' 2024-05-19T18:14:04.8516203Z Entering 'third_party/nlohmann' 2024-05-19T18:14:04.8542584Z Entering 'third_party/onnx' 2024-05-19T18:14:04.8579222Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T18:14:04.8604528Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T18:14:04.8631716Z Entering 'third_party/onnx-tensorrt' 2024-05-19T18:14:04.8656200Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:14:04.8688317Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:14:04.8716229Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:14:04.8744995Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:14:04.8775185Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T18:14:04.8804783Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:14:04.8829060Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:14:04.8856319Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:14:04.8883260Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:14:04.8910637Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:14:04.8935788Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:14:04.8961244Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:14:04.8986680Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:14:04.9015098Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:14:04.9041064Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:14:04.9083654Z Entering 'third_party/pocketfft' 2024-05-19T18:14:04.9110495Z Entering 'third_party/protobuf' 2024-05-19T18:14:04.9142494Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:14:04.9169750Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T18:14:04.9197717Z Entering 'third_party/psimd' 2024-05-19T18:14:04.9223950Z Entering 'third_party/pthreadpool' 2024-05-19T18:14:04.9249900Z Entering 'third_party/pybind11' 2024-05-19T18:14:04.9278502Z Entering 'third_party/python-peachpy' 2024-05-19T18:14:04.9306470Z Entering 'third_party/sleef' 2024-05-19T18:14:04.9332060Z Entering 'third_party/tbb' 2024-05-19T18:14:04.9360158Z Entering 'third_party/tensorpipe' 2024-05-19T18:14:04.9385354Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:14:04.9411553Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:14:04.9436552Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:14:04.9463084Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:14:04.9488230Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:14:04.9524430Z ##[endgroup] 2024-05-19T18:14:04.9554945Z [command]/usr/bin/git log -1 --format='%H' 2024-05-19T18:14:04.9575512Z 'c8c7ddfde57442cc28a394c385cab656ce9b1f5e' 2024-05-19T18:14:04.9745290Z Prepare all required actions 2024-05-19T18:14:04.9745743Z Getting action download info 2024-05-19T18:14:05.1420326Z ##[group]Run ./.github/actions/setup-linux 2024-05-19T18:14:05.1420752Z env: 2024-05-19T18:14:05.1421033Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:14:05.1421368Z ##[endgroup] 2024-05-19T18:14:05.1481193Z ##[group]Run set -euo pipefail 2024-05-19T18:14:05.1497956Z set -euo pipefail 2024-05-19T18:14:05.1498368Z function get_ec2_metadata() { 2024-05-19T18:14:05.1498902Z  # Pulled from instance metadata endpoint for EC2 2024-05-19T18:14:05.1499775Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-05-19T18:14:05.1500548Z  category=$1 2024-05-19T18:14:05.1501071Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-05-19T18:14:05.1501697Z  runner_name_str=i-01c138b15d87f731e 2024-05-19T18:14:05.1502194Z  if [[ -f /.inarc ]]; then 2024-05-19T18:14:05.1502686Z  echo "ARC Runner, no info on ec2 metadata" 2024-05-19T18:14:05.1503242Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-05-19T18:14:05.1503932Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-05-19T18:14:05.1504534Z  else 2024-05-19T18:14:05.1505026Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-05-19T18:14:05.1505594Z  fi 2024-05-19T18:14:05.1505878Z } 2024-05-19T18:14:05.1506235Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-05-19T18:14:05.1506810Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-05-19T18:14:05.1507604Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-05-19T18:14:05.1508173Z echo "system info $(uname -a)" 2024-05-19T18:14:05.1516121Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:14:05.1516617Z env: 2024-05-19T18:14:05.1516895Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:14:05.1517244Z ##[endgroup] 2024-05-19T18:14:05.1584333Z ami-id: ami-0ce0c36d7a00b20e2 2024-05-19T18:14:05.1629044Z instance-id: i-01c138b15d87f731e 2024-05-19T18:14:05.1670423Z instance-type: g5.4xlarge 2024-05-19T18:14:05.1676349Z system info Linux ip-10-0-40-27.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-05-19T18:14:05.1697047Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-05-19T18:14:05.1698140Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-05-19T18:14:05.1705698Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:14:05.1706191Z env: 2024-05-19T18:14:05.1706473Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:14:05.1706816Z ##[endgroup] 2024-05-19T18:14:05.1770358Z ##[group]Run if systemctl is-active --quiet docker; then 2024-05-19T18:14:05.1770974Z if systemctl is-active --quiet docker; then 2024-05-19T18:14:05.1771501Z  echo "Docker daemon is running..."; 2024-05-19T18:14:05.1771943Z else 2024-05-19T18:14:05.1772421Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-05-19T18:14:05.1772992Z fi 2024-05-19T18:14:05.1780827Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:14:05.1781319Z env: 2024-05-19T18:14:05.1781594Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:14:05.1781942Z ##[endgroup] 2024-05-19T18:14:05.1819230Z Docker daemon is running... 2024-05-19T18:14:05.1874471Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-05-19T18:14:05.1875029Z with: 2024-05-19T18:14:05.1875297Z shell: bash 2024-05-19T18:14:05.1875585Z timeout_minutes: 5 2024-05-19T18:14:05.1875905Z max_attempts: 3 2024-05-19T18:14:05.1876220Z retry_wait_seconds: 30 2024-05-19T18:14:05.1877672Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-05-19T18:14:05.1879064Z polling_interval_seconds: 1 2024-05-19T18:14:05.1879435Z warning_on_retry: true 2024-05-19T18:14:05.1879785Z continue_on_error: false 2024-05-19T18:14:05.1880117Z env: 2024-05-19T18:14:05.1880384Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:14:05.1880739Z AWS_RETRY_MODE: standard 2024-05-19T18:14:05.1881085Z AWS_MAX_ATTEMPTS: 5 2024-05-19T18:14:05.1881415Z AWS_DEFAULT_REGION: us-east-1 2024-05-19T18:14:05.1881784Z ##[endgroup] 2024-05-19T18:14:06.0453327Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-05-19T18:14:06.0454194Z Configure a credential helper to remove this warning. See 2024-05-19T18:14:06.0455008Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-05-19T18:14:06.0455518Z 2024-05-19T18:14:06.0455662Z Login Succeeded 2024-05-19T18:14:06.2384062Z Command completed after 1 attempt(s). 2024-05-19T18:14:06.2426153Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-05-19T18:14:06.2426861Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-05-19T18:14:06.2427502Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-05-19T18:14:06.2435216Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:14:06.2435708Z env: 2024-05-19T18:14:06.2435987Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:14:06.2436335Z ##[endgroup] 2024-05-19T18:14:06.2494624Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-05-19T18:14:06.2495569Z # ignore expansion of "docker ps -q" since it could be empty 2024-05-19T18:14:06.2496201Z # shellcheck disable=SC2046 2024-05-19T18:14:06.2496659Z docker stop $(docker ps -q) || true 2024-05-19T18:14:06.2497141Z # Prune all of the docker images 2024-05-19T18:14:06.2497778Z docker system prune -af 2024-05-19T18:14:06.2505704Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:14:06.2506435Z env: 2024-05-19T18:14:06.2506828Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:14:06.2507320Z ##[endgroup] 2024-05-19T18:14:06.2808974Z "docker stop" requires at least 1 argument. 2024-05-19T18:14:06.2810263Z See 'docker stop --help'. 2024-05-19T18:14:06.2810523Z 2024-05-19T18:14:06.2810762Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-05-19T18:14:06.2811132Z 2024-05-19T18:14:06.2811287Z Stop one or more running containers 2024-05-19T18:14:06.3133021Z Total reclaimed space: 0B 2024-05-19T18:14:06.3166333Z ##[group]Run set +e 2024-05-19T18:14:06.3166692Z set +e 2024-05-19T18:14:06.3166991Z set -x 2024-05-19T18:14:06.3167285Z  2024-05-19T18:14:06.3167599Z PT_DOMAIN=download.pytorch.org 2024-05-19T18:14:06.3168400Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-05-19T18:14:06.3169456Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-05-19T18:14:06.3170193Z # one is returned at random 2024-05-19T18:14:06.3170733Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-05-19T18:14:06.3171227Z  2024-05-19T18:14:06.3171544Z if [ -z "${RESOLVED_IP}" ]; then 2024-05-19T18:14:06.3172157Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-05-19T18:14:06.3172904Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-05-19T18:14:06.3173445Z  2024-05-19T18:14:06.3173777Z  if [ -z "${RESOLVED_IP}" ]; then 2024-05-19T18:14:06.3174323Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-05-19T18:14:06.3174824Z  exit 1 2024-05-19T18:14:06.3175126Z  fi 2024-05-19T18:14:06.3175402Z fi 2024-05-19T18:14:06.3175837Z  2024-05-19T18:14:06.3176200Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-05-19T18:14:06.3176706Z  # Clean up any old records first 2024-05-19T18:14:06.3177208Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-05-19T18:14:06.3177876Z fi 2024-05-19T18:14:06.3178159Z  2024-05-19T18:14:06.3178585Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-05-19T18:14:06.3179121Z cat /etc/hosts 2024-05-19T18:14:06.3186960Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:14:06.3187456Z env: 2024-05-19T18:14:06.3187733Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:14:06.3188084Z ##[endgroup] 2024-05-19T18:14:06.3206673Z + PT_DOMAIN=download.pytorch.org 2024-05-19T18:14:06.3210343Z ++ dig -4 +short download.pytorch.org 2024-05-19T18:14:06.3210804Z ++ tail -n1 2024-05-19T18:14:06.3556877Z + RESOLVED_IP=18.160.10.28 2024-05-19T18:14:06.3557373Z + '[' -z 18.160.10.28 ']' 2024-05-19T18:14:06.3557974Z + grep -r download.pytorch.org /etc/hosts 2024-05-19T18:14:06.3564172Z + echo '18.160.10.28 download.pytorch.org' 2024-05-19T18:14:06.3564793Z + sudo tee -a /etc/hosts 2024-05-19T18:14:06.3641081Z 18.160.10.28 download.pytorch.org 2024-05-19T18:14:06.3652404Z + cat /etc/hosts 2024-05-19T18:14:06.3658749Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-05-19T18:14:06.3672390Z ::1 localhost6 localhost6.localdomain6 2024-05-19T18:14:06.3672963Z 18.160.10.28 download.pytorch.org 2024-05-19T18:14:06.3826251Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-05-19T18:14:06.3826848Z with: 2024-05-19T18:14:06.3828016Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:14:06.3829221Z docker-build-dir: .ci/docker 2024-05-19T18:14:06.3829600Z working-directory: . 2024-05-19T18:14:06.3830072Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:14:06.3830596Z force-push: false 2024-05-19T18:14:06.3830904Z env: 2024-05-19T18:14:06.3831184Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:14:06.3831538Z ##[endgroup] 2024-05-19T18:14:06.3854767Z ##[group]Run set -ex 2024-05-19T18:14:06.3855126Z set -ex 2024-05-19T18:14:06.3855412Z  2024-05-19T18:14:06.3855964Z # If the docker build directory or the build script doesn't exist, the action will 2024-05-19T18:14:06.3856924Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-05-19T18:14:06.3872879Z # job could then download the pre-built image as usual 2024-05-19T18:14:06.3873619Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-05-19T18:14:06.3874280Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-05-19T18:14:06.3874899Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:14:06.3875438Z  2024-05-19T18:14:06.3875936Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-05-19T18:14:06.3876528Z  exit 0 2024-05-19T18:14:06.3876820Z else 2024-05-19T18:14:06.3877180Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-05-19T18:14:06.3877616Z fi 2024-05-19T18:14:06.3877888Z  2024-05-19T18:14:06.3878350Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-05-19T18:14:06.3879178Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-05-19T18:14:06.3879931Z  # use it as it is, but first let's extract the tag 2024-05-19T18:14:06.3880619Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-05-19T18:14:06.3881321Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:14:06.3882002Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:14:06.3882547Z else 2024-05-19T18:14:06.3882966Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-05-19T18:14:06.3883611Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:14:06.3884476Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:14:06.3885302Z fi 2024-05-19T18:14:06.3892794Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:14:06.3893288Z env: 2024-05-19T18:14:06.3893564Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:14:06.3893914Z REPO_NAME: pytorch 2024-05-19T18:14:06.3895027Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:14:06.3896203Z DOCKER_BUILD_DIR: .ci/docker 2024-05-19T18:14:06.3896697Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:14:06.3897205Z ##[endgroup] 2024-05-19T18:14:06.3917306Z + [[ ! -d .ci/docker ]] 2024-05-19T18:14:06.3917818Z + [[ ! -f .ci/docker/build.sh ]] 2024-05-19T18:14:06.3918199Z + echo skip=false 2024-05-19T18:14:06.3920084Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-05-19T18:14:06.3922568Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:14:06.3923815Z ++ awk -F '[:,]' '{print $2}' 2024-05-19T18:14:06.3928099Z + DOCKER_TAG=d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:14:06.3929067Z + echo docker-tag=d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:14:06.3930496Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:14:06.3959961Z ##[group]Run set +e 2024-05-19T18:14:06.3960331Z set +e 2024-05-19T18:14:06.3960637Z set -x 2024-05-19T18:14:06.3960938Z  2024-05-19T18:14:06.3961218Z login() { 2024-05-19T18:14:06.3961890Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-05-19T18:14:06.3962595Z } 2024-05-19T18:14:06.3962869Z  2024-05-19T18:14:06.3963149Z retry () { 2024-05-19T18:14:06.3963552Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-05-19T18:14:06.3963998Z } 2024-05-19T18:14:06.3964272Z  2024-05-19T18:14:06.3964591Z retry login "${DOCKER_REGISTRY}" 2024-05-19T18:14:06.3965142Z  2024-05-19T18:14:06.3965620Z # Check if image already exists, if it does then skip building it 2024-05-19T18:14:06.3966327Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-05-19T18:14:06.3966823Z  exit 0 2024-05-19T18:14:06.3967125Z fi 2024-05-19T18:14:06.3967408Z  2024-05-19T18:14:06.3967901Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-05-19T18:14:06.3968724Z # be empty. The default action would be to continue rebuild the image 2024-05-19T18:14:06.3969455Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-05-19T18:14:06.3970125Z  # if we're on the base branch then use the parent commit 2024-05-19T18:14:06.3970712Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-05-19T18:14:06.3971143Z else 2024-05-19T18:14:06.3971615Z  # otherwise we're on a PR, so use the most recent base commit 2024-05-19T18:14:06.3972298Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-05-19T18:14:06.3972792Z fi 2024-05-19T18:14:06.3973079Z  2024-05-19T18:14:06.3973398Z if [[ -z "${MERGE_BASE}" ]]; then 2024-05-19T18:14:06.3973939Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-05-19T18:14:06.3974398Z  2024-05-19T18:14:06.3975045Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-05-19T18:14:06.3975791Z  exit 0 2024-05-19T18:14:06.3976095Z fi 2024-05-19T18:14:06.3976365Z  2024-05-19T18:14:06.3976807Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-05-19T18:14:06.3978054Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-05-19T18:14:06.3978862Z  exit 1 2024-05-19T18:14:06.3979160Z fi 2024-05-19T18:14:06.3979439Z  2024-05-19T18:14:06.3979926Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-05-19T18:14:06.3980860Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-05-19T18:14:06.3981709Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-05-19T18:14:06.3982659Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-05-19T18:14:06.3983738Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-05-19T18:14:06.3984495Z fi 2024-05-19T18:14:06.3984782Z  2024-05-19T18:14:06.3985133Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-05-19T18:14:06.3992134Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:14:06.3992624Z env: 2024-05-19T18:14:06.3992896Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:14:06.3993261Z DOCKER_BUILD_DIR: .ci/docker 2024-05-19T18:14:06.3993739Z BASE_REVISION: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:14:06.3995032Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:14:06.3996255Z DOCKER_TAG: d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:14:06.3996852Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:14:06.3997363Z ##[endgroup] 2024-05-19T18:14:06.4016100Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:14:06.4016849Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:14:06.4017734Z + aws ecr get-login-password --region us-east-1 2024-05-19T18:14:06.4018714Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:14:06.8138634Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-05-19T18:14:06.8139557Z Configure a credential helper to remove this warning. See 2024-05-19T18:14:06.8140485Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-05-19T18:14:06.8140994Z 2024-05-19T18:14:06.8141115Z Login Succeeded 2024-05-19T18:14:06.8152132Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:14:07.0298198Z { 2024-05-19T18:14:07.0298599Z "schemaVersion": 2, 2024-05-19T18:14:07.0299269Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-05-19T18:14:07.0299840Z "config": { 2024-05-19T18:14:07.0300294Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-05-19T18:14:07.0300848Z "size": 45561, 2024-05-19T18:14:07.0301379Z "digest": "sha256:a0542ac5317cc182e1aac2cb8d63a2d6287450937722bab68464513bbbc603ef" 2024-05-19T18:14:07.0301981Z }, 2024-05-19T18:14:07.0302234Z "layers": [ 2024-05-19T18:14:07.0302505Z { 2024-05-19T18:14:07.0302922Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0312375Z + exit 0 2024-05-19T18:14:07.0314306Z "size": 28584317, 2024-05-19T18:14:07.0315098Z "digest": "sha256:63e9bbe323274e77e58d77c6ab6802d247458f784222fbb07a2556d6ec74ee05" 2024-05-19T18:14:07.0315933Z }, 2024-05-19T18:14:07.0316251Z { 2024-05-19T18:14:07.0316828Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0317565Z "size": 7944771, 2024-05-19T18:14:07.0318285Z "digest": "sha256:cfb3d849840ee60cee7b02bad68c1fc3c15928ebcd88f327754766b670578ed6" 2024-05-19T18:14:07.0319215Z }, 2024-05-19T18:14:07.0319540Z { 2024-05-19T18:14:07.0320113Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0320674Z "size": 57593718, 2024-05-19T18:14:07.0321197Z "digest": "sha256:968831e596a6288f0fed9b8612ee4ee8e75511037c4305058805492c5162e481" 2024-05-19T18:14:07.0321781Z }, 2024-05-19T18:14:07.0322022Z { 2024-05-19T18:14:07.0322436Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0322976Z "size": 187, 2024-05-19T18:14:07.0323506Z "digest": "sha256:ea310eb267ca1cab61b6b16f566cd28bfd59a741395a011f5e76716e15ba57c6" 2024-05-19T18:14:07.0324114Z }, 2024-05-19T18:14:07.0324350Z { 2024-05-19T18:14:07.0324987Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0325524Z "size": 6885, 2024-05-19T18:14:07.0326056Z "digest": "sha256:3af11d09e9cd1eb9c379f0a4071231e5a5642eb728b4b33bcb76be291f3c9488" 2024-05-19T18:14:07.0326881Z }, 2024-05-19T18:14:07.0327128Z { 2024-05-19T18:14:07.0327549Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0328183Z "size": 1361380219, 2024-05-19T18:14:07.0328725Z "digest": "sha256:ebfec18059b91e56882881ac34754f917861edb5f732c395d2a1a851bbd6db46" 2024-05-19T18:14:07.0329320Z }, 2024-05-19T18:14:07.0329560Z { 2024-05-19T18:14:07.0329971Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0330513Z "size": 62686, 2024-05-19T18:14:07.0331048Z "digest": "sha256:533b4aebf16914c763b7b0de3ce657590c6f979045e9fdf1f816adaf68d8a4d3" 2024-05-19T18:14:07.0331640Z }, 2024-05-19T18:14:07.0331874Z { 2024-05-19T18:14:07.0332292Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0332829Z "size": 1685, 2024-05-19T18:14:07.0333356Z "digest": "sha256:9dd75d06a0910f28cb1e484b8808724e5a6ee570ecb8fc04631368f546b39ed9" 2024-05-19T18:14:07.0333948Z }, 2024-05-19T18:14:07.0334195Z { 2024-05-19T18:14:07.0334615Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0335145Z "size": 1523, 2024-05-19T18:14:07.0335677Z "digest": "sha256:30bfca4dd3492d60ed8035b0eeb1229897041140db117f5663465a551e25851d" 2024-05-19T18:14:07.0336275Z }, 2024-05-19T18:14:07.0336519Z { 2024-05-19T18:14:07.0336935Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0337524Z "size": 2608186849, 2024-05-19T18:14:07.0338339Z "digest": "sha256:1b57ce94cad9a5097d99b8d6f7bcd51df5a162fc3c5f5686b689b76993724bc8" 2024-05-19T18:14:07.0338948Z }, 2024-05-19T18:14:07.0339182Z { 2024-05-19T18:14:07.0339599Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0340130Z "size": 86631, 2024-05-19T18:14:07.0340662Z "digest": "sha256:9ee6bdb31195dd42fd98147a75d540efe0c5708e0ecda866a0bca060084fddab" 2024-05-19T18:14:07.0341252Z }, 2024-05-19T18:14:07.0341492Z { 2024-05-19T18:14:07.0341918Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0342439Z "size": 1822, 2024-05-19T18:14:07.0342975Z "digest": "sha256:8af83a173aa33e1330fdde4efc5b96334ef85736ec0725bfeedc72c227dcfcc5" 2024-05-19T18:14:07.0343585Z }, 2024-05-19T18:14:07.0343826Z { 2024-05-19T18:14:07.0344286Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0344832Z "size": 245735746, 2024-05-19T18:14:07.0345377Z "digest": "sha256:f8281676bd20d66f882b7edad2f27bd52787f63ff5c68768681df27d74d9d07c" 2024-05-19T18:14:07.0345979Z }, 2024-05-19T18:14:07.0346213Z { 2024-05-19T18:14:07.0346629Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0347167Z "size": 544, 2024-05-19T18:14:07.0347685Z "digest": "sha256:748e3cfdd8d6f501456917dcc4a9e8b11092aa4e784a46b00c3a7a13274e2742" 2024-05-19T18:14:07.0348281Z }, 2024-05-19T18:14:07.0348523Z { 2024-05-19T18:14:07.0348945Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0349473Z "size": 1279, 2024-05-19T18:14:07.0349998Z "digest": "sha256:5fa8d379f04e0947dc4af19169168984b6dfe62a40efbd97ee9bc60015cb2970" 2024-05-19T18:14:07.0350600Z }, 2024-05-19T18:14:07.0350843Z { 2024-05-19T18:14:07.0351255Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0351793Z "size": 484, 2024-05-19T18:14:07.0352328Z "digest": "sha256:a95b9f41fb5bbac1fd00c55419da0b54b0c1b38cd46e22b6e95293bd639615b5" 2024-05-19T18:14:07.0352940Z }, 2024-05-19T18:14:07.0353176Z { 2024-05-19T18:14:07.0353596Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0354131Z "size": 91705725, 2024-05-19T18:14:07.0354660Z "digest": "sha256:8b6d0b819d19a961789f05d3c03c73f6ee67059e9464c8bbc0a1124113a32395" 2024-05-19T18:14:07.0355253Z }, 2024-05-19T18:14:07.0355497Z { 2024-05-19T18:14:07.0355916Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0356536Z "size": 3236, 2024-05-19T18:14:07.0357068Z "digest": "sha256:c61996e733996ff4a9cf89f2a69716a6add90b5c8b1902feb91d8e0dfd0f7bc4" 2024-05-19T18:14:07.0357720Z }, 2024-05-19T18:14:07.0357964Z { 2024-05-19T18:14:07.0358380Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0358917Z "size": 1905, 2024-05-19T18:14:07.0359439Z "digest": "sha256:9e69dcd7661c1bee67677240651319ec72ee977219e8097f93be00c6555c9ab5" 2024-05-19T18:14:07.0360022Z }, 2024-05-19T18:14:07.0360274Z { 2024-05-19T18:14:07.0360695Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0361235Z "size": 699, 2024-05-19T18:14:07.0361754Z "digest": "sha256:cfcaf2eb82498ef031f7d481e635b1a37af01b9e389f30555b603d75e2c00602" 2024-05-19T18:14:07.0362359Z }, 2024-05-19T18:14:07.0362605Z { 2024-05-19T18:14:07.0363025Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0363559Z "size": 2811659560, 2024-05-19T18:14:07.0364127Z "digest": "sha256:7cb10a6ae8df3c778e529febd7eef0333c2af45e2a2ed45f45967bb7b4219131" 2024-05-19T18:14:07.0364847Z }, 2024-05-19T18:14:07.0365102Z { 2024-05-19T18:14:07.0365514Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0366048Z "size": 378, 2024-05-19T18:14:07.0366574Z "digest": "sha256:07fa87a43fb8c069109d9c97743dcc016ad605a221c1df4c7d79863d0d0a2229" 2024-05-19T18:14:07.0367167Z }, 2024-05-19T18:14:07.0367410Z { 2024-05-19T18:14:07.0367838Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0368375Z "size": 11887, 2024-05-19T18:14:07.0368899Z "digest": "sha256:f16e94b048018b4677a272dcb1cb347e9c90be713acffa5bdd2d63a972224ca9" 2024-05-19T18:14:07.0369499Z }, 2024-05-19T18:14:07.0369738Z { 2024-05-19T18:14:07.0370154Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0370679Z "size": 803, 2024-05-19T18:14:07.0371207Z "digest": "sha256:c857065196d6f476be67a20b1e26f3ef6e71ae427e4feee2412e7f65cc208493" 2024-05-19T18:14:07.0371802Z }, 2024-05-19T18:14:07.0372036Z { 2024-05-19T18:14:07.0372459Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0373000Z "size": 106, 2024-05-19T18:14:07.0373532Z "digest": "sha256:9d5c38cad0c9a12d6b9b3070caf13ae1003594cf17c75170ccadf0f8719f93ab" 2024-05-19T18:14:07.0374134Z }, 2024-05-19T18:14:07.0374381Z { 2024-05-19T18:14:07.0374797Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0375340Z "size": 503, 2024-05-19T18:14:07.0375852Z "digest": "sha256:925ac0c7ff935448f88445123dc223bb4531147eec4d0e0c34f9f05bd4e14f46" 2024-05-19T18:14:07.0376451Z }, 2024-05-19T18:14:07.0376693Z { 2024-05-19T18:14:07.0377115Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0377839Z "size": 121477245, 2024-05-19T18:14:07.0378421Z "digest": "sha256:cfbf8bf4873dc1eef0b1a35dd290348be3ceb2dafccdc4820e9460ddfd8f7c54" 2024-05-19T18:14:07.0379041Z }, 2024-05-19T18:14:07.0379276Z { 2024-05-19T18:14:07.0379690Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0380227Z "size": 109, 2024-05-19T18:14:07.0380758Z "digest": "sha256:01914a189c0b5a993f1fc3ddb461abe3964eba3ae55fab9eff5a53f77de19178" 2024-05-19T18:14:07.0381353Z }, 2024-05-19T18:14:07.0381596Z { 2024-05-19T18:14:07.0382014Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0382544Z "size": 488, 2024-05-19T18:14:07.0383057Z "digest": "sha256:3c2451fbe84c6b4678a24659b7abe1ea3cca59da338f16b7563d6875accf6eee" 2024-05-19T18:14:07.0383658Z }, 2024-05-19T18:14:07.0383901Z { 2024-05-19T18:14:07.0384310Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0384843Z "size": 298, 2024-05-19T18:14:07.0385378Z "digest": "sha256:aea8a2a73b72b6c647fafb9f4b1b3d65cb7c28172d3b7a58de3bdb6335addb00" 2024-05-19T18:14:07.0386077Z }, 2024-05-19T18:14:07.0386316Z { 2024-05-19T18:14:07.0386735Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0387361Z "size": 103, 2024-05-19T18:14:07.0387896Z "digest": "sha256:c26f348bfda1e1a6e32f77a9054fa8d26cd59cf6fe8cc3ea1ff9d1816dc74d22" 2024-05-19T18:14:07.0388498Z }, 2024-05-19T18:14:07.0388738Z { 2024-05-19T18:14:07.0389152Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0389686Z "size": 1471, 2024-05-19T18:14:07.0390193Z "digest": "sha256:347c41648f51ed569232ea48943ef3e1424197b90bfae5b89898f2273f64e76b" 2024-05-19T18:14:07.0390777Z }, 2024-05-19T18:14:07.0391018Z { 2024-05-19T18:14:07.0391428Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0391960Z "size": 444189070, 2024-05-19T18:14:07.0392494Z "digest": "sha256:00953247955f346ee0a69eb263efa2135a9fc3e9680322e5c39cf79e4c06fab5" 2024-05-19T18:14:07.0393081Z }, 2024-05-19T18:14:07.0393327Z { 2024-05-19T18:14:07.0393741Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0394270Z "size": 162, 2024-05-19T18:14:07.0394809Z "digest": "sha256:df27b4a5ce7ca0be34e1295afceb581f6ac42c1c0888e253c481bbce25efde72" 2024-05-19T18:14:07.0395413Z }, 2024-05-19T18:14:07.0395653Z { 2024-05-19T18:14:07.0396072Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0396595Z "size": 564, 2024-05-19T18:14:07.0397121Z "digest": "sha256:acb7fcd74bf2071a5d257e338f2229b9a9747f6e96de138091ed80cfa845709f" 2024-05-19T18:14:07.0397721Z }, 2024-05-19T18:14:07.0397966Z { 2024-05-19T18:14:07.0398374Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0398912Z "size": 35867331, 2024-05-19T18:14:07.0399452Z "digest": "sha256:5bd6ac51ef5d3109aba547ec49aec04535079b866d45939fa098e33b7f20d94f" 2024-05-19T18:14:07.0400053Z }, 2024-05-19T18:14:07.0400290Z { 2024-05-19T18:14:07.0400715Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0401246Z "size": 104, 2024-05-19T18:14:07.0401773Z "digest": "sha256:1795bce633b0c48054bada65aede57ae82761e922b39b354aebedd2d3fc82a01" 2024-05-19T18:14:07.0402380Z }, 2024-05-19T18:14:07.0402624Z { 2024-05-19T18:14:07.0403043Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0403569Z "size": 425, 2024-05-19T18:14:07.0404119Z "digest": "sha256:e41de6016f3c0f9309d3727db3cf335f2789260369b9c02485b5a1eda2a23949" 2024-05-19T18:14:07.0404809Z }, 2024-05-19T18:14:07.0405052Z { 2024-05-19T18:14:07.0405461Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0406001Z "size": 20262012, 2024-05-19T18:14:07.0406562Z "digest": "sha256:458f5f2af8cfff41b832b1ffa0ac89536cc75afaa2b007348475812be84a3168" 2024-05-19T18:14:07.0407163Z }, 2024-05-19T18:14:07.0407396Z { 2024-05-19T18:14:07.0407818Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0408354Z "size": 437, 2024-05-19T18:14:07.0408878Z "digest": "sha256:ef99a2a9cffb05645e67c1ff7eff9d0400a37d6277a734e0534e544e6d70abc3" 2024-05-19T18:14:07.0409491Z }, 2024-05-19T18:14:07.0409732Z { 2024-05-19T18:14:07.0410152Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0410678Z "size": 699, 2024-05-19T18:14:07.0411205Z "digest": "sha256:cfcaf2eb82498ef031f7d481e635b1a37af01b9e389f30555b603d75e2c00602" 2024-05-19T18:14:07.0411800Z }, 2024-05-19T18:14:07.0412040Z { 2024-05-19T18:14:07.0412447Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0412982Z "size": 142, 2024-05-19T18:14:07.0413497Z "digest": "sha256:1ba8d08d58765b6585b8a6430120bdf33e85a118dbe004432a9515481c728a6f" 2024-05-19T18:14:07.0414086Z }, 2024-05-19T18:14:07.0414321Z { 2024-05-19T18:14:07.0414738Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0415355Z "size": 136, 2024-05-19T18:14:07.0415871Z "digest": "sha256:0c67ae0247cf4129424322fdc4c7d64de5f02712a6851839df767491d40ac668" 2024-05-19T18:14:07.0416499Z }, 2024-05-19T18:14:07.0416746Z { 2024-05-19T18:14:07.0417167Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0417881Z "size": 4981810470, 2024-05-19T18:14:07.0418427Z "digest": "sha256:3ebf7de5956e7374b72ff809508272a8ac6ff23b2293581bef5847cd0af1b4bb" 2024-05-19T18:14:07.0419020Z }, 2024-05-19T18:14:07.0419265Z { 2024-05-19T18:14:07.0419673Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0420204Z "size": 190, 2024-05-19T18:14:07.0420728Z "digest": "sha256:a6b88127ee12fade12d3202bc71d30768c0ddc0eb46ad572bd5089b6aa1bb0e2" 2024-05-19T18:14:07.0421329Z }, 2024-05-19T18:14:07.0421563Z { 2024-05-19T18:14:07.0421978Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0422506Z "size": 565, 2024-05-19T18:14:07.0423028Z "digest": "sha256:7b0cf0b8be50e7fae7398549f658c86b1e9c348102f84ef2c230bc6af5599739" 2024-05-19T18:14:07.0423630Z }, 2024-05-19T18:14:07.0423882Z { 2024-05-19T18:14:07.0424296Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0424822Z "size": 43162755, 2024-05-19T18:14:07.0425359Z "digest": "sha256:044061f29c13bd178d3113fb9ccfc367da97f96f6bf948dfcae1ff5b1e17be2f" 2024-05-19T18:14:07.0425957Z }, 2024-05-19T18:14:07.0426199Z { 2024-05-19T18:14:07.0426607Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0427139Z "size": 106, 2024-05-19T18:14:07.0427657Z "digest": "sha256:f21869a483d43c856cf8639bdb4270d8295023741b5cddebed62bcfe3b8caa7b" 2024-05-19T18:14:07.0428247Z }, 2024-05-19T18:14:07.0428495Z { 2024-05-19T18:14:07.0428907Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0429454Z "size": 1212, 2024-05-19T18:14:07.0429985Z "digest": "sha256:a155d9e6e7e7fd2243fba6789a98b96aa2db9d5d6462b50d9c11cfc7f3ac34e7" 2024-05-19T18:14:07.0430587Z }, 2024-05-19T18:14:07.0430832Z { 2024-05-19T18:14:07.0431258Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0431791Z "size": 699, 2024-05-19T18:14:07.0432315Z "digest": "sha256:cfcaf2eb82498ef031f7d481e635b1a37af01b9e389f30555b603d75e2c00602" 2024-05-19T18:14:07.0432922Z }, 2024-05-19T18:14:07.0433163Z { 2024-05-19T18:14:07.0433573Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0434102Z "size": 137, 2024-05-19T18:14:07.0434626Z "digest": "sha256:6028f518cba8d1b51ca287446cfc0a9db237fbeea2198738ee2c114a24a8c34a" 2024-05-19T18:14:07.0435218Z }, 2024-05-19T18:14:07.0435457Z { 2024-05-19T18:14:07.0435870Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0436400Z "size": 119, 2024-05-19T18:14:07.0436922Z "digest": "sha256:9f0e16ae766e08c8bf1276b07cd6781a3f9ca54131846275ffcfc9473a6312dd" 2024-05-19T18:14:07.0437516Z }, 2024-05-19T18:14:07.0437757Z { 2024-05-19T18:14:07.0438176Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0438710Z "size": 1853517530, 2024-05-19T18:14:07.0439245Z "digest": "sha256:26d45b11322ae5593c42d7929f046ca6d236f7ec5e8a13d8930360c013d32747" 2024-05-19T18:14:07.0439825Z }, 2024-05-19T18:14:07.0440063Z { 2024-05-19T18:14:07.0440478Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0441019Z "size": 172, 2024-05-19T18:14:07.0441535Z "digest": "sha256:a78a7e4425658766ba6e70bb1367f10c352dc4a7bfb16432be3d9a3496368333" 2024-05-19T18:14:07.0442116Z }, 2024-05-19T18:14:07.0442359Z { 2024-05-19T18:14:07.0442775Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0443312Z "size": 1838, 2024-05-19T18:14:07.0443844Z "digest": "sha256:df585564bf4cc28aec4957fbec751620ae4ca50edbbdcec1c5eacc306a16e2e1" 2024-05-19T18:14:07.0444544Z }, 2024-05-19T18:14:07.0444870Z { 2024-05-19T18:14:07.0445289Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0445887Z "size": 7529792, 2024-05-19T18:14:07.0446426Z "digest": "sha256:f125619c47120e1a542ddb0e822b5ffc07f9d165a0977f8c4064fdf67865e3fd" 2024-05-19T18:14:07.0447016Z }, 2024-05-19T18:14:07.0447251Z { 2024-05-19T18:14:07.0447664Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0448195Z "size": 162, 2024-05-19T18:14:07.0448716Z "digest": "sha256:4cc671ff8870320a4e5bab71f83b755b9b96c777dba43fdbfc1a0e3d001d8936" 2024-05-19T18:14:07.0449307Z }, 2024-05-19T18:14:07.0449547Z { 2024-05-19T18:14:07.0449966Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0450498Z "size": 7939, 2024-05-19T18:14:07.0451013Z "digest": "sha256:e1d3209e04ae7e31773f49e41f7cee2949fa017569383786876f6d1ba0d7d8ac" 2024-05-19T18:14:07.0451603Z }, 2024-05-19T18:14:07.0451851Z { 2024-05-19T18:14:07.0452257Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0452796Z "size": 8072, 2024-05-19T18:14:07.0453317Z "digest": "sha256:322e57f70842c48212f87b2e4c1cc094c25470b90fec1d30ad715f4d907c1f79" 2024-05-19T18:14:07.0453901Z }, 2024-05-19T18:14:07.0454137Z { 2024-05-19T18:14:07.0454556Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0455094Z "size": 301, 2024-05-19T18:14:07.0455614Z "digest": "sha256:d58f09a95d3e18b4d653b7767e371c1e9ef320625fa4cc6a92cbd1771d7ef768" 2024-05-19T18:14:07.0456201Z }, 2024-05-19T18:14:07.0456443Z { 2024-05-19T18:14:07.0456855Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0457387Z "size": 7617584, 2024-05-19T18:14:07.0458132Z "digest": "sha256:f611e28aa59ab55b92e54aac3cc04729565ccae9d0edb75cc45a534d752a9de1" 2024-05-19T18:14:07.0458743Z }, 2024-05-19T18:14:07.0458987Z { 2024-05-19T18:14:07.0459409Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0459945Z "size": 108, 2024-05-19T18:14:07.0460465Z "digest": "sha256:ff67e6bd8e2302312131632eae7033ce84ce85503282bfd8b31c008ff533990a" 2024-05-19T18:14:07.0461067Z }, 2024-05-19T18:14:07.0461305Z { 2024-05-19T18:14:07.0461730Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0462268Z "size": 54145477, 2024-05-19T18:14:07.0462818Z "digest": "sha256:d2b06b7f79e8770329516b40d5189f7c2b2d38404132638103a594685fe1d569" 2024-05-19T18:14:07.0463398Z }, 2024-05-19T18:14:07.0463648Z { 2024-05-19T18:14:07.0464065Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0464602Z "size": 507, 2024-05-19T18:14:07.0465124Z "digest": "sha256:c27bc13ce9d41097b89fa4c7f53feb417379940ba35474fede1b3322540e85e9" 2024-05-19T18:14:07.0465721Z }, 2024-05-19T18:14:07.0465967Z { 2024-05-19T18:14:07.0466393Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0466925Z "size": 1443748723, 2024-05-19T18:14:07.0467473Z "digest": "sha256:8680ed64cc192868f4e34d4dba9121684e3bba13ac365c8e8539386fe4a5147c" 2024-05-19T18:14:07.0468079Z }, 2024-05-19T18:14:07.0468324Z { 2024-05-19T18:14:07.0468737Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0469274Z "size": 106, 2024-05-19T18:14:07.0469788Z "digest": "sha256:400e9c337687d211e1a78744e24985fbaca792e988f3caf5861040d80b088662" 2024-05-19T18:14:07.0470370Z }, 2024-05-19T18:14:07.0470610Z { 2024-05-19T18:14:07.0471034Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0471572Z "size": 558, 2024-05-19T18:14:07.0472090Z "digest": "sha256:914d4657be1b6936372f45195af96a959eab5bced80ca9deeafc3d6d9db6c03c" 2024-05-19T18:14:07.0472689Z }, 2024-05-19T18:14:07.0472935Z { 2024-05-19T18:14:07.0473361Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0473980Z "size": 46248255, 2024-05-19T18:14:07.0474521Z "digest": "sha256:ecc1d0285d28b6339b9648e2195ce576f30288563ac5de0882653a42bcc0eb5b" 2024-05-19T18:14:07.0475185Z }, 2024-05-19T18:14:07.0475424Z { 2024-05-19T18:14:07.0475847Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:14:07.0476389Z "size": 111, 2024-05-19T18:14:07.0476918Z "digest": "sha256:b882dcc190622361a09ea00b970e0fae5f81ea1f0431a45afa329fd3c97f66d9" 2024-05-19T18:14:07.0477514Z } 2024-05-19T18:14:07.0477760Z ] 2024-05-19T18:14:07.0478001Z } 2024-05-19T18:14:07.0578785Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-05-19T18:14:07.0579263Z tag=${ECR_DOCKER_IMAGE##*/} 2024-05-19T18:14:07.0579789Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-05-19T18:14:07.0587639Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:14:07.0588131Z env: 2024-05-19T18:14:07.0588403Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:14:07.0589528Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:14:07.0590679Z ##[endgroup] 2024-05-19T18:14:07.0612826Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks-d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:14:07.0678660Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-05-19T18:14:07.0679231Z with: 2024-05-19T18:14:07.0680273Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:14:07.0681561Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:14:07.0682062Z env: 2024-05-19T18:14:07.0682342Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:14:07.0682694Z ##[endgroup] 2024-05-19T18:14:07.0699052Z ##[group]Run set -x 2024-05-19T18:14:07.0699396Z set -x 2024-05-19T18:14:07.0699697Z set +e 2024-05-19T18:14:07.0699991Z  2024-05-19T18:14:07.0700417Z login() { 2024-05-19T18:14:07.0701080Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-05-19T18:14:07.0701777Z } 2024-05-19T18:14:07.0702050Z  2024-05-19T18:14:07.0702346Z retry () { 2024-05-19T18:14:07.0702736Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-05-19T18:14:07.0703175Z } 2024-05-19T18:14:07.0703450Z  2024-05-19T18:14:07.0703766Z retry login "${DOCKER_REGISTRY}" 2024-05-19T18:14:07.0704171Z  2024-05-19T18:14:07.0704443Z set -e 2024-05-19T18:14:07.0704912Z # ignore output since only exit code is used for conditional 2024-05-19T18:14:07.0705601Z # only pull docker image if it's not available locally 2024-05-19T18:14:07.0706347Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-05-19T18:14:07.0707031Z  retry docker pull "${DOCKER_IMAGE}" 2024-05-19T18:14:07.0707470Z fi 2024-05-19T18:14:07.0714926Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:14:07.0715405Z env: 2024-05-19T18:14:07.0715678Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:14:07.0716802Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:14:07.0719445Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:14:07.0719956Z ##[endgroup] 2024-05-19T18:14:07.0739856Z + set +e 2024-05-19T18:14:07.0740370Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:14:07.0740996Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:14:07.0742645Z + aws ecr get-login-password --region us-east-1 2024-05-19T18:14:07.0743407Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:14:07.4909723Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-05-19T18:14:07.4910868Z Configure a credential helper to remove this warning. See 2024-05-19T18:14:07.4911686Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-05-19T18:14:07.4912221Z 2024-05-19T18:14:07.4914088Z Login Succeeded 2024-05-19T18:14:07.4925100Z + set -e 2024-05-19T18:14:07.4926749Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:14:07.5072269Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:14:07.7382289Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:14:07.7384488Z d55d6a6f08250cd2609d528116ab6fd6e787bee0: Pulling from pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks 2024-05-19T18:14:07.7385618Z 63e9bbe32327: Pulling fs layer 2024-05-19T18:14:07.7386153Z cfb3d849840e: Pulling fs layer 2024-05-19T18:14:07.7386920Z 968831e596a6: Pulling fs layer 2024-05-19T18:14:07.7387448Z ea310eb267ca: Pulling fs layer 2024-05-19T18:14:07.7387938Z 3af11d09e9cd: Pulling fs layer 2024-05-19T18:14:07.7388421Z ebfec18059b9: Pulling fs layer 2024-05-19T18:14:07.7388940Z 533b4aebf169: Pulling fs layer 2024-05-19T18:14:07.7389475Z 9dd75d06a091: Pulling fs layer 2024-05-19T18:14:07.7389983Z 30bfca4dd349: Pulling fs layer 2024-05-19T18:14:07.7390518Z 1b57ce94cad9: Pulling fs layer 2024-05-19T18:14:07.7391077Z 9ee6bdb31195: Pulling fs layer 2024-05-19T18:14:07.7391594Z 8af83a173aa3: Pulling fs layer 2024-05-19T18:14:07.7392131Z f8281676bd20: Pulling fs layer 2024-05-19T18:14:07.7392645Z 748e3cfdd8d6: Pulling fs layer 2024-05-19T18:14:07.7393103Z 5fa8d379f04e: Pulling fs layer 2024-05-19T18:14:07.7393489Z a95b9f41fb5b: Pulling fs layer 2024-05-19T18:14:07.7393889Z 8b6d0b819d19: Pulling fs layer 2024-05-19T18:14:07.7394257Z c61996e73399: Pulling fs layer 2024-05-19T18:14:07.7394642Z 9e69dcd7661c: Pulling fs layer 2024-05-19T18:14:07.7395041Z cfcaf2eb8249: Pulling fs layer 2024-05-19T18:14:07.7395415Z 7cb10a6ae8df: Pulling fs layer 2024-05-19T18:14:07.7395797Z 07fa87a43fb8: Pulling fs layer 2024-05-19T18:14:07.7396165Z 30bfca4dd349: Waiting 2024-05-19T18:14:07.7396505Z f16e94b04801: Pulling fs layer 2024-05-19T18:14:07.7396862Z ea310eb267ca: Waiting 2024-05-19T18:14:07.7397198Z f8281676bd20: Waiting 2024-05-19T18:14:07.7397615Z 3af11d09e9cd: Waiting 2024-05-19T18:14:07.7398070Z c857065196d6: Pulling fs layer 2024-05-19T18:14:07.7398620Z 9d5c38cad0c9: Pulling fs layer 2024-05-19T18:14:07.7399098Z 1b57ce94cad9: Waiting 2024-05-19T18:14:07.7399538Z 925ac0c7ff93: Pulling fs layer 2024-05-19T18:14:07.7400024Z ebfec18059b9: Waiting 2024-05-19T18:14:07.7400429Z cfbf8bf4873d: Pulling fs layer 2024-05-19T18:14:07.7400879Z 8b6d0b819d19: Waiting 2024-05-19T18:14:07.7401209Z 9ee6bdb31195: Waiting 2024-05-19T18:14:07.7401532Z 748e3cfdd8d6: Waiting 2024-05-19T18:14:07.7401845Z 8af83a173aa3: Waiting 2024-05-19T18:14:07.7402235Z 533b4aebf169: Waiting 2024-05-19T18:14:07.7402679Z a95b9f41fb5b: Waiting 2024-05-19T18:14:07.7403049Z c61996e73399: Waiting 2024-05-19T18:14:07.7403356Z 5fa8d379f04e: Waiting 2024-05-19T18:14:07.7403668Z 9d5c38cad0c9: Waiting 2024-05-19T18:14:07.7403988Z f16e94b04801: Waiting 2024-05-19T18:14:07.7404314Z 01914a189c0b: Pulling fs layer 2024-05-19T18:14:07.7404929Z cfbf8bf4873d: Waiting 2024-05-19T18:14:07.7405370Z 3c2451fbe84c: Pulling fs layer 2024-05-19T18:14:07.7405805Z aea8a2a73b72: Pulling fs layer 2024-05-19T18:14:07.7406203Z c26f348bfda1: Pulling fs layer 2024-05-19T18:14:07.7406616Z 347c41648f51: Pulling fs layer 2024-05-19T18:14:07.7406983Z 00953247955f: Pulling fs layer 2024-05-19T18:14:07.7407398Z df27b4a5ce7c: Pulling fs layer 2024-05-19T18:14:07.7407772Z acb7fcd74bf2: Pulling fs layer 2024-05-19T18:14:07.7408323Z 5bd6ac51ef5d: Pulling fs layer 2024-05-19T18:14:07.7408745Z 1795bce633b0: Pulling fs layer 2024-05-19T18:14:07.7409095Z aea8a2a73b72: Waiting 2024-05-19T18:14:07.7409410Z c857065196d6: Waiting 2024-05-19T18:14:07.7409725Z 347c41648f51: Waiting 2024-05-19T18:14:07.7410037Z 00953247955f: Waiting 2024-05-19T18:14:07.7410388Z c26f348bfda1: Waiting 2024-05-19T18:14:07.7410706Z 5bd6ac51ef5d: Waiting 2024-05-19T18:14:07.7411098Z e41de6016f3c: Pulling fs layer 2024-05-19T18:14:07.7411473Z 458f5f2af8cf: Pulling fs layer 2024-05-19T18:14:07.7411854Z ef99a2a9cffb: Pulling fs layer 2024-05-19T18:14:07.7412221Z df27b4a5ce7c: Waiting 2024-05-19T18:14:07.7412543Z e41de6016f3c: Waiting 2024-05-19T18:14:07.7412868Z 1ba8d08d5876: Pulling fs layer 2024-05-19T18:14:07.7413242Z 0c67ae0247cf: Pulling fs layer 2024-05-19T18:14:07.7413600Z 1ba8d08d5876: Waiting 2024-05-19T18:14:07.7413934Z 3ebf7de5956e: Pulling fs layer 2024-05-19T18:14:07.7414304Z a6b88127ee12: Pulling fs layer 2024-05-19T18:14:07.7414662Z 1795bce633b0: Waiting 2024-05-19T18:14:07.7415006Z 7b0cf0b8be50: Pulling fs layer 2024-05-19T18:14:07.7415364Z 3c2451fbe84c: Waiting 2024-05-19T18:14:07.7415696Z 044061f29c13: Pulling fs layer 2024-05-19T18:14:07.7416144Z a6b88127ee12: Waiting 2024-05-19T18:14:07.7416488Z f21869a483d4: Pulling fs layer 2024-05-19T18:14:07.7416844Z 7cb10a6ae8df: Waiting 2024-05-19T18:14:07.7417169Z 3ebf7de5956e: Waiting 2024-05-19T18:14:07.7417829Z a155d9e6e7e7: Pulling fs layer 2024-05-19T18:14:07.7418262Z 6028f518cba8: Pulling fs layer 2024-05-19T18:14:07.7418631Z 9f0e16ae766e: Pulling fs layer 2024-05-19T18:14:07.7419008Z 26d45b11322a: Pulling fs layer 2024-05-19T18:14:07.7419377Z a78a7e442565: Pulling fs layer 2024-05-19T18:14:07.7419725Z a155d9e6e7e7: Waiting 2024-05-19T18:14:07.7420040Z a78a7e442565: Waiting 2024-05-19T18:14:07.7420371Z df585564bf4c: Pulling fs layer 2024-05-19T18:14:07.7420743Z f125619c4712: Pulling fs layer 2024-05-19T18:14:07.7421089Z 7b0cf0b8be50: Waiting 2024-05-19T18:14:07.7421409Z df585564bf4c: Waiting 2024-05-19T18:14:07.7421748Z 4cc671ff8870: Pulling fs layer 2024-05-19T18:14:07.7422122Z e1d3209e04ae: Pulling fs layer 2024-05-19T18:14:07.7422483Z 322e57f70842: Pulling fs layer 2024-05-19T18:14:07.7422840Z 044061f29c13: Waiting 2024-05-19T18:14:07.7423174Z d58f09a95d3e: Pulling fs layer 2024-05-19T18:14:07.7423517Z f125619c4712: Waiting 2024-05-19T18:14:07.7423835Z e1d3209e04ae: Waiting 2024-05-19T18:14:07.7424155Z d58f09a95d3e: Waiting 2024-05-19T18:14:07.7424475Z 322e57f70842: Waiting 2024-05-19T18:14:07.7424783Z 6028f518cba8: Waiting 2024-05-19T18:14:07.7425120Z f611e28aa59a: Pulling fs layer 2024-05-19T18:14:07.7425505Z ff67e6bd8e23: Pulling fs layer 2024-05-19T18:14:07.7425855Z 26d45b11322a: Waiting 2024-05-19T18:14:07.7426168Z f611e28aa59a: Waiting 2024-05-19T18:14:07.7426498Z d2b06b7f79e8: Pulling fs layer 2024-05-19T18:14:07.7426873Z c27bc13ce9d4: Pulling fs layer 2024-05-19T18:14:07.7427250Z 8680ed64cc19: Pulling fs layer 2024-05-19T18:14:07.7427621Z 400e9c337687: Pulling fs layer 2024-05-19T18:14:07.7427977Z 400e9c337687: Waiting 2024-05-19T18:14:07.7428311Z 914d4657be1b: Pulling fs layer 2024-05-19T18:14:07.7428669Z ecc1d0285d28: Pulling fs layer 2024-05-19T18:14:07.7429041Z b882dcc19062: Pulling fs layer 2024-05-19T18:14:07.7429399Z ecc1d0285d28: Waiting 2024-05-19T18:14:07.7429718Z b882dcc19062: Waiting 2024-05-19T18:14:07.7430027Z 914d4657be1b: Waiting 2024-05-19T18:14:07.7430342Z ff67e6bd8e23: Waiting 2024-05-19T18:14:07.7430663Z c27bc13ce9d4: Waiting 2024-05-19T18:14:07.7430967Z d2b06b7f79e8: Waiting 2024-05-19T18:14:07.8829741Z cfb3d849840e: Verifying Checksum 2024-05-19T18:14:07.8830221Z cfb3d849840e: Download complete 2024-05-19T18:14:07.9659267Z ea310eb267ca: Verifying Checksum 2024-05-19T18:14:07.9659999Z ea310eb267ca: Download complete 2024-05-19T18:14:08.0648897Z 3af11d09e9cd: Verifying Checksum 2024-05-19T18:14:08.0649461Z 3af11d09e9cd: Download complete 2024-05-19T18:14:08.0833900Z 63e9bbe32327: Download complete 2024-05-19T18:14:08.1767774Z 533b4aebf169: Download complete 2024-05-19T18:14:08.2666181Z 9dd75d06a091: Verifying Checksum 2024-05-19T18:14:08.2666780Z 9dd75d06a091: Download complete 2024-05-19T18:14:08.3491085Z 30bfca4dd349: Verifying Checksum 2024-05-19T18:14:08.3491707Z 30bfca4dd349: Download complete 2024-05-19T18:14:08.3762860Z 968831e596a6: Verifying Checksum 2024-05-19T18:14:08.3763511Z 968831e596a6: Download complete 2024-05-19T18:14:08.4531110Z 9ee6bdb31195: Download complete 2024-05-19T18:14:08.5381874Z 8af83a173aa3: Verifying Checksum 2024-05-19T18:14:08.5382524Z 8af83a173aa3: Download complete 2024-05-19T18:14:08.7313162Z 63e9bbe32327: Pull complete 2024-05-19T18:14:08.9777041Z cfb3d849840e: Pull complete 2024-05-19T18:14:09.8781526Z 968831e596a6: Pull complete 2024-05-19T18:14:09.9729240Z ea310eb267ca: Pull complete 2024-05-19T18:14:10.0811815Z 3af11d09e9cd: Pull complete 2024-05-19T18:14:11.0565810Z f8281676bd20: Verifying Checksum 2024-05-19T18:14:11.0566389Z f8281676bd20: Download complete 2024-05-19T18:14:11.1516626Z 748e3cfdd8d6: Verifying Checksum 2024-05-19T18:14:11.1517151Z 748e3cfdd8d6: Download complete 2024-05-19T18:14:11.2492065Z 5fa8d379f04e: Download complete 2024-05-19T18:14:11.3276298Z a95b9f41fb5b: Verifying Checksum 2024-05-19T18:14:11.3276939Z a95b9f41fb5b: Download complete 2024-05-19T18:14:12.3077581Z 8b6d0b819d19: Verifying Checksum 2024-05-19T18:14:12.3078236Z 8b6d0b819d19: Download complete 2024-05-19T18:14:12.3628561Z c61996e73399: Verifying Checksum 2024-05-19T18:14:12.3629085Z c61996e73399: Download complete 2024-05-19T18:14:12.4394711Z 9e69dcd7661c: Verifying Checksum 2024-05-19T18:14:12.4395262Z 9e69dcd7661c: Download complete 2024-05-19T18:14:12.5186273Z cfcaf2eb8249: Verifying Checksum 2024-05-19T18:14:12.5186819Z cfcaf2eb8249: Download complete 2024-05-19T18:14:21.8422864Z ebfec18059b9: Download complete 2024-05-19T18:14:21.9501157Z 07fa87a43fb8: Download complete 2024-05-19T18:14:22.0490629Z f16e94b04801: Download complete 2024-05-19T18:14:22.1378632Z c857065196d6: Verifying Checksum 2024-05-19T18:14:22.1379332Z c857065196d6: Download complete 2024-05-19T18:14:23.2585669Z 9d5c38cad0c9: Verifying Checksum 2024-05-19T18:14:23.2586565Z 9d5c38cad0c9: Download complete 2024-05-19T18:14:23.3310881Z 925ac0c7ff93: Verifying Checksum 2024-05-19T18:14:23.3311575Z 925ac0c7ff93: Download complete 2024-05-19T18:14:24.6077460Z cfbf8bf4873d: Verifying Checksum 2024-05-19T18:14:24.6077955Z cfbf8bf4873d: Download complete 2024-05-19T18:14:24.6847520Z 01914a189c0b: Verifying Checksum 2024-05-19T18:14:24.6848098Z 01914a189c0b: Download complete 2024-05-19T18:14:24.7856999Z 3c2451fbe84c: Verifying Checksum 2024-05-19T18:14:24.7857723Z 3c2451fbe84c: Download complete 2024-05-19T18:14:24.8887937Z aea8a2a73b72: Download complete 2024-05-19T18:14:24.9791070Z c26f348bfda1: Verifying Checksum 2024-05-19T18:14:24.9791655Z c26f348bfda1: Download complete 2024-05-19T18:14:25.0603735Z 347c41648f51: Verifying Checksum 2024-05-19T18:14:25.0604309Z 347c41648f51: Download complete 2024-05-19T18:14:29.5582371Z 00953247955f: Verifying Checksum 2024-05-19T18:14:29.5582935Z 00953247955f: Download complete 2024-05-19T18:14:29.6631138Z df27b4a5ce7c: Verifying Checksum 2024-05-19T18:14:29.6631643Z df27b4a5ce7c: Download complete 2024-05-19T18:14:29.7406323Z acb7fcd74bf2: Verifying Checksum 2024-05-19T18:14:29.7406900Z acb7fcd74bf2: Download complete 2024-05-19T18:14:30.1571523Z 5bd6ac51ef5d: Verifying Checksum 2024-05-19T18:14:30.1571964Z 5bd6ac51ef5d: Download complete 2024-05-19T18:14:30.2420407Z 1795bce633b0: Verifying Checksum 2024-05-19T18:14:30.2421057Z 1795bce633b0: Download complete 2024-05-19T18:14:30.3117029Z e41de6016f3c: Verifying Checksum 2024-05-19T18:14:30.3117603Z e41de6016f3c: Download complete 2024-05-19T18:14:30.5757398Z 458f5f2af8cf: Verifying Checksum 2024-05-19T18:14:30.5758070Z 458f5f2af8cf: Download complete 2024-05-19T18:14:30.6660148Z ef99a2a9cffb: Download complete 2024-05-19T18:14:30.7576884Z 1ba8d08d5876: Verifying Checksum 2024-05-19T18:14:30.7577391Z 1ba8d08d5876: Download complete 2024-05-19T18:14:30.8400806Z 0c67ae0247cf: Download complete 2024-05-19T18:14:33.9460158Z ebfec18059b9: Pull complete 2024-05-19T18:14:34.1720873Z 533b4aebf169: Pull complete 2024-05-19T18:14:34.4169136Z 9dd75d06a091: Pull complete 2024-05-19T18:14:34.4874439Z 1b57ce94cad9: Verifying Checksum 2024-05-19T18:14:34.4875104Z 1b57ce94cad9: Download complete 2024-05-19T18:14:34.5686463Z a6b88127ee12: Verifying Checksum 2024-05-19T18:14:34.5687028Z a6b88127ee12: Download complete 2024-05-19T18:14:34.6468612Z 7b0cf0b8be50: Verifying Checksum 2024-05-19T18:14:34.6469150Z 7b0cf0b8be50: Download complete 2024-05-19T18:14:34.6528273Z 30bfca4dd349: Pull complete 2024-05-19T18:14:35.1364340Z 044061f29c13: Verifying Checksum 2024-05-19T18:14:35.1365080Z 044061f29c13: Download complete 2024-05-19T18:14:35.2389778Z f21869a483d4: Download complete 2024-05-19T18:14:35.3203368Z a155d9e6e7e7: Verifying Checksum 2024-05-19T18:14:35.3203955Z a155d9e6e7e7: Download complete 2024-05-19T18:14:35.3953163Z 6028f518cba8: Verifying Checksum 2024-05-19T18:14:35.3953659Z 6028f518cba8: Download complete 2024-05-19T18:14:35.5266097Z 9f0e16ae766e: Download complete 2024-05-19T18:14:40.6967390Z 7cb10a6ae8df: Verifying Checksum 2024-05-19T18:14:40.6967970Z 7cb10a6ae8df: Download complete 2024-05-19T18:14:40.7914787Z a78a7e442565: Verifying Checksum 2024-05-19T18:14:40.7915528Z a78a7e442565: Download complete 2024-05-19T18:14:40.8750656Z df585564bf4c: Verifying Checksum 2024-05-19T18:14:40.8751244Z df585564bf4c: Download complete 2024-05-19T18:14:41.0322071Z f125619c4712: Verifying Checksum 2024-05-19T18:14:41.0322739Z f125619c4712: Download complete 2024-05-19T18:14:41.1092302Z 4cc671ff8870: Verifying Checksum 2024-05-19T18:14:41.1092828Z 4cc671ff8870: Download complete 2024-05-19T18:14:41.1821848Z e1d3209e04ae: Download complete 2024-05-19T18:14:41.2866042Z 322e57f70842: Download complete 2024-05-19T18:14:41.3743205Z d58f09a95d3e: Download complete 2024-05-19T18:14:41.5098324Z f611e28aa59a: Verifying Checksum 2024-05-19T18:14:41.5098822Z f611e28aa59a: Download complete 2024-05-19T18:14:41.5983962Z ff67e6bd8e23: Verifying Checksum 2024-05-19T18:14:41.5984459Z ff67e6bd8e23: Download complete 2024-05-19T18:14:42.1967550Z d2b06b7f79e8: Verifying Checksum 2024-05-19T18:14:42.1968439Z d2b06b7f79e8: Download complete 2024-05-19T18:14:42.2690008Z c27bc13ce9d4: Download complete 2024-05-19T18:15:08.3711321Z 26d45b11322a: Verifying Checksum 2024-05-19T18:15:08.3711789Z 26d45b11322a: Download complete 2024-05-19T18:15:08.4473041Z 400e9c337687: Download complete 2024-05-19T18:15:08.5444265Z 914d4657be1b: Download complete 2024-05-19T18:15:10.0080891Z ecc1d0285d28: Verifying Checksum 2024-05-19T18:15:10.0081340Z ecc1d0285d28: Download complete 2024-05-19T18:15:10.0981098Z b882dcc19062: Verifying Checksum 2024-05-19T18:15:10.0982326Z b882dcc19062: Download complete 2024-05-19T18:15:13.4050696Z 8680ed64cc19: Verifying Checksum 2024-05-19T18:15:13.4051565Z 8680ed64cc19: Download complete 2024-05-19T18:15:37.2557979Z 3ebf7de5956e: Download complete 2024-05-19T18:15:53.5114731Z 1b57ce94cad9: Pull complete 2024-05-19T18:15:53.7437686Z 9ee6bdb31195: Pull complete 2024-05-19T18:15:53.9769555Z 8af83a173aa3: Pull complete 2024-05-19T18:15:59.0197970Z f8281676bd20: Pull complete 2024-05-19T18:15:59.2339852Z 748e3cfdd8d6: Pull complete 2024-05-19T18:15:59.4811530Z 5fa8d379f04e: Pull complete 2024-05-19T18:15:59.7155040Z a95b9f41fb5b: Pull complete 2024-05-19T18:16:01.5370293Z 8b6d0b819d19: Pull complete 2024-05-19T18:16:01.7502354Z c61996e73399: Pull complete 2024-05-19T18:16:01.9543902Z 9e69dcd7661c: Pull complete 2024-05-19T18:16:02.1875415Z cfcaf2eb8249: Pull complete 2024-05-19T18:16:39.6491572Z 7cb10a6ae8df: Pull complete 2024-05-19T18:16:39.7853262Z 07fa87a43fb8: Pull complete 2024-05-19T18:16:39.9602904Z f16e94b04801: Pull complete 2024-05-19T18:16:40.1835688Z c857065196d6: Pull complete 2024-05-19T18:16:40.3511777Z 9d5c38cad0c9: Pull complete 2024-05-19T18:16:40.4666900Z 925ac0c7ff93: Pull complete 2024-05-19T18:16:43.5298591Z cfbf8bf4873d: Pull complete 2024-05-19T18:16:43.7261649Z 01914a189c0b: Pull complete 2024-05-19T18:16:43.9047806Z 3c2451fbe84c: Pull complete 2024-05-19T18:16:44.0364090Z aea8a2a73b72: Pull complete 2024-05-19T18:16:44.1808244Z c26f348bfda1: Pull complete 2024-05-19T18:16:44.3562756Z 347c41648f51: Pull complete 2024-05-19T18:16:50.1659207Z 00953247955f: Pull complete 2024-05-19T18:16:50.3219888Z df27b4a5ce7c: Pull complete 2024-05-19T18:16:50.5088895Z acb7fcd74bf2: Pull complete 2024-05-19T18:16:51.3005463Z 5bd6ac51ef5d: Pull complete 2024-05-19T18:16:51.5035871Z 1795bce633b0: Pull complete 2024-05-19T18:16:51.7150434Z e41de6016f3c: Pull complete 2024-05-19T18:16:52.1793933Z 458f5f2af8cf: Pull complete 2024-05-19T18:16:52.4000091Z ef99a2a9cffb: Pull complete 2024-05-19T18:16:52.8651115Z 1ba8d08d5876: Pull complete 2024-05-19T18:16:53.0975752Z 0c67ae0247cf: Pull complete 2024-05-19T18:17:29.7528022Z 3ebf7de5956e: Pull complete 2024-05-19T18:17:29.9912937Z a6b88127ee12: Pull complete 2024-05-19T18:17:30.2151115Z 7b0cf0b8be50: Pull complete 2024-05-19T18:17:31.4780566Z 044061f29c13: Pull complete 2024-05-19T18:17:31.6562775Z f21869a483d4: Pull complete 2024-05-19T18:17:31.7845644Z a155d9e6e7e7: Pull complete 2024-05-19T18:17:32.1351072Z 6028f518cba8: Pull complete 2024-05-19T18:17:32.3585433Z 9f0e16ae766e: Pull complete 2024-05-19T18:18:02.1568483Z 26d45b11322a: Pull complete 2024-05-19T18:18:02.3327204Z a78a7e442565: Pull complete 2024-05-19T18:18:02.5488888Z df585564bf4c: Pull complete 2024-05-19T18:18:02.8942081Z f125619c4712: Pull complete 2024-05-19T18:18:03.1207042Z 4cc671ff8870: Pull complete 2024-05-19T18:18:03.3458042Z e1d3209e04ae: Pull complete 2024-05-19T18:18:03.5621471Z 322e57f70842: Pull complete 2024-05-19T18:18:03.7891292Z d58f09a95d3e: Pull complete 2024-05-19T18:18:04.4353777Z f611e28aa59a: Pull complete 2024-05-19T18:18:04.6493736Z ff67e6bd8e23: Pull complete 2024-05-19T18:18:06.3260305Z d2b06b7f79e8: Pull complete 2024-05-19T18:18:06.5296203Z c27bc13ce9d4: Pull complete 2024-05-19T18:18:20.7252528Z 8680ed64cc19: Pull complete 2024-05-19T18:18:20.9485540Z 400e9c337687: Pull complete 2024-05-19T18:18:21.1923038Z 914d4657be1b: Pull complete 2024-05-19T18:18:22.0139001Z ecc1d0285d28: Pull complete 2024-05-19T18:18:22.2405230Z b882dcc19062: Pull complete 2024-05-19T18:18:22.3483359Z Digest: sha256:66f062925f91f9b9204b1dc6d68caf8c89fa82fd4b12cb5f24c19aa4b1c87506 2024-05-19T18:18:22.3900054Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:18:22.4128456Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:18:22.4171707Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-05-19T18:18:22.4172649Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-05-19T18:18:22.4181462Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:18:22.4181965Z env: 2024-05-19T18:18:22.4182255Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:22.4182600Z ##[endgroup] 2024-05-19T18:18:22.4347133Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-05-19T18:18:22.4347688Z with: 2024-05-19T18:18:22.4347963Z driver-version: 550.54.15 2024-05-19T18:18:22.4348313Z env: 2024-05-19T18:18:22.4348595Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:22.4348932Z ##[endgroup] 2024-05-19T18:18:22.4484079Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-05-19T18:18:22.4484710Z with: 2024-05-19T18:18:22.4484992Z timeout_minutes: 10 2024-05-19T18:18:22.4485313Z max_attempts: 3 2024-05-19T18:18:22.4514813Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-05-19T18:18:22.4544828Z retry_wait_seconds: 10 2024-05-19T18:18:22.4545204Z polling_interval_seconds: 1 2024-05-19T18:18:22.4545574Z warning_on_retry: true 2024-05-19T18:18:22.4545925Z continue_on_error: false 2024-05-19T18:18:22.4546265Z env: 2024-05-19T18:18:22.4546543Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:18:22.4546896Z DRIVER_VERSION: 550.54.15 2024-05-19T18:18:22.4547252Z ##[endgroup] 2024-05-19T18:18:22.5066079Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-05-19T18:18:22.5067068Z + pre_install_nvidia_driver_amzn2 2024-05-19T18:18:22.5067619Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-05-19T18:18:22.8114448Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:18:22.8503721Z No Match for argument: nvidia-driver-latest-dkms 2024-05-19T18:18:22.8748138Z No Packages marked for removal 2024-05-19T18:18:22.8870518Z + install_nvidia_driver_common 2024-05-19T18:18:22.8872918Z + echo 'Before installing NVIDIA driver' 2024-05-19T18:18:22.8873445Z + lspci 2024-05-19T18:18:22.8873774Z Before installing NVIDIA driver 2024-05-19T18:18:22.9697198Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-05-19T18:18:22.9698191Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-05-19T18:18:22.9699166Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-05-19T18:18:22.9699941Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-05-19T18:18:22.9700718Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-05-19T18:18:22.9703093Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-05-19T18:18:22.9703816Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-05-19T18:18:22.9704617Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-05-19T18:18:22.9705192Z + lsmod 2024-05-19T18:18:22.9709870Z Module Size Used by 2024-05-19T18:18:22.9710348Z backlight 16384 0 2024-05-19T18:18:22.9710736Z xt_conntrack 16384 1 2024-05-19T18:18:22.9711134Z ipt_MASQUERADE 16384 1 2024-05-19T18:18:22.9711573Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-05-19T18:18:22.9712046Z nf_conntrack_netlink 49152 0 2024-05-19T18:18:22.9712492Z nfnetlink 16384 2 nf_conntrack_netlink 2024-05-19T18:18:22.9712952Z xfrm_user 45056 1 2024-05-19T18:18:22.9713364Z xfrm_algo 16384 1 xfrm_user 2024-05-19T18:18:22.9713977Z iptable_nat 16384 1 2024-05-19T18:18:22.9714376Z nf_conntrack_ipv4 16384 3 2024-05-19T18:18:22.9714925Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-05-19T18:18:22.9715436Z nf_nat_ipv4 16384 1 iptable_nat 2024-05-19T18:18:22.9716102Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-05-19T18:18:22.9717032Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-05-19T18:18:22.9718031Z xt_addrtype 16384 2 2024-05-19T18:18:22.9718533Z iptable_filter 16384 1 2024-05-19T18:18:22.9719042Z br_netfilter 24576 0 2024-05-19T18:18:22.9719460Z bridge 172032 1 br_netfilter 2024-05-19T18:18:22.9719909Z stp 16384 1 bridge 2024-05-19T18:18:22.9720325Z llc 16384 2 bridge,stp 2024-05-19T18:18:22.9720959Z overlay 86016 0 2024-05-19T18:18:22.9721343Z sunrpc 393216 1 2024-05-19T18:18:22.9721717Z dm_mirror 28672 0 2024-05-19T18:18:22.9722114Z dm_region_hash 20480 1 dm_mirror 2024-05-19T18:18:22.9722595Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-05-19T18:18:22.9723106Z dm_mod 143360 2 dm_log,dm_mirror 2024-05-19T18:18:22.9723556Z dax 69632 1 dm_mod 2024-05-19T18:18:22.9723951Z crc32_pclmul 16384 0 2024-05-19T18:18:22.9724330Z ghash_clmulni_intel 16384 0 2024-05-19T18:18:22.9724840Z pcbc 16384 0 2024-05-19T18:18:22.9725217Z aesni_intel 188416 0 2024-05-19T18:18:22.9725608Z aes_x86_64 20480 1 aesni_intel 2024-05-19T18:18:22.9726059Z crypto_simd 16384 1 aesni_intel 2024-05-19T18:18:22.9726500Z mousedev 24576 0 2024-05-19T18:18:22.9726909Z glue_helper 16384 1 aesni_intel 2024-05-19T18:18:22.9727329Z evdev 20480 3 2024-05-19T18:18:22.9727855Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-05-19T18:18:22.9728400Z psmouse 32768 0 2024-05-19T18:18:22.9728771Z button 16384 0 2024-05-19T18:18:22.9729130Z ena 139264 0 2024-05-19T18:18:22.9729501Z ptp 20480 1 ena 2024-05-19T18:18:22.9729947Z pps_core 20480 1 ptp 2024-05-19T18:18:22.9730338Z crc32c_intel 24576 0 2024-05-19T18:18:22.9730792Z autofs4 49152 2 2024-05-19T18:18:22.9731156Z + modinfo nvidia 2024-05-19T18:18:22.9731919Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-05-19T18:18:22.9732595Z firmware: nvidia/535.54.03/gsp_tu10x.bin 2024-05-19T18:18:22.9733070Z firmware: nvidia/535.54.03/gsp_ga10x.bin 2024-05-19T18:18:22.9733554Z alias: char-major-195-* 2024-05-19T18:18:22.9733937Z version: 535.54.03 2024-05-19T18:18:22.9734276Z supported: external 2024-05-19T18:18:22.9734619Z license: NVIDIA 2024-05-19T18:18:22.9734978Z srcversion: EA9C7EF32617E104C8240C4 2024-05-19T18:18:22.9735439Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-05-19T18:18:22.9735987Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-05-19T18:18:22.9736483Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-05-19T18:18:22.9736960Z depends: i2c-core,drm 2024-05-19T18:18:22.9737315Z retpoline: Y 2024-05-19T18:18:22.9737828Z name: nvidia 2024-05-19T18:18:22.9738385Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-05-19T18:18:22.9739035Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-05-19T18:18:22.9739667Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-05-19T18:18:22.9740254Z parm: NVreg_ResmanDebugLevel:int 2024-05-19T18:18:22.9740693Z parm: NVreg_RmLogonRC:int 2024-05-19T18:18:22.9741128Z parm: NVreg_ModifyDeviceFiles:int 2024-05-19T18:18:22.9741579Z parm: NVreg_DeviceFileUID:int 2024-05-19T18:18:22.9742135Z parm: NVreg_DeviceFileGID:int 2024-05-19T18:18:22.9742582Z parm: NVreg_DeviceFileMode:int 2024-05-19T18:18:22.9743117Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-05-19T18:18:22.9743667Z parm: NVreg_UsePageAttributeTable:int 2024-05-19T18:18:22.9744154Z parm: NVreg_EnablePCIeGen3:int 2024-05-19T18:18:22.9744591Z parm: NVreg_EnableMSI:int 2024-05-19T18:18:22.9745000Z parm: NVreg_TCEBypassMode:int 2024-05-19T18:18:22.9745458Z parm: NVreg_EnableStreamMemOPs:int 2024-05-19T18:18:22.9745990Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-05-19T18:18:22.9746571Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-05-19T18:18:22.9747121Z parm: NVreg_EnableS0ixPowerManagement:int 2024-05-19T18:18:22.9747732Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-05-19T18:18:22.9748425Z parm: NVreg_DynamicPowerManagement:int 2024-05-19T18:18:22.9749048Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-05-19T18:18:22.9749630Z parm: NVreg_EnableGpuFirmware:int 2024-05-19T18:18:22.9750121Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-05-19T18:18:22.9750650Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-05-19T18:18:22.9751203Z parm: NVreg_EnableUserNUMAManagement:int 2024-05-19T18:18:22.9751693Z parm: NVreg_MemoryPoolSize:int 2024-05-19T18:18:22.9752159Z parm: NVreg_KMallocHeapMaxSize:int 2024-05-19T18:18:22.9752642Z parm: NVreg_VMallocHeapMaxSize:int 2024-05-19T18:18:22.9753117Z parm: NVreg_IgnoreMMIOCheck:int 2024-05-19T18:18:22.9753556Z parm: NVreg_NvLinkDisable:int 2024-05-19T18:18:22.9754065Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-05-19T18:18:22.9754603Z parm: NVreg_RegisterPCIDriver:int 2024-05-19T18:18:22.9755093Z parm: NVreg_EnableResizableBar:int 2024-05-19T18:18:22.9755582Z parm: NVreg_EnableDbgBreakpoint:int 2024-05-19T18:18:22.9756057Z parm: NVreg_RegistryDwords:charp 2024-05-19T18:18:22.9756552Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-05-19T18:18:22.9757031Z parm: NVreg_RmMsg:charp 2024-05-19T18:18:22.9757437Z parm: NVreg_GpuBlacklist:charp 2024-05-19T18:18:22.9757912Z parm: NVreg_TemporaryFilePath:charp 2024-05-19T18:18:22.9758388Z parm: NVreg_ExcludedGpus:charp 2024-05-19T18:18:22.9758837Z parm: NVreg_DmaRemapPeerMmio:int 2024-05-19T18:18:22.9759318Z parm: NVreg_RmNvlinkBandwidth:charp 2024-05-19T18:18:22.9759781Z parm: rm_firmware_active:charp 2024-05-19T18:18:22.9760197Z + HAS_NVIDIA_DRIVER=0 2024-05-19T18:18:22.9760577Z ++ command -v nvidia-smi 2024-05-19T18:18:22.9760992Z + '[' -x /usr/bin/nvidia-smi ']' 2024-05-19T18:18:22.9761372Z + set +e 2024-05-19T18:18:22.9761874Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-05-19T18:18:25.5364866Z + INSTALLED_DRIVER_VERSION=535.54.03 2024-05-19T18:18:25.5365381Z + NVIDIA_SMI_STATUS=0 2024-05-19T18:18:25.5365889Z + '[' 0 -ne 0 ']' 2024-05-19T18:18:25.5366255Z + '[' 535.54.03 '!=' 550.54.15 ']' 2024-05-19T18:18:25.5367031Z + echo 'NVIDIA driver (535.54.03) has been installed, but we expect to have 550.54.15 instead. Continuing' 2024-05-19T18:18:25.5367745Z + set -e 2024-05-19T18:18:25.5368039Z + '[' 0 -eq 0 ']' 2024-05-19T18:18:25.5368390Z + '[' amzn2 '!=' ubuntu20.04 ']' 2024-05-19T18:18:25.5368875Z + sudo yum groupinstall -y 'Development Tools' 2024-05-19T18:18:25.5369609Z NVIDIA driver (535.54.03) has been installed, but we expect to have 550.54.15 instead. Continuing 2024-05-19T18:18:25.8402837Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:18:26.9986363Z Maybe run: yum groups mark install (see man yum) 2024-05-19T18:18:26.9987506Z No packages in any requested group available to install or update 2024-05-19T18:18:27.0911730Z ++ uname -r 2024-05-19T18:18:27.0917995Z + sudo yum install -y 'kernel-devel-uname-r == 4.14.336-257.562.amzn2.x86_64' 2024-05-19T18:18:27.4197200Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:18:28.9097331Z Package kernel-devel-4.14.336-257.562.amzn2.x86_64 already installed and latest version 2024-05-19T18:18:28.9098218Z Nothing to do 2024-05-19T18:18:29.0636242Z + sudo modprobe backlight 2024-05-19T18:18:29.0909533Z + sudo curl -fsL -o /tmp/nvidia_driver https://s3.amazonaws.com/ossci-linux/nvidia_driver/NVIDIA-Linux-x86_64-550.54.15.run 2024-05-19T18:18:32.2099074Z + set +e 2024-05-19T18:18:32.2099703Z + sudo /bin/bash /tmp/nvidia_driver -s --no-drm 2024-05-19T18:18:33.1470439Z Verifying archive integrity... OK 2024-05-19T18:18:35.7555578Z Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 550.54.15........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ 2024-05-19T18:18:35.9709162Z 2024-05-19T18:18:35.9710900Z WARNING: The nvidia-drm module will not be installed. As a result, DRM-KMS will not function with this installation of the NVIDIA driver. 2024-05-19T18:18:35.9711704Z 2024-05-19T18:18:50.0720023Z 2024-05-19T18:18:50.0723862Z WARNING: nvidia-installer was forced to guess the X library path '/usr/lib64' and X module path '/usr/lib64/xorg/modules'; these paths were not queryable from the system. If X fails to find the NVIDIA X driver module, please install the `pkg-config` utility and the X.Org SDK/development package for your distribution and reinstall the driver. 2024-05-19T18:18:50.0725795Z 2024-05-19T18:18:50.0728239Z 2024-05-19T18:18:50.0730375Z WARNING: This NVIDIA driver package includes Vulkan components, but no Vulkan ICD loader was detected on this system. The NVIDIA Vulkan ICD will not function without the loader. Most distributions package the Vulkan loader; try installing the "vulkan-loader", "vulkan-icd-loader", or "libvulkan1" package. 2024-05-19T18:18:50.0731979Z 2024-05-19T18:18:58.3796634Z + NVIDIA_INSTALLATION_STATUS=0 2024-05-19T18:18:58.3797255Z + RESET_GPU=0 2024-05-19T18:18:58.3797861Z + '[' 0 -ne 0 ']' 2024-05-19T18:18:58.3798236Z ++ command -v nvidia-smi 2024-05-19T18:18:58.3799638Z + '[' -x /usr/bin/nvidia-smi ']' 2024-05-19T18:18:58.3801582Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-05-19T18:19:00.5408726Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-05-19T18:19:00.5409265Z + NVIDIA_SMI_STATUS=0 2024-05-19T18:19:00.5410526Z + '[' 0 -ne 0 ']' 2024-05-19T18:19:00.5410985Z + '[' 0 -eq 1 ']' 2024-05-19T18:19:00.5411359Z + sudo rm -fv /tmp/nvidia_driver 2024-05-19T18:19:00.5772989Z removed '/tmp/nvidia_driver' 2024-05-19T18:19:00.5788372Z + set -e 2024-05-19T18:19:00.5790320Z + post_install_nvidia_driver_common 2024-05-19T18:19:00.5792705Z + sudo modprobe nvidia 2024-05-19T18:19:00.6186664Z + echo 'After installing NVIDIA driver' 2024-05-19T18:19:00.6187254Z + lspci 2024-05-19T18:19:00.6187645Z After installing NVIDIA driver 2024-05-19T18:19:00.6271110Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-05-19T18:19:00.6271988Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-05-19T18:19:00.6272909Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-05-19T18:19:00.6273720Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-05-19T18:19:00.6274636Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-05-19T18:19:00.6275644Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-05-19T18:19:00.6276373Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-05-19T18:19:00.6277210Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-05-19T18:19:00.6277803Z + lsmod 2024-05-19T18:19:00.6284796Z Module Size Used by 2024-05-19T18:19:00.6285504Z nvidia_uvm 4599808 0 2024-05-19T18:19:00.6286231Z nvidia 53989376 1 nvidia_uvm 2024-05-19T18:19:00.6286815Z drm 421888 1 nvidia 2024-05-19T18:19:00.6287366Z i2c_core 77824 2 nvidia,drm 2024-05-19T18:19:00.6287912Z backlight 16384 0 2024-05-19T18:19:00.6288415Z xt_conntrack 16384 1 2024-05-19T18:19:00.6288899Z ipt_MASQUERADE 16384 1 2024-05-19T18:19:00.6290891Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-05-19T18:19:00.6291504Z nf_conntrack_netlink 49152 0 2024-05-19T18:19:00.6291971Z nfnetlink 16384 2 nf_conntrack_netlink 2024-05-19T18:19:00.6292434Z xfrm_user 45056 1 2024-05-19T18:19:00.6292846Z xfrm_algo 16384 1 xfrm_user 2024-05-19T18:19:00.6293255Z iptable_nat 16384 1 2024-05-19T18:19:00.6293636Z nf_conntrack_ipv4 16384 3 2024-05-19T18:19:00.6294065Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-05-19T18:19:00.6294543Z nf_nat_ipv4 16384 1 iptable_nat 2024-05-19T18:19:00.6295068Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-05-19T18:19:00.6295998Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-05-19T18:19:00.6296956Z xt_addrtype 16384 2 2024-05-19T18:19:00.6297448Z iptable_filter 16384 1 2024-05-19T18:19:00.6298148Z br_netfilter 24576 0 2024-05-19T18:19:00.6298690Z bridge 172032 1 br_netfilter 2024-05-19T18:19:00.6299142Z stp 16384 1 bridge 2024-05-19T18:19:00.6299549Z llc 16384 2 bridge,stp 2024-05-19T18:19:00.6299964Z overlay 86016 0 2024-05-19T18:19:00.6300440Z sunrpc 393216 1 2024-05-19T18:19:00.6300808Z dm_mirror 28672 0 2024-05-19T18:19:00.6301192Z dm_region_hash 20480 1 dm_mirror 2024-05-19T18:19:00.6301681Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-05-19T18:19:00.6302185Z dm_mod 143360 2 dm_log,dm_mirror 2024-05-19T18:19:00.6302642Z dax 69632 1 dm_mod 2024-05-19T18:19:00.6303034Z crc32_pclmul 16384 0 2024-05-19T18:19:00.6303408Z ghash_clmulni_intel 16384 0 2024-05-19T18:19:00.6303788Z pcbc 16384 0 2024-05-19T18:19:00.6304153Z aesni_intel 188416 0 2024-05-19T18:19:00.6304537Z aes_x86_64 20480 1 aesni_intel 2024-05-19T18:19:00.6304984Z crypto_simd 16384 1 aesni_intel 2024-05-19T18:19:00.6305431Z mousedev 24576 0 2024-05-19T18:19:00.6305865Z glue_helper 16384 1 aesni_intel 2024-05-19T18:19:00.6306320Z evdev 20480 3 2024-05-19T18:19:00.6306945Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-05-19T18:19:00.6307526Z psmouse 32768 0 2024-05-19T18:19:00.6307905Z button 16384 0 2024-05-19T18:19:00.6308289Z ena 139264 0 2024-05-19T18:19:00.6308677Z ptp 20480 1 ena 2024-05-19T18:19:00.6309088Z pps_core 20480 1 ptp 2024-05-19T18:19:00.6309596Z crc32c_intel 24576 0 2024-05-19T18:19:00.6309973Z autofs4 49152 2 2024-05-19T18:19:00.6310334Z + modinfo nvidia 2024-05-19T18:19:00.6311009Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-05-19T18:19:00.6311681Z alias: char-major-195-* 2024-05-19T18:19:00.6312063Z version: 550.54.15 2024-05-19T18:19:00.6312412Z supported: external 2024-05-19T18:19:00.6312747Z license: NVIDIA 2024-05-19T18:19:00.6313118Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-05-19T18:19:00.6313594Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-05-19T18:19:00.6314045Z srcversion: 833721318DA517F0C2FEC97 2024-05-19T18:19:00.6314504Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-05-19T18:19:00.6314988Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-05-19T18:19:00.6315475Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-05-19T18:19:00.6316052Z depends: i2c-core,drm 2024-05-19T18:19:00.6316430Z retpoline: Y 2024-05-19T18:19:00.6316732Z name: nvidia 2024-05-19T18:19:00.6317286Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-05-19T18:19:00.6317935Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-05-19T18:19:00.6318567Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-05-19T18:19:00.6319154Z parm: NVreg_ResmanDebugLevel:int 2024-05-19T18:19:00.6319592Z parm: NVreg_RmLogonRC:int 2024-05-19T18:19:00.6320021Z parm: NVreg_ModifyDeviceFiles:int 2024-05-19T18:19:00.6320475Z parm: NVreg_DeviceFileUID:int 2024-05-19T18:19:00.6320910Z parm: NVreg_DeviceFileGID:int 2024-05-19T18:19:00.6321353Z parm: NVreg_DeviceFileMode:int 2024-05-19T18:19:00.6321879Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-05-19T18:19:00.6322430Z parm: NVreg_UsePageAttributeTable:int 2024-05-19T18:19:00.6322928Z parm: NVreg_EnablePCIeGen3:int 2024-05-19T18:19:00.6323365Z parm: NVreg_EnableMSI:int 2024-05-19T18:19:00.6323780Z parm: NVreg_TCEBypassMode:int 2024-05-19T18:19:00.6324233Z parm: NVreg_EnableStreamMemOPs:int 2024-05-19T18:19:00.6324760Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-05-19T18:19:00.6325576Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-05-19T18:19:00.6326139Z parm: NVreg_EnableS0ixPowerManagement:int 2024-05-19T18:19:00.6326736Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-05-19T18:19:00.6327328Z parm: NVreg_DynamicPowerManagement:int 2024-05-19T18:19:00.6327932Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-05-19T18:19:00.6328517Z parm: NVreg_EnableGpuFirmware:int 2024-05-19T18:19:00.6328997Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-05-19T18:19:00.6329528Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-05-19T18:19:00.6330082Z parm: NVreg_EnableUserNUMAManagement:int 2024-05-19T18:19:00.6330563Z parm: NVreg_MemoryPoolSize:int 2024-05-19T18:19:00.6331023Z parm: NVreg_KMallocHeapMaxSize:int 2024-05-19T18:19:00.6331511Z parm: NVreg_VMallocHeapMaxSize:int 2024-05-19T18:19:00.6331987Z parm: NVreg_IgnoreMMIOCheck:int 2024-05-19T18:19:00.6332423Z parm: NVreg_NvLinkDisable:int 2024-05-19T18:19:00.6332926Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-05-19T18:19:00.6333455Z parm: NVreg_RegisterPCIDriver:int 2024-05-19T18:19:00.6333931Z parm: NVreg_EnableResizableBar:int 2024-05-19T18:19:00.6334404Z parm: NVreg_EnableDbgBreakpoint:int 2024-05-19T18:19:00.6334904Z parm: NVreg_EnableNonblockingOpen:int 2024-05-19T18:19:00.6335399Z parm: NVreg_RegistryDwords:charp 2024-05-19T18:19:00.6335897Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-05-19T18:19:00.6336421Z parm: NVreg_RmMsg:charp 2024-05-19T18:19:00.6336928Z parm: NVreg_GpuBlacklist:charp 2024-05-19T18:19:00.6337400Z parm: NVreg_TemporaryFilePath:charp 2024-05-19T18:19:00.6338087Z parm: NVreg_ExcludedGpus:charp 2024-05-19T18:19:00.6338538Z parm: NVreg_DmaRemapPeerMmio:int 2024-05-19T18:19:00.6339012Z parm: NVreg_RmNvlinkBandwidth:charp 2024-05-19T18:19:00.6339487Z parm: NVreg_ImexChannelCount:int 2024-05-19T18:19:00.6339923Z parm: rm_firmware_active:charp 2024-05-19T18:19:00.6340321Z + set +e 2024-05-19T18:19:00.6340631Z + nvidia-smi 2024-05-19T18:19:02.1849484Z Sun May 19 18:19:02 2024 2024-05-19T18:19:02.1850375Z +-----------------------------------------------------------------------------------------+ 2024-05-19T18:19:02.1851205Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-05-19T18:19:02.1851973Z |-----------------------------------------+------------------------+----------------------+ 2024-05-19T18:19:02.1853094Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-05-19T18:19:02.1853963Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-05-19T18:19:02.1854650Z | | | MIG M. | 2024-05-19T18:19:02.1855203Z |=========================================+========================+======================| 2024-05-19T18:19:02.1951746Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-05-19T18:19:02.1952684Z | 0% 31C P0 58W / 300W | 0MiB / 23028MiB | 0% Default | 2024-05-19T18:19:02.1953355Z | | | N/A | 2024-05-19T18:19:02.1954231Z +-----------------------------------------+------------------------+----------------------+ 2024-05-19T18:19:02.1954827Z 2024-05-19T18:19:02.1955470Z +-----------------------------------------------------------------------------------------+ 2024-05-19T18:19:02.1956085Z | Processes: | 2024-05-19T18:19:02.1956973Z | GPU GI CI PID Type Process name GPU Memory | 2024-05-19T18:19:02.1957682Z | ID ID Usage | 2024-05-19T18:19:02.1958238Z |=========================================================================================| 2024-05-19T18:19:02.1958874Z | No running processes found | 2024-05-19T18:19:02.1959615Z +-----------------------------------------------------------------------------------------+ 2024-05-19T18:19:02.8107422Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-05-19T18:19:04.3598456Z NVIDIA A10G 2024-05-19T18:19:04.8038116Z + NVIDIA_SMI_STATUS=0 2024-05-19T18:19:04.8038728Z + '[' 0 -eq 0 ']' 2024-05-19T18:19:04.8039282Z + echo 'INFO: Ignoring allowed status 0' 2024-05-19T18:19:04.8039880Z + set -e 2024-05-19T18:19:04.8040224Z INFO: Ignoring allowed status 0 2024-05-19T18:19:04.8041862Z == Installing nvidia container toolkit for amzn2 == 2024-05-19T18:19:04.8044117Z + sudo yum install -y yum-utils 2024-05-19T18:19:05.1081619Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:19:06.6019408Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-05-19T18:19:06.6020094Z Nothing to do 2024-05-19T18:19:06.7543269Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-05-19T18:19:07.0668729Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:19:07.0941526Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-05-19T18:19:07.0943069Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-05-19T18:19:07.0944004Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-05-19T18:19:07.1059947Z + sudo yum install -y nvidia-docker2 2024-05-19T18:19:07.4167638Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:19:08.8904674Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-05-19T18:19:08.8905336Z Nothing to do 2024-05-19T18:19:09.0446309Z + sudo systemctl restart docker 2024-05-19T18:19:37.5763792Z Command completed after 1 attempt(s). 2024-05-19T18:19:37.5823892Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-05-19T18:19:37.5824652Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-05-19T18:19:37.5825334Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-05-19T18:19:37.5826110Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:19:37.5833957Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:19:37.5834444Z env: 2024-05-19T18:19:37.5834720Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:19:37.5835158Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:19:37.5835633Z ##[endgroup] 2024-05-19T18:19:37.9948758Z Defaulting to user installation because normal site-packages is not writeable 2024-05-19T18:19:38.4029696Z Collecting psutil==5.9.1 2024-05-19T18:19:38.4210333Z Downloading psutil-5.9.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-05-19T18:19:38.4671298Z Collecting nvidia-ml-py==11.525.84 2024-05-19T18:19:38.4706756Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-05-19T18:19:38.5116659Z Installing collected packages: psutil, nvidia-ml-py 2024-05-19T18:19:38.6661055Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-05-19T18:19:38.8544535Z Prepare all required actions 2024-05-19T18:19:38.8545023Z Getting action download info 2024-05-19T18:19:38.9540551Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-05-19T18:19:39.1263807Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-05-19T18:19:39.2353089Z ##[group]Run ./.github/actions/download-build-artifacts 2024-05-19T18:19:39.2353574Z with: 2024-05-19T18:19:39.2353900Z name: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:19:39.2354353Z s3-bucket: gha-artifacts 2024-05-19T18:19:39.2354695Z env: 2024-05-19T18:19:39.2354969Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:19:39.2355407Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:19:39.2355885Z ##[endgroup] 2024-05-19T18:19:39.2387726Z ##[group]Run seemethere/download-artifact-s3@v4 2024-05-19T18:19:39.2388169Z with: 2024-05-19T18:19:39.2388538Z name: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:19:39.2388996Z s3-bucket: gha-artifacts 2024-05-19T18:19:39.2389345Z region: us-east-1 2024-05-19T18:19:39.2389642Z env: 2024-05-19T18:19:39.2389915Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:19:39.2390359Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:19:39.2390833Z ##[endgroup] 2024-05-19T18:19:39.6642147Z (node:28920) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-05-19T18:19:39.6642896Z 2024-05-19T18:19:39.6643165Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-05-19T18:19:39.6643884Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-05-19T18:19:39.6644731Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-05-19T18:19:39.7387039Z Found 1 objects with prefix pytorch/pytorch/9148947323/linux-focal-cuda12.4-py3.10-gcc9-sm86/ 2024-05-19T18:19:39.7388131Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-05-19T18:19:47.2164582Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-05-19T18:19:47.2170815Z Artifact download has finished successfully 2024-05-19T18:19:47.2342203Z ##[group]Run unzip -o artifacts.zip 2024-05-19T18:19:47.2342637Z unzip -o artifacts.zip 2024-05-19T18:19:47.2350184Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:19:47.2350688Z env: 2024-05-19T18:19:47.2350964Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:19:47.2351418Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:19:47.2351895Z ##[endgroup] 2024-05-19T18:19:47.2401968Z Archive: artifacts.zip 2024-05-19T18:19:47.2403639Z creating: dist/ 2024-05-19T18:19:49.2139921Z inflating: dist/torch-2.4.0a0+gitc8c7ddf-cp310-cp310-linux_x86_64.whl 2024-05-19T18:19:49.2140682Z creating: build/custom_test_artifacts/ 2024-05-19T18:19:49.2141261Z creating: build/custom_test_artifacts/custom-op-build/ 2024-05-19T18:19:49.2142208Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-05-19T18:19:49.2143035Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-05-19T18:19:49.2147241Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-05-19T18:19:49.2148263Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-05-19T18:19:49.2149200Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-05-19T18:19:49.2150178Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-05-19T18:19:49.2151130Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-05-19T18:19:49.2152210Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-05-19T18:19:49.2153419Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-05-19T18:19:49.2154419Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-05-19T18:19:49.2155395Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-05-19T18:19:49.2156535Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-05-19T18:19:49.2157680Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-05-19T18:19:49.2158793Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-05-19T18:19:49.2160060Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-05-19T18:19:49.2161213Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-05-19T18:19:49.2162373Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-05-19T18:19:49.2163387Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-05-19T18:19:49.2164378Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-05-19T18:19:49.2199766Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-05-19T18:19:49.2239083Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-05-19T18:19:49.2240629Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-05-19T18:19:49.2284269Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-05-19T18:19:49.2285791Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-05-19T18:19:49.2287220Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-05-19T18:19:49.2288685Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-05-19T18:19:49.2290088Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-05-19T18:19:49.2291473Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-05-19T18:19:49.2292838Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-05-19T18:19:49.2294211Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-05-19T18:19:49.2295550Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-05-19T18:19:49.2296936Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-05-19T18:19:49.2298507Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-05-19T18:19:49.2299698Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-05-19T18:19:49.2300906Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-05-19T18:19:49.2302074Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-05-19T18:19:49.2303271Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-05-19T18:19:49.2365620Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-05-19T18:19:49.2435337Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-05-19T18:19:49.2436529Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-05-19T18:19:49.2437513Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-05-19T18:19:49.2438373Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-05-19T18:19:49.2439268Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-05-19T18:19:49.2440178Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-05-19T18:19:49.2441199Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-05-19T18:19:49.2442344Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-05-19T18:19:49.2443442Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-05-19T18:19:49.2444602Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-05-19T18:19:49.2445655Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-05-19T18:19:49.2446727Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-05-19T18:19:49.2447793Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-05-19T18:19:49.2448860Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-05-19T18:19:49.2449895Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-05-19T18:19:49.2460490Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-05-19T18:19:49.2588542Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-05-19T18:19:49.2589530Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-05-19T18:19:49.2590608Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-05-19T18:19:49.2591784Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-05-19T18:19:49.2592934Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-05-19T18:19:49.2594004Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-05-19T18:19:49.2595116Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-05-19T18:19:49.2596238Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-05-19T18:19:49.2597470Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-05-19T18:19:49.2598587Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-05-19T18:19:49.2599689Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-05-19T18:19:49.2613013Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-05-19T18:19:49.2692050Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-05-19T18:19:49.2693344Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-05-19T18:19:49.2694417Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-05-19T18:19:49.2695511Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-05-19T18:19:49.2696418Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-05-19T18:19:49.2697297Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-05-19T18:19:49.2698373Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-05-19T18:19:49.2699211Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-05-19T18:19:49.2699977Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-05-19T18:19:49.2700771Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-05-19T18:19:49.2806491Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-05-19T18:19:49.2867765Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-05-19T18:19:49.2869333Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-05-19T18:19:49.2870585Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-05-19T18:19:49.2871427Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-05-19T18:19:49.2874474Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-05-19T18:19:49.2875404Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-05-19T18:19:49.2876316Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-05-19T18:19:49.2877272Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-05-19T18:19:49.2878242Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-05-19T18:19:49.2879328Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-05-19T18:19:49.2880424Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-05-19T18:19:49.2881425Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-05-19T18:19:49.2882386Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-05-19T18:19:49.2883494Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-05-19T18:19:49.2884764Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-05-19T18:19:49.2885894Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-05-19T18:19:49.2887008Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-05-19T18:19:49.2888135Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-05-19T18:19:49.2889263Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-05-19T18:19:49.2890401Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-05-19T18:19:49.2891381Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-05-19T18:19:49.2928323Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-05-19T18:19:49.2967735Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-05-19T18:19:49.2969149Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-05-19T18:19:49.3014042Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-05-19T18:19:49.3015438Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-05-19T18:19:49.3016982Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-05-19T18:19:49.3018596Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-05-19T18:19:49.3019985Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-05-19T18:19:49.3021365Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-05-19T18:19:49.3022732Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-05-19T18:19:49.3024092Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-05-19T18:19:49.3025430Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-05-19T18:19:49.3026683Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-05-19T18:19:49.3027899Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-05-19T18:19:49.3029105Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-05-19T18:19:49.3030310Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-05-19T18:19:49.3031480Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-05-19T18:19:49.3032660Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-05-19T18:19:49.3093690Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-05-19T18:19:49.3164902Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-05-19T18:19:49.3166070Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-05-19T18:19:49.3167039Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-05-19T18:19:49.3167868Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-05-19T18:19:49.3168745Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-05-19T18:19:49.3169669Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-05-19T18:19:49.3170716Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-05-19T18:19:49.3171887Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-05-19T18:19:49.3173017Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-05-19T18:19:49.3174187Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-05-19T18:19:49.3175272Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-05-19T18:19:49.3176354Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-05-19T18:19:49.3177429Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-05-19T18:19:49.3179481Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-05-19T18:19:49.3181438Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-05-19T18:19:49.3190459Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-05-19T18:19:49.3251680Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-05-19T18:19:49.3252883Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-05-19T18:19:49.3253955Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-05-19T18:19:49.3254917Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-05-19T18:19:49.3255825Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-05-19T18:19:49.3256705Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-05-19T18:19:49.3257784Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-05-19T18:19:49.3258621Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-05-19T18:19:49.3259378Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-05-19T18:19:49.3260179Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-05-19T18:19:49.3305140Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-05-19T18:19:49.3305894Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-05-19T18:19:49.3306649Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-05-19T18:19:49.3307520Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-05-19T18:19:49.3312562Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-05-19T18:19:49.3313542Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-05-19T18:19:49.3314523Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-05-19T18:19:49.3315564Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-05-19T18:19:49.3316583Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-05-19T18:19:49.3317740Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-05-19T18:19:49.3318893Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-05-19T18:19:49.3319934Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-05-19T18:19:49.3320971Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-05-19T18:19:49.3322168Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-05-19T18:19:49.3323361Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-05-19T18:19:49.3324642Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-05-19T18:19:49.3326000Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-05-19T18:19:49.3327209Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-05-19T18:19:49.3328418Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-05-19T18:19:49.3329502Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-05-19T18:19:49.3330554Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-05-19T18:19:49.3365859Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-05-19T18:19:49.3405116Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-05-19T18:19:49.3406720Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-05-19T18:19:49.3450635Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-05-19T18:19:49.3452091Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-05-19T18:19:49.3453578Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-05-19T18:19:49.3455109Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-05-19T18:19:49.3456574Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-05-19T18:19:49.3458156Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-05-19T18:19:49.3459599Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-05-19T18:19:49.3461082Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-05-19T18:19:49.3462480Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-05-19T18:19:49.3463819Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-05-19T18:19:49.3465100Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-05-19T18:19:49.3466357Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-05-19T18:19:49.3467628Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-05-19T18:19:49.3468865Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-05-19T18:19:49.3470145Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-05-19T18:19:49.3530859Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-05-19T18:19:49.3601469Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-05-19T18:19:49.3602711Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-05-19T18:19:49.3603755Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-05-19T18:19:49.3604728Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-05-19T18:19:49.3605805Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-05-19T18:19:49.3606795Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-05-19T18:19:49.3607918Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-05-19T18:19:49.3609161Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-05-19T18:19:49.3610357Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-05-19T18:19:49.3611475Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-05-19T18:19:49.3612624Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-05-19T18:19:49.3613908Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-05-19T18:19:49.3615097Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-05-19T18:19:49.3616267Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-05-19T18:19:49.3617433Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-05-19T18:19:49.3618803Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-05-19T18:19:49.3731970Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-05-19T18:19:49.3733134Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-05-19T18:19:49.3734303Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-05-19T18:19:49.3735607Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-05-19T18:19:49.3736858Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-05-19T18:19:49.3738146Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-05-19T18:19:49.3739365Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-05-19T18:19:49.3740632Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-05-19T18:19:49.3741845Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-05-19T18:19:49.3743044Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-05-19T18:19:49.3744250Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-05-19T18:19:49.3755619Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-05-19T18:19:49.3809334Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-05-19T18:19:49.3810904Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-05-19T18:19:49.3812024Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-05-19T18:19:49.3813035Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-05-19T18:19:49.3813988Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-05-19T18:19:49.3814928Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-05-19T18:19:49.3815982Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-05-19T18:19:49.3833553Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-05-19T18:19:49.3834456Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-05-19T18:19:49.3835284Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-05-19T18:19:49.3913584Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-05-19T18:19:49.3954961Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-05-19T18:19:49.3955581Z creating: build/lib/ 2024-05-19T18:19:49.4040655Z inflating: build/lib/libprotobuf-lite.a 2024-05-19T18:19:49.4482097Z inflating: build/lib/libprotobuf.a 2024-05-19T18:19:49.4491137Z inflating: build/lib/libpthreadpool.a 2024-05-19T18:19:49.4498993Z inflating: build/lib/libcpuinfo.a 2024-05-19T18:19:49.4506419Z inflating: build/lib/libcpuinfo_internals.a 2024-05-19T18:19:49.4507548Z inflating: build/lib/libclog.a 2024-05-19T18:19:49.4524401Z inflating: build/lib/libnnpack.a 2024-05-19T18:19:49.4526957Z inflating: build/lib/libnnpack_reference_layers.a 2024-05-19T18:19:49.4589879Z inflating: build/lib/libgtest.a 2024-05-19T18:19:49.4661722Z inflating: build/lib/libbenchmark.a 2024-05-19T18:19:49.4723475Z inflating: build/lib/libasmjit.a 2024-05-19T18:19:49.4730642Z inflating: build/lib/libittnotify.a 2024-05-19T18:19:49.4757467Z inflating: build/lib/libtensorpipe_uv.a 2024-05-19T18:19:49.4880249Z inflating: build/lib/libgloo.a 2024-05-19T18:19:49.4901348Z inflating: build/lib/libfmt.a 2024-05-19T18:19:49.4989401Z inflating: build/lib/libc10.so 2024-05-19T18:19:49.4990861Z inflating: build/lib/libcaffe2_nvrtc.so 2024-05-19T18:19:49.4991352Z inflating: build/lib/libfoxi_loader.a 2024-05-19T18:19:49.4992438Z inflating: build/lib/libtorch_global_deps.so 2024-05-19T18:19:49.5005486Z inflating: build/lib/libqnnpack.a 2024-05-19T18:19:49.5495109Z inflating: build/lib/libprotoc.a 2024-05-19T18:19:49.5513716Z inflating: build/lib/libpytorch_qnnpack.a 2024-05-19T18:19:49.5530821Z inflating: build/lib/libgmock.a 2024-05-19T18:19:49.5531266Z inflating: build/lib/libgtest_main.a 2024-05-19T18:19:49.5531927Z inflating: build/lib/libbenchmark_main.a 2024-05-19T18:19:49.5903398Z inflating: build/lib/libgloo_cuda.a 2024-05-19T18:19:50.5684807Z inflating: build/lib/libdnnl.a 2024-05-19T18:19:50.6238135Z inflating: build/lib/libtensorpipe.a 2024-05-19T18:19:50.6290370Z inflating: build/lib/libc10_cuda.so 2024-05-19T18:19:50.6290920Z inflating: build/lib/libgmock_main.a 2024-05-19T18:19:50.7501268Z inflating: build/lib/libfbgemm.a 2024-05-19T18:19:50.7746890Z inflating: build/lib/libtensorpipe_cuda.a 2024-05-19T18:19:50.8246795Z inflating: build/lib/libkineto.a 2024-05-19T18:19:50.8283192Z inflating: build/lib/libcaffe2_protos.a 2024-05-19T18:19:50.8325588Z inflating: build/lib/libonnx_proto.a 2024-05-19T18:19:50.8503623Z inflating: build/lib/libXNNPACK.a 2024-05-19T18:19:50.9179647Z inflating: build/lib/libonnx.a 2024-05-19T18:19:53.2334775Z inflating: build/lib/libtorch_cpu.so 2024-05-19T18:19:53.2338984Z inflating: build/lib/libunbox_lib.a 2024-05-19T18:19:53.2342803Z inflating: build/lib/libshm.so 2024-05-19T18:19:55.2151228Z inflating: build/lib/libtorch_cuda.so 2024-05-19T18:19:55.2152338Z inflating: build/lib/libtorch.so 2024-05-19T18:19:55.2155122Z inflating: build/lib/libc10d_cuda_test.so 2024-05-19T18:19:56.0723676Z inflating: build/lib/libtorch_cuda_linalg.so 2024-05-19T18:19:56.2584605Z inflating: build/lib/libtorch_python.so 2024-05-19T18:19:56.2653255Z inflating: build/lib/libtorchbind_test.so 2024-05-19T18:19:56.2673494Z inflating: build/lib/libjitbackend_test.so 2024-05-19T18:19:56.2698311Z inflating: build/lib/libbackend_with_compiler.so 2024-05-19T18:19:56.2731657Z inflating: build/lib/libnnapi_backend.so 2024-05-19T18:19:56.2732099Z creating: build/bin/ 2024-05-19T18:19:56.2780566Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-05-19T18:19:56.2830712Z inflating: build/bin/c10_DeviceGuard_test 2024-05-19T18:19:56.2880098Z inflating: build/bin/c10_Device_test 2024-05-19T18:19:56.2937253Z inflating: build/bin/c10_DispatchKeySet_test 2024-05-19T18:19:56.2989515Z inflating: build/bin/c10_Scalar_test 2024-05-19T18:19:56.3037243Z inflating: build/bin/c10_StreamGuard_test 2024-05-19T18:19:56.3086803Z inflating: build/bin/c10_SymInt_test 2024-05-19T18:19:56.3139649Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-05-19T18:19:56.3194182Z inflating: build/bin/c10_InlineStreamGuard_test 2024-05-19T18:19:56.3249018Z inflating: build/bin/c10_SizesAndStrides_test 2024-05-19T18:19:56.3317892Z inflating: build/bin/c10_cow_test 2024-05-19T18:19:56.3370189Z inflating: build/bin/c10_Bitset_test 2024-05-19T18:19:56.3416997Z inflating: build/bin/c10_ConstexprCrc_test 2024-05-19T18:19:56.3465505Z inflating: build/bin/c10_DeadlockDetection_test 2024-05-19T18:19:56.3515126Z inflating: build/bin/c10_Half_test 2024-05-19T18:19:56.3569799Z inflating: build/bin/c10_LeftRight_test 2024-05-19T18:19:56.3622924Z inflating: build/bin/c10_Metaprogramming_test 2024-05-19T18:19:56.3672975Z inflating: build/bin/c10_Synchronized_test 2024-05-19T18:19:56.3727476Z inflating: build/bin/c10_ThreadLocal_test 2024-05-19T18:19:56.3777203Z inflating: build/bin/c10_TypeIndex_test 2024-05-19T18:19:56.3827758Z inflating: build/bin/c10_TypeList_test 2024-05-19T18:19:56.3875517Z inflating: build/bin/c10_TypeTraits_test 2024-05-19T18:19:56.3926240Z inflating: build/bin/c10_accumulate_test 2024-05-19T18:19:56.3979949Z inflating: build/bin/c10_bfloat16_test 2024-05-19T18:19:56.4029239Z inflating: build/bin/c10_bit_cast_test 2024-05-19T18:19:56.4085018Z inflating: build/bin/c10_complex_math_test 2024-05-19T18:19:56.4138879Z inflating: build/bin/c10_complex_test 2024-05-19T18:19:56.4190055Z inflating: build/bin/c10_exception_test 2024-05-19T18:19:56.4239542Z inflating: build/bin/c10_flags_test 2024-05-19T18:19:56.4288489Z inflating: build/bin/c10_generic_math_test 2024-05-19T18:19:56.4447626Z inflating: build/bin/c10_intrusive_ptr_test 2024-05-19T18:19:56.4496863Z inflating: build/bin/c10_irange_test 2024-05-19T18:19:56.4549302Z inflating: build/bin/c10_lazy_test 2024-05-19T18:19:56.4604900Z inflating: build/bin/c10_logging_test 2024-05-19T18:19:56.4677644Z inflating: build/bin/c10_optional_test 2024-05-19T18:19:56.4738409Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-05-19T18:19:56.4790936Z inflating: build/bin/c10_registry_test 2024-05-19T18:19:56.4938041Z inflating: build/bin/c10_small_vector_test 2024-05-19T18:19:56.4988124Z inflating: build/bin/c10_ssize_test 2024-05-19T18:19:56.5038760Z inflating: build/bin/c10_string_util_test 2024-05-19T18:19:56.5095816Z inflating: build/bin/c10_string_view_test 2024-05-19T18:19:56.5145120Z inflating: build/bin/c10_tempfile_test 2024-05-19T18:19:56.5199593Z inflating: build/bin/c10_typeid_test 2024-05-19T18:19:56.5246890Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-05-19T18:19:56.5682520Z inflating: build/bin/protoc-3.13.0.0 2024-05-19T18:19:56.6118731Z inflating: build/bin/protoc 2024-05-19T18:19:56.6170961Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-05-19T18:19:56.6221851Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-05-19T18:19:56.6273288Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-05-19T18:19:56.6323631Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-05-19T18:19:56.6375053Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-05-19T18:19:56.6426683Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-05-19T18:19:56.6474286Z inflating: build/bin/c10_cuda_CUDATest 2024-05-19T18:19:56.6525774Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-05-19T18:19:56.6845732Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-05-19T18:19:56.7180057Z inflating: build/bin/vec_test_all_types_AVX512 2024-05-19T18:19:56.7530044Z inflating: build/bin/vec_test_all_types_AVX2 2024-05-19T18:19:56.7581921Z inflating: build/bin/FileStoreTest 2024-05-19T18:19:56.7636604Z inflating: build/bin/TCPStoreTest 2024-05-19T18:19:56.7688598Z inflating: build/bin/HashStoreTest 2024-05-19T18:19:56.7701434Z inflating: build/bin/ProcessGroupMPITest 2024-05-19T18:19:56.7754266Z inflating: build/bin/test_edge_op_registration 2024-05-19T18:19:56.7758336Z inflating: build/bin/torch_shm_manager 2024-05-19T18:19:56.7761371Z inflating: build/bin/example_allreduce 2024-05-19T18:19:56.7814286Z inflating: build/bin/test_dist_autograd 2024-05-19T18:19:56.7882225Z inflating: build/bin/test_cpp_rpc 2024-05-19T18:19:56.7884311Z inflating: build/bin/parallel_benchmark 2024-05-19T18:19:56.7949540Z inflating: build/bin/test_mobile_nnc 2024-05-19T18:19:56.7958440Z inflating: build/bin/aot_model_compiler_test 2024-05-19T18:19:56.8294314Z inflating: build/bin/test_lazy 2024-05-19T18:19:56.9446084Z inflating: build/bin/test_api 2024-05-19T18:19:56.9517465Z inflating: build/bin/Dict_test 2024-05-19T18:19:56.9568262Z inflating: build/bin/Dimname_test 2024-05-19T18:19:56.9631201Z inflating: build/bin/MaybeOwned_test 2024-05-19T18:19:56.9687951Z inflating: build/bin/NamedTensor_test 2024-05-19T18:19:56.9745563Z inflating: build/bin/apply_utils_test 2024-05-19T18:19:56.9802912Z inflating: build/bin/atest 2024-05-19T18:19:56.9864415Z inflating: build/bin/basic 2024-05-19T18:19:56.9917620Z inflating: build/bin/broadcast_test 2024-05-19T18:19:56.9967358Z inflating: build/bin/cpu_allocator_test 2024-05-19T18:19:57.0023942Z inflating: build/bin/cpu_generator_test 2024-05-19T18:19:57.0076089Z inflating: build/bin/cpu_profiling_allocator_test 2024-05-19T18:19:57.0166724Z inflating: build/bin/cpu_rng_test 2024-05-19T18:19:57.0215314Z inflating: build/bin/dispatch_key_set_test 2024-05-19T18:19:57.0264703Z inflating: build/bin/dlconvertor_test 2024-05-19T18:19:57.0321441Z inflating: build/bin/extension_backend_test 2024-05-19T18:19:57.0375640Z inflating: build/bin/half_test 2024-05-19T18:19:57.0469145Z inflating: build/bin/ivalue_test 2024-05-19T18:19:57.0517781Z inflating: build/bin/lazy_tensor_test 2024-05-19T18:19:57.0571042Z inflating: build/bin/math_kernel_test 2024-05-19T18:19:57.0623909Z inflating: build/bin/memory_format_test 2024-05-19T18:19:57.0678063Z inflating: build/bin/memory_overlapping_test 2024-05-19T18:19:57.0729752Z inflating: build/bin/mobile_memory_cleanup 2024-05-19T18:19:57.0784521Z inflating: build/bin/native_test 2024-05-19T18:19:57.0833582Z inflating: build/bin/operator_name_test 2024-05-19T18:19:57.0883762Z inflating: build/bin/operators_test 2024-05-19T18:19:57.0934680Z inflating: build/bin/packedtensoraccessor_test 2024-05-19T18:19:57.1000596Z inflating: build/bin/pow_test 2024-05-19T18:19:57.1055707Z inflating: build/bin/quantized_test 2024-05-19T18:19:57.1104529Z inflating: build/bin/reduce_ops_test 2024-05-19T18:19:57.1154776Z inflating: build/bin/reportMemoryUsage_test 2024-05-19T18:19:57.1209746Z inflating: build/bin/scalar_tensor_test 2024-05-19T18:19:57.1266696Z inflating: build/bin/scalar_test 2024-05-19T18:19:57.1317754Z inflating: build/bin/StorageUtils_test 2024-05-19T18:19:57.1369395Z inflating: build/bin/stride_properties_test 2024-05-19T18:19:57.1445873Z inflating: build/bin/tensor_iterator_test 2024-05-19T18:19:57.1498642Z inflating: build/bin/test_parallel 2024-05-19T18:19:57.1501488Z inflating: build/bin/thread_init_test 2024-05-19T18:19:57.1555958Z inflating: build/bin/type_ptr_test 2024-05-19T18:19:57.1614539Z inflating: build/bin/type_test 2024-05-19T18:19:57.1665323Z inflating: build/bin/undefined_tensor_test 2024-05-19T18:19:57.1666924Z inflating: build/bin/verify_api_visibility 2024-05-19T18:19:57.1735168Z inflating: build/bin/legacy_vmap_test 2024-05-19T18:19:57.1785640Z inflating: build/bin/weakref_test 2024-05-19T18:19:57.1835811Z inflating: build/bin/wrapdim_test 2024-05-19T18:19:57.1887183Z inflating: build/bin/xla_tensor_test 2024-05-19T18:19:57.1945759Z inflating: build/bin/IListRef_test 2024-05-19T18:19:57.2048851Z inflating: build/bin/List_test 2024-05-19T18:19:57.2114402Z inflating: build/bin/KernelFunction_test 2024-05-19T18:19:57.2233051Z inflating: build/bin/kernel_function_legacy_test 2024-05-19T18:19:57.2326683Z inflating: build/bin/kernel_function_test 2024-05-19T18:19:57.2452035Z inflating: build/bin/kernel_lambda_legacy_test 2024-05-19T18:19:57.2552575Z inflating: build/bin/kernel_lambda_test 2024-05-19T18:19:57.2612388Z inflating: build/bin/kernel_stackbased_test 2024-05-19T18:19:57.2705638Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-05-19T18:19:57.2755538Z inflating: build/bin/CppSignature_test 2024-05-19T18:19:57.2802892Z inflating: build/bin/op_allowlist_test 2024-05-19T18:19:57.2858429Z inflating: build/bin/backend_fallback_test 2024-05-19T18:19:57.3151022Z inflating: build/bin/op_registration_test 2024-05-19T18:19:57.3212861Z inflating: build/bin/inline_container_test 2024-05-19T18:19:57.3264485Z inflating: build/bin/cuda_apply_test 2024-05-19T18:19:57.3313676Z inflating: build/bin/cuda_allocator_test 2024-05-19T18:19:57.3367171Z inflating: build/bin/cuda_caching_host_allocator_test 2024-05-19T18:19:57.3424938Z inflating: build/bin/cuda_atomic_ops_test 2024-05-19T18:19:57.3493198Z inflating: build/bin/cuda_complex_math_test 2024-05-19T18:19:57.3552188Z inflating: build/bin/cuda_complex_test 2024-05-19T18:19:57.3600703Z inflating: build/bin/cuda_device_test 2024-05-19T18:19:57.3658052Z inflating: build/bin/cuda_cub_test 2024-05-19T18:19:57.3707300Z inflating: build/bin/cuda_dlconvertor_test 2024-05-19T18:19:57.3770632Z inflating: build/bin/cuda_distributions_test 2024-05-19T18:19:57.3825570Z inflating: build/bin/cuda_generator_test 2024-05-19T18:19:57.3874170Z inflating: build/bin/cuda_half_test 2024-05-19T18:19:57.3923815Z inflating: build/bin/cuda_integer_divider_test 2024-05-19T18:19:57.3972292Z inflating: build/bin/cuda_optional_test 2024-05-19T18:19:57.4022382Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-05-19T18:19:57.4073675Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-05-19T18:19:57.4122194Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-05-19T18:19:57.4181480Z inflating: build/bin/cuda_stream_test 2024-05-19T18:19:57.4229321Z inflating: build/bin/cuda_cudnn_test 2024-05-19T18:19:57.4280272Z inflating: build/bin/cuda_vectorized_test 2024-05-19T18:19:57.4294696Z inflating: build/bin/tutorial_tensorexpr 2024-05-19T18:19:57.4358857Z inflating: build/bin/ProcessGroupGlooTest 2024-05-19T18:19:57.4414844Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-05-19T18:19:57.4476211Z inflating: build/bin/ProcessGroupNCCLTest 2024-05-19T18:19:57.4537881Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-05-19T18:19:57.5348666Z inflating: build/bin/test_tensorexpr 2024-05-19T18:19:57.5912832Z inflating: build/bin/test_jit 2024-05-19T18:19:57.5913704Z creating: .additional_ci_files/ 2024-05-19T18:19:57.5953913Z inflating: .additional_ci_files/test-times.json 2024-05-19T18:19:57.6116966Z inflating: .additional_ci_files/test-class-times.json 2024-05-19T18:19:57.6147694Z ##[group]Run rm artifacts.zip 2024-05-19T18:19:57.6148096Z rm artifacts.zip 2024-05-19T18:19:57.6155905Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:19:57.6156409Z env: 2024-05-19T18:19:57.6156690Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:19:57.6157148Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:19:57.6157635Z ##[endgroup] 2024-05-19T18:19:57.6840176Z ##[group]Run df -H 2024-05-19T18:19:57.6840487Z df -H 2024-05-19T18:19:57.6848027Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:19:57.6848519Z env: 2024-05-19T18:19:57.6848805Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:19:57.6849245Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:19:57.6849722Z ##[endgroup] 2024-05-19T18:19:57.6875587Z Filesystem Size Used Avail Use% Mounted on 2024-05-19T18:19:57.6876112Z devtmpfs 34G 0 34G 0% /dev 2024-05-19T18:19:57.6876910Z tmpfs 34G 0 34G 0% /dev/shm 2024-05-19T18:19:57.6877514Z tmpfs 34G 455k 34G 1% /run 2024-05-19T18:19:57.6878169Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-05-19T18:19:57.6878643Z /dev/nvme0n1p1 162G 44G 118G 27% / 2024-05-19T18:19:57.6917268Z Prepare all required actions 2024-05-19T18:19:57.6917696Z Getting action download info 2024-05-19T18:19:57.8016200Z ##[group]Run ./.github/actions/download-td-artifacts 2024-05-19T18:19:57.8016655Z with: 2024-05-19T18:19:57.8016989Z env: 2024-05-19T18:19:57.8017350Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:19:57.8017947Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:19:57.8018423Z ##[endgroup] 2024-05-19T18:19:57.8049247Z ##[group]Run seemethere/download-artifact-s3@v4 2024-05-19T18:19:57.8049690Z with: 2024-05-19T18:19:57.8049953Z name: td_results 2024-05-19T18:19:57.8050273Z s3-bucket: gha-artifacts 2024-05-19T18:19:57.8050626Z region: us-east-1 2024-05-19T18:19:57.8050923Z env: 2024-05-19T18:19:57.8051192Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:19:57.8051644Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:19:57.8052135Z ##[endgroup] 2024-05-19T18:19:58.2257322Z (node:28950) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-05-19T18:19:58.2258246Z 2024-05-19T18:19:58.2258516Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-05-19T18:19:58.2259222Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-05-19T18:19:58.2260035Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-05-19T18:19:58.3003729Z Found 0 objects with prefix pytorch/pytorch/9148947323/td_results/ 2024-05-19T18:19:58.3011179Z Artifact download has finished successfully 2024-05-19T18:19:58.3176732Z ##[group]Run mkdir -p .additional_ci_files 2024-05-19T18:19:58.3177215Z mkdir -p .additional_ci_files 2024-05-19T18:19:58.3178016Z mv td_results.json .additional_ci_files/td_results.json 2024-05-19T18:19:58.3185866Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:19:58.3186375Z env: 2024-05-19T18:19:58.3186655Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:19:58.3187118Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:19:58.3187602Z ##[endgroup] 2024-05-19T18:19:58.3222492Z mv: cannot stat 'td_results.json': No such file or directory 2024-05-19T18:19:58.3228297Z ##[error]Process completed with exit code 1. 2024-05-19T18:19:58.3261548Z ##[group]Run .github/scripts/parse_ref.py 2024-05-19T18:19:58.3262038Z .github/scripts/parse_ref.py 2024-05-19T18:19:58.3286968Z shell: /usr/bin/bash -e {0} 2024-05-19T18:19:58.3287329Z env: 2024-05-19T18:19:58.3287607Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:19:58.3288061Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:19:58.3288541Z ##[endgroup] 2024-05-19T18:19:58.3526128Z Prepare all required actions 2024-05-19T18:19:58.3563313Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-05-19T18:19:58.3563772Z with: 2024-05-19T18:19:58.3564317Z github-token: *** 2024-05-19T18:19:58.3564866Z env: 2024-05-19T18:19:58.3565138Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:19:58.3565598Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:19:58.3566086Z ##[endgroup] 2024-05-19T18:19:58.3586690Z ##[group]Run set -eux 2024-05-19T18:19:58.3587027Z set -eux 2024-05-19T18:19:58.3587603Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-05-19T18:19:58.3595427Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:19:58.3595919Z env: 2024-05-19T18:19:58.3596201Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:19:58.3596657Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:19:58.3597294Z GITHUB_TOKEN: *** 2024-05-19T18:19:58.3597606Z ##[endgroup] 2024-05-19T18:19:58.3619493Z + python3 .github/scripts/get_workflow_job_id.py 9148947323 i-01c138b15d87f731e 2024-05-19T18:19:59.6459581Z setting job-id=25153197487 2024-05-19T18:19:59.6460434Z setting job-name=cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:19:59.6647593Z Prepare all required actions 2024-05-19T18:19:59.6648034Z Getting action download info 2024-05-19T18:19:59.7674997Z ##[group]Run ./.github/actions/filter-test-configs 2024-05-19T18:19:59.7675462Z with: 2024-05-19T18:19:59.7675921Z github-token: *** 2024-05-19T18:19:59.7683523Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-05-19T18:19:59.7691872Z job-name: cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:19:59.7692595Z env: 2024-05-19T18:19:59.7692880Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:19:59.7693382Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:19:59.7693858Z ##[endgroup] 2024-05-19T18:19:59.7734158Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-05-19T18:19:59.7734711Z with: 2024-05-19T18:19:59.7734980Z shell: bash 2024-05-19T18:19:59.7735284Z timeout_minutes: 10 2024-05-19T18:19:59.7735612Z max_attempts: 5 2024-05-19T18:19:59.7735917Z retry_wait_seconds: 30 2024-05-19T18:19:59.7736684Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore python3 -m pip install requests==2.26.0 pyyaml==6.0.1 2024-05-19T18:19:59.7737773Z polling_interval_seconds: 1 2024-05-19T18:19:59.7738151Z warning_on_retry: true 2024-05-19T18:19:59.7738495Z continue_on_error: false 2024-05-19T18:19:59.7738827Z env: 2024-05-19T18:19:59.7739102Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:19:59.7739542Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:19:59.7740194Z GITHUB_TOKEN: *** 2024-05-19T18:19:59.7740508Z ##[endgroup] 2024-05-19T18:19:59.8228934Z + python3 -m pip install requests==2.26.0 pyyaml==6.0.1 2024-05-19T18:20:00.0385956Z Defaulting to user installation because normal site-packages is not writeable 2024-05-19T18:20:00.1598306Z Collecting requests==2.26.0 2024-05-19T18:20:00.1809816Z Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB) 2024-05-19T18:20:00.3347144Z Collecting pyyaml==6.0.1 2024-05-19T18:20:00.3416176Z Downloading PyYAML-6.0.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (670 kB) 2024-05-19T18:20:00.4748369Z Collecting urllib3<1.27,>=1.21.1 2024-05-19T18:20:00.4806008Z Downloading urllib3-1.26.18-py2.py3-none-any.whl (143 kB) 2024-05-19T18:20:00.5656864Z Collecting certifi>=2017.4.17 2024-05-19T18:20:00.5714350Z Downloading certifi-2024.2.2-py3-none-any.whl (163 kB) 2024-05-19T18:20:00.8896049Z Collecting charset-normalizer~=2.0.0; python_version >= "3" 2024-05-19T18:20:00.8939508Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-05-19T18:20:00.9400150Z Collecting idna<4,>=2.5; python_version >= "3" 2024-05-19T18:20:00.9445428Z Downloading idna-3.7-py3-none-any.whl (66 kB) 2024-05-19T18:20:01.0160110Z Installing collected packages: urllib3, certifi, charset-normalizer, idna, requests, pyyaml 2024-05-19T18:20:01.1132415Z WARNING: The script normalizer is installed in '/home/ec2-user/.local/bin' which is not on PATH. 2024-05-19T18:20:01.1133620Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-05-19T18:20:01.2737860Z Successfully installed certifi-2024.2.2 charset-normalizer-2.0.12 idna-3.7 pyyaml-6.0.1 requests-2.26.0 urllib3-1.26.18 2024-05-19T18:20:01.8246463Z Command completed after 1 attempt(s). 2024-05-19T18:20:01.8286662Z ##[group]Run set -x 2024-05-19T18:20:01.8286995Z set -x 2024-05-19T18:20:01.8287300Z  2024-05-19T18:20:01.8287850Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-05-19T18:20:01.8288518Z # in runner workspace 2024-05-19T18:20:01.8289048Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-05-19T18:20:01.8296730Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:20:01.8297225Z env: 2024-05-19T18:20:01.8297502Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:20:01.8298190Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:20:01.8298679Z ##[endgroup] 2024-05-19T18:20:01.8318876Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-05-19T18:20:01.8520263Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-05-19T18:20:01.8520815Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-05-19T18:20:01.8521274Z echo "Job name: ${JOB_NAME}" 2024-05-19T18:20:01.8521679Z  2024-05-19T18:20:01.8522227Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-05-19T18:20:01.8522901Z # in runner workspace 2024-05-19T18:20:01.8523476Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-05-19T18:20:01.8524117Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-05-19T18:20:01.8524730Z  --job-name "${JOB_NAME}" \ 2024-05-19T18:20:01.8532539Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-05-19T18:20:01.8540835Z  --selected-test-configs "" \ 2024-05-19T18:20:01.8541302Z  --pr-number "${PR_NUMBER}" \ 2024-05-19T18:20:01.8541731Z  --tag "${TAG}" \ 2024-05-19T18:20:01.8542125Z  --event-name "${EVENT_NAME}" \ 2024-05-19T18:20:01.8542591Z  --schedule "${SCHEDULE}" \ 2024-05-19T18:20:01.8543036Z  --branch "${HEAD_BRANCH}" 2024-05-19T18:20:01.8550623Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:20:01.8551114Z env: 2024-05-19T18:20:01.8551399Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:20:01.8551858Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:20:01.8552533Z GITHUB_TOKEN: *** 2024-05-19T18:20:01.8553183Z JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:20:01.8553913Z PR_NUMBER: 2024-05-19T18:20:01.8554243Z TAG: ciflow/inductor/121956 2024-05-19T18:20:01.8554619Z EVENT_NAME: push 2024-05-19T18:20:01.8554930Z SCHEDULE: 2024-05-19T18:20:01.8555233Z HEAD_BRANCH: 2024-05-19T18:20:01.8555542Z ##[endgroup] 2024-05-19T18:20:01.8590209Z Workflow: inductor 2024-05-19T18:20:01.8591088Z Job name: cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:20:02.1526428Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-05-19T18:20:02.2866353Z ##[group]Run echo "Filtered matrix:" 2024-05-19T18:20:02.2866827Z echo "Filtered matrix:" 2024-05-19T18:20:02.2874596Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-05-19T18:20:02.2882658Z  2024-05-19T18:20:02.2882940Z echo 2024-05-19T18:20:02.2883324Z echo "Is the current job unstable? False" 2024-05-19T18:20:02.2883776Z  2024-05-19T18:20:02.2884052Z echo 2024-05-19T18:20:02.2884414Z echo "Is keep-going label set? False" 2024-05-19T18:20:02.2885000Z  2024-05-19T18:20:02.2885267Z echo 2024-05-19T18:20:02.2885585Z echo "Renabled issues? " 2024-05-19T18:20:02.2893255Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:20:02.2893781Z env: 2024-05-19T18:20:02.2894090Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:20:02.2894548Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:20:02.2895030Z ##[endgroup] 2024-05-19T18:20:02.2914365Z Filtered matrix: 2024-05-19T18:20:02.2924308Z {include: [{config: inductor, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-05-19T18:20:02.2932144Z 2024-05-19T18:20:02.2932295Z Is the current job unstable? False 2024-05-19T18:20:02.2932582Z 2024-05-19T18:20:02.2932892Z Is keep-going label set? False 2024-05-19T18:20:02.2933152Z 2024-05-19T18:20:02.2933274Z Renabled issues? 2024-05-19T18:20:02.2973167Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-05-19T18:20:02.2973946Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-05-19T18:20:02.2980915Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:20:02.2981403Z env: 2024-05-19T18:20:02.2981688Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:20:02.2982145Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:20:02.2982637Z JOB_TIMEOUT: 240 2024-05-19T18:20:02.2982943Z ##[endgroup] 2024-05-19T18:20:02.3072119Z ##[group]Run set -x 2024-05-19T18:20:02.3072528Z set -x 2024-05-19T18:20:02.3072833Z  2024-05-19T18:20:02.3073198Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-05-19T18:20:02.3073737Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-05-19T18:20:02.3074299Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-05-19T18:20:02.3074812Z  TEST_COMMAND=.ci/onnx/test.sh 2024-05-19T18:20:02.3075228Z else 2024-05-19T18:20:02.3075583Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-05-19T18:20:02.3076005Z fi 2024-05-19T18:20:02.3076285Z  2024-05-19T18:20:02.3076756Z # detached container should get cleaned up by teardown_ec2_linux 2024-05-19T18:20:02.3077493Z # TODO: Stop building test binaries as part of the build phase 2024-05-19T18:20:02.3078161Z # Used for GPU_FLAG since that doesn't play nice 2024-05-19T18:20:02.3078737Z # shellcheck disable=SC2086,SC2090 2024-05-19T18:20:02.3079192Z container_name=$(docker run \ 2024-05-19T18:20:02.3079609Z  ${GPU_FLAG:-} \ 2024-05-19T18:20:02.3079989Z  -e BUILD_ENVIRONMENT \ 2024-05-19T18:20:02.3080395Z  -e PR_NUMBER \ 2024-05-19T18:20:02.3080751Z  -e GITHUB_ACTIONS \ 2024-05-19T18:20:02.3081147Z  -e GITHUB_REPOSITORY \ 2024-05-19T18:20:02.3081557Z  -e GITHUB_WORKFLOW \ 2024-05-19T18:20:02.3081946Z  -e GITHUB_JOB \ 2024-05-19T18:20:02.3082307Z  -e GITHUB_RUN_ID \ 2024-05-19T18:20:02.3082697Z  -e GITHUB_RUN_NUMBER \ 2024-05-19T18:20:02.3083110Z  -e GITHUB_RUN_ATTEMPT \ 2024-05-19T18:20:02.3083500Z  -e JOB_ID \ 2024-05-19T18:20:02.3083898Z  -e JOB_NAME \ 2024-05-19T18:20:02.3084249Z  -e BASE_SHA \ 2024-05-19T18:20:02.3084735Z  -e BRANCH \ 2024-05-19T18:20:02.3085063Z  -e SHA1 \ 2024-05-19T18:20:02.3085413Z  -e AWS_DEFAULT_REGION \ 2024-05-19T18:20:02.3085820Z  -e IN_WHEEL_TEST \ 2024-05-19T18:20:02.3086204Z  -e SHARD_NUMBER \ 2024-05-19T18:20:02.3086570Z  -e TEST_CONFIG \ 2024-05-19T18:20:02.3086945Z  -e NUM_TEST_SHARDS \ 2024-05-19T18:20:02.3087339Z  -e REENABLED_ISSUES \ 2024-05-19T18:20:02.3087754Z  -e CONTINUE_THROUGH_ERROR \ 2024-05-19T18:20:02.3088174Z  -e VERBOSE_TEST_LOGS \ 2024-05-19T18:20:02.3088576Z  -e NO_TEST_TIMEOUT \ 2024-05-19T18:20:02.3088962Z  -e NO_TD \ 2024-05-19T18:20:02.3089299Z  -e TD_DISTRIBUTED \ 2024-05-19T18:20:02.3089690Z  -e PR_LABELS \ 2024-05-19T18:20:02.3090106Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-05-19T18:20:02.3090567Z  -e SCCACHE_BUCKET \ 2024-05-19T18:20:02.3090958Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-05-19T18:20:02.3091374Z  -e XLA_CUDA \ 2024-05-19T18:20:02.3091788Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-05-19T18:20:02.3092301Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-05-19T18:20:02.3092818Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-05-19T18:20:02.3093338Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-05-19T18:20:02.3093848Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-05-19T18:20:02.3094283Z  -e DASHBOARD_TAG \ 2024-05-19T18:20:02.3094756Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-05-19T18:20:02.3095313Z  --security-opt seccomp=unconfined \ 2024-05-19T18:20:02.3095890Z  --cap-add=SYS_PTRACE \ 2024-05-19T18:20:02.3096283Z  --ipc=host \ 2024-05-19T18:20:02.3096643Z  --shm-size="${SHM_SIZE}" \ 2024-05-19T18:20:02.3097046Z  --tty \ 2024-05-19T18:20:02.3097366Z  --detach \ 2024-05-19T18:20:02.3097939Z  --name="${container_name}" \ 2024-05-19T18:20:02.3098359Z  --user jenkins \ 2024-05-19T18:20:02.3098924Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-05-19T18:20:02.3099495Z  -w /var/lib/jenkins/workspace \ 2024-05-19T18:20:02.3099940Z  "${DOCKER_IMAGE}" 2024-05-19T18:20:02.3100283Z ) 2024-05-19T18:20:02.3100697Z # Propagate download.pytorch.org IP to container 2024-05-19T18:20:02.3101614Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-05-19T18:20:02.3102573Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-05-19T18:20:02.3103484Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-05-19T18:20:02.3111307Z shell: /usr/bin/bash -e {0} 2024-05-19T18:20:02.3111667Z env: 2024-05-19T18:20:02.3111950Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:20:02.3112395Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:20:02.3112996Z BUILD_ENVIRONMENT: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:20:02.3113495Z PR_NUMBER: 2024-05-19T18:20:02.3113825Z GITHUB_REPOSITORY: pytorch/pytorch 2024-05-19T18:20:02.3114236Z GITHUB_WORKFLOW: inductor 2024-05-19T18:20:02.3114600Z GITHUB_JOB: test 2024-05-19T18:20:02.3114919Z GITHUB_RUN_ID: 9148947323 2024-05-19T18:20:02.3115278Z GITHUB_RUN_NUMBER: 69546 2024-05-19T18:20:02.3115624Z GITHUB_RUN_ATTEMPT: 1 2024-05-19T18:20:02.3115962Z JOB_ID: 25153197487 2024-05-19T18:20:02.3116616Z JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:20:02.3117329Z BRANCH: 2024-05-19T18:20:02.3117667Z SHA1: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:20:02.3118201Z BASE_SHA: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:20:02.3118689Z TEST_CONFIG: dynamic_inductor_timm 2024-05-19T18:20:02.3119086Z SHARD_NUMBER: 1 2024-05-19T18:20:02.3119391Z NUM_TEST_SHARDS: 2 2024-05-19T18:20:02.3119715Z REENABLED_ISSUES: 2024-05-19T18:20:02.3120071Z CONTINUE_THROUGH_ERROR: False 2024-05-19T18:20:02.3120453Z VERBOSE_TEST_LOGS: False 2024-05-19T18:20:02.3120806Z NO_TEST_TIMEOUT: False 2024-05-19T18:20:02.3121143Z NO_TD: False 2024-05-19T18:20:02.3121445Z TD_DISTRIBUTED: False 2024-05-19T18:20:02.3121851Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-05-19T18:20:02.3122340Z SCCACHE_S3_KEY_PREFIX: inductor 2024-05-19T18:20:02.3122721Z SHM_SIZE: 2g 2024-05-19T18:20:02.3123829Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:20:02.3125077Z XLA_CUDA: 2024-05-19T18:20:02.3125556Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-05-19T18:20:02.3126169Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-05-19T18:20:02.3126603Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-05-19T18:20:02.3127005Z DASHBOARD_TAG: 2024-05-19T18:20:02.3127527Z HUGGING_FACE_HUB_TOKEN: *** 2024-05-19T18:20:02.3127901Z ##[endgroup] 2024-05-19T18:20:02.3146604Z + [[ dynamic_inductor_timm == \m\u\l\t\i\g\p\u ]] 2024-05-19T18:20:02.3147378Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *onnx* ]] 2024-05-19T18:20:02.3147884Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-05-19T18:20:02.3153642Z +++ nproc --ignore=2 2024-05-19T18:20:02.3190202Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9148947323 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:20:13.7120397Z + container_name=3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T18:20:13.7121605Z + grep download.pytorch.org /etc/hosts 2024-05-19T18:20:13.7123664Z + docker exec -i 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd sudo bash -c '/bin/cat >> /etc/hosts' 2024-05-19T18:20:13.7897176Z + echo DOCKER_CONTAINER_ID=3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T18:20:13.7898922Z ++ echo dist/torch-2.4.0a0+gitc8c7ddf-cp310-cp310-linux_x86_64.whl 2024-05-19T18:20:13.7900638Z + docker exec -t 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd sh -c 'pip install dist/torch-2.4.0a0+gitc8c7ddf-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-05-19T18:20:14.1659617Z Processing ./dist/torch-2.4.0a0+gitc8c7ddf-cp310-cp310-linux_x86_64.whl (from torch==2.4.0a0+gitc8c7ddf) 2024-05-19T18:20:14.4795705Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (3.13.1) 2024-05-19T18:20:14.4800761Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (4.11.0) 2024-05-19T18:20:14.4803408Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (1.12) 2024-05-19T18:20:14.4806376Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (2.8.8) 2024-05-19T18:20:14.4809120Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (3.1.4) 2024-05-19T18:20:14.4811687Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (2024.2.0) 2024-05-19T18:20:14.4837411Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (3.3.0) 2024-05-19T18:20:14.4894369Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (1.21.2) 2024-05-19T18:20:14.5321815Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (2.1.5) 2024-05-19T18:20:14.5475508Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (1.3.0) 2024-05-19T18:20:15.8386785Z Installing collected packages: torch 2024-05-19T18:20:24.5871076Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-05-19T18:20:24.5872635Z timm 0.9.7 requires torchvision, which is not installed. 2024-05-19T18:20:24.5873310Z Successfully installed torch-2.4.0a0+gitc8c7ddf 2024-05-19T18:20:24.6675529Z ++ dirname .ci/pytorch/test.sh 2024-05-19T18:20:24.6680184Z + source .ci/pytorch/common.sh 2024-05-19T18:20:24.6682933Z +++ dirname .ci/pytorch/common.sh 2024-05-19T18:20:24.6689577Z ++ source .ci/pytorch/common_utils.sh 2024-05-19T18:20:24.6690534Z +++ declare -f -t trap_add 2024-05-19T18:20:24.6695948Z ++ set -ex 2024-05-19T18:20:24.6696580Z ++ [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]] 2024-05-19T18:20:24.6697282Z ++ BUILD_TEST_LIBTORCH=0 2024-05-19T18:20:24.6698096Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *rocm* ]] 2024-05-19T18:20:24.6698992Z ++ stat -c %u /var/lib/jenkins/workspace 2024-05-19T18:20:24.6706348Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-05-19T18:20:24.6706884Z + trap_add cleanup_workspace EXIT 2024-05-19T18:20:24.6707292Z + trap_add_cmd=cleanup_workspace 2024-05-19T18:20:24.6707672Z + shift 2024-05-19T18:20:24.6707950Z + for trap_add_name in "$@" 2024-05-19T18:20:24.6711911Z +++ trap -p EXIT 2024-05-19T18:20:24.6713994Z ++ eval 'extract_trap_cmd ' 2024-05-19T18:20:24.6714507Z +++ extract_trap_cmd 2024-05-19T18:20:24.6714930Z +++ printf '%s\n' '' 2024-05-19T18:20:24.6715324Z ++ printf '%s\n' cleanup_workspace 2024-05-19T18:20:24.6716332Z + trap -- ' 2024-05-19T18:20:24.6716795Z cleanup_workspace' EXIT 2024-05-19T18:20:24.6717330Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-05-19T18:20:25.1026972Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-05-19T18:20:25.1038766Z + echo 'Environment variables:' 2024-05-19T18:20:25.1039182Z Environment variables: 2024-05-19T18:20:25.1039501Z + env 2024-05-19T18:20:25.1044337Z INSTALLED_DB=yes 2024-05-19T18:20:25.1045035Z NV_LIBCUBLAS_VERSION=12.4.2.65-1 2024-05-19T18:20:25.1045573Z NVIDIA_VISIBLE_DEVICES=all 2024-05-19T18:20:25.1045971Z NV_NVML_DEV_VERSION=12.4.99-1 2024-05-19T18:20:25.1046581Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T18:20:25.1047154Z CONTINUE_THROUGH_ERROR=False 2024-05-19T18:20:25.1047647Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.20.5-1+cuda12.4 2024-05-19T18:20:25.1051279Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.20.5-1 2024-05-19T18:20:25.1051903Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:20:25.1052581Z HOSTNAME=3f7c218bc62a 2024-05-19T18:20:25.1053458Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_f0a2b003-d93f-4cce-8389-ae3e3ce354bc 2024-05-19T18:20:25.1054588Z GITHUB_ACTION=__self 2024-05-19T18:20:25.1054990Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-05-19T18:20:25.1060571Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536 2024-05-19T18:20:25.1066323Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.2.65-1 2024-05-19T18:20:25.1066830Z NV_NVTX_VERSION=12.4.99-1 2024-05-19T18:20:25.1067443Z GITHUB_RUN_NUMBER=69546 2024-05-19T18:20:25.1067798Z TEST_CONFIG=dynamic_inductor_timm 2024-05-19T18:20:25.1068195Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-05-19T18:20:25.1068699Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-05-19T18:20:25.1069151Z NV_CUDA_CUDART_DEV_VERSION=12.4.99-1 2024-05-19T18:20:25.1069588Z NV_LIBCUSPARSE_VERSION=12.3.0.142-1 2024-05-19T18:20:25.1069996Z NV_LIBNPP_VERSION=12.2.5.2-1 2024-05-19T18:20:25.1070538Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-05-19T18:20:25.1071032Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-05-19T18:20:25.1071497Z GITHUB_REF_TYPE=tag 2024-05-19T18:20:25.1071812Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-05-19T18:20:25.1072187Z NCCL_VERSION=2.20.5-1 2024-05-19T18:20:25.1072583Z BASE_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:20:25.1073028Z XLA_CUDA= 2024-05-19T18:20:25.1073515Z HUGGING_FACE_HUB_TOKEN=*** 2024-05-19T18:20:25.1075333Z *** 2024-05-19T18:20:25.1075631Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-05-19T18:20:25.1076042Z GITHUB_REPOSITORY_ID=65600975 2024-05-19T18:20:25.1076396Z GITHUB_ACTIONS=true 2024-05-19T18:20:25.1076730Z NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:20:25.1077230Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.99-1 2024-05-19T18:20:25.1077757Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.2-1 2024-05-19T18:20:25.1078225Z SHA1=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:20:25.1078732Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-05-19T18:20:25.1079230Z GITHUB_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:20:25.1080029Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/121956 2024-05-19T18:20:25.1080764Z UCC_HOME=/usr 2024-05-19T18:20:25.1081097Z NV_LIBCUBLAS_DEV_VERSION=12.4.2.65-1 2024-05-19T18:20:25.1081502Z VERBOSE_TEST_LOGS=False 2024-05-19T18:20:25.1081838Z NVIDIA_PRODUCT_NAME=CUDA 2024-05-19T18:20:25.1082289Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4 2024-05-19T18:20:25.1082786Z GITHUB_REF=refs/tags/ciflow/inductor/121956 2024-05-19T18:20:25.1083251Z NV_CUDA_CUDART_VERSION=12.4.99-1 2024-05-19T18:20:25.1083620Z SHARD_NUMBER=1 2024-05-19T18:20:25.1083927Z GITHUB_REF_PROTECTED=false 2024-05-19T18:20:25.1084275Z HOME=/var/lib/jenkins 2024-05-19T18:20:25.1084762Z GITHUB_API_URL=https://api.github.com 2024-05-19T18:20:25.1085195Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-05-19T18:20:25.1085673Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-05-19T18:20:25.1086151Z SCCACHE_S3_KEY_PREFIX=inductor 2024-05-19T18:20:25.1086526Z CUDA_VERSION=12.4.0 2024-05-19T18:20:25.1086940Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.2.65-1 2024-05-19T18:20:25.1087381Z NUM_TEST_SHARDS=2 2024-05-19T18:20:25.1087681Z UCX_HOME=/usr 2024-05-19T18:20:25.1088182Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.0-1 2024-05-19T18:20:25.1089238Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_f0a2b003-d93f-4cce-8389-ae3e3ce354bc 2024-05-19T18:20:25.1090454Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:20:25.1091658Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_f0a2b003-d93f-4cce-8389-ae3e3ce354bc 2024-05-19T18:20:25.1092752Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-05-19T18:20:25.1093388Z GITHUB_EVENT_NAME=push 2024-05-19T18:20:25.1093718Z DASHBOARD_TAG= 2024-05-19T18:20:25.1094023Z GITHUB_RUN_ID=9148947323 2024-05-19T18:20:25.1094472Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.2-1 2024-05-19T18:20:25.1095002Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4 2024-05-19T18:20:25.1095970Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_f0a2b003-d93f-4cce-8389-ae3e3ce354bc 2024-05-19T18:20:25.1096864Z GITHUB_ACTOR=pytorch-bot[bot] 2024-05-19T18:20:25.1097258Z NV_LIBNPP_DEV_VERSION=12.2.5.2-1 2024-05-19T18:20:25.1097870Z PR_NUMBER= 2024-05-19T18:20:25.1098171Z GITHUB_RUN_ATTEMPT=1 2024-05-19T18:20:25.1098503Z ANACONDA_PYTHON_VERSION=3.10 2024-05-19T18:20:25.1099057Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-05-19T18:20:25.1099511Z TERM=xterm 2024-05-19T18:20:25.1099851Z NV_LIBCUSPARSE_DEV_VERSION=12.3.0.142-1 2024-05-19T18:20:25.1100264Z INSTALLED_VISION=yes 2024-05-19T18:20:25.1100568Z BRANCH= 2024-05-19T18:20:25.1100848Z OPENSSL_ROOT_DIR=/opt/openssl 2024-05-19T18:20:25.1101247Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-05-19T18:20:25.1101664Z CUDA_PATH=/usr/local/cuda 2024-05-19T18:20:25.1102473Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-05-19T18:20:25.1103218Z GITHUB_SERVER_URL=https://github.com 2024-05-19T18:20:25.1103703Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-05-19T18:20:25.1104156Z REENABLED_ISSUES= 2024-05-19T18:20:25.1104457Z SHLVL=1 2024-05-19T18:20:25.1104720Z MAX_JOBS=14 2024-05-19T18:20:25.1105040Z NV_CUDA_LIB_VERSION=12.4.0-1 2024-05-19T18:20:25.1105382Z NVARCH=x86_64 2024-05-19T18:20:25.1105684Z GITHUB_ACTOR_ID=54816060 2024-05-19T18:20:25.1106152Z GITHUB_WORKFLOW_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:20:25.1106677Z GITHUB_REF_NAME=ciflow/inductor/121956 2024-05-19T18:20:25.1107155Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4 2024-05-19T18:20:25.1107800Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-05-19T18:20:25.1108366Z GITHUB_JOB=test 2024-05-19T18:20:25.1108745Z NV_LIBNCCL_PACKAGE=libnccl2=2.20.5-1+cuda12.4 2024-05-19T18:20:25.1109302Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-05-19T18:20:25.1109805Z NO_TEST_TIMEOUT=False 2024-05-19T18:20:25.1110164Z TD_DISTRIBUTED=False 2024-05-19T18:20:25.1110554Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.0-1 2024-05-19T18:20:25.1110989Z GITHUB_REPOSITORY=pytorch/pytorch 2024-05-19T18:20:25.1111400Z NV_NVPROF_VERSION=12.4.99-1 2024-05-19T18:20:25.1111765Z GITHUB_RETENTION_DAYS=90 2024-05-19T18:20:25.1112107Z OPENSSL_DIR=/opt/openssl 2024-05-19T18:20:25.1112458Z GITHUB_ACTION_REPOSITORY= 2024-05-19T18:20:25.1113481Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-05-19T18:20:25.1114516Z GITHUB_BASE_REF= 2024-05-19T18:20:25.1114846Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-05-19T18:20:25.1115209Z CI=true 2024-05-19T18:20:25.1115519Z NV_LIBNCCL_PACKAGE_VERSION=2.20.5-1 2024-05-19T18:20:25.1115918Z GITHUB_REPOSITORY_OWNER=pytorch 2024-05-19T18:20:25.1116293Z JOB_ID=25153197487 2024-05-19T18:20:25.1116605Z INSTALLED_PROTOBUF=yes 2024-05-19T18:20:25.1116935Z GITHUB_HEAD_REF= 2024-05-19T18:20:25.1117229Z GITHUB_ACTION_REF= 2024-05-19T18:20:25.1117650Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-05-19T18:20:25.1118116Z GITHUB_WORKFLOW=inductor 2024-05-19T18:20:25.1118463Z DEBIAN_FRONTEND=noninteractive 2024-05-19T18:20:25.1119343Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_f0a2b003-d93f-4cce-8389-ae3e3ce354bc 2024-05-19T18:20:25.1120145Z NO_TD=False 2024-05-19T18:20:25.1120451Z SKIP_SCCACHE_INITIALIZATION=1 2024-05-19T18:20:25.1120806Z _=/usr/bin/env 2024-05-19T18:20:25.1121275Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-05-19T18:20:25.1189587Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-05-19T18:20:25.1190769Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-05-19T18:20:25.1191841Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-05-19T18:20:25.1192697Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-05-19T18:20:25.1193297Z + BUILD_DIR=build 2024-05-19T18:20:25.1193629Z + BUILD_RENAMED_DIR=build_renamed 2024-05-19T18:20:25.1194009Z + BUILD_BIN_DIR=build/bin 2024-05-19T18:20:25.1194345Z + SHARD_NUMBER=1 2024-05-19T18:20:25.1194718Z + NUM_TEST_SHARDS=2 2024-05-19T18:20:25.1195084Z + export VALGRIND=ON 2024-05-19T18:20:25.1195395Z + VALGRIND=ON 2024-05-19T18:20:25.1195862Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *clang9* ]] 2024-05-19T18:20:25.1196507Z + [[ 0 == \1 ]] 2024-05-19T18:20:25.1196805Z + [[ False == \1 ]] 2024-05-19T18:20:25.1197263Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *bazel* ]] 2024-05-19T18:20:25.1197780Z ++ realpath build/custom_test_artifacts 2024-05-19T18:20:25.1201378Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-05-19T18:20:25.1202052Z + [[ -n '' ]] 2024-05-19T18:20:25.1202498Z + echo 'Environment variables' 2024-05-19T18:20:25.1202886Z Environment variables 2024-05-19T18:20:25.1203201Z + env 2024-05-19T18:20:25.1206351Z INSTALLED_DB=yes 2024-05-19T18:20:25.1206820Z NV_LIBCUBLAS_VERSION=12.4.2.65-1 2024-05-19T18:20:25.1207318Z NVIDIA_VISIBLE_DEVICES=all 2024-05-19T18:20:25.1207878Z NV_NVML_DEV_VERSION=12.4.99-1 2024-05-19T18:20:25.1208694Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T18:20:25.1209382Z CONTINUE_THROUGH_ERROR=False 2024-05-19T18:20:25.1209973Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.20.5-1+cuda12.4 2024-05-19T18:20:25.1210572Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.20.5-1 2024-05-19T18:20:25.1211148Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:20:25.1211666Z HOSTNAME=3f7c218bc62a 2024-05-19T18:20:25.1212804Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_f0a2b003-d93f-4cce-8389-ae3e3ce354bc 2024-05-19T18:20:25.1213877Z GITHUB_ACTION=__self 2024-05-19T18:20:25.1214346Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-05-19T18:20:25.1219928Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536 2024-05-19T18:20:25.1225555Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.2.65-1 2024-05-19T18:20:25.1226055Z NV_NVTX_VERSION=12.4.99-1 2024-05-19T18:20:25.1226402Z GITHUB_RUN_NUMBER=69546 2024-05-19T18:20:25.1226756Z TEST_CONFIG=dynamic_inductor_timm 2024-05-19T18:20:25.1227158Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-05-19T18:20:25.1227613Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-05-19T18:20:25.1228067Z NV_CUDA_CUDART_DEV_VERSION=12.4.99-1 2024-05-19T18:20:25.1228502Z NV_LIBCUSPARSE_VERSION=12.3.0.142-1 2024-05-19T18:20:25.1228919Z NV_LIBNPP_VERSION=12.2.5.2-1 2024-05-19T18:20:25.1229344Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-05-19T18:20:25.1229829Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-05-19T18:20:25.1230288Z GITHUB_REF_TYPE=tag 2024-05-19T18:20:25.1230622Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-05-19T18:20:25.1230995Z NCCL_VERSION=2.20.5-1 2024-05-19T18:20:25.1231387Z BASE_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:20:25.1231828Z XLA_CUDA= 2024-05-19T18:20:25.1232279Z HUGGING_FACE_HUB_TOKEN=*** 2024-05-19T18:20:25.1232657Z *** 2024-05-19T18:20:25.1232939Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-05-19T18:20:25.1233332Z GITHUB_REPOSITORY_ID=65600975 2024-05-19T18:20:25.1233700Z GITHUB_ACTIONS=true 2024-05-19T18:20:25.1234023Z NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:20:25.1234499Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.99-1 2024-05-19T18:20:25.1235152Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.2-1 2024-05-19T18:20:25.1235626Z SHA1=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:20:25.1236123Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-05-19T18:20:25.1236605Z GITHUB_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:20:25.1237392Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/121956 2024-05-19T18:20:25.1238092Z UCC_HOME=/usr 2024-05-19T18:20:25.1238526Z NV_LIBCUBLAS_DEV_VERSION=12.4.2.65-1 2024-05-19T18:20:25.1238935Z VERBOSE_TEST_LOGS=False 2024-05-19T18:20:25.1239278Z NVIDIA_PRODUCT_NAME=CUDA 2024-05-19T18:20:25.1239720Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4 2024-05-19T18:20:25.1240266Z GITHUB_REF=refs/tags/ciflow/inductor/121956 2024-05-19T18:20:25.1240721Z NV_CUDA_CUDART_VERSION=12.4.99-1 2024-05-19T18:20:25.1241092Z SHARD_NUMBER=1 2024-05-19T18:20:25.1241395Z GITHUB_REF_PROTECTED=false 2024-05-19T18:20:25.1241743Z HOME=/var/lib/jenkins 2024-05-19T18:20:25.1242116Z GITHUB_API_URL=https://api.github.com 2024-05-19T18:20:25.1242548Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-05-19T18:20:25.1243020Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-05-19T18:20:25.1243496Z SCCACHE_S3_KEY_PREFIX=inductor 2024-05-19T18:20:25.1243864Z CUDA_VERSION=12.4.0 2024-05-19T18:20:25.1244269Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.2.65-1 2024-05-19T18:20:25.1244813Z NUM_TEST_SHARDS=2 2024-05-19T18:20:25.1245110Z UCX_HOME=/usr 2024-05-19T18:20:25.1245618Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.0-1 2024-05-19T18:20:25.1246658Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_f0a2b003-d93f-4cce-8389-ae3e3ce354bc 2024-05-19T18:20:25.1247860Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:20:25.1249042Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_f0a2b003-d93f-4cce-8389-ae3e3ce354bc 2024-05-19T18:20:25.1250132Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-05-19T18:20:25.1250771Z GITHUB_EVENT_NAME=push 2024-05-19T18:20:25.1251106Z DASHBOARD_TAG= 2024-05-19T18:20:25.1251407Z GITHUB_RUN_ID=9148947323 2024-05-19T18:20:25.1251852Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.2-1 2024-05-19T18:20:25.1252365Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4 2024-05-19T18:20:25.1253335Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_f0a2b003-d93f-4cce-8389-ae3e3ce354bc 2024-05-19T18:20:25.1254226Z GITHUB_ACTOR=pytorch-bot[bot] 2024-05-19T18:20:25.1254626Z NV_LIBNPP_DEV_VERSION=12.2.5.2-1 2024-05-19T18:20:25.1254989Z PR_NUMBER= 2024-05-19T18:20:25.1255272Z GITHUB_RUN_ATTEMPT=1 2024-05-19T18:20:25.1255586Z VALGRIND=ON 2024-05-19T18:20:25.1255884Z ANACONDA_PYTHON_VERSION=3.10 2024-05-19T18:20:25.1256314Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-05-19T18:20:25.1256764Z TERM=xterm 2024-05-19T18:20:25.1257099Z NV_LIBCUSPARSE_DEV_VERSION=12.3.0.142-1 2024-05-19T18:20:25.1257498Z INSTALLED_VISION=yes 2024-05-19T18:20:25.1257924Z BRANCH= 2024-05-19T18:20:25.1258207Z OPENSSL_ROOT_DIR=/opt/openssl 2024-05-19T18:20:25.1258604Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-05-19T18:20:25.1259017Z CUDA_PATH=/usr/local/cuda 2024-05-19T18:20:25.1259748Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-05-19T18:20:25.1260488Z GITHUB_SERVER_URL=https://github.com 2024-05-19T18:20:25.1260970Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-05-19T18:20:25.1261422Z REENABLED_ISSUES= 2024-05-19T18:20:25.1261714Z SHLVL=1 2024-05-19T18:20:25.1261974Z MAX_JOBS=14 2024-05-19T18:20:25.1262275Z NV_CUDA_LIB_VERSION=12.4.0-1 2024-05-19T18:20:25.1262621Z NVARCH=x86_64 2024-05-19T18:20:25.1262917Z GITHUB_ACTOR_ID=54816060 2024-05-19T18:20:25.1263372Z GITHUB_WORKFLOW_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:20:25.1263896Z GITHUB_REF_NAME=ciflow/inductor/121956 2024-05-19T18:20:25.1264460Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4 2024-05-19T18:20:25.1265103Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-05-19T18:20:25.1278195Z GITHUB_JOB=test 2024-05-19T18:20:25.1278645Z NV_LIBNCCL_PACKAGE=libnccl2=2.20.5-1+cuda12.4 2024-05-19T18:20:25.1279195Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-05-19T18:20:25.1279700Z NO_TEST_TIMEOUT=False 2024-05-19T18:20:25.1280131Z TD_DISTRIBUTED=False 2024-05-19T18:20:25.1280510Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.0-1 2024-05-19T18:20:25.1280945Z GITHUB_REPOSITORY=pytorch/pytorch 2024-05-19T18:20:25.1281359Z NV_NVPROF_VERSION=12.4.99-1 2024-05-19T18:20:25.1281711Z GITHUB_RETENTION_DAYS=90 2024-05-19T18:20:25.1282060Z OPENSSL_DIR=/opt/openssl 2024-05-19T18:20:25.1282408Z GITHUB_ACTION_REPOSITORY= 2024-05-19T18:20:25.1283430Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-05-19T18:20:25.1284565Z GITHUB_BASE_REF= 2024-05-19T18:20:25.1284897Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-05-19T18:20:25.1285266Z CI=true 2024-05-19T18:20:25.1285574Z NV_LIBNCCL_PACKAGE_VERSION=2.20.5-1 2024-05-19T18:20:25.1285980Z GITHUB_REPOSITORY_OWNER=pytorch 2024-05-19T18:20:25.1286351Z JOB_ID=25153197487 2024-05-19T18:20:25.1286667Z INSTALLED_PROTOBUF=yes 2024-05-19T18:20:25.1286990Z GITHUB_HEAD_REF= 2024-05-19T18:20:25.1287295Z GITHUB_ACTION_REF= 2024-05-19T18:20:25.1287728Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-05-19T18:20:25.1288196Z GITHUB_WORKFLOW=inductor 2024-05-19T18:20:25.1288549Z DEBIAN_FRONTEND=noninteractive 2024-05-19T18:20:25.1289437Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_f0a2b003-d93f-4cce-8389-ae3e3ce354bc 2024-05-19T18:20:25.1290245Z NO_TD=False 2024-05-19T18:20:25.1290553Z SKIP_SCCACHE_INITIALIZATION=1 2024-05-19T18:20:25.1290911Z _=/usr/bin/env 2024-05-19T18:20:25.1291248Z + echo 'Testing pytorch' 2024-05-19T18:20:25.1291597Z Testing pytorch 2024-05-19T18:20:25.1291913Z + export LANG=C.UTF-8 2024-05-19T18:20:25.1292252Z + LANG=C.UTF-8 2024-05-19T18:20:25.1292544Z + PR_NUMBER= 2024-05-19T18:20:25.1292889Z + [[ dynamic_inductor_timm == \d\e\f\a\u\l\t ]] 2024-05-19T18:20:25.1293393Z + [[ dynamic_inductor_timm == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-05-19T18:20:25.1293893Z + [[ dynamic_inductor_timm == \s\l\o\w ]] 2024-05-19T18:20:25.1294514Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-05-19T18:20:25.1295188Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *cuda* ]] 2024-05-19T18:20:25.1295710Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-05-19T18:20:25.1296175Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-05-19T18:20:25.1296623Z + [[ dynamic_inductor_timm == *crossref* ]] 2024-05-19T18:20:25.1297177Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]] 2024-05-19T18:20:25.1297932Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *xpu* ]] 2024-05-19T18:20:25.1298567Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-05-19T18:20:25.1299123Z + pip_install --user ninja==1.10.2 2024-05-19T18:20:25.1299636Z + pip install --progress-bar off --user ninja==1.10.2 2024-05-19T18:20:25.4817020Z Collecting ninja==1.10.2 2024-05-19T18:20:25.4998710Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-05-19T18:20:25.5147416Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-05-19T18:20:26.7847136Z Installing collected packages: ninja 2024-05-19T18:20:26.7917779Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-05-19T18:20:26.7920207Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-05-19T18:20:26.7957793Z Successfully installed ninja-1.10.2 2024-05-19T18:20:26.8808280Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-05-19T18:20:26.8810644Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-05-19T18:20:26.8812020Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]] 2024-05-19T18:20:26.8812518Z + install_tlparse 2024-05-19T18:20:26.8812984Z + pip_install --user tlparse==0.3.7 2024-05-19T18:20:26.8813517Z + pip install --progress-bar off --user tlparse==0.3.7 2024-05-19T18:20:27.2281652Z Collecting tlparse==0.3.7 2024-05-19T18:20:27.2431076Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-05-19T18:20:27.2535065Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-05-19T18:20:28.5371395Z Installing collected packages: tlparse 2024-05-19T18:20:28.5728324Z Successfully installed tlparse-0.3.7 2024-05-19T18:20:28.6566316Z ++ python -m site --user-base 2024-05-19T18:20:28.6717932Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-05-19T18:20:28.6719517Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *asan* ]] 2024-05-19T18:20:28.6720231Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-debug* ]] 2024-05-19T18:20:28.6720889Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-05-19T18:20:28.6721805Z + echo 'We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-05-19T18:20:28.6723068Z We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass 2024-05-19T18:20:28.6723916Z + cd test 2024-05-19T18:20:28.6724583Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-05-19T18:20:30.0422547Z + [[ dynamic_inductor_timm == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-05-19T18:20:30.0423186Z + [[ dynamic_inductor_timm == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-05-19T18:20:30.0425399Z + DYNAMO_BENCHMARK_FLAGS=() 2024-05-19T18:20:30.0425980Z + [[ dynamic_inductor_timm == *dynamo_eager* ]] 2024-05-19T18:20:30.0426471Z + [[ dynamic_inductor_timm == *aot_eager* ]] 2024-05-19T18:20:30.0426958Z + [[ dynamic_inductor_timm == *aot_inductor* ]] 2024-05-19T18:20:30.0427458Z + [[ dynamic_inductor_timm == *inductor* ]] 2024-05-19T18:20:30.0427905Z + [[ dynamic_inductor_timm != *perf* ]] 2024-05-19T18:20:30.0428523Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-05-19T18:20:30.0428971Z + [[ dynamic_inductor_timm == *dynamic* ]] 2024-05-19T18:20:30.0429589Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2024-05-19T18:20:30.0430163Z + [[ dynamic_inductor_timm == *cpu_inductor* ]] 2024-05-19T18:20:30.0430676Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-05-19T18:20:30.0438473Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *tbb* ]] 2024-05-19T18:20:30.0453108Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]] 2024-05-19T18:20:30.0453871Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-05-19T18:20:30.0454371Z + cd test 2024-05-19T18:20:30.0454818Z + python -c 'import torch; print(torch.__config__.show())' 2024-05-19T18:20:31.2763558Z PyTorch built with: 2024-05-19T18:20:31.2764311Z - GCC 9.4 2024-05-19T18:20:31.2764885Z - C++ Version: 201703 2024-05-19T18:20:31.2765818Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-05-19T18:20:31.2766878Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-05-19T18:20:31.2767533Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-05-19T18:20:31.2768030Z - LAPACK is enabled (usually provided by MKL) 2024-05-19T18:20:31.2768508Z - NNPACK is enabled 2024-05-19T18:20:31.2768889Z - CPU capability usage: AVX2 2024-05-19T18:20:31.2769550Z - CUDA Runtime 12.4 2024-05-19T18:20:31.2770065Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-05-19T18:20:31.2770665Z - CuDNN 8.9.7 (built against CUDA 12.2) 2024-05-19T18:20:31.2771117Z - Magma 2.6.1 2024-05-19T18:20:31.2778399Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.4, CUDNN_VERSION=8.9.7, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.4.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-05-19T18:20:31.2784893Z 2024-05-19T18:20:31.5562448Z + cd test 2024-05-19T18:20:31.5563165Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-05-19T18:20:32.6621692Z ATen/Parallel: 2024-05-19T18:20:32.6622263Z at::get_num_threads() : 8 2024-05-19T18:20:32.6622795Z at::get_num_interop_threads() : 16 2024-05-19T18:20:32.6623261Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-05-19T18:20:32.6623662Z omp_get_max_threads() : 8 2024-05-19T18:20:32.6624662Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-05-19T18:20:32.6625446Z mkl_get_max_threads() : 8 2024-05-19T18:20:32.6626055Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-05-19T18:20:32.6626680Z std::thread::hardware_concurrency() : 16 2024-05-19T18:20:32.6627107Z Environment variables: 2024-05-19T18:20:32.6627461Z OMP_NUM_THREADS : [not set] 2024-05-19T18:20:32.6627836Z MKL_NUM_THREADS : [not set] 2024-05-19T18:20:32.6628209Z ATen parallel backend: OpenMP 2024-05-19T18:20:32.6628459Z 2024-05-19T18:20:32.9127483Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]] 2024-05-19T18:20:32.9128207Z + [[ dynamic_inductor_timm == *backward* ]] 2024-05-19T18:20:32.9128791Z + [[ dynamic_inductor_timm == *xla* ]] 2024-05-19T18:20:32.9129246Z + [[ dynamic_inductor_timm == *executorch* ]] 2024-05-19T18:20:32.9129751Z + [[ dynamic_inductor_timm == \j\i\t\_\l\e\g\a\c\y ]] 2024-05-19T18:20:32.9130390Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]] 2024-05-19T18:20:32.9130940Z + [[ dynamic_inductor_timm == distributed ]] 2024-05-19T18:20:32.9131474Z + [[ dynamic_inductor_timm == deploy ]] 2024-05-19T18:20:32.9132027Z + [[ dynamic_inductor_timm == *inductor_distributed* ]] 2024-05-19T18:20:32.9132722Z + [[ dynamic_inductor_timm == *inductor-micro-benchmark* ]] 2024-05-19T18:20:32.9133319Z + [[ dynamic_inductor_timm == *huggingface* ]] 2024-05-19T18:20:32.9133794Z + [[ dynamic_inductor_timm == *timm* ]] 2024-05-19T18:20:32.9134259Z + install_torchvision 2024-05-19T18:20:32.9134592Z + local orig_preload 2024-05-19T18:20:32.9134956Z + local commit 2024-05-19T18:20:32.9135282Z ++ get_pinned_commit vision 2024-05-19T18:20:32.9135697Z ++ cat .github/ci_commit_pins/vision.txt 2024-05-19T18:20:32.9141958Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:20:32.9142709Z + orig_preload= 2024-05-19T18:20:32.9143107Z + '[' -n '' ']' 2024-05-19T18:20:32.9143927Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:20:32.9145552Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:20:33.2247707Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:20:33.2252713Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-mjwf8pcc 2024-05-19T18:20:33.2269035Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-mjwf8pcc 2024-05-19T18:20:34.5598049Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-05-19T18:20:34.5615939Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:20:35.7677156Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:20:36.0292790Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:20:38.1935831Z Preparing metadata (setup.py) ... [?25l- \ done 2024-05-19T18:20:38.1982526Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-05-19T18:20:38.1985420Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.4.0a0+gitc8c7ddf) 2024-05-19T18:20:38.1990385Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-05-19T18:20:38.2190831Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-05-19T18:20:38.2197085Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.11.0) 2024-05-19T18:20:38.2199322Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12) 2024-05-19T18:20:38.2202461Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-05-19T18:20:38.2205377Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-05-19T18:20:38.2208253Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-05-19T18:20:38.2620034Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-05-19T18:20:38.2765480Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-05-19T18:20:38.2838219Z Building wheels for collected packages: torchvision 2024-05-19T18:21:48.9236148Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done 2024-05-19T18:21:48.9263546Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2026162 sha256=d9769181edf7ac15614a7ebd35be5a045b5bf25b587fd1ec96a244944ac44971 2024-05-19T18:21:48.9265351Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-05-19T18:21:48.9299639Z Successfully built torchvision 2024-05-19T18:21:50.1246715Z Installing collected packages: torchvision 2024-05-19T18:21:50.5202062Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-05-19T18:21:50.6338941Z + '[' -n '' ']' 2024-05-19T18:21:50.6339290Z + id=0 2024-05-19T18:21:50.6339602Z + test_dynamo_benchmark timm_models 0 2024-05-19T18:21:50.6341653Z ++ pwd 2024-05-19T18:21:50.6344413Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-05-19T18:21:50.6345986Z + local suite=timm_models 2024-05-19T18:21:50.6346389Z + shift 2024-05-19T18:21:50.6346667Z + local shard_id=0 2024-05-19T18:21:50.6347335Z + shift 2024-05-19T18:21:50.6347681Z + [[ dynamic_inductor_timm == *perf_compare* ]] 2024-05-19T18:21:50.6348158Z + [[ dynamic_inductor_timm == *perf* ]] 2024-05-19T18:21:50.6348736Z + [[ dynamic_inductor_timm == *cpu_inductor* ]] 2024-05-19T18:21:50.6349396Z + [[ dynamic_inductor_timm == *aot_inductor* ]] 2024-05-19T18:21:50.6350259Z + test_single_dynamo_benchmark inference timm_models 0 --inference --bfloat16 2024-05-19T18:21:50.6350918Z ++ pwd 2024-05-19T18:21:50.6351447Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-05-19T18:21:50.6352125Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-05-19T18:21:50.6361991Z + local name=inference 2024-05-19T18:21:50.6362440Z + shift 2024-05-19T18:21:50.6362807Z + local suite=timm_models 2024-05-19T18:21:50.6363250Z + shift 2024-05-19T18:21:50.6363605Z + local shard_id=0 2024-05-19T18:21:50.6364004Z + shift 2024-05-19T18:21:50.6364347Z + partition_flags=() 2024-05-19T18:21:50.6364787Z + local partition_flags 2024-05-19T18:21:50.6365173Z + [[ -n 2 ]] 2024-05-19T18:21:50.6365483Z + [[ -n 0 ]] 2024-05-19T18:21:50.6366049Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-05-19T18:21:50.6366726Z + [[ dynamic_inductor_timm == *perf_compare* ]] 2024-05-19T18:21:50.6367197Z + [[ dynamic_inductor_timm == *perf* ]] 2024-05-19T18:21:50.6367662Z + [[ dynamic_inductor_timm == *aot_inductor* ]] 2024-05-19T18:21:50.6369419Z + python benchmarks/dynamo/timm_models.py --ci --accuracy --timing --explain --inductor --dynamic-shapes --dynamic-batch-only --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_timm_models.csv 2024-05-19T18:21:55.9989179Z 2024-05-19T18:21:56.4119500Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:21:56.4119858Z 2024-05-19T18:21:56.5311237Z model.safetensors: 0% 0.00/95.5M [00:00 will be ignored 2024-05-19T18:42:57.8653667Z E0519 18:42:57.862000 140529272689280 torch/_dynamo/utils.py:1392] RMSE (res-fp64): 0.01333, (ref-fp64): 0.00256 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.010000 2024-05-19T18:42:57.8655363Z E0519 18:42:57.862000 140529272689280 torch/_dynamo/utils.py:1306] Accuracy failed for key name blocks.0.attn.proj.bias.grad 2024-05-19T18:42:57.8663498Z fail_accuracy 2024-05-19T18:42:57.9158669Z TIMING: entire_frame_compile:100.00842 code_gen:27.30379 inductor_compile:54.77576 backend_compile:85.65148 2024-05-19T18:42:57.9160141Z STATS: call_* op count: 1054 | FakeTensor.__torch_dispatch__:15454 | FakeTensorMode.__torch_dispatch__:108695 | attempt fast:2534 | fast is_contiguous:2534 | ProxyTorchDispatchMode.__torch_dispatch__:21953 2024-05-19T18:42:57.9161453Z Dynamo produced 3 graphs covering 1054 ops with 7 graph breaks (5 unique) 2024-05-19T18:43:05.6881966Z 2024-05-19T18:43:06.7506681Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:43:06.7508413Z loading model: 0it [00:01, ?it/s] 2024-05-19T18:43:06.7508888Z cuda train botnet26t_256 2024-05-19T18:44:18.0241896Z pass 2024-05-19T18:44:18.0344502Z TIMING: entire_frame_compile:64.67945 code_gen:20.84902 inductor_compile:38.70596 backend_compile:56.88767 2024-05-19T18:44:18.0347635Z STATS: call_* op count: 374 | FakeTensor.__torch_dispatch__:5789 | FakeTensorMode.__torch_dispatch__:43433 | attempt fast:1265 | fast is_contiguous:1239 | slow no contiguity match:26 | ProxyTorchDispatchMode.__torch_dispatch__:9047 2024-05-19T18:44:18.0350479Z Dynamo produced 2 graphs covering 374 ops with 6 graph breaks (5 unique) 2024-05-19T18:44:23.7335530Z 2024-05-19T18:44:27.5956404Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:44:27.5957087Z loading model: 0it [00:03, ?it/s] 2024-05-19T18:44:27.5957731Z cuda train cait_m36_384 2024-05-19T18:44:27.5967920Z Traceback (most recent call last): 2024-05-19T18:44:27.5968671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2280, in validate_model 2024-05-19T18:44:27.5969417Z self.model_iter_fn(model, example_inputs) 2024-05-19T18:44:27.5970253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 365, in forward_and_backward_pass 2024-05-19T18:44:27.5971054Z pred = mod(*cloned_inputs) 2024-05-19T18:44:27.5972337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T18:44:27.5973498Z return self._call_impl(*args, **kwargs) 2024-05-19T18:44:27.5974736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T18:44:27.5975670Z return forward_call(*args, **kwargs) 2024-05-19T18:44:27.5976775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 361, in forward 2024-05-19T18:44:27.5977965Z x = self.forward_features(x) 2024-05-19T18:44:27.5978862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 346, in forward_features 2024-05-19T18:44:27.5979639Z x = self.blocks(x) 2024-05-19T18:44:27.5980695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T18:44:27.5981829Z return self._call_impl(*args, **kwargs) 2024-05-19T18:44:27.5983088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T18:44:27.5984063Z return forward_call(*args, **kwargs) 2024-05-19T18:44:27.5984961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/container.py", line 217, in forward 2024-05-19T18:44:27.5985752Z input = module(input) 2024-05-19T18:44:27.5986766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T18:44:27.5987633Z return self._call_impl(*args, **kwargs) 2024-05-19T18:44:27.5988535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T18:44:27.5989349Z return forward_call(*args, **kwargs) 2024-05-19T18:44:27.5990167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 197, in forward 2024-05-19T18:44:27.5991021Z x = x + self.drop_path(self.gamma_1 * self.attn(self.norm1(x))) 2024-05-19T18:44:27.5992067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T18:44:27.5992944Z return self._call_impl(*args, **kwargs) 2024-05-19T18:44:27.5993843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T18:44:27.5994657Z return forward_call(*args, **kwargs) 2024-05-19T18:44:27.5995477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 148, in forward 2024-05-19T18:44:27.5996339Z attn = self.proj_w(attn.permute(0, 2, 3, 1)).permute(0, 3, 1, 2) 2024-05-19T18:44:27.5997414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T18:44:27.5998325Z return self._call_impl(*args, **kwargs) 2024-05-19T18:44:27.5999231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T18:44:27.6000054Z return forward_call(*args, **kwargs) 2024-05-19T18:44:27.6000916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 116, in forward 2024-05-19T18:44:27.6001754Z return F.linear(input, self.weight, self.bias) 2024-05-19T18:44:27.6005110Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 82.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 66.44 MiB is free. Process 10523 has 21.90 GiB memory in use. Of the allocated memory 21.13 GiB is allocated by PyTorch, and 430.66 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-05-19T18:44:27.6007779Z 2024-05-19T18:44:27.6008081Z The above exception was the direct cause of the following exception: 2024-05-19T18:44:27.6008525Z 2024-05-19T18:44:27.6008673Z Traceback (most recent call last): 2024-05-19T18:44:27.6009316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4109, in run 2024-05-19T18:44:27.6009962Z ) = runner.load_model( 2024-05-19T18:44:27.6010612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 297, in load_model 2024-05-19T18:44:27.6011360Z self.validate_model(model, example_inputs) 2024-05-19T18:44:27.6012111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2282, in validate_model 2024-05-19T18:44:27.6012859Z raise RuntimeError("Eager run failed") from e 2024-05-19T18:44:27.6013324Z RuntimeError: Eager run failed 2024-05-19T18:44:27.6013576Z 2024-05-19T18:44:27.6013691Z eager_fail_to_run 2024-05-19T18:44:30.8731499Z 2024-05-19T18:44:31.7531483Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:44:31.7531948Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:44:31.7532711Z cuda train coat_lite_mini 2024-05-19T18:46:18.0582314Z pass 2024-05-19T18:46:18.0695896Z TIMING: entire_frame_compile:99.68969 code_gen:26.05072 inductor_compile:57.60437 backend_compile:87.94391 2024-05-19T18:46:18.0698188Z STATS: call_* op count: 538 | FakeTensor.__torch_dispatch__:6887 | FakeTensorMode.__torch_dispatch__:65997 | attempt fast:2289 | fast is_contiguous:2091 | slow no contiguity match:198 | ProxyTorchDispatchMode.__torch_dispatch__:14338 2024-05-19T18:46:18.0700073Z Dynamo produced 2 graphs covering 538 ops with 6 graph breaks (5 unique) 2024-05-19T18:46:26.2791089Z 2024-05-19T18:46:27.9432786Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:46:27.9433407Z loading model: 0it [00:01, ?it/s] 2024-05-19T18:46:27.9433926Z cuda train convit_base 2024-05-19T18:46:27.9938116Z WARNING:common:fp64 golden ref were not generated for convit_base. Setting accuracy check to cosine 2024-05-19T18:49:30.3583476Z pass 2024-05-19T18:49:30.3807848Z TIMING: entire_frame_compile:175.90384 code_gen:30.63344 inductor_compile:88.05101 backend_compile:156.04899 2024-05-19T18:49:30.3810741Z STATS: call_* op count: 1308 | FakeTensor.__torch_dispatch__:16979 | FakeTensorMode.__torch_dispatch__:164307 | attempt fast:7025 | slow no contiguity match:310 | fast is_contiguous:6715 | ProxyTorchDispatchMode.__torch_dispatch__:40289 2024-05-19T18:49:30.3812523Z Dynamo produced 3 graphs covering 1308 ops with 7 graph breaks (5 unique) 2024-05-19T18:49:41.4648770Z 2024-05-19T18:49:42.3895495Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:49:42.3897980Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:49:42.3898512Z cuda train convmixer_768_32 2024-05-19T18:50:55.3860962Z pass 2024-05-19T18:50:55.4008816Z TIMING: entire_frame_compile:62.49897 code_gen:12.57748 inductor_compile:35.22025 backend_compile:55.8899 2024-05-19T18:50:55.4011524Z STATS: call_* op count: 235 | FakeTensor.__torch_dispatch__:12373 | FakeTensorMode.__torch_dispatch__:73702 | attempt fast:2381 | fast is_contiguous:2381 | ProxyTorchDispatchMode.__torch_dispatch__:15494 2024-05-19T18:50:55.4012883Z Dynamo produced 2 graphs covering 235 ops with 5 graph breaks (4 unique) 2024-05-19T18:51:01.4540057Z 2024-05-19T18:51:03.5889868Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:51:03.5890342Z loading model: 0it [00:02, ?it/s] 2024-05-19T18:51:03.5890765Z cuda train convnext_base 2024-05-19T18:52:45.8376017Z W0519 18:52:45.836000 140404134732416 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T18:54:16.7627229Z pass 2024-05-19T18:54:16.8104579Z TIMING: entire_frame_compile:177.62728 code_gen:44.78232 inductor_compile:97.49564 backend_compile:150.21538 2024-05-19T18:54:16.8106734Z STATS: call_* op count: 1621 | FakeTensor.__torch_dispatch__:28205 | FakeTensorMode.__torch_dispatch__:180568 | attempt fast:4737 | fast is_contiguous:4095 | fast channels_last:353 | slow no contiguity match:289 | ProxyTorchDispatchMode.__torch_dispatch__:35464 2024-05-19T18:54:16.8108325Z Dynamo produced 3 graphs covering 1621 ops with 7 graph breaks (5 unique) 2024-05-19T18:54:28.1346496Z 2024-05-19T18:54:29.0303721Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:54:29.0304194Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:54:29.0308045Z cuda train crossvit_9_240 2024-05-19T18:56:25.2668707Z W0519 18:56:25.265000 139789048509056 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T18:57:36.2822016Z pass 2024-05-19T18:57:36.2954394Z TIMING: entire_frame_compile:174.8966 code_gen:43.73902 inductor_compile:98.21898 backend_compile:152.09267 2024-05-19T18:57:36.2956390Z STATS: call_* op count: 1441 | FakeTensor.__torch_dispatch__:24421 | FakeTensorMode.__torch_dispatch__:170432 | attempt fast:4735 | fast is_contiguous:4687 | slow no contiguity match:48 | ProxyTorchDispatchMode.__torch_dispatch__:34514 2024-05-19T18:57:36.2958301Z Dynamo produced 3 graphs covering 1441 ops with 7 graph breaks (5 unique) 2024-05-19T18:57:47.6073739Z 2024-05-19T18:57:48.8660143Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:57:48.8660640Z loading model: 0it [00:01, ?it/s] 2024-05-19T18:57:48.8661087Z cuda train cspdarknet53 2024-05-19T18:59:25.0663175Z W0519 18:59:25.065000 139734911869568 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:00:22.7519375Z pass 2024-05-19T19:00:22.7679539Z TIMING: entire_frame_compile:140.40473 code_gen:36.30216 inductor_compile:79.47284 backend_compile:117.98941 2024-05-19T19:00:22.7683862Z STATS: call_* op count: 1070 | FakeTensor.__torch_dispatch__:22312 | FakeTensorMode.__torch_dispatch__:127541 | attempt fast:3258 | fast is_contiguous:3233 | slow no contiguity match:25 | ProxyTorchDispatchMode.__torch_dispatch__:24624 2024-05-19T19:00:22.7685506Z Dynamo produced 3 graphs covering 1070 ops with 7 graph breaks (5 unique) 2024-05-19T19:00:32.0791053Z 2024-05-19T19:00:33.6639941Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:00:33.6640514Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:00:33.6643087Z cuda train deit_base_distilled_patch16_224 2024-05-19T19:01:27.4365113Z W0519 19:01:27.435000 140093912875648 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:02:07.6716790Z pass 2024-05-19T19:02:07.7233624Z TIMING: entire_frame_compile:83.61899 code_gen:21.74878 inductor_compile:46.85283 backend_compile:72.21127 2024-05-19T19:02:07.7235989Z STATS: call_* op count: 869 | FakeTensor.__torch_dispatch__:13576 | FakeTensorMode.__torch_dispatch__:90443 | attempt fast:2351 | fast is_contiguous:2351 | ProxyTorchDispatchMode.__torch_dispatch__:18136 2024-05-19T19:02:07.7237302Z Dynamo produced 3 graphs covering 869 ops with 7 graph breaks (5 unique) 2024-05-19T19:02:14.8446387Z 2024-05-19T19:02:16.2632011Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:02:16.2632638Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:02:16.2635010Z cuda train dla102 2024-05-19T19:04:17.0099418Z W0519 19:04:17.009000 140383804023424 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:05:43.7930667Z pass 2024-05-19T19:05:43.8150490Z TIMING: entire_frame_compile:190.46136 code_gen:56.54804 inductor_compile:112.38055 backend_compile:167.81767 2024-05-19T19:05:43.8154706Z STATS: call_* op count: 1389 | FakeTensor.__torch_dispatch__:33508 | FakeTensorMode.__torch_dispatch__:191569 | attempt fast:3729 | fast is_contiguous:3651 | ProxyTorchDispatchMode.__torch_dispatch__:36994 | slow no contiguity match:78 2024-05-19T19:05:43.8157493Z Dynamo produced 3 graphs covering 1389 ops with 7 graph breaks (5 unique) 2024-05-19T19:05:54.9220537Z 2024-05-19T19:05:56.7017058Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:05:56.7017540Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:05:56.7018188Z cuda train dm_nfnet_f0 2024-05-19T19:07:22.9471665Z pass 2024-05-19T19:07:22.9967818Z TIMING: entire_frame_compile:78.09569 code_gen:18.48933 inductor_compile:44.274 backend_compile:65.71199 2024-05-19T19:07:22.9970523Z STATS: call_* op count: 620 | FakeTensorMode.__torch_dispatch__:67599 | attempt fast:2920 | fast is_contiguous:2920 | FakeTensor.__torch_dispatch__:11878 | ProxyTorchDispatchMode.__torch_dispatch__:16005 2024-05-19T19:07:22.9971846Z Dynamo produced 2 graphs covering 620 ops with 6 graph breaks (5 unique) 2024-05-19T19:07:29.7599880Z 2024-05-19T19:07:31.4550888Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:07:31.4551531Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:07:31.4552024Z cuda train dpn107 2024-05-19T19:10:37.2711106Z pass 2024-05-19T19:10:37.3348611Z TIMING: entire_frame_compile:165.88504 code_gen:41.37313 inductor_compile:94.17002 backend_compile:145.38476 2024-05-19T19:10:37.3350949Z STATS: call_* op count: 746 | FakeTensor.__torch_dispatch__:18918 | FakeTensorMode.__torch_dispatch__:131058 | attempt fast:4114 | fast is_contiguous:3780 | slow no contiguity match:334 | ProxyTorchDispatchMode.__torch_dispatch__:28636 2024-05-19T19:10:37.3352418Z Dynamo produced 2 graphs covering 746 ops with 6 graph breaks (5 unique) 2024-05-19T19:10:47.2532873Z 2024-05-19T19:10:48.2809803Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:10:48.2810832Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:10:48.2813932Z cuda train eca_botnext26ts_256 2024-05-19T19:12:07.4553215Z W0519 19:12:07.454000 140586084684416 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:12:39.5439384Z pass 2024-05-19T19:12:39.5553385Z TIMING: entire_frame_compile:101.50363 code_gen:29.82838 inductor_compile:58.95201 backend_compile:86.95411 2024-05-19T19:12:39.5555713Z STATS: call_* op count: 760 | FakeTensor.__torch_dispatch__:12563 | FakeTensorMode.__torch_dispatch__:80862 | attempt fast:1839 | fast is_contiguous:1777 | slow no contiguity match:62 | ProxyTorchDispatchMode.__torch_dispatch__:15134 2024-05-19T19:12:39.5557189Z Dynamo produced 3 graphs covering 760 ops with 7 graph breaks (5 unique) 2024-05-19T19:12:47.0599283Z 2024-05-19T19:12:48.1290745Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:12:48.1291221Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:12:48.1292952Z cuda train eca_halonext26ts 2024-05-19T19:14:16.2889307Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T19:14:16.2891570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 365, in torch_dynamo_resume_in_forward_and_backward_pass_at_363 2024-05-19T19:14:16.2892657Z pred = mod(*cloned_inputs) 2024-05-19T19:14:16.2893633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:14:16.2894459Z return forward_call(*args, **kwargs) 2024-05-19T19:14:16.2895342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/byobnet.py", line 1257, in forward 2024-05-19T19:14:16.2896104Z x = self.forward_features(x) 2024-05-19T19:14:16.2897153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/byobnet.py", line 1249, in forward_features 2024-05-19T19:14:16.2898392Z x = self.stages(x) 2024-05-19T19:14:16.2899255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:14:16.2900070Z return forward_call(*args, **kwargs) 2024-05-19T19:14:16.2900905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/byobnet.py", line 883, in forward 2024-05-19T19:14:16.2901640Z x = self.self_attn(x) 2024-05-19T19:14:16.2902460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:14:16.2903276Z return forward_call(*args, **kwargs) 2024-05-19T19:14:16.2906784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/layers/halo_attn.py", line 191, in forward 2024-05-19T19:14:16.2908071Z kv = kv.unfold(2, self.win_size, self.block_size).unfold(3, self.win_size, self.block_size).reshape( 2024-05-19T19:14:16.2908644Z 2024-05-19T19:14:23.5803589Z W0519 19:14:23.579000 140297505149568 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:14:55.7634517Z pass 2024-05-19T19:14:55.7705353Z TIMING: entire_frame_compile:118.56026 code_gen:38.63838 inductor_compile:72.31423 backend_compile:102.36428 2024-05-19T19:14:55.7707465Z STATS: call_* op count: 790 | FakeTensor.__torch_dispatch__:12334 | FakeTensorMode.__torch_dispatch__:83566 | attempt fast:1832 | fast is_contiguous:1770 | slow no contiguity match:62 | ProxyTorchDispatchMode.__torch_dispatch__:15740 2024-05-19T19:14:55.7708920Z Dynamo produced 3 graphs covering 790 ops with 7 graph breaks (5 unique) 2024-05-19T19:15:03.7646108Z 2024-05-19T19:15:04.6613007Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:15:04.6613477Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:15:04.6613899Z cuda train ese_vovnet19b_dw 2024-05-19T19:15:55.4131111Z W0519 19:15:55.412000 140160904438400 torch/_logging/_internal.py:1024] [7/0] Profiler function will be ignored 2024-05-19T19:16:24.8944920Z pass 2024-05-19T19:16:24.9015655Z TIMING: entire_frame_compile:72.31942 code_gen:29.17834 inductor_compile:46.81889 backend_compile:62.18521 2024-05-19T19:16:24.9017495Z STATS: call_* op count: 459 | FakeTensor.__torch_dispatch__:9284 | FakeTensorMode.__torch_dispatch__:55302 | attempt fast:927 | fast is_contiguous:903 | ProxyTorchDispatchMode.__torch_dispatch__:10392 | slow no contiguity match:24 2024-05-19T19:16:24.9019113Z Dynamo produced 4 graphs covering 459 ops with 7 graph breaks (5 unique) 2024-05-19T19:16:30.8683353Z 2024-05-19T19:16:31.8500888Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:16:31.8501364Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:16:31.8501794Z cuda train fbnetc_100 2024-05-19T19:17:45.0439501Z W0519 19:17:45.043000 139964487656064 torch/_logging/_internal.py:1024] [7/0] Profiler function will be ignored 2024-05-19T19:18:42.1848697Z pass 2024-05-19T19:18:42.1890638Z TIMING: entire_frame_compile:118.25164 code_gen:35.42789 inductor_compile:67.89927 backend_compile:100.12611 2024-05-19T19:18:42.1894018Z STATS: call_* op count: 993 | FakeTensor.__torch_dispatch__:19911 | FakeTensorMode.__torch_dispatch__:114360 | attempt fast:2264 | fast is_contiguous:2264 | ProxyTorchDispatchMode.__torch_dispatch__:22083 2024-05-19T19:18:42.1896644Z Dynamo produced 4 graphs covering 993 ops with 7 graph breaks (5 unique) 2024-05-19T19:18:50.5589700Z 2024-05-19T19:18:51.7220526Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:18:51.7221011Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:18:51.7225000Z cuda train fbnetv3_b 2024-05-19T19:21:26.9362046Z pass 2024-05-19T19:21:26.9439351Z TIMING: entire_frame_compile:144.1543 code_gen:29.64718 inductor_compile:78.45494 backend_compile:123.30793 2024-05-19T19:21:26.9441691Z STATS: call_* op count: 610 | FakeTensor.__torch_dispatch__:20898 | FakeTensorMode.__torch_dispatch__:122455 | attempt fast:5511 | fast is_contiguous:5511 | ProxyTorchDispatchMode.__torch_dispatch__:23345 2024-05-19T19:21:26.9443101Z Dynamo produced 2 graphs covering 610 ops with 6 graph breaks (5 unique) 2024-05-19T19:21:36.8956667Z 2024-05-19T19:21:38.1523999Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:21:38.1524980Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:21:38.1525813Z cuda train gernet_l 2024-05-19T19:22:44.9769467Z pass 2024-05-19T19:22:44.9988099Z TIMING: entire_frame_compile:59.92541 code_gen:13.01694 inductor_compile:32.20919 backend_compile:50.15089 2024-05-19T19:22:44.9990651Z STATS: call_* op count: 410 | FakeTensor.__torch_dispatch__:9236 | FakeTensorMode.__torch_dispatch__:55856 | attempt fast:2023 | fast is_contiguous:2023 | ProxyTorchDispatchMode.__torch_dispatch__:12116 2024-05-19T19:22:44.9992023Z Dynamo produced 2 graphs covering 410 ops with 6 graph breaks (5 unique) 2024-05-19T19:22:50.8029657Z 2024-05-19T19:22:51.9149024Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:22:51.9149604Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:22:51.9150139Z cuda train ghostnet_100 2024-05-19T19:24:49.0546946Z pass 2024-05-19T19:24:49.0592863Z TIMING: entire_frame_compile:108.98152 code_gen:28.64013 inductor_compile:66.34017 backend_compile:97.93046 2024-05-19T19:24:49.0595655Z STATS: call_* op count: 360 | FakeTensor.__torch_dispatch__:15519 | FakeTensorMode.__torch_dispatch__:95609 | attempt fast:3114 | fast is_contiguous:3018 | ProxyTorchDispatchMode.__torch_dispatch__:19864 | slow no contiguity match:96 2024-05-19T19:24:49.0597147Z Dynamo produced 2 graphs covering 360 ops with 6 graph breaks (5 unique) 2024-05-19T19:24:57.2851240Z 2024-05-19T19:24:58.6997616Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:24:58.6998200Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:24:58.6999928Z cuda train gluon_inception_v3 2024-05-19T19:25:54.0313534Z W0519 19:25:54.030000 140632465625728 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:27:10.2137727Z pass 2024-05-19T19:27:10.2360250Z TIMING: entire_frame_compile:118.87554 inductor_compile:43.41279 backend_compile:92.98053 code_gen:22.06407 2024-05-19T19:27:10.2363959Z STATS: call_* op count: 1376 | FakeTensor.__torch_dispatch__:16533 | FakeTensorMode.__torch_dispatch__:136254 | attempt fast:2205 | fast is_contiguous:2205 | ProxyTorchDispatchMode.__torch_dispatch__:33213 2024-05-19T19:27:10.2366791Z Dynamo produced 3 graphs covering 1376 ops with 7 graph breaks (5 unique) 2024-05-19T19:27:19.4780357Z 2024-05-19T19:27:20.4981910Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:27:20.4982454Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:27:20.4982953Z cuda train gmixer_24_224 2024-05-19T19:29:16.6369936Z pass 2024-05-19T19:29:16.6616484Z TIMING: entire_frame_compile:107.59044 code_gen:18.86102 inductor_compile:52.98941 backend_compile:94.47031 2024-05-19T19:29:16.6619205Z STATS: call_* op count: 684 | FakeTensor.__torch_dispatch__:12801 | FakeTensorMode.__torch_dispatch__:111341 | attempt fast:4737 | fast is_contiguous:4017 | slow no contiguity match:720 | ProxyTorchDispatchMode.__torch_dispatch__:24210 2024-05-19T19:29:16.6621362Z Dynamo produced 2 graphs covering 684 ops with 6 graph breaks (5 unique) 2024-05-19T19:29:25.5151979Z 2024-05-19T19:29:26.4242557Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:29:26.4243422Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:29:26.4244424Z cuda train gmlp_s16_224 2024-05-19T19:31:09.8699238Z W0519 19:31:09.869000 140506657849984 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:32:24.0981188Z pass 2024-05-19T19:32:24.1182284Z TIMING: entire_frame_compile:164.4095 code_gen:39.66256 inductor_compile:89.42457 backend_compile:142.44342 2024-05-19T19:32:24.1184725Z STATS: call_* op count: 1458 | FakeTensor.__torch_dispatch__:27637 | FakeTensorMode.__torch_dispatch__:180170 | attempt fast:5397 | fast is_contiguous:4917 | slow no contiguity match:480 | ProxyTorchDispatchMode.__torch_dispatch__:35364 2024-05-19T19:32:24.1186199Z Dynamo produced 3 graphs covering 1458 ops with 7 graph breaks (5 unique) 2024-05-19T19:32:35.2639629Z 2024-05-19T19:32:37.0630367Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:32:37.0630876Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:32:37.0631312Z cuda train hrnet_w18 2024-05-19T19:37:39.8897554Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T19:37:39.8900369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 365, in torch_dynamo_resume_in_forward_and_backward_pass_at_363 2024-05-19T19:37:39.8901611Z pred = mod(*cloned_inputs) 2024-05-19T19:37:39.8902958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:37:39.8903796Z return forward_call(*args, **kwargs) 2024-05-19T19:37:39.8904684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 792, in forward 2024-05-19T19:37:39.8905689Z y = self.forward_features(x) 2024-05-19T19:37:39.8906801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 770, in forward_features 2024-05-19T19:37:39.8907592Z yl = self.stages(x) 2024-05-19T19:37:39.8908350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 751, in stages 2024-05-19T19:37:39.8909097Z yl = self.stage2(xl) 2024-05-19T19:37:39.8910471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:37:39.8911510Z return forward_call(*args, **kwargs) 2024-05-19T19:37:39.8912442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 506, in forward 2024-05-19T19:37:39.8913177Z x = module(x) 2024-05-19T19:37:39.8914120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:37:39.8914958Z return forward_call(*args, **kwargs) 2024-05-19T19:37:39.8915792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 484, in forward 2024-05-19T19:37:39.8916524Z y = y + f(x[j]) 2024-05-19T19:37:39.8916718Z 2024-05-19T19:37:58.8947038Z pass 2024-05-19T19:37:58.9125972Z TIMING: entire_frame_compile:303.30147 code_gen:66.06216 inductor_compile:173.20317 backend_compile:275.70597 2024-05-19T19:37:58.9127745Z STATS: call_* op count: 1394 | FakeTensor.__torch_dispatch__:52796 | FakeTensorMode.__torch_dispatch__:322818 | attempt fast:11857 | fast is_contiguous:11857 | ProxyTorchDispatchMode.__torch_dispatch__:71274 2024-05-19T19:37:58.9129082Z Dynamo produced 2 graphs covering 1394 ops with 5 graph breaks (4 unique) 2024-05-19T19:38:15.4951979Z 2024-05-19T19:38:16.9218373Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:38:16.9218854Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:38:16.9219389Z cuda train inception_v3 2024-05-19T19:39:13.8954417Z pass 2024-05-19T19:39:13.9201767Z TIMING: entire_frame_compile:48.8345 inductor_compile:4.38064 backend_compile:35.40076 2024-05-19T19:39:13.9204851Z STATS: call_* op count: 506 | FakeTensor.__torch_dispatch__:2985 | FakeTensorMode.__torch_dispatch__:60332 | attempt fast:2205 | fast is_contiguous:2205 | ProxyTorchDispatchMode.__torch_dispatch__:20409 2024-05-19T19:39:13.9206905Z Dynamo produced 2 graphs covering 506 ops with 6 graph breaks (5 unique) 2024-05-19T19:39:19.5993479Z 2024-05-19T19:39:21.1834357Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:39:21.1834838Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:39:21.1838152Z cuda train jx_nest_base 2024-05-19T19:42:09.6231111Z W0519 19:42:09.622000 139983692923520 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:43:28.9448408Z pass 2024-05-19T19:43:28.9903288Z TIMING: entire_frame_compile:228.11714 code_gen:56.29228 inductor_compile:121.18744 backend_compile:196.76931 2024-05-19T19:43:28.9905362Z STATS: call_* op count: 1852 | FakeTensor.__torch_dispatch__:27834 | FakeTensorMode.__torch_dispatch__:214033 | attempt fast:5756 | fast is_contiguous:5756 | ProxyTorchDispatchMode.__torch_dispatch__:42081 2024-05-19T19:43:28.9906828Z Dynamo produced 3 graphs covering 1852 ops with 7 graph breaks (5 unique) 2024-05-19T19:43:42.7258472Z 2024-05-19T19:43:43.5306496Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:43:43.5307067Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:43:43.5307499Z cuda train lcnet_050 2024-05-19T19:44:33.0075054Z pass 2024-05-19T19:44:33.0100859Z TIMING: entire_frame_compile:45.37104 code_gen:10.32057 inductor_compile:24.99487 backend_compile:38.86209 2024-05-19T19:44:33.0102316Z STATS: call_* op count: 168 | FakeTensor.__torch_dispatch__:6301 | FakeTensorMode.__torch_dispatch__:36933 | attempt fast:1707 | fast is_contiguous:1707 | ProxyTorchDispatchMode.__torch_dispatch__:6908 2024-05-19T19:44:33.0103599Z Dynamo produced 2 graphs covering 168 ops with 6 graph breaks (5 unique) 2024-05-19T19:44:38.4797391Z 2024-05-19T19:44:39.4728155Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:44:39.4728798Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:44:39.4729416Z cuda train levit_128 2024-05-19T19:46:46.8834229Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T19:46:46.8836105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 365, in torch_dynamo_resume_in_forward_and_backward_pass_at_363 2024-05-19T19:46:46.8838027Z pred = mod(*cloned_inputs) 2024-05-19T19:46:46.8839577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:46:46.8840941Z return forward_call(*args, **kwargs) 2024-05-19T19:46:46.8842524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 652, in forward 2024-05-19T19:46:46.8843767Z x = self.forward_features(x) 2024-05-19T19:46:46.8845359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 643, in forward_features 2024-05-19T19:46:46.8846649Z x = self.stages(x) 2024-05-19T19:46:46.8847966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:46:46.8849337Z return forward_call(*args, **kwargs) 2024-05-19T19:46:46.8850707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 518, in forward 2024-05-19T19:46:46.8851927Z x = self.blocks(x) 2024-05-19T19:46:46.8853259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:46:46.8854597Z return forward_call(*args, **kwargs) 2024-05-19T19:46:46.8855969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 456, in forward 2024-05-19T19:46:46.8857240Z x = x + self.drop_path1(self.attn(x)) 2024-05-19T19:46:46.8859100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:46:46.8860438Z return forward_call(*args, **kwargs) 2024-05-19T19:46:46.8861811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 236, in forward 2024-05-19T19:46:46.8863273Z attn = q @ k * self.scale + self.get_attention_biases(x.device) 2024-05-19T19:46:46.8864955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 211, in get_attention_biases 2024-05-19T19:46:46.8866455Z return self.attention_biases[:, self.attention_bias_idxs] 2024-05-19T19:46:46.8867097Z 2024-05-19T19:46:54.1509444Z W0519 19:46:54.150000 140445916435072 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:47:49.6514251Z pass 2024-05-19T19:47:49.6547365Z TIMING: entire_frame_compile:180.21776 code_gen:47.79364 inductor_compile:100.31011 backend_compile:155.27128 2024-05-19T19:47:49.6550673Z STATS: call_* op count: 1257 | FakeTensor.__torch_dispatch__:23333 | FakeTensorMode.__torch_dispatch__:156596 | attempt fast:3657 | fast is_contiguous:3603 | slow no contiguity match:51 | ProxyTorchDispatchMode.__torch_dispatch__:32105 | fast channels_last:3 2024-05-19T19:47:49.6552243Z Dynamo produced 3 graphs covering 1257 ops with 7 graph breaks (5 unique) 2024-05-19T19:48:00.4936478Z 2024-05-19T19:48:01.7315668Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:48:01.7316381Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:48:01.7317062Z cuda train mixer_b16_224 2024-05-19T19:48:59.5425089Z W0519 19:48:59.541000 139637715137152 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:49:38.9191675Z pass 2024-05-19T19:49:38.9607034Z TIMING: entire_frame_compile:87.66443 code_gen:23.16454 inductor_compile:48.74715 backend_compile:76.31416 2024-05-19T19:49:38.9609074Z STATS: call_* op count: 720 | FakeTensor.__torch_dispatch__:13776 | FakeTensorMode.__torch_dispatch__:89480 | attempt fast:2769 | fast is_contiguous:2625 | slow no contiguity match:144 | ProxyTorchDispatchMode.__torch_dispatch__:18120 2024-05-19T19:49:38.9610618Z Dynamo produced 3 graphs covering 720 ops with 7 graph breaks (5 unique) 2024-05-19T19:49:43.5349344Z accuracy pass_rate=93.33% 2024-05-19T19:49:43.5353270Z calls_captured gmean=0.00x mean=852.267x 2024-05-19T19:49:43.5356981Z unique_graphs gmean=0.00x mean=2.533x 2024-05-19T19:49:43.5360311Z graph_breaks gmean=0.00x mean=6.267x 2024-05-19T19:49:43.5363447Z unique_graph_breaks gmean=0.00x mean=4.767x 2024-05-19T19:49:43.5367005Z autograd_captures gmean=0.00x mean=0.000x 2024-05-19T19:49:43.5370322Z autograd_compiles gmean=0.00x mean=0.000x 2024-05-19T19:49:43.5373690Z cudagraph_skips gmean=0.00x mean=0.133x 2024-05-19T19:49:45.1247855Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_inductor_timm_training.csv 2024-05-19T19:49:45.3669473Z adv_inception_v3 PASS 2024-05-19T19:49:45.3673898Z beit_base_patch16_224 FAIL: accuracy=fail_accuracy, expected=pass 2024-05-19T19:49:45.3677984Z botnet26t_256 PASS 2024-05-19T19:49:45.3682512Z cait_m36_384 XFAIL 2024-05-19T19:49:45.3687602Z coat_lite_mini PASS 2024-05-19T19:49:45.3692275Z convit_base PASS 2024-05-19T19:49:45.3696753Z convmixer_768_32 PASS 2024-05-19T19:49:45.3701962Z convnext_base PASS 2024-05-19T19:49:45.3706491Z crossvit_9_240 PASS 2024-05-19T19:49:45.3711144Z cspdarknet53 PASS 2024-05-19T19:49:45.3715685Z deit_base_distilled_patch16_224 PASS 2024-05-19T19:49:45.3720291Z dla102 PASS 2024-05-19T19:49:45.3724968Z dm_nfnet_f0 PASS 2024-05-19T19:49:45.3729770Z dpn107 PASS 2024-05-19T19:49:45.3734323Z eca_botnext26ts_256 PASS 2024-05-19T19:49:45.3739007Z eca_halonext26ts PASS 2024-05-19T19:49:45.3743868Z ese_vovnet19b_dw PASS 2024-05-19T19:49:45.3748260Z fbnetc_100 PASS 2024-05-19T19:49:45.3752842Z fbnetv3_b PASS 2024-05-19T19:49:45.3757391Z gernet_l PASS 2024-05-19T19:49:45.3761900Z ghostnet_100 PASS 2024-05-19T19:49:45.3766700Z gluon_inception_v3 PASS 2024-05-19T19:49:45.3771285Z gmixer_24_224 PASS 2024-05-19T19:49:45.3775795Z gmlp_s16_224 PASS 2024-05-19T19:49:45.3780607Z hrnet_w18 PASS 2024-05-19T19:49:45.3785280Z inception_v3 PASS 2024-05-19T19:49:45.3789806Z jx_nest_base PASS 2024-05-19T19:49:45.3794392Z lcnet_050 PASS 2024-05-19T19:49:45.3798876Z levit_128 PASS 2024-05-19T19:49:45.3803762Z mixer_b16_224 PASS 2024-05-19T19:49:45.3804167Z 2024-05-19T19:49:45.3804409Z Error: 1 models have accuracy status regressed: 2024-05-19T19:49:45.3805119Z beit_base_patch16_224 2024-05-19T19:49:45.3806348Z 2024-05-19T19:49:45.3806634Z 2024-05-19T19:49:45.3807469Z If this change is expected, you can update `benchmarks/dynamo/ci_expected_accuracy/dynamic_inductor_timm_training.csv` to reflect the new baseline. 2024-05-19T19:49:45.3808441Z from pytorch/pytorch root, run 2024-05-19T19:49:45.3809189Z `python benchmarks/dynamo/ci_expected_accuracy/update_expected.py c8c7ddfde57442cc28a394c385cab656ce9b1f5e` 2024-05-19T19:49:45.3810154Z and then `git add` the resulting local changes to expected CSVs to your commit. 2024-05-19T19:49:45.3810649Z 2024-05-19T19:49:45.4204324Z + cleanup_workspace 2024-05-19T19:49:45.4205391Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-05-19T19:49:45.4206650Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-05-19T19:49:45.4207649Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-05-19T19:49:45.4208336Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-05-19T19:49:45.4209386Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-05-19T19:49:45.4210302Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-05-19T19:49:45.4211041Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-05-19T19:49:45.8905577Z ##[error]Process completed with exit code 1. 2024-05-19T19:49:45.8983190Z Prepare all required actions 2024-05-19T19:49:45.8983643Z Getting action download info 2024-05-19T19:49:46.0510553Z ##[group]Run ./.github/actions/pytest-cache-upload 2024-05-19T19:49:46.0511011Z with: 2024-05-19T19:49:46.0511295Z cache_dir: .pytest_cache 2024-05-19T19:49:46.0511640Z shard: 1 2024-05-19T19:49:46.0511987Z sha: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T19:49:46.0512455Z test_config: dynamic_inductor_timm 2024-05-19T19:49:46.0513002Z job_identifier: inductor_linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T19:49:46.0513515Z env: 2024-05-19T19:49:46.0513793Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:49:46.0514265Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:49:46.0515004Z DOCKER_CONTAINER_ID: 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T19:49:46.0515650Z ##[endgroup] 2024-05-19T19:49:46.0554104Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-05-19T19:49:46.0554699Z with: 2024-05-19T19:49:46.0554961Z shell: bash 2024-05-19T19:49:46.0555256Z timeout_minutes: 5 2024-05-19T19:49:46.0555577Z max_attempts: 5 2024-05-19T19:49:46.0555888Z retry_wait_seconds: 30 2024-05-19T19:49:46.0556329Z command: set -eu python3 -m pip install boto3==1.19.12 2024-05-19T19:49:46.0556834Z polling_interval_seconds: 1 2024-05-19T19:49:46.0557201Z warning_on_retry: true 2024-05-19T19:49:46.0557549Z continue_on_error: false 2024-05-19T19:49:46.0557878Z env: 2024-05-19T19:49:46.0558150Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:49:46.0558598Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:49:46.0559346Z DOCKER_CONTAINER_ID: 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T19:49:46.0559981Z ##[endgroup] 2024-05-19T19:49:46.3258301Z Defaulting to user installation because normal site-packages is not writeable 2024-05-19T19:49:47.5270520Z Collecting boto3==1.19.12 2024-05-19T19:49:47.5487114Z Downloading boto3-1.19.12-py3-none-any.whl (131 kB) 2024-05-19T19:49:47.6060034Z Collecting s3transfer<0.6.0,>=0.5.0 2024-05-19T19:49:47.6116081Z Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 2024-05-19T19:49:47.6468030Z Collecting jmespath<1.0.0,>=0.7.1 2024-05-19T19:49:47.6518676Z Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 2024-05-19T19:49:49.0162824Z Collecting botocore<1.23.0,>=1.22.12 2024-05-19T19:49:49.0240836Z Downloading botocore-1.22.12-py3-none-any.whl (8.1 MB) 2024-05-19T19:49:49.1773608Z Collecting python-dateutil<3.0.0,>=2.1 2024-05-19T19:49:49.1826496Z Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) 2024-05-19T19:49:49.1939085Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.18) 2024-05-19T19:49:49.2460414Z Collecting six>=1.5 2024-05-19T19:49:49.2510773Z Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2024-05-19T19:49:49.3205771Z Installing collected packages: six, python-dateutil, jmespath, botocore, s3transfer, boto3 2024-05-19T19:49:49.9774896Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.9.0.post0 s3transfer-0.5.2 six-1.16.0 2024-05-19T19:49:50.1069079Z Command completed after 1 attempt(s). 2024-05-19T19:49:50.1113597Z ##[group]Run python3 .github/scripts/pytest_cache.py \ 2024-05-19T19:49:50.1114190Z python3 .github/scripts/pytest_cache.py \ 2024-05-19T19:49:50.1114674Z  --upload \ 2024-05-19T19:49:50.1115095Z  --cache_dir $GITHUB_WORKSPACE/$CACHE_DIR \ 2024-05-19T19:49:50.1115610Z  --pr_identifier $GITHUB_REF \ 2024-05-19T19:49:50.1116221Z  --job_identifier $JOB_IDENTIFIER \ 2024-05-19T19:49:50.1116664Z  --sha $SHA \ 2024-05-19T19:49:50.1117043Z  --test_config $TEST_CONFIG \ 2024-05-19T19:49:50.1117468Z  --shard $SHARD \ 2024-05-19T19:49:50.1117826Z  --repo $REPO \ 2024-05-19T19:49:50.1118308Z  --temp_dir $RUNNER_TEMP \ 2024-05-19T19:49:50.1125959Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T19:49:50.1126446Z env: 2024-05-19T19:49:50.1126724Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:49:50.1127177Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:49:50.1127918Z DOCKER_CONTAINER_ID: 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T19:49:50.1128571Z CACHE_DIR: .pytest_cache 2024-05-19T19:49:50.1129066Z JOB_IDENTIFIER: inductor_linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T19:49:50.1129653Z SHA: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T19:49:50.1130127Z TEST_CONFIG: dynamic_inductor_timm 2024-05-19T19:49:50.1130514Z SHARD: 1 2024-05-19T19:49:50.1130800Z REPO: pytorch/pytorch 2024-05-19T19:49:50.1131136Z ##[endgroup] 2024-05-19T19:49:50.2471435Z PR identifier for `refs/tags/ciflow/inductor/121956` is `5ec4bd29b9652d20d61d9e0a665b8d46` 2024-05-19T19:49:50.2474602Z Uploading cache with args Namespace(bucket=None, cache_dir='/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache', download=False, job_identifier='inductor_linux-focal-cuda12.4-py3.10-gcc9-sm86', pr_identifier='refs/tags/ciflow/inductor/121956', repo='pytorch/pytorch', sha='c8c7ddfde57442cc28a394c385cab656ce9b1f5e', shard='1', temp_dir='/home/ec2-user/actions-runner/_work/_temp', test_config='dynamic_inductor_timm', upload=True) 2024-05-19T19:49:50.2477454Z The pytest cache dir `/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache` does not exist. Skipping upload 2024-05-19T19:49:50.2657017Z ##[group]Run cat test/**/*_toprint.log || true 2024-05-19T19:49:50.2657542Z cat test/**/*_toprint.log || true 2024-05-19T19:49:50.2665545Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T19:49:50.2666024Z env: 2024-05-19T19:49:50.2666298Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:49:50.2666742Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:49:50.2667473Z DOCKER_CONTAINER_ID: 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T19:49:50.2668099Z ##[endgroup] 2024-05-19T19:49:50.2730175Z cat: test/**/*_toprint.log: No such file or directory 2024-05-19T19:49:50.2758193Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-05-19T19:49:50.2758660Z kill "$MONITOR_SCRIPT_PID" 2024-05-19T19:49:50.2765553Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T19:49:50.2766085Z env: 2024-05-19T19:49:50.2766365Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:49:50.2766838Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:49:50.2767645Z DOCKER_CONTAINER_ID: 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T19:49:50.2768387Z MONITOR_SCRIPT_PID: 28892 2024-05-19T19:49:50.2768755Z ##[endgroup] 2024-05-19T19:49:50.2950206Z Prepare all required actions 2024-05-19T19:49:50.2950648Z Getting action download info 2024-05-19T19:49:50.4146854Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-05-19T19:49:50.5763679Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-05-19T19:49:50.5764155Z with: 2024-05-19T19:49:50.5764762Z file-suffix: test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197487 2024-05-19T19:49:50.5765420Z s3-bucket: gha-artifacts 2024-05-19T19:49:50.5765761Z env: 2024-05-19T19:49:50.5766035Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:49:50.5766492Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:49:50.5767234Z DOCKER_CONTAINER_ID: 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T19:49:50.5767877Z ##[endgroup] 2024-05-19T19:49:50.5793256Z ##[group]Run # Remove any previous test jsons if they exist 2024-05-19T19:49:50.5793912Z # Remove any previous test jsons if they exist 2024-05-19T19:49:50.5794423Z rm -f test-jsons-*.zip 2024-05-19T19:49:50.5794939Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-05-19T19:49:50.5802575Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T19:49:50.5803062Z env: 2024-05-19T19:49:50.5803337Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:49:50.5803781Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:49:50.5804513Z DOCKER_CONTAINER_ID: 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T19:49:50.5805533Z FILE_SUFFIX: test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197487 2024-05-19T19:49:50.5806167Z ##[endgroup] 2024-05-19T19:49:50.5997772Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-05-19T19:49:50.6026993Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-05-19T19:49:50.6027689Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-05-19T19:49:50.6033587Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-05-19T19:49:50.6060199Z ##[group]Run # Remove any previous test reports if they exist 2024-05-19T19:49:50.6060843Z # Remove any previous test reports if they exist 2024-05-19T19:49:50.6061369Z rm -f test-reports-*.zip 2024-05-19T19:49:50.6061952Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-05-19T19:49:50.6069667Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T19:49:50.6070156Z env: 2024-05-19T19:49:50.6070432Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:49:50.6070886Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:49:50.6071615Z DOCKER_CONTAINER_ID: 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T19:49:50.6072500Z FILE_SUFFIX: test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197487 2024-05-19T19:49:50.6073152Z ##[endgroup] 2024-05-19T19:49:50.6238390Z adding: test/test-reports/inference_timm_models.csv (deflated 69%) 2024-05-19T19:49:50.6239462Z adding: test/test-reports/training_timm_models.csv (deflated 65%) 2024-05-19T19:49:50.6243373Z adding: test/test-reports/training_timm_model_graph_breaks.csv (deflated 97%) 2024-05-19T19:49:50.6244858Z adding: test/test-reports/training_timm_model_graph_break_deduped.csv (deflated 73%) 2024-05-19T19:49:50.6271922Z ##[group]Run # Remove any previous usage logs if they exist 2024-05-19T19:49:50.6272561Z # Remove any previous usage logs if they exist 2024-05-19T19:49:50.6273068Z rm -f logs-*.zip 2024-05-19T19:49:50.6273743Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-05-19T19:49:50.6274533Z # so check to see if the file exists first 2024-05-19T19:49:50.6275038Z if [ -f 'usage_log.txt' ]; then 2024-05-19T19:49:50.6275594Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-05-19T19:49:50.6276082Z fi 2024-05-19T19:49:50.6276546Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-05-19T19:49:50.6277124Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-05-19T19:49:50.6277622Z fi 2024-05-19T19:49:50.6285007Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T19:49:50.6285491Z env: 2024-05-19T19:49:50.6285772Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:49:50.6286220Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:49:50.6286960Z DOCKER_CONTAINER_ID: 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T19:49:50.6287845Z FILE_SUFFIX: test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197487 2024-05-19T19:49:50.6288486Z ##[endgroup] 2024-05-19T19:49:50.6449265Z adding: usage_log.txt (deflated 96%) 2024-05-19T19:49:50.6513971Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-05-19T19:49:50.6514850Z # Remove any previous debugging artifacts if they exist 2024-05-19T19:49:50.6515408Z rm -f debug-*.zip 2024-05-19T19:49:50.6515796Z if [ -d 'test/debug' ]; then 2024-05-19T19:49:50.6516296Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-05-19T19:49:50.6516772Z fi 2024-05-19T19:49:50.6523425Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T19:49:50.6523919Z env: 2024-05-19T19:49:50.6524187Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:49:50.6524779Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:49:50.6525524Z DOCKER_CONTAINER_ID: 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T19:49:50.6526416Z FILE_SUFFIX: test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197487 2024-05-19T19:49:50.6527041Z ##[endgroup] 2024-05-19T19:49:50.6599917Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-05-19T19:49:50.6600363Z with: 2024-05-19T19:49:50.6600644Z s3-bucket: gha-artifacts 2024-05-19T19:49:50.6601080Z s3-prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T19:49:50.6601549Z retention-days: 14 2024-05-19T19:49:50.6601877Z if-no-files-found: warn 2024-05-19T19:49:50.6602230Z path: test-jsons-*.zip 2024-05-19T19:49:50.6602567Z name: artifact 2024-05-19T19:49:50.6602860Z region: us-east-1 2024-05-19T19:49:50.6603156Z env: 2024-05-19T19:49:50.6603431Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:49:50.6603875Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:49:50.6604603Z DOCKER_CONTAINER_ID: 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T19:49:50.6605368Z ##[endgroup] 2024-05-19T19:49:50.9639223Z NOTE: s3-prefix specified, ignoring name parameter 2024-05-19T19:49:50.9640009Z With the provided path, there will be 1 file uploaded 2024-05-19T19:49:50.9640682Z Uploading to s3 prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T19:49:51.1154480Z Starting upload of test-jsons-test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197487.zip 2024-05-19T19:49:51.2231366Z Finished upload of test-jsons-test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197487.zip 2024-05-19T19:49:51.2383821Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-05-19T19:49:51.2384273Z with: 2024-05-19T19:49:51.2384566Z s3-bucket: gha-artifacts 2024-05-19T19:49:51.2384994Z s3-prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T19:49:51.2385467Z retention-days: 14 2024-05-19T19:49:51.2385812Z if-no-files-found: error 2024-05-19T19:49:51.2386181Z path: test-reports-*.zip 2024-05-19T19:49:51.2386533Z name: artifact 2024-05-19T19:49:51.2386845Z region: us-east-1 2024-05-19T19:49:51.2387152Z env: 2024-05-19T19:49:51.2387428Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:49:51.2387884Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:49:51.2388634Z DOCKER_CONTAINER_ID: 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T19:49:51.2389301Z ##[endgroup] 2024-05-19T19:49:51.5527959Z NOTE: s3-prefix specified, ignoring name parameter 2024-05-19T19:49:51.5528932Z With the provided path, there will be 1 file uploaded 2024-05-19T19:49:51.5529577Z Uploading to s3 prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T19:49:51.5555353Z Starting upload of test-reports-test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197487.zip 2024-05-19T19:49:51.6690674Z Finished upload of test-reports-test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197487.zip 2024-05-19T19:49:51.8192529Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-05-19T19:49:51.8192971Z with: 2024-05-19T19:49:51.8193245Z s3-bucket: gha-artifacts 2024-05-19T19:49:51.8193669Z s3-prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T19:49:51.8194138Z retention-days: 14 2024-05-19T19:49:51.8194478Z if-no-files-found: ignore 2024-05-19T19:49:51.8194965Z path: logs-*.zip 2024-05-19T19:49:51.8195309Z name: artifact 2024-05-19T19:49:51.8195876Z region: us-east-1 2024-05-19T19:49:51.8196216Z env: 2024-05-19T19:49:51.8196488Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:49:51.8197041Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:49:51.8197904Z DOCKER_CONTAINER_ID: 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T19:49:51.8198600Z ##[endgroup] 2024-05-19T19:49:52.1347291Z NOTE: s3-prefix specified, ignoring name parameter 2024-05-19T19:49:52.1348156Z With the provided path, there will be 1 file uploaded 2024-05-19T19:49:52.1348968Z Uploading to s3 prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T19:49:52.1375637Z Starting upload of logs-test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197487.zip 2024-05-19T19:49:52.2549568Z Finished upload of logs-test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197487.zip 2024-05-19T19:49:52.2682208Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-05-19T19:49:52.2682646Z with: 2024-05-19T19:49:52.2682933Z s3-bucket: gha-artifacts 2024-05-19T19:49:52.2683358Z s3-prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T19:49:52.2683838Z retention-days: 14 2024-05-19T19:49:52.2684168Z if-no-files-found: ignore 2024-05-19T19:49:52.2684524Z path: debug-*.zip 2024-05-19T19:49:52.2684977Z name: artifact 2024-05-19T19:49:52.2685282Z region: us-east-1 2024-05-19T19:49:52.2685571Z env: 2024-05-19T19:49:52.2685846Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:49:52.2686298Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:49:52.2687030Z DOCKER_CONTAINER_ID: 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T19:49:52.2687668Z ##[endgroup] 2024-05-19T19:49:52.5697206Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-05-19T19:49:52.5842716Z ##[group]Run # shellcheck disable=SC2156 2024-05-19T19:49:52.5843207Z # shellcheck disable=SC2156 2024-05-19T19:49:52.5844002Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-05-19T19:49:52.5852018Z shell: /usr/bin/bash -e {0} 2024-05-19T19:49:52.5852364Z env: 2024-05-19T19:49:52.5852641Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:49:52.5853095Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:49:52.5853836Z DOCKER_CONTAINER_ID: 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T19:49:52.5854472Z ##[endgroup] 2024-05-19T19:49:52.7617900Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-05-19T19:49:52.7618357Z with: 2024-05-19T19:49:52.7618816Z name: coredumps-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu 2024-05-19T19:49:52.7619393Z retention-days: 14 2024-05-19T19:49:52.7619736Z if-no-files-found: ignore 2024-05-19T19:49:52.7620090Z path: ./**/core.[1-9]* 2024-05-19T19:49:52.7620440Z s3-bucket: gha-artifacts 2024-05-19T19:49:52.7620793Z region: us-east-1 2024-05-19T19:49:52.7621096Z env: 2024-05-19T19:49:52.7621379Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:49:52.7621834Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:49:52.7622578Z DOCKER_CONTAINER_ID: 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T19:49:52.7623222Z ##[endgroup] 2024-05-19T19:50:01.8304797Z No files were found with the provided path: ./**/core.[1-9]*. No artifacts will be uploaded. 2024-05-19T19:50:01.8517437Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-05-19T19:50:01.8517990Z with: 2024-05-19T19:50:01.8518255Z env: 2024-05-19T19:50:01.8518536Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:50:01.8518997Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:50:01.8519735Z DOCKER_CONTAINER_ID: 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T19:50:01.8520380Z ##[endgroup] 2024-05-19T19:50:01.8540874Z ##[group]Run set -eou pipefail 2024-05-19T19:50:01.8541282Z set -eou pipefail 2024-05-19T19:50:01.8541631Z  2024-05-19T19:50:01.8542272Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-05-19T19:50:01.8542905Z for _ in $(seq 1440); do 2024-05-19T19:50:01.8543371Z  # Break if no ssh session exists anymore 2024-05-19T19:50:01.8543865Z  if [ "$(who)" = "" ]; then 2024-05-19T19:50:01.8544273Z  break 2024-05-19T19:50:01.8544616Z  fi 2024-05-19T19:50:01.8544921Z  echo "." 2024-05-19T19:50:01.8545243Z  sleep 5 2024-05-19T19:50:01.8545575Z done 2024-05-19T19:50:01.8553093Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T19:50:01.8553585Z env: 2024-05-19T19:50:01.8553862Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:50:01.8554315Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:50:01.8555047Z DOCKER_CONTAINER_ID: 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T19:50:01.8555687Z ##[endgroup] 2024-05-19T19:50:01.8592006Z Holding runner for 2 hours until all ssh sessions have logged out 2024-05-19T19:50:01.8638056Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-05-19T19:50:01.8638820Z # ignore expansion of "docker ps -q" since it could be empty 2024-05-19T19:50:01.8639397Z # shellcheck disable=SC2046 2024-05-19T19:50:01.8639862Z docker stop $(docker ps -q) || true 2024-05-19T19:50:01.8640338Z # Prune all of the docker images 2024-05-19T19:50:01.8640777Z docker system prune -af 2024-05-19T19:50:01.8648106Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T19:50:01.8648591Z env: 2024-05-19T19:50:01.8648864Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:50:01.8649305Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:50:01.8650032Z DOCKER_CONTAINER_ID: 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T19:50:01.8650662Z ##[endgroup] 2024-05-19T19:50:02.3467919Z 3f7c218bc62a 2024-05-19T19:50:03.0996532Z Deleted Containers: 2024-05-19T19:50:03.0997175Z 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T19:50:03.0997610Z 2024-05-19T19:50:08.3019110Z Deleted Images: 2024-05-19T19:50:08.3020963Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T19:50:08.3023853Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks@sha256:66f062925f91f9b9204b1dc6d68caf8c89fa82fd4b12cb5f24c19aa4b1c87506 2024-05-19T19:50:08.3025401Z deleted: sha256:a0542ac5317cc182e1aac2cb8d63a2d6287450937722bab68464513bbbc603ef 2024-05-19T19:50:08.3026222Z deleted: sha256:1a656f83b702d1cdc805b9509e7194f6ca9a24882785be5289f8bc566c1c1a0b 2024-05-19T19:50:08.3027054Z deleted: sha256:5b7ca0efbb8f2f6ed9ac2e375c4a52f07e2bd16e65d1914a58dcfb56ab8b1623 2024-05-19T19:50:08.3027878Z deleted: sha256:6d2255f7483ad8cfa448dc385ff2880171aacd4db919fbf38e2b3c4580f80e1e 2024-05-19T19:50:08.3028723Z deleted: sha256:a39c366330a4b21f0585d02ed65f5beb66bd03e36471ebb255c5f7766f29c118 2024-05-19T19:50:08.3029558Z deleted: sha256:5b2e96a220babb6e3a9b843ce53d3f2015dd0e505fed00a447cbe551b22d329e 2024-05-19T19:50:08.3030528Z deleted: sha256:d7247668eca42c13d2c6440886a9b43b633e5cb9304e3e3f85ac8803de7dd873 2024-05-19T19:50:08.3031498Z deleted: sha256:989398e2b08b3b7c9bf03cdbdaa903261d145b7d45bc7c1fa3a1a4a54a7ced68 2024-05-19T19:50:08.3032321Z deleted: sha256:5c2558f875db144549ed0efa5deb36d738647f1c172a9628a34677fddb6a96ee 2024-05-19T19:50:08.3033132Z deleted: sha256:b56f2c08b00650c89ac0f4d07686ba8ea22d64f666d27e8be716aeae4e042f87 2024-05-19T19:50:08.3033935Z deleted: sha256:35689f96fd20d72b6c5fd24a65e3c52c46359609c897b0552d8b36e4d62f31d9 2024-05-19T19:50:08.3034735Z deleted: sha256:8a251e66682a1ad41addcf4890ffb62aa8b85f5d9f17d433e828f170339a767f 2024-05-19T19:50:08.3035558Z deleted: sha256:9482b91acd78f655bcd9a0a6ade0f1958caa1a8274b13f7366f830fa7b4ca16e 2024-05-19T19:50:08.3036626Z deleted: sha256:1f1bf62bcfe25406d5e8f3a48198af80782c6a996bcafaae53575bcc843bd50a 2024-05-19T19:50:08.3037690Z deleted: sha256:6b680df6a8beceb613eb411958551bf0368a6d4a932eead77896d41ff535946a 2024-05-19T19:50:08.3038574Z deleted: sha256:7aa1322c73a827485b091076ffa2dc207e0a900941ca2eb2704442ca7c23ed7d 2024-05-19T19:50:08.3039515Z deleted: sha256:f0d92b27a23be6ff6a30b0974841e39e2ca9cb761797d5cb3ee5676708fbfddc 2024-05-19T19:50:08.3040351Z deleted: sha256:15e17f6dcd9e56d48cc4d19f06cf6d42e155d9dcb5d6d3289f432b1e1adb78c2 2024-05-19T19:50:08.3041156Z deleted: sha256:a9363d4f861657188bb086df9f77430971cf00513d7f3c151d954bba04092dd4 2024-05-19T19:50:08.3041969Z deleted: sha256:afad6ce30f8cd2b2fc987fb31626f14465a034dd85cea22a30e012f7b50e831c 2024-05-19T19:50:08.3042781Z deleted: sha256:a17532a675e4ec56db1fc54e2f285986b4333c89eaed6520edb8c85209ef76ea 2024-05-19T19:50:08.3043586Z deleted: sha256:55ff1a7a2aa478a7a6c00675e825d39cea2c5955f3f318832fa06cf293ddb137 2024-05-19T19:50:08.3044404Z deleted: sha256:21b7839e8103541d09ad0af189e5ecbf9df69cc798831eada41d8ce869555e94 2024-05-19T19:50:08.3045355Z deleted: sha256:3f93640e4b85ecbb5cb4239715bd521fe05dac28e2e25e53b71b808bbc193590 2024-05-19T19:50:08.3046203Z deleted: sha256:887d227bae679856dd096dad895dbcf90b807f5c82c15ff17ec5a58bf6dac184 2024-05-19T19:50:08.3047032Z deleted: sha256:d2beb8f0ce120b86322d41f423b7f42e64fdb9e4ff5d5d912252849b8146b293 2024-05-19T19:50:08.3047846Z deleted: sha256:e41d38911c997f96258de8880c0251dee4bf259158eb081eafc17f32f8948763 2024-05-19T19:50:08.3048666Z deleted: sha256:b85b0e100a5a15d221dbbccb1e864e367d917d7a79fa199c15db2c1d9a52c92a 2024-05-19T19:50:08.3049518Z deleted: sha256:75258ce71d03c6b0525b6f842fa5efd1738dbe82b883613717fcb707a60621a3 2024-05-19T19:50:08.3050363Z deleted: sha256:a12c28acec35eb76da2791220f40353777e633b6cd7d4fd187c3954e03406c46 2024-05-19T19:50:08.3051193Z deleted: sha256:eaf53ac56a05ff328d594fda97812e9db215cf74b87965669e54d4c6c1669cb3 2024-05-19T19:50:08.3052019Z deleted: sha256:795e72492c71c7cce01ac9e0408bc22fa8f7da1e41614874b812afc49c93fabf 2024-05-19T19:50:08.3052853Z deleted: sha256:1d1e16daa70a0b1c04b023ebbbe56366e3b0cbb9147851a544b43581d66ab50c 2024-05-19T19:50:08.3053672Z deleted: sha256:7752b0e90516dde8d6b7499bd8261515a2eb4063ca5d381313ea697291402d42 2024-05-19T19:50:08.3054488Z deleted: sha256:13082c1d2da03c75509cd19b1e7d211d98b359b41d53223e861fea8ec6be2366 2024-05-19T19:50:08.3055310Z deleted: sha256:fef51b8c2cf2579b04f0d82fba989e43cdd93a484741e14588a942c728328c80 2024-05-19T19:50:08.3056142Z deleted: sha256:cfd02989b5663a4fc04e8468da08dbb6202dd256e851a6e30e7370818fa65830 2024-05-19T19:50:08.3056959Z deleted: sha256:7b3853f547cd41caf732c9bf113d7f11db7fb026f979a50cf7a56ab673d8a2d9 2024-05-19T19:50:08.3058016Z deleted: sha256:8eb59cbbd7ea753bff3586c088340d20ca32887792f3f68c5b2da7d57d11447c 2024-05-19T19:50:08.3058854Z deleted: sha256:d4eb9793eb3f33e1fc18c62f8a4765625d5e2accaffffeb52543799bfe45315a 2024-05-19T19:50:08.3059665Z deleted: sha256:9b388bd2595f78854fce5ceb1a6c673a7bf3f50f84f6350351171599fcad68c4 2024-05-19T19:50:08.3060462Z deleted: sha256:1443f9602fff883313532012c5506b56993ec95031e611cace0b38b3e215413b 2024-05-19T19:50:08.3061249Z deleted: sha256:7d5157e7800e8039c647490628f0b8f64c7ed0017070e69ed1200ae786c318d2 2024-05-19T19:50:08.3062035Z deleted: sha256:3b8e4334d22e46371d21e1d80991345f300244c4932c1687b2723408f8fe213c 2024-05-19T19:50:08.3062969Z deleted: sha256:bc9b902883a611c9653e47dfcdce6f88287f0ce083995cd17612e1ee52d02f45 2024-05-19T19:50:08.3063789Z deleted: sha256:87241399dcaeb6b49174c5033c7aff0f1e914591bd2a727a80fcd0e453b1bdbc 2024-05-19T19:50:08.3064606Z deleted: sha256:9b60d0bfab86b98267697851c8871dae33fc146608b8af9764f5e3d3e459c75b 2024-05-19T19:50:08.3065425Z deleted: sha256:ad76dc292ef5f88364d52bdd2e83680e697b28b2f6df337cd9c829ab007217e1 2024-05-19T19:50:08.3066245Z deleted: sha256:7a4baeb5479a531e458c9fa438d16143efc760acf3c997b1fed3129f2d8d2fc7 2024-05-19T19:50:08.3067052Z deleted: sha256:e682379b3ae2d005fd368b974d3314ece2598d1a6928ae7341bb89f3c79e8e03 2024-05-19T19:50:08.3067951Z deleted: sha256:d8d802f39c7d9d7ddd8f5acef34ee7e07fac7f84fa106255d5228d659daf1073 2024-05-19T19:50:08.3068784Z deleted: sha256:9f893b39e0b2a09fdee21eec466f95f637ccd2e77ab254c00eeca60d4f1a4671 2024-05-19T19:50:08.3069622Z deleted: sha256:01d8e759bdd0bbea54aea263352ffd962e77eeab5d109a7fa8e279f2efdb6719 2024-05-19T19:50:08.3070462Z deleted: sha256:cd0863a43a1691ccbba624e9d63b85a341adb8bc16335dec1291e5a04147b040 2024-05-19T19:50:08.3071280Z deleted: sha256:5fec222edb2dc032d9a5203274fbd944dfb9494597fbb76b063b89ac7e84273f 2024-05-19T19:50:08.3072086Z deleted: sha256:742f8005153dd61c5f7718e2b2a8b3f934780866ef56f7ca11a48088dc638b97 2024-05-19T19:50:08.3072888Z deleted: sha256:e670dd8df73c884bf8257489df70a65828a55222a493f717eca2b9df1f3e41fa 2024-05-19T19:50:08.3073709Z deleted: sha256:e09fa8af11b0b73b35ad8f1f6ec0f0f0bf8a05ed64aa9d530377ddd2e1f94b56 2024-05-19T19:50:08.3074525Z deleted: sha256:02c8d457f61de51676900b87bd530f776b761f2b8dad7f345152b256d7bb7864 2024-05-19T19:50:08.3075335Z deleted: sha256:433544e4353c9e0269dc71fa16c00ddcecdcd5676ed0df618c7b51ac78e23053 2024-05-19T19:50:08.3076178Z deleted: sha256:fc24af9fd5121adfbc7b75df0abb1ef8c2d3868f33b7b6b1e8bf4e214fac4bd6 2024-05-19T19:50:08.3077005Z deleted: sha256:ce1492ad94b71204f8b3fb8903d0ad48a96652635e7fc9347e430639ca15a737 2024-05-19T19:50:08.3077804Z deleted: sha256:96f2ae9435e6e12f68a88dc404c8079b2796639589970c34894a86ccff3f3732 2024-05-19T19:50:08.3078613Z deleted: sha256:5310d2bc0952e7c6d8cbade7ecafd88ce3a04d9da303fa6419583e13a02b58a9 2024-05-19T19:50:08.3079440Z deleted: sha256:099076d608a55a027d60f5b572a3991a8d690a01500acf2c696933a7e6d38650 2024-05-19T19:50:08.3080269Z deleted: sha256:862b05d0f8f7e3dff2d04bd49784a9901f4e22c5dd6544ae72eb3be05e23fffe 2024-05-19T19:50:08.3081080Z deleted: sha256:f7a888e5f7904827f6d71cebcba5169dbc7a78b20f228b9b9b625f38e0c52f24 2024-05-19T19:50:08.3081899Z deleted: sha256:ab9f75a4a7a636aae480f630eaddc05133dda64f1c42445b617c97d3d209991c 2024-05-19T19:50:08.3082727Z deleted: sha256:56dda0c9cee64db5c10d5dd5f08c6f6707263f5ee98e8a075d635e5be996e855 2024-05-19T19:50:08.3083559Z deleted: sha256:5faf9c0a9efe4675ecd21a4ec417d51077d5e75da9e673161a94e7d6cd43f92c 2024-05-19T19:50:08.3084051Z 2024-05-19T19:50:08.3084204Z Total reclaimed space: 37.23GB 2024-05-19T19:50:08.3117115Z ##[group]Run set +e 2024-05-19T19:50:08.3117506Z set +e 2024-05-19T19:50:08.3117799Z set -x 2024-05-19T19:50:08.3118090Z  2024-05-19T19:50:08.3118369Z nvidia-smi 2024-05-19T19:50:08.3118984Z # NB: Surprisingly, nvidia-smi command returns successfully with return code 0 even in 2024-05-19T19:50:08.3119947Z # the case where the driver has already crashed as it still can get the driver version 2024-05-19T19:50:08.3120902Z # and some basic information like the bus ID. However, the rest of the information 2024-05-19T19:50:08.3121626Z # would be missing (ERR!), for example: 2024-05-19T19:50:08.3122071Z # 2024-05-19T19:50:08.3122497Z # +-----------------------------------------------------------------------------+ 2024-05-19T19:50:08.3123265Z # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | 2024-05-19T19:50:08.3124043Z # |-------------------------------+----------------------+----------------------+ 2024-05-19T19:50:08.3124867Z # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-05-19T19:50:08.3125728Z # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | 2024-05-19T19:50:08.3126462Z # | | | MIG M. | 2024-05-19T19:50:08.3127047Z # |===============================+======================+======================| 2024-05-19T19:50:08.3127686Z # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | 2024-05-19T19:50:08.3128426Z # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | 2024-05-19T19:50:08.3129214Z # | | | ERR! | 2024-05-19T19:50:08.3129841Z # +-------------------------------+----------------------+----------------------+ 2024-05-19T19:50:08.3130355Z # 2024-05-19T19:50:08.3130769Z # +-----------------------------------------------------------------------------+ 2024-05-19T19:50:08.3131420Z # | Processes: | 2024-05-19T19:50:08.3132143Z # | GPU GI CI PID Type Process name GPU Memory | 2024-05-19T19:50:08.3132850Z # | ID ID Usage | 2024-05-19T19:50:08.3133452Z # |=============================================================================| 2024-05-19T19:50:08.3134066Z # +-----------------------------------------------------------------------------+ 2024-05-19T19:50:08.3134573Z # 2024-05-19T19:50:08.3135118Z # This should be reported as a failure instead as it will guarantee to fail when 2024-05-19T19:50:08.3135833Z # Docker tries to run with --gpus all 2024-05-19T19:50:08.3136265Z # 2024-05-19T19:50:08.3136788Z # So, the correct check here is to query one of the missing piece of info like 2024-05-19T19:50:08.3137557Z # GPU name, so that the command can fail accordingly 2024-05-19T19:50:08.3138475Z nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-05-19T19:50:08.3139043Z NVIDIA_SMI_STATUS=$? 2024-05-19T19:50:08.3139403Z  2024-05-19T19:50:08.3140008Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2024-05-19T19:50:08.3140911Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2024-05-19T19:50:08.3141732Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2024-05-19T19:50:08.3142440Z  .github/scripts/stop_runner_service.sh 2024-05-19T19:50:08.3142884Z fi 2024-05-19T19:50:08.3143163Z  2024-05-19T19:50:08.3143822Z # For runner with multiple GPUs, we also want to confirm that the number of GPUs are the 2024-05-19T19:50:08.3144774Z # power of 2, i.e. 1, 2, 4, or 8. This is to avoid flaky test issue when one GPU fails 2024-05-19T19:50:08.3145555Z # https://github.com/pytorch/test-infra/issues/4000 2024-05-19T19:50:08.3146150Z GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) 2024-05-19T19:50:08.3146630Z NVIDIA_SMI_STATUS=$? 2024-05-19T19:50:08.3146989Z  2024-05-19T19:50:08.3147586Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2024-05-19T19:50:08.3148491Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2024-05-19T19:50:08.3149313Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2024-05-19T19:50:08.3150061Z  .github/scripts/stop_runner_service.sh 2024-05-19T19:50:08.3150505Z fi 2024-05-19T19:50:08.3150787Z  2024-05-19T19:50:08.3151144Z # Check the GPU count to be a power of 2 2024-05-19T19:50:08.3151980Z if [ "$GPU_COUNT" -le 8 ] && [ "$GPU_COUNT" -ne 1 ] && [ "$GPU_COUNT" -ne 2 ] && [ "$GPU_COUNT" -ne 4 ] && [ "$GPU_COUNT" -ne 8 ]; then 2024-05-19T19:50:08.3153067Z  echo "NVIDIA driver detects $GPU_COUNT GPUs. The runner has a broken GPU, shutting it down..." 2024-05-19T19:50:08.3153854Z  .github/scripts/stop_runner_service.sh 2024-05-19T19:50:08.3154305Z fi 2024-05-19T19:50:08.3161783Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T19:50:08.3162277Z env: 2024-05-19T19:50:08.3162553Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:50:08.3163012Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:50:08.3163819Z DOCKER_CONTAINER_ID: 3f7c218bc62af08dc9af53b39a20a662e9bceff9e3270378503a0f6bf6fa05bd 2024-05-19T19:50:08.3164596Z RUNNER_WORKSPACE: /home/ec2-user/actions-runner/_work/pytorch 2024-05-19T19:50:08.3165213Z ##[endgroup] 2024-05-19T19:50:08.3185006Z + nvidia-smi 2024-05-19T19:50:09.9557996Z Sun May 19 19:50:09 2024 2024-05-19T19:50:09.9558722Z +-----------------------------------------------------------------------------------------+ 2024-05-19T19:50:09.9559524Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-05-19T19:50:09.9560290Z |-----------------------------------------+------------------------+----------------------+ 2024-05-19T19:50:09.9561071Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-05-19T19:50:09.9561944Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-05-19T19:50:09.9562621Z | | | MIG M. | 2024-05-19T19:50:09.9563178Z |=========================================+========================+======================| 2024-05-19T19:50:09.9645200Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-05-19T19:50:09.9646499Z | 0% 32C P0 60W / 300W | 0MiB / 23028MiB | 5% Default | 2024-05-19T19:50:09.9647641Z | | | N/A | 2024-05-19T19:50:09.9648823Z +-----------------------------------------+------------------------+----------------------+ 2024-05-19T19:50:09.9649864Z 2024-05-19T19:50:09.9650858Z +-----------------------------------------------------------------------------------------+ 2024-05-19T19:50:09.9651483Z | Processes: | 2024-05-19T19:50:09.9652178Z | GPU GI CI PID Type Process name GPU Memory | 2024-05-19T19:50:09.9653151Z | ID ID Usage | 2024-05-19T19:50:09.9653720Z |=========================================================================================| 2024-05-19T19:50:09.9654361Z | No running processes found | 2024-05-19T19:50:09.9655106Z +-----------------------------------------------------------------------------------------+ 2024-05-19T19:50:10.5901213Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-05-19T19:50:12.1471668Z NVIDIA A10G 2024-05-19T19:50:12.5989261Z + NVIDIA_SMI_STATUS=0 2024-05-19T19:50:12.5989800Z + '[' 0 -ne 0 ']' 2024-05-19T19:50:12.5994037Z ++ nvidia-smi --list-gpus 2024-05-19T19:50:12.5994467Z ++ wc -l 2024-05-19T19:50:14.5983508Z + GPU_COUNT=1 2024-05-19T19:50:14.5983896Z + NVIDIA_SMI_STATUS=0 2024-05-19T19:50:14.5984445Z + '[' 0 -ne 0 ']' 2024-05-19T19:50:14.5984786Z + '[' 1 -le 8 ']' 2024-05-19T19:50:14.5985112Z + '[' 1 -ne 1 ']' 2024-05-19T19:50:14.6040301Z Post job cleanup. 2024-05-19T19:50:14.6088679Z Post job cleanup. 2024-05-19T19:50:14.6907427Z [command]/usr/bin/git version 2024-05-19T19:50:14.6941988Z git version 2.40.1 2024-05-19T19:50:14.6979805Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/14fcf66e-44b3-4c87-852d-a9371745ee18' before making global git config changes 2024-05-19T19:50:14.6981235Z Adding repository directory to the temporary git global config as a safe directory 2024-05-19T19:50:14.6985195Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T19:50:14.7014425Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-05-19T19:50:14.7039253Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-05-19T19:50:14.7249058Z Entering 'android/libs/fbjni' 2024-05-19T19:50:14.7285018Z Entering 'third_party/FP16' 2024-05-19T19:50:14.7320993Z Entering 'third_party/FXdiv' 2024-05-19T19:50:14.7356676Z Entering 'third_party/NNPACK' 2024-05-19T19:50:14.7393312Z Entering 'third_party/QNNPACK' 2024-05-19T19:50:14.7429871Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T19:50:14.7464819Z Entering 'third_party/XNNPACK' 2024-05-19T19:50:14.7513980Z Entering 'third_party/benchmark' 2024-05-19T19:50:14.7548160Z Entering 'third_party/cpp-httplib' 2024-05-19T19:50:14.7584243Z Entering 'third_party/cpuinfo' 2024-05-19T19:50:14.7618283Z Entering 'third_party/cudnn_frontend' 2024-05-19T19:50:14.7653853Z Entering 'third_party/cutlass' 2024-05-19T19:50:14.7693690Z Entering 'third_party/eigen' 2024-05-19T19:50:14.7731401Z Entering 'third_party/fbgemm' 2024-05-19T19:50:14.7767055Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T19:50:14.7802041Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T19:50:14.7838994Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T19:50:14.7877840Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T19:50:14.7914156Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T19:50:14.7949401Z Entering 'third_party/flatbuffers' 2024-05-19T19:50:14.7984693Z Entering 'third_party/fmt' 2024-05-19T19:50:14.8019709Z Entering 'third_party/foxi' 2024-05-19T19:50:14.8053025Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T19:50:14.8088435Z Entering 'third_party/gloo' 2024-05-19T19:50:14.8123558Z Entering 'third_party/googletest' 2024-05-19T19:50:14.8159507Z Entering 'third_party/ideep' 2024-05-19T19:50:14.8192934Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T19:50:14.8246442Z Entering 'third_party/ios-cmake' 2024-05-19T19:50:14.8273085Z Entering 'third_party/ittapi' 2024-05-19T19:50:14.8308796Z Entering 'third_party/kineto' 2024-05-19T19:50:14.8344596Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T19:50:14.8379560Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T19:50:14.8415906Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T19:50:14.8453921Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T19:50:14.8490106Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T19:50:14.8524576Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T19:50:14.8563254Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T19:50:14.8597759Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T19:50:14.8631161Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T19:50:14.8666265Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T19:50:14.8701183Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T19:50:14.8735748Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T19:50:14.8770735Z Entering 'third_party/mimalloc' 2024-05-19T19:50:14.8805731Z Entering 'third_party/nccl/nccl' 2024-05-19T19:50:14.8840252Z Entering 'third_party/neon2sse' 2024-05-19T19:50:14.8874993Z Entering 'third_party/nlohmann' 2024-05-19T19:50:14.8911636Z Entering 'third_party/onnx' 2024-05-19T19:50:14.8956231Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T19:50:14.8990971Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T19:50:14.9025681Z Entering 'third_party/onnx-tensorrt' 2024-05-19T19:50:14.9058482Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T19:50:14.9094263Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T19:50:14.9129134Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T19:50:14.9163637Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T19:50:14.9201806Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T19:50:14.9238289Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T19:50:14.9273631Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T19:50:14.9306538Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T19:50:14.9340437Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T19:50:14.9374501Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T19:50:14.9407885Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T19:50:14.9442557Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T19:50:14.9474875Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T19:50:14.9510249Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T19:50:14.9548400Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T19:50:14.9597762Z Entering 'third_party/pocketfft' 2024-05-19T19:50:14.9632741Z Entering 'third_party/protobuf' 2024-05-19T19:50:14.9667670Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T19:50:14.9703251Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T19:50:14.9737286Z Entering 'third_party/psimd' 2024-05-19T19:50:14.9771361Z Entering 'third_party/pthreadpool' 2024-05-19T19:50:14.9803958Z Entering 'third_party/pybind11' 2024-05-19T19:50:14.9840086Z Entering 'third_party/python-peachpy' 2024-05-19T19:50:14.9874831Z Entering 'third_party/sleef' 2024-05-19T19:50:14.9910941Z Entering 'third_party/tbb' 2024-05-19T19:50:14.9945888Z Entering 'third_party/tensorpipe' 2024-05-19T19:50:14.9980835Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T19:50:15.0012114Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T19:50:15.0046258Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T19:50:15.0080190Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T19:50:15.0112927Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T19:50:15.0163558Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-05-19T19:50:15.0181837Z http.https://github.com/.extraheader 2024-05-19T19:50:15.0189273Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-05-19T19:50:15.0218170Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-05-19T19:50:15.0428446Z Entering 'android/libs/fbjni' 2024-05-19T19:50:15.0449770Z http.https://github.com/.extraheader 2024-05-19T19:50:15.0473609Z Entering 'third_party/FP16' 2024-05-19T19:50:15.0493834Z http.https://github.com/.extraheader 2024-05-19T19:50:15.0517388Z Entering 'third_party/FXdiv' 2024-05-19T19:50:15.0537934Z http.https://github.com/.extraheader 2024-05-19T19:50:15.0560692Z Entering 'third_party/NNPACK' 2024-05-19T19:50:15.0581899Z http.https://github.com/.extraheader 2024-05-19T19:50:15.0604109Z Entering 'third_party/QNNPACK' 2024-05-19T19:50:15.0625515Z http.https://github.com/.extraheader 2024-05-19T19:50:15.0650980Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T19:50:15.0673676Z http.https://github.com/.extraheader 2024-05-19T19:50:15.0699046Z Entering 'third_party/XNNPACK' 2024-05-19T19:50:15.0721804Z http.https://github.com/.extraheader 2024-05-19T19:50:15.0756216Z Entering 'third_party/benchmark' 2024-05-19T19:50:15.0778497Z http.https://github.com/.extraheader 2024-05-19T19:50:15.0803031Z Entering 'third_party/cpp-httplib' 2024-05-19T19:50:15.0823650Z http.https://github.com/.extraheader 2024-05-19T19:50:15.0846444Z Entering 'third_party/cpuinfo' 2024-05-19T19:50:15.0868521Z http.https://github.com/.extraheader 2024-05-19T19:50:15.0892151Z Entering 'third_party/cudnn_frontend' 2024-05-19T19:50:15.0913364Z http.https://github.com/.extraheader 2024-05-19T19:50:15.0936686Z Entering 'third_party/cutlass' 2024-05-19T19:50:15.0957453Z http.https://github.com/.extraheader 2024-05-19T19:50:15.0987805Z Entering 'third_party/eigen' 2024-05-19T19:50:15.1009373Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1033749Z Entering 'third_party/fbgemm' 2024-05-19T19:50:15.1055078Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1077254Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T19:50:15.1098841Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1121197Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T19:50:15.1142498Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1166032Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T19:50:15.1186197Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1214150Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T19:50:15.1236023Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1260845Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T19:50:15.1282162Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1306158Z Entering 'third_party/flatbuffers' 2024-05-19T19:50:15.1329535Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1355769Z Entering 'third_party/fmt' 2024-05-19T19:50:15.1377066Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1402241Z Entering 'third_party/foxi' 2024-05-19T19:50:15.1422721Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1446749Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T19:50:15.1468157Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1490697Z Entering 'third_party/gloo' 2024-05-19T19:50:15.1512020Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1534400Z Entering 'third_party/googletest' 2024-05-19T19:50:15.1556422Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1579626Z Entering 'third_party/ideep' 2024-05-19T19:50:15.1600669Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1623732Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T19:50:15.1645378Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1673757Z Entering 'third_party/ios-cmake' 2024-05-19T19:50:15.1695747Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1721628Z Entering 'third_party/ittapi' 2024-05-19T19:50:15.1743306Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1766850Z Entering 'third_party/kineto' 2024-05-19T19:50:15.1789606Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1812216Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T19:50:15.1833267Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1856627Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T19:50:15.1877717Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1902632Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T19:50:15.1925162Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1949395Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T19:50:15.1970240Z http.https://github.com/.extraheader 2024-05-19T19:50:15.1992672Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T19:50:15.2013031Z http.https://github.com/.extraheader 2024-05-19T19:50:15.2035487Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T19:50:15.2055776Z http.https://github.com/.extraheader 2024-05-19T19:50:15.2081203Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T19:50:15.2101991Z http.https://github.com/.extraheader 2024-05-19T19:50:15.2126058Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T19:50:15.2147578Z http.https://github.com/.extraheader 2024-05-19T19:50:15.2171799Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T19:50:15.2191824Z http.https://github.com/.extraheader 2024-05-19T19:50:15.2215745Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T19:50:15.2237888Z http.https://github.com/.extraheader 2024-05-19T19:50:15.2264118Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T19:50:15.2285399Z http.https://github.com/.extraheader 2024-05-19T19:50:15.2307921Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T19:50:15.2328965Z http.https://github.com/.extraheader 2024-05-19T19:50:15.2351037Z Entering 'third_party/mimalloc' 2024-05-19T19:50:15.2372439Z http.https://github.com/.extraheader 2024-05-19T19:50:15.2397345Z Entering 'third_party/nccl/nccl' 2024-05-19T19:50:15.2418351Z http.https://github.com/.extraheader 2024-05-19T19:50:15.2442101Z Entering 'third_party/neon2sse' 2024-05-19T19:50:15.2463949Z http.https://github.com/.extraheader 2024-05-19T19:50:15.2486724Z Entering 'third_party/nlohmann' 2024-05-19T19:50:15.2508147Z http.https://github.com/.extraheader 2024-05-19T19:50:15.2531344Z Entering 'third_party/onnx' 2024-05-19T19:50:15.2554740Z http.https://github.com/.extraheader 2024-05-19T19:50:15.2591905Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T19:50:15.2613894Z http.https://github.com/.extraheader 2024-05-19T19:50:15.2636899Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T19:50:15.2657424Z http.https://github.com/.extraheader 2024-05-19T19:50:15.2684338Z Entering 'third_party/onnx-tensorrt' 2024-05-19T19:50:15.2706210Z http.https://github.com/.extraheader 2024-05-19T19:50:15.2729238Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T19:50:15.2750224Z http.https://github.com/.extraheader 2024-05-19T19:50:15.2776156Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T19:50:15.2797364Z http.https://github.com/.extraheader 2024-05-19T19:50:15.2820934Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T19:50:15.2842255Z http.https://github.com/.extraheader 2024-05-19T19:50:15.2864508Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T19:50:15.2884482Z http.https://github.com/.extraheader 2024-05-19T19:50:15.2913061Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T19:50:15.2934918Z http.https://github.com/.extraheader 2024-05-19T19:50:15.2961188Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T19:50:15.2981772Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3004848Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T19:50:15.3025078Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3049969Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T19:50:15.3071289Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3093100Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T19:50:15.3114815Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3139950Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T19:50:15.3159399Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3184405Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T19:50:15.3204453Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3227247Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T19:50:15.3247636Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3270321Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T19:50:15.3291421Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3316395Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T19:50:15.3336449Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3359823Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T19:50:15.3380662Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3421226Z Entering 'third_party/pocketfft' 2024-05-19T19:50:15.3443895Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3467509Z Entering 'third_party/protobuf' 2024-05-19T19:50:15.3489437Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3514967Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T19:50:15.3535492Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3559638Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T19:50:15.3581852Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3607388Z Entering 'third_party/psimd' 2024-05-19T19:50:15.3629016Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3650788Z Entering 'third_party/pthreadpool' 2024-05-19T19:50:15.3673303Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3695906Z Entering 'third_party/pybind11' 2024-05-19T19:50:15.3717583Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3742287Z Entering 'third_party/python-peachpy' 2024-05-19T19:50:15.3763044Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3786409Z Entering 'third_party/sleef' 2024-05-19T19:50:15.3807495Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3832474Z Entering 'third_party/tbb' 2024-05-19T19:50:15.3852994Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3876070Z Entering 'third_party/tensorpipe' 2024-05-19T19:50:15.3897282Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3920900Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T19:50:15.3942694Z http.https://github.com/.extraheader 2024-05-19T19:50:15.3965408Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T19:50:15.3986311Z http.https://github.com/.extraheader 2024-05-19T19:50:15.4008488Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T19:50:15.4029545Z http.https://github.com/.extraheader 2024-05-19T19:50:15.4054226Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T19:50:15.4077361Z http.https://github.com/.extraheader 2024-05-19T19:50:15.4100007Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T19:50:15.4120756Z http.https://github.com/.extraheader 2024-05-19T19:50:15.4371481Z Cleaning up orphan processes