2024-05-19T18:07:33.6786497Z Current runner version: '2.316.1' 2024-05-19T18:07:33.6792572Z Runner name: 'i-038631eb9562efe08' 2024-05-19T18:07:33.6793478Z Runner group name: 'Default' 2024-05-19T18:07:33.6794461Z Machine name: 'ip-10-0-28-22' 2024-05-19T18:07:33.6799315Z ##[group]GITHUB_TOKEN Permissions 2024-05-19T18:07:33.6801268Z Actions: read 2024-05-19T18:07:33.6801897Z Attestations: read 2024-05-19T18:07:33.6802460Z Checks: read 2024-05-19T18:07:33.6802958Z Contents: read 2024-05-19T18:07:33.6803511Z Deployments: read 2024-05-19T18:07:33.6804072Z Discussions: read 2024-05-19T18:07:33.6804862Z Issues: read 2024-05-19T18:07:33.6805433Z Metadata: read 2024-05-19T18:07:33.6805955Z Packages: read 2024-05-19T18:07:33.6806475Z Pages: read 2024-05-19T18:07:33.6806999Z PullRequests: read 2024-05-19T18:07:33.6807580Z RepositoryProjects: read 2024-05-19T18:07:33.6808209Z SecurityEvents: read 2024-05-19T18:07:33.6808787Z Statuses: read 2024-05-19T18:07:33.6809309Z ##[endgroup] 2024-05-19T18:07:33.6812504Z Secret source: Actions 2024-05-19T18:07:33.6813298Z Prepare workflow directory 2024-05-19T18:07:33.7738498Z Prepare all required actions 2024-05-19T18:07:33.7905294Z Getting action download info 2024-05-19T18:07:33.9710276Z Download action repository 'pytorch/test-infra@main' (SHA:5ab7a0ff6617c37f26f7d3ce525c072902906bdf) 2024-05-19T18:07:34.2875373Z Download action repository 'pytorch/pytorch@main' (SHA:853081a8e7e84ce3f09749929bb3cb998fd95b30) 2024-05-19T18:07:36.9702383Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-05-19T18:07:37.1005528Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-05-19T18:07:37.3570926Z Getting action download info 2024-05-19T18:07:37.4572301Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-05-19T18:07:37.6191672Z Getting action download info 2024-05-19T18:07:37.7090270Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-05-19T18:07:37.8426714Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/121956 (c8c7ddfde57442cc28a394c385cab656ce9b1f5e) 2024-05-19T18:07:37.8428930Z ##[group] Inputs 2024-05-19T18:07:37.8429402Z build-environment: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:07:37.8437528Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-05-19T18:07:37.8446841Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:07:37.8448008Z sync-tag: 2024-05-19T18:07:37.8448932Z timeout-minutes: 240 2024-05-19T18:07:37.8449269Z use-gha: 2024-05-19T18:07:37.8449566Z dashboard-tag: 2024-05-19T18:07:37.8449916Z s3-bucket: gha-artifacts 2024-05-19T18:07:37.8450274Z aws-role-to-assume: 2024-05-19T18:07:37.8450626Z ##[endgroup] 2024-05-19T18:07:37.8451489Z Complete job name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:07:37.9300647Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-05-19T18:07:37.9301219Z with: 2024-05-19T18:07:37.9301837Z github-secret: *** 2024-05-19T18:07:37.9302769Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-05-19T18:07:37.9303749Z activate-with-label: false 2024-05-19T18:07:37.9304117Z label: with-ssh 2024-05-19T18:07:37.9304444Z remove-existing-keys: true 2024-05-19T18:07:37.9304786Z env: 2024-05-19T18:07:37.9305067Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:07:37.9305411Z ##[endgroup] 2024-05-19T18:07:38.0212630Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-05-19T18:07:38.0213698Z ciflow reference detected, attempting to extract PR number 2024-05-19T18:07:39.6063736Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-05-19T18:07:39.6781117Z No SSH keys found for user pytorch-bot[bot] 2024-05-19T18:07:39.6781725Z Grabbing public ssh keys from https://github.com/nWEIdia.keys 2024-05-19T18:07:39.7379423Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-05-19T18:07:39.7393292Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-05-19T18:07:39.7417260Z Login using: ssh ec2-user@ec2-3-234-205-253.compute-1.amazonaws.com 2024-05-19T18:07:39.7418035Z All testing is done inside the container, to start an interactive session run: 2024-05-19T18:07:39.7418828Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-05-19T18:07:39.7659857Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-05-19T18:07:39.7660420Z with: 2024-05-19T18:07:39.7660714Z submodules: recursive 2024-05-19T18:07:39.7661047Z fetch-depth: 0 2024-05-19T18:07:39.7661348Z env: 2024-05-19T18:07:39.7661628Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:07:39.7661970Z ##[endgroup] 2024-05-19T18:07:39.7874448Z ##[group]Run retry () { 2024-05-19T18:07:39.7874866Z retry () { 2024-05-19T18:07:39.7875387Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-05-19T18:07:39.7875944Z } 2024-05-19T18:07:39.7876264Z echo "${GITHUB_WORKSPACE}" 2024-05-19T18:07:39.7876687Z if [ -z "${NO_SUDO}" ]; then 2024-05-19T18:07:39.7877175Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-05-19T18:07:39.7877634Z else 2024-05-19T18:07:39.7877982Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-05-19T18:07:39.7878403Z fi 2024-05-19T18:07:39.7878755Z mkdir "${GITHUB_WORKSPACE}" 2024-05-19T18:07:39.7890264Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:07:39.7890764Z env: 2024-05-19T18:07:39.7891050Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:07:39.7891402Z NO_SUDO: 2024-05-19T18:07:39.7891690Z ##[endgroup] 2024-05-19T18:07:39.7979863Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T18:07:39.8159841Z ##[group]Run malfet/checkout@silent-checkout 2024-05-19T18:07:39.8160276Z with: 2024-05-19T18:07:39.8160712Z ref: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:07:39.8161173Z fetch-depth: 0 2024-05-19T18:07:39.8161498Z submodules: recursive 2024-05-19T18:07:39.8161850Z quiet-checkout: true 2024-05-19T18:07:39.8162198Z repository: pytorch/pytorch 2024-05-19T18:07:39.8162681Z token: *** 2024-05-19T18:07:39.8162979Z ssh-strict: true 2024-05-19T18:07:39.8163324Z persist-credentials: true 2024-05-19T18:07:39.8163684Z clean: true 2024-05-19T18:07:39.8164038Z sparse-checkout-cone-mode: true 2024-05-19T18:07:39.8164435Z lfs: false 2024-05-19T18:07:39.8164958Z set-safe-directory: true 2024-05-19T18:07:39.8165311Z env: 2024-05-19T18:07:39.8165594Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:07:39.8165943Z ##[endgroup] 2024-05-19T18:07:39.9073613Z Syncing repository: pytorch/pytorch 2024-05-19T18:07:39.9075190Z ##[group]Getting Git version info 2024-05-19T18:07:39.9075947Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-05-19T18:07:39.9076805Z [command]/usr/bin/git version 2024-05-19T18:07:39.9077170Z git version 2.40.1 2024-05-19T18:07:39.9085464Z ##[endgroup] 2024-05-19T18:07:39.9099830Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/c7e461fd-3a2a-456c-bf2a-1fa7e1c25ab7' before making global git config changes 2024-05-19T18:07:39.9101061Z Adding repository directory to the temporary git global config as a safe directory 2024-05-19T18:07:39.9104435Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T18:07:39.9134339Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-05-19T18:07:39.9138137Z ##[group]Initializing the repository 2024-05-19T18:07:39.9140933Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T18:07:39.9160605Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-05-19T18:07:39.9161618Z hint: is subject to change. To configure the initial branch name to use in all 2024-05-19T18:07:39.9162413Z hint: of your new repositories, which will suppress this warning, call: 2024-05-19T18:07:39.9163174Z hint: 2024-05-19T18:07:39.9163633Z hint: git config --global init.defaultBranch 2024-05-19T18:07:39.9164108Z hint: 2024-05-19T18:07:39.9164827Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-05-19T18:07:39.9165680Z hint: 'development'. The just-created branch can be renamed via this command: 2024-05-19T18:07:39.9166492Z hint: 2024-05-19T18:07:39.9166949Z hint: git branch -m 2024-05-19T18:07:39.9167688Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-05-19T18:07:39.9170389Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-05-19T18:07:39.9190813Z ##[endgroup] 2024-05-19T18:07:39.9191413Z ##[group]Disabling automatic garbage collection 2024-05-19T18:07:39.9193458Z [command]/usr/bin/git config --local gc.auto 0 2024-05-19T18:07:39.9213669Z ##[endgroup] 2024-05-19T18:07:39.9214184Z ##[group]Setting up auth 2024-05-19T18:07:39.9219617Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-05-19T18:07:39.9244894Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-05-19T18:07:39.9449283Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-05-19T18:07:39.9471118Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-05-19T18:07:39.9664902Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-05-19T18:07:39.9708234Z ##[endgroup] 2024-05-19T18:07:39.9708784Z ##[group]Fetching the repository 2024-05-19T18:07:39.9713968Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-05-19T18:07:49.5000667Z remote: Enumerating objects: 1118700 2024-05-19T18:07:49.6003768Z remote: Enumerating objects: 1118702 2024-05-19T18:07:49.6004912Z remote: Enumerating objects: 1119785, done. 2024-05-19T18:07:49.6005821Z remote: Counting objects: 0% (1/1085) 2024-05-19T18:07:49.6006637Z remote: Counting objects: 1% (11/1085) 2024-05-19T18:07:49.6007449Z remote: Counting objects: 2% (22/1085) 2024-05-19T18:07:49.6008222Z remote: Counting objects: 3% (33/1085) 2024-05-19T18:07:49.6008985Z remote: Counting objects: 4% (44/1085) 2024-05-19T18:07:49.6009594Z remote: Counting objects: 5% (55/1085) 2024-05-19T18:07:49.6010339Z remote: Counting objects: 6% (66/1085) 2024-05-19T18:07:49.6011126Z remote: Counting objects: 7% (76/1085) 2024-05-19T18:07:49.6011896Z remote: Counting objects: 8% (87/1085) 2024-05-19T18:07:49.6012582Z remote: Counting objects: 9% (98/1085) 2024-05-19T18:07:49.6013246Z remote: Counting objects: 10% (109/1085) 2024-05-19T18:07:49.6013870Z remote: Counting objects: 11% (120/1085) 2024-05-19T18:07:49.6014512Z remote: Counting objects: 12% (131/1085) 2024-05-19T18:07:49.6015088Z remote: Counting objects: 13% (142/1085) 2024-05-19T18:07:49.6015662Z remote: Counting objects: 14% (152/1085) 2024-05-19T18:07:49.6016232Z remote: Counting objects: 15% (163/1085) 2024-05-19T18:07:49.6016806Z remote: Counting objects: 16% (174/1085) 2024-05-19T18:07:49.6017377Z remote: Counting objects: 17% (185/1085) 2024-05-19T18:07:49.6017942Z remote: Counting objects: 18% (196/1085) 2024-05-19T18:07:49.6018512Z remote: Counting objects: 19% (207/1085) 2024-05-19T18:07:49.6019100Z remote: Counting objects: 20% (217/1085) 2024-05-19T18:07:49.6019673Z remote: Counting objects: 21% (228/1085) 2024-05-19T18:07:49.6020237Z remote: Counting objects: 22% (239/1085) 2024-05-19T18:07:49.6020863Z remote: Counting objects: 23% (250/1085) 2024-05-19T18:07:49.6021434Z remote: Counting objects: 24% (261/1085) 2024-05-19T18:07:49.6022002Z remote: Counting objects: 25% (272/1085) 2024-05-19T18:07:49.6022588Z remote: Counting objects: 26% (283/1085) 2024-05-19T18:07:49.6023340Z remote: Counting objects: 27% (293/1085) 2024-05-19T18:07:49.6023918Z remote: Counting objects: 28% (304/1085) 2024-05-19T18:07:49.6024493Z remote: Counting objects: 29% (315/1085) 2024-05-19T18:07:49.6025066Z remote: Counting objects: 30% (326/1085) 2024-05-19T18:07:49.6025633Z remote: Counting objects: 31% (337/1085) 2024-05-19T18:07:49.6026215Z remote: Counting objects: 32% (348/1085) 2024-05-19T18:07:49.6026796Z remote: Counting objects: 33% (359/1085) 2024-05-19T18:07:49.6027368Z remote: Counting objects: 34% (369/1085) 2024-05-19T18:07:49.6027934Z remote: Counting objects: 35% (380/1085) 2024-05-19T18:07:49.6028506Z remote: Counting objects: 36% (391/1085) 2024-05-19T18:07:49.6029081Z remote: Counting objects: 37% (402/1085) 2024-05-19T18:07:49.6029706Z remote: Counting objects: 38% (413/1085) 2024-05-19T18:07:49.6030273Z remote: Counting objects: 39% (424/1085) 2024-05-19T18:07:49.6030860Z remote: Counting objects: 40% (434/1085) 2024-05-19T18:07:49.6031425Z remote: Counting objects: 41% (445/1085) 2024-05-19T18:07:49.6031998Z remote: Counting objects: 42% (456/1085) 2024-05-19T18:07:49.6032664Z remote: Counting objects: 43% (467/1085) 2024-05-19T18:07:49.6033249Z remote: Counting objects: 44% (478/1085) 2024-05-19T18:07:49.6033826Z remote: Counting objects: 45% (489/1085) 2024-05-19T18:07:49.6034496Z remote: Counting objects: 46% (500/1085) 2024-05-19T18:07:49.6035061Z remote: Counting objects: 47% (510/1085) 2024-05-19T18:07:49.6035634Z remote: Counting objects: 48% (521/1085) 2024-05-19T18:07:49.6036204Z remote: Counting objects: 49% (532/1085) 2024-05-19T18:07:49.6036776Z remote: Counting objects: 50% (543/1085) 2024-05-19T18:07:49.6037342Z remote: Counting objects: 51% (554/1085) 2024-05-19T18:07:49.6037913Z remote: Counting objects: 52% (565/1085) 2024-05-19T18:07:49.6038501Z remote: Counting objects: 53% (576/1085) 2024-05-19T18:07:49.6039076Z remote: Counting objects: 54% (586/1085) 2024-05-19T18:07:49.6039643Z remote: Counting objects: 55% (597/1085) 2024-05-19T18:07:49.6040217Z remote: Counting objects: 56% (608/1085) 2024-05-19T18:07:49.6040790Z remote: Counting objects: 57% (619/1085) 2024-05-19T18:07:49.6041367Z remote: Counting objects: 58% (630/1085) 2024-05-19T18:07:49.6041942Z remote: Counting objects: 59% (641/1085) 2024-05-19T18:07:49.6042517Z remote: Counting objects: 60% (651/1085) 2024-05-19T18:07:49.6043091Z remote: Counting objects: 61% (662/1085) 2024-05-19T18:07:49.6043683Z remote: Counting objects: 62% (673/1085) 2024-05-19T18:07:49.6044257Z remote: Counting objects: 63% (684/1085) 2024-05-19T18:07:49.6044970Z remote: Counting objects: 64% (695/1085) 2024-05-19T18:07:49.6045563Z remote: Counting objects: 65% (706/1085) 2024-05-19T18:07:49.6046150Z remote: Counting objects: 66% (717/1085) 2024-05-19T18:07:49.6046725Z remote: Counting objects: 67% (727/1085) 2024-05-19T18:07:49.6047294Z remote: Counting objects: 68% (738/1085) 2024-05-19T18:07:49.6047864Z remote: Counting objects: 69% (749/1085) 2024-05-19T18:07:49.6048444Z remote: Counting objects: 70% (760/1085) 2024-05-19T18:07:49.6049010Z remote: Counting objects: 71% (771/1085) 2024-05-19T18:07:49.6049585Z remote: Counting objects: 72% (782/1085) 2024-05-19T18:07:49.6050153Z remote: Counting objects: 73% (793/1085) 2024-05-19T18:07:49.6050720Z remote: Counting objects: 74% (803/1085) 2024-05-19T18:07:49.6051281Z remote: Counting objects: 75% (814/1085) 2024-05-19T18:07:49.6051861Z remote: Counting objects: 76% (825/1085) 2024-05-19T18:07:49.6052495Z remote: Counting objects: 77% (836/1085) 2024-05-19T18:07:49.6053139Z remote: Counting objects: 78% (847/1085) 2024-05-19T18:07:49.6053815Z remote: Counting objects: 79% (858/1085) 2024-05-19T18:07:49.6054459Z remote: Counting objects: 80% (868/1085) 2024-05-19T18:07:49.6055058Z remote: Counting objects: 81% (879/1085) 2024-05-19T18:07:49.6055657Z remote: Counting objects: 82% (890/1085) 2024-05-19T18:07:49.6056243Z remote: Counting objects: 83% (901/1085) 2024-05-19T18:07:49.6056848Z remote: Counting objects: 84% (912/1085) 2024-05-19T18:07:49.6057455Z remote: Counting objects: 85% (923/1085) 2024-05-19T18:07:49.6058062Z remote: Counting objects: 86% (934/1085) 2024-05-19T18:07:49.6058638Z remote: Counting objects: 87% (944/1085) 2024-05-19T18:07:49.6059224Z remote: Counting objects: 88% (955/1085) 2024-05-19T18:07:49.6059875Z remote: Counting objects: 89% (966/1085) 2024-05-19T18:07:49.6060474Z remote: Counting objects: 90% (977/1085) 2024-05-19T18:07:49.6061056Z remote: Counting objects: 91% (988/1085) 2024-05-19T18:07:49.6061652Z remote: Counting objects: 92% (999/1085) 2024-05-19T18:07:49.6062259Z remote: Counting objects: 93% (1010/1085) 2024-05-19T18:07:49.6062878Z remote: Counting objects: 94% (1020/1085) 2024-05-19T18:07:49.6063589Z remote: Counting objects: 95% (1031/1085) 2024-05-19T18:07:49.6064256Z remote: Counting objects: 96% (1042/1085) 2024-05-19T18:07:49.6064976Z remote: Counting objects: 97% (1053/1085) 2024-05-19T18:07:49.6065594Z remote: Counting objects: 98% (1064/1085) 2024-05-19T18:07:49.6066185Z remote: Counting objects: 99% (1075/1085) 2024-05-19T18:07:49.6066793Z remote: Counting objects: 100% (1085/1085) 2024-05-19T18:07:49.6067443Z remote: Counting objects: 100% (1085/1085), done. 2024-05-19T18:07:49.6068095Z remote: Compressing objects: 0% (1/372) 2024-05-19T18:07:49.6154267Z remote: Compressing objects: 1% (4/372) 2024-05-19T18:07:49.6294108Z remote: Compressing objects: 2% (8/372) 2024-05-19T18:07:49.6468901Z remote: Compressing objects: 3% (12/372) 2024-05-19T18:07:49.6858217Z remote: Compressing objects: 4% (15/372) 2024-05-19T18:07:49.7053981Z remote: Compressing objects: 5% (19/372) 2024-05-19T18:07:49.7054600Z remote: Compressing objects: 6% (23/372) 2024-05-19T18:07:49.7055203Z remote: Compressing objects: 7% (27/372) 2024-05-19T18:07:49.7055824Z remote: Compressing objects: 8% (30/372) 2024-05-19T18:07:49.7056548Z remote: Compressing objects: 9% (34/372) 2024-05-19T18:07:49.7057184Z remote: Compressing objects: 10% (38/372) 2024-05-19T18:07:49.7057838Z remote: Compressing objects: 11% (41/372) 2024-05-19T18:07:49.7110890Z remote: Compressing objects: 12% (45/372) 2024-05-19T18:07:49.7112186Z remote: Compressing objects: 13% (49/372) 2024-05-19T18:07:49.7113536Z remote: Compressing objects: 14% (53/372) 2024-05-19T18:07:49.7114888Z remote: Compressing objects: 15% (56/372) 2024-05-19T18:07:49.7116128Z remote: Compressing objects: 16% (60/372) 2024-05-19T18:07:49.7117409Z remote: Compressing objects: 17% (64/372) 2024-05-19T18:07:49.7118710Z remote: Compressing objects: 18% (67/372) 2024-05-19T18:07:49.7125667Z remote: Compressing objects: 19% (71/372) 2024-05-19T18:07:49.7136959Z remote: Compressing objects: 20% (75/372) 2024-05-19T18:07:49.7137595Z remote: Compressing objects: 21% (79/372) 2024-05-19T18:07:49.7138236Z remote: Compressing objects: 22% (82/372) 2024-05-19T18:07:49.7138889Z remote: Compressing objects: 23% (86/372) 2024-05-19T18:07:49.7143615Z remote: Compressing objects: 24% (90/372) 2024-05-19T18:07:49.7144549Z remote: Compressing objects: 25% (93/372) 2024-05-19T18:07:49.7149644Z remote: Compressing objects: 26% (97/372) 2024-05-19T18:07:49.7153272Z remote: Compressing objects: 27% (101/372) 2024-05-19T18:07:49.7154396Z remote: Compressing objects: 28% (105/372) 2024-05-19T18:07:49.7159915Z remote: Compressing objects: 29% (108/372) 2024-05-19T18:07:49.7161667Z remote: Compressing objects: 30% (112/372) 2024-05-19T18:07:49.7165555Z remote: Compressing objects: 31% (116/372) 2024-05-19T18:07:49.7168473Z remote: Compressing objects: 32% (120/372) 2024-05-19T18:07:49.7173700Z remote: Compressing objects: 33% (123/372) 2024-05-19T18:07:49.7176073Z remote: Compressing objects: 34% (127/372) 2024-05-19T18:07:49.7179481Z remote: Compressing objects: 35% (131/372) 2024-05-19T18:07:49.7213440Z remote: Compressing objects: 36% (134/372) 2024-05-19T18:07:49.7214128Z remote: Compressing objects: 37% (138/372) 2024-05-19T18:07:49.7214743Z remote: Compressing objects: 38% (142/372) 2024-05-19T18:07:49.7215486Z remote: Compressing objects: 39% (146/372) 2024-05-19T18:07:49.7216348Z remote: Compressing objects: 40% (149/372) 2024-05-19T18:07:49.7217163Z remote: Compressing objects: 41% (153/372) 2024-05-19T18:07:49.7217866Z remote: Compressing objects: 42% (157/372) 2024-05-19T18:07:49.7218681Z remote: Compressing objects: 43% (160/372) 2024-05-19T18:07:49.7219416Z remote: Compressing objects: 44% (164/372) 2024-05-19T18:07:49.7220208Z remote: Compressing objects: 45% (168/372) 2024-05-19T18:07:49.7221138Z remote: Compressing objects: 46% (172/372) 2024-05-19T18:07:49.7221822Z remote: Compressing objects: 47% (175/372) 2024-05-19T18:07:49.7222629Z remote: Compressing objects: 48% (179/372) 2024-05-19T18:07:49.7223416Z remote: Compressing objects: 49% (183/372) 2024-05-19T18:07:49.7224221Z remote: Compressing objects: 50% (186/372) 2024-05-19T18:07:49.7224958Z remote: Compressing objects: 51% (190/372) 2024-05-19T18:07:49.7225771Z remote: Compressing objects: 52% (194/372) 2024-05-19T18:07:49.7226481Z remote: Compressing objects: 53% (198/372) 2024-05-19T18:07:49.7227131Z remote: Compressing objects: 54% (201/372) 2024-05-19T18:07:49.7227912Z remote: Compressing objects: 55% (205/372) 2024-05-19T18:07:49.7228644Z remote: Compressing objects: 56% (209/372) 2024-05-19T18:07:49.7229463Z remote: Compressing objects: 57% (213/372) 2024-05-19T18:07:49.7230260Z remote: Compressing objects: 58% (216/372) 2024-05-19T18:07:49.7231014Z remote: Compressing objects: 59% (220/372) 2024-05-19T18:07:49.7231820Z remote: Compressing objects: 60% (224/372) 2024-05-19T18:07:49.7232707Z remote: Compressing objects: 61% (227/372) 2024-05-19T18:07:49.7233508Z remote: Compressing objects: 62% (231/372) 2024-05-19T18:07:49.7234316Z remote: Compressing objects: 63% (235/372) 2024-05-19T18:07:49.7235129Z remote: Compressing objects: 64% (239/372) 2024-05-19T18:07:49.7235947Z remote: Compressing objects: 65% (242/372) 2024-05-19T18:07:49.7236760Z remote: Compressing objects: 66% (246/372) 2024-05-19T18:07:49.7237577Z remote: Compressing objects: 67% (250/372) 2024-05-19T18:07:49.7238382Z remote: Compressing objects: 68% (253/372) 2024-05-19T18:07:49.7239161Z remote: Compressing objects: 69% (257/372) 2024-05-19T18:07:49.7239860Z remote: Compressing objects: 70% (261/372) 2024-05-19T18:07:49.7240590Z remote: Compressing objects: 71% (265/372) 2024-05-19T18:07:49.7241297Z remote: Compressing objects: 72% (268/372) 2024-05-19T18:07:49.7242064Z remote: Compressing objects: 73% (272/372) 2024-05-19T18:07:49.7242668Z remote: Compressing objects: 74% (276/372) 2024-05-19T18:07:49.7243273Z remote: Compressing objects: 75% (279/372) 2024-05-19T18:07:49.7243892Z remote: Compressing objects: 76% (283/372) 2024-05-19T18:07:49.7244499Z remote: Compressing objects: 77% (287/372) 2024-05-19T18:07:49.7245384Z remote: Compressing objects: 78% (291/372) 2024-05-19T18:07:49.7245992Z remote: Compressing objects: 79% (294/372) 2024-05-19T18:07:49.7246599Z remote: Compressing objects: 80% (298/372) 2024-05-19T18:07:49.7247213Z remote: Compressing objects: 81% (302/372) 2024-05-19T18:07:49.7247832Z remote: Compressing objects: 82% (306/372) 2024-05-19T18:07:49.7248432Z remote: Compressing objects: 83% (309/372) 2024-05-19T18:07:49.7249048Z remote: Compressing objects: 84% (313/372) 2024-05-19T18:07:49.7249669Z remote: Compressing objects: 85% (317/372) 2024-05-19T18:07:49.7250265Z remote: Compressing objects: 86% (320/372) 2024-05-19T18:07:49.7250875Z remote: Compressing objects: 87% (324/372) 2024-05-19T18:07:49.7251501Z remote: Compressing objects: 88% (328/372) 2024-05-19T18:07:49.7252096Z remote: Compressing objects: 89% (332/372) 2024-05-19T18:07:49.7252702Z remote: Compressing objects: 90% (335/372) 2024-05-19T18:07:49.7253318Z remote: Compressing objects: 91% (339/372) 2024-05-19T18:07:49.7253919Z remote: Compressing objects: 92% (343/372) 2024-05-19T18:07:49.7254512Z remote: Compressing objects: 93% (346/372) 2024-05-19T18:07:49.7255114Z remote: Compressing objects: 94% (350/372) 2024-05-19T18:07:49.7255718Z remote: Compressing objects: 95% (354/372) 2024-05-19T18:07:49.7256310Z remote: Compressing objects: 96% (358/372) 2024-05-19T18:07:49.7256998Z remote: Compressing objects: 97% (361/372) 2024-05-19T18:07:49.7257601Z remote: Compressing objects: 98% (365/372) 2024-05-19T18:07:49.7258206Z remote: Compressing objects: 99% (369/372) 2024-05-19T18:07:49.7258800Z remote: Compressing objects: 100% (372/372) 2024-05-19T18:07:49.7259445Z remote: Compressing objects: 100% (372/372), done. 2024-05-19T18:08:11.5339608Z remote: Total 1119785 (delta 765), reused 958 (delta 703), pack-reused 1118700 2024-05-19T18:08:37.6629075Z [command]/usr/bin/git rev-parse --verify --quiet c8c7ddfde57442cc28a394c385cab656ce9b1f5e^{object} 2024-05-19T18:08:37.6646203Z c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:08:37.6649682Z ##[endgroup] 2024-05-19T18:08:37.6650246Z ##[group]Determining the checkout info 2024-05-19T18:08:37.6651183Z ##[endgroup] 2024-05-19T18:08:37.6651717Z ##[group]Checking out the ref 2024-05-19T18:08:37.6654857Z [command]/usr/bin/git checkout --quiet --force c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:08:38.8752580Z ##[endgroup] 2024-05-19T18:08:38.8753736Z ##[group]Setting up auth for fetching submodules 2024-05-19T18:08:38.8758508Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-05-19T18:08:38.8795516Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-05-19T18:08:38.8813116Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-05-19T18:08:38.8830951Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-05-19T18:08:38.8846314Z ##[endgroup] 2024-05-19T18:08:38.8846840Z ##[group]Fetching submodules 2024-05-19T18:08:38.8849733Z [command]/usr/bin/git submodule sync --recursive 2024-05-19T18:08:38.9053252Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-05-19T18:08:38.9250873Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-05-19T18:08:38.9252296Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-05-19T18:08:38.9253657Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-05-19T18:08:38.9254989Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-05-19T18:08:38.9256772Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2024-05-19T18:08:38.9258886Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-05-19T18:08:38.9260890Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-05-19T18:08:38.9262647Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-05-19T18:08:38.9265104Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-05-19T18:08:38.9267077Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-05-19T18:08:38.9269636Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-05-19T18:08:38.9271532Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-05-19T18:08:38.9274354Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-05-19T18:08:38.9276814Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-05-19T18:08:38.9279480Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-05-19T18:08:38.9281927Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-05-19T18:08:38.9284787Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-05-19T18:08:38.9288040Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:08:38.9290569Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-05-19T18:08:38.9293470Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-05-19T18:08:38.9296356Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-05-19T18:08:38.9299625Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2024-05-19T18:08:38.9302682Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-05-19T18:08:38.9305811Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-05-19T18:08:38.9309105Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-05-19T18:08:38.9312384Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-05-19T18:08:38.9315873Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2024-05-19T18:08:38.9319206Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-05-19T18:08:38.9322611Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-05-19T18:08:38.9326631Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2024-05-19T18:08:38.9330338Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-05-19T18:08:38.9333743Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-05-19T18:08:38.9337697Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-05-19T18:08:38.9341328Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-05-19T18:08:38.9345285Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-05-19T18:08:38.9349094Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-05-19T18:08:38.9353276Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-05-19T18:08:38.9357289Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-05-19T18:08:38.9361367Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2024-05-19T18:08:38.9366790Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-05-19T18:08:38.9384580Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-05-19T18:08:39.1817301Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-05-19T18:08:56.6975028Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-05-19T18:08:56.8374496Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-05-19T18:08:57.0838713Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2024-05-19T18:08:57.2982397Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-05-19T18:08:59.3093284Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-05-19T18:09:08.0509546Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-05-19T18:09:08.4025670Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-05-19T18:09:09.0472806Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-05-19T18:09:09.6120004Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-05-19T18:09:10.8261025Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-05-19T18:09:12.5825448Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-05-19T18:09:18.2107223Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-05-19T18:09:19.5428502Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-05-19T18:09:21.1225787Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-05-19T18:09:22.1780691Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-05-19T18:09:22.3117064Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-05-19T18:09:22.6746380Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-05-19T18:09:22.9618244Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-05-19T18:09:23.8456059Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-05-19T18:09:24.1816422Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2024-05-19T18:09:24.3205159Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-05-19T18:09:24.5458449Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-05-19T18:09:25.9250814Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-05-19T18:09:26.5391803Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-05-19T18:09:27.1022651Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2024-05-19T18:09:27.3394608Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-05-19T18:09:32.8721467Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-05-19T18:09:35.2592847Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2024-05-19T18:09:35.7018918Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-05-19T18:09:39.4658954Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-05-19T18:09:39.6353755Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-05-19T18:09:46.9296785Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-05-19T18:09:47.0704091Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-05-19T18:09:47.2382258Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-05-19T18:09:48.0710001Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-05-19T18:09:48.3216290Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-05-19T18:09:48.9293804Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2024-05-19T18:10:21.0758416Z ##[error]error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8) 2024-05-19T18:10:21.0759984Z ##[error]error: 3004 bytes of body are still expected 2024-05-19T18:10:21.0760873Z fetch-pack: unexpected disconnect while reading sideband packet 2024-05-19T18:10:21.0761659Z ##[error]fatal: early EOF 2024-05-19T18:10:21.0762420Z ##[error]fatal: fetch-pack: invalid index-pack output 2024-05-19T18:10:21.0764333Z ##[error]fatal: clone of 'https://github.com/01org/tbb' into submodule path '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb' failed 2024-05-19T18:10:21.0765927Z Failed to clone 'third_party/tbb'. Retry scheduled 2024-05-19T18:10:21.0780559Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-05-19T18:10:21.4663759Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2024-05-19T18:10:23.5019074Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-05-19T18:10:23.5098501Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-05-19T18:10:23.5158266Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-05-19T18:10:23.5336302Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-05-19T18:10:23.5515475Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2024-05-19T18:10:23.5820980Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-05-19T18:10:24.3338704Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-05-19T18:10:24.3511567Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-05-19T18:10:24.3860640Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-05-19T18:10:24.4669136Z Submodule path 'third_party/cpuinfo': checked out 'd6860c477c99f1fce9e28eb206891af3c0e1a1d7' 2024-05-19T18:10:24.4891891Z Submodule path 'third_party/cudnn_frontend': checked out '150798fe976556078f443fdb059a1ff0361f58a2' 2024-05-19T18:10:24.9120943Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-05-19T18:10:25.1260429Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-05-19T18:10:25.1845987Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-05-19T18:10:25.1857038Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:10:25.1858886Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:10:25.1861275Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:10:25.1863536Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:10:25.1866025Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:10:25.1883977Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-05-19T18:10:26.1146780Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-05-19T18:10:26.6535260Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-05-19T18:10:28.3201970Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-05-19T18:10:29.2297537Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-05-19T18:10:29.5895803Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-05-19T18:10:29.6735446Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-05-19T18:10:30.0101026Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-05-19T18:10:30.0638610Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-05-19T18:10:30.0721121Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-05-19T18:10:30.1626652Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-05-19T18:10:30.1925022Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-05-19T18:10:30.1990068Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-05-19T18:10:30.2293678Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-05-19T18:10:30.2483751Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-05-19T18:10:30.2846046Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-05-19T18:10:30.2938109Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-05-19T18:10:30.2948731Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-05-19T18:10:30.2963932Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-05-19T18:10:40.9010689Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-05-19T18:10:40.9082416Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2024-05-19T18:10:40.9210228Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-05-19T18:10:40.9981237Z Submodule path 'third_party/kineto': checked out '3a81076cc97092666f319846f32f36b73ce2293e' 2024-05-19T18:10:40.9991477Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:10:40.9993183Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:10:40.9995051Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:10:41.0009599Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-05-19T18:10:41.4593466Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-05-19T18:10:42.5377289Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-05-19T18:10:43.4955162Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-05-19T18:10:43.4966546Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:10:43.4968623Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:10:43.4970455Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:10:43.4972314Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:10:43.4974236Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:10:43.4976132Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:10:43.4978047Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:10:43.4980065Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:10:43.4997719Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-05-19T18:10:44.4899339Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-05-19T18:10:44.8448089Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-05-19T18:10:45.9254391Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-05-19T18:10:46.1808088Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-05-19T18:10:46.5180234Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-05-19T18:10:47.4371387Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-05-19T18:10:52.9762182Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-05-19T18:10:53.3209863Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-05-19T18:10:53.3344560Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-05-19T18:10:53.3622788Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-05-19T18:10:53.3719107Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-05-19T18:10:53.3728151Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:10:53.3744771Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-05-19T18:10:53.6499382Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-05-19T18:10:53.6632248Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-05-19T18:10:53.6966208Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-05-19T18:10:53.7771321Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-05-19T18:10:53.7887935Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-05-19T18:10:53.8168833Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-05-19T18:10:53.8662937Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-05-19T18:10:53.8960548Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-05-19T18:10:53.9154524Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-05-19T18:10:53.9260103Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2024-05-19T18:10:54.0114020Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-05-19T18:10:54.2827769Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-05-19T18:10:54.2853639Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-05-19T18:10:54.2855463Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-05-19T18:10:54.2872831Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-05-19T18:10:54.6256536Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-05-19T18:10:55.5429338Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-05-19T18:10:55.5704094Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-05-19T18:10:55.5836591Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2024-05-19T18:10:55.5846499Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:10:55.5861349Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2024-05-19T18:10:58.0812446Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2024-05-19T18:10:58.0825484Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:10:58.0827230Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:10:58.0844289Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2024-05-19T18:10:58.4802069Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2024-05-19T18:10:59.4075873Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2024-05-19T18:10:59.4723181Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2024-05-19T18:10:59.4732764Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:10:59.4749174Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2024-05-19T18:10:59.6440162Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-05-19T18:10:59.6976854Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-05-19T18:10:59.6988495Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:10:59.6990542Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:10:59.6992398Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:10:59.6994356Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:10:59.6996523Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:10:59.6998822Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:10:59.7000894Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:10:59.7002580Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:10:59.7020918Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-05-19T18:11:00.0587480Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-05-19T18:11:01.0013349Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-05-19T18:11:01.3304555Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-05-19T18:11:06.8976202Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-05-19T18:11:07.1482074Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-05-19T18:11:07.3337643Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-05-19T18:11:07.6189436Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-05-19T18:11:13.9228495Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-05-19T18:11:13.9558014Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-05-19T18:11:13.9675340Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-05-19T18:11:14.0513533Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-05-19T18:11:14.0601412Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-05-19T18:11:14.0703308Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-05-19T18:11:14.0813073Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-05-19T18:11:14.0822695Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:11:14.0824758Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:11:14.0839366Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-05-19T18:11:15.7987574Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-05-19T18:11:16.9585242Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-05-19T18:11:16.9945615Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-05-19T18:11:17.3454523Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-05-19T18:11:17.3530629Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-05-19T18:11:17.5675541Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-05-19T18:11:17.5690880Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:11:17.5692748Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-05-19T18:11:17.5707614Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-05-19T18:11:17.9185855Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-05-19T18:11:18.8504794Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-05-19T18:11:18.9115187Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-05-19T18:11:18.9173922Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-05-19T18:11:18.9259107Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-05-19T18:11:18.9532094Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-05-19T18:11:18.9752194Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-05-19T18:11:19.0086158Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-05-19T18:11:19.0987787Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2024-05-19T18:11:19.1190545Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-05-19T18:11:19.1201958Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:11:19.1203877Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:11:19.1205867Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:11:19.1207806Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:11:19.1224856Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-05-19T18:11:20.0757465Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-05-19T18:11:20.2767276Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-05-19T18:11:21.4566595Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-05-19T18:11:22.3858001Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-05-19T18:11:22.3966507Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-05-19T18:11:22.4535078Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-05-19T18:11:22.4762208Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-05-19T18:11:22.4773413Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:11:22.4790004Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-05-19T18:11:22.6717198Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-05-19T18:11:22.8513943Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2024-05-19T18:11:22.8537942Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-05-19T18:11:22.8739621Z Entering 'android/libs/fbjni' 2024-05-19T18:11:22.8767416Z Entering 'third_party/FP16' 2024-05-19T18:11:22.8796005Z Entering 'third_party/FXdiv' 2024-05-19T18:11:22.8823270Z Entering 'third_party/NNPACK' 2024-05-19T18:11:22.8848599Z Entering 'third_party/QNNPACK' 2024-05-19T18:11:22.8877223Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T18:11:22.8903650Z Entering 'third_party/XNNPACK' 2024-05-19T18:11:22.8941408Z Entering 'third_party/benchmark' 2024-05-19T18:11:22.8969631Z Entering 'third_party/cpp-httplib' 2024-05-19T18:11:22.8997969Z Entering 'third_party/cpuinfo' 2024-05-19T18:11:22.9025671Z Entering 'third_party/cudnn_frontend' 2024-05-19T18:11:22.9051995Z Entering 'third_party/cutlass' 2024-05-19T18:11:22.9085532Z Entering 'third_party/eigen' 2024-05-19T18:11:22.9115181Z Entering 'third_party/fbgemm' 2024-05-19T18:11:22.9140915Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:11:22.9164891Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:11:22.9191432Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:11:22.9221194Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:11:22.9247217Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:11:22.9275116Z Entering 'third_party/flatbuffers' 2024-05-19T18:11:22.9305272Z Entering 'third_party/fmt' 2024-05-19T18:11:22.9332826Z Entering 'third_party/foxi' 2024-05-19T18:11:22.9360014Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:11:22.9388299Z Entering 'third_party/gloo' 2024-05-19T18:11:22.9416248Z Entering 'third_party/googletest' 2024-05-19T18:11:22.9443981Z Entering 'third_party/ideep' 2024-05-19T18:11:22.9471329Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T18:11:22.9502217Z Entering 'third_party/ios-cmake' 2024-05-19T18:11:22.9531260Z Entering 'third_party/ittapi' 2024-05-19T18:11:22.9556524Z Entering 'third_party/kineto' 2024-05-19T18:11:22.9583298Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:11:22.9609805Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:11:22.9635642Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:11:22.9663469Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:11:22.9688961Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:11:22.9715259Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:11:22.9742891Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:11:22.9769418Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:11:22.9797075Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:11:22.9824548Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:11:22.9853750Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:11:22.9881083Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:11:22.9909283Z Entering 'third_party/mimalloc' 2024-05-19T18:11:22.9936744Z Entering 'third_party/nccl/nccl' 2024-05-19T18:11:22.9961572Z Entering 'third_party/neon2sse' 2024-05-19T18:11:22.9991312Z Entering 'third_party/nlohmann' 2024-05-19T18:11:23.0018302Z Entering 'third_party/onnx' 2024-05-19T18:11:23.0058537Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T18:11:23.0087949Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T18:11:23.0117447Z Entering 'third_party/onnx-tensorrt' 2024-05-19T18:11:23.0142527Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:11:23.0171053Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:11:23.0196147Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:11:23.0223176Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:11:23.0251213Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T18:11:23.0279774Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:11:23.0307934Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:11:23.0333012Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:11:23.0359938Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:11:23.0388011Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:11:23.0413478Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:11:23.0439911Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:11:23.0463473Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:11:23.0490510Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:11:23.0518577Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:11:23.0559741Z Entering 'third_party/pocketfft' 2024-05-19T18:11:23.0587973Z Entering 'third_party/protobuf' 2024-05-19T18:11:23.0617336Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:11:23.0644320Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T18:11:23.0673385Z Entering 'third_party/psimd' 2024-05-19T18:11:23.0700126Z Entering 'third_party/pthreadpool' 2024-05-19T18:11:23.0728913Z Entering 'third_party/pybind11' 2024-05-19T18:11:23.0756785Z Entering 'third_party/python-peachpy' 2024-05-19T18:11:23.0782939Z Entering 'third_party/sleef' 2024-05-19T18:11:23.0809126Z Entering 'third_party/tbb' 2024-05-19T18:11:23.0837476Z Entering 'third_party/tensorpipe' 2024-05-19T18:11:23.0863389Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:11:23.0889002Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:11:23.0915287Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:11:23.0941444Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:11:23.0965822Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:11:23.1005707Z ##[endgroup] 2024-05-19T18:11:23.1007942Z ##[group]Persisting credentials for submodules 2024-05-19T18:11:23.1011214Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-05-19T18:11:23.1212525Z Entering 'android/libs/fbjni' 2024-05-19T18:11:23.1246197Z Entering 'third_party/FP16' 2024-05-19T18:11:23.1279722Z Entering 'third_party/FXdiv' 2024-05-19T18:11:23.1313491Z Entering 'third_party/NNPACK' 2024-05-19T18:11:23.1349137Z Entering 'third_party/QNNPACK' 2024-05-19T18:11:23.1383045Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T18:11:23.1416374Z Entering 'third_party/XNNPACK' 2024-05-19T18:11:23.1461725Z Entering 'third_party/benchmark' 2024-05-19T18:11:23.1494288Z Entering 'third_party/cpp-httplib' 2024-05-19T18:11:23.1529206Z Entering 'third_party/cpuinfo' 2024-05-19T18:11:23.1562819Z Entering 'third_party/cudnn_frontend' 2024-05-19T18:11:23.1596879Z Entering 'third_party/cutlass' 2024-05-19T18:11:23.1635781Z Entering 'third_party/eigen' 2024-05-19T18:11:23.1673407Z Entering 'third_party/fbgemm' 2024-05-19T18:11:23.1705313Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:11:23.1739041Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:11:23.1773698Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:11:23.1812315Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:11:23.1844847Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:11:23.1879681Z Entering 'third_party/flatbuffers' 2024-05-19T18:11:23.1916965Z Entering 'third_party/fmt' 2024-05-19T18:11:23.1949313Z Entering 'third_party/foxi' 2024-05-19T18:11:23.1984186Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:11:23.2017808Z Entering 'third_party/gloo' 2024-05-19T18:11:23.2052881Z Entering 'third_party/googletest' 2024-05-19T18:11:23.2086527Z Entering 'third_party/ideep' 2024-05-19T18:11:23.2119292Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T18:11:23.2156806Z Entering 'third_party/ios-cmake' 2024-05-19T18:11:23.2191926Z Entering 'third_party/ittapi' 2024-05-19T18:11:23.2223608Z Entering 'third_party/kineto' 2024-05-19T18:11:23.2254728Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:11:23.2288194Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:11:23.2322668Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:11:23.2355967Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:11:23.2390016Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:11:23.2422883Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:11:23.2456751Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:11:23.2490875Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:11:23.2523862Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:11:23.2557963Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:11:23.2592319Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:11:23.2626212Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:11:23.2659080Z Entering 'third_party/mimalloc' 2024-05-19T18:11:23.2693085Z Entering 'third_party/nccl/nccl' 2024-05-19T18:11:23.2727705Z Entering 'third_party/neon2sse' 2024-05-19T18:11:23.2762998Z Entering 'third_party/nlohmann' 2024-05-19T18:11:23.2797964Z Entering 'third_party/onnx' 2024-05-19T18:11:23.2844272Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T18:11:23.2879897Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T18:11:23.2914543Z Entering 'third_party/onnx-tensorrt' 2024-05-19T18:11:23.2946570Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:11:23.2982429Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:11:23.3014563Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:11:23.3044789Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:11:23.3082565Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T18:11:23.3115724Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:11:23.3149259Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:11:23.3181071Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:11:23.3211743Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:11:23.3245966Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:11:23.3278392Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:11:23.3313104Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:11:23.3344012Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:11:23.3378270Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:11:23.3410279Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:11:23.3458528Z Entering 'third_party/pocketfft' 2024-05-19T18:11:23.3494347Z Entering 'third_party/protobuf' 2024-05-19T18:11:23.3530686Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:11:23.3561697Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T18:11:23.3598438Z Entering 'third_party/psimd' 2024-05-19T18:11:23.3633784Z Entering 'third_party/pthreadpool' 2024-05-19T18:11:23.3668289Z Entering 'third_party/pybind11' 2024-05-19T18:11:23.3702945Z Entering 'third_party/python-peachpy' 2024-05-19T18:11:23.3735990Z Entering 'third_party/sleef' 2024-05-19T18:11:23.3769682Z Entering 'third_party/tbb' 2024-05-19T18:11:23.3803625Z Entering 'third_party/tensorpipe' 2024-05-19T18:11:23.3838075Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:11:23.3871811Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:11:23.3903076Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:11:23.3933568Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:11:23.3965346Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:11:23.4012442Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-05-19T18:11:23.4209047Z Entering 'android/libs/fbjni' 2024-05-19T18:11:23.4240460Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-05-19T18:11:23.4254012Z Entering 'third_party/FP16' 2024-05-19T18:11:23.4284853Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-05-19T18:11:23.4297860Z Entering 'third_party/FXdiv' 2024-05-19T18:11:23.4329507Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-05-19T18:11:23.4341744Z Entering 'third_party/NNPACK' 2024-05-19T18:11:23.4371739Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-05-19T18:11:23.4382478Z Entering 'third_party/QNNPACK' 2024-05-19T18:11:23.4411548Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2024-05-19T18:11:23.4423774Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T18:11:23.4454475Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-05-19T18:11:23.4467893Z Entering 'third_party/XNNPACK' 2024-05-19T18:11:23.4498057Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-05-19T18:11:23.4521314Z Entering 'third_party/benchmark' 2024-05-19T18:11:23.4552986Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-05-19T18:11:23.4566552Z Entering 'third_party/cpp-httplib' 2024-05-19T18:11:23.4597978Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-05-19T18:11:23.4609505Z Entering 'third_party/cpuinfo' 2024-05-19T18:11:23.4641511Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-05-19T18:11:23.4654290Z Entering 'third_party/cudnn_frontend' 2024-05-19T18:11:23.4684415Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-05-19T18:11:23.4698661Z Entering 'third_party/cutlass' 2024-05-19T18:11:23.4728166Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-05-19T18:11:23.4745376Z Entering 'third_party/eigen' 2024-05-19T18:11:23.4777270Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-05-19T18:11:23.4791154Z Entering 'third_party/fbgemm' 2024-05-19T18:11:23.4821917Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-05-19T18:11:23.4836077Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:11:23.4865707Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-05-19T18:11:23.4878313Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:11:23.4909928Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-05-19T18:11:23.4922380Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:11:23.4952611Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-05-19T18:11:23.4968898Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:11:23.5000083Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:11:23.5012065Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:11:23.5043464Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-05-19T18:11:23.5057539Z Entering 'third_party/flatbuffers' 2024-05-19T18:11:23.5088584Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-05-19T18:11:23.5103712Z Entering 'third_party/fmt' 2024-05-19T18:11:23.5133621Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-05-19T18:11:23.5147000Z Entering 'third_party/foxi' 2024-05-19T18:11:23.5178753Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-05-19T18:11:23.5189906Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:11:23.5221394Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-05-19T18:11:23.5232652Z Entering 'third_party/gloo' 2024-05-19T18:11:23.5262809Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-05-19T18:11:23.5275426Z Entering 'third_party/googletest' 2024-05-19T18:11:23.5306586Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:11:23.5319073Z Entering 'third_party/ideep' 2024-05-19T18:11:23.5350955Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-05-19T18:11:23.5363149Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T18:11:23.5394489Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-05-19T18:11:23.5412011Z Entering 'third_party/ios-cmake' 2024-05-19T18:11:23.5442965Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2024-05-19T18:11:23.5455196Z Entering 'third_party/ittapi' 2024-05-19T18:11:23.5485754Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-05-19T18:11:23.5498619Z Entering 'third_party/kineto' 2024-05-19T18:11:23.5528213Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-05-19T18:11:23.5540073Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:11:23.5569444Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-05-19T18:11:23.5581068Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:11:23.5613102Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-05-19T18:11:23.5625320Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:11:23.5654306Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-05-19T18:11:23.5665876Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:11:23.5696745Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-05-19T18:11:23.5708388Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:11:23.5738249Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-05-19T18:11:23.5749541Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:11:23.5780715Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-05-19T18:11:23.5794630Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:11:23.5826474Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-05-19T18:11:23.5838176Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:11:23.5869292Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:11:23.5881423Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:11:23.5913474Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-05-19T18:11:23.5925779Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:11:23.5956203Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-05-19T18:11:23.5970504Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:11:23.5999131Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-05-19T18:11:23.6010844Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:11:23.6042085Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-05-19T18:11:23.6055726Z Entering 'third_party/mimalloc' 2024-05-19T18:11:23.6087379Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-05-19T18:11:23.6100367Z Entering 'third_party/nccl/nccl' 2024-05-19T18:11:23.6129794Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-05-19T18:11:23.6141850Z Entering 'third_party/neon2sse' 2024-05-19T18:11:23.6172929Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2024-05-19T18:11:23.6185139Z Entering 'third_party/nlohmann' 2024-05-19T18:11:23.6214192Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-05-19T18:11:23.6226193Z Entering 'third_party/onnx' 2024-05-19T18:11:23.6256482Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-05-19T18:11:23.6280694Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T18:11:23.6311184Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-05-19T18:11:23.6323934Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T18:11:23.6355641Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-05-19T18:11:23.6369342Z Entering 'third_party/onnx-tensorrt' 2024-05-19T18:11:23.6401937Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2024-05-19T18:11:23.6415175Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:11:23.6445804Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2024-05-19T18:11:23.6462614Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:11:23.6493758Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-05-19T18:11:23.6506572Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:11:23.6537609Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-05-19T18:11:23.6549504Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:11:23.6580298Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-05-19T18:11:23.6596139Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T18:11:23.6627563Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-05-19T18:11:23.6641672Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:11:23.6673035Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-05-19T18:11:23.6685495Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:11:23.6715142Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:11:23.6727461Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:11:23.6758406Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-05-19T18:11:23.6770127Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:11:23.6800321Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-05-19T18:11:23.6813465Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:11:23.6841776Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-05-19T18:11:23.6852577Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:11:23.6883537Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-05-19T18:11:23.6896138Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:11:23.6927055Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-05-19T18:11:23.6939511Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:11:23.6971161Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-05-19T18:11:23.6984987Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:11:23.7016132Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-05-19T18:11:23.7029834Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:11:23.7062040Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-05-19T18:11:23.7088882Z Entering 'third_party/pocketfft' 2024-05-19T18:11:23.7120960Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-05-19T18:11:23.7132323Z Entering 'third_party/protobuf' 2024-05-19T18:11:23.7162820Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-05-19T18:11:23.7177192Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:11:23.7207460Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-05-19T18:11:23.7219071Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T18:11:23.7249634Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:11:23.7264005Z Entering 'third_party/psimd' 2024-05-19T18:11:23.7293961Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-05-19T18:11:23.7307248Z Entering 'third_party/pthreadpool' 2024-05-19T18:11:23.7337994Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-05-19T18:11:23.7350608Z Entering 'third_party/pybind11' 2024-05-19T18:11:23.7381605Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-05-19T18:11:23.7393249Z Entering 'third_party/python-peachpy' 2024-05-19T18:11:23.7423008Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-05-19T18:11:23.7436618Z Entering 'third_party/sleef' 2024-05-19T18:11:23.7466623Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-05-19T18:11:23.7480520Z Entering 'third_party/tbb' 2024-05-19T18:11:23.7512235Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2024-05-19T18:11:23.7526055Z Entering 'third_party/tensorpipe' 2024-05-19T18:11:23.7557061Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-05-19T18:11:23.7568908Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:11:23.7599207Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:11:23.7610933Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:11:23.7641641Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-05-19T18:11:23.7652894Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:11:23.7682789Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-05-19T18:11:23.7695055Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:11:23.7724563Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-05-19T18:11:23.7736711Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:11:23.7767201Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-05-19T18:11:23.8582550Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-05-19T18:11:23.8784680Z Entering 'android/libs/fbjni' 2024-05-19T18:11:23.8811180Z Entering 'third_party/FP16' 2024-05-19T18:11:23.8840575Z Entering 'third_party/FXdiv' 2024-05-19T18:11:23.8869451Z Entering 'third_party/NNPACK' 2024-05-19T18:11:23.8896813Z Entering 'third_party/QNNPACK' 2024-05-19T18:11:23.8922603Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T18:11:23.8948983Z Entering 'third_party/XNNPACK' 2024-05-19T18:11:23.8986849Z Entering 'third_party/benchmark' 2024-05-19T18:11:23.9015310Z Entering 'third_party/cpp-httplib' 2024-05-19T18:11:23.9043361Z Entering 'third_party/cpuinfo' 2024-05-19T18:11:23.9070936Z Entering 'third_party/cudnn_frontend' 2024-05-19T18:11:23.9098496Z Entering 'third_party/cutlass' 2024-05-19T18:11:23.9132779Z Entering 'third_party/eigen' 2024-05-19T18:11:23.9161409Z Entering 'third_party/fbgemm' 2024-05-19T18:11:23.9190156Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:11:23.9218186Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:11:23.9245522Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:11:23.9275152Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:11:23.9301475Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:11:23.9328012Z Entering 'third_party/flatbuffers' 2024-05-19T18:11:23.9357368Z Entering 'third_party/fmt' 2024-05-19T18:11:23.9386417Z Entering 'third_party/foxi' 2024-05-19T18:11:23.9413598Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:11:23.9440331Z Entering 'third_party/gloo' 2024-05-19T18:11:23.9469611Z Entering 'third_party/googletest' 2024-05-19T18:11:23.9497262Z Entering 'third_party/ideep' 2024-05-19T18:11:23.9524290Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T18:11:23.9557049Z Entering 'third_party/ios-cmake' 2024-05-19T18:11:23.9585132Z Entering 'third_party/ittapi' 2024-05-19T18:11:23.9611116Z Entering 'third_party/kineto' 2024-05-19T18:11:23.9640315Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:11:23.9667712Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:11:23.9695360Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:11:23.9722854Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:11:23.9751493Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:11:23.9776784Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:11:23.9804000Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:11:23.9832580Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:11:23.9858712Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:11:23.9885137Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:11:23.9912990Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:11:23.9940624Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:11:23.9969079Z Entering 'third_party/mimalloc' 2024-05-19T18:11:23.9996441Z Entering 'third_party/nccl/nccl' 2024-05-19T18:11:24.0024941Z Entering 'third_party/neon2sse' 2024-05-19T18:11:24.0054518Z Entering 'third_party/nlohmann' 2024-05-19T18:11:24.0081461Z Entering 'third_party/onnx' 2024-05-19T18:11:24.0122413Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T18:11:24.0148225Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T18:11:24.0174816Z Entering 'third_party/onnx-tensorrt' 2024-05-19T18:11:24.0202711Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:11:24.0232857Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:11:24.0260230Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:11:24.0287433Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:11:24.0319045Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T18:11:24.0350205Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:11:24.0375534Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:11:24.0403195Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:11:24.0431701Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:11:24.0458870Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:11:24.0483500Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:11:24.0510548Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:11:24.0535822Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:11:24.0562959Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:11:24.0591849Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:11:24.0632242Z Entering 'third_party/pocketfft' 2024-05-19T18:11:24.0661791Z Entering 'third_party/protobuf' 2024-05-19T18:11:24.0691358Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:11:24.0717727Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T18:11:24.0747260Z Entering 'third_party/psimd' 2024-05-19T18:11:24.0773411Z Entering 'third_party/pthreadpool' 2024-05-19T18:11:24.0799376Z Entering 'third_party/pybind11' 2024-05-19T18:11:24.0828206Z Entering 'third_party/python-peachpy' 2024-05-19T18:11:24.0854985Z Entering 'third_party/sleef' 2024-05-19T18:11:24.0881979Z Entering 'third_party/tbb' 2024-05-19T18:11:24.0911897Z Entering 'third_party/tensorpipe' 2024-05-19T18:11:24.0938550Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:11:24.0966327Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:11:24.0990998Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:11:24.1017057Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:11:24.1043941Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:11:24.1084226Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-05-19T18:11:24.1285757Z Entering 'android/libs/fbjni' 2024-05-19T18:11:24.1314457Z Entering 'third_party/FP16' 2024-05-19T18:11:24.1341641Z Entering 'third_party/FXdiv' 2024-05-19T18:11:24.1368831Z Entering 'third_party/NNPACK' 2024-05-19T18:11:24.1396867Z Entering 'third_party/QNNPACK' 2024-05-19T18:11:24.1422597Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T18:11:24.1449605Z Entering 'third_party/XNNPACK' 2024-05-19T18:11:24.1487673Z Entering 'third_party/benchmark' 2024-05-19T18:11:24.1513372Z Entering 'third_party/cpp-httplib' 2024-05-19T18:11:24.1539289Z Entering 'third_party/cpuinfo' 2024-05-19T18:11:24.1567503Z Entering 'third_party/cudnn_frontend' 2024-05-19T18:11:24.1595210Z Entering 'third_party/cutlass' 2024-05-19T18:11:24.1626922Z Entering 'third_party/eigen' 2024-05-19T18:11:24.1656734Z Entering 'third_party/fbgemm' 2024-05-19T18:11:24.1683404Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:11:24.1711270Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:11:24.1737804Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:11:24.1768664Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:11:24.1794808Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:11:24.1821198Z Entering 'third_party/flatbuffers' 2024-05-19T18:11:24.1850869Z Entering 'third_party/fmt' 2024-05-19T18:11:24.1877492Z Entering 'third_party/foxi' 2024-05-19T18:11:24.1903899Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:11:24.1929607Z Entering 'third_party/gloo' 2024-05-19T18:11:24.1956083Z Entering 'third_party/googletest' 2024-05-19T18:11:24.1981693Z Entering 'third_party/ideep' 2024-05-19T18:11:24.2007767Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T18:11:24.2039319Z Entering 'third_party/ios-cmake' 2024-05-19T18:11:24.2067965Z Entering 'third_party/ittapi' 2024-05-19T18:11:24.2093805Z Entering 'third_party/kineto' 2024-05-19T18:11:24.2118126Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:11:24.2146409Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:11:24.2172574Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:11:24.2197706Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:11:24.2225926Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:11:24.2251607Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:11:24.2279176Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:11:24.2305873Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:11:24.2333988Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:11:24.2360482Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:11:24.2389632Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:11:24.2417359Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:11:24.2446395Z Entering 'third_party/mimalloc' 2024-05-19T18:11:24.2475676Z Entering 'third_party/nccl/nccl' 2024-05-19T18:11:24.2505085Z Entering 'third_party/neon2sse' 2024-05-19T18:11:24.2533838Z Entering 'third_party/nlohmann' 2024-05-19T18:11:24.2560427Z Entering 'third_party/onnx' 2024-05-19T18:11:24.2598805Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T18:11:24.2627512Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T18:11:24.2655756Z Entering 'third_party/onnx-tensorrt' 2024-05-19T18:11:24.2683055Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:11:24.2714545Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:11:24.2741747Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:11:24.2770092Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:11:24.2801022Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T18:11:24.2830494Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:11:24.2858186Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:11:24.2883534Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:11:24.2909354Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:11:24.2936139Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:11:24.2963521Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:11:24.2990457Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:11:24.3015532Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:11:24.3041666Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:11:24.3070699Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:11:24.3111219Z Entering 'third_party/pocketfft' 2024-05-19T18:11:24.3139769Z Entering 'third_party/protobuf' 2024-05-19T18:11:24.3167707Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:11:24.3195172Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T18:11:24.3223362Z Entering 'third_party/psimd' 2024-05-19T18:11:24.3252308Z Entering 'third_party/pthreadpool' 2024-05-19T18:11:24.3278258Z Entering 'third_party/pybind11' 2024-05-19T18:11:24.3304198Z Entering 'third_party/python-peachpy' 2024-05-19T18:11:24.3331761Z Entering 'third_party/sleef' 2024-05-19T18:11:24.3359910Z Entering 'third_party/tbb' 2024-05-19T18:11:24.3390747Z Entering 'third_party/tensorpipe' 2024-05-19T18:11:24.3417129Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:11:24.3443150Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:11:24.3468745Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:11:24.3494494Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:11:24.3520640Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:11:24.3560101Z ##[endgroup] 2024-05-19T18:11:24.3586741Z [command]/usr/bin/git log -1 --format='%H' 2024-05-19T18:11:24.3603555Z 'c8c7ddfde57442cc28a394c385cab656ce9b1f5e' 2024-05-19T18:11:24.3772571Z Prepare all required actions 2024-05-19T18:11:24.3773023Z Getting action download info 2024-05-19T18:11:24.5084405Z ##[group]Run ./.github/actions/setup-linux 2024-05-19T18:11:24.5084997Z env: 2024-05-19T18:11:24.5085284Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:11:24.5085646Z ##[endgroup] 2024-05-19T18:11:24.5146743Z ##[group]Run set -euo pipefail 2024-05-19T18:11:24.5147179Z set -euo pipefail 2024-05-19T18:11:24.5147571Z function get_ec2_metadata() { 2024-05-19T18:11:24.5148119Z  # Pulled from instance metadata endpoint for EC2 2024-05-19T18:11:24.5149004Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-05-19T18:11:24.5149817Z  category=$1 2024-05-19T18:11:24.5150425Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-05-19T18:11:24.5151069Z  runner_name_str=i-038631eb9562efe08 2024-05-19T18:11:24.5151573Z  if [[ -f /.inarc ]]; then 2024-05-19T18:11:24.5152064Z  echo "ARC Runner, no info on ec2 metadata" 2024-05-19T18:11:24.5152708Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-05-19T18:11:24.5153514Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-05-19T18:11:24.5154113Z  else 2024-05-19T18:11:24.5154607Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-05-19T18:11:24.5155181Z  fi 2024-05-19T18:11:24.5155467Z } 2024-05-19T18:11:24.5155842Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-05-19T18:11:24.5156429Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-05-19T18:11:24.5157092Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-05-19T18:11:24.5157669Z echo "system info $(uname -a)" 2024-05-19T18:11:24.5165678Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:11:24.5166171Z env: 2024-05-19T18:11:24.5166474Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:11:24.5166832Z ##[endgroup] 2024-05-19T18:11:24.5231993Z ami-id: ami-0ce0c36d7a00b20e2 2024-05-19T18:11:24.5270407Z instance-id: i-038631eb9562efe08 2024-05-19T18:11:24.5308071Z instance-type: g5.4xlarge 2024-05-19T18:11:24.5313255Z system info Linux ip-10-0-28-22.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-05-19T18:11:24.5334236Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-05-19T18:11:24.5335151Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-05-19T18:11:24.5343113Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:11:24.5343600Z env: 2024-05-19T18:11:24.5343885Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:11:24.5344238Z ##[endgroup] 2024-05-19T18:11:24.5407323Z ##[group]Run if systemctl is-active --quiet docker; then 2024-05-19T18:11:24.5407930Z if systemctl is-active --quiet docker; then 2024-05-19T18:11:24.5408482Z  echo "Docker daemon is running..."; 2024-05-19T18:11:24.5408944Z else 2024-05-19T18:11:24.5409446Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-05-19T18:11:24.5410030Z fi 2024-05-19T18:11:24.5416863Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:11:24.5417373Z env: 2024-05-19T18:11:24.5417664Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:11:24.5418038Z ##[endgroup] 2024-05-19T18:11:24.5451698Z Docker daemon is running... 2024-05-19T18:11:24.5566641Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-05-19T18:11:24.5567201Z with: 2024-05-19T18:11:24.5567479Z shell: bash 2024-05-19T18:11:24.5567773Z timeout_minutes: 5 2024-05-19T18:11:24.5568103Z max_attempts: 3 2024-05-19T18:11:24.5568424Z retry_wait_seconds: 30 2024-05-19T18:11:24.5569882Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-05-19T18:11:24.5571307Z polling_interval_seconds: 1 2024-05-19T18:11:24.5571676Z warning_on_retry: true 2024-05-19T18:11:24.5572036Z continue_on_error: false 2024-05-19T18:11:24.5572376Z env: 2024-05-19T18:11:24.5572650Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:11:24.5573010Z AWS_RETRY_MODE: standard 2024-05-19T18:11:24.5573377Z AWS_MAX_ATTEMPTS: 5 2024-05-19T18:11:24.5573727Z AWS_DEFAULT_REGION: us-east-1 2024-05-19T18:11:24.5574103Z ##[endgroup] 2024-05-19T18:11:25.4076216Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-05-19T18:11:25.4077122Z Configure a credential helper to remove this warning. See 2024-05-19T18:11:25.4077961Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-05-19T18:11:25.4078478Z 2024-05-19T18:11:25.4078627Z Login Succeeded 2024-05-19T18:11:25.6054988Z Command completed after 1 attempt(s). 2024-05-19T18:11:25.6095973Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-05-19T18:11:25.6096812Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-05-19T18:11:25.6097471Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-05-19T18:11:25.6105467Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:11:25.6105966Z env: 2024-05-19T18:11:25.6106255Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:11:25.6106606Z ##[endgroup] 2024-05-19T18:11:25.6164980Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-05-19T18:11:25.6165767Z # ignore expansion of "docker ps -q" since it could be empty 2024-05-19T18:11:25.6166367Z # shellcheck disable=SC2046 2024-05-19T18:11:25.6166837Z docker stop $(docker ps -q) || true 2024-05-19T18:11:25.6167343Z # Prune all of the docker images 2024-05-19T18:11:25.6167802Z docker system prune -af 2024-05-19T18:11:25.6175540Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:11:25.6176063Z env: 2024-05-19T18:11:25.6176344Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:11:25.6176703Z ##[endgroup] 2024-05-19T18:11:25.6469379Z "docker stop" requires at least 1 argument. 2024-05-19T18:11:25.6470147Z See 'docker stop --help'. 2024-05-19T18:11:25.6470382Z 2024-05-19T18:11:25.6470605Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-05-19T18:11:25.6470980Z 2024-05-19T18:11:25.6471132Z Stop one or more running containers 2024-05-19T18:11:25.6696989Z Total reclaimed space: 0B 2024-05-19T18:11:25.6729324Z ##[group]Run set +e 2024-05-19T18:11:25.6729689Z set +e 2024-05-19T18:11:25.6730050Z set -x 2024-05-19T18:11:25.6730351Z  2024-05-19T18:11:25.6730719Z PT_DOMAIN=download.pytorch.org 2024-05-19T18:11:25.6731560Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-05-19T18:11:25.6732636Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-05-19T18:11:25.6733398Z # one is returned at random 2024-05-19T18:11:25.6733939Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-05-19T18:11:25.6734450Z  2024-05-19T18:11:25.6734788Z if [ -z "${RESOLVED_IP}" ]; then 2024-05-19T18:11:25.6735414Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-05-19T18:11:25.6736169Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-05-19T18:11:25.6736722Z  2024-05-19T18:11:25.6737066Z  if [ -z "${RESOLVED_IP}" ]; then 2024-05-19T18:11:25.6737633Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-05-19T18:11:25.6738148Z  exit 1 2024-05-19T18:11:25.6738466Z  fi 2024-05-19T18:11:25.6738770Z fi 2024-05-19T18:11:25.6739172Z  2024-05-19T18:11:25.6739544Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-05-19T18:11:25.6740058Z  # Clean up any old records first 2024-05-19T18:11:25.6740575Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-05-19T18:11:25.6741030Z fi 2024-05-19T18:11:25.6741305Z  2024-05-19T18:11:25.6741742Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-05-19T18:11:25.6742290Z cat /etc/hosts 2024-05-19T18:11:25.6749914Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:11:25.6750408Z env: 2024-05-19T18:11:25.6750691Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:11:25.6751041Z ##[endgroup] 2024-05-19T18:11:25.6771693Z + PT_DOMAIN=download.pytorch.org 2024-05-19T18:11:25.6775500Z ++ dig -4 +short download.pytorch.org 2024-05-19T18:11:25.6776110Z ++ tail -n1 2024-05-19T18:11:25.6897287Z + RESOLVED_IP=18.160.10.36 2024-05-19T18:11:25.6897764Z + '[' -z 18.160.10.36 ']' 2024-05-19T18:11:25.6898220Z + grep -r download.pytorch.org /etc/hosts 2024-05-19T18:11:25.6904172Z + echo '18.160.10.36 download.pytorch.org' 2024-05-19T18:11:25.6904890Z + sudo tee -a /etc/hosts 2024-05-19T18:11:25.6980800Z 18.160.10.36 download.pytorch.org 2024-05-19T18:11:25.6993336Z + cat /etc/hosts 2024-05-19T18:11:25.6997846Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-05-19T18:11:25.7010129Z ::1 localhost6 localhost6.localdomain6 2024-05-19T18:11:25.7010630Z 18.160.10.36 download.pytorch.org 2024-05-19T18:11:25.7157566Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-05-19T18:11:25.7158214Z with: 2024-05-19T18:11:25.7159293Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:11:25.7160494Z docker-build-dir: .ci/docker 2024-05-19T18:11:25.7160872Z working-directory: . 2024-05-19T18:11:25.7161338Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:11:25.7161857Z force-push: false 2024-05-19T18:11:25.7162156Z env: 2024-05-19T18:11:25.7162446Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:11:25.7162786Z ##[endgroup] 2024-05-19T18:11:25.7180498Z ##[group]Run set -ex 2024-05-19T18:11:25.7180870Z set -ex 2024-05-19T18:11:25.7181170Z  2024-05-19T18:11:25.7181736Z # If the docker build directory or the build script doesn't exist, the action will 2024-05-19T18:11:25.7182715Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-05-19T18:11:25.7183509Z # job could then download the pre-built image as usual 2024-05-19T18:11:25.7184255Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-05-19T18:11:25.7184926Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-05-19T18:11:25.7185568Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:11:25.7186137Z  2024-05-19T18:11:25.7186650Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-05-19T18:11:25.7187281Z  exit 0 2024-05-19T18:11:25.7187592Z else 2024-05-19T18:11:25.7187963Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-05-19T18:11:25.7188422Z fi 2024-05-19T18:11:25.7188710Z  2024-05-19T18:11:25.7189193Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-05-19T18:11:25.7190038Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-05-19T18:11:25.7190821Z  # use it as it is, but first let's extract the tag 2024-05-19T18:11:25.7191530Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-05-19T18:11:25.7192258Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:11:25.7193081Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:11:25.7193635Z else 2024-05-19T18:11:25.7194078Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-05-19T18:11:25.7194750Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:11:25.7195641Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:11:25.7196379Z fi 2024-05-19T18:11:25.7203870Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:11:25.7204368Z env: 2024-05-19T18:11:25.7204869Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:11:25.7205259Z REPO_NAME: pytorch 2024-05-19T18:11:25.7206376Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:11:25.7207577Z DOCKER_BUILD_DIR: .ci/docker 2024-05-19T18:11:25.7208080Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:11:25.7208582Z ##[endgroup] 2024-05-19T18:11:25.7227932Z + [[ ! -d .ci/docker ]] 2024-05-19T18:11:25.7228451Z + [[ ! -f .ci/docker/build.sh ]] 2024-05-19T18:11:25.7228829Z + echo skip=false 2024-05-19T18:11:25.7230669Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-05-19T18:11:25.7233272Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:11:25.7234449Z ++ awk -F '[:,]' '{print $2}' 2024-05-19T18:11:25.7239617Z + DOCKER_TAG=d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:11:25.7240516Z + echo docker-tag=d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:11:25.7241950Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:11:25.7267708Z ##[group]Run set +e 2024-05-19T18:11:25.7268087Z set +e 2024-05-19T18:11:25.7268399Z set -x 2024-05-19T18:11:25.7268690Z  2024-05-19T18:11:25.7268977Z login() { 2024-05-19T18:11:25.7269651Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-05-19T18:11:25.7270376Z } 2024-05-19T18:11:25.7270678Z  2024-05-19T18:11:25.7270991Z retry () { 2024-05-19T18:11:25.7271398Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-05-19T18:11:25.7271857Z } 2024-05-19T18:11:25.7272133Z  2024-05-19T18:11:25.7272544Z retry login "${DOCKER_REGISTRY}" 2024-05-19T18:11:25.7272970Z  2024-05-19T18:11:25.7273450Z # Check if image already exists, if it does then skip building it 2024-05-19T18:11:25.7274164Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-05-19T18:11:25.7274680Z  exit 0 2024-05-19T18:11:25.7274993Z fi 2024-05-19T18:11:25.7275275Z  2024-05-19T18:11:25.7275779Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-05-19T18:11:25.7276613Z # be empty. The default action would be to continue rebuild the image 2024-05-19T18:11:25.7277363Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-05-19T18:11:25.7278024Z  # if we're on the base branch then use the parent commit 2024-05-19T18:11:25.7278627Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-05-19T18:11:25.7279065Z else 2024-05-19T18:11:25.7279548Z  # otherwise we're on a PR, so use the most recent base commit 2024-05-19T18:11:25.7280233Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-05-19T18:11:25.7280757Z fi 2024-05-19T18:11:25.7281044Z  2024-05-19T18:11:25.7281382Z if [[ -z "${MERGE_BASE}" ]]; then 2024-05-19T18:11:25.7281891Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-05-19T18:11:25.7282362Z  2024-05-19T18:11:25.7283019Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-05-19T18:11:25.7283791Z  exit 0 2024-05-19T18:11:25.7284103Z fi 2024-05-19T18:11:25.7284399Z  2024-05-19T18:11:25.7285085Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-05-19T18:11:25.7286055Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-05-19T18:11:25.7286857Z  exit 1 2024-05-19T18:11:25.7287177Z fi 2024-05-19T18:11:25.7287465Z  2024-05-19T18:11:25.7287964Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-05-19T18:11:25.7288907Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-05-19T18:11:25.7289897Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-05-19T18:11:25.7290918Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-05-19T18:11:25.7292009Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-05-19T18:11:25.7292743Z fi 2024-05-19T18:11:25.7293030Z  2024-05-19T18:11:25.7293411Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-05-19T18:11:25.7300325Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:11:25.7300838Z env: 2024-05-19T18:11:25.7301119Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:11:25.7301500Z DOCKER_BUILD_DIR: .ci/docker 2024-05-19T18:11:25.7301995Z BASE_REVISION: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:11:25.7303277Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:11:25.7304517Z DOCKER_TAG: d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:11:25.7305143Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:11:25.7305686Z ##[endgroup] 2024-05-19T18:11:25.7325538Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:11:25.7340072Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:11:25.7340660Z + aws ecr get-login-password --region us-east-1 2024-05-19T18:11:25.7341403Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:11:26.1472847Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-05-19T18:11:26.1473690Z Configure a credential helper to remove this warning. See 2024-05-19T18:11:26.1474634Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-05-19T18:11:26.1475221Z 2024-05-19T18:11:26.1475347Z Login Succeeded 2024-05-19T18:11:26.1488097Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:11:26.4040800Z { 2024-05-19T18:11:26.4041191Z "schemaVersion": 2, 2024-05-19T18:11:26.4041794Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-05-19T18:11:26.4042582Z "config": { 2024-05-19T18:11:26.4043135Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-05-19T18:11:26.4043674Z "size": 45561, 2024-05-19T18:11:26.4044209Z "digest": "sha256:a0542ac5317cc182e1aac2cb8d63a2d6287450937722bab68464513bbbc603ef" 2024-05-19T18:11:26.4045224Z }, 2024-05-19T18:11:26.4045529Z "layers": [ 2024-05-19T18:11:26.4045807Z { 2024-05-19T18:11:26.4046247Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4046795Z "size": 28584317, 2024-05-19T18:11:26.4047342Z "digest": "sha256:63e9bbe323274e77e58d77c6ab6802d247458f784222fbb07a2556d6ec74ee05" 2024-05-19T18:11:26.4047962Z }, 2024-05-19T18:11:26.4048210Z { 2024-05-19T18:11:26.4048637Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4049182Z "size": 7944771, 2024-05-19T18:11:26.4049756Z "digest": "sha256:cfb3d849840ee60cee7b02bad68c1fc3c15928ebcd88f327754766b670578ed6" 2024-05-19T18:11:26.4050464Z }, 2024-05-19T18:11:26.4050720Z { 2024-05-19T18:11:26.4051164Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4051818Z "size": 57593718, 2024-05-19T18:11:26.4052484Z "digest": "sha256:968831e596a6288f0fed9b8612ee4ee8e75511037c4305058805492c5162e481" 2024-05-19T18:11:26.4053101Z }, 2024-05-19T18:11:26.4053427Z { 2024-05-19T18:11:26.4053860Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4054410Z "size": 187, 2024-05-19T18:11:26.4054975Z "digest": "sha256:ea310eb267ca1cab61b6b16f566cd28bfd59a741395a011f5e76716e15ba57c6" 2024-05-19T18:11:26.4055843Z }, 2024-05-19T18:11:26.4056130Z { 2024-05-19T18:11:26.4056610Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4057166Z "size": 6885, 2024-05-19T18:11:26.4057723Z "digest": "sha256:3af11d09e9cd1eb9c379f0a4071231e5a5642eb728b4b33bcb76be291f3c9488" 2024-05-19T18:11:26.4058342Z }, 2024-05-19T18:11:26.4058669Z { 2024-05-19T18:11:26.4059109Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4059672Z "size": 1361380219, 2024-05-19T18:11:26.4060344Z "digest": "sha256:ebfec18059b91e56882881ac34754f917861edb5f732c395d2a1a851bbd6db46" 2024-05-19T18:11:26.4061137Z }, 2024-05-19T18:11:26.4061479Z { 2024-05-19T18:11:26.4062048Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4062852Z "size": 62686, 2024-05-19T18:11:26.4063473Z "digest": "sha256:533b4aebf16914c763b7b0de3ce657590c6f979045e9fdf1f816adaf68d8a4d3" 2024-05-19T18:11:26.4064088Z }, 2024-05-19T18:11:26.4064348Z { 2024-05-19T18:11:26.4064773Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4065320Z "size": 1685, 2024-05-19T18:11:26.4065852Z "digest": "sha256:9dd75d06a0910f28cb1e484b8808724e5a6ee570ecb8fc04631368f546b39ed9" 2024-05-19T18:11:26.4066457Z }, 2024-05-19T18:11:26.4066703Z { 2024-05-19T18:11:26.4067138Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4067682Z "size": 1523, 2024-05-19T18:11:26.4068217Z "digest": "sha256:30bfca4dd3492d60ed8035b0eeb1229897041140db117f5663465a551e25851d" 2024-05-19T18:11:26.4068813Z }, 2024-05-19T18:11:26.4069058Z { 2024-05-19T18:11:26.4069488Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4070026Z "size": 2608186849, 2024-05-19T18:11:26.4070595Z "digest": "sha256:1b57ce94cad9a5097d99b8d6f7bcd51df5a162fc3c5f5686b689b76993724bc8" 2024-05-19T18:11:26.4071212Z }, 2024-05-19T18:11:26.4071461Z { 2024-05-19T18:11:26.4071885Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4072552Z "size": 86631, 2024-05-19T18:11:26.4073101Z "digest": "sha256:9ee6bdb31195dd42fd98147a75d540efe0c5708e0ecda866a0bca060084fddab" 2024-05-19T18:11:26.4073709Z }, 2024-05-19T18:11:26.4073955Z { 2024-05-19T18:11:26.4074397Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4074941Z "size": 1822, 2024-05-19T18:11:26.4075494Z "digest": "sha256:8af83a173aa33e1330fdde4efc5b96334ef85736ec0725bfeedc72c227dcfcc5" 2024-05-19T18:11:26.4076103Z }, 2024-05-19T18:11:26.4076359Z { 2024-05-19T18:11:26.4076787Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4077335Z "size": 245735746, 2024-05-19T18:11:26.4077899Z "digest": "sha256:f8281676bd20d66f882b7edad2f27bd52787f63ff5c68768681df27d74d9d07c" 2024-05-19T18:11:26.4078518Z }, 2024-05-19T18:11:26.4078780Z { 2024-05-19T18:11:26.4079159Z + exit 0 2024-05-19T18:11:26.4079615Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4080167Z "size": 544, 2024-05-19T18:11:26.4080719Z "digest": "sha256:748e3cfdd8d6f501456917dcc4a9e8b11092aa4e784a46b00c3a7a13274e2742" 2024-05-19T18:11:26.4081331Z }, 2024-05-19T18:11:26.4081593Z { 2024-05-19T18:11:26.4082035Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4082588Z "size": 1279, 2024-05-19T18:11:26.4083122Z "digest": "sha256:5fa8d379f04e0947dc4af19169168984b6dfe62a40efbd97ee9bc60015cb2970" 2024-05-19T18:11:26.4083746Z }, 2024-05-19T18:11:26.4084009Z { 2024-05-19T18:11:26.4084434Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4085222Z "size": 484, 2024-05-19T18:11:26.4085766Z "digest": "sha256:a95b9f41fb5bbac1fd00c55419da0b54b0c1b38cd46e22b6e95293bd639615b5" 2024-05-19T18:11:26.4086392Z }, 2024-05-19T18:11:26.4086639Z { 2024-05-19T18:11:26.4087068Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4087728Z "size": 91705725, 2024-05-19T18:11:26.4088284Z "digest": "sha256:8b6d0b819d19a961789f05d3c03c73f6ee67059e9464c8bbc0a1124113a32395" 2024-05-19T18:11:26.4088892Z }, 2024-05-19T18:11:26.4089148Z { 2024-05-19T18:11:26.4089584Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4090205Z "size": 3236, 2024-05-19T18:11:26.4090787Z "digest": "sha256:c61996e733996ff4a9cf89f2a69716a6add90b5c8b1902feb91d8e0dfd0f7bc4" 2024-05-19T18:11:26.4091399Z }, 2024-05-19T18:11:26.4091660Z { 2024-05-19T18:11:26.4092088Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4092641Z "size": 1905, 2024-05-19T18:11:26.4093180Z "digest": "sha256:9e69dcd7661c1bee67677240651319ec72ee977219e8097f93be00c6555c9ab5" 2024-05-19T18:11:26.4093790Z }, 2024-05-19T18:11:26.4094037Z { 2024-05-19T18:11:26.4094479Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4095043Z "size": 699, 2024-05-19T18:11:26.4095596Z "digest": "sha256:cfcaf2eb82498ef031f7d481e635b1a37af01b9e389f30555b603d75e2c00602" 2024-05-19T18:11:26.4096205Z }, 2024-05-19T18:11:26.4096477Z { 2024-05-19T18:11:26.4096915Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4097466Z "size": 2811659560, 2024-05-19T18:11:26.4098060Z "digest": "sha256:7cb10a6ae8df3c778e529febd7eef0333c2af45e2a2ed45f45967bb7b4219131" 2024-05-19T18:11:26.4098687Z }, 2024-05-19T18:11:26.4098954Z { 2024-05-19T18:11:26.4099384Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4099942Z "size": 378, 2024-05-19T18:11:26.4100494Z "digest": "sha256:07fa87a43fb8c069109d9c97743dcc016ad605a221c1df4c7d79863d0d0a2229" 2024-05-19T18:11:26.4101111Z }, 2024-05-19T18:11:26.4101358Z { 2024-05-19T18:11:26.4101796Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4102350Z "size": 11887, 2024-05-19T18:11:26.4102909Z "digest": "sha256:f16e94b048018b4677a272dcb1cb347e9c90be713acffa5bdd2d63a972224ca9" 2024-05-19T18:11:26.4103530Z }, 2024-05-19T18:11:26.4103793Z { 2024-05-19T18:11:26.4104232Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4104778Z "size": 803, 2024-05-19T18:11:26.4105331Z "digest": "sha256:c857065196d6f476be67a20b1e26f3ef6e71ae427e4feee2412e7f65cc208493" 2024-05-19T18:11:26.4105951Z }, 2024-05-19T18:11:26.4106210Z { 2024-05-19T18:11:26.4106649Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4107207Z "size": 106, 2024-05-19T18:11:26.4107758Z "digest": "sha256:9d5c38cad0c9a12d6b9b3070caf13ae1003594cf17c75170ccadf0f8719f93ab" 2024-05-19T18:11:26.4108385Z }, 2024-05-19T18:11:26.4108633Z { 2024-05-19T18:11:26.4109077Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4109633Z "size": 503, 2024-05-19T18:11:26.4110171Z "digest": "sha256:925ac0c7ff935448f88445123dc223bb4531147eec4d0e0c34f9f05bd4e14f46" 2024-05-19T18:11:26.4110800Z }, 2024-05-19T18:11:26.4111071Z { 2024-05-19T18:11:26.4111520Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4112074Z "size": 121477245, 2024-05-19T18:11:26.4112741Z "digest": "sha256:cfbf8bf4873dc1eef0b1a35dd290348be3ceb2dafccdc4820e9460ddfd8f7c54" 2024-05-19T18:11:26.4113375Z }, 2024-05-19T18:11:26.4113631Z { 2024-05-19T18:11:26.4114067Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4114618Z "size": 109, 2024-05-19T18:11:26.4115178Z "digest": "sha256:01914a189c0b5a993f1fc3ddb461abe3964eba3ae55fab9eff5a53f77de19178" 2024-05-19T18:11:26.4115822Z }, 2024-05-19T18:11:26.4116080Z { 2024-05-19T18:11:26.4116533Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4117087Z "size": 488, 2024-05-19T18:11:26.4117642Z "digest": "sha256:3c2451fbe84c6b4678a24659b7abe1ea3cca59da338f16b7563d6875accf6eee" 2024-05-19T18:11:26.4118322Z }, 2024-05-19T18:11:26.4118594Z { 2024-05-19T18:11:26.4119044Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4119614Z "size": 298, 2024-05-19T18:11:26.4120233Z "digest": "sha256:aea8a2a73b72b6c647fafb9f4b1b3d65cb7c28172d3b7a58de3bdb6335addb00" 2024-05-19T18:11:26.4120938Z }, 2024-05-19T18:11:26.4121204Z { 2024-05-19T18:11:26.4121636Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4122191Z "size": 103, 2024-05-19T18:11:26.4122749Z "digest": "sha256:c26f348bfda1e1a6e32f77a9054fa8d26cd59cf6fe8cc3ea1ff9d1816dc74d22" 2024-05-19T18:11:26.4123369Z }, 2024-05-19T18:11:26.4123628Z { 2024-05-19T18:11:26.4124068Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4124785Z "size": 1471, 2024-05-19T18:11:26.4125329Z "digest": "sha256:347c41648f51ed569232ea48943ef3e1424197b90bfae5b89898f2273f64e76b" 2024-05-19T18:11:26.4125942Z }, 2024-05-19T18:11:26.4126199Z { 2024-05-19T18:11:26.4126638Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4127190Z "size": 444189070, 2024-05-19T18:11:26.4127750Z "digest": "sha256:00953247955f346ee0a69eb263efa2135a9fc3e9680322e5c39cf79e4c06fab5" 2024-05-19T18:11:26.4128359Z }, 2024-05-19T18:11:26.4128614Z { 2024-05-19T18:11:26.4129053Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4129607Z "size": 162, 2024-05-19T18:11:26.4130193Z "digest": "sha256:df27b4a5ce7ca0be34e1295afceb581f6ac42c1c0888e253c481bbce25efde72" 2024-05-19T18:11:26.4130834Z }, 2024-05-19T18:11:26.4131096Z { 2024-05-19T18:11:26.4131535Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4132095Z "size": 564, 2024-05-19T18:11:26.4132643Z "digest": "sha256:acb7fcd74bf2071a5d257e338f2229b9a9747f6e96de138091ed80cfa845709f" 2024-05-19T18:11:26.4133269Z }, 2024-05-19T18:11:26.4133542Z { 2024-05-19T18:11:26.4133982Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4134534Z "size": 35867331, 2024-05-19T18:11:26.4135114Z "digest": "sha256:5bd6ac51ef5d3109aba547ec49aec04535079b866d45939fa098e33b7f20d94f" 2024-05-19T18:11:26.4135740Z }, 2024-05-19T18:11:26.4135998Z { 2024-05-19T18:11:26.4136449Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4137005Z "size": 104, 2024-05-19T18:11:26.4137565Z "digest": "sha256:1795bce633b0c48054bada65aede57ae82761e922b39b354aebedd2d3fc82a01" 2024-05-19T18:11:26.4138185Z }, 2024-05-19T18:11:26.4138450Z { 2024-05-19T18:11:26.4138905Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4139472Z "size": 425, 2024-05-19T18:11:26.4140019Z "digest": "sha256:e41de6016f3c0f9309d3727db3cf335f2789260369b9c02485b5a1eda2a23949" 2024-05-19T18:11:26.4140639Z }, 2024-05-19T18:11:26.4140914Z { 2024-05-19T18:11:26.4141349Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4141921Z "size": 20262012, 2024-05-19T18:11:26.4142494Z "digest": "sha256:458f5f2af8cfff41b832b1ffa0ac89536cc75afaa2b007348475812be84a3168" 2024-05-19T18:11:26.4143128Z }, 2024-05-19T18:11:26.4143384Z { 2024-05-19T18:11:26.4143840Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4144399Z "size": 437, 2024-05-19T18:11:26.4144957Z "digest": "sha256:ef99a2a9cffb05645e67c1ff7eff9d0400a37d6277a734e0534e544e6d70abc3" 2024-05-19T18:11:26.4145577Z }, 2024-05-19T18:11:26.4145849Z { 2024-05-19T18:11:26.4146295Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4146854Z "size": 699, 2024-05-19T18:11:26.4147392Z "digest": "sha256:cfcaf2eb82498ef031f7d481e635b1a37af01b9e389f30555b603d75e2c00602" 2024-05-19T18:11:26.4148024Z }, 2024-05-19T18:11:26.4148287Z { 2024-05-19T18:11:26.4148719Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4149373Z "size": 142, 2024-05-19T18:11:26.4149917Z "digest": "sha256:1ba8d08d58765b6585b8a6430120bdf33e85a118dbe004432a9515481c728a6f" 2024-05-19T18:11:26.4150585Z }, 2024-05-19T18:11:26.4150839Z { 2024-05-19T18:11:26.4151290Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4151961Z "size": 136, 2024-05-19T18:11:26.4152565Z "digest": "sha256:0c67ae0247cf4129424322fdc4c7d64de5f02712a6851839df767491d40ac668" 2024-05-19T18:11:26.4153161Z }, 2024-05-19T18:11:26.4153423Z { 2024-05-19T18:11:26.4153867Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4154423Z "size": 4981810470, 2024-05-19T18:11:26.4154996Z "digest": "sha256:3ebf7de5956e7374b72ff809508272a8ac6ff23b2293581bef5847cd0af1b4bb" 2024-05-19T18:11:26.4155620Z }, 2024-05-19T18:11:26.4155881Z { 2024-05-19T18:11:26.4156305Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4156870Z "size": 190, 2024-05-19T18:11:26.4157425Z "digest": "sha256:a6b88127ee12fade12d3202bc71d30768c0ddc0eb46ad572bd5089b6aa1bb0e2" 2024-05-19T18:11:26.4158049Z }, 2024-05-19T18:11:26.4158299Z { 2024-05-19T18:11:26.4158743Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4159303Z "size": 565, 2024-05-19T18:11:26.4159854Z "digest": "sha256:7b0cf0b8be50e7fae7398549f658c86b1e9c348102f84ef2c230bc6af5599739" 2024-05-19T18:11:26.4160502Z }, 2024-05-19T18:11:26.4160770Z { 2024-05-19T18:11:26.4161218Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4161758Z "size": 43162755, 2024-05-19T18:11:26.4162325Z "digest": "sha256:044061f29c13bd178d3113fb9ccfc367da97f96f6bf948dfcae1ff5b1e17be2f" 2024-05-19T18:11:26.4162950Z }, 2024-05-19T18:11:26.4163215Z { 2024-05-19T18:11:26.4163649Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4164207Z "size": 106, 2024-05-19T18:11:26.4164901Z "digest": "sha256:f21869a483d43c856cf8639bdb4270d8295023741b5cddebed62bcfe3b8caa7b" 2024-05-19T18:11:26.4165516Z }, 2024-05-19T18:11:26.4165768Z { 2024-05-19T18:11:26.4166212Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4166781Z "size": 1212, 2024-05-19T18:11:26.4167356Z "digest": "sha256:a155d9e6e7e7fd2243fba6789a98b96aa2db9d5d6462b50d9c11cfc7f3ac34e7" 2024-05-19T18:11:26.4167981Z }, 2024-05-19T18:11:26.4168248Z { 2024-05-19T18:11:26.4168691Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4169281Z "size": 699, 2024-05-19T18:11:26.4169845Z "digest": "sha256:cfcaf2eb82498ef031f7d481e635b1a37af01b9e389f30555b603d75e2c00602" 2024-05-19T18:11:26.4170467Z }, 2024-05-19T18:11:26.4170737Z { 2024-05-19T18:11:26.4171166Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4171729Z "size": 137, 2024-05-19T18:11:26.4172286Z "digest": "sha256:6028f518cba8d1b51ca287446cfc0a9db237fbeea2198738ee2c114a24a8c34a" 2024-05-19T18:11:26.4172913Z }, 2024-05-19T18:11:26.4173174Z { 2024-05-19T18:11:26.4173621Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4174171Z "size": 119, 2024-05-19T18:11:26.4174722Z "digest": "sha256:9f0e16ae766e08c8bf1276b07cd6781a3f9ca54131846275ffcfc9473a6312dd" 2024-05-19T18:11:26.4175338Z }, 2024-05-19T18:11:26.4175604Z { 2024-05-19T18:11:26.4176045Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4176600Z "size": 1853517530, 2024-05-19T18:11:26.4177162Z "digest": "sha256:26d45b11322ae5593c42d7929f046ca6d236f7ec5e8a13d8930360c013d32747" 2024-05-19T18:11:26.4177775Z }, 2024-05-19T18:11:26.4178036Z { 2024-05-19T18:11:26.4178463Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4179013Z "size": 172, 2024-05-19T18:11:26.4179549Z "digest": "sha256:a78a7e4425658766ba6e70bb1367f10c352dc4a7bfb16432be3d9a3496368333" 2024-05-19T18:11:26.4180270Z }, 2024-05-19T18:11:26.4180522Z { 2024-05-19T18:11:26.4180959Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4181536Z "size": 1838, 2024-05-19T18:11:26.4182168Z "digest": "sha256:df585564bf4cc28aec4957fbec751620ae4ca50edbbdcec1c5eacc306a16e2e1" 2024-05-19T18:11:26.4182901Z }, 2024-05-19T18:11:26.4183202Z { 2024-05-19T18:11:26.4183638Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4184195Z "size": 7529792, 2024-05-19T18:11:26.4184747Z "digest": "sha256:f125619c47120e1a542ddb0e822b5ffc07f9d165a0977f8c4064fdf67865e3fd" 2024-05-19T18:11:26.4185356Z }, 2024-05-19T18:11:26.4185625Z { 2024-05-19T18:11:26.4186070Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4186617Z "size": 162, 2024-05-19T18:11:26.4187166Z "digest": "sha256:4cc671ff8870320a4e5bab71f83b755b9b96c777dba43fdbfc1a0e3d001d8936" 2024-05-19T18:11:26.4187793Z }, 2024-05-19T18:11:26.4188050Z { 2024-05-19T18:11:26.4188493Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4189034Z "size": 7939, 2024-05-19T18:11:26.4189569Z "digest": "sha256:e1d3209e04ae7e31773f49e41f7cee2949fa017569383786876f6d1ba0d7d8ac" 2024-05-19T18:11:26.4190175Z }, 2024-05-19T18:11:26.4190435Z { 2024-05-19T18:11:26.4190871Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4191424Z "size": 8072, 2024-05-19T18:11:26.4191964Z "digest": "sha256:322e57f70842c48212f87b2e4c1cc094c25470b90fec1d30ad715f4d907c1f79" 2024-05-19T18:11:26.4192624Z }, 2024-05-19T18:11:26.4192882Z { 2024-05-19T18:11:26.4193319Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4193875Z "size": 301, 2024-05-19T18:11:26.4194408Z "digest": "sha256:d58f09a95d3e18b4d653b7767e371c1e9ef320625fa4cc6a92cbd1771d7ef768" 2024-05-19T18:11:26.4195035Z }, 2024-05-19T18:11:26.4195300Z { 2024-05-19T18:11:26.4195744Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4196293Z "size": 7617584, 2024-05-19T18:11:26.4196864Z "digest": "sha256:f611e28aa59ab55b92e54aac3cc04729565ccae9d0edb75cc45a534d752a9de1" 2024-05-19T18:11:26.4197488Z }, 2024-05-19T18:11:26.4197741Z { 2024-05-19T18:11:26.4198189Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4198746Z "size": 108, 2024-05-19T18:11:26.4199297Z "digest": "sha256:ff67e6bd8e2302312131632eae7033ce84ce85503282bfd8b31c008ff533990a" 2024-05-19T18:11:26.4199904Z }, 2024-05-19T18:11:26.4200203Z { 2024-05-19T18:11:26.4200672Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4201235Z "size": 54145477, 2024-05-19T18:11:26.4201767Z "digest": "sha256:d2b06b7f79e8770329516b40d5189f7c2b2d38404132638103a594685fe1d569" 2024-05-19T18:11:26.4202369Z }, 2024-05-19T18:11:26.4202640Z { 2024-05-19T18:11:26.4203077Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4203644Z "size": 507, 2024-05-19T18:11:26.4204194Z "digest": "sha256:c27bc13ce9d41097b89fa4c7f53feb417379940ba35474fede1b3322540e85e9" 2024-05-19T18:11:26.4204982Z }, 2024-05-19T18:11:26.4205239Z { 2024-05-19T18:11:26.4205693Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4206256Z "size": 1443748723, 2024-05-19T18:11:26.4206838Z "digest": "sha256:8680ed64cc192868f4e34d4dba9121684e3bba13ac365c8e8539386fe4a5147c" 2024-05-19T18:11:26.4207445Z }, 2024-05-19T18:11:26.4207708Z { 2024-05-19T18:11:26.4208149Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4208716Z "size": 106, 2024-05-19T18:11:26.4209246Z "digest": "sha256:400e9c337687d211e1a78744e24985fbaca792e988f3caf5861040d80b088662" 2024-05-19T18:11:26.4209867Z }, 2024-05-19T18:11:26.4210125Z { 2024-05-19T18:11:26.4210551Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4211192Z "size": 558, 2024-05-19T18:11:26.4211744Z "digest": "sha256:914d4657be1b6936372f45195af96a959eab5bced80ca9deeafc3d6d9db6c03c" 2024-05-19T18:11:26.4212365Z }, 2024-05-19T18:11:26.4212618Z { 2024-05-19T18:11:26.4213062Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4213677Z "size": 46248255, 2024-05-19T18:11:26.4214230Z "digest": "sha256:ecc1d0285d28b6339b9648e2195ce576f30288563ac5de0882653a42bcc0eb5b" 2024-05-19T18:11:26.4214828Z }, 2024-05-19T18:11:26.4215089Z { 2024-05-19T18:11:26.4215530Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:11:26.4216074Z "size": 111, 2024-05-19T18:11:26.4216615Z "digest": "sha256:b882dcc190622361a09ea00b970e0fae5f81ea1f0431a45afa329fd3c97f66d9" 2024-05-19T18:11:26.4217236Z } 2024-05-19T18:11:26.4217497Z ] 2024-05-19T18:11:26.4217746Z } 2024-05-19T18:11:26.4328328Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-05-19T18:11:26.4328822Z tag=${ECR_DOCKER_IMAGE##*/} 2024-05-19T18:11:26.4329369Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-05-19T18:11:26.4337248Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:11:26.4337748Z env: 2024-05-19T18:11:26.4338029Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:11:26.4339180Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:11:26.4340394Z ##[endgroup] 2024-05-19T18:11:26.4362164Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks-d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:11:26.4409886Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-05-19T18:11:26.4410507Z with: 2024-05-19T18:11:26.4411565Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:11:26.4412881Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:11:26.4413393Z env: 2024-05-19T18:11:26.4413680Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:11:26.4414037Z ##[endgroup] 2024-05-19T18:11:26.4429916Z ##[group]Run set -x 2024-05-19T18:11:26.4430265Z set -x 2024-05-19T18:11:26.4430578Z set +e 2024-05-19T18:11:26.4430878Z  2024-05-19T18:11:26.4431158Z login() { 2024-05-19T18:11:26.4431821Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-05-19T18:11:26.4432613Z } 2024-05-19T18:11:26.4432907Z  2024-05-19T18:11:26.4433220Z retry () { 2024-05-19T18:11:26.4433604Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-05-19T18:11:26.4434053Z } 2024-05-19T18:11:26.4434335Z  2024-05-19T18:11:26.4434650Z retry login "${DOCKER_REGISTRY}" 2024-05-19T18:11:26.4435073Z  2024-05-19T18:11:26.4435349Z set -e 2024-05-19T18:11:26.4435827Z # ignore output since only exit code is used for conditional 2024-05-19T18:11:26.4436517Z # only pull docker image if it's not available locally 2024-05-19T18:11:26.4437279Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-05-19T18:11:26.4437976Z  retry docker pull "${DOCKER_IMAGE}" 2024-05-19T18:11:26.4438421Z fi 2024-05-19T18:11:26.4445343Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:11:26.4445842Z env: 2024-05-19T18:11:26.4446127Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:11:26.4447268Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:11:26.4448555Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:11:26.4449164Z ##[endgroup] 2024-05-19T18:11:26.4471620Z + set +e 2024-05-19T18:11:26.4472202Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:11:26.4472921Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:11:26.4473857Z + aws ecr get-login-password --region us-east-1 2024-05-19T18:11:26.4474631Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:11:26.8583163Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-05-19T18:11:26.8583972Z Configure a credential helper to remove this warning. See 2024-05-19T18:11:26.8584794Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-05-19T18:11:26.8585318Z 2024-05-19T18:11:26.8586877Z Login Succeeded 2024-05-19T18:11:26.8598665Z + set -e 2024-05-19T18:11:26.8600019Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:11:26.8747625Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:11:26.8749700Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:11:27.0774423Z d55d6a6f08250cd2609d528116ab6fd6e787bee0: Pulling from pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks 2024-05-19T18:11:27.0775889Z 63e9bbe32327: Pulling fs layer 2024-05-19T18:11:27.0776466Z cfb3d849840e: Pulling fs layer 2024-05-19T18:11:27.0777307Z 968831e596a6: Pulling fs layer 2024-05-19T18:11:27.0777897Z ea310eb267ca: Pulling fs layer 2024-05-19T18:11:27.0778411Z 3af11d09e9cd: Pulling fs layer 2024-05-19T18:11:27.0778927Z ebfec18059b9: Pulling fs layer 2024-05-19T18:11:27.0779479Z 533b4aebf169: Pulling fs layer 2024-05-19T18:11:27.0780019Z 9dd75d06a091: Pulling fs layer 2024-05-19T18:11:27.0780603Z 30bfca4dd349: Pulling fs layer 2024-05-19T18:11:27.0781158Z 1b57ce94cad9: Pulling fs layer 2024-05-19T18:11:27.0781685Z 9ee6bdb31195: Pulling fs layer 2024-05-19T18:11:27.0782192Z 8af83a173aa3: Pulling fs layer 2024-05-19T18:11:27.0782705Z f8281676bd20: Pulling fs layer 2024-05-19T18:11:27.0783202Z 748e3cfdd8d6: Pulling fs layer 2024-05-19T18:11:27.0783758Z ebfec18059b9: Waiting 2024-05-19T18:11:27.0784169Z 5fa8d379f04e: Pulling fs layer 2024-05-19T18:11:27.0784540Z 533b4aebf169: Waiting 2024-05-19T18:11:27.0784875Z a95b9f41fb5b: Pulling fs layer 2024-05-19T18:11:27.0785248Z 8b6d0b819d19: Pulling fs layer 2024-05-19T18:11:27.0785722Z c61996e73399: Pulling fs layer 2024-05-19T18:11:27.0786093Z 9e69dcd7661c: Pulling fs layer 2024-05-19T18:11:27.0786467Z cfcaf2eb8249: Pulling fs layer 2024-05-19T18:11:27.0786901Z 7cb10a6ae8df: Pulling fs layer 2024-05-19T18:11:27.0787271Z 07fa87a43fb8: Pulling fs layer 2024-05-19T18:11:27.0787647Z f16e94b04801: Pulling fs layer 2024-05-19T18:11:27.0788016Z c857065196d6: Pulling fs layer 2024-05-19T18:11:27.0788372Z ea310eb267ca: Waiting 2024-05-19T18:11:27.0788693Z 9dd75d06a091: Waiting 2024-05-19T18:11:27.0804301Z a95b9f41fb5b: Waiting 2024-05-19T18:11:27.0804933Z 30bfca4dd349: Waiting 2024-05-19T18:11:27.0805287Z 9d5c38cad0c9: Pulling fs layer 2024-05-19T18:11:27.0805652Z 1b57ce94cad9: Waiting 2024-05-19T18:11:27.0805982Z 925ac0c7ff93: Pulling fs layer 2024-05-19T18:11:27.0806353Z 8b6d0b819d19: Waiting 2024-05-19T18:11:27.0806668Z 9ee6bdb31195: Waiting 2024-05-19T18:11:27.0807006Z cfbf8bf4873d: Pulling fs layer 2024-05-19T18:11:27.0807390Z 01914a189c0b: Pulling fs layer 2024-05-19T18:11:27.0807785Z 3c2451fbe84c: Pulling fs layer 2024-05-19T18:11:27.0808166Z aea8a2a73b72: Pulling fs layer 2024-05-19T18:11:27.0808529Z f8281676bd20: Waiting 2024-05-19T18:11:27.0808850Z 3af11d09e9cd: Waiting 2024-05-19T18:11:27.0809166Z 748e3cfdd8d6: Waiting 2024-05-19T18:11:27.0809489Z 5fa8d379f04e: Waiting 2024-05-19T18:11:27.0809971Z 8af83a173aa3: Waiting 2024-05-19T18:11:27.0810334Z c26f348bfda1: Pulling fs layer 2024-05-19T18:11:27.0810726Z 9e69dcd7661c: Waiting 2024-05-19T18:11:27.0811042Z f16e94b04801: Waiting 2024-05-19T18:11:27.0811363Z 9d5c38cad0c9: Waiting 2024-05-19T18:11:27.0811696Z 347c41648f51: Pulling fs layer 2024-05-19T18:11:27.0812081Z 00953247955f: Pulling fs layer 2024-05-19T18:11:27.0812458Z df27b4a5ce7c: Pulling fs layer 2024-05-19T18:11:27.0812850Z acb7fcd74bf2: Pulling fs layer 2024-05-19T18:11:27.0813232Z 5bd6ac51ef5d: Pulling fs layer 2024-05-19T18:11:27.0813626Z 1795bce633b0: Pulling fs layer 2024-05-19T18:11:27.0813993Z 07fa87a43fb8: Waiting 2024-05-19T18:11:27.0814318Z 7cb10a6ae8df: Waiting 2024-05-19T18:11:27.0814663Z e41de6016f3c: Pulling fs layer 2024-05-19T18:11:27.0815027Z 01914a189c0b: Waiting 2024-05-19T18:11:27.0815352Z cfcaf2eb8249: Waiting 2024-05-19T18:11:27.0815686Z 458f5f2af8cf: Pulling fs layer 2024-05-19T18:11:27.0816054Z c26f348bfda1: Waiting 2024-05-19T18:11:27.0816392Z 00953247955f: Waiting 2024-05-19T18:11:27.0816706Z 347c41648f51: Waiting 2024-05-19T18:11:27.0817017Z cfbf8bf4873d: Waiting 2024-05-19T18:11:27.0817345Z 5bd6ac51ef5d: Waiting 2024-05-19T18:11:27.0817667Z acb7fcd74bf2: Waiting 2024-05-19T18:11:27.0817994Z 458f5f2af8cf: Waiting 2024-05-19T18:11:27.0818299Z e41de6016f3c: Waiting 2024-05-19T18:11:27.0818611Z c61996e73399: Waiting 2024-05-19T18:11:27.0818948Z ef99a2a9cffb: Pulling fs layer 2024-05-19T18:11:27.0819318Z 1ba8d08d5876: Pulling fs layer 2024-05-19T18:11:27.0819693Z 0c67ae0247cf: Pulling fs layer 2024-05-19T18:11:27.0820070Z 3ebf7de5956e: Pulling fs layer 2024-05-19T18:11:27.0820445Z a6b88127ee12: Pulling fs layer 2024-05-19T18:11:27.0820897Z 7b0cf0b8be50: Pulling fs layer 2024-05-19T18:11:27.0821362Z 1ba8d08d5876: Waiting 2024-05-19T18:11:27.0821709Z 044061f29c13: Pulling fs layer 2024-05-19T18:11:27.0822074Z a6b88127ee12: Waiting 2024-05-19T18:11:27.0822392Z 3ebf7de5956e: Waiting 2024-05-19T18:11:27.0822733Z f21869a483d4: Pulling fs layer 2024-05-19T18:11:27.0823121Z a155d9e6e7e7: Pulling fs layer 2024-05-19T18:11:27.0823470Z 044061f29c13: Waiting 2024-05-19T18:11:27.0823788Z a155d9e6e7e7: Waiting 2024-05-19T18:11:27.0824117Z 6028f518cba8: Pulling fs layer 2024-05-19T18:11:27.0824500Z 9f0e16ae766e: Pulling fs layer 2024-05-19T18:11:27.0824874Z 26d45b11322a: Pulling fs layer 2024-05-19T18:11:27.0825236Z 6028f518cba8: Waiting 2024-05-19T18:11:27.0825573Z a78a7e442565: Pulling fs layer 2024-05-19T18:11:27.0825940Z df585564bf4c: Pulling fs layer 2024-05-19T18:11:27.0826310Z 26d45b11322a: Waiting 2024-05-19T18:11:27.0826642Z f125619c4712: Pulling fs layer 2024-05-19T18:11:27.0826999Z a78a7e442565: Waiting 2024-05-19T18:11:27.0827308Z df585564bf4c: Waiting 2024-05-19T18:11:27.0827638Z 9f0e16ae766e: Waiting 2024-05-19T18:11:27.0827973Z 4cc671ff8870: Pulling fs layer 2024-05-19T18:11:27.0828350Z e1d3209e04ae: Pulling fs layer 2024-05-19T18:11:27.0828726Z 322e57f70842: Pulling fs layer 2024-05-19T18:11:27.0829093Z e1d3209e04ae: Waiting 2024-05-19T18:11:27.0829440Z d58f09a95d3e: Pulling fs layer 2024-05-19T18:11:27.0829810Z f611e28aa59a: Pulling fs layer 2024-05-19T18:11:27.0830188Z ff67e6bd8e23: Pulling fs layer 2024-05-19T18:11:27.0830565Z d58f09a95d3e: Waiting 2024-05-19T18:11:27.0830887Z f611e28aa59a: Waiting 2024-05-19T18:11:27.0831196Z 322e57f70842: Waiting 2024-05-19T18:11:27.0831525Z d2b06b7f79e8: Pulling fs layer 2024-05-19T18:11:27.0831883Z ff67e6bd8e23: Waiting 2024-05-19T18:11:27.0832210Z c27bc13ce9d4: Pulling fs layer 2024-05-19T18:11:27.0832676Z 8680ed64cc19: Pulling fs layer 2024-05-19T18:11:27.0833039Z c27bc13ce9d4: Waiting 2024-05-19T18:11:27.0833362Z 400e9c337687: Pulling fs layer 2024-05-19T18:11:27.0833713Z 8680ed64cc19: Waiting 2024-05-19T18:11:27.0834054Z 914d4657be1b: Pulling fs layer 2024-05-19T18:11:27.0834431Z ecc1d0285d28: Pulling fs layer 2024-05-19T18:11:27.0834807Z b882dcc19062: Pulling fs layer 2024-05-19T18:11:27.0835157Z b882dcc19062: Waiting 2024-05-19T18:11:27.0835469Z 400e9c337687: Waiting 2024-05-19T18:11:27.0835856Z 914d4657be1b: Waiting 2024-05-19T18:11:27.2091754Z cfb3d849840e: Verifying Checksum 2024-05-19T18:11:27.2092397Z cfb3d849840e: Download complete 2024-05-19T18:11:27.2907153Z ea310eb267ca: Download complete 2024-05-19T18:11:27.3878394Z 3af11d09e9cd: Verifying Checksum 2024-05-19T18:11:27.3878978Z 3af11d09e9cd: Download complete 2024-05-19T18:11:27.4868040Z 63e9bbe32327: Verifying Checksum 2024-05-19T18:11:27.4868511Z 63e9bbe32327: Download complete 2024-05-19T18:11:27.5575984Z 533b4aebf169: Download complete 2024-05-19T18:11:27.7399281Z 968831e596a6: Download complete 2024-05-19T18:11:27.7632272Z 30bfca4dd349: Verifying Checksum 2024-05-19T18:11:27.7632884Z 30bfca4dd349: Download complete 2024-05-19T18:11:27.8757977Z 9ee6bdb31195: Verifying Checksum 2024-05-19T18:11:27.8758559Z 9ee6bdb31195: Download complete 2024-05-19T18:11:27.9574285Z 8af83a173aa3: Download complete 2024-05-19T18:11:28.1405531Z 63e9bbe32327: Pull complete 2024-05-19T18:11:28.3942140Z cfb3d849840e: Pull complete 2024-05-19T18:11:29.2896507Z 968831e596a6: Pull complete 2024-05-19T18:11:29.3887331Z ea310eb267ca: Pull complete 2024-05-19T18:11:29.4921266Z 3af11d09e9cd: Pull complete 2024-05-19T18:11:30.5981411Z f8281676bd20: Download complete 2024-05-19T18:11:30.6759723Z 748e3cfdd8d6: Verifying Checksum 2024-05-19T18:11:30.6760277Z 748e3cfdd8d6: Download complete 2024-05-19T18:11:30.7819945Z 5fa8d379f04e: Verifying Checksum 2024-05-19T18:11:30.7820533Z 5fa8d379f04e: Download complete 2024-05-19T18:11:30.8636338Z a95b9f41fb5b: Download complete 2024-05-19T18:11:31.9344061Z 8b6d0b819d19: Verifying Checksum 2024-05-19T18:11:31.9344536Z 8b6d0b819d19: Download complete 2024-05-19T18:11:32.0199395Z c61996e73399: Verifying Checksum 2024-05-19T18:11:32.0200167Z c61996e73399: Download complete 2024-05-19T18:11:32.1001901Z 9e69dcd7661c: Verifying Checksum 2024-05-19T18:11:32.1002388Z 9e69dcd7661c: Download complete 2024-05-19T18:11:32.1957810Z cfcaf2eb8249: Verifying Checksum 2024-05-19T18:11:32.1958250Z cfcaf2eb8249: Download complete 2024-05-19T18:11:41.0814052Z ebfec18059b9: Verifying Checksum 2024-05-19T18:11:41.0814713Z ebfec18059b9: Download complete 2024-05-19T18:11:41.1846837Z 07fa87a43fb8: Download complete 2024-05-19T18:11:41.2709575Z f16e94b04801: Download complete 2024-05-19T18:11:41.3779443Z c857065196d6: Verifying Checksum 2024-05-19T18:11:41.3780100Z c857065196d6: Download complete 2024-05-19T18:11:41.4929765Z 9d5c38cad0c9: Verifying Checksum 2024-05-19T18:11:41.4930409Z 9d5c38cad0c9: Download complete 2024-05-19T18:11:41.5703627Z 925ac0c7ff93: Download complete 2024-05-19T18:11:43.0849402Z cfbf8bf4873d: Verifying Checksum 2024-05-19T18:11:43.0850075Z cfbf8bf4873d: Download complete 2024-05-19T18:11:43.1704929Z 01914a189c0b: Download complete 2024-05-19T18:11:43.2669985Z 3c2451fbe84c: Verifying Checksum 2024-05-19T18:11:43.2670532Z 3c2451fbe84c: Download complete 2024-05-19T18:11:43.3427056Z aea8a2a73b72: Download complete 2024-05-19T18:11:43.4268770Z c26f348bfda1: Download complete 2024-05-19T18:11:43.5176730Z 347c41648f51: Download complete 2024-05-19T18:11:48.0201605Z 00953247955f: Verifying Checksum 2024-05-19T18:11:48.0202084Z 00953247955f: Download complete 2024-05-19T18:11:48.1179018Z df27b4a5ce7c: Download complete 2024-05-19T18:11:48.2267959Z acb7fcd74bf2: Download complete 2024-05-19T18:11:48.6458519Z 5bd6ac51ef5d: Verifying Checksum 2024-05-19T18:11:48.6459151Z 5bd6ac51ef5d: Download complete 2024-05-19T18:11:48.7402733Z 1795bce633b0: Verifying Checksum 2024-05-19T18:11:48.7403423Z 1795bce633b0: Download complete 2024-05-19T18:11:48.8276843Z e41de6016f3c: Verifying Checksum 2024-05-19T18:11:48.8277446Z e41de6016f3c: Download complete 2024-05-19T18:11:49.0951381Z 458f5f2af8cf: Verifying Checksum 2024-05-19T18:11:49.0951919Z 458f5f2af8cf: Download complete 2024-05-19T18:11:49.1793863Z ef99a2a9cffb: Verifying Checksum 2024-05-19T18:11:49.1794383Z ef99a2a9cffb: Download complete 2024-05-19T18:11:49.2596336Z 1ba8d08d5876: Verifying Checksum 2024-05-19T18:11:49.2597647Z 1ba8d08d5876: Download complete 2024-05-19T18:11:49.3598430Z 0c67ae0247cf: Download complete 2024-05-19T18:11:53.3020367Z ebfec18059b9: Pull complete 2024-05-19T18:11:53.5341100Z 533b4aebf169: Pull complete 2024-05-19T18:11:53.8062927Z 9dd75d06a091: Pull complete 2024-05-19T18:11:53.8907930Z 1b57ce94cad9: Verifying Checksum 2024-05-19T18:11:53.8908383Z 1b57ce94cad9: Download complete 2024-05-19T18:11:53.9721612Z a6b88127ee12: Verifying Checksum 2024-05-19T18:11:53.9722088Z a6b88127ee12: Download complete 2024-05-19T18:11:54.0445726Z 30bfca4dd349: Pull complete 2024-05-19T18:11:54.0751430Z 7b0cf0b8be50: Verifying Checksum 2024-05-19T18:11:54.0751840Z 7b0cf0b8be50: Download complete 2024-05-19T18:11:54.5816898Z 044061f29c13: Verifying Checksum 2024-05-19T18:11:54.5817356Z 044061f29c13: Download complete 2024-05-19T18:11:54.6834073Z f21869a483d4: Verifying Checksum 2024-05-19T18:11:54.6834588Z f21869a483d4: Download complete 2024-05-19T18:11:54.7652628Z a155d9e6e7e7: Verifying Checksum 2024-05-19T18:11:54.7653183Z a155d9e6e7e7: Download complete 2024-05-19T18:11:54.8797681Z 6028f518cba8: Verifying Checksum 2024-05-19T18:11:54.8798199Z 6028f518cba8: Download complete 2024-05-19T18:11:55.0229865Z 9f0e16ae766e: Verifying Checksum 2024-05-19T18:11:55.0230390Z 9f0e16ae766e: Download complete 2024-05-19T18:12:00.6133178Z 7cb10a6ae8df: Verifying Checksum 2024-05-19T18:12:00.6133714Z 7cb10a6ae8df: Download complete 2024-05-19T18:12:00.6997695Z a78a7e442565: Verifying Checksum 2024-05-19T18:12:00.6998292Z a78a7e442565: Download complete 2024-05-19T18:12:00.8105986Z df585564bf4c: Download complete 2024-05-19T18:12:00.9709192Z f125619c4712: Verifying Checksum 2024-05-19T18:12:00.9709850Z f125619c4712: Download complete 2024-05-19T18:12:01.0671883Z 4cc671ff8870: Download complete 2024-05-19T18:12:01.1591000Z e1d3209e04ae: Download complete 2024-05-19T18:12:01.2468328Z 322e57f70842: Verifying Checksum 2024-05-19T18:12:01.2468998Z 322e57f70842: Download complete 2024-05-19T18:12:01.3491650Z d58f09a95d3e: Download complete 2024-05-19T18:12:01.4907513Z f611e28aa59a: Verifying Checksum 2024-05-19T18:12:01.4908099Z f611e28aa59a: Download complete 2024-05-19T18:12:01.6107792Z ff67e6bd8e23: Verifying Checksum 2024-05-19T18:12:01.6108664Z ff67e6bd8e23: Download complete 2024-05-19T18:12:02.2457442Z d2b06b7f79e8: Verifying Checksum 2024-05-19T18:12:02.2457926Z d2b06b7f79e8: Download complete 2024-05-19T18:12:02.3827013Z c27bc13ce9d4: Verifying Checksum 2024-05-19T18:12:02.3827489Z c27bc13ce9d4: Download complete 2024-05-19T18:12:27.8425503Z 26d45b11322a: Verifying Checksum 2024-05-19T18:12:27.8426077Z 26d45b11322a: Download complete 2024-05-19T18:12:27.9259450Z 400e9c337687: Download complete 2024-05-19T18:12:28.0294877Z 914d4657be1b: Verifying Checksum 2024-05-19T18:12:28.0295496Z 914d4657be1b: Download complete 2024-05-19T18:12:29.2852866Z ecc1d0285d28: Verifying Checksum 2024-05-19T18:12:29.2853518Z ecc1d0285d28: Download complete 2024-05-19T18:12:29.3767464Z b882dcc19062: Download complete 2024-05-19T18:12:34.7497826Z 8680ed64cc19: Verifying Checksum 2024-05-19T18:12:56.5926539Z 3ebf7de5956e: Download complete 2024-05-19T18:13:12.5214263Z 1b57ce94cad9: Pull complete 2024-05-19T18:13:12.7453029Z 9ee6bdb31195: Pull complete 2024-05-19T18:13:12.9646446Z 8af83a173aa3: Pull complete 2024-05-19T18:13:18.0740353Z f8281676bd20: Pull complete 2024-05-19T18:13:18.3081372Z 748e3cfdd8d6: Pull complete 2024-05-19T18:13:18.5305000Z 5fa8d379f04e: Pull complete 2024-05-19T18:13:18.7434326Z a95b9f41fb5b: Pull complete 2024-05-19T18:13:20.5455905Z 8b6d0b819d19: Pull complete 2024-05-19T18:13:20.7681579Z c61996e73399: Pull complete 2024-05-19T18:13:20.9840372Z 9e69dcd7661c: Pull complete 2024-05-19T18:13:21.2003308Z cfcaf2eb8249: Pull complete 2024-05-19T18:13:58.2457539Z 7cb10a6ae8df: Pull complete 2024-05-19T18:13:58.4675817Z 07fa87a43fb8: Pull complete 2024-05-19T18:13:58.7119771Z f16e94b04801: Pull complete 2024-05-19T18:13:58.8943832Z c857065196d6: Pull complete 2024-05-19T18:13:59.0657085Z 9d5c38cad0c9: Pull complete 2024-05-19T18:13:59.2216544Z 925ac0c7ff93: Pull complete 2024-05-19T18:14:02.3194282Z cfbf8bf4873d: Pull complete 2024-05-19T18:14:02.4817501Z 01914a189c0b: Pull complete 2024-05-19T18:14:02.6043221Z 3c2451fbe84c: Pull complete 2024-05-19T18:14:02.7569367Z aea8a2a73b72: Pull complete 2024-05-19T18:14:02.9417530Z c26f348bfda1: Pull complete 2024-05-19T18:14:03.1188163Z 347c41648f51: Pull complete 2024-05-19T18:14:08.8791619Z 00953247955f: Pull complete 2024-05-19T18:14:09.0944025Z df27b4a5ce7c: Pull complete 2024-05-19T18:14:09.3254895Z acb7fcd74bf2: Pull complete 2024-05-19T18:14:10.0600310Z 5bd6ac51ef5d: Pull complete 2024-05-19T18:14:10.2042435Z 1795bce633b0: Pull complete 2024-05-19T18:14:10.4224824Z e41de6016f3c: Pull complete 2024-05-19T18:14:10.8487372Z 458f5f2af8cf: Pull complete 2024-05-19T18:14:10.9800788Z ef99a2a9cffb: Pull complete 2024-05-19T18:14:11.4615472Z 1ba8d08d5876: Pull complete 2024-05-19T18:14:11.6878256Z 0c67ae0247cf: Pull complete 2024-05-19T18:14:48.0224272Z 3ebf7de5956e: Pull complete 2024-05-19T18:14:48.2465832Z a6b88127ee12: Pull complete 2024-05-19T18:14:48.4638243Z 7b0cf0b8be50: Pull complete 2024-05-19T18:14:49.7941080Z 044061f29c13: Pull complete 2024-05-19T18:14:50.0185957Z f21869a483d4: Pull complete 2024-05-19T18:14:50.2427513Z a155d9e6e7e7: Pull complete 2024-05-19T18:14:50.6899128Z 6028f518cba8: Pull complete 2024-05-19T18:14:50.9186581Z 9f0e16ae766e: Pull complete 2024-05-19T18:15:20.2533785Z 26d45b11322a: Pull complete 2024-05-19T18:15:20.4569275Z a78a7e442565: Pull complete 2024-05-19T18:15:20.6890627Z df585564bf4c: Pull complete 2024-05-19T18:15:21.0609864Z f125619c4712: Pull complete 2024-05-19T18:15:21.2931524Z 4cc671ff8870: Pull complete 2024-05-19T18:15:21.5312051Z e1d3209e04ae: Pull complete 2024-05-19T18:15:21.7553398Z 322e57f70842: Pull complete 2024-05-19T18:15:21.9895738Z d58f09a95d3e: Pull complete 2024-05-19T18:15:22.6187481Z f611e28aa59a: Pull complete 2024-05-19T18:15:22.8371330Z ff67e6bd8e23: Pull complete 2024-05-19T18:15:24.5551139Z d2b06b7f79e8: Pull complete 2024-05-19T18:15:24.7913646Z c27bc13ce9d4: Pull complete 2024-05-19T18:15:38.8260567Z 8680ed64cc19: Pull complete 2024-05-19T18:15:39.0522655Z 400e9c337687: Pull complete 2024-05-19T18:15:39.2833577Z 914d4657be1b: Pull complete 2024-05-19T18:15:40.1177191Z ecc1d0285d28: Pull complete 2024-05-19T18:15:40.3559262Z b882dcc19062: Pull complete 2024-05-19T18:15:40.4629758Z Digest: sha256:66f062925f91f9b9204b1dc6d68caf8c89fa82fd4b12cb5f24c19aa4b1c87506 2024-05-19T18:15:40.5069958Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:15:40.5273500Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:15:40.5314836Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-05-19T18:15:40.5315742Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-05-19T18:15:40.5323660Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:15:40.5324160Z env: 2024-05-19T18:15:40.5324436Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:15:40.5325064Z ##[endgroup] 2024-05-19T18:15:40.5476364Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-05-19T18:15:40.5476917Z with: 2024-05-19T18:15:40.5477205Z driver-version: 550.54.15 2024-05-19T18:15:40.5477548Z env: 2024-05-19T18:15:40.5477821Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:15:40.5478166Z ##[endgroup] 2024-05-19T18:15:40.5602926Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-05-19T18:15:40.5603479Z with: 2024-05-19T18:15:40.5603762Z timeout_minutes: 10 2024-05-19T18:15:40.5604091Z max_attempts: 3 2024-05-19T18:15:40.5633864Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-05-19T18:15:40.5664059Z retry_wait_seconds: 10 2024-05-19T18:15:40.5664428Z polling_interval_seconds: 1 2024-05-19T18:15:40.5664802Z warning_on_retry: true 2024-05-19T18:15:40.5665160Z continue_on_error: false 2024-05-19T18:15:40.5665507Z env: 2024-05-19T18:15:40.5665786Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:15:40.5666136Z DRIVER_VERSION: 550.54.15 2024-05-19T18:15:40.5666495Z ##[endgroup] 2024-05-19T18:15:40.6199115Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-05-19T18:15:40.6199933Z + pre_install_nvidia_driver_amzn2 2024-05-19T18:15:40.6200453Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-05-19T18:15:40.9217889Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:15:40.9604086Z No Match for argument: nvidia-driver-latest-dkms 2024-05-19T18:15:40.9836664Z No Packages marked for removal 2024-05-19T18:15:40.9957869Z + install_nvidia_driver_common 2024-05-19T18:15:40.9960155Z + echo 'Before installing NVIDIA driver' 2024-05-19T18:15:40.9960739Z + lspci 2024-05-19T18:15:40.9961122Z Before installing NVIDIA driver 2024-05-19T18:15:41.0944029Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-05-19T18:15:41.0944979Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-05-19T18:15:41.0945893Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-05-19T18:15:41.0946894Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-05-19T18:15:41.0947794Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-05-19T18:15:41.0948536Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-05-19T18:15:41.0949255Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-05-19T18:15:41.0950041Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-05-19T18:15:41.0950618Z + lsmod 2024-05-19T18:15:41.0956757Z Module Size Used by 2024-05-19T18:15:41.0957197Z backlight 16384 0 2024-05-19T18:15:41.0957717Z xt_conntrack 16384 1 2024-05-19T18:15:41.0958232Z ipt_MASQUERADE 16384 1 2024-05-19T18:15:41.0958792Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-05-19T18:15:41.0959440Z nf_conntrack_netlink 49152 0 2024-05-19T18:15:41.0960043Z nfnetlink 16384 2 nf_conntrack_netlink 2024-05-19T18:15:41.0960690Z xfrm_user 45056 1 2024-05-19T18:15:41.0961252Z xfrm_algo 16384 1 xfrm_user 2024-05-19T18:15:41.0962107Z iptable_nat 16384 1 2024-05-19T18:15:41.0962663Z nf_conntrack_ipv4 16384 3 2024-05-19T18:15:41.0963320Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-05-19T18:15:41.0964010Z nf_nat_ipv4 16384 1 iptable_nat 2024-05-19T18:15:41.0965047Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-05-19T18:15:41.0966399Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-05-19T18:15:41.0967327Z xt_addrtype 16384 2 2024-05-19T18:15:41.0967712Z iptable_filter 16384 1 2024-05-19T18:15:41.0968097Z br_netfilter 24576 0 2024-05-19T18:15:41.0968589Z bridge 172032 1 br_netfilter 2024-05-19T18:15:41.0969235Z stp 16384 1 bridge 2024-05-19T18:15:41.0970002Z llc 16384 2 bridge,stp 2024-05-19T18:15:41.0970476Z overlay 86016 0 2024-05-19T18:15:41.0970910Z sunrpc 393216 1 2024-05-19T18:15:41.0971283Z dm_mirror 28672 0 2024-05-19T18:15:41.0971675Z dm_region_hash 20480 1 dm_mirror 2024-05-19T18:15:41.0972164Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-05-19T18:15:41.0972674Z dm_mod 143360 2 dm_log,dm_mirror 2024-05-19T18:15:41.0973119Z dax 69632 1 dm_mod 2024-05-19T18:15:41.0973512Z crc32_pclmul 16384 0 2024-05-19T18:15:41.0973891Z ghash_clmulni_intel 16384 0 2024-05-19T18:15:41.0974274Z pcbc 16384 0 2024-05-19T18:15:41.0974639Z aesni_intel 188416 0 2024-05-19T18:15:41.0975036Z aes_x86_64 20480 1 aesni_intel 2024-05-19T18:15:41.0975484Z crypto_simd 16384 1 aesni_intel 2024-05-19T18:15:41.0975916Z mousedev 24576 0 2024-05-19T18:15:41.0976316Z glue_helper 16384 1 aesni_intel 2024-05-19T18:15:41.0976875Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-05-19T18:15:41.0977414Z evdev 20480 3 2024-05-19T18:15:41.0977787Z psmouse 32768 0 2024-05-19T18:15:41.0978155Z button 16384 0 2024-05-19T18:15:41.0978523Z ena 139264 0 2024-05-19T18:15:41.0978895Z ptp 20480 1 ena 2024-05-19T18:15:41.0979282Z pps_core 20480 1 ptp 2024-05-19T18:15:41.0979778Z crc32c_intel 24576 0 2024-05-19T18:15:41.0980150Z autofs4 49152 2 2024-05-19T18:15:41.0980516Z + modinfo nvidia 2024-05-19T18:15:41.0981220Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-05-19T18:15:41.0981909Z firmware: nvidia/535.54.03/gsp_tu10x.bin 2024-05-19T18:15:41.0982397Z firmware: nvidia/535.54.03/gsp_ga10x.bin 2024-05-19T18:15:41.0982888Z alias: char-major-195-* 2024-05-19T18:15:41.0983273Z version: 535.54.03 2024-05-19T18:15:41.0983624Z supported: external 2024-05-19T18:15:41.0983967Z license: NVIDIA 2024-05-19T18:15:41.0984329Z srcversion: EA9C7EF32617E104C8240C4 2024-05-19T18:15:41.0984793Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-05-19T18:15:41.0985295Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-05-19T18:15:41.0985779Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-05-19T18:15:41.0986248Z depends: i2c-core,drm 2024-05-19T18:15:41.0986610Z retpoline: Y 2024-05-19T18:15:41.0986924Z name: nvidia 2024-05-19T18:15:41.0987470Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-05-19T18:15:41.0988113Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-05-19T18:15:41.0988761Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-05-19T18:15:41.0989351Z parm: NVreg_ResmanDebugLevel:int 2024-05-19T18:15:41.0989798Z parm: NVreg_RmLogonRC:int 2024-05-19T18:15:41.0990223Z parm: NVreg_ModifyDeviceFiles:int 2024-05-19T18:15:41.0990682Z parm: NVreg_DeviceFileUID:int 2024-05-19T18:15:41.0991223Z parm: NVreg_DeviceFileGID:int 2024-05-19T18:15:41.0991665Z parm: NVreg_DeviceFileMode:int 2024-05-19T18:15:41.0992189Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-05-19T18:15:41.0992755Z parm: NVreg_UsePageAttributeTable:int 2024-05-19T18:15:41.0993382Z parm: NVreg_EnablePCIeGen3:int 2024-05-19T18:15:41.0993813Z parm: NVreg_EnableMSI:int 2024-05-19T18:15:41.0994222Z parm: NVreg_TCEBypassMode:int 2024-05-19T18:15:41.0994682Z parm: NVreg_EnableStreamMemOPs:int 2024-05-19T18:15:41.0995211Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-05-19T18:15:41.0995797Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-05-19T18:15:41.0996347Z parm: NVreg_EnableS0ixPowerManagement:int 2024-05-19T18:15:41.0997008Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-05-19T18:15:41.0997611Z parm: NVreg_DynamicPowerManagement:int 2024-05-19T18:15:41.0998222Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-05-19T18:15:41.0998816Z parm: NVreg_EnableGpuFirmware:int 2024-05-19T18:15:41.0999307Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-05-19T18:15:41.0999840Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-05-19T18:15:41.1000387Z parm: NVreg_EnableUserNUMAManagement:int 2024-05-19T18:15:41.1000883Z parm: NVreg_MemoryPoolSize:int 2024-05-19T18:15:41.1001351Z parm: NVreg_KMallocHeapMaxSize:int 2024-05-19T18:15:41.1001836Z parm: NVreg_VMallocHeapMaxSize:int 2024-05-19T18:15:41.1002303Z parm: NVreg_IgnoreMMIOCheck:int 2024-05-19T18:15:41.1002761Z parm: NVreg_NvLinkDisable:int 2024-05-19T18:15:41.1003267Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-05-19T18:15:41.1003797Z parm: NVreg_RegisterPCIDriver:int 2024-05-19T18:15:41.1004268Z parm: NVreg_EnableResizableBar:int 2024-05-19T18:15:41.1004994Z parm: NVreg_EnableDbgBreakpoint:int 2024-05-19T18:15:41.1005481Z parm: NVreg_RegistryDwords:charp 2024-05-19T18:15:41.1005979Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-05-19T18:15:41.1006452Z parm: NVreg_RmMsg:charp 2024-05-19T18:15:41.1006870Z parm: NVreg_GpuBlacklist:charp 2024-05-19T18:15:41.1007343Z parm: NVreg_TemporaryFilePath:charp 2024-05-19T18:15:41.1007813Z parm: NVreg_ExcludedGpus:charp 2024-05-19T18:15:41.1008262Z parm: NVreg_DmaRemapPeerMmio:int 2024-05-19T18:15:41.1008740Z parm: NVreg_RmNvlinkBandwidth:charp 2024-05-19T18:15:41.1009205Z parm: rm_firmware_active:charp 2024-05-19T18:15:41.1009614Z + HAS_NVIDIA_DRIVER=0 2024-05-19T18:15:41.1009997Z ++ command -v nvidia-smi 2024-05-19T18:15:41.1010395Z + '[' -x /usr/bin/nvidia-smi ']' 2024-05-19T18:15:41.1010775Z + set +e 2024-05-19T18:15:41.1011262Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-05-19T18:15:43.6152872Z + INSTALLED_DRIVER_VERSION=535.54.03 2024-05-19T18:15:43.6153547Z + NVIDIA_SMI_STATUS=0 2024-05-19T18:15:43.6154166Z + '[' 0 -ne 0 ']' 2024-05-19T18:15:43.6154539Z + '[' 535.54.03 '!=' 550.54.15 ']' 2024-05-19T18:15:43.6155335Z + echo 'NVIDIA driver (535.54.03) has been installed, but we expect to have 550.54.15 instead. Continuing' 2024-05-19T18:15:43.6156056Z + set -e 2024-05-19T18:15:43.6156359Z + '[' 0 -eq 0 ']' 2024-05-19T18:15:43.6156722Z + '[' amzn2 '!=' ubuntu20.04 ']' 2024-05-19T18:15:43.6157203Z + sudo yum groupinstall -y 'Development Tools' 2024-05-19T18:15:43.6157950Z NVIDIA driver (535.54.03) has been installed, but we expect to have 550.54.15 instead. Continuing 2024-05-19T18:15:43.9167137Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:15:45.0684046Z Maybe run: yum groups mark install (see man yum) 2024-05-19T18:15:45.0685677Z No packages in any requested group available to install or update 2024-05-19T18:15:45.1538127Z ++ uname -r 2024-05-19T18:15:45.1545878Z + sudo yum install -y 'kernel-devel-uname-r == 4.14.336-257.562.amzn2.x86_64' 2024-05-19T18:15:45.4694591Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:15:46.9575134Z Package kernel-devel-4.14.336-257.562.amzn2.x86_64 already installed and latest version 2024-05-19T18:15:46.9576516Z Nothing to do 2024-05-19T18:15:47.1029554Z + sudo modprobe backlight 2024-05-19T18:15:47.1296454Z + sudo curl -fsL -o /tmp/nvidia_driver https://s3.amazonaws.com/ossci-linux/nvidia_driver/NVIDIA-Linux-x86_64-550.54.15.run 2024-05-19T18:15:50.2315981Z + set +e 2024-05-19T18:15:50.2316555Z + sudo /bin/bash /tmp/nvidia_driver -s --no-drm 2024-05-19T18:15:51.1586947Z Verifying archive integrity... OK 2024-05-19T18:15:53.6611458Z Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 550.54.15........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ 2024-05-19T18:15:53.8738374Z 2024-05-19T18:15:53.8739816Z WARNING: The nvidia-drm module will not be installed. As a result, DRM-KMS will not function with this installation of the NVIDIA driver. 2024-05-19T18:15:53.8740623Z 2024-05-19T18:16:07.7509032Z 2024-05-19T18:16:07.7513396Z WARNING: nvidia-installer was forced to guess the X library path '/usr/lib64' and X module path '/usr/lib64/xorg/modules'; these paths were not queryable from the system. If X fails to find the NVIDIA X driver module, please install the `pkg-config` utility and the X.Org SDK/development package for your distribution and reinstall the driver. 2024-05-19T18:16:07.7516215Z 2024-05-19T18:16:07.7518867Z 2024-05-19T18:16:07.7521372Z WARNING: This NVIDIA driver package includes Vulkan components, but no Vulkan ICD loader was detected on this system. The NVIDIA Vulkan ICD will not function without the loader. Most distributions package the Vulkan loader; try installing the "vulkan-loader", "vulkan-icd-loader", or "libvulkan1" package. 2024-05-19T18:16:07.7523733Z 2024-05-19T18:16:15.9921654Z + NVIDIA_INSTALLATION_STATUS=0 2024-05-19T18:16:15.9922342Z + RESET_GPU=0 2024-05-19T18:16:15.9922953Z + '[' 0 -ne 0 ']' 2024-05-19T18:16:15.9923481Z ++ command -v nvidia-smi 2024-05-19T18:16:15.9925021Z + '[' -x /usr/bin/nvidia-smi ']' 2024-05-19T18:16:15.9927921Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-05-19T18:16:18.1578725Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-05-19T18:16:18.1579338Z + NVIDIA_SMI_STATUS=0 2024-05-19T18:16:18.1579819Z + '[' 0 -ne 0 ']' 2024-05-19T18:16:18.1580151Z + '[' 0 -eq 1 ']' 2024-05-19T18:16:18.1580522Z + sudo rm -fv /tmp/nvidia_driver 2024-05-19T18:16:18.1936243Z removed '/tmp/nvidia_driver' 2024-05-19T18:16:18.1951711Z + set -e 2024-05-19T18:16:18.1953910Z + post_install_nvidia_driver_common 2024-05-19T18:16:18.1955991Z + sudo modprobe nvidia 2024-05-19T18:16:18.2332478Z + echo 'After installing NVIDIA driver' 2024-05-19T18:16:18.2333065Z + lspci 2024-05-19T18:16:18.2333459Z After installing NVIDIA driver 2024-05-19T18:16:18.2416590Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-05-19T18:16:18.2417518Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-05-19T18:16:18.2418671Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-05-19T18:16:18.2419465Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-05-19T18:16:18.2420218Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-05-19T18:16:18.2421015Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-05-19T18:16:18.2421931Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-05-19T18:16:18.2422714Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-05-19T18:16:18.2423300Z + lsmod 2024-05-19T18:16:18.2429365Z Module Size Used by 2024-05-19T18:16:18.2430109Z nvidia_uvm 4599808 0 2024-05-19T18:16:18.2430619Z nvidia 53989376 1 nvidia_uvm 2024-05-19T18:16:18.2431065Z drm 421888 1 nvidia 2024-05-19T18:16:18.2431487Z i2c_core 77824 2 nvidia,drm 2024-05-19T18:16:18.2431915Z backlight 16384 0 2024-05-19T18:16:18.2432294Z xt_conntrack 16384 1 2024-05-19T18:16:18.2432680Z ipt_MASQUERADE 16384 1 2024-05-19T18:16:18.2433240Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-05-19T18:16:18.2433712Z nf_conntrack_netlink 49152 0 2024-05-19T18:16:18.2434154Z nfnetlink 16384 2 nf_conntrack_netlink 2024-05-19T18:16:18.2434637Z xfrm_user 45056 1 2024-05-19T18:16:18.2435169Z xfrm_algo 16384 1 xfrm_user 2024-05-19T18:16:18.2435743Z iptable_nat 16384 1 2024-05-19T18:16:18.2436240Z nf_conntrack_ipv4 16384 3 2024-05-19T18:16:18.2436836Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-05-19T18:16:18.2437486Z nf_nat_ipv4 16384 1 iptable_nat 2024-05-19T18:16:18.2438178Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-05-19T18:16:18.2439361Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-05-19T18:16:18.2440357Z xt_addrtype 16384 2 2024-05-19T18:16:18.2440738Z iptable_filter 16384 1 2024-05-19T18:16:18.2441117Z br_netfilter 24576 0 2024-05-19T18:16:18.2441513Z bridge 172032 1 br_netfilter 2024-05-19T18:16:18.2441954Z stp 16384 1 bridge 2024-05-19T18:16:18.2442376Z llc 16384 2 bridge,stp 2024-05-19T18:16:18.2442797Z overlay 86016 0 2024-05-19T18:16:18.2443158Z sunrpc 393216 1 2024-05-19T18:16:18.2443529Z dm_mirror 28672 0 2024-05-19T18:16:18.2444028Z dm_region_hash 20480 1 dm_mirror 2024-05-19T18:16:18.2444522Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-05-19T18:16:18.2445294Z dm_mod 143360 2 dm_log,dm_mirror 2024-05-19T18:16:18.2445755Z dax 69632 1 dm_mod 2024-05-19T18:16:18.2446155Z crc32_pclmul 16384 0 2024-05-19T18:16:18.2446575Z ghash_clmulni_intel 16384 0 2024-05-19T18:16:18.2446953Z pcbc 16384 0 2024-05-19T18:16:18.2447324Z aesni_intel 188416 0 2024-05-19T18:16:18.2447718Z aes_x86_64 20480 1 aesni_intel 2024-05-19T18:16:18.2448162Z crypto_simd 16384 1 aesni_intel 2024-05-19T18:16:18.2448580Z mousedev 24576 0 2024-05-19T18:16:18.2448984Z glue_helper 16384 1 aesni_intel 2024-05-19T18:16:18.2449532Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-05-19T18:16:18.2450065Z evdev 20480 3 2024-05-19T18:16:18.2450422Z psmouse 32768 0 2024-05-19T18:16:18.2450792Z button 16384 0 2024-05-19T18:16:18.2451163Z ena 139264 0 2024-05-19T18:16:18.2451532Z ptp 20480 1 ena 2024-05-19T18:16:18.2452035Z pps_core 20480 1 ptp 2024-05-19T18:16:18.2452728Z crc32c_intel 24576 0 2024-05-19T18:16:18.2453269Z autofs4 49152 2 2024-05-19T18:16:18.2453724Z + modinfo nvidia 2024-05-19T18:16:18.2454604Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-05-19T18:16:18.2455653Z alias: char-major-195-* 2024-05-19T18:16:18.2456175Z version: 550.54.15 2024-05-19T18:16:18.2456570Z supported: external 2024-05-19T18:16:18.2456918Z license: NVIDIA 2024-05-19T18:16:18.2457299Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-05-19T18:16:18.2457786Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-05-19T18:16:18.2458242Z srcversion: 833721318DA517F0C2FEC97 2024-05-19T18:16:18.2458711Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-05-19T18:16:18.2459338Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-05-19T18:16:18.2459830Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-05-19T18:16:18.2460313Z depends: i2c-core,drm 2024-05-19T18:16:18.2460682Z retpoline: Y 2024-05-19T18:16:18.2460994Z name: nvidia 2024-05-19T18:16:18.2461546Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-05-19T18:16:18.2462201Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-05-19T18:16:18.2462842Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-05-19T18:16:18.2463434Z parm: NVreg_ResmanDebugLevel:int 2024-05-19T18:16:18.2463873Z parm: NVreg_RmLogonRC:int 2024-05-19T18:16:18.2464312Z parm: NVreg_ModifyDeviceFiles:int 2024-05-19T18:16:18.2464768Z parm: NVreg_DeviceFileUID:int 2024-05-19T18:16:18.2465451Z parm: NVreg_DeviceFileGID:int 2024-05-19T18:16:18.2465886Z parm: NVreg_DeviceFileMode:int 2024-05-19T18:16:18.2466418Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-05-19T18:16:18.2466982Z parm: NVreg_UsePageAttributeTable:int 2024-05-19T18:16:18.2467472Z parm: NVreg_EnablePCIeGen3:int 2024-05-19T18:16:18.2468021Z parm: NVreg_EnableMSI:int 2024-05-19T18:16:18.2468437Z parm: NVreg_TCEBypassMode:int 2024-05-19T18:16:18.2468894Z parm: NVreg_EnableStreamMemOPs:int 2024-05-19T18:16:18.2469430Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-05-19T18:16:18.2470002Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-05-19T18:16:18.2470563Z parm: NVreg_EnableS0ixPowerManagement:int 2024-05-19T18:16:18.2471174Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-05-19T18:16:18.2471774Z parm: NVreg_DynamicPowerManagement:int 2024-05-19T18:16:18.2472378Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-05-19T18:16:18.2473047Z parm: NVreg_EnableGpuFirmware:int 2024-05-19T18:16:18.2473550Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-05-19T18:16:18.2474090Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-05-19T18:16:18.2474643Z parm: NVreg_EnableUserNUMAManagement:int 2024-05-19T18:16:18.2475149Z parm: NVreg_MemoryPoolSize:int 2024-05-19T18:16:18.2475616Z parm: NVreg_KMallocHeapMaxSize:int 2024-05-19T18:16:18.2476097Z parm: NVreg_VMallocHeapMaxSize:int 2024-05-19T18:16:18.2476570Z parm: NVreg_IgnoreMMIOCheck:int 2024-05-19T18:16:18.2477017Z parm: NVreg_NvLinkDisable:int 2024-05-19T18:16:18.2477519Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-05-19T18:16:18.2478041Z parm: NVreg_RegisterPCIDriver:int 2024-05-19T18:16:18.2478524Z parm: NVreg_EnableResizableBar:int 2024-05-19T18:16:18.2479011Z parm: NVreg_EnableDbgBreakpoint:int 2024-05-19T18:16:18.2479511Z parm: NVreg_EnableNonblockingOpen:int 2024-05-19T18:16:18.2479995Z parm: NVreg_RegistryDwords:charp 2024-05-19T18:16:18.2480502Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-05-19T18:16:18.2480983Z parm: NVreg_RmMsg:charp 2024-05-19T18:16:18.2481480Z parm: NVreg_GpuBlacklist:charp 2024-05-19T18:16:18.2481947Z parm: NVreg_TemporaryFilePath:charp 2024-05-19T18:16:18.2482424Z parm: NVreg_ExcludedGpus:charp 2024-05-19T18:16:18.2482881Z parm: NVreg_DmaRemapPeerMmio:int 2024-05-19T18:16:18.2483359Z parm: NVreg_RmNvlinkBandwidth:charp 2024-05-19T18:16:18.2483825Z parm: NVreg_ImexChannelCount:int 2024-05-19T18:16:18.2484283Z parm: rm_firmware_active:charp 2024-05-19T18:16:18.2484900Z + set +e 2024-05-19T18:16:18.2485212Z + nvidia-smi 2024-05-19T18:16:19.8093362Z Sun May 19 18:16:19 2024 2024-05-19T18:16:19.8094169Z +-----------------------------------------------------------------------------------------+ 2024-05-19T18:16:19.8094988Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-05-19T18:16:19.8096057Z |-----------------------------------------+------------------------+----------------------+ 2024-05-19T18:16:19.8096851Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-05-19T18:16:19.8097730Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-05-19T18:16:19.8098423Z | | | MIG M. | 2024-05-19T18:16:19.8098973Z |=========================================+========================+======================| 2024-05-19T18:16:19.8186779Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-05-19T18:16:19.8187496Z | 0% 28C P0 57W / 300W | 0MiB / 23028MiB | 0% Default | 2024-05-19T18:16:19.8188117Z | | | N/A | 2024-05-19T18:16:19.8188899Z +-----------------------------------------+------------------------+----------------------+ 2024-05-19T18:16:19.8189492Z 2024-05-19T18:16:19.8190132Z +-----------------------------------------------------------------------------------------+ 2024-05-19T18:16:19.8190754Z | Processes: | 2024-05-19T18:16:19.8191451Z | GPU GI CI PID Type Process name GPU Memory | 2024-05-19T18:16:19.8192144Z | ID ID Usage | 2024-05-19T18:16:19.8192705Z |=========================================================================================| 2024-05-19T18:16:19.8193414Z | No running processes found | 2024-05-19T18:16:19.8194164Z +-----------------------------------------------------------------------------------------+ 2024-05-19T18:16:20.4278274Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-05-19T18:16:21.9914731Z NVIDIA A10G 2024-05-19T18:16:22.4326160Z + NVIDIA_SMI_STATUS=0 2024-05-19T18:16:22.4326793Z + '[' 0 -eq 0 ']' 2024-05-19T18:16:22.4327202Z + echo 'INFO: Ignoring allowed status 0' 2024-05-19T18:16:22.4327625Z + set -e 2024-05-19T18:16:22.4327930Z INFO: Ignoring allowed status 0 2024-05-19T18:16:22.4329800Z == Installing nvidia container toolkit for amzn2 == 2024-05-19T18:16:22.4331584Z + sudo yum install -y yum-utils 2024-05-19T18:16:22.7388089Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:16:24.1989491Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-05-19T18:16:24.1990167Z Nothing to do 2024-05-19T18:16:24.3455117Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-05-19T18:16:24.6743201Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:16:24.7126613Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-05-19T18:16:24.7128228Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-05-19T18:16:24.7129188Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-05-19T18:16:24.7239193Z + sudo yum install -y nvidia-docker2 2024-05-19T18:16:25.0284920Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:16:26.4784721Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-05-19T18:16:26.4786050Z Nothing to do 2024-05-19T18:16:26.6230105Z + sudo systemctl restart docker 2024-05-19T18:16:56.6876230Z Command completed after 1 attempt(s). 2024-05-19T18:16:56.6933299Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-05-19T18:16:56.6934064Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-05-19T18:16:56.6934869Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-05-19T18:16:56.6935532Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:16:56.6943307Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:16:56.6943821Z env: 2024-05-19T18:16:56.6944108Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:16:56.6944566Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:16:56.6945043Z ##[endgroup] 2024-05-19T18:16:57.0467273Z Defaulting to user installation because normal site-packages is not writeable 2024-05-19T18:16:57.4497179Z Collecting psutil==5.9.1 2024-05-19T18:16:57.4661058Z Downloading psutil-5.9.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-05-19T18:16:57.5103287Z Collecting nvidia-ml-py==11.525.84 2024-05-19T18:16:57.5137299Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-05-19T18:16:57.5516618Z Installing collected packages: psutil, nvidia-ml-py 2024-05-19T18:16:57.7038158Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-05-19T18:16:57.8865245Z Prepare all required actions 2024-05-19T18:16:57.8865737Z Getting action download info 2024-05-19T18:16:57.9839723Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-05-19T18:16:58.2227260Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-05-19T18:16:58.3619867Z ##[group]Run ./.github/actions/download-build-artifacts 2024-05-19T18:16:58.3620343Z with: 2024-05-19T18:16:58.3620671Z name: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:16:58.3621120Z s3-bucket: gha-artifacts 2024-05-19T18:16:58.3621462Z env: 2024-05-19T18:16:58.3621743Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:16:58.3622180Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:16:58.3622655Z ##[endgroup] 2024-05-19T18:16:58.3655173Z ##[group]Run seemethere/download-artifact-s3@v4 2024-05-19T18:16:58.3655631Z with: 2024-05-19T18:16:58.3656005Z name: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:16:58.3656465Z s3-bucket: gha-artifacts 2024-05-19T18:16:58.3656816Z region: us-east-1 2024-05-19T18:16:58.3657120Z env: 2024-05-19T18:16:58.3657411Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:16:58.3657852Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:16:58.3658384Z ##[endgroup] 2024-05-19T18:16:58.7782656Z (node:28709) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-05-19T18:16:58.7783344Z 2024-05-19T18:16:58.7783600Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-05-19T18:16:58.7784313Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-05-19T18:16:58.7785139Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-05-19T18:16:58.8595742Z Found 1 objects with prefix pytorch/pytorch/9148947323/linux-focal-cuda12.4-py3.10-gcc9-sm86/ 2024-05-19T18:16:58.8596829Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-05-19T18:17:06.5240911Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-05-19T18:17:06.5247764Z Artifact download has finished successfully 2024-05-19T18:17:06.5408628Z ##[group]Run unzip -o artifacts.zip 2024-05-19T18:17:06.5409079Z unzip -o artifacts.zip 2024-05-19T18:17:06.5416929Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:17:06.5417435Z env: 2024-05-19T18:17:06.5417714Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:17:06.5418168Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:17:06.5418654Z ##[endgroup] 2024-05-19T18:17:06.5460164Z Archive: artifacts.zip 2024-05-19T18:17:06.5461667Z creating: dist/ 2024-05-19T18:17:08.5191640Z inflating: dist/torch-2.4.0a0+gitc8c7ddf-cp310-cp310-linux_x86_64.whl 2024-05-19T18:17:08.5192510Z creating: build/custom_test_artifacts/ 2024-05-19T18:17:08.5193536Z creating: build/custom_test_artifacts/custom-op-build/ 2024-05-19T18:17:08.5194255Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-05-19T18:17:08.5195069Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-05-19T18:17:08.5198541Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-05-19T18:17:08.5199449Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-05-19T18:17:08.5200362Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-05-19T18:17:08.5201327Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-05-19T18:17:08.5202270Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-05-19T18:17:08.5203439Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-05-19T18:17:08.5205034Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-05-19T18:17:08.5206040Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-05-19T18:17:08.5207007Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-05-19T18:17:08.5208133Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-05-19T18:17:08.5209263Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-05-19T18:17:08.5210376Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-05-19T18:17:08.5211512Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-05-19T18:17:08.5212675Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-05-19T18:17:08.5213822Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-05-19T18:17:08.5214823Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-05-19T18:17:08.5215814Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-05-19T18:17:08.5251785Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-05-19T18:17:08.5290922Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-05-19T18:17:08.5292338Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-05-19T18:17:08.5335941Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-05-19T18:17:08.5337331Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-05-19T18:17:08.5338723Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-05-19T18:17:08.5340150Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-05-19T18:17:08.5341532Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-05-19T18:17:08.5342939Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-05-19T18:17:08.5344291Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-05-19T18:17:08.5345644Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-05-19T18:17:08.5347069Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-05-19T18:17:08.5348322Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-05-19T18:17:08.5349526Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-05-19T18:17:08.5350713Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-05-19T18:17:08.5351914Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-05-19T18:17:08.5353164Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-05-19T18:17:08.5354350Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-05-19T18:17:08.5415274Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-05-19T18:17:08.5485899Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-05-19T18:17:08.5487056Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-05-19T18:17:08.5488027Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-05-19T18:17:08.5488871Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-05-19T18:17:08.5489746Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-05-19T18:17:08.5490649Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-05-19T18:17:08.5491653Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-05-19T18:17:08.5492790Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-05-19T18:17:08.5493866Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-05-19T18:17:08.5494873Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-05-19T18:17:08.5495921Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-05-19T18:17:08.5496974Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-05-19T18:17:08.5498027Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-05-19T18:17:08.5499070Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-05-19T18:17:08.5500106Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-05-19T18:17:08.5511260Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-05-19T18:17:08.5641294Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-05-19T18:17:08.5642687Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-05-19T18:17:08.5644032Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-05-19T18:17:08.5645469Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-05-19T18:17:08.5646604Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-05-19T18:17:08.5647673Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-05-19T18:17:08.5648773Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-05-19T18:17:08.5650007Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-05-19T18:17:08.5651123Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-05-19T18:17:08.5652239Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-05-19T18:17:08.5653339Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-05-19T18:17:08.5664156Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-05-19T18:17:08.5743087Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-05-19T18:17:08.5744424Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-05-19T18:17:08.5745684Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-05-19T18:17:08.5746776Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-05-19T18:17:08.5747850Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-05-19T18:17:08.5748914Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-05-19T18:17:08.5749815Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-05-19T18:17:08.5750647Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-05-19T18:17:08.5751446Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-05-19T18:17:08.5752264Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-05-19T18:17:08.5857209Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-05-19T18:17:08.5917490Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-05-19T18:17:08.5918234Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-05-19T18:17:08.5918920Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-05-19T18:17:08.5919708Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-05-19T18:17:08.5925066Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-05-19T18:17:08.5926116Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-05-19T18:17:08.5927024Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-05-19T18:17:08.5928163Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-05-19T18:17:08.5929296Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-05-19T18:17:08.5930453Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-05-19T18:17:08.5931611Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-05-19T18:17:08.5932701Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-05-19T18:17:08.5933769Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-05-19T18:17:08.5935095Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-05-19T18:17:08.5936328Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-05-19T18:17:08.5937663Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-05-19T18:17:08.5938807Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-05-19T18:17:08.5939932Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-05-19T18:17:08.5941212Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-05-19T18:17:08.5942276Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-05-19T18:17:08.5943257Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-05-19T18:17:08.5978287Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-05-19T18:17:08.6017551Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-05-19T18:17:08.6018962Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-05-19T18:17:08.6064258Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-05-19T18:17:08.6067938Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-05-19T18:17:08.6071319Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-05-19T18:17:08.6072902Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-05-19T18:17:08.6074279Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-05-19T18:17:08.6075629Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-05-19T18:17:08.6076985Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-05-19T18:17:08.6078337Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-05-19T18:17:08.6079662Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-05-19T18:17:08.6080921Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-05-19T18:17:08.6082127Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-05-19T18:17:08.6083313Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-05-19T18:17:08.6084511Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-05-19T18:17:08.6086622Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-05-19T18:17:08.6087852Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-05-19T18:17:08.6142008Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-05-19T18:17:08.6213351Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-05-19T18:17:08.6214518Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-05-19T18:17:08.6215478Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-05-19T18:17:08.6216310Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-05-19T18:17:08.6217180Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-05-19T18:17:08.6218112Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-05-19T18:17:08.6219162Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-05-19T18:17:08.6220310Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-05-19T18:17:08.6221659Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-05-19T18:17:08.6222766Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-05-19T18:17:08.6223857Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-05-19T18:17:08.6224944Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-05-19T18:17:08.6226032Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-05-19T18:17:08.6227116Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-05-19T18:17:08.6228189Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-05-19T18:17:08.6237805Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-05-19T18:17:08.6300005Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-05-19T18:17:08.6302121Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-05-19T18:17:08.6303208Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-05-19T18:17:08.6304145Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-05-19T18:17:08.6305025Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-05-19T18:17:08.6305880Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-05-19T18:17:08.6306750Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-05-19T18:17:08.6307576Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-05-19T18:17:08.6308331Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-05-19T18:17:08.6309114Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-05-19T18:17:08.6353161Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-05-19T18:17:08.6353903Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-05-19T18:17:08.6354660Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-05-19T18:17:08.6355515Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-05-19T18:17:08.6360671Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-05-19T18:17:08.6361647Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-05-19T18:17:08.6362625Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-05-19T18:17:08.6363652Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-05-19T18:17:08.6364867Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-05-19T18:17:08.6366014Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-05-19T18:17:08.6367151Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-05-19T18:17:08.6368184Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-05-19T18:17:08.6369210Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-05-19T18:17:08.6370395Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-05-19T18:17:08.6371588Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-05-19T18:17:08.6372878Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-05-19T18:17:08.6374062Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-05-19T18:17:08.6375253Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-05-19T18:17:08.6376450Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-05-19T18:17:08.6377522Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-05-19T18:17:08.6378571Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-05-19T18:17:08.6413925Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-05-19T18:17:08.6453088Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-05-19T18:17:08.6454650Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-05-19T18:17:08.6498373Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-05-19T18:17:08.6499811Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-05-19T18:17:08.6501262Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-05-19T18:17:08.6502751Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-05-19T18:17:08.6504182Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-05-19T18:17:08.6516337Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-05-19T18:17:08.6517773Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-05-19T18:17:08.6519180Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-05-19T18:17:08.6520557Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-05-19T18:17:08.6521863Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-05-19T18:17:08.6523119Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-05-19T18:17:08.6524371Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-05-19T18:17:08.6525846Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-05-19T18:17:08.6527067Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-05-19T18:17:08.6528308Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-05-19T18:17:08.6595841Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-05-19T18:17:08.6667673Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-05-19T18:17:08.6668906Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-05-19T18:17:08.6669935Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-05-19T18:17:08.6670969Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-05-19T18:17:08.6671905Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-05-19T18:17:08.6673000Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-05-19T18:17:08.6674113Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-05-19T18:17:08.6675343Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-05-19T18:17:08.6676519Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-05-19T18:17:08.6677636Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-05-19T18:17:08.6678787Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-05-19T18:17:08.6680059Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-05-19T18:17:08.6681232Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-05-19T18:17:08.6682396Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-05-19T18:17:08.6683530Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-05-19T18:17:08.6684888Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-05-19T18:17:08.6797838Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-05-19T18:17:08.6798986Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-05-19T18:17:08.6800150Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-05-19T18:17:08.6801443Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-05-19T18:17:08.6802726Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-05-19T18:17:08.6803898Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-05-19T18:17:08.6805228Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-05-19T18:17:08.6806446Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-05-19T18:17:08.6807657Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-05-19T18:17:08.6808862Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-05-19T18:17:08.6810072Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-05-19T18:17:08.6821812Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-05-19T18:17:08.6874420Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-05-19T18:17:08.6875673Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-05-19T18:17:08.6876784Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-05-19T18:17:08.6877791Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-05-19T18:17:08.6878731Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-05-19T18:17:08.6879796Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-05-19T18:17:08.6880743Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-05-19T18:17:08.6881643Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-05-19T18:17:08.6882471Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-05-19T18:17:08.6883318Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-05-19T18:17:08.6979909Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-05-19T18:17:08.7020193Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-05-19T18:17:08.7020819Z creating: build/lib/ 2024-05-19T18:17:08.7105919Z inflating: build/lib/libprotobuf-lite.a 2024-05-19T18:17:08.7543580Z inflating: build/lib/libprotobuf.a 2024-05-19T18:17:08.7552674Z inflating: build/lib/libpthreadpool.a 2024-05-19T18:17:08.7560125Z inflating: build/lib/libcpuinfo.a 2024-05-19T18:17:08.7567769Z inflating: build/lib/libcpuinfo_internals.a 2024-05-19T18:17:08.7568725Z inflating: build/lib/libclog.a 2024-05-19T18:17:08.7585643Z inflating: build/lib/libnnpack.a 2024-05-19T18:17:08.7587597Z inflating: build/lib/libnnpack_reference_layers.a 2024-05-19T18:17:08.7649675Z inflating: build/lib/libgtest.a 2024-05-19T18:17:08.7721803Z inflating: build/lib/libbenchmark.a 2024-05-19T18:17:08.7783754Z inflating: build/lib/libasmjit.a 2024-05-19T18:17:08.7790429Z inflating: build/lib/libittnotify.a 2024-05-19T18:17:08.7816684Z inflating: build/lib/libtensorpipe_uv.a 2024-05-19T18:17:08.7938907Z inflating: build/lib/libgloo.a 2024-05-19T18:17:08.7959497Z inflating: build/lib/libfmt.a 2024-05-19T18:17:08.8047846Z inflating: build/lib/libc10.so 2024-05-19T18:17:08.8049207Z inflating: build/lib/libcaffe2_nvrtc.so 2024-05-19T18:17:08.8049695Z inflating: build/lib/libfoxi_loader.a 2024-05-19T18:17:08.8051141Z inflating: build/lib/libtorch_global_deps.so 2024-05-19T18:17:08.8063921Z inflating: build/lib/libqnnpack.a 2024-05-19T18:17:08.8548858Z inflating: build/lib/libprotoc.a 2024-05-19T18:17:08.8567598Z inflating: build/lib/libpytorch_qnnpack.a 2024-05-19T18:17:08.8584719Z inflating: build/lib/libgmock.a 2024-05-19T18:17:08.8585285Z inflating: build/lib/libgtest_main.a 2024-05-19T18:17:08.8585899Z inflating: build/lib/libbenchmark_main.a 2024-05-19T18:17:08.8955319Z inflating: build/lib/libgloo_cuda.a 2024-05-19T18:17:09.8677072Z inflating: build/lib/libdnnl.a 2024-05-19T18:17:09.9225613Z inflating: build/lib/libtensorpipe.a 2024-05-19T18:17:09.9277797Z inflating: build/lib/libc10_cuda.so 2024-05-19T18:17:09.9278599Z inflating: build/lib/libgmock_main.a 2024-05-19T18:17:10.0481433Z inflating: build/lib/libfbgemm.a 2024-05-19T18:17:10.0728210Z inflating: build/lib/libtensorpipe_cuda.a 2024-05-19T18:17:10.1224052Z inflating: build/lib/libkineto.a 2024-05-19T18:17:10.1260572Z inflating: build/lib/libcaffe2_protos.a 2024-05-19T18:17:10.1301139Z inflating: build/lib/libonnx_proto.a 2024-05-19T18:17:10.1478773Z inflating: build/lib/libXNNPACK.a 2024-05-19T18:17:10.2151532Z inflating: build/lib/libonnx.a 2024-05-19T18:17:12.5189606Z inflating: build/lib/libtorch_cpu.so 2024-05-19T18:17:12.5193251Z inflating: build/lib/libunbox_lib.a 2024-05-19T18:17:12.5196861Z inflating: build/lib/libshm.so 2024-05-19T18:17:14.4922080Z inflating: build/lib/libtorch_cuda.so 2024-05-19T18:17:14.4922930Z inflating: build/lib/libtorch.so 2024-05-19T18:17:14.4925109Z inflating: build/lib/libc10d_cuda_test.so 2024-05-19T18:17:15.3451807Z inflating: build/lib/libtorch_cuda_linalg.so 2024-05-19T18:17:15.5296740Z inflating: build/lib/libtorch_python.so 2024-05-19T18:17:15.5362563Z inflating: build/lib/libtorchbind_test.so 2024-05-19T18:17:15.5381580Z inflating: build/lib/libjitbackend_test.so 2024-05-19T18:17:15.5406020Z inflating: build/lib/libbackend_with_compiler.so 2024-05-19T18:17:15.5439223Z inflating: build/lib/libnnapi_backend.so 2024-05-19T18:17:15.5439825Z creating: build/bin/ 2024-05-19T18:17:15.5488035Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-05-19T18:17:15.5537733Z inflating: build/bin/c10_DeviceGuard_test 2024-05-19T18:17:15.5587154Z inflating: build/bin/c10_Device_test 2024-05-19T18:17:15.5643701Z inflating: build/bin/c10_DispatchKeySet_test 2024-05-19T18:17:15.5695528Z inflating: build/bin/c10_Scalar_test 2024-05-19T18:17:15.5742726Z inflating: build/bin/c10_StreamGuard_test 2024-05-19T18:17:15.5792451Z inflating: build/bin/c10_SymInt_test 2024-05-19T18:17:15.5844922Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-05-19T18:17:15.5898339Z inflating: build/bin/c10_InlineStreamGuard_test 2024-05-19T18:17:15.5953455Z inflating: build/bin/c10_SizesAndStrides_test 2024-05-19T18:17:15.6021340Z inflating: build/bin/c10_cow_test 2024-05-19T18:17:15.6072925Z inflating: build/bin/c10_Bitset_test 2024-05-19T18:17:15.6120142Z inflating: build/bin/c10_ConstexprCrc_test 2024-05-19T18:17:15.6168697Z inflating: build/bin/c10_DeadlockDetection_test 2024-05-19T18:17:15.6217730Z inflating: build/bin/c10_Half_test 2024-05-19T18:17:15.6272091Z inflating: build/bin/c10_LeftRight_test 2024-05-19T18:17:15.6325309Z inflating: build/bin/c10_Metaprogramming_test 2024-05-19T18:17:15.6373536Z inflating: build/bin/c10_Synchronized_test 2024-05-19T18:17:15.6427170Z inflating: build/bin/c10_ThreadLocal_test 2024-05-19T18:17:15.6477015Z inflating: build/bin/c10_TypeIndex_test 2024-05-19T18:17:15.6526623Z inflating: build/bin/c10_TypeList_test 2024-05-19T18:17:15.6574121Z inflating: build/bin/c10_TypeTraits_test 2024-05-19T18:17:15.6624108Z inflating: build/bin/c10_accumulate_test 2024-05-19T18:17:15.6677693Z inflating: build/bin/c10_bfloat16_test 2024-05-19T18:17:15.6726830Z inflating: build/bin/c10_bit_cast_test 2024-05-19T18:17:15.6781308Z inflating: build/bin/c10_complex_math_test 2024-05-19T18:17:15.6835314Z inflating: build/bin/c10_complex_test 2024-05-19T18:17:15.6886491Z inflating: build/bin/c10_exception_test 2024-05-19T18:17:15.6935022Z inflating: build/bin/c10_flags_test 2024-05-19T18:17:15.6983773Z inflating: build/bin/c10_generic_math_test 2024-05-19T18:17:15.7142166Z inflating: build/bin/c10_intrusive_ptr_test 2024-05-19T18:17:15.7191248Z inflating: build/bin/c10_irange_test 2024-05-19T18:17:15.7242963Z inflating: build/bin/c10_lazy_test 2024-05-19T18:17:15.7298204Z inflating: build/bin/c10_logging_test 2024-05-19T18:17:15.7371862Z inflating: build/bin/c10_optional_test 2024-05-19T18:17:15.7431955Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-05-19T18:17:15.7483777Z inflating: build/bin/c10_registry_test 2024-05-19T18:17:15.7629505Z inflating: build/bin/c10_small_vector_test 2024-05-19T18:17:15.7678815Z inflating: build/bin/c10_ssize_test 2024-05-19T18:17:15.7728814Z inflating: build/bin/c10_string_util_test 2024-05-19T18:17:15.7785870Z inflating: build/bin/c10_string_view_test 2024-05-19T18:17:15.7834588Z inflating: build/bin/c10_tempfile_test 2024-05-19T18:17:15.7888435Z inflating: build/bin/c10_typeid_test 2024-05-19T18:17:15.7935255Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-05-19T18:17:15.8367314Z inflating: build/bin/protoc-3.13.0.0 2024-05-19T18:17:15.8799356Z inflating: build/bin/protoc 2024-05-19T18:17:15.8850350Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-05-19T18:17:15.8901566Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-05-19T18:17:15.8952660Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-05-19T18:17:15.9003352Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-05-19T18:17:15.9054314Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-05-19T18:17:15.9105867Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-05-19T18:17:15.9153316Z inflating: build/bin/c10_cuda_CUDATest 2024-05-19T18:17:15.9203886Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-05-19T18:17:15.9521339Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-05-19T18:17:15.9852930Z inflating: build/bin/vec_test_all_types_AVX512 2024-05-19T18:17:16.0198433Z inflating: build/bin/vec_test_all_types_AVX2 2024-05-19T18:17:16.0249732Z inflating: build/bin/FileStoreTest 2024-05-19T18:17:16.0304712Z inflating: build/bin/TCPStoreTest 2024-05-19T18:17:16.0355951Z inflating: build/bin/HashStoreTest 2024-05-19T18:17:16.0369010Z inflating: build/bin/ProcessGroupMPITest 2024-05-19T18:17:16.0420866Z inflating: build/bin/test_edge_op_registration 2024-05-19T18:17:16.0425346Z inflating: build/bin/torch_shm_manager 2024-05-19T18:17:16.0428246Z inflating: build/bin/example_allreduce 2024-05-19T18:17:16.0481613Z inflating: build/bin/test_dist_autograd 2024-05-19T18:17:16.0548102Z inflating: build/bin/test_cpp_rpc 2024-05-19T18:17:16.0550431Z inflating: build/bin/parallel_benchmark 2024-05-19T18:17:16.0615402Z inflating: build/bin/test_mobile_nnc 2024-05-19T18:17:16.0623914Z inflating: build/bin/aot_model_compiler_test 2024-05-19T18:17:16.0956373Z inflating: build/bin/test_lazy 2024-05-19T18:17:16.2102550Z inflating: build/bin/test_api 2024-05-19T18:17:16.2173829Z inflating: build/bin/Dict_test 2024-05-19T18:17:16.2223839Z inflating: build/bin/Dimname_test 2024-05-19T18:17:16.2286286Z inflating: build/bin/MaybeOwned_test 2024-05-19T18:17:16.2341502Z inflating: build/bin/NamedTensor_test 2024-05-19T18:17:16.2399155Z inflating: build/bin/apply_utils_test 2024-05-19T18:17:16.2456006Z inflating: build/bin/atest 2024-05-19T18:17:16.2517810Z inflating: build/bin/basic 2024-05-19T18:17:16.2571215Z inflating: build/bin/broadcast_test 2024-05-19T18:17:16.2620228Z inflating: build/bin/cpu_allocator_test 2024-05-19T18:17:16.2676553Z inflating: build/bin/cpu_generator_test 2024-05-19T18:17:16.2727871Z inflating: build/bin/cpu_profiling_allocator_test 2024-05-19T18:17:16.2818260Z inflating: build/bin/cpu_rng_test 2024-05-19T18:17:16.2867389Z inflating: build/bin/dispatch_key_set_test 2024-05-19T18:17:16.2915449Z inflating: build/bin/dlconvertor_test 2024-05-19T18:17:16.2971778Z inflating: build/bin/extension_backend_test 2024-05-19T18:17:16.3025707Z inflating: build/bin/half_test 2024-05-19T18:17:16.3117813Z inflating: build/bin/ivalue_test 2024-05-19T18:17:16.3166043Z inflating: build/bin/lazy_tensor_test 2024-05-19T18:17:16.3218706Z inflating: build/bin/math_kernel_test 2024-05-19T18:17:16.3271410Z inflating: build/bin/memory_format_test 2024-05-19T18:17:16.3322745Z inflating: build/bin/memory_overlapping_test 2024-05-19T18:17:16.3374001Z inflating: build/bin/mobile_memory_cleanup 2024-05-19T18:17:16.3428445Z inflating: build/bin/native_test 2024-05-19T18:17:16.3477617Z inflating: build/bin/operator_name_test 2024-05-19T18:17:16.3526852Z inflating: build/bin/operators_test 2024-05-19T18:17:16.3576824Z inflating: build/bin/packedtensoraccessor_test 2024-05-19T18:17:16.3641729Z inflating: build/bin/pow_test 2024-05-19T18:17:16.3697118Z inflating: build/bin/quantized_test 2024-05-19T18:17:16.3745261Z inflating: build/bin/reduce_ops_test 2024-05-19T18:17:16.3794617Z inflating: build/bin/reportMemoryUsage_test 2024-05-19T18:17:16.3849172Z inflating: build/bin/scalar_tensor_test 2024-05-19T18:17:16.3907462Z inflating: build/bin/scalar_test 2024-05-19T18:17:16.3956872Z inflating: build/bin/StorageUtils_test 2024-05-19T18:17:16.4007953Z inflating: build/bin/stride_properties_test 2024-05-19T18:17:16.4083839Z inflating: build/bin/tensor_iterator_test 2024-05-19T18:17:16.4136265Z inflating: build/bin/test_parallel 2024-05-19T18:17:16.4138939Z inflating: build/bin/thread_init_test 2024-05-19T18:17:16.4193217Z inflating: build/bin/type_ptr_test 2024-05-19T18:17:16.4251195Z inflating: build/bin/type_test 2024-05-19T18:17:16.4301880Z inflating: build/bin/undefined_tensor_test 2024-05-19T18:17:16.4303268Z inflating: build/bin/verify_api_visibility 2024-05-19T18:17:16.4370234Z inflating: build/bin/legacy_vmap_test 2024-05-19T18:17:16.4420484Z inflating: build/bin/weakref_test 2024-05-19T18:17:16.4470572Z inflating: build/bin/wrapdim_test 2024-05-19T18:17:16.4520122Z inflating: build/bin/xla_tensor_test 2024-05-19T18:17:16.4578556Z inflating: build/bin/IListRef_test 2024-05-19T18:17:16.4681567Z inflating: build/bin/List_test 2024-05-19T18:17:16.4746419Z inflating: build/bin/KernelFunction_test 2024-05-19T18:17:16.4864785Z inflating: build/bin/kernel_function_legacy_test 2024-05-19T18:17:16.4958758Z inflating: build/bin/kernel_function_test 2024-05-19T18:17:16.5082765Z inflating: build/bin/kernel_lambda_legacy_test 2024-05-19T18:17:16.5182546Z inflating: build/bin/kernel_lambda_test 2024-05-19T18:17:16.5241727Z inflating: build/bin/kernel_stackbased_test 2024-05-19T18:17:16.5334687Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-05-19T18:17:16.5384003Z inflating: build/bin/CppSignature_test 2024-05-19T18:17:16.5431767Z inflating: build/bin/op_allowlist_test 2024-05-19T18:17:16.5485327Z inflating: build/bin/backend_fallback_test 2024-05-19T18:17:16.5777762Z inflating: build/bin/op_registration_test 2024-05-19T18:17:16.5840402Z inflating: build/bin/inline_container_test 2024-05-19T18:17:16.5891112Z inflating: build/bin/cuda_apply_test 2024-05-19T18:17:16.5940172Z inflating: build/bin/cuda_allocator_test 2024-05-19T18:17:16.5993818Z inflating: build/bin/cuda_caching_host_allocator_test 2024-05-19T18:17:16.6051328Z inflating: build/bin/cuda_atomic_ops_test 2024-05-19T18:17:16.6119702Z inflating: build/bin/cuda_complex_math_test 2024-05-19T18:17:16.6176339Z inflating: build/bin/cuda_complex_test 2024-05-19T18:17:16.6224401Z inflating: build/bin/cuda_device_test 2024-05-19T18:17:16.6281939Z inflating: build/bin/cuda_cub_test 2024-05-19T18:17:16.6331389Z inflating: build/bin/cuda_dlconvertor_test 2024-05-19T18:17:16.6394098Z inflating: build/bin/cuda_distributions_test 2024-05-19T18:17:16.6450761Z inflating: build/bin/cuda_generator_test 2024-05-19T18:17:16.6498905Z inflating: build/bin/cuda_half_test 2024-05-19T18:17:16.6548609Z inflating: build/bin/cuda_integer_divider_test 2024-05-19T18:17:16.6596701Z inflating: build/bin/cuda_optional_test 2024-05-19T18:17:16.6646793Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-05-19T18:17:16.6699280Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-05-19T18:17:16.6747931Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-05-19T18:17:16.6806748Z inflating: build/bin/cuda_stream_test 2024-05-19T18:17:16.6854586Z inflating: build/bin/cuda_cudnn_test 2024-05-19T18:17:16.6905508Z inflating: build/bin/cuda_vectorized_test 2024-05-19T18:17:16.6919918Z inflating: build/bin/tutorial_tensorexpr 2024-05-19T18:17:16.6983614Z inflating: build/bin/ProcessGroupGlooTest 2024-05-19T18:17:16.7039166Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-05-19T18:17:16.7099912Z inflating: build/bin/ProcessGroupNCCLTest 2024-05-19T18:17:16.7160924Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-05-19T18:17:16.7966575Z inflating: build/bin/test_tensorexpr 2024-05-19T18:17:16.8526672Z inflating: build/bin/test_jit 2024-05-19T18:17:16.8527528Z creating: .additional_ci_files/ 2024-05-19T18:17:16.8565265Z inflating: .additional_ci_files/test-times.json 2024-05-19T18:17:16.8726372Z inflating: .additional_ci_files/test-class-times.json 2024-05-19T18:17:16.8753803Z ##[group]Run rm artifacts.zip 2024-05-19T18:17:16.8754199Z rm artifacts.zip 2024-05-19T18:17:16.8761784Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:17:16.8762291Z env: 2024-05-19T18:17:16.8762576Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:17:16.8763033Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:17:16.8763510Z ##[endgroup] 2024-05-19T18:17:16.9441324Z ##[group]Run df -H 2024-05-19T18:17:16.9441647Z df -H 2024-05-19T18:17:16.9448668Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:17:16.9449176Z env: 2024-05-19T18:17:16.9449449Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:17:16.9449901Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:17:16.9450384Z ##[endgroup] 2024-05-19T18:17:16.9478211Z Filesystem Size Used Avail Use% Mounted on 2024-05-19T18:17:16.9478731Z devtmpfs 34G 0 34G 0% /dev 2024-05-19T18:17:16.9479389Z tmpfs 34G 0 34G 0% /dev/shm 2024-05-19T18:17:16.9479919Z tmpfs 34G 455k 34G 1% /run 2024-05-19T18:17:16.9480564Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-05-19T18:17:16.9481037Z /dev/nvme0n1p1 162G 44G 118G 27% / 2024-05-19T18:17:16.9519194Z Prepare all required actions 2024-05-19T18:17:16.9519638Z Getting action download info 2024-05-19T18:17:17.0557685Z ##[group]Run ./.github/actions/download-td-artifacts 2024-05-19T18:17:17.0558147Z with: 2024-05-19T18:17:17.0558419Z env: 2024-05-19T18:17:17.0558697Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:17:17.0559152Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:17:17.0559639Z ##[endgroup] 2024-05-19T18:17:17.0600664Z ##[group]Run seemethere/download-artifact-s3@v4 2024-05-19T18:17:17.0601111Z with: 2024-05-19T18:17:17.0601373Z name: td_results 2024-05-19T18:17:17.0601692Z s3-bucket: gha-artifacts 2024-05-19T18:17:17.0602045Z region: us-east-1 2024-05-19T18:17:17.0602346Z env: 2024-05-19T18:17:17.0602614Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:17:17.0603088Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:17:17.0603577Z ##[endgroup] 2024-05-19T18:17:17.4669608Z (node:28730) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-05-19T18:17:17.4670421Z 2024-05-19T18:17:17.4670692Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-05-19T18:17:17.4671401Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-05-19T18:17:17.4672226Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-05-19T18:17:17.5356527Z Found 0 objects with prefix pytorch/pytorch/9148947323/td_results/ 2024-05-19T18:17:17.5363489Z Artifact download has finished successfully 2024-05-19T18:17:17.5500712Z ##[group]Run mkdir -p .additional_ci_files 2024-05-19T18:17:17.5501200Z mkdir -p .additional_ci_files 2024-05-19T18:17:17.5501752Z mv td_results.json .additional_ci_files/td_results.json 2024-05-19T18:17:17.5509500Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:17:17.5509992Z env: 2024-05-19T18:17:17.5510279Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:17:17.5510755Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:17:17.5511236Z ##[endgroup] 2024-05-19T18:17:17.5544893Z mv: cannot stat 'td_results.json': No such file or directory 2024-05-19T18:17:17.5549304Z ##[error]Process completed with exit code 1. 2024-05-19T18:17:17.5580676Z ##[group]Run .github/scripts/parse_ref.py 2024-05-19T18:17:17.5581157Z .github/scripts/parse_ref.py 2024-05-19T18:17:17.5588068Z shell: /usr/bin/bash -e {0} 2024-05-19T18:17:17.5588431Z env: 2024-05-19T18:17:17.5588720Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:17:17.5589188Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:17:17.5589669Z ##[endgroup] 2024-05-19T18:17:17.5819155Z Prepare all required actions 2024-05-19T18:17:17.5856623Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-05-19T18:17:17.5857184Z with: 2024-05-19T18:17:17.5857751Z github-token: *** 2024-05-19T18:17:17.5858064Z env: 2024-05-19T18:17:17.5858335Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:17:17.5858796Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:17:17.5859274Z ##[endgroup] 2024-05-19T18:17:17.5874845Z ##[group]Run set -eux 2024-05-19T18:17:17.5875185Z set -eux 2024-05-19T18:17:17.5875773Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-05-19T18:17:17.5883801Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:17:17.5884298Z env: 2024-05-19T18:17:17.5884725Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:17:17.5885203Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:17:17.5885873Z GITHUB_TOKEN: *** 2024-05-19T18:17:17.5886193Z ##[endgroup] 2024-05-19T18:17:17.5905563Z + python3 .github/scripts/get_workflow_job_id.py 9148947323 i-038631eb9562efe08 2024-05-19T18:17:18.8627143Z setting job-id=25153197291 2024-05-19T18:17:18.8627967Z setting job-name=cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:17:18.8820648Z Prepare all required actions 2024-05-19T18:17:18.8821085Z Getting action download info 2024-05-19T18:17:18.9856190Z ##[group]Run ./.github/actions/filter-test-configs 2024-05-19T18:17:18.9856655Z with: 2024-05-19T18:17:18.9857113Z github-token: *** 2024-05-19T18:17:18.9864747Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-05-19T18:17:18.9873315Z job-name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:17:18.9873995Z env: 2024-05-19T18:17:18.9874275Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:17:18.9874727Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:17:18.9875205Z ##[endgroup] 2024-05-19T18:17:18.9915761Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-05-19T18:17:18.9916322Z with: 2024-05-19T18:17:18.9916596Z shell: bash 2024-05-19T18:17:18.9916906Z timeout_minutes: 10 2024-05-19T18:17:18.9917238Z max_attempts: 5 2024-05-19T18:17:18.9917562Z retry_wait_seconds: 30 2024-05-19T18:17:18.9918432Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore python3 -m pip install requests==2.26.0 pyyaml==6.0.1 2024-05-19T18:17:18.9919268Z polling_interval_seconds: 1 2024-05-19T18:17:18.9919652Z warning_on_retry: true 2024-05-19T18:17:18.9920011Z continue_on_error: false 2024-05-19T18:17:18.9920346Z env: 2024-05-19T18:17:18.9920629Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:17:18.9921093Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:17:18.9921735Z GITHUB_TOKEN: *** 2024-05-19T18:17:18.9922046Z ##[endgroup] 2024-05-19T18:17:19.0420553Z + python3 -m pip install requests==2.26.0 pyyaml==6.0.1 2024-05-19T18:17:19.2557156Z Defaulting to user installation because normal site-packages is not writeable 2024-05-19T18:17:19.3797848Z Collecting requests==2.26.0 2024-05-19T18:17:19.3982717Z Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB) 2024-05-19T18:17:19.5548260Z Collecting pyyaml==6.0.1 2024-05-19T18:17:19.5618173Z Downloading PyYAML-6.0.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (670 kB) 2024-05-19T18:17:19.6198255Z Collecting idna<4,>=2.5; python_version >= "3" 2024-05-19T18:17:19.6244157Z Downloading idna-3.7-py3-none-any.whl (66 kB) 2024-05-19T18:17:19.6871921Z Collecting certifi>=2017.4.17 2024-05-19T18:17:19.6916724Z Downloading certifi-2024.2.2-py3-none-any.whl (163 kB) 2024-05-19T18:17:19.8136847Z Collecting urllib3<1.27,>=1.21.1 2024-05-19T18:17:19.8179612Z Downloading urllib3-1.26.18-py2.py3-none-any.whl (143 kB) 2024-05-19T18:17:20.1551918Z Collecting charset-normalizer~=2.0.0; python_version >= "3" 2024-05-19T18:17:20.1595122Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-05-19T18:17:20.2316217Z Installing collected packages: idna, certifi, urllib3, charset-normalizer, requests, pyyaml 2024-05-19T18:17:20.3932379Z WARNING: The script normalizer is installed in '/home/ec2-user/.local/bin' which is not on PATH. 2024-05-19T18:17:20.3933585Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-05-19T18:17:20.4813758Z Successfully installed certifi-2024.2.2 charset-normalizer-2.0.12 idna-3.7 pyyaml-6.0.1 requests-2.26.0 urllib3-1.26.18 2024-05-19T18:17:21.0429783Z Command completed after 1 attempt(s). 2024-05-19T18:17:21.0469504Z ##[group]Run set -x 2024-05-19T18:17:21.0470075Z set -x 2024-05-19T18:17:21.0470389Z  2024-05-19T18:17:21.0470946Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-05-19T18:17:21.0471628Z # in runner workspace 2024-05-19T18:17:21.0472159Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-05-19T18:17:21.0479922Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:17:21.0480420Z env: 2024-05-19T18:17:21.0480703Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:17:21.0481151Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:17:21.0481637Z ##[endgroup] 2024-05-19T18:17:21.0502584Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-05-19T18:17:21.0698032Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-05-19T18:17:21.0698582Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-05-19T18:17:21.0699058Z echo "Job name: ${JOB_NAME}" 2024-05-19T18:17:21.0699458Z  2024-05-19T18:17:21.0699999Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-05-19T18:17:21.0700682Z # in runner workspace 2024-05-19T18:17:21.0701256Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-05-19T18:17:21.0701905Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-05-19T18:17:21.0702369Z  --job-name "${JOB_NAME}" \ 2024-05-19T18:17:21.0710301Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-05-19T18:17:21.0718489Z  --selected-test-configs "" \ 2024-05-19T18:17:21.0718945Z  --pr-number "${PR_NUMBER}" \ 2024-05-19T18:17:21.0719372Z  --tag "${TAG}" \ 2024-05-19T18:17:21.0719787Z  --event-name "${EVENT_NAME}" \ 2024-05-19T18:17:21.0720244Z  --schedule "${SCHEDULE}" \ 2024-05-19T18:17:21.0720688Z  --branch "${HEAD_BRANCH}" 2024-05-19T18:17:21.0728258Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:17:21.0728762Z env: 2024-05-19T18:17:21.0729054Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:17:21.0729518Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:17:21.0730193Z GITHUB_TOKEN: *** 2024-05-19T18:17:21.0730816Z JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:17:21.0731514Z PR_NUMBER: 2024-05-19T18:17:21.0731836Z TAG: ciflow/inductor/121956 2024-05-19T18:17:21.0732201Z EVENT_NAME: push 2024-05-19T18:17:21.0732516Z SCHEDULE: 2024-05-19T18:17:21.0732812Z HEAD_BRANCH: 2024-05-19T18:17:21.0733108Z ##[endgroup] 2024-05-19T18:17:21.0752148Z Workflow: inductor 2024-05-19T18:17:21.0753796Z Job name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:17:21.3028889Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-05-19T18:17:21.4256224Z ##[group]Run echo "Filtered matrix:" 2024-05-19T18:17:21.4256685Z echo "Filtered matrix:" 2024-05-19T18:17:21.4264484Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-05-19T18:17:21.4272529Z  2024-05-19T18:17:21.4272909Z echo 2024-05-19T18:17:21.4273293Z echo "Is the current job unstable? False" 2024-05-19T18:17:21.4273752Z  2024-05-19T18:17:21.4274023Z echo 2024-05-19T18:17:21.4274386Z echo "Is keep-going label set? False" 2024-05-19T18:17:21.4274828Z  2024-05-19T18:17:21.4275096Z echo 2024-05-19T18:17:21.4275414Z echo "Renabled issues? " 2024-05-19T18:17:21.4283029Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:17:21.4283534Z env: 2024-05-19T18:17:21.4283814Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:17:21.4284273Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:17:21.4284900Z ##[endgroup] 2024-05-19T18:17:21.4304286Z Filtered matrix: 2024-05-19T18:17:21.4314390Z {include: [{config: inductor, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-05-19T18:17:21.4322112Z 2024-05-19T18:17:21.4322265Z Is the current job unstable? False 2024-05-19T18:17:21.4322551Z 2024-05-19T18:17:21.4322833Z Is keep-going label set? False 2024-05-19T18:17:21.4323081Z 2024-05-19T18:17:21.4323350Z Renabled issues? 2024-05-19T18:17:21.4364803Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-05-19T18:17:21.4365546Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-05-19T18:17:21.4372441Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:17:21.4372937Z env: 2024-05-19T18:17:21.4373217Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:17:21.4373669Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:17:21.4374159Z JOB_TIMEOUT: 240 2024-05-19T18:17:21.4374466Z ##[endgroup] 2024-05-19T18:17:21.4474299Z ##[group]Run set -x 2024-05-19T18:17:21.4474706Z set -x 2024-05-19T18:17:21.4475017Z  2024-05-19T18:17:21.4475375Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-05-19T18:17:21.4475927Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-05-19T18:17:21.4476484Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-05-19T18:17:21.4477007Z  TEST_COMMAND=.ci/onnx/test.sh 2024-05-19T18:17:21.4477433Z else 2024-05-19T18:17:21.4477785Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-05-19T18:17:21.4478203Z fi 2024-05-19T18:17:21.4478487Z  2024-05-19T18:17:21.4478961Z # detached container should get cleaned up by teardown_ec2_linux 2024-05-19T18:17:21.4479721Z # TODO: Stop building test binaries as part of the build phase 2024-05-19T18:17:21.4480393Z # Used for GPU_FLAG since that doesn't play nice 2024-05-19T18:17:21.4480990Z # shellcheck disable=SC2086,SC2090 2024-05-19T18:17:21.4481460Z container_name=$(docker run \ 2024-05-19T18:17:21.4481879Z  ${GPU_FLAG:-} \ 2024-05-19T18:17:21.4482256Z  -e BUILD_ENVIRONMENT \ 2024-05-19T18:17:21.4482661Z  -e PR_NUMBER \ 2024-05-19T18:17:21.4483020Z  -e GITHUB_ACTIONS \ 2024-05-19T18:17:21.4483415Z  -e GITHUB_REPOSITORY \ 2024-05-19T18:17:21.4483825Z  -e GITHUB_WORKFLOW \ 2024-05-19T18:17:21.4484219Z  -e GITHUB_JOB \ 2024-05-19T18:17:21.4484583Z  -e GITHUB_RUN_ID \ 2024-05-19T18:17:21.4485256Z  -e GITHUB_RUN_NUMBER \ 2024-05-19T18:17:21.4485665Z  -e GITHUB_RUN_ATTEMPT \ 2024-05-19T18:17:21.4486061Z  -e JOB_ID \ 2024-05-19T18:17:21.4486402Z  -e JOB_NAME \ 2024-05-19T18:17:21.4486751Z  -e BASE_SHA \ 2024-05-19T18:17:21.4487100Z  -e BRANCH \ 2024-05-19T18:17:21.4487433Z  -e SHA1 \ 2024-05-19T18:17:21.4487774Z  -e AWS_DEFAULT_REGION \ 2024-05-19T18:17:21.4488175Z  -e IN_WHEEL_TEST \ 2024-05-19T18:17:21.4488553Z  -e SHARD_NUMBER \ 2024-05-19T18:17:21.4488915Z  -e TEST_CONFIG \ 2024-05-19T18:17:21.4489286Z  -e NUM_TEST_SHARDS \ 2024-05-19T18:17:21.4489679Z  -e REENABLED_ISSUES \ 2024-05-19T18:17:21.4490095Z  -e CONTINUE_THROUGH_ERROR \ 2024-05-19T18:17:21.4490515Z  -e VERBOSE_TEST_LOGS \ 2024-05-19T18:17:21.4490922Z  -e NO_TEST_TIMEOUT \ 2024-05-19T18:17:21.4491309Z  -e NO_TD \ 2024-05-19T18:17:21.4491660Z  -e TD_DISTRIBUTED \ 2024-05-19T18:17:21.4492037Z  -e PR_LABELS \ 2024-05-19T18:17:21.4492449Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-05-19T18:17:21.4492909Z  -e SCCACHE_BUCKET \ 2024-05-19T18:17:21.4493305Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-05-19T18:17:21.4493709Z  -e XLA_CUDA \ 2024-05-19T18:17:21.4494115Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-05-19T18:17:21.4494626Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-05-19T18:17:21.4495145Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-05-19T18:17:21.4495684Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-05-19T18:17:21.4496171Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-05-19T18:17:21.4496591Z  -e DASHBOARD_TAG \ 2024-05-19T18:17:21.4497072Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-05-19T18:17:21.4497737Z  --security-opt seccomp=unconfined \ 2024-05-19T18:17:21.4498206Z  --cap-add=SYS_PTRACE \ 2024-05-19T18:17:21.4498604Z  --ipc=host \ 2024-05-19T18:17:21.4498971Z  --shm-size="${SHM_SIZE}" \ 2024-05-19T18:17:21.4499368Z  --tty \ 2024-05-19T18:17:21.4499688Z  --detach \ 2024-05-19T18:17:21.4500058Z  --name="${container_name}" \ 2024-05-19T18:17:21.4500476Z  --user jenkins \ 2024-05-19T18:17:21.4501039Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-05-19T18:17:21.4501605Z  -w /var/lib/jenkins/workspace \ 2024-05-19T18:17:21.4502050Z  "${DOCKER_IMAGE}" 2024-05-19T18:17:21.4502401Z ) 2024-05-19T18:17:21.4502815Z # Propagate download.pytorch.org IP to container 2024-05-19T18:17:21.4503736Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-05-19T18:17:21.4504715Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-05-19T18:17:21.4505626Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-05-19T18:17:21.4513175Z shell: /usr/bin/bash -e {0} 2024-05-19T18:17:21.4513536Z env: 2024-05-19T18:17:21.4513819Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:17:21.4514279Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:17:21.4514886Z BUILD_ENVIRONMENT: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:17:21.4515391Z PR_NUMBER: 2024-05-19T18:17:21.4515718Z GITHUB_REPOSITORY: pytorch/pytorch 2024-05-19T18:17:21.4516136Z GITHUB_WORKFLOW: inductor 2024-05-19T18:17:21.4516485Z GITHUB_JOB: test 2024-05-19T18:17:21.4516812Z GITHUB_RUN_ID: 9148947323 2024-05-19T18:17:21.4517189Z GITHUB_RUN_NUMBER: 69546 2024-05-19T18:17:21.4517533Z GITHUB_RUN_ATTEMPT: 1 2024-05-19T18:17:21.4517873Z JOB_ID: 25153197291 2024-05-19T18:17:21.4518501Z JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:17:21.4519181Z BRANCH: 2024-05-19T18:17:21.4519525Z SHA1: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:17:21.4520060Z BASE_SHA: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:17:21.4520548Z TEST_CONFIG: inductor_timm 2024-05-19T18:17:21.4520914Z SHARD_NUMBER: 1 2024-05-19T18:17:21.4521220Z NUM_TEST_SHARDS: 2 2024-05-19T18:17:21.4521548Z REENABLED_ISSUES: 2024-05-19T18:17:21.4521896Z CONTINUE_THROUGH_ERROR: False 2024-05-19T18:17:21.4522291Z VERBOSE_TEST_LOGS: False 2024-05-19T18:17:21.4522651Z NO_TEST_TIMEOUT: False 2024-05-19T18:17:21.4522995Z NO_TD: False 2024-05-19T18:17:21.4523304Z TD_DISTRIBUTED: False 2024-05-19T18:17:21.4523723Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-05-19T18:17:21.4524214Z SCCACHE_S3_KEY_PREFIX: inductor 2024-05-19T18:17:21.4524758Z SHM_SIZE: 2g 2024-05-19T18:17:21.4525907Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:17:21.4527077Z XLA_CUDA: 2024-05-19T18:17:21.4527556Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-05-19T18:17:21.4528178Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-05-19T18:17:21.4528622Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-05-19T18:17:21.4529043Z DASHBOARD_TAG: 2024-05-19T18:17:21.4529539Z HUGGING_FACE_HUB_TOKEN: *** 2024-05-19T18:17:21.4529908Z ##[endgroup] 2024-05-19T18:17:21.4548557Z + [[ inductor_timm == \m\u\l\t\i\g\p\u ]] 2024-05-19T18:17:21.4549422Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *onnx* ]] 2024-05-19T18:17:21.4549935Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-05-19T18:17:21.4556526Z +++ nproc --ignore=2 2024-05-19T18:17:21.4577808Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9148947323 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:17:32.8132157Z + container_name=8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T18:17:32.8133157Z + grep download.pytorch.org /etc/hosts 2024-05-19T18:17:32.8135401Z + docker exec -i 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 sudo bash -c '/bin/cat >> /etc/hosts' 2024-05-19T18:17:32.8884395Z + echo DOCKER_CONTAINER_ID=8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T18:17:32.8887533Z ++ echo dist/torch-2.4.0a0+gitc8c7ddf-cp310-cp310-linux_x86_64.whl 2024-05-19T18:17:32.8889489Z + docker exec -t 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 sh -c 'pip install dist/torch-2.4.0a0+gitc8c7ddf-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-05-19T18:17:33.2784007Z Processing ./dist/torch-2.4.0a0+gitc8c7ddf-cp310-cp310-linux_x86_64.whl (from torch==2.4.0a0+gitc8c7ddf) 2024-05-19T18:17:33.5899835Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (3.13.1) 2024-05-19T18:17:33.5903874Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (4.11.0) 2024-05-19T18:17:33.5906464Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (1.12) 2024-05-19T18:17:33.5910373Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (2.8.8) 2024-05-19T18:17:33.5913596Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (3.1.4) 2024-05-19T18:17:33.5917729Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (2024.2.0) 2024-05-19T18:17:33.5944405Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (3.3.0) 2024-05-19T18:17:33.6001558Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (1.21.2) 2024-05-19T18:17:33.6429044Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (2.1.5) 2024-05-19T18:17:33.6580693Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (1.3.0) 2024-05-19T18:17:34.9403298Z Installing collected packages: torch 2024-05-19T18:17:43.5449753Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-05-19T18:17:43.5451420Z timm 0.9.7 requires torchvision, which is not installed. 2024-05-19T18:17:43.5452447Z Successfully installed torch-2.4.0a0+gitc8c7ddf 2024-05-19T18:17:43.6205074Z ++ dirname .ci/pytorch/test.sh 2024-05-19T18:17:43.6209979Z + source .ci/pytorch/common.sh 2024-05-19T18:17:43.6212481Z +++ dirname .ci/pytorch/common.sh 2024-05-19T18:17:43.6218661Z ++ source .ci/pytorch/common_utils.sh 2024-05-19T18:17:43.6220093Z +++ declare -f -t trap_add 2024-05-19T18:17:43.6225174Z ++ set -ex 2024-05-19T18:17:43.6225654Z ++ [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]] 2024-05-19T18:17:43.6226159Z ++ BUILD_TEST_LIBTORCH=0 2024-05-19T18:17:43.6227363Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *rocm* ]] 2024-05-19T18:17:43.6228036Z ++ stat -c %u /var/lib/jenkins/workspace 2024-05-19T18:17:43.6235831Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-05-19T18:17:43.6236441Z + trap_add cleanup_workspace EXIT 2024-05-19T18:17:43.6236969Z + trap_add_cmd=cleanup_workspace 2024-05-19T18:17:43.6237336Z + shift 2024-05-19T18:17:43.6237607Z + for trap_add_name in "$@" 2024-05-19T18:17:43.6240391Z +++ trap -p EXIT 2024-05-19T18:17:43.6242351Z ++ eval 'extract_trap_cmd ' 2024-05-19T18:17:43.6242791Z +++ extract_trap_cmd 2024-05-19T18:17:43.6243153Z +++ printf '%s\n' '' 2024-05-19T18:17:43.6243642Z ++ printf '%s\n' cleanup_workspace 2024-05-19T18:17:43.6244316Z + trap -- ' 2024-05-19T18:17:43.6244888Z cleanup_workspace' EXIT 2024-05-19T18:17:43.6245356Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-05-19T18:17:44.0366557Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-05-19T18:17:44.0380202Z + echo 'Environment variables:' 2024-05-19T18:17:44.0380635Z Environment variables: 2024-05-19T18:17:44.0380951Z + env 2024-05-19T18:17:44.0386457Z INSTALLED_DB=yes 2024-05-19T18:17:44.0386980Z NV_LIBCUBLAS_VERSION=12.4.2.65-1 2024-05-19T18:17:44.0387401Z NVIDIA_VISIBLE_DEVICES=all 2024-05-19T18:17:44.0387940Z NV_NVML_DEV_VERSION=12.4.99-1 2024-05-19T18:17:44.0388554Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T18:17:44.0389226Z CONTINUE_THROUGH_ERROR=False 2024-05-19T18:17:44.0389716Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.20.5-1+cuda12.4 2024-05-19T18:17:44.0390218Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.20.5-1 2024-05-19T18:17:44.0390781Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:17:44.0391321Z HOSTNAME=8e0cfbfda7b9 2024-05-19T18:17:44.0392413Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_0c804022-db42-4382-a705-81d827836dcd 2024-05-19T18:17:44.0393486Z GITHUB_ACTION=__self 2024-05-19T18:17:44.0393934Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-05-19T18:17:44.0400028Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536 2024-05-19T18:17:44.0405983Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.2.65-1 2024-05-19T18:17:44.0406736Z NV_NVTX_VERSION=12.4.99-1 2024-05-19T18:17:44.0407090Z GITHUB_RUN_NUMBER=69546 2024-05-19T18:17:44.0407493Z TEST_CONFIG=inductor_timm 2024-05-19T18:17:44.0407869Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-05-19T18:17:44.0408336Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-05-19T18:17:44.0408800Z NV_CUDA_CUDART_DEV_VERSION=12.4.99-1 2024-05-19T18:17:44.0409233Z NV_LIBCUSPARSE_VERSION=12.3.0.142-1 2024-05-19T18:17:44.0409650Z NV_LIBNPP_VERSION=12.2.5.2-1 2024-05-19T18:17:44.0410083Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-05-19T18:17:44.0410677Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-05-19T18:17:44.0411147Z GITHUB_REF_TYPE=tag 2024-05-19T18:17:44.0411469Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-05-19T18:17:44.0411841Z NCCL_VERSION=2.20.5-1 2024-05-19T18:17:44.0412234Z BASE_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:17:44.0412683Z XLA_CUDA= 2024-05-19T18:17:44.0413157Z HUGGING_FACE_HUB_TOKEN=*** 2024-05-19T18:17:44.0414661Z *** 2024-05-19T18:17:44.0414970Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-05-19T18:17:44.0415388Z GITHUB_REPOSITORY_ID=65600975 2024-05-19T18:17:44.0415749Z GITHUB_ACTIONS=true 2024-05-19T18:17:44.0416091Z NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:17:44.0416606Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.99-1 2024-05-19T18:17:44.0417215Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.2-1 2024-05-19T18:17:44.0417687Z SHA1=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:17:44.0418207Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-05-19T18:17:44.0418827Z GITHUB_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:17:44.0419680Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/121956 2024-05-19T18:17:44.0420382Z UCC_HOME=/usr 2024-05-19T18:17:44.0420725Z NV_LIBCUBLAS_DEV_VERSION=12.4.2.65-1 2024-05-19T18:17:44.0421136Z VERBOSE_TEST_LOGS=False 2024-05-19T18:17:44.0421476Z NVIDIA_PRODUCT_NAME=CUDA 2024-05-19T18:17:44.0421930Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4 2024-05-19T18:17:44.0422476Z GITHUB_REF=refs/tags/ciflow/inductor/121956 2024-05-19T18:17:44.0422948Z NV_CUDA_CUDART_VERSION=12.4.99-1 2024-05-19T18:17:44.0423429Z SHARD_NUMBER=1 2024-05-19T18:17:44.0423742Z GITHUB_REF_PROTECTED=false 2024-05-19T18:17:44.0424144Z HOME=/var/lib/jenkins 2024-05-19T18:17:44.0424517Z GITHUB_API_URL=https://api.github.com 2024-05-19T18:17:44.0424946Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-05-19T18:17:44.0425609Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-05-19T18:17:44.0426093Z SCCACHE_S3_KEY_PREFIX=inductor 2024-05-19T18:17:44.0426461Z CUDA_VERSION=12.4.0 2024-05-19T18:17:44.0426888Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.2.65-1 2024-05-19T18:17:44.0427327Z NUM_TEST_SHARDS=2 2024-05-19T18:17:44.0427630Z UCX_HOME=/usr 2024-05-19T18:17:44.0428125Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.0-1 2024-05-19T18:17:44.0429168Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_0c804022-db42-4382-a705-81d827836dcd 2024-05-19T18:17:44.0430387Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:17:44.0431533Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_0c804022-db42-4382-a705-81d827836dcd 2024-05-19T18:17:44.0432619Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-05-19T18:17:44.0433433Z GITHUB_EVENT_NAME=push 2024-05-19T18:17:44.0433806Z DASHBOARD_TAG= 2024-05-19T18:17:44.0434120Z GITHUB_RUN_ID=9148947323 2024-05-19T18:17:44.0434570Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.2-1 2024-05-19T18:17:44.0435192Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4 2024-05-19T18:17:44.0436237Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_0c804022-db42-4382-a705-81d827836dcd 2024-05-19T18:17:44.0437257Z GITHUB_ACTOR=pytorch-bot[bot] 2024-05-19T18:17:44.0437697Z NV_LIBNPP_DEV_VERSION=12.2.5.2-1 2024-05-19T18:17:44.0438121Z PR_NUMBER= 2024-05-19T18:17:44.0438402Z GITHUB_RUN_ATTEMPT=1 2024-05-19T18:17:44.0438915Z ANACONDA_PYTHON_VERSION=3.10 2024-05-19T18:17:44.0439405Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-05-19T18:17:44.0439864Z TERM=xterm 2024-05-19T18:17:44.0440205Z NV_LIBCUSPARSE_DEV_VERSION=12.3.0.142-1 2024-05-19T18:17:44.0440610Z INSTALLED_VISION=yes 2024-05-19T18:17:44.0440974Z BRANCH= 2024-05-19T18:17:44.0441265Z OPENSSL_ROOT_DIR=/opt/openssl 2024-05-19T18:17:44.0441664Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-05-19T18:17:44.0442127Z CUDA_PATH=/usr/local/cuda 2024-05-19T18:17:44.0443045Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-05-19T18:17:44.0443786Z GITHUB_SERVER_URL=https://github.com 2024-05-19T18:17:44.0444324Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-05-19T18:17:44.0447913Z REENABLED_ISSUES= 2024-05-19T18:17:44.0448221Z SHLVL=1 2024-05-19T18:17:44.0448617Z MAX_JOBS=14 2024-05-19T18:17:44.0448949Z NV_CUDA_LIB_VERSION=12.4.0-1 2024-05-19T18:17:44.0449295Z NVARCH=x86_64 2024-05-19T18:17:44.0449823Z GITHUB_ACTOR_ID=54816060 2024-05-19T18:17:44.0450854Z GITHUB_WORKFLOW_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:17:44.0451565Z GITHUB_REF_NAME=ciflow/inductor/121956 2024-05-19T18:17:44.0452052Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4 2024-05-19T18:17:44.0452700Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-05-19T18:17:44.0453258Z GITHUB_JOB=test 2024-05-19T18:17:44.0453640Z NV_LIBNCCL_PACKAGE=libnccl2=2.20.5-1+cuda12.4 2024-05-19T18:17:44.0454193Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-05-19T18:17:44.0454904Z NO_TEST_TIMEOUT=False 2024-05-19T18:17:44.0455234Z TD_DISTRIBUTED=False 2024-05-19T18:17:44.0455603Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.0-1 2024-05-19T18:17:44.0456033Z GITHUB_REPOSITORY=pytorch/pytorch 2024-05-19T18:17:44.0457122Z NV_NVPROF_VERSION=12.4.99-1 2024-05-19T18:17:44.0457483Z GITHUB_RETENTION_DAYS=90 2024-05-19T18:17:44.0457834Z OPENSSL_DIR=/opt/openssl 2024-05-19T18:17:44.0458317Z GITHUB_ACTION_REPOSITORY= 2024-05-19T18:17:44.0459666Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-05-19T18:17:44.0460983Z GITHUB_BASE_REF= 2024-05-19T18:17:44.0461500Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-05-19T18:17:44.0461871Z CI=true 2024-05-19T18:17:44.0462200Z NV_LIBNCCL_PACKAGE_VERSION=2.20.5-1 2024-05-19T18:17:44.0462611Z GITHUB_REPOSITORY_OWNER=pytorch 2024-05-19T18:17:44.0462985Z JOB_ID=25153197291 2024-05-19T18:17:44.0463784Z INSTALLED_PROTOBUF=yes 2024-05-19T18:17:44.0464121Z GITHUB_HEAD_REF= 2024-05-19T18:17:44.0464587Z GITHUB_ACTION_REF= 2024-05-19T18:17:44.0465025Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-05-19T18:17:44.0465626Z GITHUB_WORKFLOW=inductor 2024-05-19T18:17:44.0465984Z DEBIAN_FRONTEND=noninteractive 2024-05-19T18:17:44.0466873Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_0c804022-db42-4382-a705-81d827836dcd 2024-05-19T18:17:44.0467679Z NO_TD=False 2024-05-19T18:17:44.0467984Z SKIP_SCCACHE_INITIALIZATION=1 2024-05-19T18:17:44.0468341Z _=/usr/bin/env 2024-05-19T18:17:44.0468812Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-05-19T18:17:44.0536134Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-05-19T18:17:44.0537049Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-05-19T18:17:44.0537889Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-05-19T18:17:44.0538764Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-05-19T18:17:44.0539368Z + BUILD_DIR=build 2024-05-19T18:17:44.0539697Z + BUILD_RENAMED_DIR=build_renamed 2024-05-19T18:17:44.0540078Z + BUILD_BIN_DIR=build/bin 2024-05-19T18:17:44.0540417Z + SHARD_NUMBER=1 2024-05-19T18:17:44.0540722Z + NUM_TEST_SHARDS=2 2024-05-19T18:17:44.0541039Z + export VALGRIND=ON 2024-05-19T18:17:44.0541349Z + VALGRIND=ON 2024-05-19T18:17:44.0541998Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *clang9* ]] 2024-05-19T18:17:44.0542493Z + [[ 0 == \1 ]] 2024-05-19T18:17:44.0542795Z + [[ False == \1 ]] 2024-05-19T18:17:44.0543278Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *bazel* ]] 2024-05-19T18:17:44.0543801Z ++ realpath build/custom_test_artifacts 2024-05-19T18:17:44.0547278Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-05-19T18:17:44.0547955Z + [[ -n '' ]] 2024-05-19T18:17:44.0548296Z + echo 'Environment variables' 2024-05-19T18:17:44.0548784Z Environment variables 2024-05-19T18:17:44.0549106Z + env 2024-05-19T18:17:44.0552566Z INSTALLED_DB=yes 2024-05-19T18:17:44.0553133Z NV_LIBCUBLAS_VERSION=12.4.2.65-1 2024-05-19T18:17:44.0553671Z NVIDIA_VISIBLE_DEVICES=all 2024-05-19T18:17:44.0554092Z NV_NVML_DEV_VERSION=12.4.99-1 2024-05-19T18:17:44.0554879Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T18:17:44.0555570Z CONTINUE_THROUGH_ERROR=False 2024-05-19T18:17:44.0556071Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.20.5-1+cuda12.4 2024-05-19T18:17:44.0556587Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.20.5-1 2024-05-19T18:17:44.0557230Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:17:44.0557839Z HOSTNAME=8e0cfbfda7b9 2024-05-19T18:17:44.0558802Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_0c804022-db42-4382-a705-81d827836dcd 2024-05-19T18:17:44.0559826Z GITHUB_ACTION=__self 2024-05-19T18:17:44.0560275Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-05-19T18:17:44.0566573Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536 2024-05-19T18:17:44.0572295Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.2.65-1 2024-05-19T18:17:44.0572792Z NV_NVTX_VERSION=12.4.99-1 2024-05-19T18:17:44.0573139Z GITHUB_RUN_NUMBER=69546 2024-05-19T18:17:44.0573549Z TEST_CONFIG=inductor_timm 2024-05-19T18:17:44.0573915Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-05-19T18:17:44.0574382Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-05-19T18:17:44.0574836Z NV_CUDA_CUDART_DEV_VERSION=12.4.99-1 2024-05-19T18:17:44.0575278Z NV_LIBCUSPARSE_VERSION=12.3.0.142-1 2024-05-19T18:17:44.0575698Z NV_LIBNPP_VERSION=12.2.5.2-1 2024-05-19T18:17:44.0576123Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-05-19T18:17:44.0576615Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-05-19T18:17:44.0577074Z GITHUB_REF_TYPE=tag 2024-05-19T18:17:44.0577398Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-05-19T18:17:44.0577764Z NCCL_VERSION=2.20.5-1 2024-05-19T18:17:44.0578165Z BASE_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:17:44.0578615Z XLA_CUDA= 2024-05-19T18:17:44.0579087Z HUGGING_FACE_HUB_TOKEN=*** 2024-05-19T18:17:44.0579476Z *** 2024-05-19T18:17:44.0579758Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-05-19T18:17:44.0580158Z GITHUB_REPOSITORY_ID=65600975 2024-05-19T18:17:44.0580520Z GITHUB_ACTIONS=true 2024-05-19T18:17:44.0580847Z NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:17:44.0581515Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.99-1 2024-05-19T18:17:44.0582043Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.2-1 2024-05-19T18:17:44.0582519Z SHA1=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:17:44.0583022Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-05-19T18:17:44.0583511Z GITHUB_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:17:44.0584310Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/121956 2024-05-19T18:17:44.0585012Z UCC_HOME=/usr 2024-05-19T18:17:44.0585436Z NV_LIBCUBLAS_DEV_VERSION=12.4.2.65-1 2024-05-19T18:17:44.0585848Z VERBOSE_TEST_LOGS=False 2024-05-19T18:17:44.0586197Z NVIDIA_PRODUCT_NAME=CUDA 2024-05-19T18:17:44.0586648Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4 2024-05-19T18:17:44.0587152Z GITHUB_REF=refs/tags/ciflow/inductor/121956 2024-05-19T18:17:44.0587617Z NV_CUDA_CUDART_VERSION=12.4.99-1 2024-05-19T18:17:44.0587991Z SHARD_NUMBER=1 2024-05-19T18:17:44.0588298Z GITHUB_REF_PROTECTED=false 2024-05-19T18:17:44.0588660Z HOME=/var/lib/jenkins 2024-05-19T18:17:44.0589030Z GITHUB_API_URL=https://api.github.com 2024-05-19T18:17:44.0589467Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-05-19T18:17:44.0589945Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-05-19T18:17:44.0590425Z SCCACHE_S3_KEY_PREFIX=inductor 2024-05-19T18:17:44.0590797Z CUDA_VERSION=12.4.0 2024-05-19T18:17:44.0591211Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.2.65-1 2024-05-19T18:17:44.0591656Z NUM_TEST_SHARDS=2 2024-05-19T18:17:44.0591960Z UCX_HOME=/usr 2024-05-19T18:17:44.0592476Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.0-1 2024-05-19T18:17:44.0593620Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_0c804022-db42-4382-a705-81d827836dcd 2024-05-19T18:17:44.0594781Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:17:44.0595940Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_0c804022-db42-4382-a705-81d827836dcd 2024-05-19T18:17:44.0597035Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-05-19T18:17:44.0597705Z GITHUB_EVENT_NAME=push 2024-05-19T18:17:44.0598042Z DASHBOARD_TAG= 2024-05-19T18:17:44.0598347Z GITHUB_RUN_ID=9148947323 2024-05-19T18:17:44.0598803Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.2-1 2024-05-19T18:17:44.0599331Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4 2024-05-19T18:17:44.0600315Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_0c804022-db42-4382-a705-81d827836dcd 2024-05-19T18:17:44.0601209Z GITHUB_ACTOR=pytorch-bot[bot] 2024-05-19T18:17:44.0601623Z NV_LIBNPP_DEV_VERSION=12.2.5.2-1 2024-05-19T18:17:44.0601995Z PR_NUMBER= 2024-05-19T18:17:44.0602281Z GITHUB_RUN_ATTEMPT=1 2024-05-19T18:17:44.0602603Z VALGRIND=ON 2024-05-19T18:17:44.0602899Z ANACONDA_PYTHON_VERSION=3.10 2024-05-19T18:17:44.0603348Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-05-19T18:17:44.0603812Z TERM=xterm 2024-05-19T18:17:44.0604161Z NV_LIBCUSPARSE_DEV_VERSION=12.3.0.142-1 2024-05-19T18:17:44.0604570Z INSTALLED_VISION=yes 2024-05-19T18:17:44.0605176Z BRANCH= 2024-05-19T18:17:44.0605461Z OPENSSL_ROOT_DIR=/opt/openssl 2024-05-19T18:17:44.0605861Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-05-19T18:17:44.0606280Z CUDA_PATH=/usr/local/cuda 2024-05-19T18:17:44.0607036Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-05-19T18:17:44.0607785Z GITHUB_SERVER_URL=https://github.com 2024-05-19T18:17:44.0608277Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-05-19T18:17:44.0608740Z REENABLED_ISSUES= 2024-05-19T18:17:44.0609039Z SHLVL=1 2024-05-19T18:17:44.0609303Z MAX_JOBS=14 2024-05-19T18:17:44.0609611Z NV_CUDA_LIB_VERSION=12.4.0-1 2024-05-19T18:17:44.0609970Z NVARCH=x86_64 2024-05-19T18:17:44.0610272Z GITHUB_ACTOR_ID=54816060 2024-05-19T18:17:44.0610744Z GITHUB_WORKFLOW_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:17:44.0611276Z GITHUB_REF_NAME=ciflow/inductor/121956 2024-05-19T18:17:44.0611885Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4 2024-05-19T18:17:44.0612538Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-05-19T18:17:44.0613135Z GITHUB_JOB=test 2024-05-19T18:17:44.0613547Z NV_LIBNCCL_PACKAGE=libnccl2=2.20.5-1+cuda12.4 2024-05-19T18:17:44.0614107Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-05-19T18:17:44.0614620Z NO_TEST_TIMEOUT=False 2024-05-19T18:17:44.0614945Z TD_DISTRIBUTED=False 2024-05-19T18:17:44.0615395Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.0-1 2024-05-19T18:17:44.0615836Z GITHUB_REPOSITORY=pytorch/pytorch 2024-05-19T18:17:44.0616257Z NV_NVPROF_VERSION=12.4.99-1 2024-05-19T18:17:44.0616615Z GITHUB_RETENTION_DAYS=90 2024-05-19T18:17:44.0616967Z OPENSSL_DIR=/opt/openssl 2024-05-19T18:17:44.0617323Z GITHUB_ACTION_REPOSITORY= 2024-05-19T18:17:44.0618358Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-05-19T18:17:44.0619419Z GITHUB_BASE_REF= 2024-05-19T18:17:44.0619748Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-05-19T18:17:44.0620115Z CI=true 2024-05-19T18:17:44.0620429Z NV_LIBNCCL_PACKAGE_VERSION=2.20.5-1 2024-05-19T18:17:44.0620835Z GITHUB_REPOSITORY_OWNER=pytorch 2024-05-19T18:17:44.0621207Z JOB_ID=25153197291 2024-05-19T18:17:44.0621522Z INSTALLED_PROTOBUF=yes 2024-05-19T18:17:44.0621846Z GITHUB_HEAD_REF= 2024-05-19T18:17:44.0622149Z GITHUB_ACTION_REF= 2024-05-19T18:17:44.0622583Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-05-19T18:17:44.0623061Z GITHUB_WORKFLOW=inductor 2024-05-19T18:17:44.0623415Z DEBIAN_FRONTEND=noninteractive 2024-05-19T18:17:44.0624299Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_0c804022-db42-4382-a705-81d827836dcd 2024-05-19T18:17:44.0625101Z NO_TD=False 2024-05-19T18:17:44.0625407Z SKIP_SCCACHE_INITIALIZATION=1 2024-05-19T18:17:44.0625768Z _=/usr/bin/env 2024-05-19T18:17:44.0626105Z + echo 'Testing pytorch' 2024-05-19T18:17:44.0626452Z Testing pytorch 2024-05-19T18:17:44.0626767Z + export LANG=C.UTF-8 2024-05-19T18:17:44.0627106Z + LANG=C.UTF-8 2024-05-19T18:17:44.0627396Z + PR_NUMBER= 2024-05-19T18:17:44.0627723Z + [[ inductor_timm == \d\e\f\a\u\l\t ]] 2024-05-19T18:17:44.0628182Z + [[ inductor_timm == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-05-19T18:17:44.0628651Z + [[ inductor_timm == \s\l\o\w ]] 2024-05-19T18:17:44.0629244Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-05-19T18:17:44.0629937Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *cuda* ]] 2024-05-19T18:17:44.0630480Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-05-19T18:17:44.0630956Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-05-19T18:17:44.0631388Z + [[ inductor_timm == *crossref* ]] 2024-05-19T18:17:44.0631930Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]] 2024-05-19T18:17:44.0632550Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *xpu* ]] 2024-05-19T18:17:44.0633269Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-05-19T18:17:44.0633875Z + pip_install --user ninja==1.10.2 2024-05-19T18:17:44.0634404Z + pip install --progress-bar off --user ninja==1.10.2 2024-05-19T18:17:44.4203772Z Collecting ninja==1.10.2 2024-05-19T18:17:44.4573249Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-05-19T18:17:44.4726631Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-05-19T18:17:45.7502007Z Installing collected packages: ninja 2024-05-19T18:17:45.7568200Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-05-19T18:17:45.7569427Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-05-19T18:17:45.7606204Z Successfully installed ninja-1.10.2 2024-05-19T18:17:45.8408379Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-05-19T18:17:45.8410826Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-05-19T18:17:45.8412411Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]] 2024-05-19T18:17:45.8412922Z + install_tlparse 2024-05-19T18:17:45.8413305Z + pip_install --user tlparse==0.3.7 2024-05-19T18:17:45.8413985Z + pip install --progress-bar off --user tlparse==0.3.7 2024-05-19T18:17:46.1965814Z Collecting tlparse==0.3.7 2024-05-19T18:17:46.2153742Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-05-19T18:17:46.2257529Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-05-19T18:17:47.5448713Z Installing collected packages: tlparse 2024-05-19T18:17:47.5807221Z Successfully installed tlparse-0.3.7 2024-05-19T18:17:47.6603761Z ++ python -m site --user-base 2024-05-19T18:17:47.6757536Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-05-19T18:17:47.6759233Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *asan* ]] 2024-05-19T18:17:47.6759883Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-debug* ]] 2024-05-19T18:17:47.6760561Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-05-19T18:17:47.6761485Z + echo 'We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-05-19T18:17:47.6762605Z We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass 2024-05-19T18:17:47.6763295Z + cd test 2024-05-19T18:17:47.6763805Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-05-19T18:17:49.0490944Z + [[ inductor_timm == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-05-19T18:17:49.0491517Z + [[ inductor_timm == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-05-19T18:17:49.0493510Z + DYNAMO_BENCHMARK_FLAGS=() 2024-05-19T18:17:49.0494035Z + [[ inductor_timm == *dynamo_eager* ]] 2024-05-19T18:17:49.0494476Z + [[ inductor_timm == *aot_eager* ]] 2024-05-19T18:17:49.0495012Z + [[ inductor_timm == *aot_inductor* ]] 2024-05-19T18:17:49.0495522Z + [[ inductor_timm == *inductor* ]] 2024-05-19T18:17:49.0495938Z + [[ inductor_timm != *perf* ]] 2024-05-19T18:17:49.0496515Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-05-19T18:17:49.0496958Z + [[ inductor_timm == *dynamic* ]] 2024-05-19T18:17:49.0497378Z + [[ inductor_timm == *cpu_inductor* ]] 2024-05-19T18:17:49.0497877Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-05-19T18:17:49.0506482Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *tbb* ]] 2024-05-19T18:17:49.0520118Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]] 2024-05-19T18:17:49.0520786Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-05-19T18:17:49.0521418Z + cd test 2024-05-19T18:17:49.0521936Z + python -c 'import torch; print(torch.__config__.show())' 2024-05-19T18:17:50.2888436Z PyTorch built with: 2024-05-19T18:17:50.2889126Z - GCC 9.4 2024-05-19T18:17:50.2889580Z - C++ Version: 201703 2024-05-19T18:17:50.2890458Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-05-19T18:17:50.2891515Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-05-19T18:17:50.2892191Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-05-19T18:17:50.2892693Z - LAPACK is enabled (usually provided by MKL) 2024-05-19T18:17:50.2893160Z - NNPACK is enabled 2024-05-19T18:17:50.2893536Z - CPU capability usage: AVX2 2024-05-19T18:17:50.2893940Z - CUDA Runtime 12.4 2024-05-19T18:17:50.2894451Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-05-19T18:17:50.2895040Z - CuDNN 8.9.7 (built against CUDA 12.2) 2024-05-19T18:17:50.2895743Z - Magma 2.6.1 2024-05-19T18:17:50.2902686Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.4, CUDNN_VERSION=8.9.7, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.4.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-05-19T18:17:50.2909180Z 2024-05-19T18:17:50.5565471Z + cd test 2024-05-19T18:17:50.5566131Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-05-19T18:17:51.6606263Z ATen/Parallel: 2024-05-19T18:17:51.6606787Z at::get_num_threads() : 8 2024-05-19T18:17:51.6607306Z at::get_num_interop_threads() : 16 2024-05-19T18:17:51.6607843Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-05-19T18:17:51.6608324Z omp_get_max_threads() : 8 2024-05-19T18:17:51.6609278Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-05-19T18:17:51.6610064Z mkl_get_max_threads() : 8 2024-05-19T18:17:51.6610680Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-05-19T18:17:51.6611303Z std::thread::hardware_concurrency() : 16 2024-05-19T18:17:51.6611717Z Environment variables: 2024-05-19T18:17:51.6612059Z OMP_NUM_THREADS : [not set] 2024-05-19T18:17:51.6612424Z MKL_NUM_THREADS : [not set] 2024-05-19T18:17:51.6612794Z ATen parallel backend: OpenMP 2024-05-19T18:17:51.6613039Z 2024-05-19T18:17:51.8975553Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]] 2024-05-19T18:17:51.8976318Z + [[ inductor_timm == *backward* ]] 2024-05-19T18:17:51.8976859Z + [[ inductor_timm == *xla* ]] 2024-05-19T18:17:51.8977325Z + [[ inductor_timm == *executorch* ]] 2024-05-19T18:17:51.8977956Z + [[ inductor_timm == \j\i\t\_\l\e\g\a\c\y ]] 2024-05-19T18:17:51.8978652Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]] 2024-05-19T18:17:51.8979287Z + [[ inductor_timm == distributed ]] 2024-05-19T18:17:51.8979712Z + [[ inductor_timm == deploy ]] 2024-05-19T18:17:51.8980153Z + [[ inductor_timm == *inductor_distributed* ]] 2024-05-19T18:17:51.8980714Z + [[ inductor_timm == *inductor-micro-benchmark* ]] 2024-05-19T18:17:51.8981197Z + [[ inductor_timm == *huggingface* ]] 2024-05-19T18:17:51.8981615Z + [[ inductor_timm == *timm* ]] 2024-05-19T18:17:51.8981989Z + install_torchvision 2024-05-19T18:17:51.8982322Z + local orig_preload 2024-05-19T18:17:51.8982632Z + local commit 2024-05-19T18:17:51.8982943Z ++ get_pinned_commit vision 2024-05-19T18:17:51.8983328Z ++ cat .github/ci_commit_pins/vision.txt 2024-05-19T18:17:51.8989612Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:17:51.8990221Z + orig_preload= 2024-05-19T18:17:51.8990642Z + '[' -n '' ']' 2024-05-19T18:17:51.8991467Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:17:51.8992955Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:17:52.2071413Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:17:52.2075737Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-8ug9ta7z 2024-05-19T18:17:52.2093846Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-8ug9ta7z 2024-05-19T18:17:53.5168185Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-05-19T18:17:53.5176039Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:17:54.7288022Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:17:54.9586818Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:17:57.0539841Z Preparing metadata (setup.py) ... [?25l- \ done 2024-05-19T18:17:57.0586388Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-05-19T18:17:57.0589201Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.4.0a0+gitc8c7ddf) 2024-05-19T18:17:57.0594574Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-05-19T18:17:57.0797873Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-05-19T18:17:57.0804093Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.11.0) 2024-05-19T18:17:57.0806767Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12) 2024-05-19T18:17:57.0809952Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-05-19T18:17:57.0812104Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-05-19T18:17:57.0815047Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-05-19T18:17:57.1248029Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-05-19T18:17:57.1403943Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-05-19T18:17:57.1479739Z Building wheels for collected packages: torchvision 2024-05-19T18:19:07.0679736Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-05-19T18:19:07.0705603Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2025798 sha256=d043e46be3cfbf19dde4204d783793453b9ebf8c48f94ac21bb28d08cb1b7e82 2024-05-19T18:19:07.0708611Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-05-19T18:19:07.0742558Z Successfully built torchvision 2024-05-19T18:19:08.2698832Z Installing collected packages: torchvision 2024-05-19T18:19:08.6463268Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-05-19T18:19:08.7547068Z + '[' -n '' ']' 2024-05-19T18:19:08.7547506Z + id=0 2024-05-19T18:19:08.7547809Z + test_dynamo_benchmark timm_models 0 2024-05-19T18:19:08.7549053Z ++ pwd 2024-05-19T18:19:08.7551086Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-05-19T18:19:08.7552566Z + local suite=timm_models 2024-05-19T18:19:08.7553269Z + shift 2024-05-19T18:19:08.7553701Z + local shard_id=0 2024-05-19T18:19:08.7554074Z + shift 2024-05-19T18:19:08.7554385Z + [[ inductor_timm == *perf_compare* ]] 2024-05-19T18:19:08.7556187Z + [[ inductor_timm == *perf* ]] 2024-05-19T18:19:08.7556760Z + [[ inductor_timm == *cpu_inductor* ]] 2024-05-19T18:19:08.7557304Z + [[ inductor_timm == *aot_inductor* ]] 2024-05-19T18:19:08.7558286Z + test_single_dynamo_benchmark inference timm_models 0 --inference --bfloat16 2024-05-19T18:19:08.7558965Z ++ pwd 2024-05-19T18:19:08.7559491Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-05-19T18:19:08.7560186Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-05-19T18:19:08.7566963Z + local name=inference 2024-05-19T18:19:08.7567420Z + shift 2024-05-19T18:19:08.7568117Z + local suite=timm_models 2024-05-19T18:19:08.7568601Z + shift 2024-05-19T18:19:08.7568868Z + local shard_id=0 2024-05-19T18:19:08.7569169Z + shift 2024-05-19T18:19:08.7570462Z + partition_flags=() 2024-05-19T18:19:08.7570892Z + local partition_flags 2024-05-19T18:19:08.7571428Z + [[ -n 2 ]] 2024-05-19T18:19:08.7571838Z + [[ -n 0 ]] 2024-05-19T18:19:08.7572471Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-05-19T18:19:08.7573128Z + [[ inductor_timm == *perf_compare* ]] 2024-05-19T18:19:08.7573565Z + [[ inductor_timm == *perf* ]] 2024-05-19T18:19:08.7573972Z + [[ inductor_timm == *aot_inductor* ]] 2024-05-19T18:19:08.7575513Z + python benchmarks/dynamo/timm_models.py --ci --accuracy --timing --explain --inductor --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_timm_models.csv 2024-05-19T18:19:14.0306614Z 2024-05-19T18:19:14.4391601Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:19:14.4391976Z 2024-05-19T18:19:14.5500276Z model.safetensors: 0% 0.00/95.5M [00:00 will be ignored 2024-05-19T18:38:05.6029062Z E0519 18:38:05.601000 140609516171904 torch/_dynamo/utils.py:1392] RMSE (res-fp64): 0.01333, (ref-fp64): 0.00256 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.010000 2024-05-19T18:38:05.6031753Z E0519 18:38:05.602000 140609516171904 torch/_dynamo/utils.py:1306] Accuracy failed for key name blocks.0.attn.proj.bias.grad 2024-05-19T18:38:05.6053020Z fail_accuracy 2024-05-19T18:38:05.6551358Z TIMING: entire_frame_compile:63.06661 code_gen:24.76042 inductor_compile:42.87834 backend_compile:52.21182 2024-05-19T18:38:05.6552640Z STATS: call_* op count: 1028 | FakeTensor.__torch_dispatch__:15453 | FakeTensorMode.__torch_dispatch__:94224 | ProxyTorchDispatchMode.__torch_dispatch__:21953 2024-05-19T18:38:05.6553843Z Dynamo produced 3 graphs covering 1028 ops with 7 graph breaks (5 unique) 2024-05-19T18:38:11.5049667Z 2024-05-19T18:38:12.5355845Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:38:12.5356435Z loading model: 0it [00:01, ?it/s] 2024-05-19T18:38:12.5356971Z cuda train botnet26t_256 2024-05-19T18:38:59.5868541Z pass 2024-05-19T18:38:59.5971303Z TIMING: entire_frame_compile:27.9452 code_gen:21.61958 inductor_compile:29.88939 backend_compile:23.61355 2024-05-19T18:38:59.5972734Z STATS: call_* op count: 341 | FakeTensor.__torch_dispatch__:5770 | FakeTensorMode.__torch_dispatch__:34344 | ProxyTorchDispatchMode.__torch_dispatch__:9047 2024-05-19T18:38:59.5974090Z Dynamo produced 2 graphs covering 341 ops with 6 graph breaks (5 unique) 2024-05-19T18:39:03.8420676Z 2024-05-19T18:39:07.7141969Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:39:07.7142525Z loading model: 0it [00:03, ?it/s] 2024-05-19T18:39:07.7142963Z cuda train cait_m36_384 2024-05-19T18:39:07.7151497Z Traceback (most recent call last): 2024-05-19T18:39:07.7152412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2280, in validate_model 2024-05-19T18:39:07.7153489Z self.model_iter_fn(model, example_inputs) 2024-05-19T18:39:07.7154469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 365, in forward_and_backward_pass 2024-05-19T18:39:07.7155360Z pred = mod(*cloned_inputs) 2024-05-19T18:39:07.7156495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T18:39:07.7157677Z return self._call_impl(*args, **kwargs) 2024-05-19T18:39:07.7158731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T18:39:07.7159563Z return forward_call(*args, **kwargs) 2024-05-19T18:39:07.7160387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 361, in forward 2024-05-19T18:39:07.7161161Z x = self.forward_features(x) 2024-05-19T18:39:07.7162064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 346, in forward_features 2024-05-19T18:39:07.7163136Z x = self.blocks(x) 2024-05-19T18:39:07.7164273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T18:39:07.7165473Z return self._call_impl(*args, **kwargs) 2024-05-19T18:39:07.7166420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T18:39:07.7167241Z return forward_call(*args, **kwargs) 2024-05-19T18:39:07.7168137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/container.py", line 217, in forward 2024-05-19T18:39:07.7168925Z input = module(input) 2024-05-19T18:39:07.7169800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T18:39:07.7170936Z return self._call_impl(*args, **kwargs) 2024-05-19T18:39:07.7171852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T18:39:07.7172664Z return forward_call(*args, **kwargs) 2024-05-19T18:39:07.7173484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 197, in forward 2024-05-19T18:39:07.7174347Z x = x + self.drop_path(self.gamma_1 * self.attn(self.norm1(x))) 2024-05-19T18:39:07.7175483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T18:39:07.7176360Z return self._call_impl(*args, **kwargs) 2024-05-19T18:39:07.7177270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T18:39:07.7178087Z return forward_call(*args, **kwargs) 2024-05-19T18:39:07.7178910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 148, in forward 2024-05-19T18:39:07.7179793Z attn = self.proj_w(attn.permute(0, 2, 3, 1)).permute(0, 3, 1, 2) 2024-05-19T18:39:07.7180851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T18:39:07.7181783Z return self._call_impl(*args, **kwargs) 2024-05-19T18:39:07.7182688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T18:39:07.7183518Z return forward_call(*args, **kwargs) 2024-05-19T18:39:07.7184379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 116, in forward 2024-05-19T18:39:07.7185220Z return F.linear(input, self.weight, self.bias) 2024-05-19T18:39:07.7188387Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 82.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 66.44 MiB is free. Process 10456 has 21.90 GiB memory in use. Of the allocated memory 21.13 GiB is allocated by PyTorch, and 430.66 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-05-19T18:39:07.7191073Z 2024-05-19T18:39:07.7191366Z The above exception was the direct cause of the following exception: 2024-05-19T18:39:07.7191811Z 2024-05-19T18:39:07.7191962Z Traceback (most recent call last): 2024-05-19T18:39:07.7192610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4109, in run 2024-05-19T18:39:07.7193348Z ) = runner.load_model( 2024-05-19T18:39:07.7193998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 297, in load_model 2024-05-19T18:39:07.7194751Z self.validate_model(model, example_inputs) 2024-05-19T18:39:07.7195507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2282, in validate_model 2024-05-19T18:39:07.7196268Z raise RuntimeError("Eager run failed") from e 2024-05-19T18:39:07.7196731Z RuntimeError: Eager run failed 2024-05-19T18:39:07.7196990Z 2024-05-19T18:39:07.7197111Z eager_fail_to_run 2024-05-19T18:39:10.9484069Z 2024-05-19T18:39:11.8199194Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:39:11.8199756Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:39:11.8200187Z cuda train coat_lite_mini 2024-05-19T18:39:59.7208412Z pass 2024-05-19T18:39:59.7321171Z TIMING: entire_frame_compile:24.61972 code_gen:20.03362 inductor_compile:29.28053 backend_compile:21.33857 2024-05-19T18:39:59.7322787Z STATS: call_* op count: 480 | FakeTensor.__torch_dispatch__:6829 | FakeTensorMode.__torch_dispatch__:48354 | ProxyTorchDispatchMode.__torch_dispatch__:14370 2024-05-19T18:39:59.7325954Z Dynamo produced 2 graphs covering 480 ops with 6 graph breaks (5 unique) 2024-05-19T18:40:04.4114586Z 2024-05-19T18:40:06.0055645Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:40:06.0547578Z loading model: 0it [00:01, ?it/s] 2024-05-19T18:40:06.0548110Z cuda train convit_base 2024-05-19T18:40:06.0548866Z WARNING:common:fp64 golden ref were not generated for convit_base. Setting accuracy check to cosine 2024-05-19T18:41:26.0652961Z pass 2024-05-19T18:41:26.0876746Z TIMING: entire_frame_compile:48.92383 code_gen:27.79298 inductor_compile:46.15954 backend_compile:41.47383 2024-05-19T18:41:26.0879215Z STATS: call_* op count: 1216 | FakeTensor.__torch_dispatch__:17022 | FakeTensorMode.__torch_dispatch__:118378 | ProxyTorchDispatchMode.__torch_dispatch__:40289 2024-05-19T18:41:26.0880382Z Dynamo produced 3 graphs covering 1216 ops with 7 graph breaks (5 unique) 2024-05-19T18:41:31.2067035Z 2024-05-19T18:41:32.1511751Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:41:32.1512217Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:41:32.1512655Z cuda train convmixer_768_32 2024-05-19T18:42:21.1273739Z pass 2024-05-19T18:42:21.1423040Z TIMING: entire_frame_compile:27.14761 code_gen:15.2251 inductor_compile:27.83327 backend_compile:24.33471 2024-05-19T18:42:21.1425140Z STATS: call_* op count: 235 | FakeTensor.__torch_dispatch__:12638 | FakeTensorMode.__torch_dispatch__:60630 | ProxyTorchDispatchMode.__torch_dispatch__:15494 2024-05-19T18:42:21.1426257Z Dynamo produced 2 graphs covering 235 ops with 5 graph breaks (4 unique) 2024-05-19T18:42:25.5957893Z 2024-05-19T18:42:27.2961755Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:42:27.2962244Z loading model: 0it [00:01, ?it/s] 2024-05-19T18:42:27.2962680Z cuda train convnext_base 2024-05-19T18:43:22.8095343Z W0519 18:43:22.808000 139888738701952 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T18:44:49.3303388Z pass 2024-05-19T18:44:49.3778610Z TIMING: entire_frame_compile:108.00655 code_gen:46.92779 inductor_compile:78.52837 backend_compile:87.46323 2024-05-19T18:44:49.3779909Z STATS: call_* op count: 1621 | FakeTensor.__torch_dispatch__:28230 | FakeTensorMode.__torch_dispatch__:156179 | ProxyTorchDispatchMode.__torch_dispatch__:35464 2024-05-19T18:44:49.3781141Z Dynamo produced 3 graphs covering 1621 ops with 7 graph breaks (5 unique) 2024-05-19T18:44:57.1861903Z 2024-05-19T18:44:57.9705421Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:44:57.9705951Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:44:57.9706386Z cuda train crossvit_9_240 2024-05-19T18:45:49.6739011Z W0519 18:45:49.673000 139939074130560 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T18:46:56.7318607Z pass 2024-05-19T18:46:56.7451649Z TIMING: entire_frame_compile:93.98288 code_gen:39.3104 inductor_compile:66.64309 backend_compile:79.65925 2024-05-19T18:46:56.7452920Z STATS: call_* op count: 1403 | FakeTensor.__torch_dispatch__:24382 | FakeTensorMode.__torch_dispatch__:141997 | ProxyTorchDispatchMode.__torch_dispatch__:34463 2024-05-19T18:46:56.7457095Z Dynamo produced 3 graphs covering 1403 ops with 7 graph breaks (5 unique) 2024-05-19T18:47:03.7038129Z 2024-05-19T18:47:04.9297355Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:47:04.9297908Z loading model: 0it [00:01, ?it/s] 2024-05-19T18:47:04.9298366Z cuda train cspdarknet53 2024-05-19T18:48:02.7414028Z W0519 18:48:02.740000 140317532562048 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T18:48:57.9107632Z pass 2024-05-19T18:48:57.9253017Z TIMING: entire_frame_compile:87.84885 code_gen:36.75105 inductor_compile:63.23076 backend_compile:70.47613 2024-05-19T18:48:57.9255108Z STATS: call_* op count: 1070 | FakeTensor.__torch_dispatch__:22088 | FakeTensorMode.__torch_dispatch__:115546 | ProxyTorchDispatchMode.__torch_dispatch__:24624 2024-05-19T18:48:57.9256249Z Dynamo produced 3 graphs covering 1070 ops with 7 graph breaks (5 unique) 2024-05-19T18:49:04.3720205Z 2024-05-19T18:49:05.9729022Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:49:05.9729479Z loading model: 0it [00:01, ?it/s] 2024-05-19T18:49:33.5614845Z cuda train deit_base_distilled_patch16_224 2024-05-19T18:49:33.5616169Z W0519 18:49:33.560000 139769864090240 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T18:50:11.8341318Z pass 2024-05-19T18:50:11.8859083Z TIMING: entire_frame_compile:49.68995 code_gen:20.64646 inductor_compile:35.39917 backend_compile:42.06893 2024-05-19T18:50:11.8861115Z STATS: call_* op count: 841 | FakeTensor.__torch_dispatch__:13574 | FakeTensorMode.__torch_dispatch__:76457 | ProxyTorchDispatchMode.__torch_dispatch__:18136 2024-05-19T18:50:11.8862253Z Dynamo produced 3 graphs covering 841 ops with 7 graph breaks (5 unique) 2024-05-19T18:50:17.2844788Z 2024-05-19T18:50:18.6715334Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:50:18.6716211Z loading model: 0it [00:01, ?it/s] 2024-05-19T18:50:18.6717059Z cuda train dla102 2024-05-19T18:51:34.3410174Z W0519 18:51:34.340000 139839743890048 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T18:53:00.2177019Z pass 2024-05-19T18:53:00.2389989Z TIMING: entire_frame_compile:126.22312 code_gen:57.76659 inductor_compile:95.43105 backend_compile:106.33944 2024-05-19T18:53:00.2391627Z STATS: call_* op count: 1389 | FakeTensor.__torch_dispatch__:33585 | FakeTensorMode.__torch_dispatch__:174244 | ProxyTorchDispatchMode.__torch_dispatch__:36994 2024-05-19T18:53:00.2392851Z Dynamo produced 3 graphs covering 1389 ops with 7 graph breaks (5 unique) 2024-05-19T18:53:08.2597500Z 2024-05-19T18:53:10.0556195Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:53:10.0556656Z loading model: 0it [00:01, ?it/s] 2024-05-19T18:53:10.0557094Z cuda train dm_nfnet_f0 2024-05-19T18:54:05.9294907Z pass 2024-05-19T18:54:05.9790569Z TIMING: entire_frame_compile:30.13163 code_gen:17.28036 inductor_compile:30.141 backend_compile:21.17883 2024-05-19T18:54:05.9792203Z STATS: call_* op count: 620 | FakeTensorMode.__torch_dispatch__:57417 | FakeTensor.__torch_dispatch__:11878 | ProxyTorchDispatchMode.__torch_dispatch__:16005 2024-05-19T18:54:05.9793447Z Dynamo produced 2 graphs covering 620 ops with 6 graph breaks (5 unique) 2024-05-19T18:54:10.6764895Z 2024-05-19T18:54:12.3246116Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:54:12.3246983Z loading model: 0it [00:01, ?it/s] 2024-05-19T18:54:12.3247497Z cuda train dpn107 2024-05-19T18:55:51.6158093Z pass 2024-05-19T18:55:51.6796978Z TIMING: entire_frame_compile:60.72593 code_gen:33.41148 inductor_compile:54.02012 backend_compile:48.08692 2024-05-19T18:55:51.6798963Z STATS: call_* op count: 746 | FakeTensor.__torch_dispatch__:18755 | FakeTensorMode.__torch_dispatch__:104933 | ProxyTorchDispatchMode.__torch_dispatch__:28636 2024-05-19T18:55:51.6800300Z Dynamo produced 2 graphs covering 746 ops with 6 graph breaks (5 unique) 2024-05-19T18:55:57.2317503Z 2024-05-19T18:55:58.2420521Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:55:58.2421000Z loading model: 0it [00:01, ?it/s] 2024-05-19T18:55:58.2421443Z cuda train eca_botnext26ts_256 2024-05-19T18:56:45.8910944Z W0519 18:56:45.890000 140438012076672 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T18:57:16.7251987Z pass 2024-05-19T18:57:16.7365826Z TIMING: entire_frame_compile:56.01302 code_gen:31.47406 inductor_compile:47.9263 backend_compile:46.76216 2024-05-19T18:57:16.7367729Z STATS: call_* op count: 707 | FakeTensor.__torch_dispatch__:12445 | FakeTensorMode.__torch_dispatch__:67306 | ProxyTorchDispatchMode.__torch_dispatch__:15170 2024-05-19T18:57:16.7368858Z Dynamo produced 3 graphs covering 707 ops with 7 graph breaks (5 unique) 2024-05-19T18:57:21.8374838Z 2024-05-19T18:57:22.8792063Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:57:22.8793067Z loading model: 0it [00:01, ?it/s] 2024-05-19T18:57:22.8793937Z cuda train eca_halonext26ts 2024-05-19T18:58:13.8026328Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T18:58:13.8028214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 365, in torch_dynamo_resume_in_forward_and_backward_pass_at_363 2024-05-19T18:58:13.8029801Z pred = mod(*cloned_inputs) 2024-05-19T18:58:13.8031759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T18:58:13.8033269Z return forward_call(*args, **kwargs) 2024-05-19T18:58:13.8034708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/byobnet.py", line 1257, in forward 2024-05-19T18:58:13.8035980Z x = self.forward_features(x) 2024-05-19T18:58:13.8037416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/byobnet.py", line 1249, in forward_features 2024-05-19T18:58:13.8038751Z x = self.stages(x) 2024-05-19T18:58:13.8040097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T18:58:13.8041465Z return forward_call(*args, **kwargs) 2024-05-19T18:58:13.8042870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/byobnet.py", line 883, in forward 2024-05-19T18:58:13.8044104Z x = self.self_attn(x) 2024-05-19T18:58:13.8045784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T18:58:13.8047144Z return forward_call(*args, **kwargs) 2024-05-19T18:58:13.8048557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/layers/halo_attn.py", line 191, in forward 2024-05-19T18:58:13.8050293Z kv = kv.unfold(2, self.win_size, self.block_size).unfold(3, self.win_size, self.block_size).reshape( 2024-05-19T18:58:13.8051238Z 2024-05-19T18:58:14.0940370Z W0519 18:58:14.093000 139999483011712 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T18:58:44.5869642Z pass 2024-05-19T18:58:44.5939536Z TIMING: entire_frame_compile:56.56126 code_gen:33.67576 inductor_compile:50.73902 backend_compile:46.35019 2024-05-19T18:58:44.5940955Z STATS: call_* op count: 740 | FakeTensor.__torch_dispatch__:12244 | FakeTensorMode.__torch_dispatch__:69457 | ProxyTorchDispatchMode.__torch_dispatch__:15740 2024-05-19T18:58:44.5942337Z Dynamo produced 3 graphs covering 740 ops with 7 graph breaks (5 unique) 2024-05-19T18:58:49.7863372Z 2024-05-19T18:58:50.8863979Z loading model: 0it [00:00, ?it/s] 2024-05-19T18:58:50.8864446Z loading model: 0it [00:01, ?it/s] 2024-05-19T18:58:50.8864870Z cuda train ese_vovnet19b_dw 2024-05-19T18:59:25.6260987Z W0519 18:59:25.625000 139957561639552 torch/_logging/_internal.py:1024] [7/0] Profiler function will be ignored 2024-05-19T18:59:54.2431282Z pass 2024-05-19T18:59:54.2501609Z TIMING: entire_frame_compile:43.7231 code_gen:27.67463 inductor_compile:38.88386 backend_compile:34.64156 2024-05-19T18:59:54.2502870Z STATS: call_* op count: 459 | FakeTensor.__torch_dispatch__:9357 | FakeTensorMode.__torch_dispatch__:49632 | ProxyTorchDispatchMode.__torch_dispatch__:10392 2024-05-19T18:59:54.2503978Z Dynamo produced 4 graphs covering 459 ops with 7 graph breaks (5 unique) 2024-05-19T18:59:59.1025581Z 2024-05-19T19:00:00.0587601Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:00:00.0588106Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:00:00.0588524Z cuda train fbnetc_100 2024-05-19T19:00:49.0928043Z W0519 19:00:49.091000 139929636930176 torch/_logging/_internal.py:1024] [7/0] Profiler function will be ignored 2024-05-19T19:01:44.7833536Z pass 2024-05-19T19:01:44.7873186Z TIMING: entire_frame_compile:84.13481 code_gen:34.67599 inductor_compile:58.39141 backend_compile:68.01128 2024-05-19T19:01:44.7875328Z STATS: call_* op count: 993 | FakeTensor.__torch_dispatch__:19940 | FakeTensorMode.__torch_dispatch__:105832 | ProxyTorchDispatchMode.__torch_dispatch__:22083 2024-05-19T19:01:44.7876468Z Dynamo produced 4 graphs covering 993 ops with 7 graph breaks (5 unique) 2024-05-19T19:01:51.2715792Z 2024-05-19T19:01:52.4681299Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:01:52.4682112Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:01:52.4687303Z cuda train fbnetv3_b 2024-05-19T19:03:16.6264037Z pass 2024-05-19T19:03:16.6343306Z TIMING: entire_frame_compile:51.32478 code_gen:30.15133 inductor_compile:52.60593 backend_compile:41.65976 2024-05-19T19:03:16.6347374Z STATS: call_* op count: 610 | FakeTensor.__torch_dispatch__:20735 | FakeTensorMode.__torch_dispatch__:91577 | ProxyTorchDispatchMode.__torch_dispatch__:23345 2024-05-19T19:03:16.6349606Z Dynamo produced 2 graphs covering 610 ops with 6 graph breaks (5 unique) 2024-05-19T19:03:22.3026005Z 2024-05-19T19:03:23.5525766Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:03:23.5526377Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:03:23.5526902Z cuda train gernet_l 2024-05-19T19:04:08.1651951Z pass 2024-05-19T19:04:08.1852653Z TIMING: entire_frame_compile:29.16678 code_gen:13.91198 inductor_compile:24.39524 backend_compile:21.93693 2024-05-19T19:04:08.1854005Z STATS: call_* op count: 410 | FakeTensor.__torch_dispatch__:9258 | FakeTensorMode.__torch_dispatch__:47220 | ProxyTorchDispatchMode.__torch_dispatch__:12116 2024-05-19T19:04:08.1855114Z Dynamo produced 2 graphs covering 410 ops with 6 graph breaks (5 unique) 2024-05-19T19:04:12.6375382Z 2024-05-19T19:04:13.7131133Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:04:13.7131592Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:04:13.7135584Z cuda train ghostnet_100 2024-05-19T19:05:22.3394890Z pass 2024-05-19T19:05:22.3439848Z TIMING: entire_frame_compile:42.01414 code_gen:27.97833 inductor_compile:45.36768 backend_compile:37.22794 2024-05-19T19:05:22.3442058Z STATS: call_* op count: 360 | FakeTensor.__torch_dispatch__:15654 | FakeTensorMode.__torch_dispatch__:79497 | ProxyTorchDispatchMode.__torch_dispatch__:19864 2024-05-19T19:05:22.3443190Z Dynamo produced 2 graphs covering 360 ops with 6 graph breaks (5 unique) 2024-05-19T19:05:27.4579726Z 2024-05-19T19:05:28.8575641Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:05:28.8576112Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:05:28.8576571Z cuda train gluon_inception_v3 2024-05-19T19:06:00.8277987Z W0519 19:06:00.827000 140679716110976 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:07:15.9048125Z pass 2024-05-19T19:07:15.9277003Z TIMING: entire_frame_compile:95.58113 inductor_compile:43.27685 backend_compile:73.7039 code_gen:22.34536 2024-05-19T19:07:15.9279201Z STATS: call_* op count: 1376 | FakeTensor.__torch_dispatch__:16533 | FakeTensorMode.__torch_dispatch__:126519 | ProxyTorchDispatchMode.__torch_dispatch__:33213 2024-05-19T19:07:15.9280370Z Dynamo produced 3 graphs covering 1376 ops with 7 graph breaks (5 unique) 2024-05-19T19:07:23.2944760Z 2024-05-19T19:07:24.2759261Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:07:24.2759730Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:07:24.2760154Z cuda train gmixer_24_224 2024-05-19T19:08:13.7382825Z pass 2024-05-19T19:08:13.7628401Z TIMING: entire_frame_compile:27.65321 code_gen:15.69339 inductor_compile:27.50024 backend_compile:23.33618 2024-05-19T19:08:13.7629660Z STATS: call_* op count: 684 | FakeTensor.__torch_dispatch__:12928 | FakeTensorMode.__torch_dispatch__:70860 | ProxyTorchDispatchMode.__torch_dispatch__:23994 2024-05-19T19:08:13.7631283Z Dynamo produced 2 graphs covering 684 ops with 6 graph breaks (5 unique) 2024-05-19T19:08:18.5384292Z 2024-05-19T19:08:19.4412366Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:08:19.4415854Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:08:19.4416360Z cuda train gmlp_s16_224 2024-05-19T19:09:06.4256474Z W0519 19:09:06.424000 140254433055360 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:10:19.2423119Z pass 2024-05-19T19:10:19.2623054Z TIMING: entire_frame_compile:94.03957 code_gen:35.70079 inductor_compile:65.20133 backend_compile:77.18253 2024-05-19T19:10:19.2625943Z STATS: call_* op count: 1458 | FakeTensor.__torch_dispatch__:27730 | FakeTensorMode.__torch_dispatch__:149199 | ProxyTorchDispatchMode.__torch_dispatch__:35364 2024-05-19T19:10:19.2628206Z Dynamo produced 3 graphs covering 1458 ops with 7 graph breaks (5 unique) 2024-05-19T19:10:26.4653521Z 2024-05-19T19:10:28.2273321Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:10:28.2273793Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:10:28.2274215Z cuda train hrnet_w18 2024-05-19T19:13:32.4426655Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T19:13:32.4427897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 365, in torch_dynamo_resume_in_forward_and_backward_pass_at_363 2024-05-19T19:13:32.4428902Z pred = mod(*cloned_inputs) 2024-05-19T19:13:32.4430243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:13:32.4431315Z return forward_call(*args, **kwargs) 2024-05-19T19:13:32.4432433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 792, in forward 2024-05-19T19:13:32.4433491Z y = self.forward_features(x) 2024-05-19T19:13:32.4434656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 770, in forward_features 2024-05-19T19:13:32.4435456Z yl = self.stages(x) 2024-05-19T19:13:32.4436209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 751, in stages 2024-05-19T19:13:32.4436946Z yl = self.stage2(xl) 2024-05-19T19:13:32.4437764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:13:32.4438578Z return forward_call(*args, **kwargs) 2024-05-19T19:13:32.4439399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 506, in forward 2024-05-19T19:13:32.4440112Z x = module(x) 2024-05-19T19:13:32.4440908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:13:32.4441724Z return forward_call(*args, **kwargs) 2024-05-19T19:13:32.4442551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 484, in forward 2024-05-19T19:13:32.4443270Z y = y + f(x[j]) 2024-05-19T19:13:32.4443466Z 2024-05-19T19:13:37.5838289Z pass 2024-05-19T19:13:37.6018116Z TIMING: entire_frame_compile:123.22272 code_gen:62.54004 inductor_compile:115.41701 backend_compile:108.35941 2024-05-19T19:13:37.6019434Z STATS: call_* op count: 1394 | FakeTensor.__torch_dispatch__:52796 | FakeTensorMode.__torch_dispatch__:273525 | ProxyTorchDispatchMode.__torch_dispatch__:71274 2024-05-19T19:13:37.6020557Z Dynamo produced 2 graphs covering 1394 ops with 5 graph breaks (4 unique) 2024-05-19T19:13:46.1841215Z 2024-05-19T19:13:47.5733661Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:13:47.5734131Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:13:47.5734585Z cuda train inception_v3 2024-05-19T19:14:20.5849294Z pass 2024-05-19T19:14:20.6057815Z TIMING: entire_frame_compile:25.36684 inductor_compile:4.20321 backend_compile:16.1837 2024-05-19T19:14:20.6061044Z STATS: call_* op count: 506 | FakeTensor.__torch_dispatch__:2985 | FakeTensorMode.__torch_dispatch__:50578 | ProxyTorchDispatchMode.__torch_dispatch__:20409 2024-05-19T19:14:20.6062159Z Dynamo produced 2 graphs covering 506 ops with 6 graph breaks (5 unique) 2024-05-19T19:14:24.7373243Z 2024-05-19T19:14:26.2456914Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:14:26.2457388Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:14:26.2457814Z cuda train jx_nest_base 2024-05-19T19:15:45.1615603Z W0519 19:15:45.160000 139759399203456 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:16:59.0867691Z pass 2024-05-19T19:16:59.1322483Z TIMING: entire_frame_compile:112.64417 code_gen:50.44346 inductor_compile:82.85478 backend_compile:93.34999 2024-05-19T19:16:59.1324004Z STATS: call_* op count: 1700 | FakeTensor.__torch_dispatch__:27832 | FakeTensorMode.__torch_dispatch__:169110 | ProxyTorchDispatchMode.__torch_dispatch__:42081 2024-05-19T19:16:59.1326680Z Dynamo produced 3 graphs covering 1700 ops with 7 graph breaks (5 unique) 2024-05-19T19:17:06.7729931Z 2024-05-19T19:17:07.5419651Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:17:07.5420353Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:17:07.5420998Z cuda train lcnet_050 2024-05-19T19:17:35.5061707Z pass 2024-05-19T19:17:35.5083131Z TIMING: entire_frame_compile:16.72859 code_gen:10.04038 inductor_compile:17.616 backend_compile:13.78343 2024-05-19T19:17:35.5086026Z STATS: call_* op count: 168 | FakeTensor.__torch_dispatch__:6258 | FakeTensorMode.__torch_dispatch__:27787 | ProxyTorchDispatchMode.__torch_dispatch__:6908 2024-05-19T19:17:35.5087199Z Dynamo produced 2 graphs covering 168 ops with 6 graph breaks (5 unique) 2024-05-19T19:17:39.6058228Z 2024-05-19T19:17:40.6240279Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:17:40.6240818Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:17:40.6241236Z cuda train levit_128 2024-05-19T19:18:51.9700621Z skipping cudagraphs due to deterministic index put. Found from : 2024-05-19T19:18:51.9702102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 365, in torch_dynamo_resume_in_forward_and_backward_pass_at_363 2024-05-19T19:18:51.9703128Z pred = mod(*cloned_inputs) 2024-05-19T19:18:51.9705405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:18:51.9706396Z return forward_call(*args, **kwargs) 2024-05-19T19:18:51.9707341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 652, in forward 2024-05-19T19:18:51.9708131Z x = self.forward_features(x) 2024-05-19T19:18:51.9709001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 643, in forward_features 2024-05-19T19:18:51.9709851Z x = self.stages(x) 2024-05-19T19:18:51.9710780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:18:51.9711591Z return forward_call(*args, **kwargs) 2024-05-19T19:18:51.9712416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 518, in forward 2024-05-19T19:18:51.9713309Z x = self.blocks(x) 2024-05-19T19:18:51.9714154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:18:51.9714970Z return forward_call(*args, **kwargs) 2024-05-19T19:18:51.9715782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 456, in forward 2024-05-19T19:18:51.9716547Z x = x + self.drop_path1(self.attn(x)) 2024-05-19T19:18:51.9717465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:18:51.9718305Z return forward_call(*args, **kwargs) 2024-05-19T19:18:51.9719112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 236, in forward 2024-05-19T19:18:51.9719963Z attn = q @ k * self.scale + self.get_attention_biases(x.device) 2024-05-19T19:18:51.9720979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 211, in get_attention_biases 2024-05-19T19:18:51.9722195Z return self.attention_biases[:, self.attention_bias_idxs] 2024-05-19T19:18:51.9722578Z 2024-05-19T19:18:52.2804718Z W0519 19:18:52.279000 140553124065920 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:19:45.0893871Z pass 2024-05-19T19:19:45.0927230Z TIMING: entire_frame_compile:94.64654 code_gen:42.99431 inductor_compile:72.1235 backend_compile:80.11691 2024-05-19T19:19:45.0929225Z STATS: call_* op count: 1225 | FakeTensor.__torch_dispatch__:23447 | FakeTensorMode.__torch_dispatch__:136137 | ProxyTorchDispatchMode.__torch_dispatch__:32089 2024-05-19T19:19:45.0930388Z Dynamo produced 3 graphs covering 1225 ops with 7 graph breaks (5 unique) 2024-05-19T19:19:51.9139995Z 2024-05-19T19:19:53.1244570Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:19:53.1245269Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:19:53.1245716Z cuda train mixer_b16_224 2024-05-19T19:20:22.1359605Z W0519 19:20:22.135000 140058934882944 torch/_logging/_internal.py:1024] [6/0] Profiler function will be ignored 2024-05-19T19:20:59.1336104Z pass 2024-05-19T19:20:59.1753365Z TIMING: entire_frame_compile:49.71109 code_gen:21.02934 inductor_compile:35.72537 backend_compile:42.58238 2024-05-19T19:20:59.1755292Z STATS: call_* op count: 720 | FakeTensor.__torch_dispatch__:13856 | FakeTensorMode.__torch_dispatch__:74553 | ProxyTorchDispatchMode.__torch_dispatch__:18120 2024-05-19T19:20:59.1756509Z Dynamo produced 3 graphs covering 720 ops with 7 graph breaks (5 unique) 2024-05-19T19:21:01.8731879Z accuracy pass_rate=93.33% 2024-05-19T19:21:01.8735946Z calls_captured gmean=0.00x mean=833.533x 2024-05-19T19:21:01.8738996Z unique_graphs gmean=0.00x mean=2.533x 2024-05-19T19:21:01.8742547Z graph_breaks gmean=0.00x mean=6.267x 2024-05-19T19:21:01.8745866Z unique_graph_breaks gmean=0.00x mean=4.767x 2024-05-19T19:21:01.8749351Z autograd_captures gmean=0.00x mean=0.000x 2024-05-19T19:21:01.8752766Z autograd_compiles gmean=0.00x mean=0.000x 2024-05-19T19:21:01.8756163Z cudagraph_skips gmean=0.00x mean=0.133x 2024-05-19T19:21:03.5363263Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_timm_training.csv 2024-05-19T19:21:03.7796627Z adv_inception_v3 PASS 2024-05-19T19:21:03.7800871Z beit_base_patch16_224 FAIL: accuracy=fail_accuracy, expected=pass 2024-05-19T19:21:03.7805025Z botnet26t_256 PASS 2024-05-19T19:21:03.7810680Z cait_m36_384 XFAIL 2024-05-19T19:21:03.7816183Z coat_lite_mini PASS 2024-05-19T19:21:03.7821423Z convit_base PASS 2024-05-19T19:21:03.7826943Z convmixer_768_32 PASS 2024-05-19T19:21:03.7832301Z convnext_base PASS 2024-05-19T19:21:03.7837851Z crossvit_9_240 PASS 2024-05-19T19:21:03.7843166Z cspdarknet53 PASS 2024-05-19T19:21:03.7848583Z deit_base_distilled_patch16_224 PASS 2024-05-19T19:21:03.7853632Z dla102 PASS 2024-05-19T19:21:03.7859233Z dm_nfnet_f0 PASS 2024-05-19T19:21:03.7864482Z dpn107 PASS 2024-05-19T19:21:03.7869798Z eca_botnext26ts_256 PASS 2024-05-19T19:21:03.7875343Z eca_halonext26ts PASS 2024-05-19T19:21:03.7880937Z ese_vovnet19b_dw PASS 2024-05-19T19:21:03.7886481Z fbnetc_100 PASS 2024-05-19T19:21:03.7891760Z fbnetv3_b PASS 2024-05-19T19:21:03.7897073Z gernet_l PASS 2024-05-19T19:21:03.7902298Z ghostnet_100 PASS 2024-05-19T19:21:03.7907603Z gluon_inception_v3 PASS 2024-05-19T19:21:03.7912947Z gmixer_24_224 PASS 2024-05-19T19:21:03.7918342Z gmlp_s16_224 PASS 2024-05-19T19:21:03.7923617Z hrnet_w18 PASS 2024-05-19T19:21:03.7929140Z inception_v3 PASS 2024-05-19T19:21:03.7934349Z jx_nest_base PASS 2024-05-19T19:21:03.7939681Z lcnet_050 PASS 2024-05-19T19:21:03.7944943Z levit_128 PASS 2024-05-19T19:21:03.7950747Z mixer_b16_224 PASS 2024-05-19T19:21:03.7951056Z 2024-05-19T19:21:03.7951249Z Error: 1 models have accuracy status regressed: 2024-05-19T19:21:03.7951721Z beit_base_patch16_224 2024-05-19T19:21:03.7951950Z 2024-05-19T19:21:03.7956040Z 2024-05-19T19:21:03.7956958Z If this change is expected, you can update `benchmarks/dynamo/ci_expected_accuracy/inductor_timm_training.csv` to reflect the new baseline. 2024-05-19T19:21:03.7958030Z from pytorch/pytorch root, run 2024-05-19T19:21:03.7959056Z `python benchmarks/dynamo/ci_expected_accuracy/update_expected.py c8c7ddfde57442cc28a394c385cab656ce9b1f5e` 2024-05-19T19:21:03.7960063Z and then `git add` the resulting local changes to expected CSVs to your commit. 2024-05-19T19:21:03.7960554Z 2024-05-19T19:21:03.8349367Z + cleanup_workspace 2024-05-19T19:21:03.8350519Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-05-19T19:21:03.8351613Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-05-19T19:21:03.8352640Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-05-19T19:21:03.8353269Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-05-19T19:21:03.8354064Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-05-19T19:21:03.8354931Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-05-19T19:21:03.8355632Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-05-19T19:21:04.3326623Z ##[error]Process completed with exit code 1. 2024-05-19T19:21:04.3399463Z Prepare all required actions 2024-05-19T19:21:04.3399910Z Getting action download info 2024-05-19T19:21:04.4946478Z ##[group]Run ./.github/actions/pytest-cache-upload 2024-05-19T19:21:04.4946947Z with: 2024-05-19T19:21:04.4947252Z cache_dir: .pytest_cache 2024-05-19T19:21:04.4947620Z shard: 1 2024-05-19T19:21:04.4947973Z sha: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T19:21:04.4948455Z test_config: inductor_timm 2024-05-19T19:21:04.4948964Z job_identifier: inductor_linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T19:21:04.4949503Z env: 2024-05-19T19:21:04.4949788Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:21:04.4950254Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:21:04.4951011Z DOCKER_CONTAINER_ID: 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T19:21:04.4951670Z ##[endgroup] 2024-05-19T19:21:04.4991150Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-05-19T19:21:04.4991772Z with: 2024-05-19T19:21:04.4992046Z shell: bash 2024-05-19T19:21:04.4992354Z timeout_minutes: 5 2024-05-19T19:21:04.4992751Z max_attempts: 5 2024-05-19T19:21:04.4993086Z retry_wait_seconds: 30 2024-05-19T19:21:04.4993544Z command: set -eu python3 -m pip install boto3==1.19.12 2024-05-19T19:21:04.4994065Z polling_interval_seconds: 1 2024-05-19T19:21:04.4994448Z warning_on_retry: true 2024-05-19T19:21:04.4994813Z continue_on_error: false 2024-05-19T19:21:04.4995156Z env: 2024-05-19T19:21:04.4995439Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:21:04.4995899Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:21:04.4996660Z DOCKER_CONTAINER_ID: 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T19:21:04.4997320Z ##[endgroup] 2024-05-19T19:21:04.7656825Z Defaulting to user installation because normal site-packages is not writeable 2024-05-19T19:21:05.9859168Z Collecting boto3==1.19.12 2024-05-19T19:21:06.0045341Z Downloading boto3-1.19.12-py3-none-any.whl (131 kB) 2024-05-19T19:21:07.4262149Z Collecting botocore<1.23.0,>=1.22.12 2024-05-19T19:21:07.4336812Z Downloading botocore-1.22.12-py3-none-any.whl (8.1 MB) 2024-05-19T19:21:07.5889180Z Collecting s3transfer<0.6.0,>=0.5.0 2024-05-19T19:21:07.5928933Z Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 2024-05-19T19:21:07.6304725Z Collecting jmespath<1.0.0,>=0.7.1 2024-05-19T19:21:07.6346053Z Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 2024-05-19T19:21:07.6427126Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.18) 2024-05-19T19:21:07.7047785Z Collecting python-dateutil<3.0.0,>=2.1 2024-05-19T19:21:07.7089305Z Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) 2024-05-19T19:21:07.7541719Z Collecting six>=1.5 2024-05-19T19:21:07.7581621Z Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2024-05-19T19:21:07.8280366Z Installing collected packages: six, python-dateutil, jmespath, botocore, s3transfer, boto3 2024-05-19T19:21:08.4902670Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.9.0.post0 s3transfer-0.5.2 six-1.16.0 2024-05-19T19:21:08.5505876Z Command completed after 1 attempt(s). 2024-05-19T19:21:08.5547629Z ##[group]Run python3 .github/scripts/pytest_cache.py \ 2024-05-19T19:21:08.5548224Z python3 .github/scripts/pytest_cache.py \ 2024-05-19T19:21:08.5548708Z  --upload \ 2024-05-19T19:21:08.5549139Z  --cache_dir $GITHUB_WORKSPACE/$CACHE_DIR \ 2024-05-19T19:21:08.5549664Z  --pr_identifier $GITHUB_REF \ 2024-05-19T19:21:08.5550150Z  --job_identifier $JOB_IDENTIFIER \ 2024-05-19T19:21:08.5550610Z  --sha $SHA \ 2024-05-19T19:21:08.5551005Z  --test_config $TEST_CONFIG \ 2024-05-19T19:21:08.5551437Z  --shard $SHARD \ 2024-05-19T19:21:08.5551820Z  --repo $REPO \ 2024-05-19T19:21:08.5552337Z  --temp_dir $RUNNER_TEMP \ 2024-05-19T19:21:08.5560196Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T19:21:08.5560703Z env: 2024-05-19T19:21:08.5560994Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:21:08.5561461Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:21:08.5562226Z DOCKER_CONTAINER_ID: 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T19:21:08.5562910Z CACHE_DIR: .pytest_cache 2024-05-19T19:21:08.5563414Z JOB_IDENTIFIER: inductor_linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T19:21:08.5564018Z SHA: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T19:21:08.5564494Z TEST_CONFIG: inductor_timm 2024-05-19T19:21:08.5565186Z SHARD: 1 2024-05-19T19:21:08.5565533Z REPO: pytorch/pytorch 2024-05-19T19:21:08.5565943Z ##[endgroup] 2024-05-19T19:21:08.6919981Z PR identifier for `refs/tags/ciflow/inductor/121956` is `5ec4bd29b9652d20d61d9e0a665b8d46` 2024-05-19T19:21:08.6923886Z Uploading cache with args Namespace(bucket=None, cache_dir='/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache', download=False, job_identifier='inductor_linux-focal-cuda12.4-py3.10-gcc9-sm86', pr_identifier='refs/tags/ciflow/inductor/121956', repo='pytorch/pytorch', sha='c8c7ddfde57442cc28a394c385cab656ce9b1f5e', shard='1', temp_dir='/home/ec2-user/actions-runner/_work/_temp', test_config='inductor_timm', upload=True) 2024-05-19T19:21:08.6926841Z The pytest cache dir `/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache` does not exist. Skipping upload 2024-05-19T19:21:08.7091560Z ##[group]Run cat test/**/*_toprint.log || true 2024-05-19T19:21:08.7092121Z cat test/**/*_toprint.log || true 2024-05-19T19:21:08.7099839Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T19:21:08.7100333Z env: 2024-05-19T19:21:08.7100621Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:21:08.7101224Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:21:08.7102006Z DOCKER_CONTAINER_ID: 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T19:21:08.7102682Z ##[endgroup] 2024-05-19T19:21:08.7164929Z cat: test/**/*_toprint.log: No such file or directory 2024-05-19T19:21:08.7193572Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-05-19T19:21:08.7194039Z kill "$MONITOR_SCRIPT_PID" 2024-05-19T19:21:08.7200805Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T19:21:08.7201301Z env: 2024-05-19T19:21:08.7201598Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:21:08.7202059Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:21:08.7202815Z DOCKER_CONTAINER_ID: 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T19:21:08.7203480Z MONITOR_SCRIPT_PID: 28681 2024-05-19T19:21:08.7203842Z ##[endgroup] 2024-05-19T19:21:08.7399105Z Prepare all required actions 2024-05-19T19:21:08.7399550Z Getting action download info 2024-05-19T19:21:09.4215924Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-05-19T19:21:09.5815742Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-05-19T19:21:09.5816211Z with: 2024-05-19T19:21:09.5816723Z file-suffix: test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197291 2024-05-19T19:21:09.5817349Z s3-bucket: gha-artifacts 2024-05-19T19:21:09.5817688Z env: 2024-05-19T19:21:09.5817974Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:21:09.5818428Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:21:09.5819178Z DOCKER_CONTAINER_ID: 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T19:21:09.5819824Z ##[endgroup] 2024-05-19T19:21:09.5845255Z ##[group]Run # Remove any previous test jsons if they exist 2024-05-19T19:21:09.5845897Z # Remove any previous test jsons if they exist 2024-05-19T19:21:09.5846414Z rm -f test-jsons-*.zip 2024-05-19T19:21:09.5846939Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-05-19T19:21:09.5854673Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T19:21:09.5855173Z env: 2024-05-19T19:21:09.5855467Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:21:09.5855924Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:21:09.5856678Z DOCKER_CONTAINER_ID: 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T19:21:09.5857554Z FILE_SUFFIX: test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197291 2024-05-19T19:21:09.5858156Z ##[endgroup] 2024-05-19T19:21:09.6050431Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-05-19T19:21:09.6079051Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-05-19T19:21:09.6079731Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-05-19T19:21:09.6085926Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-05-19T19:21:09.6113353Z ##[group]Run # Remove any previous test reports if they exist 2024-05-19T19:21:09.6114021Z # Remove any previous test reports if they exist 2024-05-19T19:21:09.6114552Z rm -f test-reports-*.zip 2024-05-19T19:21:09.6115159Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-05-19T19:21:09.6122925Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T19:21:09.6123415Z env: 2024-05-19T19:21:09.6123702Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:21:09.6124166Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:21:09.6125214Z DOCKER_CONTAINER_ID: 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T19:21:09.6126090Z FILE_SUFFIX: test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197291 2024-05-19T19:21:09.6126697Z ##[endgroup] 2024-05-19T19:21:09.6282957Z adding: test/test-reports/inference_timm_models.csv (deflated 69%) 2024-05-19T19:21:09.6283742Z adding: test/test-reports/training_timm_models.csv (deflated 65%) 2024-05-19T19:21:09.6287036Z adding: test/test-reports/training_timm_model_graph_breaks.csv (deflated 97%) 2024-05-19T19:21:09.6287948Z adding: test/test-reports/training_timm_model_graph_break_deduped.csv (deflated 73%) 2024-05-19T19:21:09.6313480Z ##[group]Run # Remove any previous usage logs if they exist 2024-05-19T19:21:09.6314107Z # Remove any previous usage logs if they exist 2024-05-19T19:21:09.6314609Z rm -f logs-*.zip 2024-05-19T19:21:09.6315271Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-05-19T19:21:09.6316061Z # so check to see if the file exists first 2024-05-19T19:21:09.6316554Z if [ -f 'usage_log.txt' ]; then 2024-05-19T19:21:09.6317088Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-05-19T19:21:09.6317576Z fi 2024-05-19T19:21:09.6318070Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-05-19T19:21:09.6318644Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-05-19T19:21:09.6319144Z fi 2024-05-19T19:21:09.6326677Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T19:21:09.6327184Z env: 2024-05-19T19:21:09.6327474Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:21:09.6327929Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:21:09.6328684Z DOCKER_CONTAINER_ID: 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T19:21:09.6329547Z FILE_SUFFIX: test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197291 2024-05-19T19:21:09.6330151Z ##[endgroup] 2024-05-19T19:21:09.6458076Z adding: usage_log.txt (deflated 96%) 2024-05-19T19:21:09.6537323Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-05-19T19:21:09.6538040Z # Remove any previous debugging artifacts if they exist 2024-05-19T19:21:09.6538603Z rm -f debug-*.zip 2024-05-19T19:21:09.6539004Z if [ -d 'test/debug' ]; then 2024-05-19T19:21:09.6539519Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-05-19T19:21:09.6540008Z fi 2024-05-19T19:21:09.6547005Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T19:21:09.6547509Z env: 2024-05-19T19:21:09.6547797Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:21:09.6548249Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:21:09.6549011Z DOCKER_CONTAINER_ID: 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T19:21:09.6549891Z FILE_SUFFIX: test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197291 2024-05-19T19:21:09.6550500Z ##[endgroup] 2024-05-19T19:21:09.6624788Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-05-19T19:21:09.6625240Z with: 2024-05-19T19:21:09.6625519Z s3-bucket: gha-artifacts 2024-05-19T19:21:09.6625957Z s3-prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T19:21:09.6626455Z retention-days: 14 2024-05-19T19:21:09.6626797Z if-no-files-found: warn 2024-05-19T19:21:09.6627165Z path: test-jsons-*.zip 2024-05-19T19:21:09.6627528Z name: artifact 2024-05-19T19:21:09.6627841Z region: us-east-1 2024-05-19T19:21:09.6628139Z env: 2024-05-19T19:21:09.6628430Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:21:09.6628890Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:21:09.6629648Z DOCKER_CONTAINER_ID: 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T19:21:09.6630297Z ##[endgroup] 2024-05-19T19:21:09.9724164Z NOTE: s3-prefix specified, ignoring name parameter 2024-05-19T19:21:09.9725142Z With the provided path, there will be 1 file uploaded 2024-05-19T19:21:09.9725865Z Uploading to s3 prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T19:21:10.0689724Z Starting upload of test-jsons-test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197291.zip 2024-05-19T19:21:10.1890495Z Finished upload of test-jsons-test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197291.zip 2024-05-19T19:21:10.2021065Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-05-19T19:21:10.2021656Z with: 2024-05-19T19:21:10.2021948Z s3-bucket: gha-artifacts 2024-05-19T19:21:10.2022382Z s3-prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T19:21:10.2022864Z retention-days: 14 2024-05-19T19:21:10.2023198Z if-no-files-found: error 2024-05-19T19:21:10.2023570Z path: test-reports-*.zip 2024-05-19T19:21:10.2023928Z name: artifact 2024-05-19T19:21:10.2024231Z region: us-east-1 2024-05-19T19:21:10.2024588Z env: 2024-05-19T19:21:10.2024940Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:21:10.2025399Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:21:10.2026142Z DOCKER_CONTAINER_ID: 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T19:21:10.2026797Z ##[endgroup] 2024-05-19T19:21:10.5099094Z NOTE: s3-prefix specified, ignoring name parameter 2024-05-19T19:21:10.5099996Z With the provided path, there will be 1 file uploaded 2024-05-19T19:21:10.5100621Z Uploading to s3 prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T19:21:10.5125984Z Starting upload of test-reports-test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197291.zip 2024-05-19T19:21:10.6073786Z Finished upload of test-reports-test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197291.zip 2024-05-19T19:21:10.6204256Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-05-19T19:21:10.6204970Z with: 2024-05-19T19:21:10.6205268Z s3-bucket: gha-artifacts 2024-05-19T19:21:10.6205698Z s3-prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T19:21:10.6206169Z retention-days: 14 2024-05-19T19:21:10.6206505Z if-no-files-found: ignore 2024-05-19T19:21:10.6206868Z path: logs-*.zip 2024-05-19T19:21:10.6207182Z name: artifact 2024-05-19T19:21:10.6207485Z region: us-east-1 2024-05-19T19:21:10.6207790Z env: 2024-05-19T19:21:10.6208070Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:21:10.6208538Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:21:10.6209285Z DOCKER_CONTAINER_ID: 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T19:21:10.6209957Z ##[endgroup] 2024-05-19T19:21:10.9241616Z NOTE: s3-prefix specified, ignoring name parameter 2024-05-19T19:21:10.9242213Z With the provided path, there will be 1 file uploaded 2024-05-19T19:21:10.9242831Z Uploading to s3 prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T19:21:10.9269097Z Starting upload of logs-test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197291.zip 2024-05-19T19:21:11.0667060Z Finished upload of logs-test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_25153197291.zip 2024-05-19T19:21:11.0796696Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-05-19T19:21:11.0797136Z with: 2024-05-19T19:21:11.0797425Z s3-bucket: gha-artifacts 2024-05-19T19:21:11.0797854Z s3-prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T19:21:11.0798337Z retention-days: 14 2024-05-19T19:21:11.0798670Z if-no-files-found: ignore 2024-05-19T19:21:11.0799059Z path: debug-*.zip 2024-05-19T19:21:11.0799377Z name: artifact 2024-05-19T19:21:11.0799686Z region: us-east-1 2024-05-19T19:21:11.0799991Z env: 2024-05-19T19:21:11.0800269Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:21:11.0800729Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:21:11.0801469Z DOCKER_CONTAINER_ID: 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T19:21:11.0802123Z ##[endgroup] 2024-05-19T19:21:11.3803261Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-05-19T19:21:11.3934850Z ##[group]Run # shellcheck disable=SC2156 2024-05-19T19:21:11.3935331Z # shellcheck disable=SC2156 2024-05-19T19:21:11.3936131Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-05-19T19:21:11.3944026Z shell: /usr/bin/bash -e {0} 2024-05-19T19:21:11.3944396Z env: 2024-05-19T19:21:11.3944688Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:21:11.3945157Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:21:11.3946049Z DOCKER_CONTAINER_ID: 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T19:21:11.3946712Z ##[endgroup] 2024-05-19T19:21:11.5734084Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-05-19T19:21:11.5734592Z with: 2024-05-19T19:21:11.5735018Z name: coredumps-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu 2024-05-19T19:21:11.5735564Z retention-days: 14 2024-05-19T19:21:11.5735900Z if-no-files-found: ignore 2024-05-19T19:21:11.5736262Z path: ./**/core.[1-9]* 2024-05-19T19:21:11.5736617Z s3-bucket: gha-artifacts 2024-05-19T19:21:11.5736974Z region: us-east-1 2024-05-19T19:21:11.5737267Z env: 2024-05-19T19:21:11.5737540Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:21:11.5737990Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:21:11.5738723Z DOCKER_CONTAINER_ID: 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T19:21:11.5739379Z ##[endgroup] 2024-05-19T19:21:20.9798846Z No files were found with the provided path: ./**/core.[1-9]*. No artifacts will be uploaded. 2024-05-19T19:21:21.0001333Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-05-19T19:21:21.0001889Z with: 2024-05-19T19:21:21.0002143Z env: 2024-05-19T19:21:21.0002434Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:21:21.0002895Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:21:21.0003652Z DOCKER_CONTAINER_ID: 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T19:21:21.0004299Z ##[endgroup] 2024-05-19T19:21:21.0024076Z ##[group]Run set -eou pipefail 2024-05-19T19:21:21.0024485Z set -eou pipefail 2024-05-19T19:21:21.0024830Z  2024-05-19T19:21:21.0025351Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-05-19T19:21:21.0026002Z for _ in $(seq 1440); do 2024-05-19T19:21:21.0026480Z  # Break if no ssh session exists anymore 2024-05-19T19:21:21.0026979Z  if [ "$(who)" = "" ]; then 2024-05-19T19:21:21.0027421Z  break 2024-05-19T19:21:21.0027781Z  fi 2024-05-19T19:21:21.0028084Z  echo "." 2024-05-19T19:21:21.0028423Z  sleep 5 2024-05-19T19:21:21.0028754Z done 2024-05-19T19:21:21.0036223Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T19:21:21.0036778Z env: 2024-05-19T19:21:21.0037063Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:21:21.0037525Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:21:21.0038274Z DOCKER_CONTAINER_ID: 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T19:21:21.0038938Z ##[endgroup] 2024-05-19T19:21:21.0059231Z Holding runner for 2 hours until all ssh sessions have logged out 2024-05-19T19:21:21.0107794Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-05-19T19:21:21.0108546Z # ignore expansion of "docker ps -q" since it could be empty 2024-05-19T19:21:21.0109132Z # shellcheck disable=SC2046 2024-05-19T19:21:21.0109628Z docker stop $(docker ps -q) || true 2024-05-19T19:21:21.0110109Z # Prune all of the docker images 2024-05-19T19:21:21.0110555Z docker system prune -af 2024-05-19T19:21:21.0118448Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T19:21:21.0118954Z env: 2024-05-19T19:21:21.0119239Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:21:21.0119690Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:21:21.0120452Z DOCKER_CONTAINER_ID: 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T19:21:21.0121121Z ##[endgroup] 2024-05-19T19:21:21.5519258Z 8e0cfbfda7b9 2024-05-19T19:21:22.2964290Z Deleted Containers: 2024-05-19T19:21:22.2965232Z 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T19:21:22.2965708Z 2024-05-19T19:21:27.4196152Z Deleted Images: 2024-05-19T19:21:27.4198956Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T19:21:27.4201903Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks@sha256:66f062925f91f9b9204b1dc6d68caf8c89fa82fd4b12cb5f24c19aa4b1c87506 2024-05-19T19:21:27.4203430Z deleted: sha256:a0542ac5317cc182e1aac2cb8d63a2d6287450937722bab68464513bbbc603ef 2024-05-19T19:21:27.4204267Z deleted: sha256:1a656f83b702d1cdc805b9509e7194f6ca9a24882785be5289f8bc566c1c1a0b 2024-05-19T19:21:27.4205438Z deleted: sha256:5b7ca0efbb8f2f6ed9ac2e375c4a52f07e2bd16e65d1914a58dcfb56ab8b1623 2024-05-19T19:21:27.4206278Z deleted: sha256:6d2255f7483ad8cfa448dc385ff2880171aacd4db919fbf38e2b3c4580f80e1e 2024-05-19T19:21:27.4207110Z deleted: sha256:a39c366330a4b21f0585d02ed65f5beb66bd03e36471ebb255c5f7766f29c118 2024-05-19T19:21:27.4207948Z deleted: sha256:5b2e96a220babb6e3a9b843ce53d3f2015dd0e505fed00a447cbe551b22d329e 2024-05-19T19:21:27.4208779Z deleted: sha256:d7247668eca42c13d2c6440886a9b43b633e5cb9304e3e3f85ac8803de7dd873 2024-05-19T19:21:27.4209741Z deleted: sha256:989398e2b08b3b7c9bf03cdbdaa903261d145b7d45bc7c1fa3a1a4a54a7ced68 2024-05-19T19:21:27.4210692Z deleted: sha256:5c2558f875db144549ed0efa5deb36d738647f1c172a9628a34677fddb6a96ee 2024-05-19T19:21:27.4211776Z deleted: sha256:b56f2c08b00650c89ac0f4d07686ba8ea22d64f666d27e8be716aeae4e042f87 2024-05-19T19:21:27.4212760Z deleted: sha256:35689f96fd20d72b6c5fd24a65e3c52c46359609c897b0552d8b36e4d62f31d9 2024-05-19T19:21:27.4213581Z deleted: sha256:8a251e66682a1ad41addcf4890ffb62aa8b85f5d9f17d433e828f170339a767f 2024-05-19T19:21:27.4214404Z deleted: sha256:9482b91acd78f655bcd9a0a6ade0f1958caa1a8274b13f7366f830fa7b4ca16e 2024-05-19T19:21:27.4215233Z deleted: sha256:1f1bf62bcfe25406d5e8f3a48198af80782c6a996bcafaae53575bcc843bd50a 2024-05-19T19:21:27.4216063Z deleted: sha256:6b680df6a8beceb613eb411958551bf0368a6d4a932eead77896d41ff535946a 2024-05-19T19:21:27.4216885Z deleted: sha256:7aa1322c73a827485b091076ffa2dc207e0a900941ca2eb2704442ca7c23ed7d 2024-05-19T19:21:27.4217707Z deleted: sha256:f0d92b27a23be6ff6a30b0974841e39e2ca9cb761797d5cb3ee5676708fbfddc 2024-05-19T19:21:27.4218535Z deleted: sha256:15e17f6dcd9e56d48cc4d19f06cf6d42e155d9dcb5d6d3289f432b1e1adb78c2 2024-05-19T19:21:27.4219352Z deleted: sha256:a9363d4f861657188bb086df9f77430971cf00513d7f3c151d954bba04092dd4 2024-05-19T19:21:27.4220171Z deleted: sha256:afad6ce30f8cd2b2fc987fb31626f14465a034dd85cea22a30e012f7b50e831c 2024-05-19T19:21:27.4220991Z deleted: sha256:a17532a675e4ec56db1fc54e2f285986b4333c89eaed6520edb8c85209ef76ea 2024-05-19T19:21:27.4221815Z deleted: sha256:55ff1a7a2aa478a7a6c00675e825d39cea2c5955f3f318832fa06cf293ddb137 2024-05-19T19:21:27.4222632Z deleted: sha256:21b7839e8103541d09ad0af189e5ecbf9df69cc798831eada41d8ce869555e94 2024-05-19T19:21:27.4223451Z deleted: sha256:3f93640e4b85ecbb5cb4239715bd521fe05dac28e2e25e53b71b808bbc193590 2024-05-19T19:21:27.4224268Z deleted: sha256:887d227bae679856dd096dad895dbcf90b807f5c82c15ff17ec5a58bf6dac184 2024-05-19T19:21:27.4225112Z deleted: sha256:d2beb8f0ce120b86322d41f423b7f42e64fdb9e4ff5d5d912252849b8146b293 2024-05-19T19:21:27.4226037Z deleted: sha256:e41d38911c997f96258de8880c0251dee4bf259158eb081eafc17f32f8948763 2024-05-19T19:21:27.4226910Z deleted: sha256:b85b0e100a5a15d221dbbccb1e864e367d917d7a79fa199c15db2c1d9a52c92a 2024-05-19T19:21:27.4227730Z deleted: sha256:75258ce71d03c6b0525b6f842fa5efd1738dbe82b883613717fcb707a60621a3 2024-05-19T19:21:27.4228600Z deleted: sha256:a12c28acec35eb76da2791220f40353777e633b6cd7d4fd187c3954e03406c46 2024-05-19T19:21:27.4229432Z deleted: sha256:eaf53ac56a05ff328d594fda97812e9db215cf74b87965669e54d4c6c1669cb3 2024-05-19T19:21:27.4230256Z deleted: sha256:795e72492c71c7cce01ac9e0408bc22fa8f7da1e41614874b812afc49c93fabf 2024-05-19T19:21:27.4231091Z deleted: sha256:1d1e16daa70a0b1c04b023ebbbe56366e3b0cbb9147851a544b43581d66ab50c 2024-05-19T19:21:27.4231912Z deleted: sha256:7752b0e90516dde8d6b7499bd8261515a2eb4063ca5d381313ea697291402d42 2024-05-19T19:21:27.4232739Z deleted: sha256:13082c1d2da03c75509cd19b1e7d211d98b359b41d53223e861fea8ec6be2366 2024-05-19T19:21:27.4233768Z deleted: sha256:fef51b8c2cf2579b04f0d82fba989e43cdd93a484741e14588a942c728328c80 2024-05-19T19:21:27.4234590Z deleted: sha256:cfd02989b5663a4fc04e8468da08dbb6202dd256e851a6e30e7370818fa65830 2024-05-19T19:21:27.4235403Z deleted: sha256:7b3853f547cd41caf732c9bf113d7f11db7fb026f979a50cf7a56ab673d8a2d9 2024-05-19T19:21:27.4236240Z deleted: sha256:8eb59cbbd7ea753bff3586c088340d20ca32887792f3f68c5b2da7d57d11447c 2024-05-19T19:21:27.4237081Z deleted: sha256:d4eb9793eb3f33e1fc18c62f8a4765625d5e2accaffffeb52543799bfe45315a 2024-05-19T19:21:27.4237920Z deleted: sha256:9b388bd2595f78854fce5ceb1a6c673a7bf3f50f84f6350351171599fcad68c4 2024-05-19T19:21:27.4238732Z deleted: sha256:1443f9602fff883313532012c5506b56993ec95031e611cace0b38b3e215413b 2024-05-19T19:21:27.4239512Z deleted: sha256:7d5157e7800e8039c647490628f0b8f64c7ed0017070e69ed1200ae786c318d2 2024-05-19T19:21:27.4240308Z deleted: sha256:3b8e4334d22e46371d21e1d80991345f300244c4932c1687b2723408f8fe213c 2024-05-19T19:21:27.4241200Z deleted: sha256:bc9b902883a611c9653e47dfcdce6f88287f0ce083995cd17612e1ee52d02f45 2024-05-19T19:21:27.4242028Z deleted: sha256:87241399dcaeb6b49174c5033c7aff0f1e914591bd2a727a80fcd0e453b1bdbc 2024-05-19T19:21:27.4242843Z deleted: sha256:9b60d0bfab86b98267697851c8871dae33fc146608b8af9764f5e3d3e459c75b 2024-05-19T19:21:27.4243674Z deleted: sha256:ad76dc292ef5f88364d52bdd2e83680e697b28b2f6df337cd9c829ab007217e1 2024-05-19T19:21:27.4244509Z deleted: sha256:7a4baeb5479a531e458c9fa438d16143efc760acf3c997b1fed3129f2d8d2fc7 2024-05-19T19:21:27.4245486Z deleted: sha256:e682379b3ae2d005fd368b974d3314ece2598d1a6928ae7341bb89f3c79e8e03 2024-05-19T19:21:27.4246312Z deleted: sha256:d8d802f39c7d9d7ddd8f5acef34ee7e07fac7f84fa106255d5228d659daf1073 2024-05-19T19:21:27.4247153Z deleted: sha256:9f893b39e0b2a09fdee21eec466f95f637ccd2e77ab254c00eeca60d4f1a4671 2024-05-19T19:21:27.4248004Z deleted: sha256:01d8e759bdd0bbea54aea263352ffd962e77eeab5d109a7fa8e279f2efdb6719 2024-05-19T19:21:27.4248861Z deleted: sha256:cd0863a43a1691ccbba624e9d63b85a341adb8bc16335dec1291e5a04147b040 2024-05-19T19:21:27.4249692Z deleted: sha256:5fec222edb2dc032d9a5203274fbd944dfb9494597fbb76b063b89ac7e84273f 2024-05-19T19:21:27.4250507Z deleted: sha256:742f8005153dd61c5f7718e2b2a8b3f934780866ef56f7ca11a48088dc638b97 2024-05-19T19:21:27.4251325Z deleted: sha256:e670dd8df73c884bf8257489df70a65828a55222a493f717eca2b9df1f3e41fa 2024-05-19T19:21:27.4252171Z deleted: sha256:e09fa8af11b0b73b35ad8f1f6ec0f0f0bf8a05ed64aa9d530377ddd2e1f94b56 2024-05-19T19:21:27.4252998Z deleted: sha256:02c8d457f61de51676900b87bd530f776b761f2b8dad7f345152b256d7bb7864 2024-05-19T19:21:27.4253832Z deleted: sha256:433544e4353c9e0269dc71fa16c00ddcecdcd5676ed0df618c7b51ac78e23053 2024-05-19T19:21:27.4254682Z deleted: sha256:fc24af9fd5121adfbc7b75df0abb1ef8c2d3868f33b7b6b1e8bf4e214fac4bd6 2024-05-19T19:21:27.4255526Z deleted: sha256:ce1492ad94b71204f8b3fb8903d0ad48a96652635e7fc9347e430639ca15a737 2024-05-19T19:21:27.4256348Z deleted: sha256:96f2ae9435e6e12f68a88dc404c8079b2796639589970c34894a86ccff3f3732 2024-05-19T19:21:27.4257169Z deleted: sha256:5310d2bc0952e7c6d8cbade7ecafd88ce3a04d9da303fa6419583e13a02b58a9 2024-05-19T19:21:27.4258022Z deleted: sha256:099076d608a55a027d60f5b572a3991a8d690a01500acf2c696933a7e6d38650 2024-05-19T19:21:27.4258884Z deleted: sha256:862b05d0f8f7e3dff2d04bd49784a9901f4e22c5dd6544ae72eb3be05e23fffe 2024-05-19T19:21:27.4259718Z deleted: sha256:f7a888e5f7904827f6d71cebcba5169dbc7a78b20f228b9b9b625f38e0c52f24 2024-05-19T19:21:27.4260545Z deleted: sha256:ab9f75a4a7a636aae480f630eaddc05133dda64f1c42445b617c97d3d209991c 2024-05-19T19:21:27.4261391Z deleted: sha256:56dda0c9cee64db5c10d5dd5f08c6f6707263f5ee98e8a075d635e5be996e855 2024-05-19T19:21:27.4262230Z deleted: sha256:5faf9c0a9efe4675ecd21a4ec417d51077d5e75da9e673161a94e7d6cd43f92c 2024-05-19T19:21:27.4262729Z 2024-05-19T19:21:27.4262894Z Total reclaimed space: 37.23GB 2024-05-19T19:21:27.4295647Z ##[group]Run set +e 2024-05-19T19:21:27.4296036Z set +e 2024-05-19T19:21:27.4296427Z set -x 2024-05-19T19:21:27.4296712Z  2024-05-19T19:21:27.4296995Z nvidia-smi 2024-05-19T19:21:27.4297612Z # NB: Surprisingly, nvidia-smi command returns successfully with return code 0 even in 2024-05-19T19:21:27.4298584Z # the case where the driver has already crashed as it still can get the driver version 2024-05-19T19:21:27.4299534Z # and some basic information like the bus ID. However, the rest of the information 2024-05-19T19:21:27.4300258Z # would be missing (ERR!), for example: 2024-05-19T19:21:27.4300706Z # 2024-05-19T19:21:27.4301130Z # +-----------------------------------------------------------------------------+ 2024-05-19T19:21:27.4301888Z # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | 2024-05-19T19:21:27.4302665Z # |-------------------------------+----------------------+----------------------+ 2024-05-19T19:21:27.4303412Z # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-05-19T19:21:27.4304273Z # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | 2024-05-19T19:21:27.4305012Z # | | | MIG M. | 2024-05-19T19:21:27.4305604Z # |===============================+======================+======================| 2024-05-19T19:21:27.4306248Z # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | 2024-05-19T19:21:27.4306988Z # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | 2024-05-19T19:21:27.4307684Z # | | | ERR! | 2024-05-19T19:21:27.4308311Z # +-------------------------------+----------------------+----------------------+ 2024-05-19T19:21:27.4308836Z # 2024-05-19T19:21:27.4309256Z # +-----------------------------------------------------------------------------+ 2024-05-19T19:21:27.4309913Z # | Processes: | 2024-05-19T19:21:27.4310636Z # | GPU GI CI PID Type Process name GPU Memory | 2024-05-19T19:21:27.4311360Z # | ID ID Usage | 2024-05-19T19:21:27.4311966Z # |=============================================================================| 2024-05-19T19:21:27.4312578Z # +-----------------------------------------------------------------------------+ 2024-05-19T19:21:27.4313149Z # 2024-05-19T19:21:27.4313699Z # This should be reported as a failure instead as it will guarantee to fail when 2024-05-19T19:21:27.4314418Z # Docker tries to run with --gpus all 2024-05-19T19:21:27.4314859Z # 2024-05-19T19:21:27.4315385Z # So, the correct check here is to query one of the missing piece of info like 2024-05-19T19:21:27.4316161Z # GPU name, so that the command can fail accordingly 2024-05-19T19:21:27.4316848Z nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-05-19T19:21:27.4317420Z NVIDIA_SMI_STATUS=$? 2024-05-19T19:21:27.4317782Z  2024-05-19T19:21:27.4318396Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2024-05-19T19:21:27.4319304Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2024-05-19T19:21:27.4320129Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2024-05-19T19:21:27.4320828Z  .github/scripts/stop_runner_service.sh 2024-05-19T19:21:27.4321293Z fi 2024-05-19T19:21:27.4321579Z  2024-05-19T19:21:27.4322220Z # For runner with multiple GPUs, we also want to confirm that the number of GPUs are the 2024-05-19T19:21:27.4323208Z # power of 2, i.e. 1, 2, 4, or 8. This is to avoid flaky test issue when one GPU fails 2024-05-19T19:21:27.4323995Z # https://github.com/pytorch/test-infra/issues/4000 2024-05-19T19:21:27.4324726Z GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) 2024-05-19T19:21:27.4325208Z NVIDIA_SMI_STATUS=$? 2024-05-19T19:21:27.4325566Z  2024-05-19T19:21:27.4326196Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2024-05-19T19:21:27.4327110Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2024-05-19T19:21:27.4327943Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2024-05-19T19:21:27.4328655Z  .github/scripts/stop_runner_service.sh 2024-05-19T19:21:27.4329119Z fi 2024-05-19T19:21:27.4329425Z  2024-05-19T19:21:27.4329801Z # Check the GPU count to be a power of 2 2024-05-19T19:21:27.4330658Z if [ "$GPU_COUNT" -le 8 ] && [ "$GPU_COUNT" -ne 1 ] && [ "$GPU_COUNT" -ne 2 ] && [ "$GPU_COUNT" -ne 4 ] && [ "$GPU_COUNT" -ne 8 ]; then 2024-05-19T19:21:27.4331755Z  echo "NVIDIA driver detects $GPU_COUNT GPUs. The runner has a broken GPU, shutting it down..." 2024-05-19T19:21:27.4332559Z  .github/scripts/stop_runner_service.sh 2024-05-19T19:21:27.4333021Z fi 2024-05-19T19:21:27.4340746Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T19:21:27.4341236Z env: 2024-05-19T19:21:27.4341517Z GIT_DEFAULT_BRANCH: main 2024-05-19T19:21:27.4341973Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T19:21:27.4342719Z DOCKER_CONTAINER_ID: 8e0cfbfda7b9b65fe7bac7a0896c965b71d3fe5387e9d2e7a49bc38dc16e86e2 2024-05-19T19:21:27.4343495Z RUNNER_WORKSPACE: /home/ec2-user/actions-runner/_work/pytorch 2024-05-19T19:21:27.4344020Z ##[endgroup] 2024-05-19T19:21:27.4363838Z + nvidia-smi 2024-05-19T19:21:29.0108420Z Sun May 19 19:21:29 2024 2024-05-19T19:21:29.0109231Z +-----------------------------------------------------------------------------------------+ 2024-05-19T19:21:29.0110061Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-05-19T19:21:29.0110842Z |-----------------------------------------+------------------------+----------------------+ 2024-05-19T19:21:29.0111640Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-05-19T19:21:29.0112525Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-05-19T19:21:29.0113276Z | | | MIG M. | 2024-05-19T19:21:29.0113818Z |=========================================+========================+======================| 2024-05-19T19:21:29.0251391Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-05-19T19:21:29.0252102Z | 0% 29C P0 58W / 300W | 0MiB / 23028MiB | 5% Default | 2024-05-19T19:21:29.0252741Z | | | N/A | 2024-05-19T19:21:29.0253407Z +-----------------------------------------+------------------------+----------------------+ 2024-05-19T19:21:29.0254064Z 2024-05-19T19:21:29.0254833Z +-----------------------------------------------------------------------------------------+ 2024-05-19T19:21:29.0255449Z | Processes: | 2024-05-19T19:21:29.0256146Z | GPU GI CI PID Type Process name GPU Memory | 2024-05-19T19:21:29.0257074Z | ID ID Usage | 2024-05-19T19:21:29.0257659Z |=========================================================================================| 2024-05-19T19:21:29.0258416Z | No running processes found | 2024-05-19T19:21:29.0259165Z +-----------------------------------------------------------------------------------------+ 2024-05-19T19:21:29.6412744Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-05-19T19:21:31.2120118Z NVIDIA A10G 2024-05-19T19:21:31.6479405Z + NVIDIA_SMI_STATUS=0 2024-05-19T19:21:31.6479966Z + '[' 0 -ne 0 ']' 2024-05-19T19:21:31.6483159Z ++ nvidia-smi --list-gpus 2024-05-19T19:21:31.6483521Z ++ wc -l 2024-05-19T19:21:33.6873106Z + GPU_COUNT=1 2024-05-19T19:21:33.6873586Z + NVIDIA_SMI_STATUS=0 2024-05-19T19:21:33.6874208Z + '[' 0 -ne 0 ']' 2024-05-19T19:21:33.6874554Z + '[' 1 -le 8 ']' 2024-05-19T19:21:33.6874885Z + '[' 1 -ne 1 ']' 2024-05-19T19:21:33.6929658Z Post job cleanup. 2024-05-19T19:21:33.6978066Z Post job cleanup. 2024-05-19T19:21:33.7793384Z [command]/usr/bin/git version 2024-05-19T19:21:33.7833912Z git version 2.40.1 2024-05-19T19:21:33.7877621Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/8ca1b227-f53c-41c7-b4ea-e78e720d4634' before making global git config changes 2024-05-19T19:21:33.7879478Z Adding repository directory to the temporary git global config as a safe directory 2024-05-19T19:21:33.7884483Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T19:21:33.7913981Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-05-19T19:21:33.7942772Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-05-19T19:21:33.8154112Z Entering 'android/libs/fbjni' 2024-05-19T19:21:33.8187662Z Entering 'third_party/FP16' 2024-05-19T19:21:33.8223283Z Entering 'third_party/FXdiv' 2024-05-19T19:21:33.8258922Z Entering 'third_party/NNPACK' 2024-05-19T19:21:33.8294036Z Entering 'third_party/QNNPACK' 2024-05-19T19:21:33.8329495Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T19:21:33.8362276Z Entering 'third_party/XNNPACK' 2024-05-19T19:21:33.8409126Z Entering 'third_party/benchmark' 2024-05-19T19:21:33.8440508Z Entering 'third_party/cpp-httplib' 2024-05-19T19:21:33.8476272Z Entering 'third_party/cpuinfo' 2024-05-19T19:21:33.8512131Z Entering 'third_party/cudnn_frontend' 2024-05-19T19:21:33.8549861Z Entering 'third_party/cutlass' 2024-05-19T19:21:33.8590710Z Entering 'third_party/eigen' 2024-05-19T19:21:33.8629132Z Entering 'third_party/fbgemm' 2024-05-19T19:21:33.8664749Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T19:21:33.8699655Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T19:21:33.8732142Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T19:21:33.8770324Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T19:21:33.8803595Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T19:21:33.8839785Z Entering 'third_party/flatbuffers' 2024-05-19T19:21:33.8877132Z Entering 'third_party/fmt' 2024-05-19T19:21:33.8912897Z Entering 'third_party/foxi' 2024-05-19T19:21:33.8946102Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T19:21:33.8981697Z Entering 'third_party/gloo' 2024-05-19T19:21:33.9016391Z Entering 'third_party/googletest' 2024-05-19T19:21:33.9053222Z Entering 'third_party/ideep' 2024-05-19T19:21:33.9088502Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T19:21:33.9126289Z Entering 'third_party/ios-cmake' 2024-05-19T19:21:33.9159208Z Entering 'third_party/ittapi' 2024-05-19T19:21:33.9195081Z Entering 'third_party/kineto' 2024-05-19T19:21:33.9231697Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T19:21:33.9266700Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T19:21:33.9301113Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T19:21:33.9335931Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T19:21:33.9370715Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T19:21:33.9405624Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T19:21:33.9441413Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T19:21:33.9475798Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T19:21:33.9510583Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T19:21:33.9544707Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T19:21:33.9580201Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T19:21:33.9612460Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T19:21:33.9647929Z Entering 'third_party/mimalloc' 2024-05-19T19:21:33.9684348Z Entering 'third_party/nccl/nccl' 2024-05-19T19:21:33.9720798Z Entering 'third_party/neon2sse' 2024-05-19T19:21:33.9757335Z Entering 'third_party/nlohmann' 2024-05-19T19:21:33.9794102Z Entering 'third_party/onnx' 2024-05-19T19:21:33.9841716Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T19:21:33.9877001Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T19:21:33.9914461Z Entering 'third_party/onnx-tensorrt' 2024-05-19T19:21:33.9947539Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T19:21:33.9984552Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T19:21:34.0019219Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T19:21:34.0051750Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T19:21:34.0088669Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T19:21:34.0124186Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T19:21:34.0158969Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T19:21:34.0192545Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T19:21:34.0225805Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T19:21:34.0259106Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T19:21:34.0290922Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T19:21:34.0324844Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T19:21:34.0358123Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T19:21:34.0394597Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T19:21:34.0431276Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T19:21:34.0478978Z Entering 'third_party/pocketfft' 2024-05-19T19:21:34.0515657Z Entering 'third_party/protobuf' 2024-05-19T19:21:34.0553386Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T19:21:34.0588077Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T19:21:34.0622101Z Entering 'third_party/psimd' 2024-05-19T19:21:34.0657671Z Entering 'third_party/pthreadpool' 2024-05-19T19:21:34.0692362Z Entering 'third_party/pybind11' 2024-05-19T19:21:34.0726656Z Entering 'third_party/python-peachpy' 2024-05-19T19:21:34.0758178Z Entering 'third_party/sleef' 2024-05-19T19:21:34.0792585Z Entering 'third_party/tbb' 2024-05-19T19:21:34.0829564Z Entering 'third_party/tensorpipe' 2024-05-19T19:21:34.0862547Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T19:21:34.0896030Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T19:21:34.0931148Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T19:21:34.0965051Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T19:21:34.0998845Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T19:21:34.1050040Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-05-19T19:21:34.1071982Z http.https://github.com/.extraheader 2024-05-19T19:21:34.1080952Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-05-19T19:21:34.1110776Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-05-19T19:21:34.1314617Z Entering 'android/libs/fbjni' 2024-05-19T19:21:34.1338905Z http.https://github.com/.extraheader 2024-05-19T19:21:34.1362811Z Entering 'third_party/FP16' 2024-05-19T19:21:34.1383904Z http.https://github.com/.extraheader 2024-05-19T19:21:34.1407304Z Entering 'third_party/FXdiv' 2024-05-19T19:21:34.1428913Z http.https://github.com/.extraheader 2024-05-19T19:21:34.1451690Z Entering 'third_party/NNPACK' 2024-05-19T19:21:34.1473085Z http.https://github.com/.extraheader 2024-05-19T19:21:34.1497057Z Entering 'third_party/QNNPACK' 2024-05-19T19:21:34.1517088Z http.https://github.com/.extraheader 2024-05-19T19:21:34.1542024Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T19:21:34.1563020Z http.https://github.com/.extraheader 2024-05-19T19:21:34.1586043Z Entering 'third_party/XNNPACK' 2024-05-19T19:21:34.1609113Z http.https://github.com/.extraheader 2024-05-19T19:21:34.1641799Z Entering 'third_party/benchmark' 2024-05-19T19:21:34.1663054Z http.https://github.com/.extraheader 2024-05-19T19:21:34.1686724Z Entering 'third_party/cpp-httplib' 2024-05-19T19:21:34.1707879Z http.https://github.com/.extraheader 2024-05-19T19:21:34.1731587Z Entering 'third_party/cpuinfo' 2024-05-19T19:21:34.1753129Z http.https://github.com/.extraheader 2024-05-19T19:21:34.1776016Z Entering 'third_party/cudnn_frontend' 2024-05-19T19:21:34.1797790Z http.https://github.com/.extraheader 2024-05-19T19:21:34.1821678Z Entering 'third_party/cutlass' 2024-05-19T19:21:34.1842655Z http.https://github.com/.extraheader 2024-05-19T19:21:34.1871723Z Entering 'third_party/eigen' 2024-05-19T19:21:34.1893655Z http.https://github.com/.extraheader 2024-05-19T19:21:34.1918968Z Entering 'third_party/fbgemm' 2024-05-19T19:21:34.1941147Z http.https://github.com/.extraheader 2024-05-19T19:21:34.1963983Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T19:21:34.1985125Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2007926Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T19:21:34.2028452Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2052829Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T19:21:34.2073826Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2101753Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T19:21:34.2122745Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2145361Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T19:21:34.2165864Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2189936Z Entering 'third_party/flatbuffers' 2024-05-19T19:21:34.2210511Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2235284Z Entering 'third_party/fmt' 2024-05-19T19:21:34.2257702Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2281143Z Entering 'third_party/foxi' 2024-05-19T19:21:34.2303109Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2328256Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T19:21:34.2349221Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2372356Z Entering 'third_party/gloo' 2024-05-19T19:21:34.2393837Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2416992Z Entering 'third_party/googletest' 2024-05-19T19:21:34.2438602Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2462698Z Entering 'third_party/ideep' 2024-05-19T19:21:34.2484141Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2506603Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T19:21:34.2526382Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2554688Z Entering 'third_party/ios-cmake' 2024-05-19T19:21:34.2577406Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2600571Z Entering 'third_party/ittapi' 2024-05-19T19:21:34.2622446Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2644905Z Entering 'third_party/kineto' 2024-05-19T19:21:34.2665817Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2690975Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T19:21:34.2711573Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2734133Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T19:21:34.2754018Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2778169Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T19:21:34.2797691Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2822616Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T19:21:34.2843237Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2865974Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T19:21:34.2887475Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2909135Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T19:21:34.2931484Z http.https://github.com/.extraheader 2024-05-19T19:21:34.2955091Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T19:21:34.2976412Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3000112Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T19:21:34.3021696Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3045374Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T19:21:34.3066184Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3090057Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T19:21:34.3111218Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3134322Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T19:21:34.3155245Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3178328Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T19:21:34.3198306Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3221727Z Entering 'third_party/mimalloc' 2024-05-19T19:21:34.3243726Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3266942Z Entering 'third_party/nccl/nccl' 2024-05-19T19:21:34.3288634Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3311874Z Entering 'third_party/neon2sse' 2024-05-19T19:21:34.3333341Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3355787Z Entering 'third_party/nlohmann' 2024-05-19T19:21:34.3376654Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3399624Z Entering 'third_party/onnx' 2024-05-19T19:21:34.3420724Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3456847Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T19:21:34.3478048Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3501760Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T19:21:34.3524017Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3549131Z Entering 'third_party/onnx-tensorrt' 2024-05-19T19:21:34.3570393Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3593210Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T19:21:34.3615229Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3641997Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T19:21:34.3662800Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3685542Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T19:21:34.3706727Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3728867Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T19:21:34.3749533Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3777086Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T19:21:34.3799061Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3825196Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T19:21:34.3846195Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3868648Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T19:21:34.3890019Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3912718Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T19:21:34.3934129Z http.https://github.com/.extraheader 2024-05-19T19:21:34.3957983Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T19:21:34.3979096Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4003011Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T19:21:34.4024482Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4048760Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T19:21:34.4069048Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4092616Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T19:21:34.4114195Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4136605Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T19:21:34.4157532Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4182237Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T19:21:34.4203097Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4226988Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T19:21:34.4248368Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4287308Z Entering 'third_party/pocketfft' 2024-05-19T19:21:34.4308581Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4331529Z Entering 'third_party/protobuf' 2024-05-19T19:21:34.4352323Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4375709Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T19:21:34.4396598Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4420556Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T19:21:34.4441804Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4467171Z Entering 'third_party/psimd' 2024-05-19T19:21:34.4488521Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4511617Z Entering 'third_party/pthreadpool' 2024-05-19T19:21:34.4532311Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4556655Z Entering 'third_party/pybind11' 2024-05-19T19:21:34.4578235Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4599033Z Entering 'third_party/python-peachpy' 2024-05-19T19:21:34.4620155Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4645991Z Entering 'third_party/sleef' 2024-05-19T19:21:34.4665963Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4688726Z Entering 'third_party/tbb' 2024-05-19T19:21:34.4709093Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4732436Z Entering 'third_party/tensorpipe' 2024-05-19T19:21:34.4753847Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4776852Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T19:21:34.4796881Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4818779Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T19:21:34.4839645Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4863001Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T19:21:34.4883150Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4906042Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T19:21:34.4927137Z http.https://github.com/.extraheader 2024-05-19T19:21:34.4949634Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T19:21:34.4971766Z http.https://github.com/.extraheader 2024-05-19T19:21:34.5219955Z Cleaning up orphan processes