2024-06-26T07:04:14.5122368Z Current runner version: '2.317.0' 2024-06-26T07:04:14.5128386Z Runner name: 'i-0f01143c0858f0e6b' 2024-06-26T07:04:14.5129287Z Runner group name: 'Default' 2024-06-26T07:04:14.5130156Z Machine name: 'ip-10-0-21-121' 2024-06-26T07:04:14.5134841Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T07:04:14.5136819Z Actions: read 2024-06-26T07:04:14.5137298Z Attestations: read 2024-06-26T07:04:14.5137780Z Checks: read 2024-06-26T07:04:14.5138267Z Contents: read 2024-06-26T07:04:14.5138741Z Deployments: read 2024-06-26T07:04:14.5139244Z Discussions: read 2024-06-26T07:04:14.5139749Z Issues: read 2024-06-26T07:04:14.5140198Z Metadata: read 2024-06-26T07:04:14.5140636Z Packages: read 2024-06-26T07:04:14.5141122Z Pages: read 2024-06-26T07:04:14.5141579Z PullRequests: read 2024-06-26T07:04:14.5142156Z RepositoryProjects: read 2024-06-26T07:04:14.5142695Z SecurityEvents: read 2024-06-26T07:04:14.5143195Z Statuses: read 2024-06-26T07:04:14.5143667Z ##[endgroup] 2024-06-26T07:04:14.5146668Z Secret source: Actions 2024-06-26T07:04:14.5147470Z Prepare workflow directory 2024-06-26T07:04:14.8866226Z Prepare all required actions 2024-06-26T07:04:14.9052907Z Getting action download info 2024-06-26T07:04:15.0735985Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T07:04:15.3731827Z Download action repository 'pytorch/pytorch@main' (SHA:c422a9549d2b320b47a9391f6cbcf592b2876c88) 2024-06-26T07:04:18.1866274Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-26T07:04:18.3229472Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T07:04:18.6086221Z Getting action download info 2024-06-26T07:04:18.7054522Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T07:04:18.8858057Z Getting action download info 2024-06-26T07:04:18.9874874Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T07:04:19.1421233Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/129470 (b8c4c54d347aa776934c60784e35936878ef18dc) 2024-06-26T07:04:19.1423368Z ##[group] Inputs 2024-06-26T07:04:19.1423856Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:04:19.1430754Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:04:19.1438426Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:19.1439599Z sync-tag: 2024-06-26T07:04:19.1440413Z timeout-minutes: 240 2024-06-26T07:04:19.1440764Z use-gha: 2024-06-26T07:04:19.1441066Z dashboard-tag: 2024-06-26T07:04:19.1441391Z s3-bucket: gha-artifacts 2024-06-26T07:04:19.1441765Z aws-role-to-assume: 2024-06-26T07:04:19.1442162Z ##[endgroup] 2024-06-26T07:04:19.1443224Z Complete job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:04:19.2012850Z A job started hook has been configured by the self-hosted runner administrator 2024-06-26T07:04:19.2148446Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T07:04:19.2158601Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:19.2159134Z ##[endgroup] 2024-06-26T07:04:19.9204984Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-26T07:04:19.9205550Z with: 2024-06-26T07:04:19.9206157Z github-secret: *** 2024-06-26T07:04:19.9207098Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-26T07:04:19.9208096Z activate-with-label: false 2024-06-26T07:04:19.9208633Z label: with-ssh 2024-06-26T07:04:19.9208974Z remove-existing-keys: true 2024-06-26T07:04:19.9209341Z fail-silently: true 2024-06-26T07:04:19.9209667Z env: 2024-06-26T07:04:19.9209951Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:19.9210303Z ##[endgroup] 2024-06-26T07:04:20.0065243Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-26T07:04:20.0066265Z ciflow reference detected, attempting to extract PR number 2024-06-26T07:04:20.3161465Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-06-26T07:04:20.3868382Z No SSH keys found for user pytorch-bot[bot] 2024-06-26T07:04:20.3869228Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T07:04:20.4518123Z No SSH keys found for user leslie-fang-intel 2024-06-26T07:04:20.4621148Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T07:04:20.4621709Z with: 2024-06-26T07:04:20.4622003Z submodules: recursive 2024-06-26T07:04:20.4622345Z fetch-depth: 0 2024-06-26T07:04:20.4622649Z env: 2024-06-26T07:04:20.4622929Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:20.4623275Z ##[endgroup] 2024-06-26T07:04:20.4809891Z ##[group]Run retry () { 2024-06-26T07:04:20.4810380Z retry () { 2024-06-26T07:04:20.4810966Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T07:04:20.4811598Z } 2024-06-26T07:04:20.4812017Z echo "${GITHUB_WORKSPACE}" 2024-06-26T07:04:20.4812481Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T07:04:20.4813076Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:04:20.4813616Z else 2024-06-26T07:04:20.4814022Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:04:20.4814495Z fi 2024-06-26T07:04:20.4814971Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T07:04:20.4823026Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:20.4823571Z env: 2024-06-26T07:04:20.4823925Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:20.4824364Z NO_SUDO: 2024-06-26T07:04:20.4824692Z ##[endgroup] 2024-06-26T07:04:20.4851441Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:04:22.6825449Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T07:04:22.6825893Z with: 2024-06-26T07:04:22.6826217Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:04:22.6826666Z fetch-depth: 0 2024-06-26T07:04:22.6826984Z submodules: recursive 2024-06-26T07:04:22.6827324Z quiet-checkout: true 2024-06-26T07:04:22.6827801Z repository: pytorch/pytorch 2024-06-26T07:04:22.6828287Z token: *** 2024-06-26T07:04:22.6828588Z ssh-strict: true 2024-06-26T07:04:22.6828915Z persist-credentials: true 2024-06-26T07:04:22.6829279Z clean: true 2024-06-26T07:04:22.6829622Z sparse-checkout-cone-mode: true 2024-06-26T07:04:22.6830021Z lfs: false 2024-06-26T07:04:22.6830324Z set-safe-directory: true 2024-06-26T07:04:22.6830674Z env: 2024-06-26T07:04:22.6830954Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:22.6831299Z ##[endgroup] 2024-06-26T07:04:22.7732326Z Syncing repository: pytorch/pytorch 2024-06-26T07:04:22.7733937Z ##[group]Getting Git version info 2024-06-26T07:04:22.7734706Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:04:22.7735646Z [command]/usr/bin/git version 2024-06-26T07:04:22.7736029Z git version 2.40.1 2024-06-26T07:04:22.7737452Z ##[endgroup] 2024-06-26T07:04:22.7748721Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/53f3d4b1-43cf-49df-9550-b731849119f8' before making global git config changes 2024-06-26T07:04:22.7749958Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T07:04:22.7751090Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:04:22.7782121Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:04:22.7785985Z ##[group]Initializing the repository 2024-06-26T07:04:22.7788885Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:04:22.7809453Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T07:04:22.7810634Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T07:04:22.7811687Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T07:04:22.7812387Z hint: 2024-06-26T07:04:22.7813027Z hint: git config --global init.defaultBranch 2024-06-26T07:04:22.7813532Z hint: 2024-06-26T07:04:22.7814044Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T07:04:22.7814895Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T07:04:22.7815818Z hint: 2024-06-26T07:04:22.7816305Z hint: git branch -m 2024-06-26T07:04:22.7817064Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T07:04:22.7819276Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T07:04:22.7841621Z ##[endgroup] 2024-06-26T07:04:22.7842255Z ##[group]Disabling automatic garbage collection 2024-06-26T07:04:22.7844251Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T07:04:22.7865752Z ##[endgroup] 2024-06-26T07:04:22.7866290Z ##[group]Setting up auth 2024-06-26T07:04:22.7871632Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T07:04:22.7894549Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T07:04:22.8089241Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T07:04:22.8114621Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T07:04:22.8307827Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:04:22.8379211Z ##[endgroup] 2024-06-26T07:04:22.8379784Z ##[group]Fetching the repository 2024-06-26T07:04:22.8385450Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T07:04:24.9796135Z remote: Enumerating objects: 987612 2024-06-26T07:04:24.9797633Z remote: Enumerating objects: 990209, done. 2024-06-26T07:04:24.9801278Z remote: Counting objects: 0% (1/2597) 2024-06-26T07:04:24.9805025Z remote: Counting objects: 1% (26/2597) 2024-06-26T07:04:24.9805677Z remote: Counting objects: 2% (52/2597) 2024-06-26T07:04:24.9806192Z remote: Counting objects: 3% (78/2597) 2024-06-26T07:04:24.9806804Z remote: Counting objects: 4% (104/2597) 2024-06-26T07:04:24.9807382Z remote: Counting objects: 5% (130/2597) 2024-06-26T07:04:24.9808055Z remote: Counting objects: 6% (156/2597) 2024-06-26T07:04:24.9814812Z remote: Counting objects: 7% (182/2597) 2024-06-26T07:04:24.9815438Z remote: Counting objects: 8% (208/2597) 2024-06-26T07:04:24.9816035Z remote: Counting objects: 9% (234/2597) 2024-06-26T07:04:24.9816541Z remote: Counting objects: 10% (260/2597) 2024-06-26T07:04:24.9817044Z remote: Counting objects: 11% (286/2597) 2024-06-26T07:04:24.9817552Z remote: Counting objects: 12% (312/2597) 2024-06-26T07:04:24.9818041Z remote: Counting objects: 13% (338/2597) 2024-06-26T07:04:24.9818538Z remote: Counting objects: 14% (364/2597) 2024-06-26T07:04:24.9819045Z remote: Counting objects: 15% (390/2597) 2024-06-26T07:04:24.9826015Z remote: Counting objects: 16% (416/2597) 2024-06-26T07:04:24.9826662Z remote: Counting objects: 17% (442/2597) 2024-06-26T07:04:24.9827163Z remote: Counting objects: 18% (468/2597) 2024-06-26T07:04:24.9827664Z remote: Counting objects: 19% (494/2597) 2024-06-26T07:04:24.9828162Z remote: Counting objects: 20% (520/2597) 2024-06-26T07:04:24.9828658Z remote: Counting objects: 21% (546/2597) 2024-06-26T07:04:24.9829149Z remote: Counting objects: 22% (572/2597) 2024-06-26T07:04:24.9829651Z remote: Counting objects: 23% (598/2597) 2024-06-26T07:04:24.9830151Z remote: Counting objects: 24% (624/2597) 2024-06-26T07:04:24.9830660Z remote: Counting objects: 25% (650/2597) 2024-06-26T07:04:24.9831151Z remote: Counting objects: 26% (676/2597) 2024-06-26T07:04:24.9836716Z remote: Counting objects: 27% (702/2597) 2024-06-26T07:04:24.9837605Z remote: Counting objects: 28% (728/2597) 2024-06-26T07:04:24.9838274Z remote: Counting objects: 29% (754/2597) 2024-06-26T07:04:24.9838911Z remote: Counting objects: 30% (780/2597) 2024-06-26T07:04:24.9839542Z remote: Counting objects: 31% (806/2597) 2024-06-26T07:04:24.9840045Z remote: Counting objects: 32% (832/2597) 2024-06-26T07:04:24.9840538Z remote: Counting objects: 33% (858/2597) 2024-06-26T07:04:24.9841057Z remote: Counting objects: 34% (883/2597) 2024-06-26T07:04:24.9841659Z remote: Counting objects: 35% (909/2597) 2024-06-26T07:04:24.9842359Z remote: Counting objects: 36% (935/2597) 2024-06-26T07:04:24.9842936Z remote: Counting objects: 37% (961/2597) 2024-06-26T07:04:24.9843470Z remote: Counting objects: 38% (987/2597) 2024-06-26T07:04:24.9844071Z remote: Counting objects: 39% (1013/2597) 2024-06-26T07:04:24.9844694Z remote: Counting objects: 40% (1039/2597) 2024-06-26T07:04:24.9845223Z remote: Counting objects: 41% (1065/2597) 2024-06-26T07:04:24.9845736Z remote: Counting objects: 42% (1091/2597) 2024-06-26T07:04:24.9846248Z remote: Counting objects: 43% (1117/2597) 2024-06-26T07:04:24.9846850Z remote: Counting objects: 44% (1143/2597) 2024-06-26T07:04:24.9847513Z remote: Counting objects: 45% (1169/2597) 2024-06-26T07:04:24.9848027Z remote: Counting objects: 46% (1195/2597) 2024-06-26T07:04:24.9848714Z remote: Counting objects: 47% (1221/2597) 2024-06-26T07:04:24.9849234Z remote: Counting objects: 48% (1247/2597) 2024-06-26T07:04:24.9849747Z remote: Counting objects: 49% (1273/2597) 2024-06-26T07:04:24.9850264Z remote: Counting objects: 50% (1299/2597) 2024-06-26T07:04:24.9852943Z remote: Counting objects: 51% (1325/2597) 2024-06-26T07:04:24.9853668Z remote: Counting objects: 52% (1351/2597) 2024-06-26T07:04:24.9854228Z remote: Counting objects: 53% (1377/2597) 2024-06-26T07:04:24.9854881Z remote: Counting objects: 54% (1403/2597) 2024-06-26T07:04:24.9855632Z remote: Counting objects: 55% (1429/2597) 2024-06-26T07:04:24.9856153Z remote: Counting objects: 56% (1455/2597) 2024-06-26T07:04:24.9856673Z remote: Counting objects: 57% (1481/2597) 2024-06-26T07:04:24.9857289Z remote: Counting objects: 58% (1507/2597) 2024-06-26T07:04:24.9857932Z remote: Counting objects: 59% (1533/2597) 2024-06-26T07:04:24.9858567Z remote: Counting objects: 60% (1559/2597) 2024-06-26T07:04:24.9859223Z remote: Counting objects: 61% (1585/2597) 2024-06-26T07:04:24.9859832Z remote: Counting objects: 62% (1611/2597) 2024-06-26T07:04:24.9860395Z remote: Counting objects: 63% (1637/2597) 2024-06-26T07:04:24.9860999Z remote: Counting objects: 64% (1663/2597) 2024-06-26T07:04:24.9861730Z remote: Counting objects: 65% (1689/2597) 2024-06-26T07:04:24.9862478Z remote: Counting objects: 66% (1715/2597) 2024-06-26T07:04:24.9863283Z remote: Counting objects: 67% (1740/2597) 2024-06-26T07:04:24.9864063Z remote: Counting objects: 68% (1766/2597) 2024-06-26T07:04:24.9864828Z remote: Counting objects: 69% (1792/2597) 2024-06-26T07:04:24.9865529Z remote: Counting objects: 70% (1818/2597) 2024-06-26T07:04:24.9866052Z remote: Counting objects: 71% (1844/2597) 2024-06-26T07:04:24.9866582Z remote: Counting objects: 72% (1870/2597) 2024-06-26T07:04:24.9867099Z remote: Counting objects: 73% (1896/2597) 2024-06-26T07:04:24.9867615Z remote: Counting objects: 74% (1922/2597) 2024-06-26T07:04:24.9868135Z remote: Counting objects: 75% (1948/2597) 2024-06-26T07:04:24.9868655Z remote: Counting objects: 76% (1974/2597) 2024-06-26T07:04:24.9869175Z remote: Counting objects: 77% (2000/2597) 2024-06-26T07:04:24.9869694Z remote: Counting objects: 78% (2026/2597) 2024-06-26T07:04:24.9870349Z remote: Counting objects: 79% (2052/2597) 2024-06-26T07:04:24.9870864Z remote: Counting objects: 80% (2078/2597) 2024-06-26T07:04:24.9871383Z remote: Counting objects: 81% (2104/2597) 2024-06-26T07:04:24.9871898Z remote: Counting objects: 82% (2130/2597) 2024-06-26T07:04:24.9872412Z remote: Counting objects: 83% (2156/2597) 2024-06-26T07:04:24.9872922Z remote: Counting objects: 84% (2182/2597) 2024-06-26T07:04:24.9873440Z remote: Counting objects: 85% (2208/2597) 2024-06-26T07:04:24.9873958Z remote: Counting objects: 86% (2234/2597) 2024-06-26T07:04:24.9874472Z remote: Counting objects: 87% (2260/2597) 2024-06-26T07:04:24.9874982Z remote: Counting objects: 88% (2286/2597) 2024-06-26T07:04:24.9875505Z remote: Counting objects: 89% (2312/2597) 2024-06-26T07:04:24.9876021Z remote: Counting objects: 90% (2338/2597) 2024-06-26T07:04:24.9876537Z remote: Counting objects: 91% (2364/2597) 2024-06-26T07:04:24.9877057Z remote: Counting objects: 92% (2390/2597) 2024-06-26T07:04:24.9877573Z remote: Counting objects: 93% (2416/2597) 2024-06-26T07:04:24.9878090Z remote: Counting objects: 94% (2442/2597) 2024-06-26T07:04:24.9878606Z remote: Counting objects: 95% (2468/2597) 2024-06-26T07:04:24.9879113Z remote: Counting objects: 96% (2494/2597) 2024-06-26T07:04:24.9879632Z remote: Counting objects: 97% (2520/2597) 2024-06-26T07:04:24.9880148Z remote: Counting objects: 98% (2546/2597) 2024-06-26T07:04:24.9880665Z remote: Counting objects: 99% (2572/2597) 2024-06-26T07:04:24.9881173Z remote: Counting objects: 100% (2597/2597) 2024-06-26T07:04:24.9881725Z remote: Counting objects: 100% (2597/2597), done. 2024-06-26T07:04:24.9985342Z remote: Compressing objects: 0% (1/1074) 2024-06-26T07:04:25.0374363Z remote: Compressing objects: 1% (11/1074) 2024-06-26T07:04:25.0792305Z remote: Compressing objects: 2% (22/1074) 2024-06-26T07:04:25.1151189Z remote: Compressing objects: 3% (33/1074) 2024-06-26T07:04:25.1355912Z remote: Compressing objects: 4% (43/1074) 2024-06-26T07:04:25.1598825Z remote: Compressing objects: 5% (54/1074) 2024-06-26T07:04:25.1794585Z remote: Compressing objects: 6% (65/1074) 2024-06-26T07:04:25.2173929Z remote: Compressing objects: 7% (76/1074) 2024-06-26T07:04:25.2955651Z remote: Compressing objects: 8% (86/1074) 2024-06-26T07:04:25.3705667Z remote: Compressing objects: 9% (97/1074) 2024-06-26T07:04:25.4077795Z remote: Compressing objects: 10% (108/1074) 2024-06-26T07:04:25.4104571Z remote: Compressing objects: 11% (119/1074) 2024-06-26T07:04:25.4203588Z remote: Compressing objects: 12% (129/1074) 2024-06-26T07:04:25.4281111Z remote: Compressing objects: 13% (140/1074) 2024-06-26T07:04:25.4318455Z remote: Compressing objects: 14% (151/1074) 2024-06-26T07:04:25.4323168Z remote: Compressing objects: 15% (162/1074) 2024-06-26T07:04:25.4379932Z remote: Compressing objects: 16% (172/1074) 2024-06-26T07:04:25.4380485Z remote: Compressing objects: 17% (183/1074) 2024-06-26T07:04:25.4381657Z remote: Compressing objects: 18% (194/1074) 2024-06-26T07:04:25.4393604Z remote: Compressing objects: 19% (205/1074) 2024-06-26T07:04:25.4411698Z remote: Compressing objects: 20% (215/1074) 2024-06-26T07:04:25.4427504Z remote: Compressing objects: 21% (226/1074) 2024-06-26T07:04:25.4440469Z remote: Compressing objects: 22% (237/1074) 2024-06-26T07:04:25.4458696Z remote: Compressing objects: 23% (248/1074) 2024-06-26T07:04:25.4468863Z remote: Compressing objects: 24% (258/1074) 2024-06-26T07:04:25.4477207Z remote: Compressing objects: 25% (269/1074) 2024-06-26T07:04:25.4485497Z remote: Compressing objects: 26% (280/1074) 2024-06-26T07:04:25.4492461Z remote: Compressing objects: 27% (290/1074) 2024-06-26T07:04:25.4494371Z remote: Compressing objects: 28% (301/1074) 2024-06-26T07:04:25.4504906Z remote: Compressing objects: 29% (312/1074) 2024-06-26T07:04:25.4514067Z remote: Compressing objects: 30% (323/1074) 2024-06-26T07:04:25.4516368Z remote: Compressing objects: 31% (333/1074) 2024-06-26T07:04:25.4519412Z remote: Compressing objects: 32% (344/1074) 2024-06-26T07:04:25.4525898Z remote: Compressing objects: 33% (355/1074) 2024-06-26T07:04:25.4537680Z remote: Compressing objects: 34% (366/1074) 2024-06-26T07:04:25.4555320Z remote: Compressing objects: 35% (376/1074) 2024-06-26T07:04:25.4570320Z remote: Compressing objects: 36% (387/1074) 2024-06-26T07:04:25.4585255Z remote: Compressing objects: 37% (398/1074) 2024-06-26T07:04:25.4599569Z remote: Compressing objects: 38% (409/1074) 2024-06-26T07:04:25.4614654Z remote: Compressing objects: 39% (419/1074) 2024-06-26T07:04:25.4636020Z remote: Compressing objects: 40% (430/1074) 2024-06-26T07:04:25.4656897Z remote: Compressing objects: 41% (441/1074) 2024-06-26T07:04:25.4669708Z remote: Compressing objects: 42% (452/1074) 2024-06-26T07:04:25.4677029Z remote: Compressing objects: 43% (462/1074) 2024-06-26T07:04:25.4677586Z remote: Compressing objects: 44% (473/1074) 2024-06-26T07:04:25.4682313Z remote: Compressing objects: 45% (484/1074) 2024-06-26T07:04:25.4687607Z remote: Compressing objects: 46% (495/1074) 2024-06-26T07:04:25.4700010Z remote: Compressing objects: 47% (505/1074) 2024-06-26T07:04:25.4710053Z remote: Compressing objects: 48% (516/1074) 2024-06-26T07:04:25.4713183Z remote: Compressing objects: 49% (527/1074) 2024-06-26T07:04:25.4717764Z remote: Compressing objects: 50% (537/1074) 2024-06-26T07:04:25.4727185Z remote: Compressing objects: 51% (548/1074) 2024-06-26T07:04:25.4740210Z remote: Compressing objects: 52% (559/1074) 2024-06-26T07:04:25.4751123Z remote: Compressing objects: 53% (570/1074) 2024-06-26T07:04:25.4771352Z remote: Compressing objects: 54% (580/1074) 2024-06-26T07:04:25.4779102Z remote: Compressing objects: 55% (591/1074) 2024-06-26T07:04:25.4790044Z remote: Compressing objects: 56% (602/1074) 2024-06-26T07:04:25.4796330Z remote: Compressing objects: 57% (613/1074) 2024-06-26T07:04:25.4816617Z remote: Compressing objects: 58% (623/1074) 2024-06-26T07:04:25.4823501Z remote: Compressing objects: 59% (634/1074) 2024-06-26T07:04:25.4826632Z remote: Compressing objects: 60% (645/1074) 2024-06-26T07:04:25.4836650Z remote: Compressing objects: 61% (656/1074) 2024-06-26T07:04:25.4846030Z remote: Compressing objects: 62% (666/1074) 2024-06-26T07:04:25.4856223Z remote: Compressing objects: 63% (677/1074) 2024-06-26T07:04:25.4862138Z remote: Compressing objects: 64% (688/1074) 2024-06-26T07:04:25.4868262Z remote: Compressing objects: 65% (699/1074) 2024-06-26T07:04:25.4872296Z remote: Compressing objects: 66% (709/1074) 2024-06-26T07:04:25.4877561Z remote: Compressing objects: 67% (720/1074) 2024-06-26T07:04:25.4883821Z remote: Compressing objects: 68% (731/1074) 2024-06-26T07:04:25.4887175Z remote: Compressing objects: 69% (742/1074) 2024-06-26T07:04:25.4898896Z remote: Compressing objects: 70% (752/1074) 2024-06-26T07:04:25.4899478Z remote: Compressing objects: 71% (763/1074) 2024-06-26T07:04:25.4900093Z remote: Compressing objects: 72% (774/1074) 2024-06-26T07:04:25.4900639Z remote: Compressing objects: 73% (785/1074) 2024-06-26T07:04:25.4901280Z remote: Compressing objects: 74% (795/1074) 2024-06-26T07:04:25.4902030Z remote: Compressing objects: 75% (806/1074) 2024-06-26T07:04:25.4902578Z remote: Compressing objects: 76% (817/1074) 2024-06-26T07:04:25.4903355Z remote: Compressing objects: 77% (827/1074) 2024-06-26T07:04:25.4904144Z remote: Compressing objects: 78% (838/1074) 2024-06-26T07:04:25.4904790Z remote: Compressing objects: 79% (849/1074) 2024-06-26T07:04:25.4905522Z remote: Compressing objects: 80% (860/1074) 2024-06-26T07:04:25.4906060Z remote: Compressing objects: 81% (870/1074) 2024-06-26T07:04:25.4907898Z remote: Compressing objects: 82% (881/1074) 2024-06-26T07:04:25.4910589Z remote: Compressing objects: 83% (892/1074) 2024-06-26T07:04:25.4911183Z remote: Compressing objects: 84% (903/1074) 2024-06-26T07:04:25.4914087Z remote: Compressing objects: 85% (913/1074) 2024-06-26T07:04:25.4914640Z remote: Compressing objects: 86% (924/1074) 2024-06-26T07:04:25.4917018Z remote: Compressing objects: 87% (935/1074) 2024-06-26T07:04:25.4919902Z remote: Compressing objects: 88% (946/1074) 2024-06-26T07:04:25.4920455Z remote: Compressing objects: 89% (956/1074) 2024-06-26T07:04:25.4924074Z remote: Compressing objects: 90% (967/1074) 2024-06-26T07:04:25.4924628Z remote: Compressing objects: 91% (978/1074) 2024-06-26T07:04:25.4925157Z remote: Compressing objects: 92% (989/1074) 2024-06-26T07:04:25.4925807Z remote: Compressing objects: 93% (999/1074) 2024-06-26T07:04:25.4926799Z remote: Compressing objects: 94% (1010/1074) 2024-06-26T07:04:25.4927402Z remote: Compressing objects: 95% (1021/1074) 2024-06-26T07:04:25.4927959Z remote: Compressing objects: 96% (1032/1074) 2024-06-26T07:04:25.4930564Z remote: Compressing objects: 97% (1042/1074) 2024-06-26T07:04:25.4931136Z remote: Compressing objects: 98% (1053/1074) 2024-06-26T07:04:25.4931694Z remote: Compressing objects: 99% (1064/1074) 2024-06-26T07:04:25.4932389Z remote: Compressing objects: 100% (1074/1074) 2024-06-26T07:04:25.4933579Z remote: Compressing objects: 100% (1074/1074), done. 2024-06-26T07:04:45.6458160Z remote: Total 990209 (delta 2062), reused 1863 (delta 1522), pack-reused 987612 2024-06-26T07:05:05.2876494Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T07:05:05.2896476Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:05:05.2902448Z ##[endgroup] 2024-06-26T07:05:05.2903264Z ##[group]Determining the checkout info 2024-06-26T07:05:05.2904215Z ##[endgroup] 2024-06-26T07:05:05.2904993Z ##[group]Checking out the ref 2024-06-26T07:05:05.2906083Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:05:06.5034975Z ##[endgroup] 2024-06-26T07:05:06.5035585Z ##[group]Setting up auth for fetching submodules 2024-06-26T07:05:06.5039129Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:05:06.5080154Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T07:05:06.5105698Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T07:05:06.5130206Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T07:05:06.5151269Z ##[endgroup] 2024-06-26T07:05:06.5152078Z ##[group]Fetching submodules 2024-06-26T07:05:06.5155538Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T07:05:06.5359359Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T07:05:06.5556565Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T07:05:06.5559701Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T07:05:06.5562812Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T07:05:06.5565234Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T07:05:06.5567087Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T07:05:06.5569045Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T07:05:06.5570392Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T07:05:06.5571872Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T07:05:06.5573247Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T07:05:06.5574783Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T07:05:06.5576428Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T07:05:06.5577698Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T07:05:06.5579053Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T07:05:06.5581801Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T07:05:06.5584187Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T07:05:06.5586709Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T07:05:06.5589558Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:05:06.5592071Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T07:05:06.5594896Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T07:05:06.5597607Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T07:05:06.5600562Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T07:05:06.5603577Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T07:05:06.5606769Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T07:05:06.5610247Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T07:05:06.5613487Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T07:05:06.5616770Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T07:05:06.5620422Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T07:05:06.5623600Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T07:05:06.5627006Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T07:05:06.5630488Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T07:05:06.5634188Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T07:05:06.5637849Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T07:05:06.5641416Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T07:05:06.5645136Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T07:05:06.5649270Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T07:05:06.5668388Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T07:05:06.8253977Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T07:05:07.0130337Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T07:05:07.1994771Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T07:05:07.4138675Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T07:05:09.4564776Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T07:05:19.2672384Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T07:05:19.7308482Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T07:05:20.2726462Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T07:05:20.9341027Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T07:05:22.1525489Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T07:05:23.8494327Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T07:05:29.7268357Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T07:05:31.0377227Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T07:05:32.7477430Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T07:05:33.9136155Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T07:05:34.0564656Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T07:05:34.4344286Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T07:05:34.7308780Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T07:05:35.7013786Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T07:05:36.0229853Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T07:05:36.2715363Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T07:05:37.7355322Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T07:05:38.4434146Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T07:05:39.0895670Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T07:05:44.6417595Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T07:05:47.0700418Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T07:05:52.5855248Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T07:05:52.7775964Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T07:06:01.0187685Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T07:06:01.1644905Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T07:06:01.3878135Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T07:06:02.3625542Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T07:06:02.6504159Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T07:06:03.3284407Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T07:06:03.7465698Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T07:06:03.7549314Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T07:06:03.7609033Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T07:06:03.7787505Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T07:06:03.8076332Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T07:06:04.5782642Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T07:06:04.5949188Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T07:06:04.6301481Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T07:06:04.7145618Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T07:06:04.7412556Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T07:06:05.1685459Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T07:06:05.3800149Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T07:06:05.4425274Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T07:06:05.4437802Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:06:05.4439713Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:06:05.4442311Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:06:05.4444785Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:06:05.4447543Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:06:05.4465717Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T07:06:06.4465314Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T07:06:07.0013824Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T07:06:08.9293194Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T07:06:09.8487244Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T07:06:10.1739523Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T07:06:10.2577188Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T07:06:10.5982085Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T07:06:10.6524929Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T07:06:10.6611486Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T07:06:10.7537308Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T07:06:10.7814090Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T07:06:10.7879821Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T07:06:10.8182357Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T07:06:10.8378112Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T07:06:10.8743798Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:06:10.8837998Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T07:06:10.8848382Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T07:06:10.8864228Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T07:06:22.5663142Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T07:06:22.5796393Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T07:06:22.6573819Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T07:06:22.6584914Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:06:22.6586579Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:06:22.6588352Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:06:22.6605649Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T07:06:23.1550327Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T07:06:24.3485134Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T07:06:25.3935863Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T07:06:25.3946308Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:06:25.3948375Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:06:25.3950467Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:06:25.3952484Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:06:25.3954519Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:06:25.3956846Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:06:25.3958911Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:06:25.3960802Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:06:25.3979139Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T07:06:26.4754067Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T07:06:26.8478839Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T07:06:28.0395912Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T07:06:28.3030630Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T07:06:28.7650138Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T07:06:29.7317031Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T07:06:35.5154420Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T07:06:35.9384507Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T07:06:35.9523511Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T07:06:35.9812354Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T07:06:35.9908693Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T07:06:35.9919349Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:06:35.9935594Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T07:06:36.2749991Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T07:06:36.2885410Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T07:06:36.3221699Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T07:06:36.4020910Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T07:06:36.4141027Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T07:06:36.4442528Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T07:06:36.4932981Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T07:06:36.5234084Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T07:06:36.5431015Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T07:06:36.6292876Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T07:06:36.9030508Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T07:06:36.9054931Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T07:06:36.9056876Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T07:06:36.9073026Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T07:06:37.3283304Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T07:06:38.3210292Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T07:06:38.3487678Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T07:06:38.4020110Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T07:06:38.4032588Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:06:38.4039559Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:06:38.4041127Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:06:38.4042936Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:06:38.4044787Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:06:38.4046752Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:06:38.4048862Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:06:38.4050452Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:06:38.4063511Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T07:06:38.7823478Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T07:06:39.8861415Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T07:06:40.2908741Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T07:06:45.8077606Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T07:06:46.0935278Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T07:06:46.2836817Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T07:06:46.5700514Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T07:06:54.0087356Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T07:06:54.0422240Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T07:06:54.0545165Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T07:06:54.1387821Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T07:06:54.1478292Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T07:06:54.1587828Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T07:06:54.1701416Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T07:06:54.1711598Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:06:54.1713685Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:06:54.1730191Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T07:06:55.9500306Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T07:06:57.1397590Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T07:06:57.1768210Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:06:57.5444415Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T07:06:57.5521575Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T07:06:57.7671100Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T07:06:57.7685052Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:06:57.7686761Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T07:06:57.7705197Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T07:06:58.1634350Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T07:06:59.1313661Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T07:06:59.1933925Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T07:06:59.1996582Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T07:06:59.2085198Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T07:06:59.2363732Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T07:06:59.2581989Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T07:06:59.2915605Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T07:06:59.3113742Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T07:06:59.3123555Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:06:59.3126254Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:06:59.3127863Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:06:59.3129881Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:06:59.3146860Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T07:07:00.2905255Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T07:07:00.5190343Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T07:07:01.5274577Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T07:07:02.6661747Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T07:07:02.6774940Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T07:07:02.7330824Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T07:07:02.7556566Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T07:07:02.7567473Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:07:02.7584483Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T07:07:02.9580842Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T07:07:02.9602763Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T07:07:02.9808218Z Entering 'android/libs/fbjni' 2024-06-26T07:07:02.9833136Z Entering 'third_party/FP16' 2024-06-26T07:07:02.9861913Z Entering 'third_party/FXdiv' 2024-06-26T07:07:02.9886721Z Entering 'third_party/NNPACK' 2024-06-26T07:07:02.9914207Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:07:02.9941005Z Entering 'third_party/XNNPACK' 2024-06-26T07:07:02.9977416Z Entering 'third_party/benchmark' 2024-06-26T07:07:03.0006440Z Entering 'third_party/cpp-httplib' 2024-06-26T07:07:03.0033221Z Entering 'third_party/cpuinfo' 2024-06-26T07:07:03.0060456Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:07:03.0086049Z Entering 'third_party/cutlass' 2024-06-26T07:07:03.0117811Z Entering 'third_party/eigen' 2024-06-26T07:07:03.0147166Z Entering 'third_party/fbgemm' 2024-06-26T07:07:03.0173576Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:07:03.0198904Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:07:03.0226173Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:07:03.0258732Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:07:03.0284467Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:07:03.0313516Z Entering 'third_party/flatbuffers' 2024-06-26T07:07:03.0342719Z Entering 'third_party/fmt' 2024-06-26T07:07:03.0372326Z Entering 'third_party/foxi' 2024-06-26T07:07:03.0401107Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:07:03.0427254Z Entering 'third_party/gloo' 2024-06-26T07:07:03.0452984Z Entering 'third_party/googletest' 2024-06-26T07:07:03.0480840Z Entering 'third_party/ideep' 2024-06-26T07:07:03.0505855Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:07:03.0539565Z Entering 'third_party/ittapi' 2024-06-26T07:07:03.0566945Z Entering 'third_party/kineto' 2024-06-26T07:07:03.0595300Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:07:03.0623282Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:07:03.0651688Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:07:03.0678766Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:07:03.0708310Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:07:03.0734129Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:07:03.0761763Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:07:03.0788119Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:07:03.0815644Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:07:03.0841333Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:07:03.0872570Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:07:03.0900168Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:07:03.0926329Z Entering 'third_party/mimalloc' 2024-06-26T07:07:03.0956515Z Entering 'third_party/nccl/nccl' 2024-06-26T07:07:03.0983722Z Entering 'third_party/nlohmann' 2024-06-26T07:07:03.1012964Z Entering 'third_party/onnx' 2024-06-26T07:07:03.1051165Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:07:03.1080462Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:07:03.1109678Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:07:03.1139414Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:07:03.1164161Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:07:03.1191824Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:07:03.1217359Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:07:03.1245954Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:07:03.1273706Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:07:03.1299072Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:07:03.1324925Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:07:03.1352148Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:07:03.1378364Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:07:03.1420853Z Entering 'third_party/pocketfft' 2024-06-26T07:07:03.1448056Z Entering 'third_party/protobuf' 2024-06-26T07:07:03.1478287Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:07:03.1504739Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:07:03.1534699Z Entering 'third_party/psimd' 2024-06-26T07:07:03.1561035Z Entering 'third_party/pthreadpool' 2024-06-26T07:07:03.1590201Z Entering 'third_party/pybind11' 2024-06-26T07:07:03.1619045Z Entering 'third_party/python-peachpy' 2024-06-26T07:07:03.1645449Z Entering 'third_party/sleef' 2024-06-26T07:07:03.1673543Z Entering 'third_party/tensorpipe' 2024-06-26T07:07:03.1701277Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:07:03.1728371Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:07:03.1756289Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:07:03.1783333Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:07:03.1810083Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:07:03.1849059Z ##[endgroup] 2024-06-26T07:07:03.1851068Z ##[group]Persisting credentials for submodules 2024-06-26T07:07:03.1854507Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T07:07:03.2062591Z Entering 'android/libs/fbjni' 2024-06-26T07:07:03.2095779Z Entering 'third_party/FP16' 2024-06-26T07:07:03.2131386Z Entering 'third_party/FXdiv' 2024-06-26T07:07:03.2164969Z Entering 'third_party/NNPACK' 2024-06-26T07:07:03.2201539Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:07:03.2236368Z Entering 'third_party/XNNPACK' 2024-06-26T07:07:03.2281857Z Entering 'third_party/benchmark' 2024-06-26T07:07:03.2318317Z Entering 'third_party/cpp-httplib' 2024-06-26T07:07:03.2353501Z Entering 'third_party/cpuinfo' 2024-06-26T07:07:03.2387791Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:07:03.2420121Z Entering 'third_party/cutlass' 2024-06-26T07:07:03.2460828Z Entering 'third_party/eigen' 2024-06-26T07:07:03.2497174Z Entering 'third_party/fbgemm' 2024-06-26T07:07:03.2533238Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:07:03.2567535Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:07:03.2602663Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:07:03.2641357Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:07:03.2675360Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:07:03.2709375Z Entering 'third_party/flatbuffers' 2024-06-26T07:07:03.2747059Z Entering 'third_party/fmt' 2024-06-26T07:07:03.2781506Z Entering 'third_party/foxi' 2024-06-26T07:07:03.2816491Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:07:03.2847725Z Entering 'third_party/gloo' 2024-06-26T07:07:03.2882672Z Entering 'third_party/googletest' 2024-06-26T07:07:03.2917699Z Entering 'third_party/ideep' 2024-06-26T07:07:03.2951380Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:07:03.2991197Z Entering 'third_party/ittapi' 2024-06-26T07:07:03.3025654Z Entering 'third_party/kineto' 2024-06-26T07:07:03.3061013Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:07:03.3094568Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:07:03.3126788Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:07:03.3160601Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:07:03.3194457Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:07:03.3227582Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:07:03.3261956Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:07:03.3296038Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:07:03.3327866Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:07:03.3363535Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:07:03.3398947Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:07:03.3435434Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:07:03.3468825Z Entering 'third_party/mimalloc' 2024-06-26T07:07:03.3505044Z Entering 'third_party/nccl/nccl' 2024-06-26T07:07:03.3539763Z Entering 'third_party/nlohmann' 2024-06-26T07:07:03.3574768Z Entering 'third_party/onnx' 2024-06-26T07:07:03.3622042Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:07:03.3655870Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:07:03.3692319Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:07:03.3728413Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:07:03.3762959Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:07:03.3795534Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:07:03.3828565Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:07:03.3861304Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:07:03.3891943Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:07:03.3924987Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:07:03.3955309Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:07:03.3989278Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:07:03.4023609Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:07:03.4074141Z Entering 'third_party/pocketfft' 2024-06-26T07:07:03.4108923Z Entering 'third_party/protobuf' 2024-06-26T07:07:03.4143900Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:07:03.4177508Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:07:03.4211361Z Entering 'third_party/psimd' 2024-06-26T07:07:03.4246834Z Entering 'third_party/pthreadpool' 2024-06-26T07:07:03.4282821Z Entering 'third_party/pybind11' 2024-06-26T07:07:03.4318270Z Entering 'third_party/python-peachpy' 2024-06-26T07:07:03.4353455Z Entering 'third_party/sleef' 2024-06-26T07:07:03.4387113Z Entering 'third_party/tensorpipe' 2024-06-26T07:07:03.4422362Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:07:03.4454346Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:07:03.4487109Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:07:03.4521630Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:07:03.4555542Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:07:03.4602939Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T07:07:03.4809461Z Entering 'android/libs/fbjni' 2024-06-26T07:07:03.4839498Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T07:07:03.4852143Z Entering 'third_party/FP16' 2024-06-26T07:07:03.4883373Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T07:07:03.4895631Z Entering 'third_party/FXdiv' 2024-06-26T07:07:03.4926116Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T07:07:03.4939180Z Entering 'third_party/NNPACK' 2024-06-26T07:07:03.4970501Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T07:07:03.4984746Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:07:03.5015636Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T07:07:03.5027613Z Entering 'third_party/XNNPACK' 2024-06-26T07:07:03.5057549Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T07:07:03.5080602Z Entering 'third_party/benchmark' 2024-06-26T07:07:03.5111879Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:07:03.5124685Z Entering 'third_party/cpp-httplib' 2024-06-26T07:07:03.5155649Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T07:07:03.5169425Z Entering 'third_party/cpuinfo' 2024-06-26T07:07:03.5199801Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:07:03.5213479Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:07:03.5243621Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T07:07:03.5256085Z Entering 'third_party/cutlass' 2024-06-26T07:07:03.5286988Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:07:03.5305832Z Entering 'third_party/eigen' 2024-06-26T07:07:03.5338265Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T07:07:03.5352518Z Entering 'third_party/fbgemm' 2024-06-26T07:07:03.5383457Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T07:07:03.5397341Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:07:03.5428316Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T07:07:03.5441291Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:07:03.5473331Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:07:03.5485233Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:07:03.5518250Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:07:03.5535191Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:07:03.5565688Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:07:03.5577917Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:07:03.5609965Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T07:07:03.5625121Z Entering 'third_party/flatbuffers' 2024-06-26T07:07:03.5656799Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T07:07:03.5672412Z Entering 'third_party/fmt' 2024-06-26T07:07:03.5703444Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:07:03.5717182Z Entering 'third_party/foxi' 2024-06-26T07:07:03.5749793Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T07:07:03.5762554Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:07:03.5795043Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T07:07:03.5808082Z Entering 'third_party/gloo' 2024-06-26T07:07:03.5839927Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T07:07:03.5853301Z Entering 'third_party/googletest' 2024-06-26T07:07:03.5884169Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:07:03.5896468Z Entering 'third_party/ideep' 2024-06-26T07:07:03.5926311Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T07:07:03.5937680Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:07:03.5968942Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T07:07:03.5986549Z Entering 'third_party/ittapi' 2024-06-26T07:07:03.6017399Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T07:07:03.6031086Z Entering 'third_party/kineto' 2024-06-26T07:07:03.6061630Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T07:07:03.6075011Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:07:03.6107076Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T07:07:03.6118964Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:07:03.6156729Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T07:07:03.6164714Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:07:03.6195649Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T07:07:03.6207642Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:07:03.6239885Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:07:03.6252974Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:07:03.6284477Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T07:07:03.6295558Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:07:03.6327203Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T07:07:03.6341345Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:07:03.6371928Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T07:07:03.6383697Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:07:03.6416652Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:07:03.6427877Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:07:03.6458170Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T07:07:03.6470028Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:07:03.6502178Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T07:07:03.6516839Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:07:03.6548220Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T07:07:03.6559910Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:07:03.6590749Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T07:07:03.6604862Z Entering 'third_party/mimalloc' 2024-06-26T07:07:03.6635828Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T07:07:03.6649658Z Entering 'third_party/nccl/nccl' 2024-06-26T07:07:03.6680900Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T07:07:03.6694350Z Entering 'third_party/nlohmann' 2024-06-26T07:07:03.6725870Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T07:07:03.6740790Z Entering 'third_party/onnx' 2024-06-26T07:07:03.6772109Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T07:07:03.6796839Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:07:03.6827451Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:07:03.6840659Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:07:03.6872408Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:07:03.6886767Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:07:03.6920534Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T07:07:03.6934918Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:07:03.6964916Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:07:03.6977636Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:07:03.7009696Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:07:03.7021112Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:07:03.7051367Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T07:07:03.7063431Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:07:03.7094561Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T07:07:03.7107898Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:07:03.7139306Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T07:07:03.7151950Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:07:03.7183346Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T07:07:03.7195099Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:07:03.7226597Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T07:07:03.7237555Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:07:03.7268346Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T07:07:03.7282624Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:07:03.7314571Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T07:07:03.7327750Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:07:03.7358446Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T07:07:03.7387398Z Entering 'third_party/pocketfft' 2024-06-26T07:07:03.7419468Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T07:07:03.7431843Z Entering 'third_party/protobuf' 2024-06-26T07:07:03.7463759Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T07:07:03.7478380Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:07:03.7509858Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:07:03.7522107Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:07:03.7553659Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:07:03.7567647Z Entering 'third_party/psimd' 2024-06-26T07:07:03.7599111Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T07:07:03.7610786Z Entering 'third_party/pthreadpool' 2024-06-26T07:07:03.7642943Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T07:07:03.7655075Z Entering 'third_party/pybind11' 2024-06-26T07:07:03.7684815Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:07:03.7697030Z Entering 'third_party/python-peachpy' 2024-06-26T07:07:03.7729297Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T07:07:03.7743069Z Entering 'third_party/sleef' 2024-06-26T07:07:03.7773585Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T07:07:03.7787648Z Entering 'third_party/tensorpipe' 2024-06-26T07:07:03.7819333Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T07:07:03.7831773Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:07:03.7862370Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:07:03.7875556Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:07:03.7905227Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T07:07:03.7915535Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:07:03.7947879Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T07:07:03.7959223Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:07:03.7990542Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:07:03.8002518Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:07:03.8033787Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T07:07:03.9024675Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T07:07:03.9226210Z Entering 'android/libs/fbjni' 2024-06-26T07:07:03.9251491Z Entering 'third_party/FP16' 2024-06-26T07:07:03.9278508Z Entering 'third_party/FXdiv' 2024-06-26T07:07:03.9306211Z Entering 'third_party/NNPACK' 2024-06-26T07:07:03.9331808Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:07:03.9357739Z Entering 'third_party/XNNPACK' 2024-06-26T07:07:03.9397036Z Entering 'third_party/benchmark' 2024-06-26T07:07:03.9423451Z Entering 'third_party/cpp-httplib' 2024-06-26T07:07:03.9451987Z Entering 'third_party/cpuinfo' 2024-06-26T07:07:03.9480751Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:07:03.9510634Z Entering 'third_party/cutlass' 2024-06-26T07:07:03.9544996Z Entering 'third_party/eigen' 2024-06-26T07:07:03.9573991Z Entering 'third_party/fbgemm' 2024-06-26T07:07:03.9600033Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:07:03.9626388Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:07:03.9653946Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:07:03.9684736Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:07:03.9713164Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:07:03.9740028Z Entering 'third_party/flatbuffers' 2024-06-26T07:07:03.9768388Z Entering 'third_party/fmt' 2024-06-26T07:07:03.9799403Z Entering 'third_party/foxi' 2024-06-26T07:07:03.9827603Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:07:03.9855842Z Entering 'third_party/gloo' 2024-06-26T07:07:03.9882165Z Entering 'third_party/googletest' 2024-06-26T07:07:03.9908924Z Entering 'third_party/ideep' 2024-06-26T07:07:03.9936544Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:07:03.9969663Z Entering 'third_party/ittapi' 2024-06-26T07:07:04.0000170Z Entering 'third_party/kineto' 2024-06-26T07:07:04.0027556Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:07:04.0053687Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:07:04.0083314Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:07:04.0110459Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:07:04.0138513Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:07:04.0164976Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:07:04.0194565Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:07:04.0221891Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:07:04.0249612Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:07:04.0277374Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:07:04.0306999Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:07:04.0334036Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:07:04.0362478Z Entering 'third_party/mimalloc' 2024-06-26T07:07:04.0392949Z Entering 'third_party/nccl/nccl' 2024-06-26T07:07:04.0421003Z Entering 'third_party/nlohmann' 2024-06-26T07:07:04.0447752Z Entering 'third_party/onnx' 2024-06-26T07:07:04.0489570Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:07:04.0518588Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:07:04.0550018Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:07:04.0578614Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:07:04.0605446Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:07:04.0634399Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:07:04.0659806Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:07:04.0687789Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:07:04.0716741Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:07:04.0742903Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:07:04.0769400Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:07:04.0798016Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:07:04.0826440Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:07:04.0868523Z Entering 'third_party/pocketfft' 2024-06-26T07:07:04.0897723Z Entering 'third_party/protobuf' 2024-06-26T07:07:04.0929102Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:07:04.0957016Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:07:04.0984921Z Entering 'third_party/psimd' 2024-06-26T07:07:04.1011522Z Entering 'third_party/pthreadpool' 2024-06-26T07:07:04.1040578Z Entering 'third_party/pybind11' 2024-06-26T07:07:04.1070331Z Entering 'third_party/python-peachpy' 2024-06-26T07:07:04.1098676Z Entering 'third_party/sleef' 2024-06-26T07:07:04.1125780Z Entering 'third_party/tensorpipe' 2024-06-26T07:07:04.1156004Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:07:04.1181647Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:07:04.1208752Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:07:04.1236537Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:07:04.1263671Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:07:04.1302999Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T07:07:04.1504356Z Entering 'android/libs/fbjni' 2024-06-26T07:07:04.1532599Z Entering 'third_party/FP16' 2024-06-26T07:07:04.1561281Z Entering 'third_party/FXdiv' 2024-06-26T07:07:04.1588328Z Entering 'third_party/NNPACK' 2024-06-26T07:07:04.1617205Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:07:04.1646206Z Entering 'third_party/XNNPACK' 2024-06-26T07:07:04.1683811Z Entering 'third_party/benchmark' 2024-06-26T07:07:04.1712560Z Entering 'third_party/cpp-httplib' 2024-06-26T07:07:04.1739454Z Entering 'third_party/cpuinfo' 2024-06-26T07:07:04.1765671Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:07:04.1793311Z Entering 'third_party/cutlass' 2024-06-26T07:07:04.1824754Z Entering 'third_party/eigen' 2024-06-26T07:07:04.1855468Z Entering 'third_party/fbgemm' 2024-06-26T07:07:04.1881580Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:07:04.1909375Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:07:04.1935245Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:07:04.1966539Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:07:04.1994290Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:07:04.2021345Z Entering 'third_party/flatbuffers' 2024-06-26T07:07:04.2051422Z Entering 'third_party/fmt' 2024-06-26T07:07:04.2080124Z Entering 'third_party/foxi' 2024-06-26T07:07:04.2106412Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:07:04.2131357Z Entering 'third_party/gloo' 2024-06-26T07:07:04.2158361Z Entering 'third_party/googletest' 2024-06-26T07:07:04.2185275Z Entering 'third_party/ideep' 2024-06-26T07:07:04.2212913Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:07:04.2244483Z Entering 'third_party/ittapi' 2024-06-26T07:07:04.2273745Z Entering 'third_party/kineto' 2024-06-26T07:07:04.2300146Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:07:04.2326352Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:07:04.2356364Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:07:04.2384362Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:07:04.2409890Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:07:04.2436542Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:07:04.2464148Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:07:04.2490153Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:07:04.2517277Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:07:04.2545239Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:07:04.2574099Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:07:04.2600629Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:07:04.2628241Z Entering 'third_party/mimalloc' 2024-06-26T07:07:04.2655574Z Entering 'third_party/nccl/nccl' 2024-06-26T07:07:04.2681053Z Entering 'third_party/nlohmann' 2024-06-26T07:07:04.2709313Z Entering 'third_party/onnx' 2024-06-26T07:07:04.2747782Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:07:04.2775279Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:07:04.2804098Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:07:04.2831348Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:07:04.2857026Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:07:04.2884520Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:07:04.2911498Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:07:04.2938942Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:07:04.2966434Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:07:04.2993760Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:07:04.3021672Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:07:04.3049978Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:07:04.3078830Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:07:04.3121379Z Entering 'third_party/pocketfft' 2024-06-26T07:07:04.3151755Z Entering 'third_party/protobuf' 2024-06-26T07:07:04.3181020Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:07:04.3208745Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:07:04.3237032Z Entering 'third_party/psimd' 2024-06-26T07:07:04.3264077Z Entering 'third_party/pthreadpool' 2024-06-26T07:07:04.3292310Z Entering 'third_party/pybind11' 2024-06-26T07:07:04.3320351Z Entering 'third_party/python-peachpy' 2024-06-26T07:07:04.3350001Z Entering 'third_party/sleef' 2024-06-26T07:07:04.3375662Z Entering 'third_party/tensorpipe' 2024-06-26T07:07:04.3401538Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:07:04.3428705Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:07:04.3454044Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:07:04.3480521Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:07:04.3506053Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:07:04.3545189Z ##[endgroup] 2024-06-26T07:07:04.3578112Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T07:07:04.3599515Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T07:07:04.3765892Z Prepare all required actions 2024-06-26T07:07:04.3766356Z Getting action download info 2024-06-26T07:07:04.5093195Z ##[group]Run ./.github/actions/setup-linux 2024-06-26T07:07:04.5093620Z env: 2024-06-26T07:07:04.5093891Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:04.5094235Z ##[endgroup] 2024-06-26T07:07:04.5151590Z ##[group]Run set -euo pipefail 2024-06-26T07:07:04.5152029Z set -euo pipefail 2024-06-26T07:07:04.5152417Z function get_ec2_metadata() { 2024-06-26T07:07:04.5152948Z  # Pulled from instance metadata endpoint for EC2 2024-06-26T07:07:04.5153829Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-26T07:07:04.5154584Z  category=$1 2024-06-26T07:07:04.5155110Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-26T07:07:04.5155736Z  runner_name_str=i-0f01143c0858f0e6b 2024-06-26T07:07:04.5156219Z  if [[ -f /.inarc ]]; then 2024-06-26T07:07:04.5156710Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-26T07:07:04.5157317Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-26T07:07:04.5158024Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-26T07:07:04.5158624Z  else 2024-06-26T07:07:04.5159118Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-26T07:07:04.5159700Z  fi 2024-06-26T07:07:04.5159992Z } 2024-06-26T07:07:04.5160359Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-26T07:07:04.5160935Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-26T07:07:04.5161578Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-26T07:07:04.5162139Z echo "system info $(uname -a)" 2024-06-26T07:07:04.5170020Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:04.5170511Z env: 2024-06-26T07:07:04.5170796Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:04.5171139Z ##[endgroup] 2024-06-26T07:07:04.5239345Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-26T07:07:04.5278902Z instance-id: i-0f01143c0858f0e6b 2024-06-26T07:07:04.5316789Z instance-type: g5.4xlarge 2024-06-26T07:07:04.5322035Z system info Linux ip-10-0-21-121.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-26T07:07:04.5342464Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:07:04.5343381Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:07:04.5351273Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:04.5351769Z env: 2024-06-26T07:07:04.5352061Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:04.5352419Z ##[endgroup] 2024-06-26T07:07:04.5416409Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-26T07:07:04.5417024Z if systemctl is-active --quiet docker; then 2024-06-26T07:07:04.5417567Z  echo "Docker daemon is running..."; 2024-06-26T07:07:04.5418129Z else 2024-06-26T07:07:04.5418621Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-26T07:07:04.5419205Z fi 2024-06-26T07:07:04.5425855Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:04.5426355Z env: 2024-06-26T07:07:04.5426669Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:04.5427035Z ##[endgroup] 2024-06-26T07:07:04.5462514Z Docker daemon is running... 2024-06-26T07:07:04.5506535Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:07:04.5507117Z with: 2024-06-26T07:07:04.5507396Z shell: bash 2024-06-26T07:07:04.5507688Z timeout_minutes: 5 2024-06-26T07:07:04.5508011Z max_attempts: 3 2024-06-26T07:07:04.5508330Z retry_wait_seconds: 30 2024-06-26T07:07:04.5509803Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-26T07:07:04.5511225Z polling_interval_seconds: 1 2024-06-26T07:07:04.5511607Z warning_on_retry: true 2024-06-26T07:07:04.5511963Z continue_on_error: false 2024-06-26T07:07:04.5512302Z env: 2024-06-26T07:07:04.5512569Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:04.5512928Z AWS_RETRY_MODE: standard 2024-06-26T07:07:04.5513280Z AWS_MAX_ATTEMPTS: 5 2024-06-26T07:07:04.5513613Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:07:04.5513989Z ##[endgroup] 2024-06-26T07:07:05.4114032Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:07:05.4114890Z Configure a credential helper to remove this warning. See 2024-06-26T07:07:05.4115752Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:07:05.4116278Z 2024-06-26T07:07:05.4116428Z Login Succeeded 2024-06-26T07:07:05.5996266Z Command completed after 1 attempt(s). 2024-06-26T07:07:05.6038106Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:07:05.6038840Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:07:05.6039497Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:07:05.6047182Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:05.6047680Z env: 2024-06-26T07:07:05.6047963Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:05.6048314Z ##[endgroup] 2024-06-26T07:07:05.6105279Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:07:05.6106050Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:07:05.6106645Z # shellcheck disable=SC2046 2024-06-26T07:07:05.6107108Z docker stop $(docker ps -q) || true 2024-06-26T07:07:05.6107594Z # Prune all of the docker images 2024-06-26T07:07:05.6108048Z docker system prune -af 2024-06-26T07:07:05.6115016Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:05.6115523Z env: 2024-06-26T07:07:05.6115803Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:05.6116163Z ##[endgroup] 2024-06-26T07:07:05.6353937Z "docker stop" requires at least 1 argument. 2024-06-26T07:07:05.6354575Z See 'docker stop --help'. 2024-06-26T07:07:05.6354816Z 2024-06-26T07:07:05.6355040Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T07:07:05.6355420Z 2024-06-26T07:07:05.6359020Z Stop one or more running containers 2024-06-26T07:07:05.6509331Z Total reclaimed space: 0B 2024-06-26T07:07:05.6539060Z ##[group]Run set +e 2024-06-26T07:07:05.6539428Z set +e 2024-06-26T07:07:05.6539736Z set -x 2024-06-26T07:07:05.6540033Z  2024-06-26T07:07:05.6540356Z PT_DOMAIN=download.pytorch.org 2024-06-26T07:07:05.6541151Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-26T07:07:05.6542221Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-26T07:07:05.6543083Z # one is returned at random 2024-06-26T07:07:05.6543630Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-26T07:07:05.6544137Z  2024-06-26T07:07:05.6544459Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:07:05.6545087Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-26T07:07:05.6545847Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-26T07:07:05.6546397Z  2024-06-26T07:07:05.6546729Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:07:05.6547284Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-26T07:07:05.6547796Z  exit 1 2024-06-26T07:07:05.6548107Z  fi 2024-06-26T07:07:05.6548401Z fi 2024-06-26T07:07:05.6548779Z  2024-06-26T07:07:05.6549151Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-26T07:07:05.6549672Z  # Clean up any old records first 2024-06-26T07:07:05.6550181Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-26T07:07:05.6550641Z fi 2024-06-26T07:07:05.6550915Z  2024-06-26T07:07:05.6551346Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-26T07:07:05.6551887Z cat /etc/hosts 2024-06-26T07:07:05.6559386Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:05.6559882Z env: 2024-06-26T07:07:05.6560169Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:05.6560522Z ##[endgroup] 2024-06-26T07:07:05.6580198Z + PT_DOMAIN=download.pytorch.org 2024-06-26T07:07:05.6584297Z ++ dig -4 +short download.pytorch.org 2024-06-26T07:07:05.6585400Z ++ tail -n1 2024-06-26T07:07:05.6647479Z + RESOLVED_IP=18.160.10.28 2024-06-26T07:07:05.6648024Z + '[' -z 18.160.10.28 ']' 2024-06-26T07:07:05.6648819Z + grep -r download.pytorch.org /etc/hosts 2024-06-26T07:07:05.6654407Z 18.160.18.37 download.pytorch.org 2024-06-26T07:07:05.6655338Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-06-26T07:07:05.6752547Z + echo '18.160.10.28 download.pytorch.org' 2024-06-26T07:07:05.6753239Z + sudo tee -a /etc/hosts 2024-06-26T07:07:05.7082856Z 18.160.10.28 download.pytorch.org 2024-06-26T07:07:05.7095789Z + cat /etc/hosts 2024-06-26T07:07:05.7101112Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-26T07:07:05.7112593Z ::1 localhost6 localhost6.localdomain6 2024-06-26T07:07:05.7113118Z 18.160.10.28 download.pytorch.org 2024-06-26T07:07:05.7260089Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T07:07:05.7260689Z with: 2024-06-26T07:07:05.7261786Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:05.7263000Z docker-build-dir: .ci/docker 2024-06-26T07:07:05.7263385Z working-directory: . 2024-06-26T07:07:05.7263870Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:05.7264398Z force-push: false 2024-06-26T07:07:05.7264710Z env: 2024-06-26T07:07:05.7264991Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:05.7280392Z ##[endgroup] 2024-06-26T07:07:05.7298597Z ##[group]Run set -ex 2024-06-26T07:07:05.7298967Z set -ex 2024-06-26T07:07:05.7299272Z  2024-06-26T07:07:05.7299831Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T07:07:05.7300815Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T07:07:05.7301635Z # job could then download the pre-built image as usual 2024-06-26T07:07:05.7302379Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T07:07:05.7303063Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:05.7303702Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:05.7304376Z  2024-06-26T07:07:05.7304880Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T07:07:05.7305500Z  exit 0 2024-06-26T07:07:05.7305806Z else 2024-06-26T07:07:05.7306178Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:05.7306629Z fi 2024-06-26T07:07:05.7306945Z  2024-06-26T07:07:05.7307442Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T07:07:05.7308286Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T07:07:05.7309053Z  # use it as it is, but first let's extract the tag 2024-06-26T07:07:05.7309757Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T07:07:05.7310481Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:05.7311195Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:05.7311755Z else 2024-06-26T07:07:05.7312196Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T07:07:05.7312865Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:05.7313758Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:05.7314501Z fi 2024-06-26T07:07:05.7322189Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:05.7322694Z env: 2024-06-26T07:07:05.7322972Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:05.7323331Z REPO_NAME: pytorch 2024-06-26T07:07:05.7324463Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:05.7325650Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:07:05.7326158Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:05.7326678Z ##[endgroup] 2024-06-26T07:07:05.7347205Z + [[ ! -d .ci/docker ]] 2024-06-26T07:07:05.7347612Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T07:07:05.7347990Z + echo skip=false 2024-06-26T07:07:05.7349898Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T07:07:05.7352413Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:05.7353579Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T07:07:05.7358763Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:05.7359418Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:05.7361116Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:05.7386271Z ##[group]Run set +e 2024-06-26T07:07:05.7386647Z set +e 2024-06-26T07:07:05.7386975Z set -x 2024-06-26T07:07:05.7387301Z  2024-06-26T07:07:05.7387594Z login() { 2024-06-26T07:07:05.7388259Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:07:05.7388972Z } 2024-06-26T07:07:05.7389247Z  2024-06-26T07:07:05.7389538Z retry () { 2024-06-26T07:07:05.7389938Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:07:05.7390389Z } 2024-06-26T07:07:05.7390667Z  2024-06-26T07:07:05.7390994Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:07:05.7391419Z  2024-06-26T07:07:05.7391890Z # Check if image already exists, if it does then skip building it 2024-06-26T07:07:05.7392737Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T07:07:05.7393245Z  exit 0 2024-06-26T07:07:05.7393562Z fi 2024-06-26T07:07:05.7393841Z  2024-06-26T07:07:05.7394348Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T07:07:05.7395178Z # be empty. The default action would be to continue rebuild the image 2024-06-26T07:07:05.7395920Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T07:07:05.7396591Z  # if we're on the base branch then use the parent commit 2024-06-26T07:07:05.7397242Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T07:07:05.7397683Z else 2024-06-26T07:07:05.7398162Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T07:07:05.7398847Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T07:07:05.7399365Z fi 2024-06-26T07:07:05.7399650Z  2024-06-26T07:07:05.7399976Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T07:07:05.7400478Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:05.7400939Z  2024-06-26T07:07:05.7401598Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T07:07:05.7402361Z  exit 0 2024-06-26T07:07:05.7402658Z fi 2024-06-26T07:07:05.7402944Z  2024-06-26T07:07:05.7403392Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T07:07:05.7404353Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T07:07:05.7405140Z  exit 1 2024-06-26T07:07:05.7405450Z fi 2024-06-26T07:07:05.7405732Z  2024-06-26T07:07:05.7406220Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T07:07:05.7407208Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T07:07:05.7408069Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T07:07:05.7409270Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T07:07:05.7410358Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T07:07:05.7411143Z fi 2024-06-26T07:07:05.7411422Z  2024-06-26T07:07:05.7411783Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:05.7418732Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:05.7419234Z env: 2024-06-26T07:07:05.7419514Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:05.7419900Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:07:05.7420377Z BASE_REVISION: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:07:05.7421643Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:05.7422874Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:05.7423480Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:05.7424007Z ##[endgroup] 2024-06-26T07:07:05.7444192Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:05.7445514Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:05.7446682Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:07:05.7447809Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:06.1573805Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:07:06.1574650Z Configure a credential helper to remove this warning. See 2024-06-26T07:07:06.1575949Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:07:06.1576659Z 2024-06-26T07:07:06.1576789Z Login Succeeded 2024-06-26T07:07:06.1586665Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:06.3761720Z { 2024-06-26T07:07:06.3762104Z "schemaVersion": 2, 2024-06-26T07:07:06.3762908Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T07:07:06.3763628Z "config": { 2024-06-26T07:07:06.3764218Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T07:07:06.3764892Z "size": 48681, 2024-06-26T07:07:06.3765624Z "digest": "sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd" 2024-06-26T07:07:06.3766329Z }, 2024-06-26T07:07:06.3766663Z "layers": [ 2024-06-26T07:07:06.3767054Z { 2024-06-26T07:07:06.3767601Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3768185Z "size": 28580681, 2024-06-26T07:07:06.3769107Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-06-26T07:07:06.3769740Z }, 2024-06-26T07:07:06.3769995Z { 2024-06-26T07:07:06.3770431Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3770980Z "size": 7943451, 2024-06-26T07:07:06.3771521Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-06-26T07:07:06.3772145Z }, 2024-06-26T07:07:06.3772400Z { 2024-06-26T07:07:06.3772828Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3773501Z "size": 55728572, 2024-06-26T07:07:06.3774235Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-06-26T07:07:06.3775115Z }, 2024-06-26T07:07:06.3775465Z { 2024-06-26T07:07:06.3776053Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3777037Z + exit 0 2024-06-26T07:07:06.3777446Z "size": 186, 2024-06-26T07:07:06.3778244Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-06-26T07:07:06.3779217Z }, 2024-06-26T07:07:06.3779588Z { 2024-06-26T07:07:06.3780252Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3781114Z "size": 6886, 2024-06-26T07:07:06.3781880Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-06-26T07:07:06.3783009Z }, 2024-06-26T07:07:06.3783400Z { 2024-06-26T07:07:06.3784046Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3784897Z "size": 1329001756, 2024-06-26T07:07:06.3785799Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-06-26T07:07:06.3786761Z }, 2024-06-26T07:07:06.3787129Z { 2024-06-26T07:07:06.3787673Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3788321Z "size": 62501, 2024-06-26T07:07:06.3788935Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-06-26T07:07:06.3789897Z }, 2024-06-26T07:07:06.3790187Z { 2024-06-26T07:07:06.3790758Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3791504Z "size": 1684, 2024-06-26T07:07:06.3792193Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-06-26T07:07:06.3792981Z }, 2024-06-26T07:07:06.3793320Z { 2024-06-26T07:07:06.3793896Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3794622Z "size": 1523, 2024-06-26T07:07:06.3795290Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-06-26T07:07:06.3796132Z }, 2024-06-26T07:07:06.3796452Z { 2024-06-26T07:07:06.3796999Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3797737Z "size": 2528295403, 2024-06-26T07:07:06.3798447Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-06-26T07:07:06.3799068Z }, 2024-06-26T07:07:06.3799311Z { 2024-06-26T07:07:06.3799754Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3800314Z "size": 86016, 2024-06-26T07:07:06.3800870Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-06-26T07:07:06.3801482Z }, 2024-06-26T07:07:06.3801741Z { 2024-06-26T07:07:06.3802185Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3802727Z "size": 1823, 2024-06-26T07:07:06.3803265Z "digest": "sha256:309b872da805fcdf74137a887464876fa0615dc8b24199307119eb65fc2bfb49" 2024-06-26T07:07:06.3803870Z }, 2024-06-26T07:07:06.3804122Z { 2024-06-26T07:07:06.3804553Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3805114Z "size": 246737542, 2024-06-26T07:07:06.3805680Z "digest": "sha256:e82932cb002c06a782077b8e5965f8733eb8563406a23cab260d9d8c4a994f7d" 2024-06-26T07:07:06.3806290Z }, 2024-06-26T07:07:06.3806538Z { 2024-06-26T07:07:06.3806978Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3807577Z "size": 545, 2024-06-26T07:07:06.3808120Z "digest": "sha256:ca9dd73e3e9378225ebdcfd12199ef44649e34fe55003cfbc8afe2e8a4527c5b" 2024-06-26T07:07:06.3808990Z }, 2024-06-26T07:07:06.3809244Z { 2024-06-26T07:07:06.3809682Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3810218Z "size": 1283, 2024-06-26T07:07:06.3810747Z "digest": "sha256:16d76a79d5d6616e228537785ec83a869f68a402520bbc6e826201621ca6cb98" 2024-06-26T07:07:06.3811344Z }, 2024-06-26T07:07:06.3811591Z { 2024-06-26T07:07:06.3812019Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3812566Z "size": 485, 2024-06-26T07:07:06.3813111Z "digest": "sha256:4bdf41c9ecbe1651e0986ccbea4663a0556af27b7145a679cfa45c35dc96b891" 2024-06-26T07:07:06.3813728Z }, 2024-06-26T07:07:06.3813975Z { 2024-06-26T07:07:06.3814402Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3814959Z "size": 91710937, 2024-06-26T07:07:06.3815521Z "digest": "sha256:f4863801e30cdbcd1d4e6cb1cbbcde7dbcaf6ae317ca80429295741ce60ac31d" 2024-06-26T07:07:06.3816233Z }, 2024-06-26T07:07:06.3816499Z { 2024-06-26T07:07:06.3816937Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3817585Z "size": 3233, 2024-06-26T07:07:06.3818148Z "digest": "sha256:d37dd9ed8948a0d02e1198bbedadbca36477a0980dd553196385aab0c025581d" 2024-06-26T07:07:06.3818771Z }, 2024-06-26T07:07:06.3819021Z { 2024-06-26T07:07:06.3819448Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3820003Z "size": 1905, 2024-06-26T07:07:06.3820566Z "digest": "sha256:cf2f34f225dbed17c1a9a8b9ea4ad030aba797774a716dc0a82fe212324aa751" 2024-06-26T07:07:06.3821186Z }, 2024-06-26T07:07:06.3821436Z { 2024-06-26T07:07:06.3821874Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3822422Z "size": 702, 2024-06-26T07:07:06.3822954Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:07:06.3823571Z }, 2024-06-26T07:07:06.3823820Z { 2024-06-26T07:07:06.3824253Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3824797Z "size": 2784789002, 2024-06-26T07:07:06.3825370Z "digest": "sha256:3d914950f8addfc4d657a2365c76e1143eb80b710cd3f6e9ae05640217fc6347" 2024-06-26T07:07:06.3825977Z }, 2024-06-26T07:07:06.3826229Z { 2024-06-26T07:07:06.3826657Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3827208Z "size": 381, 2024-06-26T07:07:06.3827749Z "digest": "sha256:96d0e2032184c57e62a652aaffcf0241f0d0c7a23e894ca0ad192c4a64a36583" 2024-06-26T07:07:06.3828357Z }, 2024-06-26T07:07:06.3828709Z { 2024-06-26T07:07:06.3829142Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3829694Z "size": 11879, 2024-06-26T07:07:06.3830226Z "digest": "sha256:06e12ce59790cb8b091e5d619c2a62a28329008ee184a8b9a562e3db12471f30" 2024-06-26T07:07:06.3830828Z }, 2024-06-26T07:07:06.3831085Z { 2024-06-26T07:07:06.3831516Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3832056Z "size": 803, 2024-06-26T07:07:06.3832607Z "digest": "sha256:ed712f63cde7070bdec2a6179c4c35e752c6991ddbc69fe9f2724a9903250ceb" 2024-06-26T07:07:06.3833225Z }, 2024-06-26T07:07:06.3833470Z { 2024-06-26T07:07:06.3833910Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3834464Z "size": 106, 2024-06-26T07:07:06.3834997Z "digest": "sha256:a97a71ef8a4e66234dfc2098834f8395ac5630240e60164d1a251fdb0115ba85" 2024-06-26T07:07:06.3835590Z }, 2024-06-26T07:07:06.3835843Z { 2024-06-26T07:07:06.3836283Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3836840Z "size": 503, 2024-06-26T07:07:06.3837422Z "digest": "sha256:25cd0c482ad612f936b35b4f220c580e16be982226aac8801a6a849a6bfdc221" 2024-06-26T07:07:06.3838034Z }, 2024-06-26T07:07:06.3838284Z { 2024-06-26T07:07:06.3838713Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3839266Z "size": 121477177, 2024-06-26T07:07:06.3839835Z "digest": "sha256:9bf94bc456584365fda21d0ed83b7cc690b08a71727b71c4c1e667e15028a386" 2024-06-26T07:07:06.3840439Z }, 2024-06-26T07:07:06.3840684Z { 2024-06-26T07:07:06.3841115Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3841668Z "size": 109, 2024-06-26T07:07:06.3842207Z "digest": "sha256:de45a14c9b227e4b0c9d45d42223b50a71c03522ddd9e2ca0d9517835206c823" 2024-06-26T07:07:06.3842807Z }, 2024-06-26T07:07:06.3843058Z { 2024-06-26T07:07:06.3843491Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3844047Z "size": 491, 2024-06-26T07:07:06.3844574Z "digest": "sha256:e99241e77209b0b35a0fb59a61c0821fc161190178af7e4af45e6eeaa043375e" 2024-06-26T07:07:06.3845181Z }, 2024-06-26T07:07:06.3845432Z { 2024-06-26T07:07:06.3845855Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3846401Z "size": 291, 2024-06-26T07:07:06.3846957Z "digest": "sha256:35e8f4e2a42f9df0c969dedd2199cf71719cf5ecfa676dfa014bbdeb4fdb295f" 2024-06-26T07:07:06.3847650Z }, 2024-06-26T07:07:06.3847898Z { 2024-06-26T07:07:06.3848331Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3849086Z "size": 101, 2024-06-26T07:07:06.3849619Z "digest": "sha256:a823ffa4c541394b52953391dc0a6867e26c37fe4a71fa022a2cc3f07f0ba70d" 2024-06-26T07:07:06.3850218Z }, 2024-06-26T07:07:06.3850467Z { 2024-06-26T07:07:06.3850896Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3851453Z "size": 1473, 2024-06-26T07:07:06.3851981Z "digest": "sha256:44ba272eae2064603605a6238b3f35cc6578859d660068e702a047a9cd8ab2fd" 2024-06-26T07:07:06.3852577Z }, 2024-06-26T07:07:06.3852830Z { 2024-06-26T07:07:06.3853254Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3853804Z "size": 446710781, 2024-06-26T07:07:06.3854365Z "digest": "sha256:da02d278e5240c285bc855ce4e42cd6121d5b903787f5c14f03e575ad1268650" 2024-06-26T07:07:06.3854969Z }, 2024-06-26T07:07:06.3855222Z { 2024-06-26T07:07:06.3855653Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3856258Z "size": 161, 2024-06-26T07:07:06.3856778Z "digest": "sha256:73a0ce681c295135456e190578733273276b38b4dd5babbb39298f384d43b1c6" 2024-06-26T07:07:06.3857414Z }, 2024-06-26T07:07:06.3857654Z { 2024-06-26T07:07:06.3858073Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3858604Z "size": 564, 2024-06-26T07:07:06.3859241Z "digest": "sha256:49737bd5ae709dbf7956e21fc8b05ba009e9e6c9b130b0126a73bd3d3f5efc78" 2024-06-26T07:07:06.3859854Z }, 2024-06-26T07:07:06.3860105Z { 2024-06-26T07:07:06.3860527Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3861074Z "size": 35874600, 2024-06-26T07:07:06.3861629Z "digest": "sha256:9cd4c745f8e4f9d5c9bb8050923575afad21f5b7852975a4a1981bc9eff282c5" 2024-06-26T07:07:06.3862229Z }, 2024-06-26T07:07:06.3862465Z { 2024-06-26T07:07:06.3862896Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3863432Z "size": 104, 2024-06-26T07:07:06.3863957Z "digest": "sha256:b2950a0365345d4c17a48627c905fd8930b3b3154c727f0526eb9ca547102bb7" 2024-06-26T07:07:06.3864544Z }, 2024-06-26T07:07:06.3864784Z { 2024-06-26T07:07:06.3865208Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3865738Z "size": 425, 2024-06-26T07:07:06.3866258Z "digest": "sha256:25e19d81d9fc081962ff4cbc09e098f4035419822712c19a410f5282300dfb9e" 2024-06-26T07:07:06.3866857Z }, 2024-06-26T07:07:06.3867096Z { 2024-06-26T07:07:06.3867514Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3868059Z "size": 20262084, 2024-06-26T07:07:06.3868633Z "digest": "sha256:60156b40fcbec2c25f0170b0a82fa8cd9303e0daae8bb1811fd0fc6562e88e8e" 2024-06-26T07:07:06.3869241Z }, 2024-06-26T07:07:06.3869477Z { 2024-06-26T07:07:06.3869914Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3870451Z "size": 439, 2024-06-26T07:07:06.3870966Z "digest": "sha256:7a868239227fd19b30a19927bbc8f058d247403fe5b343c55f4b4b8987918911" 2024-06-26T07:07:06.3871560Z }, 2024-06-26T07:07:06.3871803Z { 2024-06-26T07:07:06.3872222Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3872756Z "size": 702, 2024-06-26T07:07:06.3873290Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:07:06.3873900Z }, 2024-06-26T07:07:06.3874139Z { 2024-06-26T07:07:06.3874558Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3875097Z "size": 143, 2024-06-26T07:07:06.3875618Z "digest": "sha256:83911d7c7ecf11b5932bb7449975c52e7f737a5b867691935e1ccb85ceb25b52" 2024-06-26T07:07:06.3876214Z }, 2024-06-26T07:07:06.3876450Z { 2024-06-26T07:07:06.3876878Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3877548Z "size": 135, 2024-06-26T07:07:06.3878088Z "digest": "sha256:b3c5d66c679f58c71febdd502889bd8cd3d36e5001754cdf1e30eabe21135cce" 2024-06-26T07:07:06.3878697Z }, 2024-06-26T07:07:06.3878940Z { 2024-06-26T07:07:06.3879365Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3879905Z "size": 4911495247, 2024-06-26T07:07:06.3880457Z "digest": "sha256:84b2c7a899820f6acdb22148c5e8346d8b26c33269fd01842c9573515251e1e1" 2024-06-26T07:07:06.3881064Z }, 2024-06-26T07:07:06.3881306Z { 2024-06-26T07:07:06.3881724Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3882263Z "size": 190, 2024-06-26T07:07:06.3882807Z "digest": "sha256:c1068d03a811d8dead5da9e82fdc8869205fc4ee9cb92d4f7a4acfbb5ba83950" 2024-06-26T07:07:06.3883424Z }, 2024-06-26T07:07:06.3883661Z { 2024-06-26T07:07:06.3884085Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3884622Z "size": 565, 2024-06-26T07:07:06.3885145Z "digest": "sha256:671b4a301d81074fca17d36851f81801552c4ecb2f90ff3412aa9ab6a1a91371" 2024-06-26T07:07:06.3885739Z }, 2024-06-26T07:07:06.3885981Z { 2024-06-26T07:07:06.3886470Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3887010Z "size": 43163095, 2024-06-26T07:07:06.3887613Z "digest": "sha256:67760cf5ec46ab58d3ca98637fd9b26cb47a67446bfd0db176fe5e023606aa0d" 2024-06-26T07:07:06.3888212Z }, 2024-06-26T07:07:06.3888693Z { 2024-06-26T07:07:06.3889132Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3889680Z "size": 106, 2024-06-26T07:07:06.3890208Z "digest": "sha256:ddc9740bcea2b301e99b50525059aea8d9c7121a1c8eab566561a8968438a8c2" 2024-06-26T07:07:06.3890820Z }, 2024-06-26T07:07:06.3891072Z { 2024-06-26T07:07:06.3891496Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3892043Z "size": 1213, 2024-06-26T07:07:06.3892580Z "digest": "sha256:fb2e977685609fcc01de170ba7f50a573e73e6520e47253717e92a28952c5ee8" 2024-06-26T07:07:06.3893181Z }, 2024-06-26T07:07:06.3893424Z { 2024-06-26T07:07:06.3893853Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3894419Z "size": 702, 2024-06-26T07:07:06.3894959Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:07:06.3895563Z }, 2024-06-26T07:07:06.3895814Z { 2024-06-26T07:07:06.3896320Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3896873Z "size": 138, 2024-06-26T07:07:06.3897413Z "digest": "sha256:b599c5a8c7cb461cbd7b3c98b99960b8eb04506e05e79db1cc0cfadb14a47e91" 2024-06-26T07:07:06.3898030Z }, 2024-06-26T07:07:06.3898283Z { 2024-06-26T07:07:06.3898709Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3899258Z "size": 120, 2024-06-26T07:07:06.3899805Z "digest": "sha256:e08e88c1366827455ee7ddbf92b6ae615ceb3cbd3fb11cd03ddd31bd3fa94a9c" 2024-06-26T07:07:06.3900423Z }, 2024-06-26T07:07:06.3900665Z { 2024-06-26T07:07:06.3901095Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3901642Z "size": 1904157495, 2024-06-26T07:07:06.3902215Z "digest": "sha256:12ba7431ff38de8cf6af3052a2a52e79f4e712b8c99f5bafde3c20e63b1f570f" 2024-06-26T07:07:06.3902819Z }, 2024-06-26T07:07:06.3903069Z { 2024-06-26T07:07:06.3903500Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3904039Z "size": 172, 2024-06-26T07:07:06.3904582Z "digest": "sha256:cf7c2cb896cba6a594028cc0fd246e127a2f4481d48905a3aa12c5f5872332d4" 2024-06-26T07:07:06.3905190Z }, 2024-06-26T07:07:06.3905438Z { 2024-06-26T07:07:06.3905861Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3906407Z "size": 908, 2024-06-26T07:07:06.3906957Z "digest": "sha256:3a778ddfac2af5b3a11f49ddf2f3f1ad1731f74ba691f434ab69966d1e2a189f" 2024-06-26T07:07:06.3907670Z }, 2024-06-26T07:07:06.3907925Z { 2024-06-26T07:07:06.3908356Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3908907Z "size": 702, 2024-06-26T07:07:06.3909451Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:07:06.3910056Z }, 2024-06-26T07:07:06.3910312Z { 2024-06-26T07:07:06.3910747Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3911296Z "size": 134, 2024-06-26T07:07:06.3911832Z "digest": "sha256:236c8f0245f4b40616a4322eaf1e1b3e178d5990db1fcaba39622ea4e79c06a1" 2024-06-26T07:07:06.3912440Z }, 2024-06-26T07:07:06.3912689Z { 2024-06-26T07:07:06.3913115Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3913662Z "size": 32, 2024-06-26T07:07:06.3914206Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:07:06.3914820Z }, 2024-06-26T07:07:06.3915074Z { 2024-06-26T07:07:06.3915517Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3916069Z "size": 156, 2024-06-26T07:07:06.3916612Z "digest": "sha256:dda1ce302b9a9df0c8f1145d2e3eaa03c96909bb45a792451f1fcb555b06738d" 2024-06-26T07:07:06.3917233Z }, 2024-06-26T07:07:06.3917485Z { 2024-06-26T07:07:06.3917918Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3918462Z "size": 1841, 2024-06-26T07:07:06.3919091Z "digest": "sha256:e797fe86caef19319e03f8dca92cf96838ab3bb3644f4447c6dfc04230118afa" 2024-06-26T07:07:06.3919711Z }, 2024-06-26T07:07:06.3919963Z { 2024-06-26T07:07:06.3920391Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3920944Z "size": 7529784, 2024-06-26T07:07:06.3921497Z "digest": "sha256:967190076fc0ab2bb2fc0d74e58edd3d662a537d702ff3f8b6e876230154a16f" 2024-06-26T07:07:06.3922107Z }, 2024-06-26T07:07:06.3922355Z { 2024-06-26T07:07:06.3922796Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3923343Z "size": 165, 2024-06-26T07:07:06.3923877Z "digest": "sha256:4fb0a83f06855534ef82b80dcf7c685303edb2bcbd446196066dc2ad2dd71281" 2024-06-26T07:07:06.3924490Z }, 2024-06-26T07:07:06.3924742Z { 2024-06-26T07:07:06.3925177Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3925717Z "size": 7944, 2024-06-26T07:07:06.3926259Z "digest": "sha256:68af2ec4e30a33f26947ece9f3a31b7d3b04c2bca5865d4b48618457832a1da2" 2024-06-26T07:07:06.3926879Z }, 2024-06-26T07:07:06.3927131Z { 2024-06-26T07:07:06.3927557Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3928108Z "size": 8070, 2024-06-26T07:07:06.3928839Z "digest": "sha256:cc91933e985a044c6df53caa15488d2285896f5e991f38ec068c253ceec53f14" 2024-06-26T07:07:06.3929460Z }, 2024-06-26T07:07:06.3929710Z { 2024-06-26T07:07:06.3930155Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3930709Z "size": 303, 2024-06-26T07:07:06.3931246Z "digest": "sha256:9f3187916e7dcad33cc572f8f0e949d2f507b887f99cfd6acefd14826b44960d" 2024-06-26T07:07:06.3931869Z }, 2024-06-26T07:07:06.3932123Z { 2024-06-26T07:07:06.3932558Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3933104Z "size": 7629071, 2024-06-26T07:07:06.3933639Z "digest": "sha256:52608399485738d10d779254946c9e1d536010dd45593c69bf1eb9f62f85dc24" 2024-06-26T07:07:06.3934234Z }, 2024-06-26T07:07:06.3934490Z { 2024-06-26T07:07:06.3934920Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3935469Z "size": 108, 2024-06-26T07:07:06.3936015Z "digest": "sha256:05a6f6dcc1c00bac6660923a22adc7d7c34416718f8c89daf9b3351d9365f694" 2024-06-26T07:07:06.3936685Z }, 2024-06-26T07:07:06.3936940Z { 2024-06-26T07:07:06.3937374Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3938038Z "size": 54145775, 2024-06-26T07:07:06.3938603Z "digest": "sha256:6bc9dc8315b09b27c976bfba9700767ba412d72fd0528b18bbe8f8e62005ec22" 2024-06-26T07:07:06.3939220Z }, 2024-06-26T07:07:06.3939474Z { 2024-06-26T07:07:06.3939910Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3940454Z "size": 474, 2024-06-26T07:07:06.3940996Z "digest": "sha256:1b0e27c35f07b3fbf0aeb548d1949bf1d60a420136da21789d94d196c891d981" 2024-06-26T07:07:06.3941618Z }, 2024-06-26T07:07:06.3941864Z { 2024-06-26T07:07:06.3942300Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3942856Z "size": 1374858747, 2024-06-26T07:07:06.3943409Z "digest": "sha256:95237e7311a1c5ce6937db7196c866632f44e8788514a5c0eb4a3f04da4017e6" 2024-06-26T07:07:06.3943999Z }, 2024-06-26T07:07:06.3944258Z { 2024-06-26T07:07:06.3944691Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3945241Z "size": 106, 2024-06-26T07:07:06.3945792Z "digest": "sha256:789eff6c51aada70bf75eb1acafed63c0d839b768638001500616ff6e53b854a" 2024-06-26T07:07:06.3946418Z }, 2024-06-26T07:07:06.3946674Z { 2024-06-26T07:07:06.3947109Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3947658Z "size": 558, 2024-06-26T07:07:06.3948212Z "digest": "sha256:de0f8a8c45c6077b3c5ba3b84d7f07de98353fd618dad9a1e2f368ffe00493c7" 2024-06-26T07:07:06.3948835Z }, 2024-06-26T07:07:06.3949156Z { 2024-06-26T07:07:06.3949592Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3950145Z "size": 46248513, 2024-06-26T07:07:06.3950697Z "digest": "sha256:2060028f74888d218bebdd10d723f66c33ee1c708935c4ffd4654d34912ac4dc" 2024-06-26T07:07:06.3951297Z }, 2024-06-26T07:07:06.3951550Z { 2024-06-26T07:07:06.3951997Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3952553Z "size": 111, 2024-06-26T07:07:06.3953092Z "digest": "sha256:e412c7f3b2ded440992955bd92512345cbece8a8894cd3592f714ebc758ad1a0" 2024-06-26T07:07:06.3953705Z }, 2024-06-26T07:07:06.3953957Z { 2024-06-26T07:07:06.3954385Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3954939Z "size": 32, 2024-06-26T07:07:06.3955485Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:07:06.3956102Z }, 2024-06-26T07:07:06.3956353Z { 2024-06-26T07:07:06.3956794Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3957348Z "size": 32, 2024-06-26T07:07:06.3957892Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:07:06.3958499Z }, 2024-06-26T07:07:06.3958758Z { 2024-06-26T07:07:06.3959199Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3959747Z "size": 32, 2024-06-26T07:07:06.3960285Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:07:06.3960903Z }, 2024-06-26T07:07:06.3961156Z { 2024-06-26T07:07:06.3961582Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:06.3962135Z "size": 32, 2024-06-26T07:07:06.3962676Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:07:06.3963290Z } 2024-06-26T07:07:06.3963537Z ] 2024-06-26T07:07:06.3963797Z } 2024-06-26T07:07:06.4079143Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:07:06.4079654Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:07:06.4080209Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-26T07:07:06.4088045Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:06.4088704Z env: 2024-06-26T07:07:06.4088994Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:06.4090139Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:06.4091319Z ##[endgroup] 2024-06-26T07:07:06.4113620Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:06.4163356Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T07:07:06.4163938Z with: 2024-06-26T07:07:06.4165014Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:06.4166340Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:06.4166857Z env: 2024-06-26T07:07:06.4167140Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:06.4167506Z ##[endgroup] 2024-06-26T07:07:06.4183432Z ##[group]Run set -x 2024-06-26T07:07:06.4183789Z set -x 2024-06-26T07:07:06.4184107Z set +e 2024-06-26T07:07:06.4184579Z  2024-06-26T07:07:06.4184858Z login() { 2024-06-26T07:07:06.4185517Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:07:06.4186230Z } 2024-06-26T07:07:06.4186521Z  2024-06-26T07:07:06.4186828Z retry () { 2024-06-26T07:07:06.4187236Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:07:06.4187688Z } 2024-06-26T07:07:06.4187972Z  2024-06-26T07:07:06.4188396Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:07:06.4188818Z  2024-06-26T07:07:06.4189095Z set -e 2024-06-26T07:07:06.4189580Z # ignore output since only exit code is used for conditional 2024-06-26T07:07:06.4190284Z # only pull docker image if it's not available locally 2024-06-26T07:07:06.4191047Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T07:07:06.4191740Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T07:07:06.4192184Z fi 2024-06-26T07:07:06.4199658Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:06.4200148Z env: 2024-06-26T07:07:06.4200433Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:06.4201566Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:06.4202852Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:06.4203380Z ##[endgroup] 2024-06-26T07:07:06.4222783Z + set +e 2024-06-26T07:07:06.4223271Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:06.4223901Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:06.4225676Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:07:06.4226445Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:06.8289899Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:07:06.8290817Z Configure a credential helper to remove this warning. See 2024-06-26T07:07:06.8291791Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:07:06.8292418Z 2024-06-26T07:07:06.8292556Z Login Succeeded 2024-06-26T07:07:06.8303772Z + set -e 2024-06-26T07:07:06.8305081Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:06.8429620Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:06.8431742Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:07.0668109Z 91382da70d5719cd7007b6b80b71d2f48398f6b7: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-06-26T07:07:07.0669159Z 7a2c55901189: Pulling fs layer 2024-06-26T07:07:07.0669617Z 224fe954d725: Pulling fs layer 2024-06-26T07:07:07.0670233Z 75722010b82e: Pulling fs layer 2024-06-26T07:07:07.0670632Z d527cbbb87e3: Pulling fs layer 2024-06-26T07:07:07.0671094Z b57676e46aee: Pulling fs layer 2024-06-26T07:07:07.0671479Z a8c1e85b5e14: Pulling fs layer 2024-06-26T07:07:07.0671927Z a41a8d1c11c8: Pulling fs layer 2024-06-26T07:07:07.0672301Z 0c1227890755: Pulling fs layer 2024-06-26T07:07:07.0672737Z d8d1234baab3: Pulling fs layer 2024-06-26T07:07:07.0673120Z 7ed32bc8e469: Pulling fs layer 2024-06-26T07:07:07.0673555Z ec1e7978c1fe: Pulling fs layer 2024-06-26T07:07:07.0673932Z 309b872da805: Pulling fs layer 2024-06-26T07:07:07.0674368Z e82932cb002c: Pulling fs layer 2024-06-26T07:07:07.0674745Z ca9dd73e3e93: Pulling fs layer 2024-06-26T07:07:07.0675172Z 16d76a79d5d6: Pulling fs layer 2024-06-26T07:07:07.0675556Z 4bdf41c9ecbe: Pulling fs layer 2024-06-26T07:07:07.0675986Z f4863801e30c: Pulling fs layer 2024-06-26T07:07:07.0676366Z d37dd9ed8948: Pulling fs layer 2024-06-26T07:07:07.0676773Z cf2f34f225db: Pulling fs layer 2024-06-26T07:07:07.0677174Z a8c1e85b5e14: Waiting 2024-06-26T07:07:07.0677736Z df546d7b3e34: Pulling fs layer 2024-06-26T07:07:07.0678198Z 3d914950f8ad: Pulling fs layer 2024-06-26T07:07:07.0678683Z 96d0e2032184: Pulling fs layer 2024-06-26T07:07:07.0679256Z 06e12ce59790: Pulling fs layer 2024-06-26T07:07:07.0679919Z ed712f63cde7: Pulling fs layer 2024-06-26T07:07:07.0680411Z a97a71ef8a4e: Pulling fs layer 2024-06-26T07:07:07.0680915Z 25cd0c482ad6: Pulling fs layer 2024-06-26T07:07:07.0681348Z 9bf94bc45658: Pulling fs layer 2024-06-26T07:07:07.0681782Z 0c1227890755: Waiting 2024-06-26T07:07:07.0682111Z de45a14c9b22: Pulling fs layer 2024-06-26T07:07:07.0682485Z e99241e77209: Pulling fs layer 2024-06-26T07:07:07.0682839Z d8d1234baab3: Waiting 2024-06-26T07:07:07.0683181Z 35e8f4e2a42f: Pulling fs layer 2024-06-26T07:07:07.0683554Z a823ffa4c541: Pulling fs layer 2024-06-26T07:07:07.0683927Z 44ba272eae20: Pulling fs layer 2024-06-26T07:07:07.0684286Z 7ed32bc8e469: Waiting 2024-06-26T07:07:07.0684650Z da02d278e524: Pulling fs layer 2024-06-26T07:07:07.0685017Z ec1e7978c1fe: Waiting 2024-06-26T07:07:07.0685350Z 73a0ce681c29: Pulling fs layer 2024-06-26T07:07:07.0685726Z 49737bd5ae70: Pulling fs layer 2024-06-26T07:07:07.0686098Z 9cd4c745f8e4: Pulling fs layer 2024-06-26T07:07:07.0686459Z d527cbbb87e3: Waiting 2024-06-26T07:07:07.0686779Z 309b872da805: Waiting 2024-06-26T07:07:07.0687109Z b2950a036534: Pulling fs layer 2024-06-26T07:07:07.0687469Z b57676e46aee: Waiting 2024-06-26T07:07:07.0687793Z 25e19d81d9fc: Pulling fs layer 2024-06-26T07:07:07.0688150Z e82932cb002c: Waiting 2024-06-26T07:07:07.0688703Z 60156b40fcbe: Pulling fs layer 2024-06-26T07:07:07.0689080Z 7a868239227f: Pulling fs layer 2024-06-26T07:07:07.0689441Z ca9dd73e3e93: Waiting 2024-06-26T07:07:07.0689846Z 83911d7c7ecf: Pulling fs layer 2024-06-26T07:07:07.0690283Z b3c5d66c679f: Pulling fs layer 2024-06-26T07:07:07.0690646Z 16d76a79d5d6: Waiting 2024-06-26T07:07:07.0691028Z 84b2c7a89982: Pulling fs layer 2024-06-26T07:07:07.0691408Z c1068d03a811: Pulling fs layer 2024-06-26T07:07:07.0691767Z 96d0e2032184: Waiting 2024-06-26T07:07:07.0692100Z 4bdf41c9ecbe: Waiting 2024-06-26T07:07:07.0692439Z 671b4a301d81: Pulling fs layer 2024-06-26T07:07:07.0692817Z 67760cf5ec46: Pulling fs layer 2024-06-26T07:07:07.0693183Z 06e12ce59790: Waiting 2024-06-26T07:07:07.0693502Z f4863801e30c: Waiting 2024-06-26T07:07:07.0693840Z ddc9740bcea2: Pulling fs layer 2024-06-26T07:07:07.0694230Z fb2e97768560: Pulling fs layer 2024-06-26T07:07:07.0694588Z ed712f63cde7: Waiting 2024-06-26T07:07:07.0694914Z d37dd9ed8948: Waiting 2024-06-26T07:07:07.0695252Z b599c5a8c7cb: Pulling fs layer 2024-06-26T07:07:07.0695614Z a97a71ef8a4e: Waiting 2024-06-26T07:07:07.0695936Z 3d914950f8ad: Waiting 2024-06-26T07:07:07.0696362Z e08e88c13668: Pulling fs layer 2024-06-26T07:07:07.0696728Z cf2f34f225db: Waiting 2024-06-26T07:07:07.0697042Z de45a14c9b22: Waiting 2024-06-26T07:07:07.0697377Z 12ba7431ff38: Pulling fs layer 2024-06-26T07:07:07.0697742Z df546d7b3e34: Waiting 2024-06-26T07:07:07.0698065Z 25cd0c482ad6: Waiting 2024-06-26T07:07:07.0698510Z cf7c2cb896cb: Pulling fs layer 2024-06-26T07:07:07.0698956Z 9bf94bc45658: Waiting 2024-06-26T07:07:07.0699298Z 3a778ddfac2a: Pulling fs layer 2024-06-26T07:07:07.0699659Z 49737bd5ae70: Waiting 2024-06-26T07:07:07.0699990Z 44ba272eae20: Waiting 2024-06-26T07:07:07.0700308Z e99241e77209: Waiting 2024-06-26T07:07:07.0700640Z 236c8f0245f4: Pulling fs layer 2024-06-26T07:07:07.0700998Z da02d278e524: Waiting 2024-06-26T07:07:07.0701333Z 4f4fb700ef54: Pulling fs layer 2024-06-26T07:07:07.0701695Z 60156b40fcbe: Waiting 2024-06-26T07:07:07.0702029Z dda1ce302b9a: Pulling fs layer 2024-06-26T07:07:07.0702402Z b2950a036534: Waiting 2024-06-26T07:07:07.0702718Z 7a868239227f: Waiting 2024-06-26T07:07:07.0703039Z 73a0ce681c29: Waiting 2024-06-26T07:07:07.0703370Z e797fe86caef: Pulling fs layer 2024-06-26T07:07:07.0703748Z 967190076fc0: Pulling fs layer 2024-06-26T07:07:07.0704106Z 35e8f4e2a42f: Waiting 2024-06-26T07:07:07.0704425Z 25e19d81d9fc: Waiting 2024-06-26T07:07:07.0704764Z 4fb0a83f0685: Pulling fs layer 2024-06-26T07:07:07.0705144Z 68af2ec4e30a: Pulling fs layer 2024-06-26T07:07:07.0705529Z cc91933e985a: Pulling fs layer 2024-06-26T07:07:07.0705897Z 9f3187916e7d: Pulling fs layer 2024-06-26T07:07:07.0706269Z 526083994857: Pulling fs layer 2024-06-26T07:07:07.0706715Z a823ffa4c541: Waiting 2024-06-26T07:07:07.0707038Z b599c5a8c7cb: Waiting 2024-06-26T07:07:07.0707368Z 05a6f6dcc1c0: Pulling fs layer 2024-06-26T07:07:07.0707732Z 236c8f0245f4: Waiting 2024-06-26T07:07:07.0708066Z 6bc9dc8315b0: Pulling fs layer 2024-06-26T07:07:07.0708428Z e08e88c13668: Waiting 2024-06-26T07:07:07.0708757Z 1b0e27c35f07: Pulling fs layer 2024-06-26T07:07:07.0709121Z b3c5d66c679f: Waiting 2024-06-26T07:07:07.0709441Z 4f4fb700ef54: Waiting 2024-06-26T07:07:07.0709769Z 95237e7311a1: Pulling fs layer 2024-06-26T07:07:07.0710151Z 789eff6c51aa: Pulling fs layer 2024-06-26T07:07:07.0710521Z 12ba7431ff38: Waiting 2024-06-26T07:07:07.0710840Z 84b2c7a89982: Waiting 2024-06-26T07:07:07.0711151Z fb2e97768560: Waiting 2024-06-26T07:07:07.0711479Z cf7c2cb896cb: Waiting 2024-06-26T07:07:07.0711807Z 67760cf5ec46: Waiting 2024-06-26T07:07:07.0712121Z 3a778ddfac2a: Waiting 2024-06-26T07:07:07.0712463Z de0f8a8c45c6: Pulling fs layer 2024-06-26T07:07:07.0712831Z dda1ce302b9a: Waiting 2024-06-26T07:07:07.0713159Z c1068d03a811: Waiting 2024-06-26T07:07:07.0713482Z 2060028f7488: Pulling fs layer 2024-06-26T07:07:07.0713862Z e412c7f3b2de: Pulling fs layer 2024-06-26T07:07:07.0714228Z ddc9740bcea2: Waiting 2024-06-26T07:07:07.0714553Z 6bc9dc8315b0: Waiting 2024-06-26T07:07:07.0714866Z e797fe86caef: Waiting 2024-06-26T07:07:07.0715184Z 05a6f6dcc1c0: Waiting 2024-06-26T07:07:07.0715498Z 526083994857: Waiting 2024-06-26T07:07:07.0715803Z 9f3187916e7d: Waiting 2024-06-26T07:07:07.0716119Z de0f8a8c45c6: Waiting 2024-06-26T07:07:07.0716435Z 2060028f7488: Waiting 2024-06-26T07:07:07.0716748Z 967190076fc0: Waiting 2024-06-26T07:07:07.0717061Z 4fb0a83f0685: Waiting 2024-06-26T07:07:07.0717381Z 68af2ec4e30a: Waiting 2024-06-26T07:07:07.0717702Z 83911d7c7ecf: Waiting 2024-06-26T07:07:07.0718014Z e412c7f3b2de: Waiting 2024-06-26T07:07:07.0718332Z cc91933e985a: Waiting 2024-06-26T07:07:07.0718649Z a41a8d1c11c8: Waiting 2024-06-26T07:07:07.0718967Z 789eff6c51aa: Waiting 2024-06-26T07:07:07.2006414Z 224fe954d725: Verifying Checksum 2024-06-26T07:07:07.2006906Z 224fe954d725: Download complete 2024-06-26T07:07:07.2752547Z d527cbbb87e3: Verifying Checksum 2024-06-26T07:07:07.2753037Z d527cbbb87e3: Download complete 2024-06-26T07:07:07.3549467Z b57676e46aee: Verifying Checksum 2024-06-26T07:07:07.3550068Z b57676e46aee: Download complete 2024-06-26T07:07:07.4200520Z 7a2c55901189: Verifying Checksum 2024-06-26T07:07:07.4201163Z 7a2c55901189: Download complete 2024-06-26T07:07:07.5086834Z a41a8d1c11c8: Verifying Checksum 2024-06-26T07:07:07.5087328Z a41a8d1c11c8: Download complete 2024-06-26T07:07:07.5841241Z 0c1227890755: Verifying Checksum 2024-06-26T07:07:07.5841726Z 0c1227890755: Download complete 2024-06-26T07:07:07.6490153Z d8d1234baab3: Download complete 2024-06-26T07:07:07.6920382Z 75722010b82e: Verifying Checksum 2024-06-26T07:07:07.6920831Z 75722010b82e: Download complete 2024-06-26T07:07:07.7716861Z ec1e7978c1fe: Download complete 2024-06-26T07:07:07.8613309Z 309b872da805: Verifying Checksum 2024-06-26T07:07:07.8613815Z 309b872da805: Download complete 2024-06-26T07:07:08.4435350Z 7a2c55901189: Pull complete 2024-06-26T07:07:08.7547114Z 224fe954d725: Pull complete 2024-06-26T07:07:09.7120160Z 75722010b82e: Pull complete 2024-06-26T07:07:09.7580645Z d527cbbb87e3: Pull complete 2024-06-26T07:07:09.7960441Z b57676e46aee: Pull complete 2024-06-26T07:07:10.3957609Z e82932cb002c: Verifying Checksum 2024-06-26T07:07:10.3958606Z e82932cb002c: Download complete 2024-06-26T07:07:10.4891163Z ca9dd73e3e93: Verifying Checksum 2024-06-26T07:07:10.5850328Z 16d76a79d5d6: Verifying Checksum 2024-06-26T07:07:10.5850818Z 16d76a79d5d6: Download complete 2024-06-26T07:07:10.6748972Z 4bdf41c9ecbe: Download complete 2024-06-26T07:07:11.6563361Z f4863801e30c: Verifying Checksum 2024-06-26T07:07:11.6563867Z f4863801e30c: Download complete 2024-06-26T07:07:11.7721758Z d37dd9ed8948: Download complete 2024-06-26T07:07:11.8267931Z cf2f34f225db: Verifying Checksum 2024-06-26T07:07:11.8268394Z cf2f34f225db: Download complete 2024-06-26T07:07:11.8991047Z df546d7b3e34: Verifying Checksum 2024-06-26T07:07:11.8991776Z df546d7b3e34: Download complete 2024-06-26T07:07:20.7018731Z a8c1e85b5e14: Verifying Checksum 2024-06-26T07:07:20.7019402Z a8c1e85b5e14: Download complete 2024-06-26T07:07:20.7844689Z 96d0e2032184: Verifying Checksum 2024-06-26T07:07:20.7845328Z 96d0e2032184: Download complete 2024-06-26T07:07:20.8534688Z 06e12ce59790: Verifying Checksum 2024-06-26T07:07:20.8535245Z 06e12ce59790: Download complete 2024-06-26T07:07:20.9358028Z ed712f63cde7: Download complete 2024-06-26T07:07:21.0008081Z a97a71ef8a4e: Verifying Checksum 2024-06-26T07:07:21.0008891Z a97a71ef8a4e: Download complete 2024-06-26T07:07:21.0817446Z 25cd0c482ad6: Download complete 2024-06-26T07:07:22.3560104Z 9bf94bc45658: Verifying Checksum 2024-06-26T07:07:22.3560593Z 9bf94bc45658: Download complete 2024-06-26T07:07:22.4253002Z de45a14c9b22: Verifying Checksum 2024-06-26T07:07:22.4253488Z de45a14c9b22: Download complete 2024-06-26T07:07:22.5040309Z e99241e77209: Verifying Checksum 2024-06-26T07:07:22.5040994Z e99241e77209: Download complete 2024-06-26T07:07:22.5790811Z 35e8f4e2a42f: Verifying Checksum 2024-06-26T07:07:22.5791381Z 35e8f4e2a42f: Download complete 2024-06-26T07:07:22.6541003Z a823ffa4c541: Verifying Checksum 2024-06-26T07:07:22.6541567Z a823ffa4c541: Download complete 2024-06-26T07:07:22.7364041Z 44ba272eae20: Verifying Checksum 2024-06-26T07:07:22.7364600Z 44ba272eae20: Download complete 2024-06-26T07:07:27.2545616Z da02d278e524: Verifying Checksum 2024-06-26T07:07:27.2546290Z da02d278e524: Download complete 2024-06-26T07:07:27.3375966Z 73a0ce681c29: Download complete 2024-06-26T07:07:27.4098048Z 49737bd5ae70: Verifying Checksum 2024-06-26T07:07:27.4098525Z 49737bd5ae70: Download complete 2024-06-26T07:07:27.8253079Z 9cd4c745f8e4: Verifying Checksum 2024-06-26T07:07:27.8253600Z 9cd4c745f8e4: Download complete 2024-06-26T07:07:27.9016729Z b2950a036534: Verifying Checksum 2024-06-26T07:07:27.9017347Z b2950a036534: Download complete 2024-06-26T07:07:27.9720356Z 25e19d81d9fc: Verifying Checksum 2024-06-26T07:07:28.2352066Z 60156b40fcbe: Verifying Checksum 2024-06-26T07:07:28.2352637Z 60156b40fcbe: Download complete 2024-06-26T07:07:28.3138867Z 7a868239227f: Verifying Checksum 2024-06-26T07:07:28.3139507Z 7a868239227f: Download complete 2024-06-26T07:07:28.3771409Z 83911d7c7ecf: Verifying Checksum 2024-06-26T07:07:28.3771971Z 83911d7c7ecf: Download complete 2024-06-26T07:07:28.4453429Z b3c5d66c679f: Verifying Checksum 2024-06-26T07:07:28.4453985Z b3c5d66c679f: Download complete 2024-06-26T07:07:32.9888381Z 7ed32bc8e469: Verifying Checksum 2024-06-26T07:07:32.9889230Z 7ed32bc8e469: Download complete 2024-06-26T07:07:33.0592402Z c1068d03a811: Download complete 2024-06-26T07:07:33.1444962Z 671b4a301d81: Verifying Checksum 2024-06-26T07:07:33.1445879Z 671b4a301d81: Download complete 2024-06-26T07:07:33.5834484Z a8c1e85b5e14: Pull complete 2024-06-26T07:07:33.6588952Z 67760cf5ec46: Verifying Checksum 2024-06-26T07:07:33.6589485Z 67760cf5ec46: Download complete 2024-06-26T07:07:33.7419939Z ddc9740bcea2: Verifying Checksum 2024-06-26T07:07:33.7420447Z ddc9740bcea2: Download complete 2024-06-26T07:07:33.8182645Z fb2e97768560: Download complete 2024-06-26T07:07:33.8195100Z a41a8d1c11c8: Pull complete 2024-06-26T07:07:33.8808894Z b599c5a8c7cb: Verifying Checksum 2024-06-26T07:07:33.8809505Z b599c5a8c7cb: Download complete 2024-06-26T07:07:33.9573562Z e08e88c13668: Verifying Checksum 2024-06-26T07:07:33.9574038Z e08e88c13668: Download complete 2024-06-26T07:07:34.0660714Z 0c1227890755: Pull complete 2024-06-26T07:07:34.3126563Z d8d1234baab3: Pull complete 2024-06-26T07:07:39.7976491Z 3d914950f8ad: Download complete 2024-06-26T07:07:39.8855227Z cf7c2cb896cb: Verifying Checksum 2024-06-26T07:07:39.8855932Z cf7c2cb896cb: Download complete 2024-06-26T07:07:39.9621838Z 3a778ddfac2a: Verifying Checksum 2024-06-26T07:07:39.9622534Z 3a778ddfac2a: Download complete 2024-06-26T07:07:40.0377074Z 236c8f0245f4: Download complete 2024-06-26T07:07:40.0479007Z 4f4fb700ef54: Verifying Checksum 2024-06-26T07:07:40.0479770Z 4f4fb700ef54: Download complete 2024-06-26T07:07:40.1332939Z dda1ce302b9a: Download complete 2024-06-26T07:07:40.2136866Z e797fe86caef: Verifying Checksum 2024-06-26T07:07:40.3623910Z 967190076fc0: Verifying Checksum 2024-06-26T07:07:40.3624404Z 967190076fc0: Download complete 2024-06-26T07:07:40.4382955Z 4fb0a83f0685: Download complete 2024-06-26T07:07:40.5240582Z 68af2ec4e30a: Download complete 2024-06-26T07:07:40.6134270Z cc91933e985a: Download complete 2024-06-26T07:07:40.6908832Z 9f3187916e7d: Verifying Checksum 2024-06-26T07:07:40.8098401Z 9f3187916e7d: Download complete 2024-06-26T07:07:40.8099047Z 526083994857: Verifying Checksum 2024-06-26T07:07:40.8099531Z 526083994857: Download complete 2024-06-26T07:07:40.8849548Z 05a6f6dcc1c0: Verifying Checksum 2024-06-26T07:07:40.8850082Z 05a6f6dcc1c0: Download complete 2024-06-26T07:07:41.4788034Z 6bc9dc8315b0: Verifying Checksum 2024-06-26T07:07:41.4788604Z 6bc9dc8315b0: Download complete 2024-06-26T07:07:41.5649547Z 1b0e27c35f07: Verifying Checksum 2024-06-26T07:07:41.5650289Z 1b0e27c35f07: Download complete 2024-06-26T07:08:04.2353591Z 12ba7431ff38: Verifying Checksum 2024-06-26T07:08:04.2355142Z 12ba7431ff38: Download complete 2024-06-26T07:08:04.3190412Z 789eff6c51aa: Verifying Checksum 2024-06-26T07:08:04.3191065Z 789eff6c51aa: Download complete 2024-06-26T07:08:04.3901430Z de0f8a8c45c6: Verifying Checksum 2024-06-26T07:08:04.3901898Z de0f8a8c45c6: Download complete 2024-06-26T07:08:05.8013889Z 2060028f7488: Verifying Checksum 2024-06-26T07:08:05.8014454Z 2060028f7488: Download complete 2024-06-26T07:08:05.8921296Z e412c7f3b2de: Verifying Checksum 2024-06-26T07:08:05.8921966Z e412c7f3b2de: Download complete 2024-06-26T07:08:09.6570305Z 95237e7311a1: Verifying Checksum 2024-06-26T07:08:32.5180653Z 84b2c7a89982: Verifying Checksum 2024-06-26T07:08:32.5181128Z 84b2c7a89982: Download complete 2024-06-26T07:08:49.8529748Z 7ed32bc8e469: Pull complete 2024-06-26T07:08:50.0711052Z ec1e7978c1fe: Pull complete 2024-06-26T07:08:50.2996968Z 309b872da805: Pull complete 2024-06-26T07:08:57.6712900Z e82932cb002c: Pull complete 2024-06-26T07:08:57.8767015Z ca9dd73e3e93: Pull complete 2024-06-26T07:08:58.1019114Z 16d76a79d5d6: Pull complete 2024-06-26T07:08:58.3251995Z 4bdf41c9ecbe: Pull complete 2024-06-26T07:09:00.7111944Z f4863801e30c: Pull complete 2024-06-26T07:09:00.9159625Z d37dd9ed8948: Pull complete 2024-06-26T07:09:01.1454321Z cf2f34f225db: Pull complete 2024-06-26T07:09:01.3518605Z df546d7b3e34: Pull complete 2024-06-26T07:09:50.4750525Z 3d914950f8ad: Pull complete 2024-06-26T07:09:50.6506672Z 96d0e2032184: Pull complete 2024-06-26T07:09:50.8104119Z 06e12ce59790: Pull complete 2024-06-26T07:09:50.9668256Z ed712f63cde7: Pull complete 2024-06-26T07:09:51.0807955Z a97a71ef8a4e: Pull complete 2024-06-26T07:09:51.1862522Z 25cd0c482ad6: Pull complete 2024-06-26T07:09:54.6483223Z 9bf94bc45658: Pull complete 2024-06-26T07:09:54.7596752Z de45a14c9b22: Pull complete 2024-06-26T07:09:54.9791365Z e99241e77209: Pull complete 2024-06-26T07:09:55.1925519Z 35e8f4e2a42f: Pull complete 2024-06-26T07:09:55.4192809Z a823ffa4c541: Pull complete 2024-06-26T07:09:55.6416959Z 44ba272eae20: Pull complete 2024-06-26T07:10:03.9895044Z da02d278e524: Pull complete 2024-06-26T07:10:04.1421651Z 73a0ce681c29: Pull complete 2024-06-26T07:10:04.3395592Z 49737bd5ae70: Pull complete 2024-06-26T07:10:05.1966446Z 9cd4c745f8e4: Pull complete 2024-06-26T07:10:05.4124595Z b2950a036534: Pull complete 2024-06-26T07:10:05.6301552Z 25e19d81d9fc: Pull complete 2024-06-26T07:10:06.0299464Z 60156b40fcbe: Pull complete 2024-06-26T07:10:06.1775011Z 7a868239227f: Pull complete 2024-06-26T07:10:06.6393885Z 83911d7c7ecf: Pull complete 2024-06-26T07:10:06.8194475Z b3c5d66c679f: Pull complete 2024-06-26T07:10:46.0085174Z 84b2c7a89982: Pull complete 2024-06-26T07:10:46.1442758Z c1068d03a811: Pull complete 2024-06-26T07:10:46.1779261Z 671b4a301d81: Pull complete 2024-06-26T07:10:48.1540827Z 67760cf5ec46: Pull complete 2024-06-26T07:10:48.2005085Z ddc9740bcea2: Pull complete 2024-06-26T07:10:48.2387061Z fb2e97768560: Pull complete 2024-06-26T07:10:48.3359774Z b599c5a8c7cb: Pull complete 2024-06-26T07:10:48.3793821Z e08e88c13668: Pull complete 2024-06-26T07:11:22.4187949Z 12ba7431ff38: Pull complete 2024-06-26T07:11:22.5742678Z cf7c2cb896cb: Pull complete 2024-06-26T07:11:22.7712999Z 3a778ddfac2a: Pull complete 2024-06-26T07:11:23.2640513Z 236c8f0245f4: Pull complete 2024-06-26T07:11:23.4894467Z 4f4fb700ef54: Pull complete 2024-06-26T07:11:23.7165247Z dda1ce302b9a: Pull complete 2024-06-26T07:11:23.9341816Z e797fe86caef: Pull complete 2024-06-26T07:11:24.3231721Z 967190076fc0: Pull complete 2024-06-26T07:11:24.5492966Z 4fb0a83f0685: Pull complete 2024-06-26T07:11:24.7738370Z 68af2ec4e30a: Pull complete 2024-06-26T07:11:24.9861551Z cc91933e985a: Pull complete 2024-06-26T07:11:25.2106321Z 9f3187916e7d: Pull complete 2024-06-26T07:11:26.2241281Z 526083994857: Pull complete 2024-06-26T07:11:26.4079547Z 05a6f6dcc1c0: Pull complete 2024-06-26T07:11:28.4863292Z 6bc9dc8315b0: Pull complete 2024-06-26T07:11:28.7110042Z 1b0e27c35f07: Pull complete 2024-06-26T07:11:42.4839727Z 95237e7311a1: Pull complete 2024-06-26T07:11:42.7082161Z 789eff6c51aa: Pull complete 2024-06-26T07:11:42.9345971Z de0f8a8c45c6: Pull complete 2024-06-26T07:11:43.8302974Z 2060028f7488: Pull complete 2024-06-26T07:11:44.0519215Z e412c7f3b2de: Pull complete 2024-06-26T07:11:45.1687965Z Digest: sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:11:45.2087831Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:11:45.2272731Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:11:45.2329133Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:11:45.2330166Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:11:45.2338338Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:45.2338838Z env: 2024-06-26T07:11:45.2339119Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:45.2339475Z ##[endgroup] 2024-06-26T07:11:45.2491482Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-26T07:11:45.2492032Z with: 2024-06-26T07:11:45.2492326Z driver-version: 550.54.15 2024-06-26T07:11:45.2492678Z env: 2024-06-26T07:11:45.2492961Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:45.2493365Z ##[endgroup] 2024-06-26T07:11:45.2600867Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:11:45.2601422Z with: 2024-06-26T07:11:45.2601705Z timeout_minutes: 10 2024-06-26T07:11:45.2602027Z max_attempts: 3 2024-06-26T07:11:45.2632999Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-26T07:11:45.2664148Z retry_wait_seconds: 10 2024-06-26T07:11:45.2664621Z polling_interval_seconds: 1 2024-06-26T07:11:45.2664996Z warning_on_retry: true 2024-06-26T07:11:45.2665353Z continue_on_error: false 2024-06-26T07:11:45.2665698Z env: 2024-06-26T07:11:45.2665975Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:45.2666329Z DRIVER_VERSION: 550.54.15 2024-06-26T07:11:45.2666684Z ##[endgroup] 2024-06-26T07:11:45.3187110Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-26T07:11:45.3187951Z + pre_install_nvidia_driver_amzn2 2024-06-26T07:11:45.3188472Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-26T07:11:45.6228723Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:11:45.6616420Z No Match for argument: nvidia-driver-latest-dkms 2024-06-26T07:11:45.6856778Z No Packages marked for removal 2024-06-26T07:11:45.6972367Z + install_nvidia_driver_common 2024-06-26T07:11:45.6974941Z + echo 'Before installing NVIDIA driver' 2024-06-26T07:11:45.6975633Z + lspci 2024-06-26T07:11:45.6976031Z Before installing NVIDIA driver 2024-06-26T07:11:45.7056298Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:11:45.7057090Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:11:45.7058043Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:11:45.7058854Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:11:45.7059620Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:11:45.7060389Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:11:45.7061128Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:11:45.7061918Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:11:45.7062506Z + lsmod 2024-06-26T07:11:45.7070201Z Module Size Used by 2024-06-26T07:11:45.7070633Z ib_core 266240 0 2024-06-26T07:11:45.7071032Z nvidia_modeset 1339392 0 2024-06-26T07:11:45.7071563Z veth 16384 0 2024-06-26T07:11:45.7071996Z nvidia_uvm 4599808 0 2024-06-26T07:11:45.7072450Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:11:45.7073115Z drm 421888 1 nvidia 2024-06-26T07:11:45.7073619Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:11:45.7074146Z backlight 16384 1 nvidia_modeset 2024-06-26T07:11:45.7074592Z xt_conntrack 16384 1 2024-06-26T07:11:45.7074978Z ipt_MASQUERADE 16384 1 2024-06-26T07:11:45.7075416Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:11:45.7075892Z nf_conntrack_netlink 49152 0 2024-06-26T07:11:45.7076348Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:11:45.7076812Z xfrm_user 45056 1 2024-06-26T07:11:45.7077207Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:11:45.7077626Z iptable_nat 16384 1 2024-06-26T07:11:45.7078024Z nf_conntrack_ipv4 16384 3 2024-06-26T07:11:45.7078515Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:11:45.7079031Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:11:45.7079751Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T07:11:45.7080733Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T07:11:45.7081570Z xt_addrtype 16384 2 2024-06-26T07:11:45.7081951Z iptable_filter 16384 1 2024-06-26T07:11:45.7082336Z br_netfilter 24576 0 2024-06-26T07:11:45.7082748Z bridge 172032 1 br_netfilter 2024-06-26T07:11:45.7083191Z stp 16384 1 bridge 2024-06-26T07:11:45.7083611Z llc 16384 2 bridge,stp 2024-06-26T07:11:45.7084088Z overlay 86016 0 2024-06-26T07:11:45.7084461Z sunrpc 393216 1 2024-06-26T07:11:45.7084963Z dm_mirror 28672 0 2024-06-26T07:11:45.7085361Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:11:45.7085854Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:11:45.7086364Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:11:45.7086815Z dax 69632 1 dm_mod 2024-06-26T07:11:45.7087220Z crc32_pclmul 16384 0 2024-06-26T07:11:45.7087604Z ghash_clmulni_intel 16384 0 2024-06-26T07:11:45.7088207Z pcbc 16384 0 2024-06-26T07:11:45.7088772Z mousedev 24576 0 2024-06-26T07:11:45.7089312Z aesni_intel 188416 0 2024-06-26T07:11:45.7089722Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:11:45.7090171Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:11:45.7090631Z glue_helper 16384 1 aesni_intel 2024-06-26T07:11:45.7091189Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:11:45.7091734Z psmouse 32768 0 2024-06-26T07:11:45.7092113Z evdev 20480 3 2024-06-26T07:11:45.7092562Z button 16384 0 2024-06-26T07:11:45.7092929Z ena 139264 0 2024-06-26T07:11:45.7093347Z ptp 20480 1 ena 2024-06-26T07:11:45.7093746Z pps_core 20480 1 ptp 2024-06-26T07:11:45.7094140Z crc32c_intel 24576 0 2024-06-26T07:11:45.7094512Z autofs4 49152 2 2024-06-26T07:11:45.7094873Z + modinfo nvidia 2024-06-26T07:11:45.7095610Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:11:45.7096308Z alias: char-major-195-* 2024-06-26T07:11:45.7096692Z version: 550.54.15 2024-06-26T07:11:45.7097036Z supported: external 2024-06-26T07:11:45.7097381Z license: NVIDIA 2024-06-26T07:11:45.7097810Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:11:45.7098298Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:11:45.7098759Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:11:45.7099471Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:11:45.7099985Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:11:45.7100477Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:11:45.7101083Z depends: i2c-core,drm 2024-06-26T07:11:45.7101447Z retpoline: Y 2024-06-26T07:11:45.7101757Z name: nvidia 2024-06-26T07:11:45.7102313Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:11:45.7102969Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:11:45.7103620Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:11:45.7104218Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:11:45.7104666Z parm: NVreg_RmLogonRC:int 2024-06-26T07:11:45.7105095Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:11:45.7105554Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:11:45.7106000Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:11:45.7106442Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:11:45.7106971Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:11:45.7107544Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:11:45.7108033Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:11:45.7108468Z parm: NVreg_EnableMSI:int 2024-06-26T07:11:45.7108880Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:11:45.7109339Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:11:45.7109876Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:11:45.7110468Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:11:45.7111031Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:11:45.7111645Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:11:45.7112253Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:11:45.7112873Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:11:45.7113623Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:11:45.7114119Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:11:45.7114672Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:11:45.7115225Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:11:45.7115730Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:11:45.7116198Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:11:45.7116694Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:11:45.7117164Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:11:45.7117615Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:11:45.7118124Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:11:45.7118659Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:11:45.7119138Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:11:45.7119635Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:11:45.7120148Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:11:45.7120639Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:11:45.7121146Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:11:45.7121646Z parm: NVreg_RmMsg:charp 2024-06-26T07:11:45.7122060Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:11:45.7122532Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:11:45.7123000Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:11:45.7123510Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:11:45.7123996Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:11:45.7124470Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:11:45.7124930Z parm: rm_firmware_active:charp 2024-06-26T07:11:45.7125345Z + HAS_NVIDIA_DRIVER=0 2024-06-26T07:11:45.7125716Z ++ command -v nvidia-smi 2024-06-26T07:11:45.7126107Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-26T07:11:45.7126477Z + set +e 2024-06-26T07:11:45.7126982Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-26T07:11:47.6982431Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-26T07:11:47.6982996Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:11:47.6983499Z + '[' 0 -ne 0 ']' 2024-06-26T07:11:47.6984410Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-26T07:11:47.6984856Z + HAS_NVIDIA_DRIVER=1 2024-06-26T07:11:47.6985608Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-26T07:11:47.6986484Z + set -e 2024-06-26T07:11:47.6986814Z + '[' 1 -eq 0 ']' 2024-06-26T07:11:47.6987394Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-26T07:11:47.6988090Z + post_install_nvidia_driver_common 2024-06-26T07:11:47.6988495Z + sudo modprobe nvidia 2024-06-26T07:11:47.7084759Z + echo 'After installing NVIDIA driver' 2024-06-26T07:11:47.7085357Z + lspci 2024-06-26T07:11:47.7085807Z After installing NVIDIA driver 2024-06-26T07:11:47.7163187Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:11:47.7163962Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:11:47.7165103Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:11:47.7165916Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:11:47.7166679Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:11:47.7167426Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:11:47.7168154Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:11:47.7169213Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:11:47.7169810Z + lsmod 2024-06-26T07:11:47.7177454Z Module Size Used by 2024-06-26T07:11:47.7177933Z ib_core 266240 0 2024-06-26T07:11:47.7178496Z nvidia_modeset 1339392 0 2024-06-26T07:11:47.7179221Z veth 16384 0 2024-06-26T07:11:47.7179717Z nvidia_uvm 4599808 0 2024-06-26T07:11:47.7180331Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:11:47.7181029Z drm 421888 1 nvidia 2024-06-26T07:11:47.7181557Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:11:47.7182030Z backlight 16384 1 nvidia_modeset 2024-06-26T07:11:47.7182479Z xt_conntrack 16384 1 2024-06-26T07:11:47.7182868Z ipt_MASQUERADE 16384 1 2024-06-26T07:11:47.7183311Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:11:47.7183780Z nf_conntrack_netlink 49152 0 2024-06-26T07:11:47.7184234Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:11:47.7184702Z xfrm_user 45056 1 2024-06-26T07:11:47.7185126Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:11:47.7185731Z iptable_nat 16384 1 2024-06-26T07:11:47.7186285Z nf_conntrack_ipv4 16384 3 2024-06-26T07:11:47.7186925Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:11:47.7187597Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:11:47.7188130Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T07:11:47.7189074Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T07:11:47.7189922Z xt_addrtype 16384 2 2024-06-26T07:11:47.7190307Z iptable_filter 16384 1 2024-06-26T07:11:47.7190685Z br_netfilter 24576 0 2024-06-26T07:11:47.7191101Z bridge 172032 1 br_netfilter 2024-06-26T07:11:47.7191549Z stp 16384 1 bridge 2024-06-26T07:11:47.7191966Z llc 16384 2 bridge,stp 2024-06-26T07:11:47.7192389Z overlay 86016 0 2024-06-26T07:11:47.7192768Z sunrpc 393216 1 2024-06-26T07:11:47.7193142Z dm_mirror 28672 0 2024-06-26T07:11:47.7193646Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:11:47.7194147Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:11:47.7194715Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:11:47.7195170Z dax 69632 1 dm_mod 2024-06-26T07:11:47.7195713Z crc32_pclmul 16384 0 2024-06-26T07:11:47.7196109Z ghash_clmulni_intel 16384 0 2024-06-26T07:11:47.7196498Z pcbc 16384 0 2024-06-26T07:11:47.7196871Z mousedev 24576 0 2024-06-26T07:11:47.7197241Z aesni_intel 188416 0 2024-06-26T07:11:47.7197645Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:11:47.7198109Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:11:47.7198563Z glue_helper 16384 1 aesni_intel 2024-06-26T07:11:47.7199132Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:11:47.7199689Z psmouse 32768 0 2024-06-26T07:11:47.7200065Z evdev 20480 3 2024-06-26T07:11:47.7200437Z button 16384 0 2024-06-26T07:11:47.7200813Z ena 139264 0 2024-06-26T07:11:47.7201192Z ptp 20480 1 ena 2024-06-26T07:11:47.7201602Z pps_core 20480 1 ptp 2024-06-26T07:11:47.7202004Z crc32c_intel 24576 0 2024-06-26T07:11:47.7202385Z autofs4 49152 2 2024-06-26T07:11:47.7202753Z + modinfo nvidia 2024-06-26T07:11:47.7203443Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:11:47.7204136Z alias: char-major-195-* 2024-06-26T07:11:47.7204527Z version: 550.54.15 2024-06-26T07:11:47.7204881Z supported: external 2024-06-26T07:11:47.7205230Z license: NVIDIA 2024-06-26T07:11:47.7205609Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:11:47.7206102Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:11:47.7206567Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:11:47.7207051Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:11:47.7207621Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:11:47.7208126Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:11:47.7208883Z depends: i2c-core,drm 2024-06-26T07:11:47.7209247Z retpoline: Y 2024-06-26T07:11:47.7209577Z name: nvidia 2024-06-26T07:11:47.7210135Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:11:47.7210800Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:11:47.7211453Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:11:47.7212060Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:11:47.7212521Z parm: NVreg_RmLogonRC:int 2024-06-26T07:11:47.7212967Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:11:47.7213430Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:11:47.7213882Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:11:47.7214333Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:11:47.7214882Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:11:47.7215537Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:11:47.7216040Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:11:47.7216487Z parm: NVreg_EnableMSI:int 2024-06-26T07:11:47.7216917Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:11:47.7217379Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:11:47.7217930Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:11:47.7218527Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:11:47.7219104Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:11:47.7219721Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:11:47.7220340Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:11:47.7220967Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:11:47.7221581Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:11:47.7222084Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:11:47.7222631Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:11:47.7223198Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:11:47.7223796Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:11:47.7224278Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:11:47.7224772Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:11:47.7225257Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:11:47.7225707Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:11:47.7226222Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:11:47.7226769Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:11:47.7227257Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:11:47.7227745Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:11:47.7228255Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:11:47.7228772Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:11:47.7229290Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:11:47.7229781Z parm: NVreg_RmMsg:charp 2024-06-26T07:11:47.7230202Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:11:47.7230687Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:11:47.7231171Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:11:47.7231633Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:11:47.7232123Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:11:47.7232624Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:11:47.7233081Z parm: rm_firmware_active:charp 2024-06-26T07:11:47.7233493Z + set +e 2024-06-26T07:11:47.7233836Z + nvidia-smi 2024-06-26T07:11:49.2827621Z Wed Jun 26 07:11:49 2024 2024-06-26T07:11:49.2828329Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:11:49.2829241Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-26T07:11:49.2830268Z |-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:11:49.2831095Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-26T07:11:49.2831997Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-26T07:11:49.2832702Z | | | MIG M. | 2024-06-26T07:11:49.2833260Z |=========================================+========================+======================| 2024-06-26T07:11:49.2926008Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-06-26T07:11:49.2926750Z | 0% 33C P0 60W / 300W | 0MiB / 23028MiB | 0% Default | 2024-06-26T07:11:49.2927400Z | | | N/A | 2024-06-26T07:11:49.2928096Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:11:49.2928986Z 2024-06-26T07:11:49.2929666Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:11:49.2930295Z | Processes: | 2024-06-26T07:11:49.2931011Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-26T07:11:49.2931709Z | ID ID Usage | 2024-06-26T07:11:49.2932285Z |=========================================================================================| 2024-06-26T07:11:49.2932948Z | No running processes found | 2024-06-26T07:11:49.2933700Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:11:49.9189828Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-26T07:11:51.4797320Z NVIDIA A10G 2024-06-26T07:11:51.9357035Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:11:51.9357975Z + '[' 0 -eq 0 ']' 2024-06-26T07:11:51.9358422Z + echo 'INFO: Ignoring allowed status 0' 2024-06-26T07:11:51.9358862Z + set -e 2024-06-26T07:11:51.9359177Z INFO: Ignoring allowed status 0 2024-06-26T07:11:51.9360771Z == Installing nvidia container toolkit for amzn2 == 2024-06-26T07:11:51.9362641Z + sudo yum install -y yum-utils 2024-06-26T07:11:52.2391037Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:11:53.8838875Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-26T07:11:53.8839590Z Nothing to do 2024-06-26T07:11:54.0341993Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-06-26T07:11:54.0343185Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:11:54.0344596Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:11:54.3515364Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:11:54.3758327Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:11:54.3759516Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:11:54.3760494Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:11:54.3871214Z + sudo yum install -y nvidia-docker2 2024-06-26T07:11:54.6843842Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:11:56.1411021Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-26T07:11:56.1411695Z Nothing to do 2024-06-26T07:11:56.2877186Z + sudo systemctl restart docker 2024-06-26T07:12:32.3627704Z Command completed after 1 attempt(s). 2024-06-26T07:12:32.3683703Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:12:32.3684485Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:12:32.3685180Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T07:12:32.3685844Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:12:32.3693892Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:32.3694385Z env: 2024-06-26T07:12:32.3694668Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:32.3695127Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:32.3695708Z ##[endgroup] 2024-06-26T07:12:32.5883891Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:12:32.6042435Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-06-26T07:12:32.6140974Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-06-26T07:12:32.7068281Z Prepare all required actions 2024-06-26T07:12:32.7068774Z Getting action download info 2024-06-26T07:12:32.8092287Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T07:12:32.9721012Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T07:12:33.0927593Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T07:12:33.0928113Z with: 2024-06-26T07:12:33.0928668Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:12:33.0929156Z s3-bucket: gha-artifacts 2024-06-26T07:12:33.0929520Z env: 2024-06-26T07:12:33.0929814Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:33.0930276Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:33.0930812Z ##[endgroup] 2024-06-26T07:12:33.0963454Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:12:33.0963918Z with: 2024-06-26T07:12:33.0964303Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:12:33.0964775Z s3-bucket: gha-artifacts 2024-06-26T07:12:33.0965133Z region: us-east-1 2024-06-26T07:12:33.0965443Z env: 2024-06-26T07:12:33.0965729Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:33.0966189Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:33.0966725Z ##[endgroup] 2024-06-26T07:12:33.5105286Z (node:7647) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:12:33.5106075Z 2024-06-26T07:12:33.5106376Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:12:33.5107130Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:12:33.5108374Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:12:34.2508634Z Found 1 objects with prefix pytorch/pytorch/9673646806/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-06-26T07:12:34.2509758Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:12:41.7678769Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:12:41.7684613Z Artifact download has finished successfully 2024-06-26T07:12:41.7837302Z ##[group]Run unzip -o artifacts.zip 2024-06-26T07:12:41.7837761Z unzip -o artifacts.zip 2024-06-26T07:12:41.7845433Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:41.7845952Z env: 2024-06-26T07:12:41.7846235Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:41.7846709Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:41.7847210Z ##[endgroup] 2024-06-26T07:12:41.7875347Z Archive: artifacts.zip 2024-06-26T07:12:41.7877338Z creating: dist/ 2024-06-26T07:12:43.7667882Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:12:43.7668831Z creating: build/custom_test_artifacts/ 2024-06-26T07:12:43.7669928Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T07:12:43.7670913Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T07:12:43.7672081Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:12:43.7676625Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:12:43.7677985Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T07:12:43.7679311Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:12:43.7680729Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:12:43.7682097Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:12:43.7683700Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:12:43.7685448Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:12:43.7686873Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:12:43.7688296Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:12:43.7690300Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:12:43.7691986Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:12:43.7693644Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:12:43.7695323Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:12:43.7697139Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:12:43.7698787Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:12:43.7700338Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:12:43.7701767Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:12:43.7735045Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:12:43.7773547Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:12:43.7775711Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:12:43.7818541Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:12:43.7820609Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:12:43.7822722Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:12:43.7824855Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:12:43.7826926Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:12:43.7828953Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:12:43.7831137Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:12:43.7833142Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:12:43.7835258Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:12:43.7837150Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:12:43.7838925Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:12:43.7840677Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:12:43.7842515Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:12:43.7844282Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:12:43.7846022Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:12:43.7891604Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:12:43.7950035Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:12:43.7951786Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:12:43.7953224Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:12:43.7954492Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:12:43.7955815Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T07:12:43.7957177Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T07:12:43.7958681Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T07:12:43.7960298Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T07:12:43.7961900Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T07:12:43.7963350Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T07:12:43.7964899Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T07:12:43.7966400Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T07:12:43.7967944Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T07:12:43.7969769Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T07:12:43.7971295Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T07:12:43.7979875Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T07:12:43.8108406Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T07:12:43.8109922Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T07:12:43.8111520Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T07:12:43.8113290Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T07:12:43.8115006Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T07:12:43.8116559Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T07:12:43.8118159Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T07:12:43.8119822Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T07:12:43.8121583Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T07:12:43.8123170Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T07:12:43.8124788Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T07:12:43.8136437Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T07:12:43.8216849Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T07:12:43.8218565Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:12:43.8220180Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:12:43.8221694Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T07:12:43.8222984Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T07:12:43.8224321Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T07:12:43.8225678Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-26T07:12:43.8226911Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T07:12:43.8228019Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T07:12:43.8229143Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T07:12:43.8333800Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T07:12:43.8394764Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T07:12:43.8395827Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T07:12:43.8396805Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T07:12:43.8397923Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:12:43.8403236Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:12:43.8404583Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T07:12:43.8405894Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:12:43.8407303Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:12:43.8408965Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:12:43.8410611Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:12:43.8412180Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:12:43.8413610Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:12:43.8415034Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:12:43.8416830Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:12:43.8418457Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:12:43.8420093Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:12:43.8421712Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:12:43.8423342Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:12:43.8424969Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:12:43.8426584Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:12:43.8427983Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:12:43.8460698Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:12:43.8499109Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:12:43.8501224Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:12:43.8544331Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:12:43.8546406Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:12:43.8548642Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:12:43.8550834Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:12:43.8552907Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:12:43.8554902Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:12:43.8557008Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:12:43.8558995Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:12:43.8560922Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:12:43.8562811Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:12:43.8564590Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:12:43.8566299Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:12:43.8568114Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:12:43.8570230Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:12:43.8571962Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:12:43.8617396Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:12:43.8675757Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:12:43.8677474Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:12:43.8678901Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:12:43.8680203Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:12:43.8681499Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T07:12:43.8682892Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T07:12:43.8684445Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T07:12:43.8686101Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T07:12:43.8687913Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T07:12:43.8689694Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T07:12:43.8691293Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T07:12:43.8692821Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T07:12:43.8694417Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T07:12:43.8696022Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T07:12:43.8697587Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T07:12:43.8706287Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T07:12:43.8769020Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T07:12:43.8770734Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:12:43.8772290Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:12:43.8773652Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T07:12:43.8774918Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T07:12:43.8776322Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T07:12:43.8777644Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-26T07:12:43.8778862Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T07:12:43.8780015Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T07:12:43.8781119Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T07:12:43.8825662Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T07:12:43.8826763Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T07:12:43.8827820Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T07:12:43.8829030Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:12:43.8834164Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:12:43.8835599Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T07:12:43.8836996Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:12:43.8838507Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:12:43.8840014Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:12:43.8841662Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:12:43.8843351Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:12:43.8844871Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:12:43.8846368Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:12:43.8848103Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:12:43.8850132Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:12:43.8851345Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:12:43.8852742Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:12:43.8853962Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:12:43.8855185Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:12:43.8856374Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:12:43.8857431Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:12:43.8889011Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:12:43.8926590Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:12:43.8928239Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:12:43.8972028Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:12:43.8974954Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:12:43.8978012Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:12:43.8980091Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:12:43.8981540Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:12:43.8982965Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:12:43.8984394Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:12:43.8985811Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:12:43.8987199Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:12:43.8988519Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:12:43.8989849Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:12:43.8991099Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:12:43.8992375Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:12:43.8993608Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:12:43.8994871Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:12:43.9036924Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:12:43.9095977Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:12:43.9098514Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:12:43.9099850Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:12:43.9100768Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:12:43.9101887Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T07:12:43.9102895Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T07:12:43.9104024Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T07:12:43.9105265Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T07:12:43.9106468Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T07:12:43.9107601Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T07:12:43.9108775Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T07:12:43.9110061Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T07:12:43.9111238Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T07:12:43.9112419Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T07:12:43.9113590Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T07:12:43.9114828Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T07:12:43.9224466Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T07:12:43.9225702Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T07:12:43.9226884Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T07:12:43.9228268Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T07:12:43.9229536Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T07:12:43.9230729Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T07:12:43.9231951Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T07:12:43.9233178Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T07:12:43.9234409Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T07:12:43.9235641Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T07:12:43.9236873Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T07:12:43.9248249Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T07:12:43.9301706Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T07:12:43.9303028Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:12:43.9304174Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:12:43.9305211Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T07:12:43.9306171Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T07:12:43.9307155Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T07:12:43.9308353Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-26T07:12:43.9309265Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T07:12:43.9310092Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T07:12:43.9310959Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T07:12:43.9406338Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T07:12:43.9448369Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T07:12:43.9449691Z creating: build/lib/ 2024-06-26T07:12:43.9532888Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T07:12:43.9971220Z inflating: build/lib/libprotobuf.a 2024-06-26T07:12:43.9980595Z inflating: build/lib/libpthreadpool.a 2024-06-26T07:12:43.9988557Z inflating: build/lib/libcpuinfo.a 2024-06-26T07:12:43.9996043Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T07:12:43.9996552Z inflating: build/lib/libclog.a 2024-06-26T07:12:44.0014314Z inflating: build/lib/libnnpack.a 2024-06-26T07:12:44.0016342Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T07:12:44.0078717Z inflating: build/lib/libgtest.a 2024-06-26T07:12:44.0150767Z inflating: build/lib/libbenchmark.a 2024-06-26T07:12:44.0212235Z inflating: build/lib/libasmjit.a 2024-06-26T07:12:44.0219345Z inflating: build/lib/libittnotify.a 2024-06-26T07:12:44.0246346Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T07:12:44.0368219Z inflating: build/lib/libgloo.a 2024-06-26T07:12:44.0389354Z inflating: build/lib/libfmt.a 2024-06-26T07:12:44.0478235Z inflating: build/lib/libc10.so 2024-06-26T07:12:44.0479953Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T07:12:44.0480433Z inflating: build/lib/libfoxi_loader.a 2024-06-26T07:12:44.0481321Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T07:12:44.0500200Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T07:12:44.0517252Z inflating: build/lib/libgmock.a 2024-06-26T07:12:44.0517705Z inflating: build/lib/libgtest_main.a 2024-06-26T07:12:44.0518397Z inflating: build/lib/libbenchmark_main.a 2024-06-26T07:12:44.1003379Z inflating: build/lib/libprotoc.a 2024-06-26T07:12:44.1374273Z inflating: build/lib/libgloo_cuda.a 2024-06-26T07:12:45.1120611Z inflating: build/lib/libdnnl.a 2024-06-26T07:12:45.1670782Z inflating: build/lib/libtensorpipe.a 2024-06-26T07:12:45.1724244Z inflating: build/lib/libc10_cuda.so 2024-06-26T07:12:45.1725313Z inflating: build/lib/libgmock_main.a 2024-06-26T07:12:45.2927719Z inflating: build/lib/libfbgemm.a 2024-06-26T07:12:45.3172442Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-26T07:12:45.3673339Z inflating: build/lib/libkineto.a 2024-06-26T07:12:45.3850998Z inflating: build/lib/libXNNPACK.a 2024-06-26T07:12:45.3891679Z inflating: build/lib/libonnx_proto.a 2024-06-26T07:12:45.4557861Z inflating: build/lib/libonnx.a 2024-06-26T07:12:47.7871475Z inflating: build/lib/libtorch_cpu.so 2024-06-26T07:12:47.7875809Z inflating: build/lib/libunbox_lib.a 2024-06-26T07:12:47.7879647Z inflating: build/lib/libshm.so 2024-06-26T07:12:49.7790297Z inflating: build/lib/libtorch_cuda.so 2024-06-26T07:12:49.7791310Z inflating: build/lib/libtorch.so 2024-06-26T07:12:50.5747114Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-26T07:12:50.5749057Z inflating: build/lib/libc10d_cuda_test.so 2024-06-26T07:12:50.7649561Z inflating: build/lib/libtorch_python.so 2024-06-26T07:12:50.7717674Z inflating: build/lib/libtorchbind_test.so 2024-06-26T07:12:50.7736823Z inflating: build/lib/libjitbackend_test.so 2024-06-26T07:12:50.7761593Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T07:12:50.7794622Z inflating: build/lib/libnnapi_backend.so 2024-06-26T07:12:50.7795347Z creating: build/bin/ 2024-06-26T07:12:50.7843594Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T07:12:50.7893398Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T07:12:50.7942219Z inflating: build/bin/c10_Device_test 2024-06-26T07:12:50.7999376Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T07:12:50.8051594Z inflating: build/bin/c10_Scalar_test 2024-06-26T07:12:50.8099869Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T07:12:50.8151932Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T07:12:50.8203187Z inflating: build/bin/c10_SymInt_test 2024-06-26T07:12:50.8256691Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T07:12:50.8311135Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T07:12:50.8362048Z inflating: build/bin/c10_Bitset_test 2024-06-26T07:12:50.8430641Z inflating: build/bin/c10_cow_test 2024-06-26T07:12:50.8478060Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T07:12:50.8527036Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T07:12:50.8575333Z inflating: build/bin/c10_Half_test 2024-06-26T07:12:50.8629832Z inflating: build/bin/c10_LeftRight_test 2024-06-26T07:12:50.8682936Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T07:12:50.8731426Z inflating: build/bin/c10_Synchronized_test 2024-06-26T07:12:50.8785312Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T07:12:50.8834958Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T07:12:50.8884147Z inflating: build/bin/c10_TypeList_test 2024-06-26T07:12:50.8931811Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T07:12:50.8981913Z inflating: build/bin/c10_accumulate_test 2024-06-26T07:12:50.9035333Z inflating: build/bin/c10_bfloat16_test 2024-06-26T07:12:50.9084097Z inflating: build/bin/c10_bit_cast_test 2024-06-26T07:12:50.9138747Z inflating: build/bin/c10_complex_math_test 2024-06-26T07:12:50.9192855Z inflating: build/bin/c10_complex_test 2024-06-26T07:12:50.9244293Z inflating: build/bin/c10_exception_test 2024-06-26T07:12:50.9293013Z inflating: build/bin/c10_flags_test 2024-06-26T07:12:50.9341585Z inflating: build/bin/c10_generic_math_test 2024-06-26T07:12:50.9500168Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T07:12:50.9549461Z inflating: build/bin/c10_irange_test 2024-06-26T07:12:50.9602018Z inflating: build/bin/c10_lazy_test 2024-06-26T07:12:50.9656627Z inflating: build/bin/c10_logging_test 2024-06-26T07:12:50.9729279Z inflating: build/bin/c10_optional_test 2024-06-26T07:12:50.9789609Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T07:12:50.9842057Z inflating: build/bin/c10_registry_test 2024-06-26T07:12:50.9989211Z inflating: build/bin/c10_small_vector_test 2024-06-26T07:12:51.0039036Z inflating: build/bin/c10_ssize_test 2024-06-26T07:12:51.0089517Z inflating: build/bin/c10_string_util_test 2024-06-26T07:12:51.0146560Z inflating: build/bin/c10_string_view_test 2024-06-26T07:12:51.0195554Z inflating: build/bin/c10_tempfile_test 2024-06-26T07:12:51.0243018Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T07:12:51.0296755Z inflating: build/bin/c10_typeid_test 2024-06-26T07:12:51.0730316Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T07:12:51.1163292Z inflating: build/bin/protoc 2024-06-26T07:12:51.1214900Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-26T07:12:51.1266332Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-26T07:12:51.1317550Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-26T07:12:51.1367965Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-26T07:12:51.1419937Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T07:12:51.1471276Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T07:12:51.1518805Z inflating: build/bin/c10_cuda_CUDATest 2024-06-26T07:12:51.1570042Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-26T07:12:51.1889886Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T07:12:51.2225060Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T07:12:51.2571881Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T07:12:51.2623210Z inflating: build/bin/FileStoreTest 2024-06-26T07:12:51.2677771Z inflating: build/bin/TCPStoreTest 2024-06-26T07:12:51.2729658Z inflating: build/bin/HashStoreTest 2024-06-26T07:12:51.2742754Z inflating: build/bin/ProcessGroupMPITest 2024-06-26T07:12:51.2794901Z inflating: build/bin/test_edge_op_registration 2024-06-26T07:12:51.2798848Z inflating: build/bin/torch_shm_manager 2024-06-26T07:12:51.2801556Z inflating: build/bin/example_allreduce 2024-06-26T07:12:51.2854914Z inflating: build/bin/test_dist_autograd 2024-06-26T07:12:51.2923317Z inflating: build/bin/test_cpp_rpc 2024-06-26T07:12:51.2924922Z inflating: build/bin/parallel_benchmark 2024-06-26T07:12:51.2989693Z inflating: build/bin/test_mobile_nnc 2024-06-26T07:12:51.2998117Z inflating: build/bin/aot_model_compiler_test 2024-06-26T07:12:51.3331999Z inflating: build/bin/test_lazy 2024-06-26T07:12:51.4478506Z inflating: build/bin/test_api 2024-06-26T07:12:51.4549353Z inflating: build/bin/Dict_test 2024-06-26T07:12:51.4600397Z inflating: build/bin/Dimname_test 2024-06-26T07:12:51.4662703Z inflating: build/bin/MaybeOwned_test 2024-06-26T07:12:51.4718227Z inflating: build/bin/NamedTensor_test 2024-06-26T07:12:51.4775785Z inflating: build/bin/apply_utils_test 2024-06-26T07:12:51.4832949Z inflating: build/bin/atest 2024-06-26T07:12:51.4894258Z inflating: build/bin/basic 2024-06-26T07:12:51.4947487Z inflating: build/bin/broadcast_test 2024-06-26T07:12:51.4996731Z inflating: build/bin/cpu_allocator_test 2024-06-26T07:12:51.5053344Z inflating: build/bin/cpu_generator_test 2024-06-26T07:12:51.5105097Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T07:12:51.5195661Z inflating: build/bin/cpu_rng_test 2024-06-26T07:12:51.5244293Z inflating: build/bin/dispatch_key_set_test 2024-06-26T07:12:51.5293651Z inflating: build/bin/dlconvertor_test 2024-06-26T07:12:51.5349963Z inflating: build/bin/extension_backend_test 2024-06-26T07:12:51.5403869Z inflating: build/bin/half_test 2024-06-26T07:12:51.5496445Z inflating: build/bin/ivalue_test 2024-06-26T07:12:51.5544827Z inflating: build/bin/lazy_tensor_test 2024-06-26T07:12:51.5597891Z inflating: build/bin/math_kernel_test 2024-06-26T07:12:51.5651337Z inflating: build/bin/memory_format_test 2024-06-26T07:12:51.5702756Z inflating: build/bin/memory_overlapping_test 2024-06-26T07:12:51.5754335Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T07:12:51.5809248Z inflating: build/bin/native_test 2024-06-26T07:12:51.5858181Z inflating: build/bin/operator_name_test 2024-06-26T07:12:51.5908522Z inflating: build/bin/operators_test 2024-06-26T07:12:51.5958920Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T07:12:51.6024455Z inflating: build/bin/pow_test 2024-06-26T07:12:51.6080159Z inflating: build/bin/quantized_test 2024-06-26T07:12:51.6129028Z inflating: build/bin/reduce_ops_test 2024-06-26T07:12:51.6179184Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T07:12:51.6233769Z inflating: build/bin/scalar_tensor_test 2024-06-26T07:12:51.6290676Z inflating: build/bin/scalar_test 2024-06-26T07:12:51.6341406Z inflating: build/bin/StorageUtils_test 2024-06-26T07:12:51.6392075Z inflating: build/bin/stride_properties_test 2024-06-26T07:12:51.6468477Z inflating: build/bin/tensor_iterator_test 2024-06-26T07:12:51.6523304Z inflating: build/bin/test_parallel 2024-06-26T07:12:51.6526274Z inflating: build/bin/thread_init_test 2024-06-26T07:12:51.6580157Z inflating: build/bin/type_ptr_test 2024-06-26T07:12:51.6638606Z inflating: build/bin/type_test 2024-06-26T07:12:51.6689347Z inflating: build/bin/undefined_tensor_test 2024-06-26T07:12:51.6690912Z inflating: build/bin/verify_api_visibility 2024-06-26T07:12:51.6758466Z inflating: build/bin/legacy_vmap_test 2024-06-26T07:12:51.6809071Z inflating: build/bin/weakref_test 2024-06-26T07:12:51.6859260Z inflating: build/bin/wrapdim_test 2024-06-26T07:12:51.6909274Z inflating: build/bin/xla_tensor_test 2024-06-26T07:12:51.6967584Z inflating: build/bin/IListRef_test 2024-06-26T07:12:51.7070590Z inflating: build/bin/List_test 2024-06-26T07:12:51.7135521Z inflating: build/bin/KernelFunction_test 2024-06-26T07:12:51.7253886Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T07:12:51.7347598Z inflating: build/bin/kernel_function_test 2024-06-26T07:12:51.7472277Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T07:12:51.7572759Z inflating: build/bin/kernel_lambda_test 2024-06-26T07:12:51.7633722Z inflating: build/bin/kernel_stackbased_test 2024-06-26T07:12:51.7727633Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T07:12:51.7777370Z inflating: build/bin/CppSignature_test 2024-06-26T07:12:51.7831693Z inflating: build/bin/backend_fallback_test 2024-06-26T07:12:51.7879689Z inflating: build/bin/op_allowlist_test 2024-06-26T07:12:51.8172293Z inflating: build/bin/op_registration_test 2024-06-26T07:12:51.8234418Z inflating: build/bin/inline_container_test 2024-06-26T07:12:51.8285856Z inflating: build/bin/cuda_apply_test 2024-06-26T07:12:51.8335485Z inflating: build/bin/cuda_allocator_test 2024-06-26T07:12:51.8392969Z inflating: build/bin/cuda_atomic_ops_test 2024-06-26T07:12:51.8446535Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-26T07:12:51.8513961Z inflating: build/bin/cuda_complex_math_test 2024-06-26T07:12:51.8571677Z inflating: build/bin/cuda_complex_test 2024-06-26T07:12:51.8620693Z inflating: build/bin/cuda_device_test 2024-06-26T07:12:51.8676970Z inflating: build/bin/cuda_cub_test 2024-06-26T07:12:51.8726415Z inflating: build/bin/cuda_dlconvertor_test 2024-06-26T07:12:51.8788847Z inflating: build/bin/cuda_distributions_test 2024-06-26T07:12:51.8844124Z inflating: build/bin/cuda_generator_test 2024-06-26T07:12:51.8893048Z inflating: build/bin/cuda_half_test 2024-06-26T07:12:51.8942339Z inflating: build/bin/cuda_integer_divider_test 2024-06-26T07:12:51.8990122Z inflating: build/bin/cuda_optional_test 2024-06-26T07:12:51.9040421Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-26T07:12:51.9092132Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-26T07:12:51.9140207Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-26T07:12:51.9199962Z inflating: build/bin/cuda_stream_test 2024-06-26T07:12:51.9247664Z inflating: build/bin/cuda_cudnn_test 2024-06-26T07:12:51.9297670Z inflating: build/bin/cuda_vectorized_test 2024-06-26T07:12:51.9311976Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T07:12:51.9375875Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T07:12:51.9431496Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-26T07:12:51.9492904Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-26T07:12:51.9553179Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-26T07:12:52.0360745Z inflating: build/bin/test_tensorexpr 2024-06-26T07:12:52.0921441Z inflating: build/bin/test_jit 2024-06-26T07:12:52.0922081Z creating: .additional_ci_files/ 2024-06-26T07:12:52.0976184Z inflating: .additional_ci_files/test-times.json 2024-06-26T07:12:52.1186530Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T07:12:52.1212270Z ##[group]Run rm artifacts.zip 2024-06-26T07:12:52.1212669Z rm artifacts.zip 2024-06-26T07:12:52.1222071Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:52.1222701Z env: 2024-06-26T07:12:52.1222994Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:52.1223462Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:52.1223959Z ##[endgroup] 2024-06-26T07:12:52.1938003Z ##[group]Run df -H 2024-06-26T07:12:52.1938497Z df -H 2024-06-26T07:12:52.1948488Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:52.1949235Z env: 2024-06-26T07:12:52.1949659Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:52.1950320Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:52.1951032Z ##[endgroup] 2024-06-26T07:12:52.1979560Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T07:12:52.1980096Z devtmpfs 34G 0 34G 0% /dev 2024-06-26T07:12:52.1980576Z tmpfs 34G 0 34G 0% /dev/shm 2024-06-26T07:12:52.1981466Z tmpfs 34G 455k 34G 1% /run 2024-06-26T07:12:52.1981973Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-06-26T07:12:52.1982517Z /dev/nvme0n1p1 162G 43G 119G 27% / 2024-06-26T07:12:52.2021832Z Prepare all required actions 2024-06-26T07:12:52.2022284Z Getting action download info 2024-06-26T07:12:52.3099598Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T07:12:52.3100063Z with: 2024-06-26T07:12:52.3100333Z env: 2024-06-26T07:12:52.3100618Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:52.3101080Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:52.3101568Z ##[endgroup] 2024-06-26T07:12:52.3132437Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:12:52.3132882Z with: 2024-06-26T07:12:52.3133154Z name: td_results 2024-06-26T07:12:52.3133473Z s3-bucket: gha-artifacts 2024-06-26T07:12:52.3133827Z region: us-east-1 2024-06-26T07:12:52.3134135Z env: 2024-06-26T07:12:52.3134414Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:52.3134954Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:52.3135535Z ##[endgroup] 2024-06-26T07:12:52.7314182Z (node:7669) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:12:52.7314968Z 2024-06-26T07:12:52.7315243Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:12:52.7315986Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:12:52.7316827Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:12:52.8014831Z Found 0 objects with prefix pytorch/pytorch/9673646806/td_results/ 2024-06-26T07:12:52.8021448Z Artifact download has finished successfully 2024-06-26T07:12:52.8150526Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T07:12:52.8151024Z mkdir -p .additional_ci_files 2024-06-26T07:12:52.8151597Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T07:12:52.8159307Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:52.8159795Z env: 2024-06-26T07:12:52.8160073Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:52.8160548Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:52.8161033Z ##[endgroup] 2024-06-26T07:12:52.8195389Z mv: cannot stat 'td_results.json': No such file or directory 2024-06-26T07:12:52.8204907Z ##[error]Process completed with exit code 1. 2024-06-26T07:12:52.8237376Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T07:12:52.8237867Z .github/scripts/parse_ref.py 2024-06-26T07:12:52.8245336Z shell: /usr/bin/bash -e {0} 2024-06-26T07:12:52.8245690Z env: 2024-06-26T07:12:52.8245982Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:52.8246449Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:52.8246941Z ##[endgroup] 2024-06-26T07:12:52.8472794Z Prepare all required actions 2024-06-26T07:12:52.8509346Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T07:12:52.8509812Z with: 2024-06-26T07:12:52.8510356Z github-token: *** 2024-06-26T07:12:52.8510667Z env: 2024-06-26T07:12:52.8510944Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:52.8511519Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:52.8512004Z ##[endgroup] 2024-06-26T07:12:52.8527248Z ##[group]Run set -eux 2024-06-26T07:12:52.8527593Z set -eux 2024-06-26T07:12:52.8528186Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T07:12:52.8536099Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:52.8536595Z env: 2024-06-26T07:12:52.8536880Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:52.8537338Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:52.8537978Z GITHUB_TOKEN: *** 2024-06-26T07:12:52.8538292Z ##[endgroup] 2024-06-26T07:12:52.8557558Z + python3 .github/scripts/get_workflow_job_id.py 9673646806 i-0f01143c0858f0e6b 2024-06-26T07:12:53.4228862Z setting job-id=26688915598 2024-06-26T07:12:53.4229927Z setting job-name=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:53.4431777Z Prepare all required actions 2024-06-26T07:12:53.4432235Z Getting action download info 2024-06-26T07:12:53.5486423Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T07:12:53.5486884Z with: 2024-06-26T07:12:53.5487342Z github-token: *** 2024-06-26T07:12:53.5494215Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:12:53.5501600Z job-name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:53.5502547Z env: 2024-06-26T07:12:53.5502819Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:53.5503277Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:53.5503762Z ##[endgroup] 2024-06-26T07:12:53.5544007Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:12:53.5544563Z with: 2024-06-26T07:12:53.5544839Z shell: bash 2024-06-26T07:12:53.5545150Z timeout_minutes: 10 2024-06-26T07:12:53.5545486Z max_attempts: 5 2024-06-26T07:12:53.5545802Z retry_wait_seconds: 30 2024-06-26T07:12:53.5546917Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:12:53.5548073Z polling_interval_seconds: 1 2024-06-26T07:12:53.5548450Z warning_on_retry: true 2024-06-26T07:12:53.5548799Z continue_on_error: false 2024-06-26T07:12:53.5549287Z env: 2024-06-26T07:12:53.5549570Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:53.5550029Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:53.5550672Z GITHUB_TOKEN: *** 2024-06-26T07:12:53.5550992Z ##[endgroup] 2024-06-26T07:12:53.6031507Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:12:53.8193101Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:12:53.8349808Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-06-26T07:12:53.8494770Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-06-26T07:12:53.8503932Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-06-26T07:12:53.8516201Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-06-26T07:12:53.8536430Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-06-26T07:12:53.8734685Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2) 2024-06-26T07:12:54.6034590Z Command completed after 1 attempt(s). 2024-06-26T07:12:54.6073219Z ##[group]Run set -x 2024-06-26T07:12:54.6073555Z set -x 2024-06-26T07:12:54.6073858Z  2024-06-26T07:12:54.6074408Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:12:54.6075079Z # in runner workspace 2024-06-26T07:12:54.6075612Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T07:12:54.6083462Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:54.6083966Z env: 2024-06-26T07:12:54.6084262Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:54.6084720Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:54.6085198Z ##[endgroup] 2024-06-26T07:12:54.6106672Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T07:12:54.6303649Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:12:54.6304215Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:12:54.6304691Z echo "Job name: ${JOB_NAME}" 2024-06-26T07:12:54.6305107Z  2024-06-26T07:12:54.6305668Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:12:54.6306349Z # in runner workspace 2024-06-26T07:12:54.6306948Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T07:12:54.6307601Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T07:12:54.6308091Z  --job-name "${JOB_NAME}" \ 2024-06-26T07:12:54.6314727Z  --test-matrix "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-06-26T07:12:54.6321752Z  --selected-test-configs "" \ 2024-06-26T07:12:54.6322260Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T07:12:54.6322707Z  --tag "${TAG}" \ 2024-06-26T07:12:54.6323116Z  --event-name "${EVENT_NAME}" \ 2024-06-26T07:12:54.6323575Z  --schedule "${SCHEDULE}" \ 2024-06-26T07:12:54.6324135Z  --branch "${HEAD_BRANCH}" 2024-06-26T07:12:54.6332100Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:54.6349981Z env: 2024-06-26T07:12:54.6350313Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:54.6350792Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:54.6351476Z GITHUB_TOKEN: *** 2024-06-26T07:12:54.6352317Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:54.6353214Z PR_NUMBER: 2024-06-26T07:12:54.6353529Z TAG: ciflow/inductor/129470 2024-06-26T07:12:54.6353890Z EVENT_NAME: push 2024-06-26T07:12:54.6354200Z SCHEDULE: 2024-06-26T07:12:54.6354483Z HEAD_BRANCH: 2024-06-26T07:12:54.6354782Z ##[endgroup] 2024-06-26T07:12:54.6374325Z Workflow: inductor-periodic 2024-06-26T07:12:54.6375338Z Job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:54.8666933Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T07:12:54.9962806Z ##[group]Run echo "Filtered matrix:" 2024-06-26T07:12:54.9963269Z echo "Filtered matrix:" 2024-06-26T07:12:54.9970508Z echo "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-06-26T07:12:54.9977378Z  2024-06-26T07:12:54.9977660Z echo 2024-06-26T07:12:54.9978059Z echo "Is the current job unstable? False" 2024-06-26T07:12:54.9978652Z  2024-06-26T07:12:54.9978920Z echo 2024-06-26T07:12:54.9979283Z echo "Is keep-going label set? False" 2024-06-26T07:12:54.9979725Z  2024-06-26T07:12:54.9979995Z echo 2024-06-26T07:12:54.9980317Z echo "Renabled issues? " 2024-06-26T07:12:54.9988083Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:54.9988587Z env: 2024-06-26T07:12:54.9988859Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:54.9989318Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:54.9989807Z ##[endgroup] 2024-06-26T07:12:55.0013330Z Filtered matrix: 2024-06-26T07:12:55.0021422Z {include: [{config: dynamo_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-06-26T07:12:55.0027994Z 2024-06-26T07:12:55.0028149Z Is the current job unstable? False 2024-06-26T07:12:55.0028438Z 2024-06-26T07:12:55.0028665Z Is keep-going label set? False 2024-06-26T07:12:55.0028920Z 2024-06-26T07:12:55.0029051Z Renabled issues? 2024-06-26T07:12:55.0068861Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:12:55.0069576Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:12:55.0077202Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:55.0077696Z env: 2024-06-26T07:12:55.0077986Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:55.0078445Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:55.0078937Z JOB_TIMEOUT: 240 2024-06-26T07:12:55.0079293Z ##[endgroup] 2024-06-26T07:12:55.0146256Z ##[group]Run set -x 2024-06-26T07:12:55.0146651Z set -x 2024-06-26T07:12:55.0146965Z  2024-06-26T07:12:55.0147317Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T07:12:55.0147857Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T07:12:55.0148422Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T07:12:55.0149079Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-26T07:12:55.0149507Z else 2024-06-26T07:12:55.0149854Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:12:55.0150278Z fi 2024-06-26T07:12:55.0150562Z  2024-06-26T07:12:55.0151039Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T07:12:55.0151791Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T07:12:55.0152470Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T07:12:55.0153116Z # shellcheck disable=SC2086,SC2090 2024-06-26T07:12:55.0153585Z container_name=$(docker run \ 2024-06-26T07:12:55.0154132Z  ${GPU_FLAG:-} \ 2024-06-26T07:12:55.0154515Z  -e BUILD_ENVIRONMENT \ 2024-06-26T07:12:55.0154922Z  -e PR_NUMBER \ 2024-06-26T07:12:55.0155283Z  -e GITHUB_ACTIONS \ 2024-06-26T07:12:55.0155683Z  -e GITHUB_REPOSITORY \ 2024-06-26T07:12:55.0156100Z  -e GITHUB_WORKFLOW \ 2024-06-26T07:12:55.0156498Z  -e GITHUB_JOB \ 2024-06-26T07:12:55.0156863Z  -e GITHUB_RUN_ID \ 2024-06-26T07:12:55.0157257Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T07:12:55.0157672Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T07:12:55.0158067Z  -e JOB_ID \ 2024-06-26T07:12:55.0158417Z  -e JOB_NAME \ 2024-06-26T07:12:55.0158772Z  -e BASE_SHA \ 2024-06-26T07:12:55.0159128Z  -e BRANCH \ 2024-06-26T07:12:55.0159468Z  -e SHA1 \ 2024-06-26T07:12:55.0159824Z  -e AWS_DEFAULT_REGION \ 2024-06-26T07:12:55.0160243Z  -e IN_WHEEL_TEST \ 2024-06-26T07:12:55.0160641Z  -e SHARD_NUMBER \ 2024-06-26T07:12:55.0161015Z  -e TEST_CONFIG \ 2024-06-26T07:12:55.0161394Z  -e NUM_TEST_SHARDS \ 2024-06-26T07:12:55.0161795Z  -e REENABLED_ISSUES \ 2024-06-26T07:12:55.0162215Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T07:12:55.0162662Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T07:12:55.0163101Z  -e NO_TEST_TIMEOUT \ 2024-06-26T07:12:55.0163488Z  -e NO_TD \ 2024-06-26T07:12:55.0163838Z  -e TD_DISTRIBUTED \ 2024-06-26T07:12:55.0164218Z  -e PR_LABELS \ 2024-06-26T07:12:55.0164633Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T07:12:55.0165097Z  -e SCCACHE_BUCKET \ 2024-06-26T07:12:55.0165493Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-26T07:12:55.0165910Z  -e XLA_CUDA \ 2024-06-26T07:12:55.0166323Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T07:12:55.0166846Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T07:12:55.0167379Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T07:12:55.0167899Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-26T07:12:55.0168371Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-26T07:12:55.0168961Z  -e DASHBOARD_TAG \ 2024-06-26T07:12:55.0169445Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T07:12:55.0170008Z  --security-opt seccomp=unconfined \ 2024-06-26T07:12:55.0170485Z  --cap-add=SYS_PTRACE \ 2024-06-26T07:12:55.0170883Z  --ipc=host \ 2024-06-26T07:12:55.0171247Z  --shm-size="${SHM_SIZE}" \ 2024-06-26T07:12:55.0171652Z  --tty \ 2024-06-26T07:12:55.0171977Z  --detach \ 2024-06-26T07:12:55.0172349Z  --name="${container_name}" \ 2024-06-26T07:12:55.0172773Z  --user jenkins \ 2024-06-26T07:12:55.0173353Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T07:12:55.0173922Z  -w /var/lib/jenkins/workspace \ 2024-06-26T07:12:55.0174376Z  "${DOCKER_IMAGE}" 2024-06-26T07:12:55.0174725Z ) 2024-06-26T07:12:55.0175135Z # Propagate download.pytorch.org IP to container 2024-06-26T07:12:55.0176110Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-26T07:12:55.0177071Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-26T07:12:55.0177974Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-26T07:12:55.0185134Z shell: /usr/bin/bash -e {0} 2024-06-26T07:12:55.0185497Z env: 2024-06-26T07:12:55.0185779Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:55.0186242Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:55.0186851Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:12:55.0187363Z PR_NUMBER: 2024-06-26T07:12:55.0187690Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T07:12:55.0188209Z GITHUB_WORKFLOW: inductor-periodic 2024-06-26T07:12:55.0188610Z GITHUB_JOB: test 2024-06-26T07:12:55.0188930Z GITHUB_RUN_ID: 9673646806 2024-06-26T07:12:55.0189291Z GITHUB_RUN_NUMBER: 45567 2024-06-26T07:12:55.0189641Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T07:12:55.0189977Z JOB_ID: 26688915598 2024-06-26T07:12:55.0190824Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:55.0191726Z BRANCH: 2024-06-26T07:12:55.0192060Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:55.0192576Z BASE_SHA: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:55.0193119Z TEST_CONFIG: dynamo_eager_torchbench 2024-06-26T07:12:55.0193529Z SHARD_NUMBER: 2 2024-06-26T07:12:55.0193838Z NUM_TEST_SHARDS: 2 2024-06-26T07:12:55.0194170Z REENABLED_ISSUES: 2024-06-26T07:12:55.0194516Z CONTINUE_THROUGH_ERROR: False 2024-06-26T07:12:55.0194906Z VERBOSE_TEST_LOGS: False 2024-06-26T07:12:55.0195266Z NO_TEST_TIMEOUT: False 2024-06-26T07:12:55.0195603Z NO_TD: False 2024-06-26T07:12:55.0195910Z TD_DISTRIBUTED: False 2024-06-26T07:12:55.0196330Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T07:12:55.0196841Z SCCACHE_S3_KEY_PREFIX: inductor-periodic 2024-06-26T07:12:55.0197269Z SHM_SIZE: 2g 2024-06-26T07:12:55.0198352Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:12:55.0199498Z XLA_CUDA: 2024-06-26T07:12:55.0199978Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:12:55.0200591Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T07:12:55.0201030Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T07:12:55.0201440Z DASHBOARD_TAG: 2024-06-26T07:12:55.0201949Z HUGGING_FACE_HUB_TOKEN: *** 2024-06-26T07:12:55.0202328Z ##[endgroup] 2024-06-26T07:12:55.0220916Z + [[ dynamo_eager_torchbench == \m\u\l\t\i\g\p\u ]] 2024-06-26T07:12:55.0221646Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-06-26T07:12:55.0222169Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:12:55.0228327Z +++ nproc --ignore=2 2024-06-26T07:12:55.0241296Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9673646806 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:13:06.2340758Z + container_name=fe8dad447a4030c1c55cf9472d5e553b966e0b723306a0c787595c538bab5320 2024-06-26T07:13:06.2342082Z + grep download.pytorch.org /etc/hosts 2024-06-26T07:13:06.2343323Z + docker exec -i fe8dad447a4030c1c55cf9472d5e553b966e0b723306a0c787595c538bab5320 sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-26T07:13:06.2888744Z + echo DOCKER_CONTAINER_ID=fe8dad447a4030c1c55cf9472d5e553b966e0b723306a0c787595c538bab5320 2024-06-26T07:13:06.2891011Z ++ echo dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:13:06.2893029Z + docker exec -t fe8dad447a4030c1c55cf9472d5e553b966e0b723306a0c787595c538bab5320 sh -c 'pip install dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-26T07:13:06.6734500Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb8c4c54) 2024-06-26T07:13:06.9849446Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T07:13:06.9851763Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T07:13:06.9855611Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T07:13:06.9859186Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T07:13:06.9862143Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T07:13:06.9865331Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T07:13:06.9892336Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.3.0) 2024-06-26T07:13:06.9951796Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.21.2) 2024-06-26T07:13:07.0395315Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T07:13:07.0554602Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T07:13:08.3669716Z Installing collected packages: torch 2024-06-26T07:13:17.0408060Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-06-26T07:13:17.0409891Z timm 0.9.7 requires torchvision, which is not installed. 2024-06-26T07:13:17.0411140Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T07:13:17.1185514Z ++ dirname .ci/pytorch/test.sh 2024-06-26T07:13:17.1190669Z + source .ci/pytorch/common.sh 2024-06-26T07:13:17.1192965Z +++ dirname .ci/pytorch/common.sh 2024-06-26T07:13:17.1198212Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T07:13:17.1199678Z +++ declare -f -t trap_add 2024-06-26T07:13:17.1204867Z ++ set -ex 2024-06-26T07:13:17.1205473Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:13:17.1205993Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T07:13:17.1206796Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-06-26T07:13:17.1208355Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-26T07:13:17.1216396Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-26T07:13:17.1216862Z + trap_add cleanup_workspace EXIT 2024-06-26T07:13:17.1217377Z + trap_add_cmd=cleanup_workspace 2024-06-26T07:13:17.1219154Z + shift 2024-06-26T07:13:17.1219484Z + for trap_add_name in "$@" 2024-06-26T07:13:17.1221236Z +++ trap -p EXIT 2024-06-26T07:13:17.1222574Z ++ eval 'extract_trap_cmd ' 2024-06-26T07:13:17.1222949Z +++ extract_trap_cmd 2024-06-26T07:13:17.1223307Z +++ printf '%s\n' '' 2024-06-26T07:13:17.1223693Z ++ printf '%s\n' cleanup_workspace 2024-06-26T07:13:17.1224762Z + trap -- ' 2024-06-26T07:13:17.1225108Z cleanup_workspace' EXIT 2024-06-26T07:13:17.1225581Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-26T07:13:17.6091143Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-26T07:13:17.6103018Z + echo 'Environment variables:' 2024-06-26T07:13:17.6103440Z Environment variables: 2024-06-26T07:13:17.6103769Z + env 2024-06-26T07:13:17.6109278Z INSTALLED_DB=yes 2024-06-26T07:13:17.6109788Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:13:17.6110340Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:13:17.6110891Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:13:17.6111749Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:13:17.6112627Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:13:17.6113351Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:13:17.6114065Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:13:17.6114897Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:13:17.6115898Z HOSTNAME=fe8dad447a40 2024-06-26T07:13:17.6117166Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_ec9348dc-853b-4411-ada6-3b7062436196 2024-06-26T07:13:17.6118386Z GITHUB_ACTION=__self 2024-06-26T07:13:17.6118936Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:13:17.6124675Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:13:17.6131403Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:13:17.6132189Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:13:17.6132737Z GITHUB_RUN_NUMBER=45567 2024-06-26T07:13:17.6133269Z TEST_CONFIG=dynamo_eager_torchbench 2024-06-26T07:13:17.6133891Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:13:17.6134598Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:13:17.6135291Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:13:17.6136077Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:13:17.6136742Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:13:17.6137453Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:13:17.6138214Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:13:17.6138907Z GITHUB_REF_TYPE=tag 2024-06-26T07:13:17.6139410Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:13:17.6140001Z NCCL_VERSION=2.17.1-1 2024-06-26T07:13:17.6140913Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:17.6141573Z XLA_CUDA= 2024-06-26T07:13:17.6142286Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:13:17.6144646Z *** 2024-06-26T07:13:17.6145108Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:13:17.6145710Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:13:17.6146240Z GITHUB_ACTIONS=true 2024-06-26T07:13:17.6146753Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:17.6147525Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:13:17.6148342Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:13:17.6149093Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:17.6149877Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:13:17.6150630Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:17.6152069Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:13:17.6153256Z UCC_HOME=/usr 2024-06-26T07:13:17.6153800Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:13:17.6154376Z VERBOSE_TEST_LOGS=False 2024-06-26T07:13:17.6154831Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:13:17.6155699Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:13:17.6156463Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:13:17.6157200Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:13:17.6157785Z SHARD_NUMBER=2 2024-06-26T07:13:17.6158275Z GITHUB_REF_PROTECTED=false 2024-06-26T07:13:17.6158830Z HOME=/var/lib/jenkins 2024-06-26T07:13:17.6159415Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:13:17.6160072Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:13:17.6160793Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:13:17.6161647Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-06-26T07:13:17.6162280Z CUDA_VERSION=12.1.1 2024-06-26T07:13:17.6162908Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:13:17.6163600Z NUM_TEST_SHARDS=2 2024-06-26T07:13:17.6164035Z UCX_HOME=/usr 2024-06-26T07:13:17.6164804Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:13:17.6166482Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_ec9348dc-853b-4411-ada6-3b7062436196 2024-06-26T07:13:17.6168937Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:13:17.6171098Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_ec9348dc-853b-4411-ada6-3b7062436196 2024-06-26T07:13:17.6172873Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:13:17.6173888Z GITHUB_EVENT_NAME=push 2024-06-26T07:13:17.6174406Z DASHBOARD_TAG= 2024-06-26T07:13:17.6174871Z GITHUB_RUN_ID=9673646806 2024-06-26T07:13:17.6175751Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:13:17.6176584Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:13:17.6178148Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_ec9348dc-853b-4411-ada6-3b7062436196 2024-06-26T07:13:17.6179636Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:13:17.6180295Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:13:17.6180871Z PR_NUMBER= 2024-06-26T07:13:17.6181312Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:13:17.6181815Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:13:17.6182484Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:13:17.6183135Z TERM=xterm 2024-06-26T07:13:17.6183658Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:13:17.6184318Z INSTALLED_VISION=yes 2024-06-26T07:13:17.6184792Z BRANCH= 2024-06-26T07:13:17.6185239Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:13:17.6185881Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:13:17.6186544Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:13:17.6187728Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:13:17.6188908Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:13:17.6189850Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:13:17.6190584Z REENABLED_ISSUES= 2024-06-26T07:13:17.6191050Z SHLVL=1 2024-06-26T07:13:17.6191459Z MAX_JOBS=14 2024-06-26T07:13:17.6191952Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:13:17.6192467Z NVARCH=x86_64 2024-06-26T07:13:17.6192914Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:13:17.6193593Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:17.6194414Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:13:17.6195166Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:13:17.6196195Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:13:17.6197104Z GITHUB_JOB=test 2024-06-26T07:13:17.6197728Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:13:17.6198585Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:13:17.6199410Z NO_TEST_TIMEOUT=False 2024-06-26T07:13:17.6199928Z TD_DISTRIBUTED=False 2024-06-26T07:13:17.6200511Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:13:17.6201200Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:13:17.6201847Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:13:17.6202579Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:13:17.6203127Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:13:17.6203671Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:13:17.6205264Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:13:17.6206725Z GITHUB_BASE_REF= 2024-06-26T07:13:17.6207180Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:13:17.6207741Z CI=true 2024-06-26T07:13:17.6208234Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:13:17.6209161Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:13:17.6209712Z JOB_ID=26688915598 2024-06-26T07:13:17.6210215Z INSTALLED_PROTOBUF=yes 2024-06-26T07:13:17.6210690Z GITHUB_HEAD_REF= 2024-06-26T07:13:17.6211146Z GITHUB_ACTION_REF= 2024-06-26T07:13:17.6211803Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:13:17.6212489Z GITHUB_WORKFLOW=inductor-periodic 2024-06-26T07:13:17.6213114Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:13:17.6214475Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_ec9348dc-853b-4411-ada6-3b7062436196 2024-06-26T07:13:17.6215779Z NO_TD=False 2024-06-26T07:13:17.6216242Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:13:17.6216792Z _=/usr/bin/env 2024-06-26T07:13:17.6217557Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T07:13:17.6279601Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-26T07:13:17.6280791Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T07:13:17.6281826Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-26T07:13:17.6282769Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-26T07:13:17.6283380Z + BUILD_DIR=build 2024-06-26T07:13:17.6283736Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T07:13:17.6284204Z + BUILD_BIN_DIR=build/bin 2024-06-26T07:13:17.6284655Z + SHARD_NUMBER=2 2024-06-26T07:13:17.6284968Z + NUM_TEST_SHARDS=2 2024-06-26T07:13:17.6285287Z + export VALGRIND=ON 2024-06-26T07:13:17.6285601Z + VALGRIND=ON 2024-06-26T07:13:17.6286067Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-06-26T07:13:17.6286557Z + [[ 0 == \1 ]] 2024-06-26T07:13:17.6286853Z + [[ False == \1 ]] 2024-06-26T07:13:17.6287313Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-06-26T07:13:17.6287842Z ++ realpath build/custom_test_artifacts 2024-06-26T07:13:17.6290839Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-26T07:13:17.6291751Z + [[ -n '' ]] 2024-06-26T07:13:17.6292102Z + echo 'Environment variables' 2024-06-26T07:13:17.6292517Z Environment variables 2024-06-26T07:13:17.6292860Z + env 2024-06-26T07:13:17.6295898Z INSTALLED_DB=yes 2024-06-26T07:13:17.6296411Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:13:17.6297153Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:13:17.6297689Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:13:17.6298411Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:13:17.6298982Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:13:17.6299468Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:13:17.6299990Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:13:17.6300568Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:13:17.6301169Z HOSTNAME=fe8dad447a40 2024-06-26T07:13:17.6302163Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_ec9348dc-853b-4411-ada6-3b7062436196 2024-06-26T07:13:17.6302979Z GITHUB_ACTION=__self 2024-06-26T07:13:17.6303343Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:13:17.6307060Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:13:17.6311014Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:13:17.6311522Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:13:17.6311890Z GITHUB_RUN_NUMBER=45567 2024-06-26T07:13:17.6312260Z TEST_CONFIG=dynamo_eager_torchbench 2024-06-26T07:13:17.6312687Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:13:17.6313157Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:13:17.6313635Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:13:17.6314094Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:13:17.6314534Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:13:17.6314976Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:13:17.6315477Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:13:17.6315950Z GITHUB_REF_TYPE=tag 2024-06-26T07:13:17.6316280Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:13:17.6316662Z NCCL_VERSION=2.17.1-1 2024-06-26T07:13:17.6317062Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:17.6317520Z XLA_CUDA= 2024-06-26T07:13:17.6318009Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:13:17.6318411Z *** 2024-06-26T07:13:17.6318699Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:13:17.6319110Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:13:17.6319479Z GITHUB_ACTIONS=true 2024-06-26T07:13:17.6319828Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:17.6320321Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:13:17.6320863Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:13:17.6321333Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:17.6321848Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:13:17.6322338Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:17.6323254Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:13:17.6324015Z UCC_HOME=/usr 2024-06-26T07:13:17.6324361Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:13:17.6324770Z VERBOSE_TEST_LOGS=False 2024-06-26T07:13:17.6325127Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:13:17.6325582Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:13:17.6326091Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:13:17.6326564Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:13:17.6326948Z SHARD_NUMBER=2 2024-06-26T07:13:17.6327267Z GITHUB_REF_PROTECTED=false 2024-06-26T07:13:17.6327629Z HOME=/var/lib/jenkins 2024-06-26T07:13:17.6328004Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:13:17.6328922Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:13:17.6330682Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:13:17.6331426Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-06-26T07:13:17.6331888Z CUDA_VERSION=12.1.1 2024-06-26T07:13:17.6332334Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:13:17.6332948Z NUM_TEST_SHARDS=2 2024-06-26T07:13:17.6333369Z UCX_HOME=/usr 2024-06-26T07:13:17.6334011Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:13:17.6335102Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_ec9348dc-853b-4411-ada6-3b7062436196 2024-06-26T07:13:17.6336629Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:13:17.6338015Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_ec9348dc-853b-4411-ada6-3b7062436196 2024-06-26T07:13:17.6339143Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:13:17.6340048Z GITHUB_EVENT_NAME=push 2024-06-26T07:13:17.6340384Z DASHBOARD_TAG= 2024-06-26T07:13:17.6340693Z GITHUB_RUN_ID=9673646806 2024-06-26T07:13:17.6341153Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:13:17.6341682Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:13:17.6342666Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_ec9348dc-853b-4411-ada6-3b7062436196 2024-06-26T07:13:17.6343565Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:13:17.6343973Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:13:17.6344348Z PR_NUMBER= 2024-06-26T07:13:17.6344634Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:13:17.6344955Z VALGRIND=ON 2024-06-26T07:13:17.6345249Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:13:17.6345701Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:13:17.6346162Z TERM=xterm 2024-06-26T07:13:17.6346504Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:13:17.6346916Z INSTALLED_VISION=yes 2024-06-26T07:13:17.6347239Z BRANCH= 2024-06-26T07:13:17.6347528Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:13:17.6347954Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:13:17.6348374Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:13:17.6349167Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:13:17.6349911Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:13:17.6350405Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:13:17.6350865Z REENABLED_ISSUES= 2024-06-26T07:13:17.6363952Z SHLVL=1 2024-06-26T07:13:17.6364268Z MAX_JOBS=14 2024-06-26T07:13:17.6364608Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:13:17.6364976Z NVARCH=x86_64 2024-06-26T07:13:17.6365280Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:13:17.6365746Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:17.6366289Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:13:17.6366798Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:13:17.6367458Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:13:17.6368042Z GITHUB_JOB=test 2024-06-26T07:13:17.6368639Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:13:17.6369202Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:13:17.6369722Z NO_TEST_TIMEOUT=False 2024-06-26T07:13:17.6370055Z TD_DISTRIBUTED=False 2024-06-26T07:13:17.6370444Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:13:17.6370885Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:13:17.6371311Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:13:17.6371686Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:13:17.6372043Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:13:17.6372393Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:13:17.6373428Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:13:17.6374633Z GITHUB_BASE_REF= 2024-06-26T07:13:17.6374975Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:13:17.6375350Z CI=true 2024-06-26T07:13:17.6375763Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:13:17.6376209Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:13:17.6376576Z JOB_ID=26688915598 2024-06-26T07:13:17.6376892Z INSTALLED_PROTOBUF=yes 2024-06-26T07:13:17.6377221Z GITHUB_HEAD_REF= 2024-06-26T07:13:17.6377517Z GITHUB_ACTION_REF= 2024-06-26T07:13:17.6377938Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:13:17.6378472Z GITHUB_WORKFLOW=inductor-periodic 2024-06-26T07:13:17.6378915Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:13:17.6379812Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_ec9348dc-853b-4411-ada6-3b7062436196 2024-06-26T07:13:17.6380612Z NO_TD=False 2024-06-26T07:13:17.6380917Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:13:17.6381283Z _=/usr/bin/env 2024-06-26T07:13:17.6381613Z + echo 'Testing pytorch' 2024-06-26T07:13:17.6381969Z Testing pytorch 2024-06-26T07:13:17.6382297Z + export LANG=C.UTF-8 2024-06-26T07:13:17.6382722Z + LANG=C.UTF-8 2024-06-26T07:13:17.6383005Z + PR_NUMBER= 2024-06-26T07:13:17.6383362Z + [[ dynamo_eager_torchbench == \d\e\f\a\u\l\t ]] 2024-06-26T07:13:17.6383902Z + [[ dynamo_eager_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T07:13:17.6384420Z + [[ dynamo_eager_torchbench == \s\l\o\w ]] 2024-06-26T07:13:17.6385047Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-06-26T07:13:17.6385731Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-06-26T07:13:17.6386274Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:13:17.6386745Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:13:17.6387202Z + [[ dynamo_eager_torchbench == *crossref* ]] 2024-06-26T07:13:17.6387779Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:13:17.6388409Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-06-26T07:13:17.6389045Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:13:17.6389598Z + pip_install --user ninja==1.10.2 2024-06-26T07:13:17.6390133Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T07:13:17.9892711Z Collecting ninja==1.10.2 2024-06-26T07:13:18.0047855Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T07:13:18.0179676Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T07:13:19.3078256Z Installing collected packages: ninja 2024-06-26T07:13:19.3145507Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T07:13:19.3146775Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T07:13:19.3214910Z Successfully installed ninja-1.10.2 2024-06-26T07:13:19.4044016Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:13:19.4046129Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:13:19.4047658Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:13:19.4048176Z + install_tlparse 2024-06-26T07:13:19.4048842Z + pip_install --user tlparse==0.3.7 2024-06-26T07:13:19.4049436Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T07:13:19.7565928Z Collecting tlparse==0.3.7 2024-06-26T07:13:19.7712700Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T07:13:19.7824192Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T07:13:21.1181486Z Installing collected packages: tlparse 2024-06-26T07:13:21.1558359Z Successfully installed tlparse-0.3.7 2024-06-26T07:13:21.2383965Z ++ python -m site --user-base 2024-06-26T07:13:21.2538994Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:13:21.2540895Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-06-26T07:13:21.2541646Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-06-26T07:13:21.2542388Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:13:21.2543348Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-06-26T07:13:21.2544470Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-06-26T07:13:21.2545176Z + cd test 2024-06-26T07:13:21.2545711Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T07:13:22.7130305Z + [[ dynamo_eager_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T07:13:22.7131274Z + [[ dynamo_eager_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T07:13:22.7132124Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:13:22.7132758Z + pushd test 2024-06-26T07:13:22.7133084Z ~/workspace/test ~/workspace 2024-06-26T07:13:22.7133908Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T07:13:24.2171087Z + CUDA_VERSION=12.1 2024-06-26T07:13:24.2171719Z + '[' 12.1 == 12.4 ']' 2024-06-26T07:13:24.2172067Z + ISCUDA124= 2024-06-26T07:13:24.2172357Z + popd 2024-06-26T07:13:24.2172633Z ~/workspace 2024-06-26T07:13:24.2175128Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T07:13:24.2175755Z + [[ dynamo_eager_torchbench == *dynamo_eager* ]] 2024-06-26T07:13:24.2176365Z + DYNAMO_BENCHMARK_FLAGS+=(--backend eager) 2024-06-26T07:13:24.2176860Z + [[ dynamo_eager_torchbench == *dynamic* ]] 2024-06-26T07:13:24.2177357Z + [[ dynamo_eager_torchbench == *cpu_inductor* ]] 2024-06-26T07:13:24.2177913Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T07:13:24.2202156Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:13:24.2202859Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-06-26T07:13:24.2203615Z + cd test 2024-06-26T07:13:24.2204125Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T07:13:25.5648045Z PyTorch built with: 2024-06-26T07:13:25.5649057Z - GCC 9.4 2024-06-26T07:13:25.5649457Z - C++ Version: 201703 2024-06-26T07:13:25.5650322Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:13:25.5651397Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:13:25.5652071Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:13:25.5652583Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T07:13:25.5653078Z - NNPACK is enabled 2024-06-26T07:13:25.5653467Z - CPU capability usage: AVX2 2024-06-26T07:13:25.5653900Z - CUDA Runtime 12.1 2024-06-26T07:13:25.5654419Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-06-26T07:13:25.5655051Z - CuDNN 90.1 (built against CUDA 12.4) 2024-06-26T07:13:25.5655507Z - Magma 2.6.1 2024-06-26T07:13:25.5662733Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T07:13:25.5669181Z 2024-06-26T07:13:25.8342848Z + cd test 2024-06-26T07:13:25.8343611Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T07:13:27.0274096Z ATen/Parallel: 2024-06-26T07:13:27.0274759Z at::get_num_threads() : 8 2024-06-26T07:13:27.0275323Z at::get_num_interop_threads() : 16 2024-06-26T07:13:27.0275896Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:13:27.0276422Z omp_get_max_threads() : 8 2024-06-26T07:13:27.0277425Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:13:27.0278542Z mkl_get_max_threads() : 8 2024-06-26T07:13:27.0279172Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:13:27.0279813Z std::thread::hardware_concurrency() : 16 2024-06-26T07:13:27.0280247Z Environment variables: 2024-06-26T07:13:27.0280610Z OMP_NUM_THREADS : [not set] 2024-06-26T07:13:27.0281000Z MKL_NUM_THREADS : [not set] 2024-06-26T07:13:27.0281428Z ATen parallel backend: OpenMP 2024-06-26T07:13:27.0281700Z 2024-06-26T07:13:27.2675965Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:13:27.2676798Z + [[ dynamo_eager_torchbench == *backward* ]] 2024-06-26T07:13:27.2677288Z + [[ dynamo_eager_torchbench == *xla* ]] 2024-06-26T07:13:27.2677772Z + [[ dynamo_eager_torchbench == *executorch* ]] 2024-06-26T07:13:27.2678300Z + [[ dynamo_eager_torchbench == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T07:13:27.2678974Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:13:27.2679540Z + [[ dynamo_eager_torchbench == distributed ]] 2024-06-26T07:13:27.2680072Z + [[ dynamo_eager_torchbench == *inductor_distributed* ]] 2024-06-26T07:13:27.2680679Z + [[ dynamo_eager_torchbench == *inductor-halide* ]] 2024-06-26T07:13:27.2681366Z + [[ dynamo_eager_torchbench == *inductor-micro-benchmark* ]] 2024-06-26T07:13:27.2681925Z + [[ dynamo_eager_torchbench == *huggingface* ]] 2024-06-26T07:13:27.2682407Z + [[ dynamo_eager_torchbench == *timm* ]] 2024-06-26T07:13:27.2682881Z + [[ dynamo_eager_torchbench == *torchbench* ]] 2024-06-26T07:13:27.2683382Z + [[ dynamo_eager_torchbench == *cpu_inductor* ]] 2024-06-26T07:13:27.2683832Z + install_torchaudio cuda 2024-06-26T07:13:27.2684175Z + local commit 2024-06-26T07:13:27.2686582Z ++ get_pinned_commit audio 2024-06-26T07:13:27.2687071Z ++ cat .github/ci_commit_pins/audio.txt 2024-06-26T07:13:27.2688730Z + commit=b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:27.2689334Z + [[ cuda == \c\u\d\a ]] 2024-06-26T07:13:27.2689860Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-06-26T07:13:27.2690779Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:27.2692236Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:27.5777488Z Collecting git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:27.5780918Z Cloning https://github.com/pytorch/audio.git (to revision b829e936f7cc61b48149f5f957a451a38bf2a178) to /tmp/pip-req-build-p8qyhqi8 2024-06-26T07:13:27.5796279Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-p8qyhqi8 2024-06-26T07:13:28.2222737Z Running command git rev-parse -q --verify 'sha^b829e936f7cc61b48149f5f957a451a38bf2a178' 2024-06-26T07:13:28.2241892Z Running command git fetch -q https://github.com/pytorch/audio.git b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:28.5997751Z Running command git checkout -q b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:28.7142597Z Resolved https://github.com/pytorch/audio.git to commit b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:28.7143681Z Running command git submodule update --init --recursive -q 2024-06-26T07:13:30.8668383Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:13:30.8698424Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.4.0a0+b829e93) (2.5.0a0+gitb8c4c54) 2024-06-26T07:13:30.8760813Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (3.13.1) 2024-06-26T07:13:30.8765246Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (4.12.2) 2024-06-26T07:13:30.8768292Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (1.12.1) 2024-06-26T07:13:30.8771624Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (2.8.8) 2024-06-26T07:13:30.8774649Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (3.1.4) 2024-06-26T07:13:30.8777982Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (2024.2.0) 2024-06-26T07:13:30.9204780Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.4.0a0+b829e93) (2.1.5) 2024-06-26T07:13:30.9361646Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchaudio==2.4.0a0+b829e93) (1.3.0) 2024-06-26T07:13:30.9429837Z Building wheels for collected packages: torchaudio 2024-06-26T07:15:57.7604945Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-06-26T07:15:57.7634811Z [?25h Created wheel for torchaudio: filename=torchaudio-2.4.0a0+b829e93-cp310-cp310-linux_x86_64.whl size=2308817 sha256=ce2f767b9a8584f3cae292802124fba173bd747212aff6ec25a12cdc6001316d 2024-06-26T07:15:57.7636955Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/30/b3/9c/882f183f00af2e582825654a92a04400924519922db53ff522 2024-06-26T07:15:57.7663431Z Successfully built torchaudio 2024-06-26T07:15:59.0188736Z Installing collected packages: torchaudio 2024-06-26T07:15:59.2127378Z Successfully installed torchaudio-2.4.0a0+b829e93 2024-06-26T07:15:59.3432917Z + install_torchtext 2024-06-26T07:15:59.3433271Z + local data_commit 2024-06-26T07:15:59.3433608Z + local text_commit 2024-06-26T07:15:59.3436726Z ++ get_pinned_commit data 2024-06-26T07:15:59.3437230Z ++ cat .github/ci_commit_pins/data.txt 2024-06-26T07:15:59.3447460Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:59.3449465Z ++ get_pinned_commit text 2024-06-26T07:15:59.3451426Z ++ cat .github/ci_commit_pins/text.txt 2024-06-26T07:15:59.3461934Z + text_commit=b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:15:59.3463189Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:59.3464825Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:59.6575235Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:59.6581634Z Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-h0i3mcfs 2024-06-26T07:15:59.6597474Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-h0i3mcfs 2024-06-26T07:16:00.1547562Z Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f' 2024-06-26T07:16:00.1565532Z Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:16:00.3850000Z Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:16:00.5183061Z Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:16:00.5184138Z Running command git submodule update --init --recursive -q 2024-06-26T07:17:12.4561289Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:17:12.4614683Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.19) 2024-06-26T07:17:12.4616946Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.32.3) 2024-06-26T07:17:12.4622584Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.5.0a0+gitb8c4c54) 2024-06-26T07:17:12.4683521Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1) 2024-06-26T07:17:12.4689316Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.12.2) 2024-06-26T07:17:12.4692842Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.12.1) 2024-06-26T07:17:12.4694867Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8) 2024-06-26T07:17:12.4698004Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4) 2024-06-26T07:17:12.4700552Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.2.0) 2024-06-26T07:17:12.4889323Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2) 2024-06-26T07:17:12.4895680Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7) 2024-06-26T07:17:12.4901710Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.6.2) 2024-06-26T07:17:12.5365389Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5) 2024-06-26T07:17:12.5529656Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0) 2024-06-26T07:17:12.5611982Z Building wheels for collected packages: torchdata 2024-06-26T07:17:14.2831505Z Building wheel for torchdata (setup.py) ... [?25l- \ | done 2024-06-26T07:17:14.2838122Z [?25h Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182956 sha256=8f54880d036660afd595a182da450443e5bd7cd3daf6aaef1dfb999308a1cefd 2024-06-26T07:17:14.2840771Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce 2024-06-26T07:17:14.2866131Z Successfully built torchdata 2024-06-26T07:17:15.5042859Z Installing collected packages: torchdata 2024-06-26T07:17:15.5955656Z Successfully installed torchdata-0.7.0a0+11bb5b8 2024-06-26T07:17:18.1004021Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:17:18.1005480Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:17:18.4160108Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:17:18.4164173Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-g8ij705p 2024-06-26T07:17:18.4181820Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-g8ij705p 2024-06-26T07:17:19.4495076Z Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db' 2024-06-26T07:17:19.4509391Z Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:17:19.8979459Z Running command git checkout -q b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:17:20.0486343Z Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:17:20.0487443Z Running command git submodule update --init --recursive -q 2024-06-26T07:17:27.0588502Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:17:27.0639165Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.4) 2024-06-26T07:17:27.0642322Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.32.3) 2024-06-26T07:17:27.0645846Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.5.0a0+gitb8c4c54) 2024-06-26T07:17:27.0649055Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.21.2) 2024-06-26T07:17:27.0652261Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8) 2024-06-26T07:17:27.0713105Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2) 2024-06-26T07:17:27.0718382Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7) 2024-06-26T07:17:27.0724952Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.19) 2024-06-26T07:17:27.0730775Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.6.2) 2024-06-26T07:17:27.0786276Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1) 2024-06-26T07:17:27.0792690Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.12.2) 2024-06-26T07:17:27.0795629Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.12.1) 2024-06-26T07:17:27.0798721Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8) 2024-06-26T07:17:27.0801905Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4) 2024-06-26T07:17:27.0804670Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.2.0) 2024-06-26T07:17:27.1490014Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5) 2024-06-26T07:17:27.1649377Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0) 2024-06-26T07:17:27.1730405Z Building wheels for collected packages: torchtext 2024-06-26T07:18:03.2964886Z Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-06-26T07:18:03.2990419Z [?25h Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2045303 sha256=2255e6fe3ae7809752599828abab0382ff5e68193ac598dcb7169007b4b247c2 2024-06-26T07:18:03.2992238Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf 2024-06-26T07:18:03.3016961Z Successfully built torchtext 2024-06-26T07:18:04.4984524Z Installing collected packages: torchtext 2024-06-26T07:18:04.5969162Z Successfully installed torchtext-0.16.0a0+b0ebddc 2024-06-26T07:18:04.7181499Z + install_torchvision 2024-06-26T07:18:04.7181997Z + local orig_preload 2024-06-26T07:18:04.7182452Z + local commit 2024-06-26T07:18:04.7183162Z ++ get_pinned_commit vision 2024-06-26T07:18:04.7183644Z ++ cat .github/ci_commit_pins/vision.txt 2024-06-26T07:18:04.7193018Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:18:04.7193598Z + orig_preload= 2024-06-26T07:18:04.7194048Z + '[' -n '' ']' 2024-06-26T07:18:04.7194899Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:18:04.7196334Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:18:05.0306370Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:18:05.0310160Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-g9sf0zfn 2024-06-26T07:18:05.0326512Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-g9sf0zfn 2024-06-26T07:18:06.3977756Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-06-26T07:18:06.3996959Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:18:07.6255256Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:18:07.8620369Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:18:10.1516200Z Preparing metadata (setup.py) ... [?25l- \ done 2024-06-26T07:18:10.1565105Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-06-26T07:18:10.1567232Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+gitb8c4c54) 2024-06-26T07:18:10.1573464Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-06-26T07:18:10.1777672Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-06-26T07:18:10.1784800Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-06-26T07:18:10.1787760Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1) 2024-06-26T07:18:10.1791411Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-06-26T07:18:10.1794535Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-06-26T07:18:10.1797729Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-06-26T07:18:10.2222230Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-06-26T07:18:10.2373377Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-06-26T07:18:10.2447425Z Building wheels for collected packages: torchvision 2024-06-26T07:19:22.5618241Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-06-26T07:19:22.5644898Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2113266 sha256=786caae59e3b7289f3b48d8c85ab06683fa4b54162b3410db1af8ddd2a7b464f 2024-06-26T07:19:22.5646689Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-06-26T07:19:22.5682059Z Successfully built torchvision 2024-06-26T07:19:23.7792722Z Installing collected packages: torchvision 2024-06-26T07:19:24.1599767Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-06-26T07:19:24.2696083Z + '[' -n '' ']' 2024-06-26T07:19:24.2696450Z + id=1 2024-06-26T07:19:24.2696826Z + pip_install opencv-python==4.8.0.74 2024-06-26T07:19:24.2697414Z + pip install --progress-bar off opencv-python==4.8.0.74 2024-06-26T07:19:24.7012752Z Collecting opencv-python==4.8.0.74 2024-06-26T07:19:24.7186770Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2024-06-26T07:19:24.7361853Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.21.2) 2024-06-26T07:19:24.7426383Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2024-06-26T07:19:26.5187019Z Installing collected packages: opencv-python 2024-06-26T07:19:27.3385505Z Successfully installed opencv-python-4.8.0.74 2024-06-26T07:19:27.4313755Z + [[ dynamo_eager_torchbench == *inductor_torchbench_smoketest_perf* ]] 2024-06-26T07:19:27.4314793Z + [[ dynamo_eager_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]] 2024-06-26T07:19:27.4315718Z + [[ dynamo_eager_torchbench == *torchbench_gcp_smoketest* ]] 2024-06-26T07:19:27.4316403Z + checkout_install_torchbench 2024-06-26T07:19:27.4316771Z + local commit 2024-06-26T07:19:27.4317096Z ++ get_pinned_commit torchbench 2024-06-26T07:19:27.4317534Z ++ cat .github/ci_commit_pins/torchbench.txt 2024-06-26T07:19:27.4328009Z + commit=23512dbebd44a11eb84afbf53c3c071dd105297e 2024-06-26T07:19:27.4328855Z + git clone https://github.com/pytorch/benchmark torchbench 2024-06-26T07:19:27.4337613Z Cloning into 'torchbench'... 2024-06-26T07:19:27.5757238Z remote: Enumerating objects: 30187, done. 2024-06-26T07:19:27.5764737Z remote: Counting objects: 0% (1/546) 2024-06-26T07:19:27.5765453Z remote: Counting objects: 1% (6/546) 2024-06-26T07:19:27.5766414Z remote: Counting objects: 2% (11/546) 2024-06-26T07:19:27.5767141Z remote: Counting objects: 3% (17/546) 2024-06-26T07:19:27.5767835Z remote: Counting objects: 4% (22/546) 2024-06-26T07:19:27.5768384Z remote: Counting objects: 5% (28/546) 2024-06-26T07:19:27.5769069Z remote: Counting objects: 6% (33/546) 2024-06-26T07:19:27.5769573Z remote: Counting objects: 7% (39/546) 2024-06-26T07:19:27.5770092Z remote: Counting objects: 8% (44/546) 2024-06-26T07:19:27.5770797Z remote: Counting objects: 9% (50/546) 2024-06-26T07:19:27.5771503Z remote: Counting objects: 10% (55/546) 2024-06-26T07:19:27.5772122Z remote: Counting objects: 11% (61/546) 2024-06-26T07:19:27.5772670Z remote: Counting objects: 12% (66/546) 2024-06-26T07:19:27.5773209Z remote: Counting objects: 13% (71/546) 2024-06-26T07:19:27.5773859Z remote: Counting objects: 14% (77/546) 2024-06-26T07:19:27.5774458Z remote: Counting objects: 15% (82/546) 2024-06-26T07:19:27.5774964Z remote: Counting objects: 16% (88/546) 2024-06-26T07:19:27.5775573Z remote: Counting objects: 17% (93/546) 2024-06-26T07:19:27.5776831Z remote: Counting objects: 18% (99/546) 2024-06-26T07:19:27.5777519Z remote: Counting objects: 19% (104/546) 2024-06-26T07:19:27.5778125Z remote: Counting objects: 20% (110/546) 2024-06-26T07:19:27.5778768Z remote: Counting objects: 21% (115/546) 2024-06-26T07:19:27.5779551Z remote: Counting objects: 22% (121/546) 2024-06-26T07:19:27.5780255Z remote: Counting objects: 23% (126/546) 2024-06-26T07:19:27.5780961Z remote: Counting objects: 24% (132/546) 2024-06-26T07:19:27.5781671Z remote: Counting objects: 25% (137/546) 2024-06-26T07:19:27.5782346Z remote: Counting objects: 26% (142/546) 2024-06-26T07:19:27.5783021Z remote: Counting objects: 27% (148/546) 2024-06-26T07:19:27.5783615Z remote: Counting objects: 28% (153/546) 2024-06-26T07:19:27.5784119Z remote: Counting objects: 29% (159/546) 2024-06-26T07:19:27.5784630Z remote: Counting objects: 30% (164/546) 2024-06-26T07:19:27.5785137Z remote: Counting objects: 31% (170/546) 2024-06-26T07:19:27.5785654Z remote: Counting objects: 32% (175/546) 2024-06-26T07:19:27.5786241Z remote: Counting objects: 33% (181/546) 2024-06-26T07:19:27.5786747Z remote: Counting objects: 34% (186/546) 2024-06-26T07:19:27.5787253Z remote: Counting objects: 35% (192/546) 2024-06-26T07:19:27.5787768Z remote: Counting objects: 36% (197/546) 2024-06-26T07:19:27.5788265Z remote: Counting objects: 37% (203/546) 2024-06-26T07:19:27.5788771Z remote: Counting objects: 38% (208/546) 2024-06-26T07:19:27.5789272Z remote: Counting objects: 39% (213/546) 2024-06-26T07:19:27.5789775Z remote: Counting objects: 40% (219/546) 2024-06-26T07:19:27.5790274Z remote: Counting objects: 41% (224/546) 2024-06-26T07:19:27.5790776Z remote: Counting objects: 42% (230/546) 2024-06-26T07:19:27.5791277Z remote: Counting objects: 43% (235/546) 2024-06-26T07:19:27.5791790Z remote: Counting objects: 44% (241/546) 2024-06-26T07:19:27.5792284Z remote: Counting objects: 45% (246/546) 2024-06-26T07:19:27.5792808Z remote: Counting objects: 46% (252/546) 2024-06-26T07:19:27.5793312Z remote: Counting objects: 47% (257/546) 2024-06-26T07:19:27.5793816Z remote: Counting objects: 48% (263/546) 2024-06-26T07:19:27.5794309Z remote: Counting objects: 49% (268/546) 2024-06-26T07:19:27.5794809Z remote: Counting objects: 50% (273/546) 2024-06-26T07:19:27.5795316Z remote: Counting objects: 51% (279/546) 2024-06-26T07:19:27.5795818Z remote: Counting objects: 52% (284/546) 2024-06-26T07:19:27.5796314Z remote: Counting objects: 53% (290/546) 2024-06-26T07:19:27.5796819Z remote: Counting objects: 54% (295/546) 2024-06-26T07:19:27.5797329Z remote: Counting objects: 55% (301/546) 2024-06-26T07:19:27.5797840Z remote: Counting objects: 56% (306/546) 2024-06-26T07:19:27.5798337Z remote: Counting objects: 57% (312/546) 2024-06-26T07:19:27.5799010Z remote: Counting objects: 58% (317/546) 2024-06-26T07:19:27.5799523Z remote: Counting objects: 59% (323/546) 2024-06-26T07:19:27.5800024Z remote: Counting objects: 60% (328/546) 2024-06-26T07:19:27.5800539Z remote: Counting objects: 61% (334/546) 2024-06-26T07:19:27.5801083Z remote: Counting objects: 62% (339/546) 2024-06-26T07:19:27.5801582Z remote: Counting objects: 63% (344/546) 2024-06-26T07:19:27.5802076Z remote: Counting objects: 64% (350/546) 2024-06-26T07:19:27.5802583Z remote: Counting objects: 65% (355/546) 2024-06-26T07:19:27.5803084Z remote: Counting objects: 66% (361/546) 2024-06-26T07:19:27.5803585Z remote: Counting objects: 67% (366/546) 2024-06-26T07:19:27.5804079Z remote: Counting objects: 68% (372/546) 2024-06-26T07:19:27.5804583Z remote: Counting objects: 69% (377/546) 2024-06-26T07:19:27.5805085Z remote: Counting objects: 70% (383/546) 2024-06-26T07:19:27.5805588Z remote: Counting objects: 71% (388/546) 2024-06-26T07:19:27.5806088Z remote: Counting objects: 72% (394/546) 2024-06-26T07:19:27.5806591Z remote: Counting objects: 73% (399/546) 2024-06-26T07:19:27.5807171Z remote: Counting objects: 74% (405/546) 2024-06-26T07:19:27.5807672Z remote: Counting objects: 75% (410/546) 2024-06-26T07:19:27.5808167Z remote: Counting objects: 76% (415/546) 2024-06-26T07:19:27.5808994Z remote: Counting objects: 77% (421/546) 2024-06-26T07:19:27.5809504Z remote: Counting objects: 78% (426/546) 2024-06-26T07:19:27.5809997Z remote: Counting objects: 79% (432/546) 2024-06-26T07:19:27.5810500Z remote: Counting objects: 80% (437/546) 2024-06-26T07:19:27.5811052Z remote: Counting objects: 81% (443/546) 2024-06-26T07:19:27.5811554Z remote: Counting objects: 82% (448/546) 2024-06-26T07:19:27.5812054Z remote: Counting objects: 83% (454/546) 2024-06-26T07:19:27.5812559Z remote: Counting objects: 84% (459/546) 2024-06-26T07:19:27.5813060Z remote: Counting objects: 85% (465/546) 2024-06-26T07:19:27.5813568Z remote: Counting objects: 86% (470/546) 2024-06-26T07:19:27.5814063Z remote: Counting objects: 87% (476/546) 2024-06-26T07:19:27.5814572Z remote: Counting objects: 88% (481/546) 2024-06-26T07:19:27.5815074Z remote: Counting objects: 89% (486/546) 2024-06-26T07:19:27.5815574Z remote: Counting objects: 90% (492/546) 2024-06-26T07:19:27.5816168Z remote: Counting objects: 91% (497/546) 2024-06-26T07:19:27.5816668Z remote: Counting objects: 92% (503/546) 2024-06-26T07:19:27.5817173Z remote: Counting objects: 93% (508/546) 2024-06-26T07:19:27.5817673Z remote: Counting objects: 94% (514/546) 2024-06-26T07:19:27.5818166Z remote: Counting objects: 95% (519/546) 2024-06-26T07:19:27.5818663Z remote: Counting objects: 96% (525/546) 2024-06-26T07:19:27.5819163Z remote: Counting objects: 97% (530/546) 2024-06-26T07:19:27.5819655Z remote: Counting objects: 98% (536/546) 2024-06-26T07:19:27.5820158Z remote: Counting objects: 99% (541/546) 2024-06-26T07:19:27.5820662Z remote: Counting objects: 100% (546/546) 2024-06-26T07:19:27.5821196Z remote: Counting objects: 100% (546/546), done. 2024-06-26T07:19:27.5821750Z remote: Compressing objects: 0% (1/319) 2024-06-26T07:19:27.5822282Z remote: Compressing objects: 1% (4/319) 2024-06-26T07:19:27.5822900Z remote: Compressing objects: 2% (7/319) 2024-06-26T07:19:27.5823439Z remote: Compressing objects: 3% (10/319) 2024-06-26T07:19:27.5823970Z remote: Compressing objects: 4% (13/319) 2024-06-26T07:19:27.5824501Z remote: Compressing objects: 5% (16/319) 2024-06-26T07:19:27.5825036Z remote: Compressing objects: 6% (20/319) 2024-06-26T07:19:27.5825572Z remote: Compressing objects: 7% (23/319) 2024-06-26T07:19:27.5826097Z remote: Compressing objects: 8% (26/319) 2024-06-26T07:19:27.5826623Z remote: Compressing objects: 9% (29/319) 2024-06-26T07:19:27.5828763Z remote: Compressing objects: 10% (32/319) 2024-06-26T07:19:27.5831295Z remote: Compressing objects: 11% (36/319) 2024-06-26T07:19:27.5833532Z remote: Compressing objects: 12% (39/319) 2024-06-26T07:19:27.5836344Z remote: Compressing objects: 13% (42/319) 2024-06-26T07:19:27.5838883Z remote: Compressing objects: 14% (45/319) 2024-06-26T07:19:27.5839725Z remote: Compressing objects: 15% (48/319) 2024-06-26T07:19:27.5840438Z remote: Compressing objects: 16% (52/319) 2024-06-26T07:19:27.5841186Z remote: Compressing objects: 17% (55/319) 2024-06-26T07:19:27.5841978Z remote: Compressing objects: 18% (58/319) 2024-06-26T07:19:27.5842690Z remote: Compressing objects: 19% (61/319) 2024-06-26T07:19:27.5843316Z remote: Compressing objects: 20% (64/319) 2024-06-26T07:19:27.5844009Z remote: Compressing objects: 21% (67/319) 2024-06-26T07:19:27.5844715Z remote: Compressing objects: 22% (71/319) 2024-06-26T07:19:27.5845242Z remote: Compressing objects: 23% (74/319) 2024-06-26T07:19:27.5845764Z remote: Compressing objects: 24% (77/319) 2024-06-26T07:19:27.5846348Z remote: Compressing objects: 25% (80/319) 2024-06-26T07:19:27.5846940Z remote: Compressing objects: 26% (83/319) 2024-06-26T07:19:27.5847679Z remote: Compressing objects: 27% (87/319) 2024-06-26T07:19:27.5848204Z remote: Compressing objects: 28% (90/319) 2024-06-26T07:19:27.5848992Z remote: Compressing objects: 29% (93/319) 2024-06-26T07:19:27.5849566Z remote: Compressing objects: 30% (96/319) 2024-06-26T07:19:27.5850202Z remote: Compressing objects: 31% (99/319) 2024-06-26T07:19:27.5850750Z remote: Compressing objects: 32% (103/319) 2024-06-26T07:19:27.5851294Z remote: Compressing objects: 33% (106/319) 2024-06-26T07:19:27.5851915Z remote: Compressing objects: 34% (109/319) 2024-06-26T07:19:27.5852602Z remote: Compressing objects: 35% (112/319) 2024-06-26T07:19:27.5853139Z remote: Compressing objects: 36% (115/319) 2024-06-26T07:19:27.5853679Z remote: Compressing objects: 37% (119/319) 2024-06-26T07:19:27.5854396Z remote: Compressing objects: 38% (122/319) 2024-06-26T07:19:27.5855053Z remote: Compressing objects: 39% (125/319) 2024-06-26T07:19:27.5855634Z remote: Compressing objects: 40% (128/319) 2024-06-26T07:19:27.5856301Z remote: Compressing objects: 41% (131/319) 2024-06-26T07:19:27.5856880Z remote: Compressing objects: 42% (134/319) 2024-06-26T07:19:27.5857698Z remote: Compressing objects: 43% (138/319) 2024-06-26T07:19:27.5858423Z remote: Compressing objects: 44% (141/319) 2024-06-26T07:19:27.5858982Z remote: Compressing objects: 45% (144/319) 2024-06-26T07:19:27.5859514Z remote: Compressing objects: 46% (147/319) 2024-06-26T07:19:27.5860314Z remote: Compressing objects: 47% (150/319) 2024-06-26T07:19:27.5860999Z remote: Compressing objects: 48% (154/319) 2024-06-26T07:19:27.5861539Z remote: Compressing objects: 49% (157/319) 2024-06-26T07:19:27.5862120Z remote: Compressing objects: 50% (160/319) 2024-06-26T07:19:27.5862825Z remote: Compressing objects: 51% (163/319) 2024-06-26T07:19:27.5863378Z remote: Compressing objects: 52% (166/319) 2024-06-26T07:19:27.5863906Z remote: Compressing objects: 53% (170/319) 2024-06-26T07:19:27.5864446Z remote: Compressing objects: 54% (173/319) 2024-06-26T07:19:27.5864970Z remote: Compressing objects: 55% (176/319) 2024-06-26T07:19:27.5865493Z remote: Compressing objects: 56% (179/319) 2024-06-26T07:19:27.5866021Z remote: Compressing objects: 57% (182/319) 2024-06-26T07:19:27.5866545Z remote: Compressing objects: 58% (186/319) 2024-06-26T07:19:27.5867121Z remote: Compressing objects: 59% (189/319) 2024-06-26T07:19:27.5867902Z remote: Compressing objects: 60% (192/319) 2024-06-26T07:19:27.5868437Z remote: Compressing objects: 61% (195/319) 2024-06-26T07:19:27.5868962Z remote: Compressing objects: 62% (198/319) 2024-06-26T07:19:27.5869529Z remote: Compressing objects: 63% (201/319) 2024-06-26T07:19:27.5870240Z remote: Compressing objects: 64% (205/319) 2024-06-26T07:19:27.5870858Z remote: Compressing objects: 65% (208/319) 2024-06-26T07:19:27.5871496Z remote: Compressing objects: 66% (211/319) 2024-06-26T07:19:27.5872057Z remote: Compressing objects: 67% (214/319) 2024-06-26T07:19:27.5872779Z remote: Compressing objects: 68% (217/319) 2024-06-26T07:19:27.5873325Z remote: Compressing objects: 69% (221/319) 2024-06-26T07:19:27.5873850Z remote: Compressing objects: 70% (224/319) 2024-06-26T07:19:27.5874368Z remote: Compressing objects: 71% (227/319) 2024-06-26T07:19:27.5874891Z remote: Compressing objects: 72% (230/319) 2024-06-26T07:19:27.5875414Z remote: Compressing objects: 73% (233/319) 2024-06-26T07:19:27.5875936Z remote: Compressing objects: 74% (237/319) 2024-06-26T07:19:27.5876448Z remote: Compressing objects: 75% (240/319) 2024-06-26T07:19:27.5876974Z remote: Compressing objects: 76% (243/319) 2024-06-26T07:19:27.5877496Z remote: Compressing objects: 77% (246/319) 2024-06-26T07:19:27.5878016Z remote: Compressing objects: 78% (249/319) 2024-06-26T07:19:27.5878540Z remote: Compressing objects: 79% (253/319) 2024-06-26T07:19:27.5879246Z remote: Compressing objects: 80% (256/319) 2024-06-26T07:19:27.5880081Z remote: Compressing objects: 81% (259/319) 2024-06-26T07:19:27.5880660Z remote: Compressing objects: 82% (262/319) 2024-06-26T07:19:27.5881180Z remote: Compressing objects: 83% (265/319) 2024-06-26T07:19:27.5881707Z remote: Compressing objects: 84% (268/319) 2024-06-26T07:19:27.5882243Z remote: Compressing objects: 85% (272/319) 2024-06-26T07:19:27.5882790Z remote: Compressing objects: 86% (275/319) 2024-06-26T07:19:27.5883315Z remote: Compressing objects: 87% (278/319) 2024-06-26T07:19:27.5883853Z remote: Compressing objects: 88% (281/319) 2024-06-26T07:19:27.5884386Z remote: Compressing objects: 89% (284/319) 2024-06-26T07:19:27.5884974Z remote: Compressing objects: 90% (288/319) 2024-06-26T07:19:27.5885494Z remote: Compressing objects: 91% (291/319) 2024-06-26T07:19:27.5886021Z remote: Compressing objects: 92% (294/319) 2024-06-26T07:19:27.5886552Z remote: Compressing objects: 93% (297/319) 2024-06-26T07:19:27.5887089Z remote: Compressing objects: 94% (300/319) 2024-06-26T07:19:27.5887608Z remote: Compressing objects: 95% (304/319) 2024-06-26T07:19:27.5888135Z remote: Compressing objects: 96% (307/319) 2024-06-26T07:19:27.5888901Z remote: Compressing objects: 97% (310/319) 2024-06-26T07:19:27.5889426Z remote: Compressing objects: 98% (313/319) 2024-06-26T07:19:27.5889959Z remote: Compressing objects: 99% (316/319) 2024-06-26T07:19:27.5890489Z remote: Compressing objects: 100% (319/319) 2024-06-26T07:19:27.5891055Z remote: Compressing objects: 100% (319/319), done. 2024-06-26T07:19:27.5936169Z Receiving objects: 0% (1/30187) 2024-06-26T07:19:27.5970451Z Receiving objects: 1% (302/30187) 2024-06-26T07:19:27.6007994Z Receiving objects: 2% (604/30187) 2024-06-26T07:19:27.6042643Z Receiving objects: 3% (906/30187) 2024-06-26T07:19:27.6072845Z Receiving objects: 4% (1208/30187) 2024-06-26T07:19:27.6105101Z Receiving objects: 5% (1510/30187) 2024-06-26T07:19:27.6136777Z Receiving objects: 6% (1812/30187) 2024-06-26T07:19:27.6175361Z Receiving objects: 7% (2114/30187) 2024-06-26T07:19:27.6207232Z Receiving objects: 8% (2415/30187) 2024-06-26T07:19:27.6238588Z Receiving objects: 9% (2717/30187) 2024-06-26T07:19:27.6274717Z Receiving objects: 10% (3019/30187) 2024-06-26T07:19:27.6336716Z Receiving objects: 11% (3321/30187) 2024-06-26T07:19:27.6399481Z Receiving objects: 12% (3623/30187) 2024-06-26T07:19:27.6455875Z Receiving objects: 13% (3925/30187) 2024-06-26T07:19:27.6626082Z Receiving objects: 14% (4227/30187) 2024-06-26T07:19:27.6726891Z Receiving objects: 15% (4529/30187) 2024-06-26T07:19:27.6818329Z Receiving objects: 16% (4830/30187) 2024-06-26T07:19:27.6870278Z Receiving objects: 17% (5132/30187) 2024-06-26T07:19:27.6910998Z Receiving objects: 18% (5434/30187) 2024-06-26T07:19:27.6936308Z Receiving objects: 19% (5736/30187) 2024-06-26T07:19:27.6999728Z Receiving objects: 20% (6038/30187) 2024-06-26T07:19:27.7019170Z Receiving objects: 21% (6340/30187) 2024-06-26T07:19:27.7038715Z Receiving objects: 22% (6642/30187) 2024-06-26T07:19:27.8098216Z Receiving objects: 23% (6944/30187) 2024-06-26T07:19:28.5894303Z Receiving objects: 24% (7245/30187) 2024-06-26T07:19:28.7117493Z Receiving objects: 24% (7431/30187), 79.89 MiB | 79.97 MiB/s 2024-06-26T07:19:29.1998347Z Receiving objects: 25% (7547/30187), 79.89 MiB | 79.97 MiB/s 2024-06-26T07:19:29.2672149Z Receiving objects: 26% (7849/30187), 117.23 MiB | 78.20 MiB/s 2024-06-26T07:19:29.3425283Z Receiving objects: 27% (8151/30187), 117.23 MiB | 78.20 MiB/s 2024-06-26T07:19:29.4098615Z Receiving objects: 28% (8453/30187), 117.23 MiB | 78.20 MiB/s 2024-06-26T07:19:29.4768993Z Receiving objects: 29% (8755/30187), 117.23 MiB | 78.20 MiB/s 2024-06-26T07:19:29.5536789Z Receiving objects: 30% (9057/30187), 117.23 MiB | 78.20 MiB/s 2024-06-26T07:19:29.5893924Z Receiving objects: 31% (9358/30187), 117.23 MiB | 78.20 MiB/s 2024-06-26T07:19:29.6244794Z Receiving objects: 31% (9509/30187), 156.77 MiB | 78.42 MiB/s 2024-06-26T07:19:29.6953786Z Receiving objects: 32% (9660/30187), 156.77 MiB | 78.42 MiB/s 2024-06-26T07:19:29.9304551Z Receiving objects: 33% (9962/30187), 156.77 MiB | 78.42 MiB/s 2024-06-26T07:19:30.0064567Z Receiving objects: 34% (10264/30187), 156.77 MiB | 78.42 MiB/s 2024-06-26T07:19:30.0206598Z Receiving objects: 35% (10566/30187), 156.77 MiB | 78.42 MiB/s 2024-06-26T07:19:30.2688715Z Receiving objects: 36% (10868/30187), 156.77 MiB | 78.42 MiB/s 2024-06-26T07:19:30.5342747Z Receiving objects: 37% (11170/30187), 186.62 MiB | 74.67 MiB/s 2024-06-26T07:19:30.5544676Z Receiving objects: 38% (11472/30187), 186.62 MiB | 74.67 MiB/s 2024-06-26T07:19:30.5558362Z Receiving objects: 39% (11773/30187), 186.62 MiB | 74.67 MiB/s 2024-06-26T07:19:30.5583019Z Receiving objects: 40% (12075/30187), 186.62 MiB | 74.67 MiB/s 2024-06-26T07:19:30.5595946Z Receiving objects: 41% (12377/30187), 186.62 MiB | 74.67 MiB/s 2024-06-26T07:19:30.5609555Z Receiving objects: 42% (12679/30187), 186.62 MiB | 74.67 MiB/s 2024-06-26T07:19:30.5619958Z Receiving objects: 43% (12981/30187), 186.62 MiB | 74.67 MiB/s 2024-06-26T07:19:30.5628848Z Receiving objects: 44% (13283/30187), 186.62 MiB | 74.67 MiB/s 2024-06-26T07:19:30.5648099Z Receiving objects: 45% (13585/30187), 186.62 MiB | 74.67 MiB/s 2024-06-26T07:19:30.5659364Z Receiving objects: 46% (13887/30187), 186.62 MiB | 74.67 MiB/s 2024-06-26T07:19:30.5663318Z Receiving objects: 47% (14188/30187), 186.62 MiB | 74.67 MiB/s 2024-06-26T07:19:30.5669184Z Receiving objects: 48% (14490/30187), 186.62 MiB | 74.67 MiB/s 2024-06-26T07:19:30.5690177Z Receiving objects: 49% (14792/30187), 186.62 MiB | 74.67 MiB/s 2024-06-26T07:19:30.5706017Z Receiving objects: 50% (15094/30187), 186.62 MiB | 74.67 MiB/s 2024-06-26T07:19:30.5716294Z Receiving objects: 51% (15396/30187), 186.62 MiB | 74.67 MiB/s 2024-06-26T07:19:30.5894580Z Receiving objects: 52% (15698/30187), 186.62 MiB | 74.67 MiB/s 2024-06-26T07:19:30.5915151Z Receiving objects: 52% (15764/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.5919619Z Receiving objects: 53% (16000/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.5926643Z Receiving objects: 54% (16301/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.5932594Z Receiving objects: 55% (16603/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.5937754Z Receiving objects: 56% (16905/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.5953938Z Receiving objects: 57% (17207/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.5963981Z Receiving objects: 58% (17509/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.5972518Z Receiving objects: 59% (17811/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.5985709Z Receiving objects: 60% (18113/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.5993823Z Receiving objects: 61% (18415/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.6003289Z Receiving objects: 62% (18716/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.6130190Z Receiving objects: 63% (19018/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.6135837Z Receiving objects: 64% (19320/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.6142074Z Receiving objects: 65% (19622/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.6148003Z Receiving objects: 66% (19924/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.6156547Z Receiving objects: 67% (20226/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.6164135Z Receiving objects: 68% (20528/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.6405706Z Receiving objects: 69% (20830/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.6792788Z Receiving objects: 70% (21131/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.6798436Z Receiving objects: 71% (21433/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.6809557Z Receiving objects: 72% (21735/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.6822592Z Receiving objects: 73% (22037/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.6834927Z Receiving objects: 74% (22339/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.6844712Z Receiving objects: 75% (22641/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.6853285Z Receiving objects: 76% (22943/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.6871318Z Receiving objects: 77% (23244/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.6877210Z Receiving objects: 78% (23546/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.6885008Z Receiving objects: 79% (23848/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.6907144Z Receiving objects: 80% (24150/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.8839006Z Receiving objects: 81% (24452/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.9047701Z Receiving objects: 82% (24754/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.9079829Z Receiving objects: 83% (25056/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.9089704Z Receiving objects: 84% (25358/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.9100465Z Receiving objects: 85% (25659/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.9129411Z Receiving objects: 86% (25961/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.9157874Z Receiving objects: 87% (26263/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.9284810Z Receiving objects: 88% (26565/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.9290615Z Receiving objects: 89% (26867/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.9298426Z Receiving objects: 90% (27169/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.9303385Z Receiving objects: 91% (27471/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.9309589Z Receiving objects: 92% (27773/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.9315522Z Receiving objects: 93% (28074/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.9425216Z Receiving objects: 94% (28376/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.9439881Z Receiving objects: 95% (28678/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.9449761Z Receiving objects: 96% (28980/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.9467860Z Receiving objects: 97% (29282/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.9482233Z Receiving objects: 98% (29584/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.9521210Z Receiving objects: 99% (29886/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.9522277Z remote: Total 30187 (delta 255), reused 486 (delta 215), pack-reused 29641 2024-06-26T07:19:30.9539435Z Receiving objects: 100% (30187/30187), 223.23 MiB | 74.43 MiB/s 2024-06-26T07:19:30.9540205Z Receiving objects: 100% (30187/30187), 245.25 MiB | 72.90 MiB/s, done. 2024-06-26T07:19:30.9566879Z Resolving deltas: 0% (0/16212) 2024-06-26T07:19:30.9580261Z Resolving deltas: 1% (163/16212) 2024-06-26T07:19:30.9590382Z Resolving deltas: 2% (403/16212) 2024-06-26T07:19:30.9608788Z Resolving deltas: 3% (503/16212) 2024-06-26T07:19:30.9629567Z Resolving deltas: 4% (649/16212) 2024-06-26T07:19:30.9637272Z Resolving deltas: 5% (903/16212) 2024-06-26T07:19:30.9646395Z Resolving deltas: 6% (973/16212) 2024-06-26T07:19:30.9656535Z Resolving deltas: 7% (1137/16212) 2024-06-26T07:19:30.9665970Z Resolving deltas: 8% (1297/16212) 2024-06-26T07:19:30.9674631Z Resolving deltas: 9% (1468/16212) 2024-06-26T07:19:30.9686200Z Resolving deltas: 10% (1632/16212) 2024-06-26T07:19:30.9697155Z Resolving deltas: 11% (1790/16212) 2024-06-26T07:19:30.9706727Z Resolving deltas: 12% (1948/16212) 2024-06-26T07:19:30.9720041Z Resolving deltas: 13% (2109/16212) 2024-06-26T07:19:30.9728811Z Resolving deltas: 14% (2281/16212) 2024-06-26T07:19:30.9747040Z Resolving deltas: 15% (2438/16212) 2024-06-26T07:19:30.9764300Z Resolving deltas: 16% (2610/16212) 2024-06-26T07:19:30.9780342Z Resolving deltas: 17% (2758/16212) 2024-06-26T07:19:30.9820855Z Resolving deltas: 18% (2919/16212) 2024-06-26T07:19:30.9837647Z Resolving deltas: 19% (3081/16212) 2024-06-26T07:19:30.9855558Z Resolving deltas: 20% (3245/16212) 2024-06-26T07:19:30.9857841Z Resolving deltas: 21% (3527/16212) 2024-06-26T07:19:30.9868720Z Resolving deltas: 22% (3568/16212) 2024-06-26T07:19:30.9880709Z Resolving deltas: 23% (3773/16212) 2024-06-26T07:19:30.9883944Z Resolving deltas: 24% (3999/16212) 2024-06-26T07:19:30.9897751Z Resolving deltas: 25% (4059/16212) 2024-06-26T07:19:30.9931512Z Resolving deltas: 26% (4218/16212) 2024-06-26T07:19:30.9952433Z Resolving deltas: 27% (4378/16212) 2024-06-26T07:19:30.9964935Z Resolving deltas: 28% (4540/16212) 2024-06-26T07:19:30.9979149Z Resolving deltas: 29% (4708/16212) 2024-06-26T07:19:30.9993426Z Resolving deltas: 30% (4881/16212) 2024-06-26T07:19:31.0009661Z Resolving deltas: 31% (5026/16212) 2024-06-26T07:19:31.0017950Z Resolving deltas: 32% (5189/16212) 2024-06-26T07:19:31.0028870Z Resolving deltas: 33% (5354/16212) 2024-06-26T07:19:31.0039398Z Resolving deltas: 34% (5514/16212) 2024-06-26T07:19:31.0060881Z Resolving deltas: 35% (5681/16212) 2024-06-26T07:19:31.0082524Z Resolving deltas: 36% (5837/16212) 2024-06-26T07:19:31.0094449Z Resolving deltas: 37% (6044/16212) 2024-06-26T07:19:31.0109388Z Resolving deltas: 38% (6194/16212) 2024-06-26T07:19:31.0123617Z Resolving deltas: 39% (6331/16212) 2024-06-26T07:19:31.0133425Z Resolving deltas: 40% (6489/16212) 2024-06-26T07:19:31.0150234Z Resolving deltas: 41% (6656/16212) 2024-06-26T07:19:31.0161249Z Resolving deltas: 42% (6843/16212) 2024-06-26T07:19:31.0167012Z Resolving deltas: 43% (6981/16212) 2024-06-26T07:19:31.0179572Z Resolving deltas: 44% (7139/16212) 2024-06-26T07:19:31.0192293Z Resolving deltas: 45% (7317/16212) 2024-06-26T07:19:31.0216866Z Resolving deltas: 46% (7458/16212) 2024-06-26T07:19:31.0227185Z Resolving deltas: 47% (7622/16212) 2024-06-26T07:19:31.0251504Z Resolving deltas: 48% (7785/16212) 2024-06-26T07:19:31.0264350Z Resolving deltas: 49% (7945/16212) 2024-06-26T07:19:31.0273597Z Resolving deltas: 50% (8118/16212) 2024-06-26T07:19:31.0279672Z Resolving deltas: 51% (8280/16212) 2024-06-26T07:19:31.0289953Z Resolving deltas: 52% (8432/16212) 2024-06-26T07:19:31.0306265Z Resolving deltas: 53% (8599/16212) 2024-06-26T07:19:31.0309485Z Resolving deltas: 54% (8848/16212) 2024-06-26T07:19:31.0316252Z Resolving deltas: 55% (8926/16212) 2024-06-26T07:19:31.0328747Z Resolving deltas: 56% (9079/16212) 2024-06-26T07:19:31.0338800Z Resolving deltas: 57% (9247/16212) 2024-06-26T07:19:31.0345580Z Resolving deltas: 58% (9404/16212) 2024-06-26T07:19:31.0355280Z Resolving deltas: 59% (9592/16212) 2024-06-26T07:19:31.0361770Z Resolving deltas: 60% (9864/16212) 2024-06-26T07:19:31.0365715Z Resolving deltas: 61% (9979/16212) 2024-06-26T07:19:31.0379403Z Resolving deltas: 62% (10064/16212) 2024-06-26T07:19:31.0389756Z Resolving deltas: 63% (10281/16212) 2024-06-26T07:19:31.0392059Z Resolving deltas: 64% (10486/16212) 2024-06-26T07:19:31.0399834Z Resolving deltas: 65% (10538/16212) 2024-06-26T07:19:31.0427093Z Resolving deltas: 66% (10700/16212) 2024-06-26T07:19:31.0433645Z Resolving deltas: 68% (11156/16212) 2024-06-26T07:19:31.0449758Z Resolving deltas: 69% (11262/16212) 2024-06-26T07:19:31.0458617Z Resolving deltas: 70% (11353/16212) 2024-06-26T07:19:31.0470489Z Resolving deltas: 71% (11511/16212) 2024-06-26T07:19:31.0479537Z Resolving deltas: 72% (11703/16212) 2024-06-26T07:19:31.0492064Z Resolving deltas: 73% (11863/16212) 2024-06-26T07:19:31.0505900Z Resolving deltas: 74% (11999/16212) 2024-06-26T07:19:31.0538577Z Resolving deltas: 75% (12182/16212) 2024-06-26T07:19:31.0542302Z Resolving deltas: 78% (12722/16212) 2024-06-26T07:19:31.0552548Z Resolving deltas: 79% (12809/16212) 2024-06-26T07:19:31.0560205Z Resolving deltas: 80% (12986/16212) 2024-06-26T07:19:31.0571663Z Resolving deltas: 81% (13143/16212) 2024-06-26T07:19:31.0579687Z Resolving deltas: 82% (13310/16212) 2024-06-26T07:19:31.0587551Z Resolving deltas: 83% (13457/16212) 2024-06-26T07:19:31.0598175Z Resolving deltas: 84% (13627/16212) 2024-06-26T07:19:31.0619159Z Resolving deltas: 85% (13781/16212) 2024-06-26T07:19:31.0629603Z Resolving deltas: 86% (13944/16212) 2024-06-26T07:19:31.0636940Z Resolving deltas: 87% (14111/16212) 2024-06-26T07:19:31.0644041Z Resolving deltas: 88% (14280/16212) 2024-06-26T07:19:31.0652172Z Resolving deltas: 89% (14443/16212) 2024-06-26T07:19:31.0662872Z Resolving deltas: 90% (14604/16212) 2024-06-26T07:19:31.0670201Z Resolving deltas: 91% (14776/16212) 2024-06-26T07:19:31.0681930Z Resolving deltas: 92% (14923/16212) 2024-06-26T07:19:31.0686251Z Resolving deltas: 93% (15134/16212) 2024-06-26T07:19:31.0695590Z Resolving deltas: 94% (15253/16212) 2024-06-26T07:19:31.0703587Z Resolving deltas: 95% (15423/16212) 2024-06-26T07:19:31.0710312Z Resolving deltas: 96% (15569/16212) 2024-06-26T07:19:31.0716248Z Resolving deltas: 97% (15726/16212) 2024-06-26T07:19:31.0723151Z Resolving deltas: 98% (15893/16212) 2024-06-26T07:19:31.0757782Z Resolving deltas: 99% (16050/16212) 2024-06-26T07:19:31.0758299Z Resolving deltas: 100% (16212/16212) 2024-06-26T07:19:31.0758886Z Resolving deltas: 100% (16212/16212), done. 2024-06-26T07:19:32.1476718Z + pushd torchbench 2024-06-26T07:19:32.1477185Z ~/workspace/torchbench ~/workspace 2024-06-26T07:19:32.1477725Z + git checkout 23512dbebd44a11eb84afbf53c3c071dd105297e 2024-06-26T07:19:32.2045890Z Note: switching to '23512dbebd44a11eb84afbf53c3c071dd105297e'. 2024-06-26T07:19:32.2046352Z 2024-06-26T07:19:32.2046841Z You are in 'detached HEAD' state. You can look around, make experimental 2024-06-26T07:19:32.2047849Z changes and commit them, and you can discard any commits you make in this 2024-06-26T07:19:32.2048941Z state without impacting any branches by switching back to a branch. 2024-06-26T07:19:32.2049388Z 2024-06-26T07:19:32.2049711Z If you want to create a new branch to retain commits you create, you may 2024-06-26T07:19:32.2050676Z do so (now or later) by using -c with the switch command. Example: 2024-06-26T07:19:32.2051110Z 2024-06-26T07:19:32.2051295Z git switch -c 2024-06-26T07:19:32.2051568Z 2024-06-26T07:19:32.2051738Z Or undo this operation with: 2024-06-26T07:19:32.2051988Z 2024-06-26T07:19:32.2052125Z git switch - 2024-06-26T07:19:32.2052310Z 2024-06-26T07:19:32.2052648Z Turn off this advice by setting config variable advice.detachedHead to false 2024-06-26T07:19:32.2053143Z 2024-06-26T07:19:32.2053315Z HEAD is now at 23512dbe Pin numpy version 2024-06-26T07:19:32.2053763Z + '[' '' ']' 2024-06-26T07:19:32.2054130Z + python install.py --continue_on_fail 2024-06-26T07:19:34.1620535Z checking packages torch, torchvision, torchaudio are installed...OK 2024-06-26T07:19:49.6290592Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2024-06-26T07:19:51.9943112Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2024-06-26T07:19:54.5809714Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2024-06-26T07:19:56.9382249Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2024-06-26T07:19:56.9535565Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2024-06-26T07:20:05.6226347Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2024-06-26T07:20:08.9641538Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2024-06-26T07:20:12.3124668Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2024-06-26T07:20:15.6636918Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2024-06-26T07:20:15.6638451Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2024-06-26T07:20:17.4936772Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2024-06-26T07:20:21.0950913Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2024-06-26T07:20:21.1101573Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2024-06-26T07:21:15.7249839Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2024-06-26T07:21:24.6859491Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2024-06-26T07:21:32.3494596Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2024-06-26T07:21:39.8016977Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2024-06-26T07:21:48.7068560Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2024-06-26T07:21:56.3294644Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2024-06-26T07:22:03.5601306Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2024-06-26T07:22:10.9009236Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2024-06-26T07:22:18.5337356Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2024-06-26T07:22:26.2988047Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2024-06-26T07:22:33.6846338Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2024-06-26T07:22:41.2630639Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2024-06-26T07:22:45.2633330Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2024-06-26T07:23:25.7671849Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2024-06-26T07:23:31.7439605Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2024-06-26T07:23:38.5694216Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2024-06-26T07:23:49.5496418Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2024-06-26T07:23:51.7465334Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2024-06-26T07:23:54.2051366Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2024-06-26T07:24:02.1048397Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2024-06-26T07:24:10.1680531Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2024-06-26T07:24:16.8381847Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2024-06-26T07:24:25.4608108Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2024-06-26T07:24:32.4886412Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2024-06-26T07:24:38.7730997Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2024-06-26T07:24:46.1089858Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2024-06-26T07:25:02.2919052Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2024-06-26T07:25:09.5749241Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2024-06-26T07:25:14.8818780Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2024-06-26T07:25:20.9614073Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2024-06-26T07:25:28.7562560Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2024-06-26T07:25:28.7569469Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2024-06-26T07:25:42.1192610Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2024-06-26T07:25:47.3882852Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2024-06-26T07:25:47.3884554Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2024-06-26T07:25:55.9311485Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2024-06-26T07:25:58.1873182Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2024-06-26T07:26:00.4764973Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2024-06-26T07:26:40.1905883Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2024-06-26T07:28:01.2242407Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2024-06-26T07:28:01.2398335Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2024-06-26T07:28:03.8100988Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2024-06-26T07:28:03.8254766Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2024-06-26T07:28:03.8401267Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2024-06-26T07:28:03.8556531Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2024-06-26T07:28:03.8707703Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2024-06-26T07:28:03.8861859Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2024-06-26T07:28:24.2699936Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2024-06-26T07:28:24.2701838Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2024-06-26T07:28:26.5729332Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2024-06-26T07:28:29.9875804Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2024-06-26T07:28:30.0032153Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2024-06-26T07:28:30.0187650Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2024-06-26T07:28:30.0340478Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2024-06-26T07:28:30.0493803Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2024-06-26T07:28:30.0649561Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2024-06-26T07:28:37.2673449Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2024-06-26T07:28:39.7842187Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2024-06-26T07:28:44.2975422Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2024-06-26T07:28:44.3126353Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2024-06-26T07:28:44.3284479Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2024-06-26T07:28:44.3438955Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2024-06-26T07:28:44.3587893Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2024-06-26T07:28:44.3740379Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2024-06-26T07:28:56.1904071Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2024-06-26T07:29:08.3183567Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2024-06-26T07:29:08.3339968Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2024-06-26T07:29:08.3343108Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2024-06-26T07:29:08.3344807Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2024-06-26T07:29:11.7524786Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2024-06-26T07:29:14.4611297Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2024-06-26T07:29:14.4766019Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2024-06-26T07:29:36.7154771Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2024-06-26T07:29:43.4608214Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2024-06-26T07:29:49.1150652Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2024-06-26T07:29:57.4923662Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2024-06-26T07:29:57.5084006Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2024-06-26T07:29:57.5241478Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2024-06-26T07:29:57.5397522Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2024-06-26T07:29:57.5547190Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2024-06-26T07:29:57.5699183Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2024-06-26T07:29:57.5854045Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2024-06-26T07:29:57.6007706Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2024-06-26T07:30:08.1261819Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2024-06-26T07:30:21.0161540Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2024-06-26T07:30:21.0314500Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2024-06-26T07:30:24.0322505Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2024-06-26T07:30:27.2581391Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2024-06-26T07:30:27.6761962Z + popd 2024-06-26T07:30:27.6762316Z ~/workspace 2024-06-26T07:30:27.6762707Z + [[ dynamo_eager_torchbench != *cpu_inductor* ]] 2024-06-26T07:30:27.6763235Z + install_torchrec_and_fbgemm 2024-06-26T07:30:27.6763735Z + local torchrec_commit 2024-06-26T07:30:27.6764630Z ++ get_pinned_commit torchrec 2024-06-26T07:30:27.6765198Z ++ cat .github/ci_commit_pins/torchrec.txt 2024-06-26T07:30:27.6800935Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T07:30:27.6801463Z + local fbgemm_commit 2024-06-26T07:30:27.6803562Z ++ get_pinned_commit fbgemm 2024-06-26T07:30:27.6804038Z ++ cat .github/ci_commit_pins/fbgemm.txt 2024-06-26T07:30:27.6814126Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:30:27.6814837Z + pip_uninstall torchrec-nightly 2024-06-26T07:30:27.6815292Z + pip uninstall -y torchrec-nightly 2024-06-26T07:30:28.0381239Z WARNING: Skipping torchrec-nightly as it is not installed. 2024-06-26T07:30:28.0844173Z + pip_uninstall fbgemm-gpu-nightly 2024-06-26T07:30:28.0844738Z + pip uninstall -y fbgemm-gpu-nightly 2024-06-26T07:30:28.4294246Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed. 2024-06-26T07:30:28.4709874Z + pip_install setuptools-git-versioning scikit-build pyre-extensions 2024-06-26T07:30:28.4710861Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions 2024-06-26T07:30:28.9142587Z Collecting setuptools-git-versioning 2024-06-26T07:30:28.9296382Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB) 2024-06-26T07:30:29.0269960Z Collecting scikit-build 2024-06-26T07:30:29.0298421Z Downloading scikit_build-0.18.0-py3-none-any.whl.metadata (17 kB) 2024-06-26T07:30:29.1319246Z Collecting pyre-extensions 2024-06-26T07:30:29.1345767Z Downloading pyre_extensions-0.0.30-py3-none-any.whl.metadata (4.0 kB) 2024-06-26T07:30:29.1775528Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.1) 2024-06-26T07:30:29.1778747Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (69.5.1) 2024-06-26T07:30:29.2411711Z Collecting toml>=0.10.2 (from setuptools-git-versioning) 2024-06-26T07:30:29.2440088Z Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) 2024-06-26T07:30:29.2990322Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0) 2024-06-26T07:30:29.3001223Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1) 2024-06-26T07:30:29.3006269Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.43.0) 2024-06-26T07:30:29.3613763Z Collecting typing-inspect (from pyre-extensions) 2024-06-26T07:30:29.3643179Z Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB) 2024-06-26T07:30:29.4086737Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.2) 2024-06-26T07:30:29.4311074Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0) 2024-06-26T07:30:29.4356079Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB) 2024-06-26T07:30:29.4954343Z Downloading scikit_build-0.18.0-py3-none-any.whl (85 kB) 2024-06-26T07:30:29.5509821Z Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB) 2024-06-26T07:30:29.6087011Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2024-06-26T07:30:29.6656389Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2024-06-26T07:30:31.8551062Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions 2024-06-26T07:30:32.1860224Z Successfully installed pyre-extensions-0.0.30 scikit-build-0.18.0 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0 2024-06-26T07:30:32.3010154Z + CUDA_PATH=/usr/local/cuda-12.1 2024-06-26T07:30:32.3011406Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-06-26T07:30:32.3013255Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-06-26T07:30:32.6365595Z Collecting fbgemm-gpu 2024-06-26T07:30:32.6366906Z Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-4ua5apq0/fbgemm-gpu_82c159c64b2b4f3f9273ff9150e15abf 2024-06-26T07:30:32.6383369Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-4ua5apq0/fbgemm-gpu_82c159c64b2b4f3f9273ff9150e15abf 2024-06-26T07:30:33.8012096Z Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd' 2024-06-26T07:30:33.8025912Z Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:30:34.2063683Z Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:30:34.6856896Z Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:30:34.6857874Z Running command git submodule update --init --recursive -q 2024-06-26T07:30:42.7458457Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:30:42.7479473Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.21.2) 2024-06-26T07:30:42.7485662Z Building wheels for collected packages: fbgemm-gpu 2024-06-26T08:12:52.4024296Z Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-06-26T08:12:52.5819588Z [?25h Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=252634753 sha256=f91fe3c9d492c53000d70a90dcfa63885990bcd7f7aab404f7446ba31bd9a6e7 2024-06-26T08:12:52.5821250Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39 2024-06-26T08:12:52.5842086Z Successfully built fbgemm-gpu 2024-06-26T08:12:54.7378573Z Installing collected packages: fbgemm-gpu 2024-06-26T08:12:57.8712516Z Successfully installed fbgemm-gpu-0.4.1rc0.post421 2024-06-26T08:12:58.4900939Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:58.4902445Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:58.8268623Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:58.8273142Z Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-a4d0huoe 2024-06-26T08:12:58.8290895Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-a4d0huoe 2024-06-26T08:12:59.4293240Z Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2' 2024-06-26T08:12:59.4307915Z Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:59.7052022Z Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:13:00.0125066Z Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:13:00.2420635Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T08:13:00.2469416Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9) 2024-06-26T08:13:00.2472373Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3) 2024-06-26T08:13:00.2476881Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0) 2024-06-26T08:13:00.2975296Z Collecting torchmetrics (from torchrec==0.3.2) 2024-06-26T08:13:00.3137933Z Downloading torchmetrics-1.4.0.post0-py3-none-any.whl.metadata (19 kB) 2024-06-26T08:13:00.3190658Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.4) 2024-06-26T08:13:00.3193820Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421) 2024-06-26T08:13:00.3206664Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.21.2) 2024-06-26T08:13:00.3230029Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.10.0) 2024-06-26T08:13:00.3892374Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0) 2024-06-26T08:13:00.3897743Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-06-26T08:13:00.3902637Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-06-26T08:13:00.4670105Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.1) 2024-06-26T08:13:00.4674877Z Requirement already satisfied: torch>=1.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.5.0a0+gitb8c4c54) 2024-06-26T08:13:00.4855794Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2) 2024-06-26T08:13:00.4884742Z Downloading lightning_utilities-0.11.2-py3-none-any.whl.metadata (4.7 kB) 2024-06-26T08:13:00.5049578Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (69.5.1) 2024-06-26T08:13:00.5051840Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.2) 2024-06-26T08:13:00.5083885Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0) 2024-06-26T08:13:00.5152393Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.13.1) 2024-06-26T08:13:00.5156125Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.12.1) 2024-06-26T08:13:00.5159037Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.8.8) 2024-06-26T08:13:00.5161669Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.1.4) 2024-06-26T08:13:00.5164807Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2024.2.0) 2024-06-26T08:13:00.5820733Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.1.5) 2024-06-26T08:13:00.6126471Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.3.0) 2024-06-26T08:13:00.6266670Z Downloading torchmetrics-1.4.0.post0-py3-none-any.whl (868 kB) 2024-06-26T08:13:00.6457194Z Downloading lightning_utilities-0.11.2-py3-none-any.whl (26 kB) 2024-06-26T08:13:00.6550066Z Building wheels for collected packages: torchrec 2024-06-26T08:13:01.0990099Z Building wheel for torchrec (setup.py) ... [?25l- \ | done 2024-06-26T08:13:01.0999796Z [?25h Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374488 sha256=eafc3066979d5f1eb2e7e4af74e691e94c5f5690d655674407f686cf947d9cb6 2024-06-26T08:13:01.1001499Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8 2024-06-26T08:13:01.1023254Z Successfully built torchrec 2024-06-26T08:13:03.1861957Z Installing collected packages: lightning-utilities, torchmetrics, torchrec 2024-06-26T08:13:03.7365303Z Successfully installed lightning-utilities-0.11.2 torchmetrics-1.4.0.post0 torchrec-0.3.2 2024-06-26T08:13:03.8691993Z ++ pwd 2024-06-26T08:13:03.8694649Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2024-06-26T08:13:03.8695321Z + test_dynamo_benchmark torchbench 1 2024-06-26T08:13:03.8697926Z ++ pwd 2024-06-26T08:13:03.8700327Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:13:03.8700897Z + local suite=torchbench 2024-06-26T08:13:03.8701243Z + shift 2024-06-26T08:13:03.8701526Z + local shard_id=1 2024-06-26T08:13:03.8701854Z + shift 2024-06-26T08:13:03.8705071Z + [[ dynamo_eager_torchbench == *perf_compare* ]] 2024-06-26T08:13:03.8705788Z + [[ dynamo_eager_torchbench == *perf* ]] 2024-06-26T08:13:03.8706321Z + [[ dynamo_eager_torchbench == *cpu_inductor* ]] 2024-06-26T08:13:03.8706827Z + [[ dynamo_eager_torchbench == *aot_inductor* ]] 2024-06-26T08:13:03.8707562Z + test_single_dynamo_benchmark inference torchbench 1 --inference --bfloat16 2024-06-26T08:13:03.8708142Z ++ pwd 2024-06-26T08:13:03.8708597Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:13:03.8709268Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:13:03.8731843Z + local name=inference 2024-06-26T08:13:03.8732324Z + shift 2024-06-26T08:13:03.8732717Z + local suite=torchbench 2024-06-26T08:13:03.8733123Z + shift 2024-06-26T08:13:03.8733399Z + local shard_id=1 2024-06-26T08:13:03.8733896Z + shift 2024-06-26T08:13:03.8734175Z + partition_flags=() 2024-06-26T08:13:03.8734524Z + local partition_flags 2024-06-26T08:13:03.8734914Z + [[ -n 2 ]] 2024-06-26T08:13:03.8735230Z + [[ -n 1 ]] 2024-06-26T08:13:03.8735915Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-06-26T08:13:03.8736614Z + [[ dynamo_eager_torchbench == *perf_compare* ]] 2024-06-26T08:13:03.8737111Z + [[ dynamo_eager_torchbench == *perf* ]] 2024-06-26T08:13:03.8737597Z + [[ dynamo_eager_torchbench == *aot_inductor* ]] 2024-06-26T08:13:03.8739216Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --backend eager --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2024-06-26T08:13:09.4995141Z 2024-06-26T08:13:09.9803987Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:09.9804432Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:09.9839105Z cuda eval llama 2024-06-26T08:13:12.5180323Z pass 2024-06-26T08:13:12.5324586Z TIMING: backend_compile:0.00024 entire_frame_compile:2.02314 2024-06-26T08:13:12.5326286Z STATS: call_* op count: 1058 | FakeTensor.__torch_dispatch__:364 | FakeTensorMode.__torch_dispatch__:3224 2024-06-26T08:13:12.5328088Z Dynamo produced 1 graphs covering 1058 ops with 0 graph breaks (0 unique) 2024-06-26T08:13:15.8901294Z 2024-06-26T08:13:16.1590880Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:16.1591480Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:16.1592011Z cuda eval llama_v2_7b_16h 2024-06-26T08:13:16.1596869Z Traceback (most recent call last): 2024-06-26T08:13:16.1597805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:13:16.1598653Z ) = runner.load_model( 2024-06-26T08:13:16.1599384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 327, in load_model 2024-06-26T08:13:16.1600390Z benchmark = benchmark_cls( 2024-06-26T08:13:16.1601155Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-26T08:13:16.1602058Z obj = type.__call__(cls, *args, **kwargs) 2024-06-26T08:13:16.1602957Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h/__init__.py", line 12, in __init__ 2024-06-26T08:13:16.1604157Z super().__init__(name="llama_v2_7b_16h", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:13:16.1605450Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-06-26T08:13:16.1606643Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:13:16.1607625Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-06-26T08:13:16.1608367Z self._skip_by_device_name() 2024-06-26T08:13:16.1609442Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-06-26T08:13:16.1610670Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-06-26T08:13:16.1611854Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llama_v2_7b_16h/metadata.yaml`. 2024-06-26T08:13:16.1612457Z 2024-06-26T08:13:16.1612579Z model_fail_to_load 2024-06-26T08:13:19.3134683Z 2024-06-26T08:13:19.5836295Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:19.5836843Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:19.5837282Z cuda eval llava 2024-06-26T08:13:19.5841854Z Traceback (most recent call last): 2024-06-26T08:13:19.5842602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:13:19.5843497Z ) = runner.load_model( 2024-06-26T08:13:19.5844646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 327, in load_model 2024-06-26T08:13:19.5845485Z benchmark = benchmark_cls( 2024-06-26T08:13:19.5846284Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-26T08:13:19.5847141Z obj = type.__call__(cls, *args, **kwargs) 2024-06-26T08:13:19.5848064Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 11, in __init__ 2024-06-26T08:13:19.5849438Z super().__init__(name="llava", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:13:19.5850828Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-06-26T08:13:19.5852113Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:13:19.5853210Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-06-26T08:13:19.5854046Z self._skip_by_device_name() 2024-06-26T08:13:19.5854924Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-06-26T08:13:19.5856494Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-06-26T08:13:19.5857674Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`. 2024-06-26T08:13:19.5858300Z 2024-06-26T08:13:19.5858441Z model_fail_to_load 2024-06-26T08:13:22.7353127Z 2024-06-26T08:13:23.3017859Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:23.3018335Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:23.3024697Z cuda eval maml 2024-06-26T08:13:23.3027654Z pass_due_to_skip 2024-06-26T08:13:23.3035621Z TIMING: 2024-06-26T08:13:23.3044588Z STATS: call_* op count: 0 2024-06-26T08:13:23.3045172Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T08:13:26.5224819Z 2024-06-26T08:13:26.9918466Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:26.9918946Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:26.9925141Z cuda eval maml_omniglot 2024-06-26T08:13:27.3334382Z pass 2024-06-26T08:13:27.3335184Z TIMING: backend_compile:5e-05 entire_frame_compile:0.16848 2024-06-26T08:13:27.3336171Z STATS: call_* op count: 28 | FakeTensor.__torch_dispatch__:89 | FakeTensorMode.__torch_dispatch__:354 2024-06-26T08:13:27.3337056Z Dynamo produced 1 graphs covering 28 ops with 0 graph breaks (0 unique) 2024-06-26T08:13:30.5687144Z 2024-06-26T08:13:30.7652878Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/mnasnet1.0_top1_73.512-f206786ef8.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/mnasnet1.0_top1_73.512-f206786ef8.pth 2024-06-26T08:13:30.7823621Z 2024-06-26T08:13:30.7823907Z 2024-06-26T08:13:30.8275146Z 0% 0.00/16.9M [00:00 will be ignored 2024-06-26T08:14:03.9896934Z [rank0]:W0626 08:14:03.988000 140439397220992 torch/_dynamo/backends/distributed.py:88] [2/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance. 2024-06-26T08:14:04.0085201Z [rank0]:W0626 08:14:04.008000 140439397220992 torch/_dynamo/backends/distributed.py:105] [2/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes: 2024-06-26T08:14:04.0088819Z [rank0]:W0626 08:14:04.008000 140439397220992 torch/_dynamo/backends/distributed.py:105] [2/0_1] ┌─────────┬─────────────┬────────────────────────┐ 2024-06-26T08:14:04.0090444Z [rank0]:W0626 08:14:04.008000 140439397220992 torch/_dynamo/backends/distributed.py:105] [2/0_1] │ Index │ Extra Ops │ Extra Param Size (b) │ 2024-06-26T08:14:04.0091988Z [rank0]:W0626 08:14:04.008000 140439397220992 torch/_dynamo/backends/distributed.py:105] [2/0_1] ├─────────┼─────────────┼────────────────────────┤ 2024-06-26T08:14:04.0093465Z [rank0]:W0626 08:14:04.008000 140439397220992 torch/_dynamo/backends/distributed.py:105] [2/0_1] │ 0 │ 157 │ 44910720 │ 2024-06-26T08:14:04.0094971Z [rank0]:W0626 08:14:04.008000 140439397220992 torch/_dynamo/backends/distributed.py:105] [2/0_1] └─────────┴─────────────┴────────────────────────┘ 2024-06-26T08:14:07.4137640Z [rank0]:W0626 08:14:07.413000 140439397220992 torch/_dynamo/variables/tensor.py:714] [5/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T08:14:07.4139124Z [rank0]:W0626 08:14:07.413000 140439397220992 torch/_dynamo/variables/tensor.py:714] [5/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T08:14:07.4140300Z [rank0]:W0626 08:14:07.413000 140439397220992 torch/_dynamo/variables/tensor.py:714] [5/0] or: 2024-06-26T08:14:07.4141416Z [rank0]:W0626 08:14:07.413000 140439397220992 torch/_dynamo/variables/tensor.py:714] [5/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T08:14:07.4142741Z [rank0]:W0626 08:14:07.413000 140439397220992 torch/_dynamo/variables/tensor.py:714] [5/0] to include these operations in the captured graph. 2024-06-26T08:14:07.4143853Z [rank0]:W0626 08:14:07.413000 140439397220992 torch/_dynamo/variables/tensor.py:714] [5/0] 2024-06-26T08:14:08.4174337Z pass 2024-06-26T08:14:08.4180916Z TIMING: entire_frame_compile:11.7982 backend_compile:2.15591 2024-06-26T08:14:08.4181800Z STATS: call_* op count: 877 | FakeTensor.__torch_dispatch__:2446 | FakeTensorMode.__torch_dispatch__:13631 2024-06-26T08:14:08.4182718Z Dynamo produced 7 graphs covering 877 ops with 5 graph breaks (3 unique) 2024-06-26T08:14:08.7687728Z [rank0]:[W626 08:14:08.819141964 ProcessGroupNCCL.cpp:1161] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) 2024-06-26T08:14:12.3340541Z 2024-06-26T08:14:12.6030955Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:12.6031917Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:12.6032813Z cuda eval moondream 2024-06-26T08:14:12.6035496Z Traceback (most recent call last): 2024-06-26T08:14:12.6036212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:14:12.6037158Z ) = runner.load_model( 2024-06-26T08:14:12.6037869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 327, in load_model 2024-06-26T08:14:12.6038817Z benchmark = benchmark_cls( 2024-06-26T08:14:12.6039724Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-26T08:14:12.6040516Z obj = type.__call__(cls, *args, **kwargs) 2024-06-26T08:14:12.6041401Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream/__init__.py", line 11, in __init__ 2024-06-26T08:14:12.6042773Z super().__init__(name="moondream", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:14:12.6044040Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-06-26T08:14:12.6045210Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:14:12.6046187Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-06-26T08:14:12.6046942Z self._skip_by_device_name() 2024-06-26T08:14:12.6047750Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-06-26T08:14:12.6049221Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-06-26T08:14:12.6050320Z NotImplementedError: The current device NVIDIA A10G is skipped by its `moondream/metadata.yaml`. 2024-06-26T08:14:12.6050910Z 2024-06-26T08:14:12.6051032Z model_fail_to_load 2024-06-26T08:14:15.7502747Z 2024-06-26T08:14:17.6236428Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M 2024-06-26T08:14:17.8405936Z num decayed parameter tensors: 50, with 124,354,560 parameters 2024-06-26T08:14:17.8407204Z num non-decayed parameter tensors: 98, with 121,344 parameters 2024-06-26T08:14:17.8409730Z using fused AdamW: True 2024-06-26T08:14:18.0983308Z 2024-06-26T08:14:18.0984068Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:14:18.1022723Z cuda eval nanogpt 2024-06-26T08:14:20.6115067Z pass 2024-06-26T08:14:20.6117208Z TIMING: backend_compile:0.00022 entire_frame_compile:1.97645 2024-06-26T08:14:20.6118684Z STATS: call_* op count: 786 | FakeTensorMode.__torch_dispatch__:2980 | FakeTensor.__torch_dispatch__:675 2024-06-26T08:14:20.6119660Z Dynamo produced 1 graphs covering 786 ops with 0 graph breaks (0 unique) 2024-06-26T08:14:23.8473445Z 2024-06-26T08:14:26.1533946Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:26.1534505Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:14:26.1537698Z cuda eval nvidia_deeprecommender 2024-06-26T08:14:26.6180788Z pass 2024-06-26T08:14:26.6183932Z TIMING: backend_compile:6e-05 entire_frame_compile:0.19053 2024-06-26T08:14:26.6184666Z STATS: call_* op count: 26 | FakeTensorMode.__torch_dispatch__:138 2024-06-26T08:14:26.6185405Z Dynamo produced 1 graphs covering 26 ops with 0 graph breaks (0 unique) 2024-06-26T08:14:29.7795457Z 2024-06-26T08:14:30.5769053Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:30.5769626Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:30.5792502Z cuda eval opacus_cifar10 2024-06-26T08:14:33.9518422Z pass 2024-06-26T08:14:33.9519489Z TIMING: backend_compile:8e-05 entire_frame_compile:3.04855 2024-06-26T08:14:33.9520185Z STATS: call_* op count: 138 | FakeTensorMode.__torch_dispatch__:379 2024-06-26T08:14:33.9520933Z Dynamo produced 1 graphs covering 138 ops with 0 graph breaks (0 unique) 2024-06-26T08:14:37.2261219Z 2024-06-26T08:14:37.7759823Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:37.7760298Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:37.7825278Z cuda eval phlippe_densenet 2024-06-26T08:14:39.9902485Z pass 2024-06-26T08:14:39.9904361Z TIMING: backend_compile:0.00011 entire_frame_compile:1.85664 2024-06-26T08:14:39.9905215Z STATS: call_* op count: 372 | FakeTensor.__torch_dispatch__:993 | FakeTensorMode.__torch_dispatch__:4848 2024-06-26T08:14:39.9906135Z Dynamo produced 1 graphs covering 372 ops with 0 graph breaks (0 unique) 2024-06-26T08:14:43.2652406Z 2024-06-26T08:14:43.7313126Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:43.7313635Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:43.7345448Z cuda eval phlippe_resnet 2024-06-26T08:14:44.6172832Z pass 2024-06-26T08:14:44.6173602Z TIMING: backend_compile:7e-05 entire_frame_compile:0.63573 2024-06-26T08:14:44.6175972Z STATS: call_* op count: 142 | FakeTensor.__torch_dispatch__:391 | FakeTensorMode.__torch_dispatch__:1804 2024-06-26T08:14:44.6176991Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-06-26T08:14:47.8703033Z 2024-06-26T08:14:48.1924130Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:48.1924773Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:48.1925543Z cuda eval pyhpc_equation_of_state 2024-06-26T08:14:48.7763609Z pass 2024-06-26T08:14:48.7764493Z TIMING: backend_compile:0.00015 entire_frame_compile:0.41286 2024-06-26T08:14:48.7765847Z STATS: call_* op count: 732 | FakeTensorMode.__torch_dispatch__:744 2024-06-26T08:14:48.7766734Z Dynamo produced 1 graphs covering 732 ops with 0 graph breaks (0 unique) 2024-06-26T08:14:51.9927076Z 2024-06-26T08:14:52.2948721Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:52.2949259Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:52.2952027Z cuda eval pyhpc_isoneutral_mixing 2024-06-26T08:14:53.9714063Z pass 2024-06-26T08:14:53.9714638Z TIMING: backend_compile:0.0003 entire_frame_compile:1.48152 2024-06-26T08:14:53.9717341Z STATS: call_* op count: 1488 | FakeTensorMode.__torch_dispatch__:4719 2024-06-26T08:14:53.9718184Z Dynamo produced 1 graphs covering 1488 ops with 0 graph breaks (0 unique) 2024-06-26T08:14:57.2055673Z 2024-06-26T08:14:57.9511591Z loading model: 0it [00:00, ?it/s]WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-06-26T08:14:58.0315676Z 2024-06-26T08:14:58.0316400Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:58.0317211Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-06-26T08:14:58.0318012Z cuda eval pyhpc_turbulent_kinetic_energy 2024-06-26T08:14:58.0772412Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-06-26T08:14:59.9878829Z pass 2024-06-26T08:15:00.0061477Z TIMING: backend_compile:0.00037 entire_frame_compile:1.66017 2024-06-26T08:15:00.0062449Z STATS: call_* op count: 1992 | FakeTensorMode.__torch_dispatch__:5445 | FakeTensor.__torch_dispatch__:1 2024-06-26T08:15:00.0063343Z Dynamo produced 1 graphs covering 1992 ops with 0 graph breaks (0 unique) 2024-06-26T08:15:03.2287201Z 2024-06-26T08:15:03.8997113Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:03.8997625Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:03.9010438Z cuda eval pytorch_CycleGAN_and_pix2pix 2024-06-26T08:15:10.1025289Z pass 2024-06-26T08:15:10.1030008Z TIMING: backend_compile:8e-05 entire_frame_compile:0.56529 2024-06-26T08:15:10.1030936Z STATS: call_* op count: 182 | FakeTensorMode.__torch_dispatch__:1466 | FakeTensor.__torch_dispatch__:297 2024-06-26T08:15:10.1031850Z Dynamo produced 1 graphs covering 182 ops with 0 graph breaks (0 unique) 2024-06-26T08:15:13.3891163Z 2024-06-26T08:15:14.4019728Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:14.4020314Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:15:14.4041498Z cuda eval pytorch_stargan 2024-06-26T08:15:29.6772491Z pass 2024-06-26T08:15:29.6816673Z TIMING: backend_compile:7e-05 entire_frame_compile:0.61218 2024-06-26T08:15:29.6817613Z STATS: call_* op count: 112 | FakeTensorMode.__torch_dispatch__:2222 | FakeTensor.__torch_dispatch__:326 2024-06-26T08:15:29.6818652Z Dynamo produced 1 graphs covering 112 ops with 0 graph breaks (0 unique) 2024-06-26T08:15:32.9127143Z 2024-06-26T08:15:33.5116842Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:33.5117573Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:33.5139627Z cuda eval pytorch_unet 2024-06-26T08:15:41.7988610Z pass 2024-06-26T08:15:41.8067522Z TIMING: backend_compile:7e-05 entire_frame_compile:0.85374 2024-06-26T08:15:41.8068734Z STATS: call_* op count: 142 | FakeTensor.__torch_dispatch__:462 | FakeTensorMode.__torch_dispatch__:2196 2024-06-26T08:15:41.8069993Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-06-26T08:15:45.0382147Z 2024-06-26T08:15:45.8046472Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnet152-394f9c45.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnet152-394f9c45.pth 2024-06-26T08:15:45.8225351Z 2024-06-26T08:15:45.8225361Z 2024-06-26T08:15:45.9228815Z 0% 0.00/230M [00:00 will be ignored 2024-06-26T08:27:48.8975008Z pass 2024-06-26T08:27:48.8975859Z TIMING: entire_frame_compile:0.81683 backend_compile:0.00017 2024-06-26T08:27:48.8976980Z STATS: call_* op count: 78 | FakeTensor.__torch_dispatch__:375 | FakeTensorMode.__torch_dispatch__:554 2024-06-26T08:27:48.8977958Z Dynamo produced 3 graphs covering 78 ops with 7 graph breaks (5 unique) 2024-06-26T08:27:52.1660386Z 2024-06-26T08:27:54.3914475Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:27:54.3915421Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:27:54.3915979Z cuda train mnasnet1_0 2024-06-26T08:27:57.7650963Z W0626 08:27:57.764000 139776725656192 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:28:05.0692520Z pass 2024-06-26T08:28:05.0721269Z TIMING: entire_frame_compile:8.91435 backend_compile:0.00055 2024-06-26T08:28:05.0722131Z STATS: call_* op count: 648 | FakeTensor.__torch_dispatch__:4021 | FakeTensorMode.__torch_dispatch__:6304 2024-06-26T08:28:05.0723082Z Dynamo produced 3 graphs covering 648 ops with 7 graph breaks (5 unique) 2024-06-26T08:28:08.5569880Z 2024-06-26T08:28:10.3299237Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:28:10.3299718Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:28:10.3300177Z cuda train mobilenet_v2 2024-06-26T08:28:13.8986840Z pass 2024-06-26T08:28:13.9033317Z TIMING: entire_frame_compile:1.80658 backend_compile:0.00011 2024-06-26T08:28:13.9034166Z STATS: call_* op count: 157 | FakeTensor.__torch_dispatch__:1034 | FakeTensorMode.__torch_dispatch__:6340 2024-06-26T08:28:13.9035383Z Dynamo produced 2 graphs covering 157 ops with 6 graph breaks (5 unique) 2024-06-26T08:28:17.2203287Z 2024-06-26T08:28:18.9988381Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:28:18.9989025Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:28:18.9989603Z cuda train mobilenet_v2_quantized_qat 2024-06-26T08:28:18.9990913Z Traceback (most recent call last): 2024-06-26T08:28:18.9991881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:28:18.9992841Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:28:18.9993848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in forward_and_backward_pass 2024-06-26T08:28:18.9994692Z pred = mod(*cloned_inputs) 2024-06-26T08:28:18.9995815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 738, in call_wrapped 2024-06-26T08:28:18.9996798Z return self._wrapped_call(self, *args, **kwargs) 2024-06-26T08:28:18.9998111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 316, in __call__ 2024-06-26T08:28:18.9998960Z raise e 2024-06-26T08:28:18.9999710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 303, in __call__ 2024-06-26T08:28:19.0000664Z return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] 2024-06-26T08:28:19.0002146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:28:19.0003298Z return self._call_impl(*args, **kwargs) 2024-06-26T08:28:19.0004239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:28:19.0005111Z return forward_call(*args, **kwargs) 2024-06-26T08:28:19.0005746Z File ".3", line 207, in forward 2024-06-26T08:28:19.0006571Z activation_post_process_101 = self.activation_post_process_101(classifier_1); classifier_1 = None 2024-06-26T08:28:19.0007832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:28:19.0008994Z return self._call_impl(*args, **kwargs) 2024-06-26T08:28:19.0009927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:28:19.0010760Z return forward_call(*args, **kwargs) 2024-06-26T08:28:19.0011723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 343, in forward 2024-06-26T08:28:19.0012633Z return torch.fused_moving_avg_obs_fake_quant( 2024-06-26T08:28:19.0013191Z RuntimeError: expected scalar type Float but found Half 2024-06-26T08:28:19.0013568Z 2024-06-26T08:28:19.0014150Z The above exception was the direct cause of the following exception: 2024-06-26T08:28:19.0014609Z 2024-06-26T08:28:19.0014765Z Traceback (most recent call last): 2024-06-26T08:28:19.0015422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:28:19.0016197Z ) = runner.load_model( 2024-06-26T08:28:19.0016860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:28:19.0017630Z self.validate_model(model, example_inputs) 2024-06-26T08:28:19.0018388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:28:19.0019156Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:28:19.0019624Z RuntimeError: Eager run failed 2024-06-26T08:28:19.0019890Z 2024-06-26T08:28:19.0020013Z eager_fail_to_run 2024-06-26T08:28:22.2923644Z 2024-06-26T08:28:24.4636112Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:28:24.4636682Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:28:24.4637151Z cuda train mobilenet_v3_large 2024-06-26T08:28:28.2119699Z W0626 08:28:28.211000 140581633430144 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:28:36.4841505Z pass 2024-06-26T08:28:36.4876446Z TIMING: entire_frame_compile:10.11857 backend_compile:0.0006 2024-06-26T08:28:36.4877303Z STATS: call_* op count: 731 | FakeTensor.__torch_dispatch__:4453 | FakeTensorMode.__torch_dispatch__:6696 2024-06-26T08:28:36.4878210Z Dynamo produced 3 graphs covering 731 ops with 7 graph breaks (5 unique) 2024-06-26T08:28:40.0530150Z 2024-06-26T08:28:42.0326834Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:28:42.0327327Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:28:42.0327765Z cuda train moco 2024-06-26T08:28:46.6087010Z [rank0]:W0626 08:28:46.607000 139774544757376 torch/_logging/_internal.py:1040] [3/0] Profiler function will be ignored 2024-06-26T08:28:52.2306873Z [rank0]:W0626 08:28:52.229000 139774544757376 torch/_dynamo/backends/distributed.py:88] [4/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance. 2024-06-26T08:28:52.2497228Z [rank0]:W0626 08:28:52.249000 139774544757376 torch/_dynamo/backends/distributed.py:105] [4/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes: 2024-06-26T08:28:52.2503711Z [rank0]:W0626 08:28:52.249000 139774544757376 torch/_dynamo/backends/distributed.py:105] [4/0_1] ┌─────────┬─────────────┬────────────────────────┐ 2024-06-26T08:28:52.2505292Z [rank0]:W0626 08:28:52.249000 139774544757376 torch/_dynamo/backends/distributed.py:105] [4/0_1] │ Index │ Extra Ops │ Extra Param Size (b) │ 2024-06-26T08:28:52.2506831Z [rank0]:W0626 08:28:52.249000 139774544757376 torch/_dynamo/backends/distributed.py:105] [4/0_1] ├─────────┼─────────────┼────────────────────────┤ 2024-06-26T08:28:52.2508295Z [rank0]:W0626 08:28:52.249000 139774544757376 torch/_dynamo/backends/distributed.py:105] [4/0_1] │ 0 │ 161 │ 94032128 │ 2024-06-26T08:28:52.2509772Z [rank0]:W0626 08:28:52.249000 139774544757376 torch/_dynamo/backends/distributed.py:105] [4/0_1] └─────────┴─────────────┴────────────────────────┘ 2024-06-26T08:28:55.7067333Z [rank0]:W0626 08:28:55.705000 139774544757376 torch/_dynamo/variables/tensor.py:714] [7/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T08:28:55.7068935Z [rank0]:W0626 08:28:55.705000 139774544757376 torch/_dynamo/variables/tensor.py:714] [7/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T08:28:55.7070373Z [rank0]:W0626 08:28:55.705000 139774544757376 torch/_dynamo/variables/tensor.py:714] [7/0] or: 2024-06-26T08:28:55.7071485Z [rank0]:W0626 08:28:55.705000 139774544757376 torch/_dynamo/variables/tensor.py:714] [7/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T08:28:55.7072826Z [rank0]:W0626 08:28:55.705000 139774544757376 torch/_dynamo/variables/tensor.py:714] [7/0] to include these operations in the captured graph. 2024-06-26T08:28:55.7073927Z [rank0]:W0626 08:28:55.705000 139774544757376 torch/_dynamo/variables/tensor.py:714] [7/0] 2024-06-26T08:28:57.0194782Z pass 2024-06-26T08:28:57.0516678Z TIMING: entire_frame_compile:10.12525 backend_compile:2.17511 2024-06-26T08:28:57.0517654Z STATS: call_* op count: 887 | FakeTensor.__torch_dispatch__:2470 | FakeTensorMode.__torch_dispatch__:16729 2024-06-26T08:28:57.0518647Z Dynamo produced 10 graphs covering 887 ops with 11 graph breaks (8 unique) 2024-06-26T08:28:57.4422275Z [rank0]:[W626 08:28:57.492607553 ProcessGroupNCCL.cpp:1161] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) 2024-06-26T08:29:01.0789767Z 2024-06-26T08:29:02.9651352Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M 2024-06-26T08:29:03.1716584Z num decayed parameter tensors: 50, with 124,354,560 parameters 2024-06-26T08:29:03.1717655Z num non-decayed parameter tensors: 98, with 121,344 parameters 2024-06-26T08:29:03.1720160Z using fused AdamW: True 2024-06-26T08:29:03.5001225Z 2024-06-26T08:29:03.5001747Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:29:03.5002340Z cuda train nanogpt 2024-06-26T08:29:07.1685667Z W0626 08:29:07.167000 140078009856640 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:29:12.8067958Z pass 2024-06-26T08:29:12.8790556Z TIMING: entire_frame_compile:7.70762 backend_compile:0.00056 2024-06-26T08:29:12.8792147Z STATS: call_* op count: 859 | FakeTensorMode.__torch_dispatch__:5860 | FakeTensor.__torch_dispatch__:3507 2024-06-26T08:29:12.8793057Z Dynamo produced 3 graphs covering 859 ops with 7 graph breaks (5 unique) 2024-06-26T08:29:16.2670999Z 2024-06-26T08:29:18.6031452Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:29:18.6032012Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:29:18.6032481Z cuda train nvidia_deeprecommender 2024-06-26T08:29:19.6019351Z W0626 08:29:19.601000 139916439859840 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:29:20.2644050Z pass 2024-06-26T08:29:20.2867244Z TIMING: entire_frame_compile:0.76263 backend_compile:0.00016 2024-06-26T08:29:20.2868119Z STATS: call_* op count: 71 | FakeTensorMode.__torch_dispatch__:364 | FakeTensor.__torch_dispatch__:248 2024-06-26T08:29:20.2869036Z Dynamo produced 3 graphs covering 71 ops with 7 graph breaks (5 unique) 2024-06-26T08:29:23.5121976Z 2024-06-26T08:29:24.3048128Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:29:24.3048829Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:29:24.3049351Z cuda train opacus_cifar10 2024-06-26T08:29:24.3071698Z Traceback (most recent call last): 2024-06-26T08:29:24.3072529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:29:24.3073382Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:29:24.3074319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 458, in forward_and_backward_pass 2024-06-26T08:29:24.3075237Z self.grad_scaler.scale(loss).backward() 2024-06-26T08:29:24.3076636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-06-26T08:29:24.3077578Z torch.autograd.backward( 2024-06-26T08:29:24.3078733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 288, in backward 2024-06-26T08:29:24.3079686Z _engine_run_backward( 2024-06-26T08:29:24.3080662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 768, in _engine_run_backward 2024-06-26T08:29:24.3081965Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-06-26T08:29:24.3083222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 97, in __call__ 2024-06-26T08:29:24.3084121Z return self.hook(module, *args, **kwargs) 2024-06-26T08:29:24.3085313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 327, in capture_backprops_hook 2024-06-26T08:29:24.3086409Z activations, backprops = self.rearrange_grad_samples( 2024-06-26T08:29:24.3087611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 383, in rearrange_grad_samples 2024-06-26T08:29:24.3088950Z raise ValueError( 2024-06-26T08:29:24.3089873Z ValueError: No activations detected for , run forward after add_hooks(model) 2024-06-26T08:29:24.3090571Z 2024-06-26T08:29:24.3090872Z The above exception was the direct cause of the following exception: 2024-06-26T08:29:24.3091324Z 2024-06-26T08:29:24.3091473Z Traceback (most recent call last): 2024-06-26T08:29:24.3092199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:29:24.3092866Z ) = runner.load_model( 2024-06-26T08:29:24.3093534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:29:24.3094300Z self.validate_model(model, example_inputs) 2024-06-26T08:29:24.3095114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:29:24.3096041Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:29:24.3096515Z RuntimeError: Eager run failed 2024-06-26T08:29:24.3096787Z 2024-06-26T08:29:24.3096913Z eager_fail_to_run 2024-06-26T08:29:27.5041278Z 2024-06-26T08:29:28.1846846Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:29:28.1847456Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:29:28.1848014Z cuda train phlippe_densenet 2024-06-26T08:29:30.8939635Z pass 2024-06-26T08:29:30.8943398Z TIMING: entire_frame_compile:2.08124 backend_compile:0.00012 2024-06-26T08:29:30.8944366Z STATS: call_* op count: 190 | FakeTensor.__torch_dispatch__:1013 | FakeTensorMode.__torch_dispatch__:6468 2024-06-26T08:29:30.8945274Z Dynamo produced 2 graphs covering 190 ops with 6 graph breaks (5 unique) 2024-06-26T08:29:34.2387208Z 2024-06-26T08:29:34.8221428Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:29:34.8221945Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:29:34.8222401Z cuda train phlippe_resnet 2024-06-26T08:29:35.9455353Z pass 2024-06-26T08:29:35.9456467Z TIMING: entire_frame_compile:0.75865 backend_compile:0.0001 2024-06-26T08:29:35.9457978Z STATS: call_* op count: 75 | FakeTensor.__torch_dispatch__:411 | FakeTensorMode.__torch_dispatch__:2480 2024-06-26T08:29:35.9458863Z Dynamo produced 2 graphs covering 75 ops with 6 graph breaks (5 unique) 2024-06-26T08:29:39.2525312Z 2024-06-26T08:29:40.3542553Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:29:40.3543045Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:29:40.3543514Z cuda train pytorch_CycleGAN_and_pix2pix 2024-06-26T08:29:49.9005427Z pass 2024-06-26T08:29:49.9045874Z TIMING: entire_frame_compile:0.70009 backend_compile:0.00026 2024-06-26T08:29:49.9046899Z STATS: call_* op count: 95 | FakeTensorMode.__torch_dispatch__:2450 | FakeTensor.__torch_dispatch__:317 2024-06-26T08:29:49.9048130Z Dynamo produced 2 graphs covering 95 ops with 6 graph breaks (5 unique) 2024-06-26T08:29:53.2055610Z 2024-06-26T08:29:54.5453518Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:29:54.5454106Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:29:54.5454574Z cuda train pytorch_stargan 2024-06-26T08:30:21.4518301Z pass 2024-06-26T08:30:21.4714212Z TIMING: entire_frame_compile:0.73587 backend_compile:0.0001 2024-06-26T08:30:21.4715179Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:2962 | FakeTensor.__torch_dispatch__:346 2024-06-26T08:30:21.4718375Z Dynamo produced 2 graphs covering 60 ops with 6 graph breaks (5 unique) 2024-06-26T08:30:24.8075895Z 2024-06-26T08:30:25.5903508Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:30:25.5903985Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:30:25.5904445Z cuda train pytorch_unet 2024-06-26T08:30:46.0528395Z W0626 08:30:46.052000 140147107193472 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:30:49.2434044Z pass_due_to_skip 2024-06-26T08:30:49.4422494Z TIMING: entire_frame_compile:4.14471 backend_compile:0.00033 2024-06-26T08:30:49.4423780Z STATS: call_* op count: 315 | FakeTensor.__torch_dispatch__:1888 | FakeTensorMode.__torch_dispatch__:3174 2024-06-26T08:30:49.4424679Z Dynamo produced 3 graphs covering 315 ops with 7 graph breaks (5 unique) 2024-06-26T08:30:52.7943421Z 2024-06-26T08:30:54.9330467Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:30:54.9330946Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:30:54.9333302Z cuda train resnet152 2024-06-26T08:31:05.1121553Z W0626 08:31:05.111000 140113121256064 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:31:26.7912280Z pass 2024-06-26T08:31:26.8326968Z TIMING: entire_frame_compile:25.72197 backend_compile:0.00141 2024-06-26T08:31:26.8328970Z STATS: call_* op count: 1938 | FakeTensor.__torch_dispatch__:11723 | FakeTensorMode.__torch_dispatch__:18408 2024-06-26T08:31:26.8329921Z Dynamo produced 3 graphs covering 1938 ops with 7 graph breaks (5 unique) 2024-06-26T08:31:30.7609958Z 2024-06-26T08:31:31.8195255Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:31:31.8195741Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:31:31.8197322Z cuda train resnet18 2024-06-26T08:31:34.1282284Z pass 2024-06-26T08:31:34.1347768Z TIMING: entire_frame_compile:0.75872 backend_compile:9e-05 2024-06-26T08:31:34.1348634Z STATS: call_* op count: 73 | FakeTensor.__torch_dispatch__:409 | FakeTensorMode.__torch_dispatch__:2508 2024-06-26T08:31:34.1349528Z Dynamo produced 2 graphs covering 73 ops with 6 graph breaks (5 unique) 2024-06-26T08:31:37.4554722Z 2024-06-26T08:31:39.0214672Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:31:39.0215283Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:31:39.0215833Z cuda train resnet50 2024-06-26T08:31:45.0501362Z pass 2024-06-26T08:31:45.0685386Z TIMING: entire_frame_compile:1.73773 backend_compile:0.00011 2024-06-26T08:31:45.0686632Z STATS: call_* op count: 179 | FakeTensor.__torch_dispatch__:1014 | FakeTensorMode.__torch_dispatch__:6422 2024-06-26T08:31:45.0687991Z Dynamo produced 2 graphs covering 179 ops with 6 graph breaks (5 unique) 2024-06-26T08:31:48.4167262Z 2024-06-26T08:31:50.4618321Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:31:50.4618893Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:31:50.4619397Z cuda train resnet50_quantized_qat 2024-06-26T08:31:50.4624283Z Traceback (most recent call last): 2024-06-26T08:31:50.4625369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:31:50.4626231Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:31:50.4627178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in forward_and_backward_pass 2024-06-26T08:31:50.4628058Z pred = mod(*cloned_inputs) 2024-06-26T08:31:50.4630049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 738, in call_wrapped 2024-06-26T08:31:50.4631205Z return self._wrapped_call(self, *args, **kwargs) 2024-06-26T08:31:50.4632220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 316, in __call__ 2024-06-26T08:31:50.4633119Z raise e 2024-06-26T08:31:50.4633998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 303, in __call__ 2024-06-26T08:31:50.4635019Z return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] 2024-06-26T08:31:50.4636152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:31:50.4637162Z return self._call_impl(*args, **kwargs) 2024-06-26T08:31:50.4638247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:31:50.4639175Z return forward_call(*args, **kwargs) 2024-06-26T08:31:50.4639717Z File ".3", line 167, in forward 2024-06-26T08:31:50.4640609Z activation_post_process_73 = self.activation_post_process_73(fc); fc = None 2024-06-26T08:31:50.4641836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:31:50.4642798Z return self._call_impl(*args, **kwargs) 2024-06-26T08:31:50.4643820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:31:50.4644732Z return forward_call(*args, **kwargs) 2024-06-26T08:31:50.4645787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 343, in forward 2024-06-26T08:31:50.4646775Z return torch.fused_moving_avg_obs_fake_quant( 2024-06-26T08:31:50.4647395Z RuntimeError: expected scalar type Float but found Half 2024-06-26T08:31:50.4647777Z 2024-06-26T08:31:50.4648112Z The above exception was the direct cause of the following exception: 2024-06-26T08:31:50.4648922Z 2024-06-26T08:31:50.4649136Z Traceback (most recent call last): 2024-06-26T08:31:50.4649811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:31:50.4650540Z ) = runner.load_model( 2024-06-26T08:31:50.4651269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:31:50.4652092Z self.validate_model(model, example_inputs) 2024-06-26T08:31:50.4652906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:31:50.4653729Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:31:50.4654242Z RuntimeError: Eager run failed 2024-06-26T08:31:50.4654524Z 2024-06-26T08:31:50.4654655Z eager_fail_to_run 2024-06-26T08:31:53.6895951Z 2024-06-26T08:31:55.4172470Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:31:55.4173419Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:31:55.4174307Z cuda train resnext50_32x4d 2024-06-26T08:32:02.8381388Z W0626 08:32:02.837000 140163532993152 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:32:10.3573867Z pass 2024-06-26T08:32:10.3731096Z TIMING: entire_frame_compile:8.97471 backend_compile:0.00055 2024-06-26T08:32:10.3731954Z STATS: call_* op count: 680 | FakeTensor.__torch_dispatch__:4071 | FakeTensorMode.__torch_dispatch__:6432 2024-06-26T08:32:10.3732854Z Dynamo produced 3 graphs covering 680 ops with 7 graph breaks (5 unique) 2024-06-26T08:32:13.8923347Z 2024-06-26T08:32:19.7971629Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:19.7972167Z loading model: 0it [00:05, ?it/s] 2024-06-26T08:32:19.7972703Z cuda train sam 2024-06-26T08:32:19.7980058Z Traceback (most recent call last): 2024-06-26T08:32:19.7981111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:32:19.7981895Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:32:19.7982761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 458, in forward_and_backward_pass 2024-06-26T08:32:19.7983611Z self.grad_scaler.scale(loss).backward() 2024-06-26T08:32:19.7984921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-06-26T08:32:19.7985888Z torch.autograd.backward( 2024-06-26T08:32:19.7986986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 288, in backward 2024-06-26T08:32:19.7987885Z _engine_run_backward( 2024-06-26T08:32:19.7988783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 768, in _engine_run_backward 2024-06-26T08:32:19.7989941Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-06-26T08:32:19.7990902Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn 2024-06-26T08:32:19.7991592Z 2024-06-26T08:32:19.7991893Z The above exception was the direct cause of the following exception: 2024-06-26T08:32:19.7992346Z 2024-06-26T08:32:19.7992494Z Traceback (most recent call last): 2024-06-26T08:32:19.7993149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:32:19.7993807Z ) = runner.load_model( 2024-06-26T08:32:19.7994475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:32:19.7995240Z self.validate_model(model, example_inputs) 2024-06-26T08:32:19.7995997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:32:19.7996763Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:32:19.7997234Z RuntimeError: Eager run failed 2024-06-26T08:32:19.7997500Z 2024-06-26T08:32:19.7997620Z eager_fail_to_run 2024-06-26T08:32:23.0760689Z 2024-06-26T08:32:24.1494505Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:24.1495025Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:32:24.1495563Z cuda train shufflenet_v2_x1_0 2024-06-26T08:32:27.0572660Z pass 2024-06-26T08:32:27.0581228Z TIMING: entire_frame_compile:1.97032 backend_compile:0.00013 2024-06-26T08:32:27.0582311Z STATS: call_* op count: 275 | FakeTensor.__torch_dispatch__:1057 | FakeTensorMode.__torch_dispatch__:7321 2024-06-26T08:32:27.0583224Z Dynamo produced 2 graphs covering 275 ops with 6 graph breaks (5 unique) 2024-06-26T08:32:30.4067671Z 2024-06-26T08:32:31.1146039Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:31.1146527Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:31.1146985Z cuda train soft_actor_critic 2024-06-26T08:32:31.5576255Z W0626 08:32:31.556000 140287466295936 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:32:31.8468515Z pass 2024-06-26T08:32:31.8469293Z TIMING: entire_frame_compile:0.52595 backend_compile:0.0002 2024-06-26T08:32:31.8470157Z STATS: call_* op count: 57 | FakeTensor.__torch_dispatch__:174 | FakeTensorMode.__torch_dispatch__:320 2024-06-26T08:32:31.8471055Z Dynamo produced 4 graphs covering 57 ops with 6 graph breaks (5 unique) 2024-06-26T08:32:35.0848322Z 2024-06-26T08:32:36.2748022Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:36.2748646Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:32:36.2749213Z cuda train speech_transformer 2024-06-26T08:32:41.3888882Z pass 2024-06-26T08:32:41.4342056Z TIMING: entire_frame_compile:2.56662 backend_compile:0.00057 2024-06-26T08:32:41.4343832Z STATS: call_* op count: 771 | FakeTensorMode.__torch_dispatch__:5823 | FakeTensor.__torch_dispatch__:847 2024-06-26T08:32:41.4345654Z Dynamo produced 10 graphs covering 771 ops with 16 graph breaks (7 unique) 2024-06-26T08:32:44.7497052Z 2024-06-26T08:32:45.7176252Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:45.7177323Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:45.7178039Z cuda train squeezenet1_1 2024-06-26T08:32:47.3617672Z pass 2024-06-26T08:32:47.3631421Z TIMING: entire_frame_compile:0.55434 backend_compile:0.0001 2024-06-26T08:32:47.3632640Z STATS: call_* op count: 70 | FakeTensor.__torch_dispatch__:364 | FakeTensorMode.__torch_dispatch__:1422 2024-06-26T08:32:47.3633981Z Dynamo produced 2 graphs covering 70 ops with 6 graph breaks (5 unique) 2024-06-26T08:32:50.6840240Z 2024-06-26T08:32:51.8396495Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:51.8396859Z 2024-06-26T08:32:52.4397435Z Loading pipeline components...: 0% 0/6 [00:00 will be ignored 2024-06-26T08:33:30.2411378Z pass 2024-06-26T08:33:30.2458736Z TIMING: entire_frame_compile:14.14908 backend_compile:0.00077 2024-06-26T08:33:30.2459632Z STATS: call_* op count: 974 | FakeTensor.__torch_dispatch__:4856 | FakeTensorMode.__torch_dispatch__:7018 2024-06-26T08:33:30.2460538Z Dynamo produced 3 graphs covering 974 ops with 7 graph breaks (5 unique) 2024-06-26T08:33:34.0234941Z 2024-06-26T08:33:36.5396508Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:33:36.5397596Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:33:36.5398585Z cuda train timm_regnet 2024-06-26T08:33:52.4616144Z pass 2024-06-26T08:33:52.5024698Z TIMING: entire_frame_compile:5.99555 backend_compile:0.00021 2024-06-26T08:33:52.5026394Z STATS: call_* op count: 463 | FakeTensor.__torch_dispatch__:896 | FakeTensorMode.__torch_dispatch__:8504 2024-06-26T08:33:52.5027854Z Dynamo produced 2 graphs covering 463 ops with 6 graph breaks (5 unique) 2024-06-26T08:33:56.0515324Z 2024-06-26T08:33:57.7822876Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:33:57.7823400Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:33:57.7823848Z cuda train timm_resnest 2024-06-26T08:34:02.9624347Z W0626 08:34:02.961000 140086661026432 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:34:06.8764575Z pass 2024-06-26T08:34:06.8845230Z TIMING: entire_frame_compile:4.97932 backend_compile:0.00037 2024-06-26T08:34:06.8846094Z STATS: call_* op count: 419 | FakeTensor.__torch_dispatch__:2125 | FakeTensorMode.__torch_dispatch__:3566 2024-06-26T08:34:06.8846993Z Dynamo produced 3 graphs covering 419 ops with 7 graph breaks (5 unique) 2024-06-26T08:34:10.4610105Z 2024-06-26T08:34:11.9783431Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:34:11.9783918Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:34:11.9784362Z cuda train timm_vision_transformer 2024-06-26T08:34:14.9375762Z pass 2024-06-26T08:34:14.9388402Z TIMING: entire_frame_compile:1.73644 backend_compile:0.00014 2024-06-26T08:34:14.9389592Z STATS: call_* op count: 355 | FakeTensor.__torch_dispatch__:1002 | FakeTensorMode.__torch_dispatch__:5315 2024-06-26T08:34:14.9390498Z Dynamo produced 2 graphs covering 355 ops with 6 graph breaks (5 unique) 2024-06-26T08:34:18.3467078Z 2024-06-26T08:34:30.7285102Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:34:30.7286199Z loading model: 0it [00:12, ?it/s] 2024-06-26T08:34:30.7287439Z cuda train timm_vision_transformer_large 2024-06-26T08:34:30.7288286Z pass_due_to_skip 2024-06-26T08:34:30.9914237Z TIMING: 2024-06-26T08:34:30.9914555Z STATS: call_* op count: 0 2024-06-26T08:34:30.9915114Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T08:34:34.3104360Z 2024-06-26T08:34:36.1139679Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:34:36.1140285Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:34:36.1140856Z cuda train timm_vovnet 2024-06-26T08:34:45.6532745Z pass 2024-06-26T08:34:45.6703016Z TIMING: entire_frame_compile:3.38546 backend_compile:0.00014 2024-06-26T08:34:45.6704113Z STATS: call_* op count: 174 | FakeTensor.__torch_dispatch__:286 | FakeTensorMode.__torch_dispatch__:4156 2024-06-26T08:34:45.6706143Z Dynamo produced 2 graphs covering 174 ops with 6 graph breaks (5 unique) 2024-06-26T08:34:49.1598418Z 2024-06-26T08:34:50.8689116Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:34:50.8689865Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:34:50.8690475Z cuda train torch_multimodal_clip 2024-06-26T08:34:57.3237235Z W0626 08:34:57.322000 140473757983360 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:35:08.6435879Z pass 2024-06-26T08:35:08.7252964Z TIMING: entire_frame_compile:13.01531 backend_compile:0.00088 2024-06-26T08:35:08.7253908Z STATS: call_* op count: 961 | FakeTensor.__torch_dispatch__:7687 | FakeTensorMode.__torch_dispatch__:12910 2024-06-26T08:35:08.7254831Z Dynamo produced 3 graphs covering 961 ops with 7 graph breaks (5 unique) 2024-06-26T08:35:12.2970542Z 2024-06-26T08:35:12.8422836Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:35:12.8423332Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:35:12.8423781Z cuda train tts_angular 2024-06-26T08:35:13.4323313Z W0626 08:35:13.431000 140206084149888 torch/_logging/_internal.py:1040] [10/0] Profiler function will be ignored 2024-06-26T08:35:14.0585323Z pass 2024-06-26T08:35:14.0655198Z TIMING: entire_frame_compile:0.78077 backend_compile:0.00024 2024-06-26T08:35:14.0657021Z STATS: call_* op count: 71 | FakeTensor.__torch_dispatch__:317 | FakeTensorMode.__torch_dispatch__:70 2024-06-26T08:35:14.0658782Z Dynamo produced 5 graphs covering 71 ops with 9 graph breaks (6 unique) 2024-06-26T08:35:17.3643535Z 2024-06-26T08:35:19.4735775Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:35:19.4736578Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:35:19.4737037Z cuda train vgg16 2024-06-26T08:35:37.1689620Z pass 2024-06-26T08:35:37.1940928Z TIMING: entire_frame_compile:0.34488 backend_compile:9e-05 2024-06-26T08:35:37.1941937Z STATS: call_* op count: 44 | FakeTensor.__torch_dispatch__:229 | FakeTensorMode.__torch_dispatch__:948 2024-06-26T08:35:37.1942835Z Dynamo produced 2 graphs covering 44 ops with 6 graph breaks (5 unique) 2024-06-26T08:35:40.4647582Z 2024-06-26T08:35:43.5193299Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:35:43.5193815Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:35:43.5195311Z cuda train vision_maskrcnn 2024-06-26T08:36:01.6521319Z W0626 08:36:01.651000 139895875445376 torch/_dynamo/variables/tensor.py:714] [18/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T08:36:01.6523228Z W0626 08:36:01.651000 139895875445376 torch/_dynamo/variables/tensor.py:714] [18/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T08:36:01.6524651Z W0626 08:36:01.651000 139895875445376 torch/_dynamo/variables/tensor.py:714] [18/0] or: 2024-06-26T08:36:01.6526256Z W0626 08:36:01.651000 139895875445376 torch/_dynamo/variables/tensor.py:714] [18/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T08:36:01.6527605Z W0626 08:36:01.651000 139895875445376 torch/_dynamo/variables/tensor.py:714] [18/0] to include these operations in the captured graph. 2024-06-26T08:36:01.6529093Z W0626 08:36:01.651000 139895875445376 torch/_dynamo/variables/tensor.py:714] [18/0] 2024-06-26T08:36:27.0817148Z W0626 08:36:27.081000 139895875445376 torch/_dynamo/convert_frame.py:762] [30/8] torch._dynamo hit config.cache_size_limit (8) 2024-06-26T08:36:27.0819172Z W0626 08:36:27.081000 139895875445376 torch/_dynamo/convert_frame.py:762] [30/8] function: 'roi_align' (/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py:186) 2024-06-26T08:36:27.0820998Z W0626 08:36:27.081000 139895875445376 torch/_dynamo/convert_frame.py:762] [30/8] last reason: tensor 'L['boxes']' size mismatch at index 0. expected 648, actual 619 2024-06-26T08:36:27.0822475Z W0626 08:36:27.081000 139895875445376 torch/_dynamo/convert_frame.py:762] [30/8] To log all recompilation reasons, use TORCH_LOGS="recompiles". 2024-06-26T08:36:27.0824056Z W0626 08:36:27.081000 139895875445376 torch/_dynamo/convert_frame.py:762] [30/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html. 2024-06-26T08:36:31.6820565Z pass 2024-06-26T08:36:31.7452455Z TIMING: entire_frame_compile:40.00281 backend_compile:0.00336 2024-06-26T08:36:31.7453846Z STATS: call_* op count: 2515 | FakeTensorMode.__torch_dispatch__:29856 | FakeTensor.__torch_dispatch__:622 | attempt fast:517 | slow no contiguity match:38 | fast is_contiguous:359 | slow both tensors nontrivially broadcast:120 2024-06-26T08:36:31.7455280Z Dynamo produced 51 graphs covering 2515 ops with 33 graph breaks (8 unique) 2024-06-26T08:36:35.7408187Z 2024-06-26T08:36:38.3724846Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:36:38.3725450Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:36:38.3725933Z cuda train yolov3 2024-06-26T08:37:00.1233305Z W0626 08:37:00.122000 139897322013312 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:37:13.1488683Z pass 2024-06-26T08:37:13.2016279Z TIMING: entire_frame_compile:16.50541 backend_compile:0.00083 2024-06-26T08:37:13.2018513Z STATS: call_* op count: 1232 | FakeTensor.__torch_dispatch__:7586 | FakeTensorMode.__torch_dispatch__:22469 2024-06-26T08:37:13.2019581Z Dynamo produced 4 graphs covering 1232 ops with 8 graph breaks (5 unique) 2024-06-26T08:37:14.3472379Z accuracy pass_rate=78.95% 2024-06-26T08:37:14.3475507Z calls_captured gmean=0.00x mean=427.500x 2024-06-26T08:37:14.3479004Z unique_graphs gmean=0.00x mean=3.816x 2024-06-26T08:37:14.3483014Z graph_breaks gmean=0.00x mean=6.395x 2024-06-26T08:37:14.3485915Z unique_graph_breaks gmean=0.00x mean=4.316x 2024-06-26T08:37:14.3489432Z autograd_captures gmean=0.00x mean=0.000x 2024-06-26T08:37:14.3492968Z autograd_compiles gmean=0.00x mean=0.000x 2024-06-26T08:37:14.3496360Z cudagraph_skips gmean=0.00x mean=0.000x 2024-06-26T08:37:14.8882682Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamo_eager_torchbench_training.csv 2024-06-26T08:37:15.1991943Z llava XFAIL 2024-06-26T08:37:15.1996452Z maml_omniglot PASS 2024-06-26T08:37:15.2001051Z mnasnet1_0 PASS 2024-06-26T08:37:15.2005411Z mobilenet_v2 PASS 2024-06-26T08:37:15.2009885Z mobilenet_v2_quantized_qat XFAIL 2024-06-26T08:37:15.2014521Z mobilenet_v3_large PASS 2024-06-26T08:37:15.2019500Z moco PASS 2024-06-26T08:37:15.2023590Z nanogpt PASS 2024-06-26T08:37:15.2027975Z nvidia_deeprecommender PASS 2024-06-26T08:37:15.2032338Z opacus_cifar10 XFAIL 2024-06-26T08:37:15.2036805Z phlippe_densenet PASS 2024-06-26T08:37:15.2041606Z phlippe_resnet PASS 2024-06-26T08:37:15.2045906Z pytorch_CycleGAN_and_pix2pix PASS 2024-06-26T08:37:15.2050489Z pytorch_stargan PASS 2024-06-26T08:37:15.2054873Z pytorch_unet XFAIL 2024-06-26T08:37:15.2059541Z resnet152 PASS 2024-06-26T08:37:15.2064152Z resnet18 PASS 2024-06-26T08:37:15.2068645Z resnet50 PASS 2024-06-26T08:37:15.2073132Z resnet50_quantized_qat XFAIL 2024-06-26T08:37:15.2077787Z resnext50_32x4d PASS 2024-06-26T08:37:15.2082354Z sam XFAIL 2024-06-26T08:37:15.2086844Z shufflenet_v2_x1_0 PASS 2024-06-26T08:37:15.2091570Z soft_actor_critic PASS 2024-06-26T08:37:15.2096135Z speech_transformer PASS 2024-06-26T08:37:15.2100885Z squeezenet1_1 PASS 2024-06-26T08:37:15.2105323Z stable_diffusion_text_encoder PASS 2024-06-26T08:37:15.2109782Z stable_diffusion_unet XFAIL 2024-06-26T08:37:15.2114322Z timm_efficientnet PASS 2024-06-26T08:37:15.2118855Z timm_regnet PASS 2024-06-26T08:37:15.2123343Z timm_resnest PASS 2024-06-26T08:37:15.2127835Z timm_vision_transformer PASS 2024-06-26T08:37:15.2132532Z timm_vision_transformer_large XFAIL 2024-06-26T08:37:15.2137024Z timm_vovnet PASS 2024-06-26T08:37:15.2141968Z torch_multimodal_clip PASS 2024-06-26T08:37:15.2146397Z tts_angular PASS 2024-06-26T08:37:15.2150876Z vgg16 PASS 2024-06-26T08:37:15.2155367Z vision_maskrcnn PASS 2024-06-26T08:37:15.2159855Z yolov3 PASS 2024-06-26T08:37:15.2614899Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamo_eager_torchbench_training.csv 2024-06-26T08:37:15.5746701Z llava PASS 2024-06-26T08:37:15.5750333Z maml_omniglot PASS 2024-06-26T08:37:15.5755353Z mnasnet1_0 PASS 2024-06-26T08:37:15.5760104Z mobilenet_v2 PASS 2024-06-26T08:37:15.5764787Z mobilenet_v2_quantized_qat PASS 2024-06-26T08:37:15.5769947Z mobilenet_v3_large PASS 2024-06-26T08:37:15.5775630Z moco PASS 2024-06-26T08:37:15.5781683Z nanogpt PASS 2024-06-26T08:37:15.5786944Z nvidia_deeprecommender PASS 2024-06-26T08:37:15.5792393Z opacus_cifar10 PASS 2024-06-26T08:37:15.5797651Z phlippe_densenet PASS 2024-06-26T08:37:15.5802586Z phlippe_resnet PASS 2024-06-26T08:37:15.5807510Z pytorch_CycleGAN_and_pix2pix PASS 2024-06-26T08:37:15.5812224Z pytorch_stargan PASS 2024-06-26T08:37:15.5816917Z pytorch_unet PASS 2024-06-26T08:37:15.5821792Z resnet152 PASS 2024-06-26T08:37:15.5826370Z resnet18 PASS 2024-06-26T08:37:15.5831188Z resnet50 PASS 2024-06-26T08:37:15.5835570Z resnet50_quantized_qat PASS 2024-06-26T08:37:15.5840292Z resnext50_32x4d PASS 2024-06-26T08:37:15.5844733Z sam PASS 2024-06-26T08:37:15.5849646Z shufflenet_v2_x1_0 PASS 2024-06-26T08:37:15.5854790Z soft_actor_critic PASS 2024-06-26T08:37:15.5860088Z speech_transformer PASS 2024-06-26T08:37:15.5864857Z squeezenet1_1 PASS 2024-06-26T08:37:15.5869650Z stable_diffusion_text_encoder PASS 2024-06-26T08:37:15.5874662Z stable_diffusion_unet PASS 2024-06-26T08:37:15.5879414Z timm_efficientnet PASS 2024-06-26T08:37:15.5884386Z timm_regnet PASS 2024-06-26T08:37:15.5889379Z timm_resnest PASS 2024-06-26T08:37:15.5894548Z timm_vision_transformer PASS 2024-06-26T08:37:15.5899779Z timm_vision_transformer_large PASS 2024-06-26T08:37:15.5904683Z timm_vovnet PASS 2024-06-26T08:37:15.5909579Z torch_multimodal_clip PASS 2024-06-26T08:37:15.5914573Z tts_angular PASS 2024-06-26T08:37:15.5919388Z vgg16 PASS 2024-06-26T08:37:15.5924299Z vision_maskrcnn PASS 2024-06-26T08:37:15.5928939Z yolov3 PASS 2024-06-26T08:37:15.6385917Z + cleanup_workspace 2024-06-26T08:37:15.6386885Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-26T08:37:15.6387979Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-26T08:37:15.6388902Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-26T08:37:15.6389735Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T08:37:15.6390806Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-26T08:37:15.6393839Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-26T08:37:15.6394563Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-26T08:37:16.7195750Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T08:37:16.7196280Z cat test/**/*_toprint.log || true 2024-06-26T08:37:16.7204146Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:37:16.7204646Z env: 2024-06-26T08:37:16.7204932Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:37:16.7205405Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:37:16.7206147Z DOCKER_CONTAINER_ID: fe8dad447a4030c1c55cf9472d5e553b966e0b723306a0c787595c538bab5320 2024-06-26T08:37:16.7206787Z ##[endgroup] 2024-06-26T08:37:16.7269399Z cat: test/**/*_toprint.log: No such file or directory 2024-06-26T08:37:16.7461249Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:37:16.7461727Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:37:16.7469277Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:37:16.7469783Z env: 2024-06-26T08:37:16.7470067Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:37:16.7470529Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:37:16.7471261Z DOCKER_CONTAINER_ID: fe8dad447a4030c1c55cf9472d5e553b966e0b723306a0c787595c538bab5320 2024-06-26T08:37:16.7471920Z MONITOR_SCRIPT_PID: 7617 2024-06-26T08:37:16.7472273Z ##[endgroup] 2024-06-26T08:37:16.7665670Z Prepare all required actions 2024-06-26T08:37:16.7666120Z Getting action download info 2024-06-26T08:37:16.9237407Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T08:37:17.2332940Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T08:37:17.2333415Z with: 2024-06-26T08:37:17.2333969Z file-suffix: test-dynamo_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688915598 2024-06-26T08:37:17.2334654Z s3-bucket: gha-artifacts 2024-06-26T08:37:17.2335002Z env: 2024-06-26T08:37:17.2335287Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:37:17.2335745Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:37:17.2336542Z DOCKER_CONTAINER_ID: fe8dad447a4030c1c55cf9472d5e553b966e0b723306a0c787595c538bab5320 2024-06-26T08:37:17.2337181Z ##[endgroup] 2024-06-26T08:37:17.2361522Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-26T08:37:17.2362173Z # Remove any previous test jsons if they exist 2024-06-26T08:37:17.2362701Z rm -f test-jsons-*.zip 2024-06-26T08:37:17.2363400Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-26T08:37:17.2371642Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:37:17.2372150Z env: 2024-06-26T08:37:17.2372438Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:37:17.2372897Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:37:17.2373688Z DOCKER_CONTAINER_ID: fe8dad447a4030c1c55cf9472d5e553b966e0b723306a0c787595c538bab5320 2024-06-26T08:37:17.2374599Z FILE_SUFFIX: test-dynamo_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688915598 2024-06-26T08:37:17.2375268Z ##[endgroup] 2024-06-26T08:37:17.2546118Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-26T08:37:17.2575009Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-26T08:37:17.2575861Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-26T08:37:17.2581898Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-26T08:37:17.2623184Z ##[group]Run # Remove any previous test reports if they exist 2024-06-26T08:37:17.2623837Z # Remove any previous test reports if they exist 2024-06-26T08:37:17.2624374Z rm -f test-reports-*.zip 2024-06-26T08:37:17.2624972Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-26T08:37:17.2632515Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:37:17.2633014Z env: 2024-06-26T08:37:17.2633285Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:37:17.2633741Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:37:17.2634471Z DOCKER_CONTAINER_ID: fe8dad447a4030c1c55cf9472d5e553b966e0b723306a0c787595c538bab5320 2024-06-26T08:37:17.2635381Z FILE_SUFFIX: test-dynamo_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688915598 2024-06-26T08:37:17.2636031Z ##[endgroup] 2024-06-26T08:37:17.2791411Z adding: test/test-reports/inference_torchbench.csv (deflated 69%) 2024-06-26T08:37:17.2793051Z adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 93%) 2024-06-26T08:37:17.2794390Z adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 74%) 2024-06-26T08:37:17.2795295Z adding: test/test-reports/training_torchbench.csv (deflated 66%) 2024-06-26T08:37:17.2801585Z adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 96%) 2024-06-26T08:37:17.2802826Z adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 79%) 2024-06-26T08:37:17.2828428Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-26T08:37:17.2829074Z # Remove any previous usage logs if they exist 2024-06-26T08:37:17.2829580Z rm -f logs-*.zip 2024-06-26T08:37:17.2830257Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-26T08:37:17.2831179Z # so check to see if the file exists first 2024-06-26T08:37:17.2831700Z if [ -f 'usage_log.txt' ]; then 2024-06-26T08:37:17.2832252Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-26T08:37:17.2832748Z fi 2024-06-26T08:37:17.2833144Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-26T08:37:17.2833757Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-26T08:37:17.2834250Z fi 2024-06-26T08:37:17.2841706Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:37:17.2842197Z env: 2024-06-26T08:37:17.2842474Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:37:17.2842936Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:37:17.2843725Z DOCKER_CONTAINER_ID: fe8dad447a4030c1c55cf9472d5e553b966e0b723306a0c787595c538bab5320 2024-06-26T08:37:17.2844631Z FILE_SUFFIX: test-dynamo_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688915598 2024-06-26T08:37:17.2845289Z ##[endgroup] 2024-06-26T08:37:17.2917060Z adding: usage_log.txt (deflated 92%) 2024-06-26T08:37:17.2982333Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-26T08:37:17.2983192Z # Remove any previous debugging artifacts if they exist 2024-06-26T08:37:17.2983747Z rm -f debug-*.zip 2024-06-26T08:37:17.2984144Z if [ -d 'test/debug' ]; then 2024-06-26T08:37:17.2984651Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-26T08:37:17.2985132Z fi 2024-06-26T08:37:17.2991700Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:37:17.2992213Z env: 2024-06-26T08:37:17.2992492Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:37:17.2992961Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:37:17.2993703Z DOCKER_CONTAINER_ID: fe8dad447a4030c1c55cf9472d5e553b966e0b723306a0c787595c538bab5320 2024-06-26T08:37:17.2994616Z FILE_SUFFIX: test-dynamo_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688915598 2024-06-26T08:37:17.2995271Z ##[endgroup] 2024-06-26T08:37:17.3067427Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:37:17.3067857Z with: 2024-06-26T08:37:17.3068142Z s3-bucket: gha-artifacts 2024-06-26T08:37:17.3068566Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:37:17.3069030Z retention-days: 14 2024-06-26T08:37:17.3069354Z if-no-files-found: warn 2024-06-26T08:37:17.3069705Z path: test-jsons-*.zip 2024-06-26T08:37:17.3070040Z name: artifact 2024-06-26T08:37:17.3070337Z region: us-east-1 2024-06-26T08:37:17.3070629Z env: 2024-06-26T08:37:17.3070907Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:37:17.3071355Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:37:17.3072077Z DOCKER_CONTAINER_ID: fe8dad447a4030c1c55cf9472d5e553b966e0b723306a0c787595c538bab5320 2024-06-26T08:37:17.3072718Z ##[endgroup] 2024-06-26T08:37:17.6091863Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:37:17.6092661Z With the provided path, there will be 1 file uploaded 2024-06-26T08:37:17.6093541Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:37:17.6118049Z Starting upload of test-jsons-test-dynamo_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688915598.zip 2024-06-26T08:37:17.7464539Z Finished upload of test-jsons-test-dynamo_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688915598.zip 2024-06-26T08:37:17.7600551Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:37:17.7600996Z with: 2024-06-26T08:37:17.7601288Z s3-bucket: gha-artifacts 2024-06-26T08:37:17.7601712Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:37:17.7602180Z retention-days: 14 2024-06-26T08:37:17.7602512Z if-no-files-found: error 2024-06-26T08:37:17.7602878Z path: test-reports-*.zip 2024-06-26T08:37:17.7603227Z name: artifact 2024-06-26T08:37:17.7603524Z region: us-east-1 2024-06-26T08:37:17.7603821Z env: 2024-06-26T08:37:17.7604094Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:37:17.7604547Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:37:17.7605409Z DOCKER_CONTAINER_ID: fe8dad447a4030c1c55cf9472d5e553b966e0b723306a0c787595c538bab5320 2024-06-26T08:37:17.7606062Z ##[endgroup] 2024-06-26T08:37:18.0672155Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:37:18.0672952Z With the provided path, there will be 1 file uploaded 2024-06-26T08:37:18.0673657Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:37:18.0698803Z Starting upload of test-reports-test-dynamo_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688915598.zip 2024-06-26T08:37:18.1584086Z Finished upload of test-reports-test-dynamo_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688915598.zip 2024-06-26T08:37:18.1722191Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:37:18.1722641Z with: 2024-06-26T08:37:18.1722937Z s3-bucket: gha-artifacts 2024-06-26T08:37:18.1723391Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:37:18.1723897Z retention-days: 14 2024-06-26T08:37:18.1724248Z if-no-files-found: ignore 2024-06-26T08:37:18.1724618Z path: logs-*.zip 2024-06-26T08:37:18.1724936Z name: artifact 2024-06-26T08:37:18.1725400Z region: us-east-1 2024-06-26T08:37:18.1725716Z env: 2024-06-26T08:37:18.1725998Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:37:18.1726457Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:37:18.1727187Z DOCKER_CONTAINER_ID: fe8dad447a4030c1c55cf9472d5e553b966e0b723306a0c787595c538bab5320 2024-06-26T08:37:18.1727822Z ##[endgroup] 2024-06-26T08:37:18.4849898Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:37:18.4850583Z With the provided path, there will be 1 file uploaded 2024-06-26T08:37:18.4851210Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:37:18.4875695Z Starting upload of logs-test-dynamo_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688915598.zip 2024-06-26T08:37:18.5890262Z Finished upload of logs-test-dynamo_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688915598.zip 2024-06-26T08:37:18.6028669Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:37:18.6029111Z with: 2024-06-26T08:37:18.6029415Z s3-bucket: gha-artifacts 2024-06-26T08:37:18.6029846Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:37:18.6030318Z retention-days: 14 2024-06-26T08:37:18.6030652Z if-no-files-found: ignore 2024-06-26T08:37:18.6031015Z path: debug-*.zip 2024-06-26T08:37:18.6031333Z name: artifact 2024-06-26T08:37:18.6031634Z region: us-east-1 2024-06-26T08:37:18.6031937Z env: 2024-06-26T08:37:18.6032216Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:37:18.6032671Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:37:18.6033399Z DOCKER_CONTAINER_ID: fe8dad447a4030c1c55cf9472d5e553b966e0b723306a0c787595c538bab5320 2024-06-26T08:37:18.6034035Z ##[endgroup] 2024-06-26T08:37:18.9000343Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-26T08:37:18.9143013Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T08:37:18.9143508Z # shellcheck disable=SC2156 2024-06-26T08:37:18.9144361Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T08:37:18.9152183Z shell: /usr/bin/bash -e {0} 2024-06-26T08:37:18.9152544Z env: 2024-06-26T08:37:18.9152828Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:37:18.9153292Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:37:18.9154073Z DOCKER_CONTAINER_ID: fe8dad447a4030c1c55cf9472d5e553b966e0b723306a0c787595c538bab5320 2024-06-26T08:37:18.9154714Z ##[endgroup] 2024-06-26T08:37:19.0804975Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-26T08:37:19.0805534Z with: 2024-06-26T08:37:19.0805791Z env: 2024-06-26T08:37:19.0806065Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:37:19.0806526Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:37:19.0807255Z DOCKER_CONTAINER_ID: fe8dad447a4030c1c55cf9472d5e553b966e0b723306a0c787595c538bab5320 2024-06-26T08:37:19.0807883Z ##[endgroup] 2024-06-26T08:37:19.0826550Z ##[group]Run set -eou pipefail 2024-06-26T08:37:19.0827139Z set -eou pipefail 2024-06-26T08:37:19.0827612Z  2024-06-26T08:37:19.0828134Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-26T08:37:19.0828767Z for _ in $(seq 1440); do 2024-06-26T08:37:19.0829252Z  # Break if no ssh session exists anymore 2024-06-26T08:37:19.0829755Z  if [ "$(who)" = "" ]; then 2024-06-26T08:37:19.0830173Z  break 2024-06-26T08:37:19.0830533Z  fi 2024-06-26T08:37:19.0830837Z  echo "." 2024-06-26T08:37:19.0831174Z  sleep 5 2024-06-26T08:37:19.0831500Z done 2024-06-26T08:37:19.0838872Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:37:19.0839364Z env: 2024-06-26T08:37:19.0839637Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:37:19.0840091Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:37:19.0840869Z DOCKER_CONTAINER_ID: fe8dad447a4030c1c55cf9472d5e553b966e0b723306a0c787595c538bab5320 2024-06-26T08:37:19.0841690Z ##[endgroup] 2024-06-26T08:37:19.0866725Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-26T08:37:19.0903844Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:37:19.0904613Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:37:19.0905204Z # shellcheck disable=SC2046 2024-06-26T08:37:19.0905692Z docker stop $(docker ps -q) || true 2024-06-26T08:37:19.0906173Z # Prune all of the docker images 2024-06-26T08:37:19.0906632Z docker system prune -af 2024-06-26T08:37:19.0917376Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:37:19.0917877Z env: 2024-06-26T08:37:19.0918149Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:37:19.0918605Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:37:19.0919335Z DOCKER_CONTAINER_ID: fe8dad447a4030c1c55cf9472d5e553b966e0b723306a0c787595c538bab5320 2024-06-26T08:37:19.0919980Z ##[endgroup] 2024-06-26T08:37:19.4540734Z fe8dad447a40 2024-06-26T08:37:21.7394127Z Deleted Containers: 2024-06-26T08:37:21.7394750Z fe8dad447a4030c1c55cf9472d5e553b966e0b723306a0c787595c538bab5320 2024-06-26T08:37:21.7395200Z 2024-06-26T08:37:25.6143885Z Deleted Images: 2024-06-26T08:37:25.6145787Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T08:37:25.6148440Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T08:37:25.6150558Z deleted: sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd 2024-06-26T08:37:25.6151726Z deleted: sha256:754ead7012f0b7691885a9660f02e80be6cee8c5665486fb995c1fb06adb6e92 2024-06-26T08:37:25.6152879Z deleted: sha256:fbe4496d8830e11be86d8f222522e7956f32f85bfc06b05ea53961f34a58af74 2024-06-26T08:37:25.6153730Z deleted: sha256:1c83814aa1a1a7e0cc998e3479bdab8e8766b975560cbf00690acf39ed899a80 2024-06-26T08:37:25.6154569Z deleted: sha256:e9ea8e61942c7f8ae4aab97b8d102b01d6bb4b6cf2a605808cd34130a17d8a0f 2024-06-26T08:37:25.6155414Z deleted: sha256:acc5fc32588cd95e082a6ce35e48d0f303ae8f96e3ab65c4e375b85e1f3c4a92 2024-06-26T08:37:25.6156298Z deleted: sha256:a93e53400904d29e4185d3be942ea34f19572ea438962788ae4e9b3d9bf4c498 2024-06-26T08:37:25.6157359Z deleted: sha256:c97092b88b5e4c77c68bbc4b51afab64e38719af1ff58013ec49a696cdd40a6e 2024-06-26T08:37:25.6158189Z deleted: sha256:b132889c90d67ffa53655846603e3fcc2372c23f04dbdbc54690a39d304eb89b 2024-06-26T08:37:25.6159001Z deleted: sha256:915d54081ed607710624abe3f2490114c745858831f4fc13396bd3b4b9a121bc 2024-06-26T08:37:25.6159806Z deleted: sha256:bfa8d446c498c87f535aef102695564de1b22c755e6c63944ae2f5073a2b31ec 2024-06-26T08:37:25.6160630Z deleted: sha256:e71460fb363c29e70bbe357d680aaa7ae694f094ef2f2713609158ad90611744 2024-06-26T08:37:25.6161451Z deleted: sha256:84079bb1b4ee1d0119ae3d3e701de460a1324912297c675425c9a36d30fcdb0f 2024-06-26T08:37:25.6162264Z deleted: sha256:d86ac3ca148d4758cb03b493fdd49109d8f1a2c7c85c4b755493c3c4529553e5 2024-06-26T08:37:25.6163091Z deleted: sha256:532b3ab97c44a673a0d5f62ee46fa6a83248a84de8877754a612355be56ed0a3 2024-06-26T08:37:25.6163917Z deleted: sha256:617e2a35f25ae24500c88c0c9b8475f0687fbcbed3d200013b4c99c6d8941aff 2024-06-26T08:37:25.6164748Z deleted: sha256:b88120bcd5a26a660c09c5df13176e0ef098fb2c8cdb57d74ab0796a3d1194d6 2024-06-26T08:37:25.6165567Z deleted: sha256:c016b2e7764735243205fb76fa9a273bb3dd4bf05532288e3a613ce022131ac3 2024-06-26T08:37:25.6166402Z deleted: sha256:bcaca462c390deddd2995de2d245a7495e7d2e1e94f0edad30a573c83d0588fd 2024-06-26T08:37:25.6167236Z deleted: sha256:9e260eba003d5c25d7b410e5c4c3093845d9e46611896840901c689d7fbf8ee6 2024-06-26T08:37:25.6168270Z deleted: sha256:3e62b86f8b911a9b59c62d55f063e71683ae6a01c22418db5d27930712adaad6 2024-06-26T08:37:25.6169440Z deleted: sha256:905920ccb1d88449b8dde686c34a4b2a8b6f59e78ac44916b189ea3593b9fbbb 2024-06-26T08:37:25.6170261Z deleted: sha256:961201237e03acd5061406238dd7a3c1343ac888d7583212cd0ab5ae1b4f0540 2024-06-26T08:37:25.6171177Z deleted: sha256:70047a2aac62159fa28fb4dbd4567ec2ffbab68fe046967e09acb0eb3e5cb9cb 2024-06-26T08:37:25.6172030Z deleted: sha256:1ec680ad781436ae65fa987f5fa1ed032ecd5c82a6e35eea6707de501f62c0f4 2024-06-26T08:37:25.6172874Z deleted: sha256:ef859623a50e85abf6e6c1dbd6188aa765a66cc7b2bd618a1cbe4825b7b2bd91 2024-06-26T08:37:25.6173724Z deleted: sha256:404ade5cacff59773b93ab438bfa64cd3a171d07f6f168bdc4a5d4bfa0b43526 2024-06-26T08:37:25.6174569Z deleted: sha256:5b27de6ee3d9f2f18217bf18dec725f7b4954c0e48c85d35752634c22d62209b 2024-06-26T08:37:25.6175406Z deleted: sha256:498e39c575b3fa069acaac7e46b91b4e54ae3a05f2ada5c4347091f346f25d4c 2024-06-26T08:37:25.6176332Z deleted: sha256:1405919d2694d65669b1f4c842282021afe89208ce86570e96464903733da2d0 2024-06-26T08:37:25.6177131Z deleted: sha256:d5b41e5d70896c0472fc3e6da0516a17d89cd6635dc75010ecddc31ce9690fad 2024-06-26T08:37:25.6177964Z deleted: sha256:18e9379b62b9d0bb40293d28a652eafaee66a499f66f0776bf14e37a513101b7 2024-06-26T08:37:25.6178797Z deleted: sha256:a781f80ed00ff2926a699fd01a309bb0b4fb7d5655eb652c205bb9c7f811035f 2024-06-26T08:37:25.6179614Z deleted: sha256:c0430973b7cf759075330ee29c79bfd1312c91e6033d9dd883b4b1c29743a171 2024-06-26T08:37:25.6180437Z deleted: sha256:e733b3d66efb9342e5a283376c0e1ddffa47a42fecc26e61ecaf0a3472885d97 2024-06-26T08:37:25.6181268Z deleted: sha256:8929c0b2047f74b84c82c39eff9077a92ae1a828cde00c9ab78fb3e0ed5e0283 2024-06-26T08:37:25.6182084Z deleted: sha256:f5b56155fb58a0a60fe6391d3581cf91610ed69d4f0a321257c19944da296a52 2024-06-26T08:37:25.6182973Z deleted: sha256:b54d2cb355238d1aa91e58df93064447b80f53706d1ca62823acc834f5ac54fe 2024-06-26T08:37:25.6183981Z deleted: sha256:e7311285af9eeddb51aa5629aa98386e89f1456ff32843b6aee6443247edf66c 2024-06-26T08:37:25.6184817Z deleted: sha256:67657ceec347da8dc8085d5731633efeac115ba677fa4b7c5966c4260f7abab0 2024-06-26T08:37:25.6185660Z deleted: sha256:b496e50409dbf414930fc9b4437aa0aa4ce27bc7840e050ae459aaffcf9aca86 2024-06-26T08:37:25.6186526Z deleted: sha256:8f5276170cc4a60d36ad7b964a502306b29546254cedcbcb101ea9123497d2c0 2024-06-26T08:37:25.6187330Z deleted: sha256:53d524d1564745e96a7e515462171b73184d9f1ff5454b32fcfbbca39d361467 2024-06-26T08:37:25.6188163Z deleted: sha256:2acbadb00d69b6aca575f04baba4ea26ae2b5c2d85f09945004ad93a78a8eb75 2024-06-26T08:37:25.6189139Z deleted: sha256:02665efeff878aeb603c1edb2e21cb700c171c025d64f5aae1b2b20b12f3964a 2024-06-26T08:37:25.6189987Z deleted: sha256:19bd23e9a51dfe883a8b8d2ff6d66ff1b3476b8912856406bb79b81762aa6d4b 2024-06-26T08:37:25.6190810Z deleted: sha256:73aa8637bf9299d6298dabd858480c274a62117388f2ca1d5e9eb15c6d1e4155 2024-06-26T08:37:25.6191630Z deleted: sha256:5f8968406305f348b1caebc0e035187d6e9965ace7b9d5f2f21114d37c9fe2e1 2024-06-26T08:37:25.6192453Z deleted: sha256:d827104914db0c503cf5c81f13e2456bf4efce42d4cf1f8d5d70838207278bee 2024-06-26T08:37:25.6193289Z deleted: sha256:867b219724f2f1aaf56f42e3e9f03a5d64bdca8a6a72407efd10328aeea5d510 2024-06-26T08:37:25.6194373Z deleted: sha256:b9ff2a7cceb9237b26bcb17728edf8a558eec5a76f63626e84f80541dfc0f1d4 2024-06-26T08:37:25.6195205Z deleted: sha256:be73f509106819007ce2b0f5a1301ac56ae583cb6c1fb69bee93efa0840368f1 2024-06-26T08:37:25.6196045Z deleted: sha256:2a11c07c915c3fdbb4e73096dea73ea950c970d50fe1a5764060bd62b7b7cf2c 2024-06-26T08:37:25.6196880Z deleted: sha256:5e9989622e2abc1c72f8184343507b45d75e4ff971edb68b96cbcd75ae484ff5 2024-06-26T08:37:25.6197715Z deleted: sha256:05ce642bd9cf5a9205bff107df28b1d6cef7d5f6ab3ea944d28c032ec61270df 2024-06-26T08:37:25.6198532Z deleted: sha256:017702617f6c07b73f2088678f4710b794cfe897a213a5cf122a0e2519eae0ac 2024-06-26T08:37:25.6199344Z deleted: sha256:44b3b37bb1ca39dc5cf356de41a23a5c12f605a1992813e408087fb2b0ad6d1c 2024-06-26T08:37:25.6200162Z deleted: sha256:8c387982e9f27db357a1d149dc149cabc0bd04668ad6956387a5072682dada67 2024-06-26T08:37:25.6200985Z deleted: sha256:10d415dca4e3fbb88a747b40317f3c6333c041ba8f73855f67fb9fe981fae5ee 2024-06-26T08:37:25.6201894Z deleted: sha256:6c65469bf0f53dcd625faefe363b8f3136c08388a6cf64d9158e8576e4ee89b1 2024-06-26T08:37:25.6202728Z deleted: sha256:027db5d8b76deeefbd8c34b485ab7ff14ed2a3e6800425c1ce0d77c2e9fdd4ab 2024-06-26T08:37:25.6203558Z deleted: sha256:24941234ad28cf5c8174d7cf18326c10b4e17f2c2137672ea0322df461597dd7 2024-06-26T08:37:25.6204388Z deleted: sha256:240ea2eebfa8519d6c2d88f4c99e66ce4e038d000f93ce8d03726c01d81180ef 2024-06-26T08:37:25.6205211Z deleted: sha256:09e823cb391a69f6a195e3ca8a41624b5f8b34eee14c6eb36277de814a91966e 2024-06-26T08:37:25.6206066Z deleted: sha256:730479d695169a6593bf0ac16b377491dce65175a6d158e012422367d04577fd 2024-06-26T08:37:25.6206875Z deleted: sha256:b0a30f9f7bd5459ff662453b9346a7ba8ea8db1191528345c1953eab4aa2f54e 2024-06-26T08:37:25.6207702Z deleted: sha256:75cefa87effaa3ad361c74642839923101f0f9982f814f3c9e2ed4ad52f2a008 2024-06-26T08:37:25.6208658Z deleted: sha256:6f97db87ba9512e615e871c6982000602982e1e5a5be334a56e7292251e84e00 2024-06-26T08:37:25.6209499Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-06-26T08:37:25.6210328Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-06-26T08:37:25.6211138Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-06-26T08:37:25.6211960Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-06-26T08:37:25.6212788Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-06-26T08:37:25.6213610Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-06-26T08:37:25.6214446Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-06-26T08:37:25.6215291Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-06-26T08:37:25.6216264Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-06-26T08:37:25.6217114Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-06-26T08:37:25.6217947Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-06-26T08:37:25.6218442Z 2024-06-26T08:37:25.6218598Z Total reclaimed space: 41.17GB 2024-06-26T08:37:25.6280199Z Post job cleanup. 2024-06-26T08:37:25.6327743Z Post job cleanup. 2024-06-26T08:37:25.7138753Z [command]/usr/bin/git version 2024-06-26T08:37:25.7179145Z git version 2.40.1 2024-06-26T08:37:25.7218145Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/892b2e43-d8db-499a-8d18-fa1b056d082b' before making global git config changes 2024-06-26T08:37:25.7219633Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T08:37:25.7223612Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T08:37:25.7249431Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T08:37:25.7271246Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T08:37:25.7474553Z Entering 'android/libs/fbjni' 2024-06-26T08:37:25.7512950Z Entering 'third_party/FP16' 2024-06-26T08:37:25.7548911Z Entering 'third_party/FXdiv' 2024-06-26T08:37:25.7583709Z Entering 'third_party/NNPACK' 2024-06-26T08:37:25.7619182Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:37:25.7653961Z Entering 'third_party/XNNPACK' 2024-06-26T08:37:25.7701523Z Entering 'third_party/benchmark' 2024-06-26T08:37:25.7736966Z Entering 'third_party/cpp-httplib' 2024-06-26T08:37:25.7773116Z Entering 'third_party/cpuinfo' 2024-06-26T08:37:25.7807418Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:37:25.7842548Z Entering 'third_party/cutlass' 2024-06-26T08:37:25.7882134Z Entering 'third_party/eigen' 2024-06-26T08:37:25.7918963Z Entering 'third_party/fbgemm' 2024-06-26T08:37:25.7956180Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:37:25.7988172Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:37:25.8024994Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:37:25.8064634Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:37:25.8099965Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:37:25.8140368Z Entering 'third_party/flatbuffers' 2024-06-26T08:37:25.8175918Z Entering 'third_party/fmt' 2024-06-26T08:37:25.8211878Z Entering 'third_party/foxi' 2024-06-26T08:37:25.8247505Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:37:25.8282912Z Entering 'third_party/gloo' 2024-06-26T08:37:25.8319490Z Entering 'third_party/googletest' 2024-06-26T08:37:25.8356272Z Entering 'third_party/ideep' 2024-06-26T08:37:25.8388906Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:37:25.8428972Z Entering 'third_party/ittapi' 2024-06-26T08:37:25.8464432Z Entering 'third_party/kineto' 2024-06-26T08:37:25.8498049Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:37:25.8532414Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:37:25.8567375Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:37:25.8602515Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:37:25.8636059Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:37:25.8669518Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:37:25.8706249Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:37:25.8740144Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:37:25.8775198Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:37:25.8811068Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:37:25.8846870Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:37:25.8881055Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:37:25.8917590Z Entering 'third_party/mimalloc' 2024-06-26T08:37:25.8954288Z Entering 'third_party/nccl/nccl' 2024-06-26T08:37:25.8988735Z Entering 'third_party/nlohmann' 2024-06-26T08:37:25.9023951Z Entering 'third_party/onnx' 2024-06-26T08:37:25.9068339Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:37:25.9103048Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:37:25.9140273Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:37:25.9174644Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:37:25.9207747Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:37:25.9241835Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:37:25.9276550Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:37:25.9310015Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:37:25.9343376Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:37:25.9375336Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:37:25.9408429Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:37:25.9444572Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:37:25.9479416Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:37:25.9531397Z Entering 'third_party/pocketfft' 2024-06-26T08:37:25.9564952Z Entering 'third_party/protobuf' 2024-06-26T08:37:25.9601205Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:37:25.9636613Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:37:25.9670855Z Entering 'third_party/psimd' 2024-06-26T08:37:25.9703181Z Entering 'third_party/pthreadpool' 2024-06-26T08:37:25.9737794Z Entering 'third_party/pybind11' 2024-06-26T08:37:25.9771813Z Entering 'third_party/python-peachpy' 2024-06-26T08:37:25.9806955Z Entering 'third_party/sleef' 2024-06-26T08:37:25.9841453Z Entering 'third_party/tensorpipe' 2024-06-26T08:37:25.9876092Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:37:25.9910801Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:37:25.9944937Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:37:25.9981002Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:37:26.0012670Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:37:26.0067917Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T08:37:26.0086123Z http.https://github.com/.extraheader 2024-06-26T08:37:26.0094008Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T08:37:26.0118636Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T08:37:26.0326073Z Entering 'android/libs/fbjni' 2024-06-26T08:37:26.0348318Z http.https://github.com/.extraheader 2024-06-26T08:37:26.0372582Z Entering 'third_party/FP16' 2024-06-26T08:37:26.0393904Z http.https://github.com/.extraheader 2024-06-26T08:37:26.0416586Z Entering 'third_party/FXdiv' 2024-06-26T08:37:26.0437421Z http.https://github.com/.extraheader 2024-06-26T08:37:26.0459409Z Entering 'third_party/NNPACK' 2024-06-26T08:37:26.0480835Z http.https://github.com/.extraheader 2024-06-26T08:37:26.0504797Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:37:26.0525627Z http.https://github.com/.extraheader 2024-06-26T08:37:26.0549986Z Entering 'third_party/XNNPACK' 2024-06-26T08:37:26.0571794Z http.https://github.com/.extraheader 2024-06-26T08:37:26.0605849Z Entering 'third_party/benchmark' 2024-06-26T08:37:26.0629031Z http.https://github.com/.extraheader 2024-06-26T08:37:26.0650498Z Entering 'third_party/cpp-httplib' 2024-06-26T08:37:26.0672570Z http.https://github.com/.extraheader 2024-06-26T08:37:26.0695914Z Entering 'third_party/cpuinfo' 2024-06-26T08:37:26.0717513Z http.https://github.com/.extraheader 2024-06-26T08:37:26.0740108Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:37:26.0761440Z http.https://github.com/.extraheader 2024-06-26T08:37:26.0785039Z Entering 'third_party/cutlass' 2024-06-26T08:37:26.0806514Z http.https://github.com/.extraheader 2024-06-26T08:37:26.0835767Z Entering 'third_party/eigen' 2024-06-26T08:37:26.0856805Z http.https://github.com/.extraheader 2024-06-26T08:37:26.0880945Z Entering 'third_party/fbgemm' 2024-06-26T08:37:26.0902063Z http.https://github.com/.extraheader 2024-06-26T08:37:26.0930154Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:37:26.0950183Z http.https://github.com/.extraheader 2024-06-26T08:37:26.0974019Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:37:26.0995252Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1017495Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:37:26.1037653Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1065172Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:37:26.1085816Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1110103Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:37:26.1130649Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1154571Z Entering 'third_party/flatbuffers' 2024-06-26T08:37:26.1176872Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1202899Z Entering 'third_party/fmt' 2024-06-26T08:37:26.1225140Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1248340Z Entering 'third_party/foxi' 2024-06-26T08:37:26.1269795Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1293080Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:37:26.1314038Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1338091Z Entering 'third_party/gloo' 2024-06-26T08:37:26.1359440Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1382941Z Entering 'third_party/googletest' 2024-06-26T08:37:26.1404552Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1428309Z Entering 'third_party/ideep' 2024-06-26T08:37:26.1451084Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1474277Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:37:26.1495252Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1523061Z Entering 'third_party/ittapi' 2024-06-26T08:37:26.1546417Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1569803Z Entering 'third_party/kineto' 2024-06-26T08:37:26.1590971Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1614300Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:37:26.1635623Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1658030Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:37:26.1678408Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1704053Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:37:26.1725155Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1747418Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:37:26.1766704Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1789436Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:37:26.1811215Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1832711Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:37:26.1855227Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1879108Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:37:26.1900197Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1924160Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:37:26.1946062Z http.https://github.com/.extraheader 2024-06-26T08:37:26.1969627Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:37:26.1991128Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2015969Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:37:26.2036850Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2060519Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:37:26.2081017Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2104073Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:37:26.2124998Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2150404Z Entering 'third_party/mimalloc' 2024-06-26T08:37:26.2172235Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2196477Z Entering 'third_party/nccl/nccl' 2024-06-26T08:37:26.2218699Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2241470Z Entering 'third_party/nlohmann' 2024-06-26T08:37:26.2262955Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2285940Z Entering 'third_party/onnx' 2024-06-26T08:37:26.2307465Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2343196Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:37:26.2364241Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2386536Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:37:26.2408300Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2434214Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:37:26.2455891Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2482306Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:37:26.2504571Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2529598Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:37:26.2550157Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2572158Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:37:26.2593068Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2615787Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:37:26.2636061Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2658279Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:37:26.2678711Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2701480Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:37:26.2722468Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2745591Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:37:26.2766124Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2788661Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:37:26.2809297Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2833554Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:37:26.2854461Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2878652Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:37:26.2899546Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2937295Z Entering 'third_party/pocketfft' 2024-06-26T08:37:26.2959458Z http.https://github.com/.extraheader 2024-06-26T08:37:26.2982941Z Entering 'third_party/protobuf' 2024-06-26T08:37:26.3004380Z http.https://github.com/.extraheader 2024-06-26T08:37:26.3030714Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:37:26.3051685Z http.https://github.com/.extraheader 2024-06-26T08:37:26.3075596Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:37:26.3097184Z http.https://github.com/.extraheader 2024-06-26T08:37:26.3121273Z Entering 'third_party/psimd' 2024-06-26T08:37:26.3144541Z http.https://github.com/.extraheader 2024-06-26T08:37:26.3167376Z Entering 'third_party/pthreadpool' 2024-06-26T08:37:26.3189982Z http.https://github.com/.extraheader 2024-06-26T08:37:26.3214656Z Entering 'third_party/pybind11' 2024-06-26T08:37:26.3235969Z http.https://github.com/.extraheader 2024-06-26T08:37:26.3259731Z Entering 'third_party/python-peachpy' 2024-06-26T08:37:26.3280928Z http.https://github.com/.extraheader 2024-06-26T08:37:26.3303416Z Entering 'third_party/sleef' 2024-06-26T08:37:26.3324982Z http.https://github.com/.extraheader 2024-06-26T08:37:26.3348624Z Entering 'third_party/tensorpipe' 2024-06-26T08:37:26.3369954Z http.https://github.com/.extraheader 2024-06-26T08:37:26.3393341Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:37:26.3414345Z http.https://github.com/.extraheader 2024-06-26T08:37:26.3437662Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:37:26.3457758Z http.https://github.com/.extraheader 2024-06-26T08:37:26.3479105Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:37:26.3501409Z http.https://github.com/.extraheader 2024-06-26T08:37:26.3524601Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:37:26.3546717Z http.https://github.com/.extraheader 2024-06-26T08:37:26.3568203Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:37:26.3589537Z http.https://github.com/.extraheader 2024-06-26T08:37:26.3685822Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-26T08:37:26.3703527Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T08:37:26.3710185Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:37:26.3710716Z ##[endgroup] 2024-06-26T08:37:27.1590548Z Cleaning up orphan processes