2024-06-26T07:00:56.4831862Z Current runner version: '2.317.0' 2024-06-26T07:00:56.4837982Z Runner name: 'i-0bab72a3f8092178a' 2024-06-26T07:00:56.4838875Z Runner group name: 'Default' 2024-06-26T07:00:56.4839843Z Machine name: 'ip-10-0-43-34' 2024-06-26T07:00:56.4844718Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T07:00:56.4846705Z Actions: read 2024-06-26T07:00:56.4847363Z Attestations: read 2024-06-26T07:00:56.4847938Z Checks: read 2024-06-26T07:00:56.4848463Z Contents: read 2024-06-26T07:00:56.4849001Z Deployments: read 2024-06-26T07:00:56.4849577Z Discussions: read 2024-06-26T07:00:56.4850169Z Issues: read 2024-06-26T07:00:56.4850696Z Metadata: read 2024-06-26T07:00:56.4851404Z Packages: read 2024-06-26T07:00:56.4851945Z Pages: read 2024-06-26T07:00:56.4852506Z PullRequests: read 2024-06-26T07:00:56.4853081Z RepositoryProjects: read 2024-06-26T07:00:56.4853762Z SecurityEvents: read 2024-06-26T07:00:56.4854364Z Statuses: read 2024-06-26T07:00:56.4854903Z ##[endgroup] 2024-06-26T07:00:56.4858266Z Secret source: Actions 2024-06-26T07:00:56.4859112Z Prepare workflow directory 2024-06-26T07:00:56.8555231Z Prepare all required actions 2024-06-26T07:00:56.8763549Z Getting action download info 2024-06-26T07:00:57.0773285Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T07:00:57.3739964Z Download action repository 'pytorch/pytorch@main' (SHA:c422a9549d2b320b47a9391f6cbcf592b2876c88) 2024-06-26T07:01:00.0094401Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-26T07:01:00.1437542Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T07:01:00.3809330Z Getting action download info 2024-06-26T07:01:00.4930596Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T07:01:00.6606326Z Getting action download info 2024-06-26T07:01:00.7847245Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T07:01:00.9061272Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/129470 (b8c4c54d347aa776934c60784e35936878ef18dc) 2024-06-26T07:01:00.9063539Z ##[group] Inputs 2024-06-26T07:01:00.9064013Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:01:00.9072518Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:01:00.9081886Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:01:00.9083054Z sync-tag: 2024-06-26T07:01:00.9083849Z timeout-minutes: 240 2024-06-26T07:01:00.9084187Z use-gha: 2024-06-26T07:01:00.9084466Z dashboard-tag: 2024-06-26T07:01:00.9084786Z s3-bucket: gha-artifacts 2024-06-26T07:01:00.9085144Z aws-role-to-assume: 2024-06-26T07:01:00.9085475Z ##[endgroup] 2024-06-26T07:01:00.9086330Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:01:00.9644126Z A job started hook has been configured by the self-hosted runner administrator 2024-06-26T07:01:00.9779972Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T07:01:00.9790363Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:01:00.9790898Z ##[endgroup] 2024-06-26T07:01:01.9899808Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-26T07:01:01.9900368Z with: 2024-06-26T07:01:01.9901061Z github-secret: *** 2024-06-26T07:01:01.9902011Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-26T07:01:01.9903118Z activate-with-label: false 2024-06-26T07:01:01.9903490Z label: with-ssh 2024-06-26T07:01:01.9903817Z remove-existing-keys: true 2024-06-26T07:01:01.9904189Z fail-silently: true 2024-06-26T07:01:01.9904501Z env: 2024-06-26T07:01:01.9904791Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:01:01.9905140Z ##[endgroup] 2024-06-26T07:01:02.0755543Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-26T07:01:02.0756559Z ciflow reference detected, attempting to extract PR number 2024-06-26T07:01:02.3595379Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-06-26T07:01:02.4321875Z No SSH keys found for user pytorch-bot[bot] 2024-06-26T07:01:02.4322727Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T07:01:02.4962918Z No SSH keys found for user leslie-fang-intel 2024-06-26T07:01:02.5067870Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T07:01:02.5068421Z with: 2024-06-26T07:01:02.5068694Z submodules: recursive 2024-06-26T07:01:02.5069017Z fetch-depth: 0 2024-06-26T07:01:02.5069336Z env: 2024-06-26T07:01:02.5069624Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:01:02.5069952Z ##[endgroup] 2024-06-26T07:01:02.5259398Z ##[group]Run retry () { 2024-06-26T07:01:02.5259839Z retry () { 2024-06-26T07:01:02.5260426Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T07:01:02.5261049Z } 2024-06-26T07:01:02.5261385Z echo "${GITHUB_WORKSPACE}" 2024-06-26T07:01:02.5261882Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T07:01:02.5262402Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:01:02.5263007Z else 2024-06-26T07:01:02.5263380Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:01:02.5263897Z fi 2024-06-26T07:01:02.5264344Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T07:01:02.5272384Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:01:02.5273010Z env: 2024-06-26T07:01:02.5273373Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:01:02.5273790Z NO_SUDO: 2024-06-26T07:01:02.5274094Z ##[endgroup] 2024-06-26T07:01:02.5297050Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:01:04.8001523Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T07:01:04.8001960Z with: 2024-06-26T07:01:04.8002283Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:01:04.8002710Z fetch-depth: 0 2024-06-26T07:01:04.8003020Z submodules: recursive 2024-06-26T07:01:04.8003360Z quiet-checkout: true 2024-06-26T07:01:04.8003705Z repository: pytorch/pytorch 2024-06-26T07:01:04.8004175Z token: *** 2024-06-26T07:01:04.8004466Z ssh-strict: true 2024-06-26T07:01:04.8004792Z persist-credentials: true 2024-06-26T07:01:04.8005145Z clean: true 2024-06-26T07:01:04.8005472Z sparse-checkout-cone-mode: true 2024-06-26T07:01:04.8005858Z lfs: false 2024-06-26T07:01:04.8006159Z set-safe-directory: true 2024-06-26T07:01:04.8006492Z env: 2024-06-26T07:01:04.8006754Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:01:04.8007091Z ##[endgroup] 2024-06-26T07:01:04.8921966Z Syncing repository: pytorch/pytorch 2024-06-26T07:01:04.8923528Z ##[group]Getting Git version info 2024-06-26T07:01:04.8924301Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:01:04.8925138Z [command]/usr/bin/git version 2024-06-26T07:01:04.8925502Z git version 2.40.1 2024-06-26T07:01:04.8927167Z ##[endgroup] 2024-06-26T07:01:04.8938703Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/38470ba3-a581-4c97-8c47-3de38ef1366d' before making global git config changes 2024-06-26T07:01:04.8939929Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T07:01:04.8941096Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:01:04.8965584Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:01:04.8969471Z ##[group]Initializing the repository 2024-06-26T07:01:04.8972796Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:01:04.8992184Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T07:01:04.8993209Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T07:01:04.8994006Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T07:01:04.8994577Z hint: 2024-06-26T07:01:04.8995022Z hint: git config --global init.defaultBranch 2024-06-26T07:01:04.8995493Z hint: 2024-06-26T07:01:04.8995999Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T07:01:04.8996841Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T07:01:04.8997588Z hint: 2024-06-26T07:01:04.8997925Z hint: git branch -m 2024-06-26T07:01:04.8998650Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T07:01:04.9001468Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T07:01:04.9021703Z ##[endgroup] 2024-06-26T07:01:04.9022290Z ##[group]Disabling automatic garbage collection 2024-06-26T07:01:04.9024689Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T07:01:04.9043484Z ##[endgroup] 2024-06-26T07:01:04.9044008Z ##[group]Setting up auth 2024-06-26T07:01:04.9049358Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T07:01:04.9070023Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T07:01:04.9261846Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T07:01:04.9283035Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T07:01:04.9476263Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:01:04.9517048Z ##[endgroup] 2024-06-26T07:01:04.9517607Z ##[group]Fetching the repository 2024-06-26T07:01:04.9522826Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T07:01:07.1231494Z remote: Enumerating objects: 987612 2024-06-26T07:01:07.1232111Z remote: Enumerating objects: 990209, done. 2024-06-26T07:01:07.1237026Z remote: Counting objects: 0% (1/2597) 2024-06-26T07:01:07.1239924Z remote: Counting objects: 1% (26/2597) 2024-06-26T07:01:07.1240541Z remote: Counting objects: 2% (52/2597) 2024-06-26T07:01:07.1241302Z remote: Counting objects: 3% (78/2597) 2024-06-26T07:01:07.1242288Z remote: Counting objects: 4% (104/2597) 2024-06-26T07:01:07.1242888Z remote: Counting objects: 5% (130/2597) 2024-06-26T07:01:07.1243402Z remote: Counting objects: 6% (156/2597) 2024-06-26T07:01:07.1244124Z remote: Counting objects: 7% (182/2597) 2024-06-26T07:01:07.1244629Z remote: Counting objects: 8% (208/2597) 2024-06-26T07:01:07.1245495Z remote: Counting objects: 9% (234/2597) 2024-06-26T07:01:07.1246101Z remote: Counting objects: 10% (260/2597) 2024-06-26T07:01:07.1246602Z remote: Counting objects: 11% (286/2597) 2024-06-26T07:01:07.1247109Z remote: Counting objects: 12% (312/2597) 2024-06-26T07:01:07.1249652Z remote: Counting objects: 13% (338/2597) 2024-06-26T07:01:07.1250260Z remote: Counting objects: 14% (364/2597) 2024-06-26T07:01:07.1250768Z remote: Counting objects: 15% (390/2597) 2024-06-26T07:01:07.1251538Z remote: Counting objects: 16% (416/2597) 2024-06-26T07:01:07.1252221Z remote: Counting objects: 17% (442/2597) 2024-06-26T07:01:07.1252752Z remote: Counting objects: 18% (468/2597) 2024-06-26T07:01:07.1253249Z remote: Counting objects: 19% (494/2597) 2024-06-26T07:01:07.1254311Z remote: Counting objects: 20% (520/2597) 2024-06-26T07:01:07.1254871Z remote: Counting objects: 21% (546/2597) 2024-06-26T07:01:07.1255367Z remote: Counting objects: 22% (572/2597) 2024-06-26T07:01:07.1255859Z remote: Counting objects: 23% (598/2597) 2024-06-26T07:01:07.1256340Z remote: Counting objects: 24% (624/2597) 2024-06-26T07:01:07.1259269Z remote: Counting objects: 25% (650/2597) 2024-06-26T07:01:07.1259899Z remote: Counting objects: 26% (676/2597) 2024-06-26T07:01:07.1260403Z remote: Counting objects: 27% (702/2597) 2024-06-26T07:01:07.1261161Z remote: Counting objects: 28% (728/2597) 2024-06-26T07:01:07.1261664Z remote: Counting objects: 29% (754/2597) 2024-06-26T07:01:07.1262250Z remote: Counting objects: 30% (780/2597) 2024-06-26T07:01:07.1263343Z remote: Counting objects: 31% (806/2597) 2024-06-26T07:01:07.1263839Z remote: Counting objects: 32% (832/2597) 2024-06-26T07:01:07.1264344Z remote: Counting objects: 33% (858/2597) 2024-06-26T07:01:07.1264941Z remote: Counting objects: 34% (883/2597) 2024-06-26T07:01:07.1265567Z remote: Counting objects: 35% (909/2597) 2024-06-26T07:01:07.1267876Z remote: Counting objects: 36% (935/2597) 2024-06-26T07:01:07.1268453Z remote: Counting objects: 37% (961/2597) 2024-06-26T07:01:07.1268945Z remote: Counting objects: 38% (987/2597) 2024-06-26T07:01:07.1269451Z remote: Counting objects: 39% (1013/2597) 2024-06-26T07:01:07.1270114Z remote: Counting objects: 40% (1039/2597) 2024-06-26T07:01:07.1270727Z remote: Counting objects: 41% (1065/2597) 2024-06-26T07:01:07.1271241Z remote: Counting objects: 42% (1091/2597) 2024-06-26T07:01:07.1272805Z remote: Counting objects: 43% (1117/2597) 2024-06-26T07:01:07.1273382Z remote: Counting objects: 44% (1143/2597) 2024-06-26T07:01:07.1273889Z remote: Counting objects: 45% (1169/2597) 2024-06-26T07:01:07.1274924Z remote: Counting objects: 46% (1195/2597) 2024-06-26T07:01:07.1275464Z remote: Counting objects: 47% (1221/2597) 2024-06-26T07:01:07.1277656Z remote: Counting objects: 48% (1247/2597) 2024-06-26T07:01:07.1278235Z remote: Counting objects: 49% (1273/2597) 2024-06-26T07:01:07.1278737Z remote: Counting objects: 50% (1299/2597) 2024-06-26T07:01:07.1279250Z remote: Counting objects: 51% (1325/2597) 2024-06-26T07:01:07.1280194Z remote: Counting objects: 52% (1351/2597) 2024-06-26T07:01:07.1280785Z remote: Counting objects: 53% (1377/2597) 2024-06-26T07:01:07.1281285Z remote: Counting objects: 54% (1403/2597) 2024-06-26T07:01:07.1282451Z remote: Counting objects: 55% (1429/2597) 2024-06-26T07:01:07.1283067Z remote: Counting objects: 56% (1455/2597) 2024-06-26T07:01:07.1283583Z remote: Counting objects: 57% (1481/2597) 2024-06-26T07:01:07.1284086Z remote: Counting objects: 58% (1507/2597) 2024-06-26T07:01:07.1284608Z remote: Counting objects: 59% (1533/2597) 2024-06-26T07:01:07.1285115Z remote: Counting objects: 60% (1559/2597) 2024-06-26T07:01:07.1285992Z remote: Counting objects: 61% (1585/2597) 2024-06-26T07:01:07.1286641Z remote: Counting objects: 62% (1611/2597) 2024-06-26T07:01:07.1287287Z remote: Counting objects: 63% (1637/2597) 2024-06-26T07:01:07.1287799Z remote: Counting objects: 64% (1663/2597) 2024-06-26T07:01:07.1288305Z remote: Counting objects: 65% (1689/2597) 2024-06-26T07:01:07.1288815Z remote: Counting objects: 66% (1715/2597) 2024-06-26T07:01:07.1289324Z remote: Counting objects: 67% (1740/2597) 2024-06-26T07:01:07.1289914Z remote: Counting objects: 68% (1766/2597) 2024-06-26T07:01:07.1290613Z remote: Counting objects: 69% (1792/2597) 2024-06-26T07:01:07.1291310Z remote: Counting objects: 70% (1818/2597) 2024-06-26T07:01:07.1291831Z remote: Counting objects: 71% (1844/2597) 2024-06-26T07:01:07.1292335Z remote: Counting objects: 72% (1870/2597) 2024-06-26T07:01:07.1292974Z remote: Counting objects: 73% (1896/2597) 2024-06-26T07:01:07.1293593Z remote: Counting objects: 74% (1922/2597) 2024-06-26T07:01:07.1294108Z remote: Counting objects: 75% (1948/2597) 2024-06-26T07:01:07.1294628Z remote: Counting objects: 76% (1974/2597) 2024-06-26T07:01:07.1295130Z remote: Counting objects: 77% (2000/2597) 2024-06-26T07:01:07.1295637Z remote: Counting objects: 78% (2026/2597) 2024-06-26T07:01:07.1296260Z remote: Counting objects: 79% (2052/2597) 2024-06-26T07:01:07.1296771Z remote: Counting objects: 80% (2078/2597) 2024-06-26T07:01:07.1297279Z remote: Counting objects: 81% (2104/2597) 2024-06-26T07:01:07.1297786Z remote: Counting objects: 82% (2130/2597) 2024-06-26T07:01:07.1298284Z remote: Counting objects: 83% (2156/2597) 2024-06-26T07:01:07.1298800Z remote: Counting objects: 84% (2182/2597) 2024-06-26T07:01:07.1299304Z remote: Counting objects: 85% (2208/2597) 2024-06-26T07:01:07.1299810Z remote: Counting objects: 86% (2234/2597) 2024-06-26T07:01:07.1300307Z remote: Counting objects: 87% (2260/2597) 2024-06-26T07:01:07.1300822Z remote: Counting objects: 88% (2286/2597) 2024-06-26T07:01:07.1301326Z remote: Counting objects: 89% (2312/2597) 2024-06-26T07:01:07.1301824Z remote: Counting objects: 90% (2338/2597) 2024-06-26T07:01:07.1302329Z remote: Counting objects: 91% (2364/2597) 2024-06-26T07:01:07.1302974Z remote: Counting objects: 92% (2390/2597) 2024-06-26T07:01:07.1303486Z remote: Counting objects: 93% (2416/2597) 2024-06-26T07:01:07.1303984Z remote: Counting objects: 94% (2442/2597) 2024-06-26T07:01:07.1304503Z remote: Counting objects: 95% (2468/2597) 2024-06-26T07:01:07.1305002Z remote: Counting objects: 96% (2494/2597) 2024-06-26T07:01:07.1305591Z remote: Counting objects: 97% (2520/2597) 2024-06-26T07:01:07.1306099Z remote: Counting objects: 98% (2546/2597) 2024-06-26T07:01:07.1306595Z remote: Counting objects: 99% (2572/2597) 2024-06-26T07:01:07.1307103Z remote: Counting objects: 100% (2597/2597) 2024-06-26T07:01:07.1307651Z remote: Counting objects: 100% (2597/2597), done. 2024-06-26T07:01:07.1411538Z remote: Compressing objects: 0% (1/1074) 2024-06-26T07:01:07.1774891Z remote: Compressing objects: 1% (11/1074) 2024-06-26T07:01:07.2167489Z remote: Compressing objects: 2% (22/1074) 2024-06-26T07:01:07.2492962Z remote: Compressing objects: 3% (33/1074) 2024-06-26T07:01:07.2688612Z remote: Compressing objects: 4% (43/1074) 2024-06-26T07:01:07.2919275Z remote: Compressing objects: 5% (54/1074) 2024-06-26T07:01:07.3096815Z remote: Compressing objects: 6% (65/1074) 2024-06-26T07:01:07.3438084Z remote: Compressing objects: 7% (76/1074) 2024-06-26T07:01:07.4180521Z remote: Compressing objects: 8% (86/1074) 2024-06-26T07:01:07.4848751Z remote: Compressing objects: 9% (97/1074) 2024-06-26T07:01:07.5198988Z remote: Compressing objects: 10% (108/1074) 2024-06-26T07:01:07.5224829Z remote: Compressing objects: 11% (119/1074) 2024-06-26T07:01:07.5314578Z remote: Compressing objects: 12% (129/1074) 2024-06-26T07:01:07.5381893Z remote: Compressing objects: 13% (140/1074) 2024-06-26T07:01:07.5429464Z remote: Compressing objects: 14% (151/1074) 2024-06-26T07:01:07.5430076Z remote: Compressing objects: 15% (162/1074) 2024-06-26T07:01:07.5488490Z remote: Compressing objects: 16% (172/1074) 2024-06-26T07:01:07.5489100Z remote: Compressing objects: 17% (183/1074) 2024-06-26T07:01:07.5489639Z remote: Compressing objects: 18% (194/1074) 2024-06-26T07:01:07.5502105Z remote: Compressing objects: 19% (205/1074) 2024-06-26T07:01:07.5517221Z remote: Compressing objects: 20% (215/1074) 2024-06-26T07:01:07.5535843Z remote: Compressing objects: 21% (226/1074) 2024-06-26T07:01:07.5549096Z remote: Compressing objects: 22% (237/1074) 2024-06-26T07:01:07.5571233Z remote: Compressing objects: 23% (248/1074) 2024-06-26T07:01:07.5590122Z remote: Compressing objects: 24% (258/1074) 2024-06-26T07:01:07.5602986Z remote: Compressing objects: 25% (269/1074) 2024-06-26T07:01:07.5603597Z remote: Compressing objects: 26% (280/1074) 2024-06-26T07:01:07.5604257Z remote: Compressing objects: 27% (290/1074) 2024-06-26T07:01:07.5617969Z remote: Compressing objects: 28% (301/1074) 2024-06-26T07:01:07.5618574Z remote: Compressing objects: 29% (312/1074) 2024-06-26T07:01:07.5619103Z remote: Compressing objects: 30% (323/1074) 2024-06-26T07:01:07.5634437Z remote: Compressing objects: 31% (333/1074) 2024-06-26T07:01:07.5635000Z remote: Compressing objects: 32% (344/1074) 2024-06-26T07:01:07.5644014Z remote: Compressing objects: 33% (355/1074) 2024-06-26T07:01:07.5658234Z remote: Compressing objects: 34% (366/1074) 2024-06-26T07:01:07.5671901Z remote: Compressing objects: 35% (376/1074) 2024-06-26T07:01:07.5677857Z remote: Compressing objects: 36% (387/1074) 2024-06-26T07:01:07.5690418Z remote: Compressing objects: 37% (398/1074) 2024-06-26T07:01:07.5706170Z remote: Compressing objects: 38% (409/1074) 2024-06-26T07:01:07.5721076Z remote: Compressing objects: 39% (419/1074) 2024-06-26T07:01:07.5738232Z remote: Compressing objects: 40% (430/1074) 2024-06-26T07:01:07.5751880Z remote: Compressing objects: 41% (441/1074) 2024-06-26T07:01:07.5765177Z remote: Compressing objects: 42% (452/1074) 2024-06-26T07:01:07.5778144Z remote: Compressing objects: 43% (462/1074) 2024-06-26T07:01:07.5780548Z remote: Compressing objects: 44% (473/1074) 2024-06-26T07:01:07.5783538Z remote: Compressing objects: 45% (484/1074) 2024-06-26T07:01:07.5790964Z remote: Compressing objects: 46% (495/1074) 2024-06-26T07:01:07.5803459Z remote: Compressing objects: 47% (505/1074) 2024-06-26T07:01:07.5828245Z remote: Compressing objects: 48% (516/1074) 2024-06-26T07:01:07.5830776Z remote: Compressing objects: 49% (527/1074) 2024-06-26T07:01:07.5832917Z remote: Compressing objects: 50% (537/1074) 2024-06-26T07:01:07.5836362Z remote: Compressing objects: 51% (548/1074) 2024-06-26T07:01:07.5842585Z remote: Compressing objects: 52% (559/1074) 2024-06-26T07:01:07.5853954Z remote: Compressing objects: 53% (570/1074) 2024-06-26T07:01:07.5871899Z remote: Compressing objects: 54% (580/1074) 2024-06-26T07:01:07.5885566Z remote: Compressing objects: 55% (591/1074) 2024-06-26T07:01:07.5898643Z remote: Compressing objects: 56% (602/1074) 2024-06-26T07:01:07.5909617Z remote: Compressing objects: 57% (613/1074) 2024-06-26T07:01:07.5915430Z remote: Compressing objects: 58% (623/1074) 2024-06-26T07:01:07.5932761Z remote: Compressing objects: 59% (634/1074) 2024-06-26T07:01:07.5937998Z remote: Compressing objects: 60% (645/1074) 2024-06-26T07:01:07.5952143Z remote: Compressing objects: 61% (656/1074) 2024-06-26T07:01:07.5957178Z remote: Compressing objects: 62% (666/1074) 2024-06-26T07:01:07.5963982Z remote: Compressing objects: 63% (677/1074) 2024-06-26T07:01:07.5973739Z remote: Compressing objects: 64% (688/1074) 2024-06-26T07:01:07.5977363Z remote: Compressing objects: 65% (699/1074) 2024-06-26T07:01:07.5979901Z remote: Compressing objects: 66% (709/1074) 2024-06-26T07:01:07.5983399Z remote: Compressing objects: 67% (720/1074) 2024-06-26T07:01:07.5987737Z remote: Compressing objects: 68% (731/1074) 2024-06-26T07:01:07.5991697Z remote: Compressing objects: 69% (742/1074) 2024-06-26T07:01:07.5994359Z remote: Compressing objects: 70% (752/1074) 2024-06-26T07:01:07.5997366Z remote: Compressing objects: 71% (763/1074) 2024-06-26T07:01:07.6003254Z remote: Compressing objects: 72% (774/1074) 2024-06-26T07:01:07.6006299Z remote: Compressing objects: 73% (785/1074) 2024-06-26T07:01:07.6006932Z remote: Compressing objects: 74% (795/1074) 2024-06-26T07:01:07.6007461Z remote: Compressing objects: 75% (806/1074) 2024-06-26T07:01:07.6007982Z remote: Compressing objects: 76% (817/1074) 2024-06-26T07:01:07.6008649Z remote: Compressing objects: 77% (827/1074) 2024-06-26T07:01:07.6009180Z remote: Compressing objects: 78% (838/1074) 2024-06-26T07:01:07.6011711Z remote: Compressing objects: 79% (849/1074) 2024-06-26T07:01:07.6014817Z remote: Compressing objects: 80% (860/1074) 2024-06-26T07:01:07.6015350Z remote: Compressing objects: 81% (870/1074) 2024-06-26T07:01:07.6018711Z remote: Compressing objects: 82% (881/1074) 2024-06-26T07:01:07.6019355Z remote: Compressing objects: 83% (892/1074) 2024-06-26T07:01:07.6019878Z remote: Compressing objects: 84% (903/1074) 2024-06-26T07:01:07.6023698Z remote: Compressing objects: 85% (913/1074) 2024-06-26T07:01:07.6027363Z remote: Compressing objects: 86% (924/1074) 2024-06-26T07:01:07.6029866Z remote: Compressing objects: 87% (935/1074) 2024-06-26T07:01:07.6032499Z remote: Compressing objects: 88% (946/1074) 2024-06-26T07:01:07.6033120Z remote: Compressing objects: 89% (956/1074) 2024-06-26T07:01:07.6033715Z remote: Compressing objects: 90% (967/1074) 2024-06-26T07:01:07.6034408Z remote: Compressing objects: 91% (978/1074) 2024-06-26T07:01:07.6035201Z remote: Compressing objects: 92% (989/1074) 2024-06-26T07:01:07.6035770Z remote: Compressing objects: 93% (999/1074) 2024-06-26T07:01:07.6036373Z remote: Compressing objects: 94% (1010/1074) 2024-06-26T07:01:07.6037193Z remote: Compressing objects: 95% (1021/1074) 2024-06-26T07:01:07.6037732Z remote: Compressing objects: 96% (1032/1074) 2024-06-26T07:01:07.6038351Z remote: Compressing objects: 97% (1042/1074) 2024-06-26T07:01:07.6038994Z remote: Compressing objects: 98% (1053/1074) 2024-06-26T07:01:07.6039777Z remote: Compressing objects: 99% (1064/1074) 2024-06-26T07:01:07.6040538Z remote: Compressing objects: 100% (1074/1074) 2024-06-26T07:01:07.6041117Z remote: Compressing objects: 100% (1074/1074), done. 2024-06-26T07:01:27.7234958Z remote: Total 990209 (delta 2062), reused 1863 (delta 1522), pack-reused 987612 2024-06-26T07:01:50.3942902Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T07:01:50.3964333Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:01:50.3967858Z ##[endgroup] 2024-06-26T07:01:50.3968400Z ##[group]Determining the checkout info 2024-06-26T07:01:50.3970471Z ##[endgroup] 2024-06-26T07:01:50.3971182Z ##[group]Checking out the ref 2024-06-26T07:01:50.3973271Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:01:51.6180011Z ##[endgroup] 2024-06-26T07:01:51.6180727Z ##[group]Setting up auth for fetching submodules 2024-06-26T07:01:51.6184450Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:01:51.6228005Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T07:01:51.6253821Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T07:01:51.6280345Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T07:01:51.6301549Z ##[endgroup] 2024-06-26T07:01:51.6302074Z ##[group]Fetching submodules 2024-06-26T07:01:51.6305619Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T07:01:51.6516254Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T07:01:51.6727832Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T07:01:51.6729590Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T07:01:51.6731586Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T07:01:51.6733436Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T07:01:51.6735401Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T07:01:51.6737140Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T07:01:51.6738649Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T07:01:51.6740617Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T07:01:51.6743020Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T07:01:51.6745378Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T07:01:51.6747526Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T07:01:51.6749871Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T07:01:51.6752294Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T07:01:51.6754970Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T07:01:51.6757364Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T07:01:51.6760030Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T07:01:51.6762873Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:01:51.6765467Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T07:01:51.6768353Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T07:01:51.6771434Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T07:01:51.6774809Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T07:01:51.6777755Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T07:01:51.6781000Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T07:01:51.6784277Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T07:01:51.6787561Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T07:01:51.6790736Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T07:01:51.6794408Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T07:01:51.6797599Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T07:01:51.6801116Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T07:01:51.6804610Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T07:01:51.6808643Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T07:01:51.6813363Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T07:01:51.6817316Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T07:01:51.6821223Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T07:01:51.6825612Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T07:01:51.6846839Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T07:01:51.9197450Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T07:01:52.0983176Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T07:01:52.2849423Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T07:01:52.5012837Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T07:01:54.5013043Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T07:02:04.2144290Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T07:02:04.6118628Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T07:02:05.1166280Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T07:02:05.7070100Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T07:02:06.8295266Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T07:02:08.5417072Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T07:02:14.3631121Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T07:02:15.6418998Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T07:02:17.2681091Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T07:02:18.4731345Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T07:02:18.7648105Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T07:02:19.1265859Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T07:02:19.4762773Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T07:02:20.4734789Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T07:02:20.8004655Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T07:02:21.0144041Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T07:02:22.4864820Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T07:02:23.1950624Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T07:02:23.9462570Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T07:02:29.5704813Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T07:02:31.9495693Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T07:02:37.4449632Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T07:02:37.6356123Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T07:02:45.6732296Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T07:02:45.8198858Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T07:02:46.1025239Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T07:02:47.0744736Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T07:02:47.3510966Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T07:02:47.9244656Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T07:02:48.3373443Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T07:02:48.3456365Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T07:02:48.3515740Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T07:02:48.3699148Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T07:02:48.3997849Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T07:02:49.1737787Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T07:02:49.1917837Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T07:02:49.2274905Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T07:02:49.3127220Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T07:02:49.3395750Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T07:02:49.7732185Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T07:02:49.9842815Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T07:02:50.0480422Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T07:02:50.0491326Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:02:50.0494079Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:02:50.0497159Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:02:50.0498667Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:02:50.0500331Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:02:50.0517284Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T07:02:50.9518365Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T07:02:51.5254451Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T07:02:53.1923847Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T07:02:54.1210070Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T07:02:54.4335223Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T07:02:54.5174382Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T07:02:54.8596519Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T07:02:54.9135928Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T07:02:54.9224292Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T07:02:55.0168378Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T07:02:55.0454296Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T07:02:55.0516005Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T07:02:55.0820248Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T07:02:55.1017804Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T07:02:55.1376370Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:02:55.1471238Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T07:02:55.1481753Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T07:02:55.1497001Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T07:03:06.7941489Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T07:03:06.8082510Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T07:03:06.8861352Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T07:03:06.8872600Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:03:06.8875139Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:03:06.8876854Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:03:06.8893489Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T07:03:07.3905968Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T07:03:08.6089752Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T07:03:09.6406827Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T07:03:09.6417489Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:03:09.6419166Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:03:09.6420842Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:03:09.6422875Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:03:09.6424861Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:03:09.6426942Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:03:09.6428862Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:03:09.6431122Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:03:09.6450964Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T07:03:10.6991926Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T07:03:11.0599097Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T07:03:12.2611192Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T07:03:12.5168498Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T07:03:12.9725941Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T07:03:13.9469640Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T07:03:19.5172603Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T07:03:19.8574178Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T07:03:19.8709806Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T07:03:19.9006737Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T07:03:19.9107450Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T07:03:19.9118244Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:03:19.9134318Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T07:03:20.1817077Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T07:03:20.1953425Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T07:03:20.2296029Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T07:03:20.3106687Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T07:03:20.3227477Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T07:03:20.3532229Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T07:03:20.4027258Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T07:03:20.4335950Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T07:03:20.4541602Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T07:03:20.5391922Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T07:03:20.8147391Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T07:03:20.8169634Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T07:03:20.8173042Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T07:03:20.8191683Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T07:03:21.1830912Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T07:03:22.1836597Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T07:03:22.2117924Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T07:03:22.2683238Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T07:03:22.2685196Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:03:22.2686813Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:03:22.2688624Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:03:22.2690368Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:03:22.2693072Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:03:22.2695046Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:03:22.2696906Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:03:22.2698613Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:03:22.2699996Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T07:03:22.6827660Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T07:03:23.8294659Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T07:03:24.2082040Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T07:03:29.8980987Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T07:03:30.1910423Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T07:03:30.4267561Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T07:03:30.7057080Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T07:03:38.1123951Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T07:03:38.1461666Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T07:03:38.1586409Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T07:03:38.2426870Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T07:03:38.2523848Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T07:03:38.2630795Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T07:03:38.2744622Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T07:03:38.2754419Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:03:38.2756254Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:03:38.2774329Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T07:03:40.0725253Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T07:03:41.2657398Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T07:03:41.3023271Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:03:41.6685820Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T07:03:41.6764248Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T07:03:41.8930284Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T07:03:41.8947664Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:03:41.8949407Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T07:03:41.8965716Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T07:03:42.2625203Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T07:03:43.1918964Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T07:03:43.2543669Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T07:03:43.2603466Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T07:03:43.2692554Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T07:03:43.2972348Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T07:03:43.3201445Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T07:03:43.3535078Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T07:03:43.3738460Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T07:03:43.3750932Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:03:43.3753342Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:03:43.3754872Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:03:43.3756710Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:03:43.3775188Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T07:03:44.3350728Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T07:03:44.5365619Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T07:03:45.5888037Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T07:03:46.6461167Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T07:03:46.6578188Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T07:03:46.7141833Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T07:03:46.7375261Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T07:03:46.7387066Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:03:46.7404790Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T07:03:46.9430300Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T07:03:46.9453785Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T07:03:46.9662849Z Entering 'android/libs/fbjni' 2024-06-26T07:03:46.9690071Z Entering 'third_party/FP16' 2024-06-26T07:03:46.9717932Z Entering 'third_party/FXdiv' 2024-06-26T07:03:46.9745948Z Entering 'third_party/NNPACK' 2024-06-26T07:03:46.9774151Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:03:46.9801598Z Entering 'third_party/XNNPACK' 2024-06-26T07:03:46.9840790Z Entering 'third_party/benchmark' 2024-06-26T07:03:46.9868893Z Entering 'third_party/cpp-httplib' 2024-06-26T07:03:46.9896972Z Entering 'third_party/cpuinfo' 2024-06-26T07:03:46.9926758Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:03:46.9953606Z Entering 'third_party/cutlass' 2024-06-26T07:03:46.9985203Z Entering 'third_party/eigen' 2024-06-26T07:03:47.0015001Z Entering 'third_party/fbgemm' 2024-06-26T07:03:47.0041370Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:03:47.0069131Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:03:47.0097119Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:03:47.0128376Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:03:47.0155611Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:03:47.0183434Z Entering 'third_party/flatbuffers' 2024-06-26T07:03:47.0213866Z Entering 'third_party/fmt' 2024-06-26T07:03:47.0242944Z Entering 'third_party/foxi' 2024-06-26T07:03:47.0270905Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:03:47.0300234Z Entering 'third_party/gloo' 2024-06-26T07:03:47.0330368Z Entering 'third_party/googletest' 2024-06-26T07:03:47.0360590Z Entering 'third_party/ideep' 2024-06-26T07:03:47.0388547Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:03:47.0422331Z Entering 'third_party/ittapi' 2024-06-26T07:03:47.0450338Z Entering 'third_party/kineto' 2024-06-26T07:03:47.0480884Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:03:47.0507821Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:03:47.0534137Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:03:47.0563768Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:03:47.0589850Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:03:47.0614726Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:03:47.0639936Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:03:47.0668030Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:03:47.0693127Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:03:47.0718349Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:03:47.0748393Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:03:47.0772771Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:03:47.0800635Z Entering 'third_party/mimalloc' 2024-06-26T07:03:47.0828829Z Entering 'third_party/nccl/nccl' 2024-06-26T07:03:47.0855862Z Entering 'third_party/nlohmann' 2024-06-26T07:03:47.0883373Z Entering 'third_party/onnx' 2024-06-26T07:03:47.0922942Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:03:47.0950706Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:03:47.0978582Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:03:47.1007592Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:03:47.1034945Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:03:47.1062834Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:03:47.1088336Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:03:47.1117674Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:03:47.1145731Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:03:47.1173460Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:03:47.1201184Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:03:47.1229602Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:03:47.1258502Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:03:47.1300277Z Entering 'third_party/pocketfft' 2024-06-26T07:03:47.1328310Z Entering 'third_party/protobuf' 2024-06-26T07:03:47.1358555Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:03:47.1386456Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:03:47.1415829Z Entering 'third_party/psimd' 2024-06-26T07:03:47.1442859Z Entering 'third_party/pthreadpool' 2024-06-26T07:03:47.1469774Z Entering 'third_party/pybind11' 2024-06-26T07:03:47.1495119Z Entering 'third_party/python-peachpy' 2024-06-26T07:03:47.1523884Z Entering 'third_party/sleef' 2024-06-26T07:03:47.1550821Z Entering 'third_party/tensorpipe' 2024-06-26T07:03:47.1580675Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:03:47.1607587Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:03:47.1637416Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:03:47.1665370Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:03:47.1690355Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:03:47.1732135Z ##[endgroup] 2024-06-26T07:03:47.1733982Z ##[group]Persisting credentials for submodules 2024-06-26T07:03:47.1737655Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T07:03:47.1946390Z Entering 'android/libs/fbjni' 2024-06-26T07:03:47.1983806Z Entering 'third_party/FP16' 2024-06-26T07:03:47.2019302Z Entering 'third_party/FXdiv' 2024-06-26T07:03:47.2053176Z Entering 'third_party/NNPACK' 2024-06-26T07:03:47.2087839Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:03:47.2121047Z Entering 'third_party/XNNPACK' 2024-06-26T07:03:47.2168687Z Entering 'third_party/benchmark' 2024-06-26T07:03:47.2203941Z Entering 'third_party/cpp-httplib' 2024-06-26T07:03:47.2240166Z Entering 'third_party/cpuinfo' 2024-06-26T07:03:47.2276774Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:03:47.2310666Z Entering 'third_party/cutlass' 2024-06-26T07:03:47.2352667Z Entering 'third_party/eigen' 2024-06-26T07:03:47.2390376Z Entering 'third_party/fbgemm' 2024-06-26T07:03:47.2423705Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:03:47.2459178Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:03:47.2494820Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:03:47.2535475Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:03:47.2568995Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:03:47.2603891Z Entering 'third_party/flatbuffers' 2024-06-26T07:03:47.2641891Z Entering 'third_party/fmt' 2024-06-26T07:03:47.2679095Z Entering 'third_party/foxi' 2024-06-26T07:03:47.2715582Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:03:47.2750607Z Entering 'third_party/gloo' 2024-06-26T07:03:47.2786751Z Entering 'third_party/googletest' 2024-06-26T07:03:47.2820270Z Entering 'third_party/ideep' 2024-06-26T07:03:47.2855549Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:03:47.2894601Z Entering 'third_party/ittapi' 2024-06-26T07:03:47.2928685Z Entering 'third_party/kineto' 2024-06-26T07:03:47.2963476Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:03:47.2998334Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:03:47.3034760Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:03:47.3068377Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:03:47.3104521Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:03:47.3135428Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:03:47.3172432Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:03:47.3205171Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:03:47.3241169Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:03:47.3276457Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:03:47.3310652Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:03:47.3346850Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:03:47.3381140Z Entering 'third_party/mimalloc' 2024-06-26T07:03:47.3416368Z Entering 'third_party/nccl/nccl' 2024-06-26T07:03:47.3451371Z Entering 'third_party/nlohmann' 2024-06-26T07:03:47.3488416Z Entering 'third_party/onnx' 2024-06-26T07:03:47.3535128Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:03:47.3569836Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:03:47.3606110Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:03:47.3643974Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:03:47.3678755Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:03:47.3712464Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:03:47.3747758Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:03:47.3779265Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:03:47.3815263Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:03:47.3848658Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:03:47.3882296Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:03:47.3917692Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:03:47.3952746Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:03:47.4003315Z Entering 'third_party/pocketfft' 2024-06-26T07:03:47.4040253Z Entering 'third_party/protobuf' 2024-06-26T07:03:47.4078791Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:03:47.4112680Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:03:47.4146902Z Entering 'third_party/psimd' 2024-06-26T07:03:47.4178692Z Entering 'third_party/pthreadpool' 2024-06-26T07:03:47.4213704Z Entering 'third_party/pybind11' 2024-06-26T07:03:47.4245501Z Entering 'third_party/python-peachpy' 2024-06-26T07:03:47.4279844Z Entering 'third_party/sleef' 2024-06-26T07:03:47.4314064Z Entering 'third_party/tensorpipe' 2024-06-26T07:03:47.4351304Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:03:47.4385361Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:03:47.4418511Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:03:47.4452919Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:03:47.4488097Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:03:47.4537445Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T07:03:47.4747237Z Entering 'android/libs/fbjni' 2024-06-26T07:03:47.4779871Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T07:03:47.4793226Z Entering 'third_party/FP16' 2024-06-26T07:03:47.4826593Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T07:03:47.4839708Z Entering 'third_party/FXdiv' 2024-06-26T07:03:47.4871733Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T07:03:47.4885859Z Entering 'third_party/NNPACK' 2024-06-26T07:03:47.4918656Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T07:03:47.4930355Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:03:47.4963565Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T07:03:47.4975783Z Entering 'third_party/XNNPACK' 2024-06-26T07:03:47.5007652Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T07:03:47.5030655Z Entering 'third_party/benchmark' 2024-06-26T07:03:47.5065176Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:03:47.5079067Z Entering 'third_party/cpp-httplib' 2024-06-26T07:03:47.5110481Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T07:03:47.5124115Z Entering 'third_party/cpuinfo' 2024-06-26T07:03:47.5155035Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:03:47.5168060Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:03:47.5202861Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T07:03:47.5217182Z Entering 'third_party/cutlass' 2024-06-26T07:03:47.5248294Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:03:47.5268350Z Entering 'third_party/eigen' 2024-06-26T07:03:47.5300040Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T07:03:47.5314490Z Entering 'third_party/fbgemm' 2024-06-26T07:03:47.5346515Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T07:03:47.5358669Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:03:47.5389233Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T07:03:47.5401761Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:03:47.5434316Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:03:47.5445338Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:03:47.5476886Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:03:47.5494723Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:03:47.5528115Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:03:47.5541234Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:03:47.5572868Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T07:03:47.5587987Z Entering 'third_party/flatbuffers' 2024-06-26T07:03:47.5620256Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T07:03:47.5635816Z Entering 'third_party/fmt' 2024-06-26T07:03:47.5666762Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:03:47.5680251Z Entering 'third_party/foxi' 2024-06-26T07:03:47.5713340Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T07:03:47.5726815Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:03:47.5759728Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T07:03:47.5774150Z Entering 'third_party/gloo' 2024-06-26T07:03:47.5807639Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T07:03:47.5821031Z Entering 'third_party/googletest' 2024-06-26T07:03:47.5853207Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:03:47.5866954Z Entering 'third_party/ideep' 2024-06-26T07:03:47.5899108Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T07:03:47.5911601Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:03:47.5943923Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T07:03:47.5961728Z Entering 'third_party/ittapi' 2024-06-26T07:03:47.5995521Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T07:03:47.6007474Z Entering 'third_party/kineto' 2024-06-26T07:03:47.6040575Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T07:03:47.6056488Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:03:47.6088622Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T07:03:47.6101181Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:03:47.6134754Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T07:03:47.6148573Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:03:47.6180552Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T07:03:47.6193702Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:03:47.6225678Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:03:47.6238624Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:03:47.6270466Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T07:03:47.6280672Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:03:47.6314712Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T07:03:47.6329118Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:03:47.6363612Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T07:03:47.6377996Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:03:47.6409532Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:03:47.6420790Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:03:47.6451763Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T07:03:47.6465492Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:03:47.6502946Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T07:03:47.6517066Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:03:47.6548491Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T07:03:47.6562028Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:03:47.6595035Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T07:03:47.6608299Z Entering 'third_party/mimalloc' 2024-06-26T07:03:47.6641965Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T07:03:47.6655461Z Entering 'third_party/nccl/nccl' 2024-06-26T07:03:47.6686550Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T07:03:47.6700853Z Entering 'third_party/nlohmann' 2024-06-26T07:03:47.6733301Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T07:03:47.6746889Z Entering 'third_party/onnx' 2024-06-26T07:03:47.6778627Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T07:03:47.6803768Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:03:47.6835467Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:03:47.6849193Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:03:47.6881202Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:03:47.6897511Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:03:47.6930129Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T07:03:47.6944608Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:03:47.6977417Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:03:47.6990006Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:03:47.7020725Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:03:47.7034023Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:03:47.7066999Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T07:03:47.7079081Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:03:47.7108979Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T07:03:47.7122719Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:03:47.7155798Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T07:03:47.7167226Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:03:47.7199059Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T07:03:47.7211068Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:03:47.7243474Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T07:03:47.7255071Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:03:47.7285949Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T07:03:47.7300551Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:03:47.7331993Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T07:03:47.7347029Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:03:47.7376828Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T07:03:47.7404721Z Entering 'third_party/pocketfft' 2024-06-26T07:03:47.7437733Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T07:03:47.7451505Z Entering 'third_party/protobuf' 2024-06-26T07:03:47.7483603Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T07:03:47.7500098Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:03:47.7530721Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:03:47.7545099Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:03:47.7576455Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:03:47.7591863Z Entering 'third_party/psimd' 2024-06-26T07:03:47.7623409Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T07:03:47.7635719Z Entering 'third_party/pthreadpool' 2024-06-26T07:03:47.7668562Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T07:03:47.7682579Z Entering 'third_party/pybind11' 2024-06-26T07:03:47.7715086Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:03:47.7726839Z Entering 'third_party/python-peachpy' 2024-06-26T07:03:47.7759344Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T07:03:47.7773291Z Entering 'third_party/sleef' 2024-06-26T07:03:47.7806355Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T07:03:47.7819698Z Entering 'third_party/tensorpipe' 2024-06-26T07:03:47.7851646Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T07:03:47.7864200Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:03:47.7893832Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:03:47.7906174Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:03:47.7936576Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T07:03:47.7948489Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:03:47.7979444Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T07:03:47.7993194Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:03:47.8024768Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:03:47.8037372Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:03:47.8069718Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T07:03:47.8917486Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T07:03:47.9122662Z Entering 'android/libs/fbjni' 2024-06-26T07:03:47.9149624Z Entering 'third_party/FP16' 2024-06-26T07:03:47.9176339Z Entering 'third_party/FXdiv' 2024-06-26T07:03:47.9202110Z Entering 'third_party/NNPACK' 2024-06-26T07:03:47.9228263Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:03:47.9255778Z Entering 'third_party/XNNPACK' 2024-06-26T07:03:47.9292014Z Entering 'third_party/benchmark' 2024-06-26T07:03:47.9320136Z Entering 'third_party/cpp-httplib' 2024-06-26T07:03:47.9348968Z Entering 'third_party/cpuinfo' 2024-06-26T07:03:47.9375133Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:03:47.9403726Z Entering 'third_party/cutlass' 2024-06-26T07:03:47.9438436Z Entering 'third_party/eigen' 2024-06-26T07:03:47.9468391Z Entering 'third_party/fbgemm' 2024-06-26T07:03:47.9497117Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:03:47.9523633Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:03:47.9552224Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:03:47.9583634Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:03:47.9611246Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:03:47.9639066Z Entering 'third_party/flatbuffers' 2024-06-26T07:03:47.9671340Z Entering 'third_party/fmt' 2024-06-26T07:03:47.9700390Z Entering 'third_party/foxi' 2024-06-26T07:03:47.9730231Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:03:47.9760150Z Entering 'third_party/gloo' 2024-06-26T07:03:47.9789106Z Entering 'third_party/googletest' 2024-06-26T07:03:47.9818255Z Entering 'third_party/ideep' 2024-06-26T07:03:47.9845954Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:03:47.9879967Z Entering 'third_party/ittapi' 2024-06-26T07:03:47.9907690Z Entering 'third_party/kineto' 2024-06-26T07:03:47.9937319Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:03:47.9966066Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:03:47.9997017Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:03:48.0024689Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:03:48.0051403Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:03:48.0080500Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:03:48.0108212Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:03:48.0137712Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:03:48.0166155Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:03:48.0195317Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:03:48.0222268Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:03:48.0249594Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:03:48.0280270Z Entering 'third_party/mimalloc' 2024-06-26T07:03:48.0309202Z Entering 'third_party/nccl/nccl' 2024-06-26T07:03:48.0338401Z Entering 'third_party/nlohmann' 2024-06-26T07:03:48.0367149Z Entering 'third_party/onnx' 2024-06-26T07:03:48.0405898Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:03:48.0436114Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:03:48.0465247Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:03:48.0495897Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:03:48.0524547Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:03:48.0552393Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:03:48.0579333Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:03:48.0607819Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:03:48.0633363Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:03:48.0658609Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:03:48.0686217Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:03:48.0716179Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:03:48.0745706Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:03:48.0789255Z Entering 'third_party/pocketfft' 2024-06-26T07:03:48.0818455Z Entering 'third_party/protobuf' 2024-06-26T07:03:48.0847358Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:03:48.0876503Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:03:48.0905800Z Entering 'third_party/psimd' 2024-06-26T07:03:48.0932251Z Entering 'third_party/pthreadpool' 2024-06-26T07:03:48.0960943Z Entering 'third_party/pybind11' 2024-06-26T07:03:48.0991136Z Entering 'third_party/python-peachpy' 2024-06-26T07:03:48.1020484Z Entering 'third_party/sleef' 2024-06-26T07:03:48.1050050Z Entering 'third_party/tensorpipe' 2024-06-26T07:03:48.1080153Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:03:48.1107566Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:03:48.1132174Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:03:48.1159596Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:03:48.1187585Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:03:48.1229187Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T07:03:48.1440680Z Entering 'android/libs/fbjni' 2024-06-26T07:03:48.1469177Z Entering 'third_party/FP16' 2024-06-26T07:03:48.1498339Z Entering 'third_party/FXdiv' 2024-06-26T07:03:48.1527461Z Entering 'third_party/NNPACK' 2024-06-26T07:03:48.1557467Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:03:48.1585793Z Entering 'third_party/XNNPACK' 2024-06-26T07:03:48.1624278Z Entering 'third_party/benchmark' 2024-06-26T07:03:48.1653637Z Entering 'third_party/cpp-httplib' 2024-06-26T07:03:48.1681554Z Entering 'third_party/cpuinfo' 2024-06-26T07:03:48.1709833Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:03:48.1739007Z Entering 'third_party/cutlass' 2024-06-26T07:03:48.1774347Z Entering 'third_party/eigen' 2024-06-26T07:03:48.1805944Z Entering 'third_party/fbgemm' 2024-06-26T07:03:48.1836011Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:03:48.1863830Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:03:48.1891940Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:03:48.1924905Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:03:48.1951657Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:03:48.1981840Z Entering 'third_party/flatbuffers' 2024-06-26T07:03:48.2012122Z Entering 'third_party/fmt' 2024-06-26T07:03:48.2041941Z Entering 'third_party/foxi' 2024-06-26T07:03:48.2069090Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:03:48.2099502Z Entering 'third_party/gloo' 2024-06-26T07:03:48.2126949Z Entering 'third_party/googletest' 2024-06-26T07:03:48.2155254Z Entering 'third_party/ideep' 2024-06-26T07:03:48.2181514Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:03:48.2214003Z Entering 'third_party/ittapi' 2024-06-26T07:03:48.2239558Z Entering 'third_party/kineto' 2024-06-26T07:03:48.2266815Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:03:48.2295589Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:03:48.2325037Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:03:48.2353062Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:03:48.2381366Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:03:48.2407790Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:03:48.2436803Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:03:48.2465385Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:03:48.2491574Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:03:48.2521000Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:03:48.2548541Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:03:48.2574239Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:03:48.2603113Z Entering 'third_party/mimalloc' 2024-06-26T07:03:48.2633884Z Entering 'third_party/nccl/nccl' 2024-06-26T07:03:48.2661475Z Entering 'third_party/nlohmann' 2024-06-26T07:03:48.2690505Z Entering 'third_party/onnx' 2024-06-26T07:03:48.2732828Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:03:48.2760721Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:03:48.2791839Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:03:48.2819820Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:03:48.2846385Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:03:48.2875364Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:03:48.2904031Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:03:48.2937595Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:03:48.2964426Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:03:48.2990755Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:03:48.3017294Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:03:48.3046369Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:03:48.3076381Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:03:48.3119166Z Entering 'third_party/pocketfft' 2024-06-26T07:03:48.3147479Z Entering 'third_party/protobuf' 2024-06-26T07:03:48.3175932Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:03:48.3204432Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:03:48.3234801Z Entering 'third_party/psimd' 2024-06-26T07:03:48.3264060Z Entering 'third_party/pthreadpool' 2024-06-26T07:03:48.3293874Z Entering 'third_party/pybind11' 2024-06-26T07:03:48.3323539Z Entering 'third_party/python-peachpy' 2024-06-26T07:03:48.3352280Z Entering 'third_party/sleef' 2024-06-26T07:03:48.3381301Z Entering 'third_party/tensorpipe' 2024-06-26T07:03:48.3409992Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:03:48.3438009Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:03:48.3464838Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:03:48.3490007Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:03:48.3516917Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:03:48.3554827Z ##[endgroup] 2024-06-26T07:03:48.3587143Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T07:03:48.3604445Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T07:03:48.3777677Z Prepare all required actions 2024-06-26T07:03:48.3778127Z Getting action download info 2024-06-26T07:03:48.5138418Z ##[group]Run ./.github/actions/setup-linux 2024-06-26T07:03:48.5138834Z env: 2024-06-26T07:03:48.5139113Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:48.5139458Z ##[endgroup] 2024-06-26T07:03:48.5197357Z ##[group]Run set -euo pipefail 2024-06-26T07:03:48.5197802Z set -euo pipefail 2024-06-26T07:03:48.5198191Z function get_ec2_metadata() { 2024-06-26T07:03:48.5198715Z  # Pulled from instance metadata endpoint for EC2 2024-06-26T07:03:48.5199583Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-26T07:03:48.5200339Z  category=$1 2024-06-26T07:03:48.5200871Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-26T07:03:48.5201504Z  runner_name_str=i-0bab72a3f8092178a 2024-06-26T07:03:48.5201991Z  if [[ -f /.inarc ]]; then 2024-06-26T07:03:48.5202484Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-26T07:03:48.5203048Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-26T07:03:48.5203723Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-26T07:03:48.5204321Z  else 2024-06-26T07:03:48.5204800Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-26T07:03:48.5205363Z  fi 2024-06-26T07:03:48.5205647Z } 2024-06-26T07:03:48.5205995Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-26T07:03:48.5206573Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-26T07:03:48.5207215Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-26T07:03:48.5207780Z echo "system info $(uname -a)" 2024-06-26T07:03:48.5215756Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:48.5216243Z env: 2024-06-26T07:03:48.5216527Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:48.5216877Z ##[endgroup] 2024-06-26T07:03:48.5284398Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-26T07:03:48.5323719Z instance-id: i-0bab72a3f8092178a 2024-06-26T07:03:48.5361857Z instance-type: g5.4xlarge 2024-06-26T07:03:48.5368230Z system info Linux ip-10-0-43-34.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-26T07:03:48.5387278Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:03:48.5388188Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:03:48.5396149Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:48.5396749Z env: 2024-06-26T07:03:48.5397015Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:48.5397362Z ##[endgroup] 2024-06-26T07:03:48.5459050Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-26T07:03:48.5459670Z if systemctl is-active --quiet docker; then 2024-06-26T07:03:48.5460208Z  echo "Docker daemon is running..."; 2024-06-26T07:03:48.5460667Z else 2024-06-26T07:03:48.5461160Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-26T07:03:48.5461746Z fi 2024-06-26T07:03:48.5469398Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:48.5469909Z env: 2024-06-26T07:03:48.5470180Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:48.5470539Z ##[endgroup] 2024-06-26T07:03:48.5506096Z Docker daemon is running... 2024-06-26T07:03:48.5549175Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:03:48.5549739Z with: 2024-06-26T07:03:48.5550007Z shell: bash 2024-06-26T07:03:48.5550318Z timeout_minutes: 5 2024-06-26T07:03:48.5550634Z max_attempts: 3 2024-06-26T07:03:48.5550950Z retry_wait_seconds: 30 2024-06-26T07:03:48.5552445Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-26T07:03:48.5553869Z polling_interval_seconds: 1 2024-06-26T07:03:48.5554246Z warning_on_retry: true 2024-06-26T07:03:48.5554586Z continue_on_error: false 2024-06-26T07:03:48.5554921Z env: 2024-06-26T07:03:48.5555193Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:48.5555536Z AWS_RETRY_MODE: standard 2024-06-26T07:03:48.5555884Z AWS_MAX_ATTEMPTS: 5 2024-06-26T07:03:48.5556223Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:03:48.5556589Z ##[endgroup] 2024-06-26T07:03:49.3981821Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:03:49.3982947Z Configure a credential helper to remove this warning. See 2024-06-26T07:03:49.3983778Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:03:49.3984387Z 2024-06-26T07:03:49.3984634Z Login Succeeded 2024-06-26T07:03:49.6046134Z Command completed after 1 attempt(s). 2024-06-26T07:03:49.6091257Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:03:49.6091980Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:03:49.6092630Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:03:49.6100400Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:49.6100889Z env: 2024-06-26T07:03:49.6101157Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:49.6101514Z ##[endgroup] 2024-06-26T07:03:49.6160718Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:03:49.6161482Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:03:49.6162097Z # shellcheck disable=SC2046 2024-06-26T07:03:49.6162764Z docker stop $(docker ps -q) || true 2024-06-26T07:03:49.6163331Z # Prune all of the docker images 2024-06-26T07:03:49.6163775Z docker system prune -af 2024-06-26T07:03:49.6171360Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:49.6171866Z env: 2024-06-26T07:03:49.6172144Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:49.6172488Z ##[endgroup] 2024-06-26T07:03:49.6418749Z "docker stop" requires at least 1 argument. 2024-06-26T07:03:49.6419402Z See 'docker stop --help'. 2024-06-26T07:03:49.6419646Z 2024-06-26T07:03:49.6419869Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T07:03:49.6420302Z 2024-06-26T07:03:49.6420453Z Stop one or more running containers 2024-06-26T07:03:49.6567825Z Total reclaimed space: 0B 2024-06-26T07:03:49.6597407Z ##[group]Run set +e 2024-06-26T07:03:49.6597777Z set +e 2024-06-26T07:03:49.6598185Z set -x 2024-06-26T07:03:49.6598487Z  2024-06-26T07:03:49.6598813Z PT_DOMAIN=download.pytorch.org 2024-06-26T07:03:49.6599599Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-26T07:03:49.6600716Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-26T07:03:49.6601468Z # one is returned at random 2024-06-26T07:03:49.6602006Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-26T07:03:49.6602514Z  2024-06-26T07:03:49.6602823Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:03:49.6603446Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-26T07:03:49.6604206Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-26T07:03:49.6604758Z  2024-06-26T07:03:49.6605080Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:03:49.6605637Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-26T07:03:49.6606144Z  exit 1 2024-06-26T07:03:49.6606461Z  fi 2024-06-26T07:03:49.6606745Z fi 2024-06-26T07:03:49.6607129Z  2024-06-26T07:03:49.6607489Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-26T07:03:49.6607995Z  # Clean up any old records first 2024-06-26T07:03:49.6608507Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-26T07:03:49.6608962Z fi 2024-06-26T07:03:49.6609244Z  2024-06-26T07:03:49.6609669Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-26T07:03:49.6610214Z cat /etc/hosts 2024-06-26T07:03:49.6618120Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:49.6618613Z env: 2024-06-26T07:03:49.6618878Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:49.6619219Z ##[endgroup] 2024-06-26T07:03:49.6638244Z + PT_DOMAIN=download.pytorch.org 2024-06-26T07:03:49.6642038Z ++ dig -4 +short download.pytorch.org 2024-06-26T07:03:49.6642752Z ++ tail -n1 2024-06-26T07:03:49.7126392Z + RESOLVED_IP=18.160.10.22 2024-06-26T07:03:49.7126838Z + '[' -z 18.160.10.22 ']' 2024-06-26T07:03:49.7127283Z + grep -r download.pytorch.org /etc/hosts 2024-06-26T07:03:49.7132788Z 18.160.10.22 download.pytorch.org 2024-06-26T07:03:49.7133885Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-06-26T07:03:49.7224421Z + echo '18.160.10.22 download.pytorch.org' 2024-06-26T07:03:49.7224938Z + sudo tee -a /etc/hosts 2024-06-26T07:03:49.7606424Z 18.160.10.22 download.pytorch.org 2024-06-26T07:03:49.7619837Z + cat /etc/hosts 2024-06-26T07:03:49.7623858Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-26T07:03:49.7636527Z ::1 localhost6 localhost6.localdomain6 2024-06-26T07:03:49.7637027Z 18.160.10.22 download.pytorch.org 2024-06-26T07:03:49.7842170Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T07:03:49.7842848Z with: 2024-06-26T07:03:49.7843929Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:49.7845135Z docker-build-dir: .ci/docker 2024-06-26T07:03:49.7845511Z working-directory: . 2024-06-26T07:03:49.7845975Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:49.7846491Z force-push: false 2024-06-26T07:03:49.7846796Z env: 2024-06-26T07:03:49.7847063Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:49.7847390Z ##[endgroup] 2024-06-26T07:03:49.7867385Z ##[group]Run set -ex 2024-06-26T07:03:49.7867735Z set -ex 2024-06-26T07:03:49.7868017Z  2024-06-26T07:03:49.7868559Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T07:03:49.7869508Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T07:03:49.7870410Z # job could then download the pre-built image as usual 2024-06-26T07:03:49.7871145Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T07:03:49.7871806Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:03:49.7872431Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:03:49.7872974Z  2024-06-26T07:03:49.7873470Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T07:03:49.7874081Z  exit 0 2024-06-26T07:03:49.7874371Z else 2024-06-26T07:03:49.7874729Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T07:03:49.7875170Z fi 2024-06-26T07:03:49.7875439Z  2024-06-26T07:03:49.7875904Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T07:03:49.7876750Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T07:03:49.7877524Z  # use it as it is, but first let's extract the tag 2024-06-26T07:03:49.7878213Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T07:03:49.7878927Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:03:49.7879625Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:03:49.7880182Z else 2024-06-26T07:03:49.7880616Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T07:03:49.7881264Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:03:49.7882135Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:03:49.7882866Z fi 2024-06-26T07:03:49.7890470Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:49.7891216Z env: 2024-06-26T07:03:49.7891491Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:49.7891829Z REPO_NAME: pytorch 2024-06-26T07:03:49.7892941Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:49.7894138Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:03:49.7894624Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:49.7895137Z ##[endgroup] 2024-06-26T07:03:49.7915487Z + [[ ! -d .ci/docker ]] 2024-06-26T07:03:49.7915891Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T07:03:49.7916261Z + echo skip=false 2024-06-26T07:03:49.7918183Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T07:03:49.7920925Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:49.7922105Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T07:03:49.7926770Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:49.7927415Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:49.7929087Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:49.7953944Z ##[group]Run set +e 2024-06-26T07:03:49.7954303Z set +e 2024-06-26T07:03:49.7954605Z set -x 2024-06-26T07:03:49.7954886Z  2024-06-26T07:03:49.7955158Z login() { 2024-06-26T07:03:49.7955818Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:03:49.7956526Z } 2024-06-26T07:03:49.7956797Z  2024-06-26T07:03:49.7957168Z retry () { 2024-06-26T07:03:49.7957557Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:03:49.7958001Z } 2024-06-26T07:03:49.7958270Z  2024-06-26T07:03:49.7958584Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:03:49.7959000Z  2024-06-26T07:03:49.7959455Z # Check if image already exists, if it does then skip building it 2024-06-26T07:03:49.7960153Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T07:03:49.7960652Z  exit 0 2024-06-26T07:03:49.7960952Z fi 2024-06-26T07:03:49.7961220Z  2024-06-26T07:03:49.7961715Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T07:03:49.7962532Z # be empty. The default action would be to continue rebuild the image 2024-06-26T07:03:49.7963264Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T07:03:49.7963923Z  # if we're on the base branch then use the parent commit 2024-06-26T07:03:49.7964517Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T07:03:49.7964944Z else 2024-06-26T07:03:49.7965409Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T07:03:49.7966084Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T07:03:49.7966582Z fi 2024-06-26T07:03:49.7966859Z  2024-06-26T07:03:49.7967175Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T07:03:49.7967661Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:03:49.7968113Z  2024-06-26T07:03:49.7968756Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T07:03:49.7969507Z  exit 0 2024-06-26T07:03:49.7969798Z fi 2024-06-26T07:03:49.7970072Z  2024-06-26T07:03:49.7970522Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T07:03:49.7971715Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T07:03:49.7972500Z  exit 1 2024-06-26T07:03:49.7972802Z fi 2024-06-26T07:03:49.7973079Z  2024-06-26T07:03:49.7973561Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T07:03:49.7974496Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T07:03:49.7975361Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T07:03:49.7976308Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T07:03:49.7977388Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T07:03:49.7978161Z fi 2024-06-26T07:03:49.7978438Z  2024-06-26T07:03:49.7978800Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:03:49.7985713Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:49.7986210Z env: 2024-06-26T07:03:49.7986478Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:49.7986840Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:03:49.7987301Z BASE_REVISION: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:03:49.7988547Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:49.7989780Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:49.7990425Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:49.7990938Z ##[endgroup] 2024-06-26T07:03:49.8010474Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:49.8011330Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:49.8011918Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:03:49.8012808Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:50.2107554Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:03:50.2108359Z Configure a credential helper to remove this warning. See 2024-06-26T07:03:50.2109176Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:03:50.2109693Z 2024-06-26T07:03:50.2109810Z Login Succeeded 2024-06-26T07:03:50.2120637Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:50.4161077Z { 2024-06-26T07:03:50.4162030Z "schemaVersion": 2, 2024-06-26T07:03:50.4162874Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T07:03:50.4163667Z "config": { 2024-06-26T07:03:50.4164154Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T07:03:50.4164811Z "size": 48681, 2024-06-26T07:03:50.4165500Z "digest": "sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd" 2024-06-26T07:03:50.4166298Z }, 2024-06-26T07:03:50.4166606Z "layers": [ 2024-06-26T07:03:50.4166960Z { 2024-06-26T07:03:50.4167549Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4168294Z "size": 28580681, 2024-06-26T07:03:50.4169042Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-06-26T07:03:50.4169845Z }, 2024-06-26T07:03:50.4170172Z { 2024-06-26T07:03:50.4170734Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4171730Z "size": 7943451, 2024-06-26T07:03:50.4172438Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-06-26T07:03:50.4173247Z }, 2024-06-26T07:03:50.4173680Z { 2024-06-26T07:03:50.4174143Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4174708Z "size": 55728572, 2024-06-26T07:03:50.4175137Z + exit 0 2024-06-26T07:03:50.4175635Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-06-26T07:03:50.4176230Z }, 2024-06-26T07:03:50.4176471Z { 2024-06-26T07:03:50.4176894Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4177430Z "size": 186, 2024-06-26T07:03:50.4177968Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-06-26T07:03:50.4178595Z }, 2024-06-26T07:03:50.4178836Z { 2024-06-26T07:03:50.4179258Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4179801Z "size": 6886, 2024-06-26T07:03:50.4180368Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-06-26T07:03:50.4181281Z }, 2024-06-26T07:03:50.4181537Z { 2024-06-26T07:03:50.4181962Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4182518Z "size": 1329001756, 2024-06-26T07:03:50.4183193Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-06-26T07:03:50.4183818Z }, 2024-06-26T07:03:50.4184061Z { 2024-06-26T07:03:50.4184476Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4185022Z "size": 62501, 2024-06-26T07:03:50.4185561Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-06-26T07:03:50.4186163Z }, 2024-06-26T07:03:50.4186399Z { 2024-06-26T07:03:50.4186820Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4187362Z "size": 1684, 2024-06-26T07:03:50.4187888Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-06-26T07:03:50.4188483Z }, 2024-06-26T07:03:50.4188733Z { 2024-06-26T07:03:50.4189157Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4189815Z "size": 1523, 2024-06-26T07:03:50.4190359Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-06-26T07:03:50.4190980Z }, 2024-06-26T07:03:50.4191230Z { 2024-06-26T07:03:50.4191650Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4192195Z "size": 2528295403, 2024-06-26T07:03:50.4192749Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-06-26T07:03:50.4193348Z }, 2024-06-26T07:03:50.4193583Z { 2024-06-26T07:03:50.4194005Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4194544Z "size": 86016, 2024-06-26T07:03:50.4195082Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-06-26T07:03:50.4195677Z }, 2024-06-26T07:03:50.4195922Z { 2024-06-26T07:03:50.4196350Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4196888Z "size": 1823, 2024-06-26T07:03:50.4197409Z "digest": "sha256:309b872da805fcdf74137a887464876fa0615dc8b24199307119eb65fc2bfb49" 2024-06-26T07:03:50.4198001Z }, 2024-06-26T07:03:50.4198246Z { 2024-06-26T07:03:50.4198663Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4199201Z "size": 246737542, 2024-06-26T07:03:50.4199738Z "digest": "sha256:e82932cb002c06a782077b8e5965f8733eb8563406a23cab260d9d8c4a994f7d" 2024-06-26T07:03:50.4200325Z }, 2024-06-26T07:03:50.4200559Z { 2024-06-26T07:03:50.4200980Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4201519Z "size": 545, 2024-06-26T07:03:50.4202056Z "digest": "sha256:ca9dd73e3e9378225ebdcfd12199ef44649e34fe55003cfbc8afe2e8a4527c5b" 2024-06-26T07:03:50.4202654Z }, 2024-06-26T07:03:50.4202906Z { 2024-06-26T07:03:50.4203338Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4203874Z "size": 1283, 2024-06-26T07:03:50.4204396Z "digest": "sha256:16d76a79d5d6616e228537785ec83a869f68a402520bbc6e826201621ca6cb98" 2024-06-26T07:03:50.4204980Z }, 2024-06-26T07:03:50.4205221Z { 2024-06-26T07:03:50.4205637Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4206173Z "size": 485, 2024-06-26T07:03:50.4206709Z "digest": "sha256:4bdf41c9ecbe1651e0986ccbea4663a0556af27b7145a679cfa45c35dc96b891" 2024-06-26T07:03:50.4207317Z }, 2024-06-26T07:03:50.4207552Z { 2024-06-26T07:03:50.4207973Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4208516Z "size": 91710937, 2024-06-26T07:03:50.4209076Z "digest": "sha256:f4863801e30cdbcd1d4e6cb1cbbcde7dbcaf6ae317ca80429295741ce60ac31d" 2024-06-26T07:03:50.4209687Z }, 2024-06-26T07:03:50.4209930Z { 2024-06-26T07:03:50.4210377Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4211300Z "size": 3233, 2024-06-26T07:03:50.4211893Z "digest": "sha256:d37dd9ed8948a0d02e1198bbedadbca36477a0980dd553196385aab0c025581d" 2024-06-26T07:03:50.4212509Z }, 2024-06-26T07:03:50.4212748Z { 2024-06-26T07:03:50.4213164Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4213703Z "size": 1905, 2024-06-26T07:03:50.4214248Z "digest": "sha256:cf2f34f225dbed17c1a9a8b9ea4ad030aba797774a716dc0a82fe212324aa751" 2024-06-26T07:03:50.4214859Z }, 2024-06-26T07:03:50.4215092Z { 2024-06-26T07:03:50.4215520Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4216057Z "size": 702, 2024-06-26T07:03:50.4216578Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:03:50.4217182Z }, 2024-06-26T07:03:50.4217424Z { 2024-06-26T07:03:50.4217842Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4218374Z "size": 2784789002, 2024-06-26T07:03:50.4218934Z "digest": "sha256:3d914950f8addfc4d657a2365c76e1143eb80b710cd3f6e9ae05640217fc6347" 2024-06-26T07:03:50.4219602Z }, 2024-06-26T07:03:50.4219844Z { 2024-06-26T07:03:50.4220262Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4220806Z "size": 381, 2024-06-26T07:03:50.4221332Z "digest": "sha256:96d0e2032184c57e62a652aaffcf0241f0d0c7a23e894ca0ad192c4a64a36583" 2024-06-26T07:03:50.4221933Z }, 2024-06-26T07:03:50.4222169Z { 2024-06-26T07:03:50.4222714Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4223282Z "size": 11879, 2024-06-26T07:03:50.4223804Z "digest": "sha256:06e12ce59790cb8b091e5d619c2a62a28329008ee184a8b9a562e3db12471f30" 2024-06-26T07:03:50.4224395Z }, 2024-06-26T07:03:50.4224639Z { 2024-06-26T07:03:50.4225058Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4225590Z "size": 803, 2024-06-26T07:03:50.4226133Z "digest": "sha256:ed712f63cde7070bdec2a6179c4c35e752c6991ddbc69fe9f2724a9903250ceb" 2024-06-26T07:03:50.4226748Z }, 2024-06-26T07:03:50.4226990Z { 2024-06-26T07:03:50.4227402Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4227939Z "size": 106, 2024-06-26T07:03:50.4228460Z "digest": "sha256:a97a71ef8a4e66234dfc2098834f8395ac5630240e60164d1a251fdb0115ba85" 2024-06-26T07:03:50.4229051Z }, 2024-06-26T07:03:50.4229294Z { 2024-06-26T07:03:50.4229713Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4230247Z "size": 503, 2024-06-26T07:03:50.4230761Z "digest": "sha256:25cd0c482ad612f936b35b4f220c580e16be982226aac8801a6a849a6bfdc221" 2024-06-26T07:03:50.4231354Z }, 2024-06-26T07:03:50.4231595Z { 2024-06-26T07:03:50.4232015Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4232546Z "size": 121477177, 2024-06-26T07:03:50.4233099Z "digest": "sha256:9bf94bc456584365fda21d0ed83b7cc690b08a71727b71c4c1e667e15028a386" 2024-06-26T07:03:50.4233699Z }, 2024-06-26T07:03:50.4233947Z { 2024-06-26T07:03:50.4234363Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4234895Z "size": 109, 2024-06-26T07:03:50.4235423Z "digest": "sha256:de45a14c9b227e4b0c9d45d42223b50a71c03522ddd9e2ca0d9517835206c823" 2024-06-26T07:03:50.4236017Z }, 2024-06-26T07:03:50.4236260Z { 2024-06-26T07:03:50.4236686Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4237223Z "size": 491, 2024-06-26T07:03:50.4237737Z "digest": "sha256:e99241e77209b0b35a0fb59a61c0821fc161190178af7e4af45e6eeaa043375e" 2024-06-26T07:03:50.4238334Z }, 2024-06-26T07:03:50.4238577Z { 2024-06-26T07:03:50.4239000Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4239532Z "size": 291, 2024-06-26T07:03:50.4240073Z "digest": "sha256:35e8f4e2a42f9df0c969dedd2199cf71719cf5ecfa676dfa014bbdeb4fdb295f" 2024-06-26T07:03:50.4240791Z }, 2024-06-26T07:03:50.4241029Z { 2024-06-26T07:03:50.4241454Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4242001Z "size": 101, 2024-06-26T07:03:50.4242523Z "digest": "sha256:a823ffa4c541394b52953391dc0a6867e26c37fe4a71fa022a2cc3f07f0ba70d" 2024-06-26T07:03:50.4243116Z }, 2024-06-26T07:03:50.4243360Z { 2024-06-26T07:03:50.4243778Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4244319Z "size": 1473, 2024-06-26T07:03:50.4244833Z "digest": "sha256:44ba272eae2064603605a6238b3f35cc6578859d660068e702a047a9cd8ab2fd" 2024-06-26T07:03:50.4245426Z }, 2024-06-26T07:03:50.4245667Z { 2024-06-26T07:03:50.4246092Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4246626Z "size": 446710781, 2024-06-26T07:03:50.4247172Z "digest": "sha256:da02d278e5240c285bc855ce4e42cd6121d5b903787f5c14f03e575ad1268650" 2024-06-26T07:03:50.4247770Z }, 2024-06-26T07:03:50.4248016Z { 2024-06-26T07:03:50.4248436Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4249040Z "size": 161, 2024-06-26T07:03:50.4249552Z "digest": "sha256:73a0ce681c295135456e190578733273276b38b4dd5babbb39298f384d43b1c6" 2024-06-26T07:03:50.4250131Z }, 2024-06-26T07:03:50.4250374Z { 2024-06-26T07:03:50.4250795Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4251559Z "size": 564, 2024-06-26T07:03:50.4252082Z "digest": "sha256:49737bd5ae709dbf7956e21fc8b05ba009e9e6c9b130b0126a73bd3d3f5efc78" 2024-06-26T07:03:50.4252687Z }, 2024-06-26T07:03:50.4252927Z { 2024-06-26T07:03:50.4253338Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4253881Z "size": 35874600, 2024-06-26T07:03:50.4254429Z "digest": "sha256:9cd4c745f8e4f9d5c9bb8050923575afad21f5b7852975a4a1981bc9eff282c5" 2024-06-26T07:03:50.4255025Z }, 2024-06-26T07:03:50.4255259Z { 2024-06-26T07:03:50.4255682Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4256222Z "size": 104, 2024-06-26T07:03:50.4256739Z "digest": "sha256:b2950a0365345d4c17a48627c905fd8930b3b3154c727f0526eb9ca547102bb7" 2024-06-26T07:03:50.4257319Z }, 2024-06-26T07:03:50.4257561Z { 2024-06-26T07:03:50.4257978Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4258518Z "size": 425, 2024-06-26T07:03:50.4259023Z "digest": "sha256:25e19d81d9fc081962ff4cbc09e098f4035419822712c19a410f5282300dfb9e" 2024-06-26T07:03:50.4259613Z }, 2024-06-26T07:03:50.4259857Z { 2024-06-26T07:03:50.4260271Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4260810Z "size": 20262084, 2024-06-26T07:03:50.4261358Z "digest": "sha256:60156b40fcbec2c25f0170b0a82fa8cd9303e0daae8bb1811fd0fc6562e88e8e" 2024-06-26T07:03:50.4261959Z }, 2024-06-26T07:03:50.4262199Z { 2024-06-26T07:03:50.4262708Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4263245Z "size": 439, 2024-06-26T07:03:50.4263769Z "digest": "sha256:7a868239227fd19b30a19927bbc8f058d247403fe5b343c55f4b4b8987918911" 2024-06-26T07:03:50.4264355Z }, 2024-06-26T07:03:50.4264604Z { 2024-06-26T07:03:50.4265024Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4265551Z "size": 702, 2024-06-26T07:03:50.4266075Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:03:50.4266679Z }, 2024-06-26T07:03:50.4266922Z { 2024-06-26T07:03:50.4267334Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4267872Z "size": 143, 2024-06-26T07:03:50.4268390Z "digest": "sha256:83911d7c7ecf11b5932bb7449975c52e7f737a5b867691935e1ccb85ceb25b52" 2024-06-26T07:03:50.4268983Z }, 2024-06-26T07:03:50.4269216Z { 2024-06-26T07:03:50.4269637Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4270292Z "size": 135, 2024-06-26T07:03:50.4270852Z "digest": "sha256:b3c5d66c679f58c71febdd502889bd8cd3d36e5001754cdf1e30eabe21135cce" 2024-06-26T07:03:50.4271459Z }, 2024-06-26T07:03:50.4271700Z { 2024-06-26T07:03:50.4272122Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4272657Z "size": 4911495247, 2024-06-26T07:03:50.4273201Z "digest": "sha256:84b2c7a899820f6acdb22148c5e8346d8b26c33269fd01842c9573515251e1e1" 2024-06-26T07:03:50.4273796Z }, 2024-06-26T07:03:50.4274043Z { 2024-06-26T07:03:50.4274455Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4274999Z "size": 190, 2024-06-26T07:03:50.4275535Z "digest": "sha256:c1068d03a811d8dead5da9e82fdc8869205fc4ee9cb92d4f7a4acfbb5ba83950" 2024-06-26T07:03:50.4276148Z }, 2024-06-26T07:03:50.4276385Z { 2024-06-26T07:03:50.4276806Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4277346Z "size": 565, 2024-06-26T07:03:50.4277863Z "digest": "sha256:671b4a301d81074fca17d36851f81801552c4ecb2f90ff3412aa9ab6a1a91371" 2024-06-26T07:03:50.4278535Z }, 2024-06-26T07:03:50.4278784Z { 2024-06-26T07:03:50.4279211Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4279747Z "size": 43163095, 2024-06-26T07:03:50.4280335Z "digest": "sha256:67760cf5ec46ab58d3ca98637fd9b26cb47a67446bfd0db176fe5e023606aa0d" 2024-06-26T07:03:50.4280954Z }, 2024-06-26T07:03:50.4281196Z { 2024-06-26T07:03:50.4281608Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4282148Z "size": 106, 2024-06-26T07:03:50.4282679Z "digest": "sha256:ddc9740bcea2b301e99b50525059aea8d9c7121a1c8eab566561a8968438a8c2" 2024-06-26T07:03:50.4283277Z }, 2024-06-26T07:03:50.4283511Z { 2024-06-26T07:03:50.4283926Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4284482Z "size": 1213, 2024-06-26T07:03:50.4285003Z "digest": "sha256:fb2e977685609fcc01de170ba7f50a573e73e6520e47253717e92a28952c5ee8" 2024-06-26T07:03:50.4285600Z }, 2024-06-26T07:03:50.4285850Z { 2024-06-26T07:03:50.4286271Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4286805Z "size": 702, 2024-06-26T07:03:50.4287334Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:03:50.4287943Z }, 2024-06-26T07:03:50.4288186Z { 2024-06-26T07:03:50.4288597Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4289131Z "size": 138, 2024-06-26T07:03:50.4289659Z "digest": "sha256:b599c5a8c7cb461cbd7b3c98b99960b8eb04506e05e79db1cc0cfadb14a47e91" 2024-06-26T07:03:50.4290265Z }, 2024-06-26T07:03:50.4290499Z { 2024-06-26T07:03:50.4291096Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4291647Z "size": 120, 2024-06-26T07:03:50.4292179Z "digest": "sha256:e08e88c1366827455ee7ddbf92b6ae615ceb3cbd3fb11cd03ddd31bd3fa94a9c" 2024-06-26T07:03:50.4292786Z }, 2024-06-26T07:03:50.4293030Z { 2024-06-26T07:03:50.4293455Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4293989Z "size": 1904157495, 2024-06-26T07:03:50.4294542Z "digest": "sha256:12ba7431ff38de8cf6af3052a2a52e79f4e712b8c99f5bafde3c20e63b1f570f" 2024-06-26T07:03:50.4295153Z }, 2024-06-26T07:03:50.4295396Z { 2024-06-26T07:03:50.4295807Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4296349Z "size": 172, 2024-06-26T07:03:50.4296879Z "digest": "sha256:cf7c2cb896cba6a594028cc0fd246e127a2f4481d48905a3aa12c5f5872332d4" 2024-06-26T07:03:50.4297488Z }, 2024-06-26T07:03:50.4297723Z { 2024-06-26T07:03:50.4298140Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4298675Z "size": 908, 2024-06-26T07:03:50.4299204Z "digest": "sha256:3a778ddfac2af5b3a11f49ddf2f3f1ad1731f74ba691f434ab69966d1e2a189f" 2024-06-26T07:03:50.4299898Z }, 2024-06-26T07:03:50.4300146Z { 2024-06-26T07:03:50.4300571Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4301109Z "size": 702, 2024-06-26T07:03:50.4301636Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:03:50.4302246Z }, 2024-06-26T07:03:50.4302492Z { 2024-06-26T07:03:50.4303024Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4303569Z "size": 134, 2024-06-26T07:03:50.4304097Z "digest": "sha256:236c8f0245f4b40616a4322eaf1e1b3e178d5990db1fcaba39622ea4e79c06a1" 2024-06-26T07:03:50.4304694Z }, 2024-06-26T07:03:50.4304940Z { 2024-06-26T07:03:50.4305365Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4305911Z "size": 32, 2024-06-26T07:03:50.4306439Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:03:50.4307051Z }, 2024-06-26T07:03:50.4307305Z { 2024-06-26T07:03:50.4307733Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4308404Z "size": 156, 2024-06-26T07:03:50.4308945Z "digest": "sha256:dda1ce302b9a9df0c8f1145d2e3eaa03c96909bb45a792451f1fcb555b06738d" 2024-06-26T07:03:50.4309564Z }, 2024-06-26T07:03:50.4309804Z { 2024-06-26T07:03:50.4310229Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4310772Z "size": 1841, 2024-06-26T07:03:50.4311310Z "digest": "sha256:e797fe86caef19319e03f8dca92cf96838ab3bb3644f4447c6dfc04230118afa" 2024-06-26T07:03:50.4311908Z }, 2024-06-26T07:03:50.4312154Z { 2024-06-26T07:03:50.4312583Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4313126Z "size": 7529784, 2024-06-26T07:03:50.4313666Z "digest": "sha256:967190076fc0ab2bb2fc0d74e58edd3d662a537d702ff3f8b6e876230154a16f" 2024-06-26T07:03:50.4314272Z }, 2024-06-26T07:03:50.4314519Z { 2024-06-26T07:03:50.4314952Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4315489Z "size": 165, 2024-06-26T07:03:50.4316032Z "digest": "sha256:4fb0a83f06855534ef82b80dcf7c685303edb2bcbd446196066dc2ad2dd71281" 2024-06-26T07:03:50.4316639Z }, 2024-06-26T07:03:50.4316878Z { 2024-06-26T07:03:50.4317310Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4317855Z "size": 7944, 2024-06-26T07:03:50.4318396Z "digest": "sha256:68af2ec4e30a33f26947ece9f3a31b7d3b04c2bca5865d4b48618457832a1da2" 2024-06-26T07:03:50.4318999Z }, 2024-06-26T07:03:50.4319245Z { 2024-06-26T07:03:50.4319670Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4320216Z "size": 8070, 2024-06-26T07:03:50.4320741Z "digest": "sha256:cc91933e985a044c6df53caa15488d2285896f5e991f38ec068c253ceec53f14" 2024-06-26T07:03:50.4321342Z }, 2024-06-26T07:03:50.4321590Z { 2024-06-26T07:03:50.4322013Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4322553Z "size": 303, 2024-06-26T07:03:50.4323090Z "digest": "sha256:9f3187916e7dcad33cc572f8f0e949d2f507b887f99cfd6acefd14826b44960d" 2024-06-26T07:03:50.4323710Z }, 2024-06-26T07:03:50.4323947Z { 2024-06-26T07:03:50.4324373Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4324919Z "size": 7629071, 2024-06-26T07:03:50.4325438Z "digest": "sha256:52608399485738d10d779254946c9e1d536010dd45593c69bf1eb9f62f85dc24" 2024-06-26T07:03:50.4326016Z }, 2024-06-26T07:03:50.4326261Z { 2024-06-26T07:03:50.4326682Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4327219Z "size": 108, 2024-06-26T07:03:50.4327749Z "digest": "sha256:05a6f6dcc1c00bac6660923a22adc7d7c34416718f8c89daf9b3351d9365f694" 2024-06-26T07:03:50.4328358Z }, 2024-06-26T07:03:50.4328602Z { 2024-06-26T07:03:50.4329023Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4329634Z "size": 54145775, 2024-06-26T07:03:50.4330191Z "digest": "sha256:6bc9dc8315b09b27c976bfba9700767ba412d72fd0528b18bbe8f8e62005ec22" 2024-06-26T07:03:50.4330942Z }, 2024-06-26T07:03:50.4331245Z { 2024-06-26T07:03:50.4331676Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4332211Z "size": 474, 2024-06-26T07:03:50.4332740Z "digest": "sha256:1b0e27c35f07b3fbf0aeb548d1949bf1d60a420136da21789d94d196c891d981" 2024-06-26T07:03:50.4333342Z }, 2024-06-26T07:03:50.4333584Z { 2024-06-26T07:03:50.4334010Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4334548Z "size": 1374858747, 2024-06-26T07:03:50.4335092Z "digest": "sha256:95237e7311a1c5ce6937db7196c866632f44e8788514a5c0eb4a3f04da4017e6" 2024-06-26T07:03:50.4335685Z }, 2024-06-26T07:03:50.4335928Z { 2024-06-26T07:03:50.4336345Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4336888Z "size": 106, 2024-06-26T07:03:50.4337437Z "digest": "sha256:789eff6c51aada70bf75eb1acafed63c0d839b768638001500616ff6e53b854a" 2024-06-26T07:03:50.4338136Z }, 2024-06-26T07:03:50.4338374Z { 2024-06-26T07:03:50.4338800Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4339346Z "size": 558, 2024-06-26T07:03:50.4339886Z "digest": "sha256:de0f8a8c45c6077b3c5ba3b84d7f07de98353fd618dad9a1e2f368ffe00493c7" 2024-06-26T07:03:50.4340494Z }, 2024-06-26T07:03:50.4340737Z { 2024-06-26T07:03:50.4341160Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4341697Z "size": 46248513, 2024-06-26T07:03:50.4342235Z "digest": "sha256:2060028f74888d218bebdd10d723f66c33ee1c708935c4ffd4654d34912ac4dc" 2024-06-26T07:03:50.4342917Z }, 2024-06-26T07:03:50.4343160Z { 2024-06-26T07:03:50.4343576Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4344116Z "size": 111, 2024-06-26T07:03:50.4344655Z "digest": "sha256:e412c7f3b2ded440992955bd92512345cbece8a8894cd3592f714ebc758ad1a0" 2024-06-26T07:03:50.4345256Z }, 2024-06-26T07:03:50.4345500Z { 2024-06-26T07:03:50.4345921Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4346466Z "size": 32, 2024-06-26T07:03:50.4346991Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:03:50.4347597Z }, 2024-06-26T07:03:50.4347841Z { 2024-06-26T07:03:50.4348266Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4348806Z "size": 32, 2024-06-26T07:03:50.4349339Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:03:50.4349950Z }, 2024-06-26T07:03:50.4350200Z { 2024-06-26T07:03:50.4350622Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4351164Z "size": 32, 2024-06-26T07:03:50.4351699Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:03:50.4352304Z }, 2024-06-26T07:03:50.4352543Z { 2024-06-26T07:03:50.4352975Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:50.4353519Z "size": 32, 2024-06-26T07:03:50.4354040Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:03:50.4354653Z } 2024-06-26T07:03:50.4354901Z ] 2024-06-26T07:03:50.4355143Z } 2024-06-26T07:03:50.4454631Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:03:50.4455118Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:03:50.4455658Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-26T07:03:50.4463599Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:50.4464097Z env: 2024-06-26T07:03:50.4464372Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:50.4479054Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:50.4480213Z ##[endgroup] 2024-06-26T07:03:50.4503067Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:50.4550716Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T07:03:50.4551284Z with: 2024-06-26T07:03:50.4552326Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:50.4553604Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:50.4554104Z env: 2024-06-26T07:03:50.4554374Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:50.4554715Z ##[endgroup] 2024-06-26T07:03:50.4569850Z ##[group]Run set -x 2024-06-26T07:03:50.4570187Z set -x 2024-06-26T07:03:50.4570492Z set +e 2024-06-26T07:03:50.4570932Z  2024-06-26T07:03:50.4571208Z login() { 2024-06-26T07:03:50.4571872Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:03:50.4572683Z } 2024-06-26T07:03:50.4572968Z  2024-06-26T07:03:50.4573263Z retry () { 2024-06-26T07:03:50.4573654Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:03:50.4574094Z } 2024-06-26T07:03:50.4574368Z  2024-06-26T07:03:50.4574679Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:03:50.4575089Z  2024-06-26T07:03:50.4575359Z set -e 2024-06-26T07:03:50.4575826Z # ignore output since only exit code is used for conditional 2024-06-26T07:03:50.4576523Z # only pull docker image if it's not available locally 2024-06-26T07:03:50.4577270Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T07:03:50.4577968Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T07:03:50.4578413Z fi 2024-06-26T07:03:50.4585150Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:50.4585652Z env: 2024-06-26T07:03:50.4585922Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:50.4587049Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:50.4588316Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:50.4588824Z ##[endgroup] 2024-06-26T07:03:50.4608007Z + set +e 2024-06-26T07:03:50.4608614Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:50.4609244Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:50.4610198Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:03:50.4611265Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:50.8678255Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:03:50.8679157Z Configure a credential helper to remove this warning. See 2024-06-26T07:03:50.8680107Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:03:50.8680643Z 2024-06-26T07:03:50.8680765Z Login Succeeded 2024-06-26T07:03:50.8688601Z + set -e 2024-06-26T07:03:50.8689897Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:50.8812908Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:50.8815015Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:51.0759397Z 91382da70d5719cd7007b6b80b71d2f48398f6b7: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-06-26T07:03:51.0766248Z 7a2c55901189: Pulling fs layer 2024-06-26T07:03:51.0767943Z 224fe954d725: Pulling fs layer 2024-06-26T07:03:51.0769018Z 75722010b82e: Pulling fs layer 2024-06-26T07:03:51.0769639Z d527cbbb87e3: Pulling fs layer 2024-06-26T07:03:51.0770197Z b57676e46aee: Pulling fs layer 2024-06-26T07:03:51.0770747Z a8c1e85b5e14: Pulling fs layer 2024-06-26T07:03:51.0771489Z a41a8d1c11c8: Pulling fs layer 2024-06-26T07:03:51.0771986Z 0c1227890755: Pulling fs layer 2024-06-26T07:03:51.0772478Z d8d1234baab3: Pulling fs layer 2024-06-26T07:03:51.0772964Z 7ed32bc8e469: Pulling fs layer 2024-06-26T07:03:51.0773439Z ec1e7978c1fe: Pulling fs layer 2024-06-26T07:03:51.0773814Z 309b872da805: Pulling fs layer 2024-06-26T07:03:51.0774190Z e82932cb002c: Pulling fs layer 2024-06-26T07:03:51.0774587Z ca9dd73e3e93: Pulling fs layer 2024-06-26T07:03:51.0774959Z 16d76a79d5d6: Pulling fs layer 2024-06-26T07:03:51.0775342Z 4bdf41c9ecbe: Pulling fs layer 2024-06-26T07:03:51.0775726Z f4863801e30c: Pulling fs layer 2024-06-26T07:03:51.0776235Z d37dd9ed8948: Pulling fs layer 2024-06-26T07:03:51.0776718Z cf2f34f225db: Pulling fs layer 2024-06-26T07:03:51.0777121Z df546d7b3e34: Pulling fs layer 2024-06-26T07:03:51.0777536Z 3d914950f8ad: Pulling fs layer 2024-06-26T07:03:51.0778045Z 96d0e2032184: Pulling fs layer 2024-06-26T07:03:51.0778537Z 06e12ce59790: Pulling fs layer 2024-06-26T07:03:51.0779034Z ed712f63cde7: Pulling fs layer 2024-06-26T07:03:51.0779530Z a97a71ef8a4e: Pulling fs layer 2024-06-26T07:03:51.0780034Z 25cd0c482ad6: Pulling fs layer 2024-06-26T07:03:51.0780520Z a41a8d1c11c8: Waiting 2024-06-26T07:03:51.0780952Z 9bf94bc45658: Pulling fs layer 2024-06-26T07:03:51.0781428Z d527cbbb87e3: Waiting 2024-06-26T07:03:51.0781853Z 0c1227890755: Waiting 2024-06-26T07:03:51.0782290Z de45a14c9b22: Pulling fs layer 2024-06-26T07:03:51.0782892Z e99241e77209: Pulling fs layer 2024-06-26T07:03:51.0783389Z 35e8f4e2a42f: Pulling fs layer 2024-06-26T07:03:51.0783859Z a823ffa4c541: Pulling fs layer 2024-06-26T07:03:51.0784237Z 44ba272eae20: Pulling fs layer 2024-06-26T07:03:51.0784630Z e82932cb002c: Waiting 2024-06-26T07:03:51.0784951Z a8c1e85b5e14: Waiting 2024-06-26T07:03:51.0785290Z da02d278e524: Pulling fs layer 2024-06-26T07:03:51.0785650Z b57676e46aee: Waiting 2024-06-26T07:03:51.0785958Z ca9dd73e3e93: Waiting 2024-06-26T07:03:51.0786288Z 73a0ce681c29: Pulling fs layer 2024-06-26T07:03:51.0786662Z 49737bd5ae70: Pulling fs layer 2024-06-26T07:03:51.0787023Z 9cd4c745f8e4: Pulling fs layer 2024-06-26T07:03:51.0787381Z 4bdf41c9ecbe: Waiting 2024-06-26T07:03:51.0787709Z b2950a036534: Pulling fs layer 2024-06-26T07:03:51.0788108Z f4863801e30c: Waiting 2024-06-26T07:03:51.0788419Z 309b872da805: Waiting 2024-06-26T07:03:51.0788737Z 25e19d81d9fc: Pulling fs layer 2024-06-26T07:03:51.0789089Z 25cd0c482ad6: Waiting 2024-06-26T07:03:51.0789418Z 60156b40fcbe: Pulling fs layer 2024-06-26T07:03:51.0789776Z d37dd9ed8948: Waiting 2024-06-26T07:03:51.0790084Z da02d278e524: Waiting 2024-06-26T07:03:51.0790409Z 7a868239227f: Pulling fs layer 2024-06-26T07:03:51.0790768Z 9bf94bc45658: Waiting 2024-06-26T07:03:51.0791085Z 83911d7c7ecf: Pulling fs layer 2024-06-26T07:03:51.0791457Z b3c5d66c679f: Pulling fs layer 2024-06-26T07:03:51.0791809Z 73a0ce681c29: Waiting 2024-06-26T07:03:51.0792174Z de45a14c9b22: Waiting 2024-06-26T07:03:51.0792497Z 84b2c7a89982: Pulling fs layer 2024-06-26T07:03:51.0792866Z c1068d03a811: Pulling fs layer 2024-06-26T07:03:51.0793233Z 671b4a301d81: Pulling fs layer 2024-06-26T07:03:51.0793591Z 67760cf5ec46: Pulling fs layer 2024-06-26T07:03:51.0793959Z ddc9740bcea2: Pulling fs layer 2024-06-26T07:03:51.0794328Z fb2e97768560: Pulling fs layer 2024-06-26T07:03:51.0794696Z b599c5a8c7cb: Pulling fs layer 2024-06-26T07:03:51.0795056Z e08e88c13668: Pulling fs layer 2024-06-26T07:03:51.0795420Z 12ba7431ff38: Pulling fs layer 2024-06-26T07:03:51.0795788Z cf7c2cb896cb: Pulling fs layer 2024-06-26T07:03:51.0796139Z 49737bd5ae70: Waiting 2024-06-26T07:03:51.0796449Z ec1e7978c1fe: Waiting 2024-06-26T07:03:51.0796788Z 3a778ddfac2a: Pulling fs layer 2024-06-26T07:03:51.0797155Z 9cd4c745f8e4: Waiting 2024-06-26T07:03:51.0797475Z 236c8f0245f4: Pulling fs layer 2024-06-26T07:03:51.0797976Z 4f4fb700ef54: Pulling fs layer 2024-06-26T07:03:51.0798337Z 44ba272eae20: Waiting 2024-06-26T07:03:51.0798647Z c1068d03a811: Waiting 2024-06-26T07:03:51.0798966Z dda1ce302b9a: Pulling fs layer 2024-06-26T07:03:51.0799321Z 83911d7c7ecf: Waiting 2024-06-26T07:03:51.0799632Z a97a71ef8a4e: Waiting 2024-06-26T07:03:51.0799941Z 84b2c7a89982: Waiting 2024-06-26T07:03:51.0800247Z b3c5d66c679f: Waiting 2024-06-26T07:03:51.0800558Z 671b4a301d81: Waiting 2024-06-26T07:03:51.0800866Z a823ffa4c541: Waiting 2024-06-26T07:03:51.0801167Z 60156b40fcbe: Waiting 2024-06-26T07:03:51.0801477Z ed712f63cde7: Waiting 2024-06-26T07:03:51.0801804Z e797fe86caef: Pulling fs layer 2024-06-26T07:03:51.0802173Z 967190076fc0: Pulling fs layer 2024-06-26T07:03:51.0802534Z 4fb0a83f0685: Pulling fs layer 2024-06-26T07:03:51.0802904Z 68af2ec4e30a: Pulling fs layer 2024-06-26T07:03:51.0803258Z b2950a036534: Waiting 2024-06-26T07:03:51.0803635Z cc91933e985a: Pulling fs layer 2024-06-26T07:03:51.0804007Z 9f3187916e7d: Pulling fs layer 2024-06-26T07:03:51.0804378Z 526083994857: Pulling fs layer 2024-06-26T07:03:51.0804738Z df546d7b3e34: Waiting 2024-06-26T07:03:51.0805060Z 05a6f6dcc1c0: Pulling fs layer 2024-06-26T07:03:51.0805415Z 3d914950f8ad: Waiting 2024-06-26T07:03:51.0805730Z 35e8f4e2a42f: Waiting 2024-06-26T07:03:51.0806059Z 6bc9dc8315b0: Pulling fs layer 2024-06-26T07:03:51.0806419Z 1b0e27c35f07: Pulling fs layer 2024-06-26T07:03:51.0806785Z 95237e7311a1: Pulling fs layer 2024-06-26T07:03:51.0807150Z 789eff6c51aa: Pulling fs layer 2024-06-26T07:03:51.0807512Z de0f8a8c45c6: Pulling fs layer 2024-06-26T07:03:51.0807882Z 2060028f7488: Pulling fs layer 2024-06-26T07:03:51.0808256Z e412c7f3b2de: Pulling fs layer 2024-06-26T07:03:51.0808616Z e99241e77209: Waiting 2024-06-26T07:03:51.0808927Z 3a778ddfac2a: Waiting 2024-06-26T07:03:51.0809243Z ddc9740bcea2: Waiting 2024-06-26T07:03:51.0809552Z 967190076fc0: Waiting 2024-06-26T07:03:51.0809865Z b599c5a8c7cb: Waiting 2024-06-26T07:03:51.0810171Z 4fb0a83f0685: Waiting 2024-06-26T07:03:51.0810483Z 1b0e27c35f07: Waiting 2024-06-26T07:03:51.0810986Z 236c8f0245f4: Waiting 2024-06-26T07:03:51.0811398Z 95237e7311a1: Waiting 2024-06-26T07:03:51.0811717Z 526083994857: Waiting 2024-06-26T07:03:51.0812022Z 4f4fb700ef54: Waiting 2024-06-26T07:03:51.0812360Z 68af2ec4e30a: Waiting 2024-06-26T07:03:51.0812693Z dda1ce302b9a: Waiting 2024-06-26T07:03:51.0813006Z cc91933e985a: Waiting 2024-06-26T07:03:51.0813317Z e08e88c13668: Waiting 2024-06-26T07:03:51.0813620Z 12ba7431ff38: Waiting 2024-06-26T07:03:51.0813937Z de0f8a8c45c6: Waiting 2024-06-26T07:03:51.0814245Z 2060028f7488: Waiting 2024-06-26T07:03:51.0814558Z cf7c2cb896cb: Waiting 2024-06-26T07:03:51.0814865Z 6bc9dc8315b0: Waiting 2024-06-26T07:03:51.0815178Z e797fe86caef: Waiting 2024-06-26T07:03:51.0815490Z 789eff6c51aa: Waiting 2024-06-26T07:03:51.0815798Z fb2e97768560: Waiting 2024-06-26T07:03:51.0816108Z e412c7f3b2de: Waiting 2024-06-26T07:03:51.2141513Z 224fe954d725: Verifying Checksum 2024-06-26T07:03:51.2141978Z 224fe954d725: Download complete 2024-06-26T07:03:51.2819827Z d527cbbb87e3: Download complete 2024-06-26T07:03:51.3638236Z b57676e46aee: Download complete 2024-06-26T07:03:51.4094637Z 7a2c55901189: Verifying Checksum 2024-06-26T07:03:51.4096089Z 7a2c55901189: Download complete 2024-06-26T07:03:51.4929369Z a41a8d1c11c8: Download complete 2024-06-26T07:03:51.5560694Z 0c1227890755: Download complete 2024-06-26T07:03:51.6242728Z d8d1234baab3: Download complete 2024-06-26T07:03:51.7017579Z 75722010b82e: Verifying Checksum 2024-06-26T07:03:51.7018121Z 75722010b82e: Download complete 2024-06-26T07:03:51.7923465Z ec1e7978c1fe: Download complete 2024-06-26T07:03:51.9006976Z 309b872da805: Download complete 2024-06-26T07:03:52.4355762Z 7a2c55901189: Pull complete 2024-06-26T07:03:52.7507523Z 224fe954d725: Pull complete 2024-06-26T07:03:53.7267814Z 75722010b82e: Pull complete 2024-06-26T07:03:53.7695871Z d527cbbb87e3: Pull complete 2024-06-26T07:03:53.8050320Z b57676e46aee: Pull complete 2024-06-26T07:03:54.4281414Z e82932cb002c: Verifying Checksum 2024-06-26T07:03:54.4282232Z e82932cb002c: Download complete 2024-06-26T07:03:54.5033879Z ca9dd73e3e93: Verifying Checksum 2024-06-26T07:03:54.5034944Z ca9dd73e3e93: Download complete 2024-06-26T07:03:54.5834069Z 16d76a79d5d6: Download complete 2024-06-26T07:03:54.6670539Z 4bdf41c9ecbe: Verifying Checksum 2024-06-26T07:03:54.6670966Z 4bdf41c9ecbe: Download complete 2024-06-26T07:03:55.6459680Z f4863801e30c: Verifying Checksum 2024-06-26T07:03:55.6460213Z f4863801e30c: Download complete 2024-06-26T07:03:55.7187561Z d37dd9ed8948: Verifying Checksum 2024-06-26T07:03:55.7188107Z d37dd9ed8948: Download complete 2024-06-26T07:03:55.7910719Z cf2f34f225db: Verifying Checksum 2024-06-26T07:03:55.7911137Z cf2f34f225db: Download complete 2024-06-26T07:03:55.8752961Z df546d7b3e34: Verifying Checksum 2024-06-26T07:04:04.7093321Z a8c1e85b5e14: Verifying Checksum 2024-06-26T07:04:04.7093851Z a8c1e85b5e14: Download complete 2024-06-26T07:04:04.7881739Z 96d0e2032184: Verifying Checksum 2024-06-26T07:04:04.7882601Z 96d0e2032184: Download complete 2024-06-26T07:04:04.8700476Z 06e12ce59790: Download complete 2024-06-26T07:04:04.9386573Z ed712f63cde7: Verifying Checksum 2024-06-26T07:04:04.9387129Z ed712f63cde7: Download complete 2024-06-26T07:04:05.0270187Z a97a71ef8a4e: Verifying Checksum 2024-06-26T07:04:05.0270756Z a97a71ef8a4e: Download complete 2024-06-26T07:04:05.1174729Z 25cd0c482ad6: Verifying Checksum 2024-06-26T07:04:05.1175213Z 25cd0c482ad6: Download complete 2024-06-26T07:04:06.5054321Z 9bf94bc45658: Verifying Checksum 2024-06-26T07:04:06.5054940Z 9bf94bc45658: Download complete 2024-06-26T07:04:06.5770672Z de45a14c9b22: Verifying Checksum 2024-06-26T07:04:06.5771394Z de45a14c9b22: Download complete 2024-06-26T07:04:06.6891341Z e99241e77209: Verifying Checksum 2024-06-26T07:04:06.6891906Z e99241e77209: Download complete 2024-06-26T07:04:06.7834069Z 35e8f4e2a42f: Verifying Checksum 2024-06-26T07:04:06.7834503Z 35e8f4e2a42f: Download complete 2024-06-26T07:04:06.8775005Z a823ffa4c541: Verifying Checksum 2024-06-26T07:04:06.8775511Z a823ffa4c541: Download complete 2024-06-26T07:04:06.9481381Z 44ba272eae20: Verifying Checksum 2024-06-26T07:04:06.9481973Z 44ba272eae20: Download complete 2024-06-26T07:04:11.4735278Z da02d278e524: Verifying Checksum 2024-06-26T07:04:11.4735875Z da02d278e524: Download complete 2024-06-26T07:04:11.5464786Z 73a0ce681c29: Verifying Checksum 2024-06-26T07:04:11.5465245Z 73a0ce681c29: Download complete 2024-06-26T07:04:11.6168552Z 49737bd5ae70: Download complete 2024-06-26T07:04:12.0255550Z 9cd4c745f8e4: Verifying Checksum 2024-06-26T07:04:12.0256010Z 9cd4c745f8e4: Download complete 2024-06-26T07:04:12.1025779Z b2950a036534: Download complete 2024-06-26T07:04:12.1751974Z 25e19d81d9fc: Verifying Checksum 2024-06-26T07:04:12.5057477Z 60156b40fcbe: Verifying Checksum 2024-06-26T07:04:12.5057937Z 60156b40fcbe: Download complete 2024-06-26T07:04:12.5798600Z 7a868239227f: Verifying Checksum 2024-06-26T07:04:12.5799264Z 7a868239227f: Download complete 2024-06-26T07:04:12.8638110Z 83911d7c7ecf: Verifying Checksum 2024-06-26T07:04:12.8638584Z 83911d7c7ecf: Download complete 2024-06-26T07:04:12.9420268Z b3c5d66c679f: Verifying Checksum 2024-06-26T07:04:12.9420689Z b3c5d66c679f: Download complete 2024-06-26T07:04:16.9536184Z 7ed32bc8e469: Verifying Checksum 2024-06-26T07:04:16.9536675Z 7ed32bc8e469: Download complete 2024-06-26T07:04:17.0512079Z c1068d03a811: Verifying Checksum 2024-06-26T07:04:17.0512733Z c1068d03a811: Download complete 2024-06-26T07:04:17.1367717Z 671b4a301d81: Download complete 2024-06-26T07:04:17.6426769Z a8c1e85b5e14: Pull complete 2024-06-26T07:04:17.6780797Z 67760cf5ec46: Verifying Checksum 2024-06-26T07:04:17.6781288Z 67760cf5ec46: Download complete 2024-06-26T07:04:17.7560172Z ddc9740bcea2: Download complete 2024-06-26T07:04:17.7958576Z a41a8d1c11c8: Pull complete 2024-06-26T07:04:17.8220202Z fb2e97768560: Download complete 2024-06-26T07:04:17.8915805Z b599c5a8c7cb: Download complete 2024-06-26T07:04:17.9553373Z 0c1227890755: Pull complete 2024-06-26T07:04:17.9772295Z e08e88c13668: Download complete 2024-06-26T07:04:18.1941363Z d8d1234baab3: Pull complete 2024-06-26T07:04:23.7850212Z 3d914950f8ad: Verifying Checksum 2024-06-26T07:04:23.7851043Z 3d914950f8ad: Download complete 2024-06-26T07:04:23.8581543Z cf7c2cb896cb: Verifying Checksum 2024-06-26T07:04:23.8583915Z cf7c2cb896cb: Download complete 2024-06-26T07:04:23.9467118Z 3a778ddfac2a: Verifying Checksum 2024-06-26T07:04:23.9467707Z 3a778ddfac2a: Download complete 2024-06-26T07:04:24.0184745Z 236c8f0245f4: Verifying Checksum 2024-06-26T07:04:24.0185239Z 236c8f0245f4: Download complete 2024-06-26T07:04:24.0265986Z 4f4fb700ef54: Verifying Checksum 2024-06-26T07:04:24.0266723Z 4f4fb700ef54: Download complete 2024-06-26T07:04:24.1092239Z dda1ce302b9a: Verifying Checksum 2024-06-26T07:04:24.1092840Z dda1ce302b9a: Download complete 2024-06-26T07:04:24.1852306Z e797fe86caef: Verifying Checksum 2024-06-26T07:04:24.1852772Z e797fe86caef: Download complete 2024-06-26T07:04:24.3161648Z 967190076fc0: Verifying Checksum 2024-06-26T07:04:24.4006920Z 4fb0a83f0685: Verifying Checksum 2024-06-26T07:04:24.4007842Z 4fb0a83f0685: Download complete 2024-06-26T07:04:24.4807536Z 68af2ec4e30a: Verifying Checksum 2024-06-26T07:04:24.4808097Z 68af2ec4e30a: Download complete 2024-06-26T07:04:24.5656361Z cc91933e985a: Download complete 2024-06-26T07:04:24.6404986Z 9f3187916e7d: Download complete 2024-06-26T07:04:24.7763636Z 526083994857: Verifying Checksum 2024-06-26T07:04:24.7764182Z 526083994857: Download complete 2024-06-26T07:04:24.8705759Z 05a6f6dcc1c0: Verifying Checksum 2024-06-26T07:04:24.8706355Z 05a6f6dcc1c0: Download complete 2024-06-26T07:04:25.4710635Z 6bc9dc8315b0: Verifying Checksum 2024-06-26T07:04:25.4711121Z 6bc9dc8315b0: Download complete 2024-06-26T07:04:25.5442863Z 1b0e27c35f07: Verifying Checksum 2024-06-26T07:04:25.5443518Z 1b0e27c35f07: Download complete 2024-06-26T07:04:48.6276838Z 12ba7431ff38: Verifying Checksum 2024-06-26T07:04:48.6278344Z 12ba7431ff38: Download complete 2024-06-26T07:04:48.7030755Z 789eff6c51aa: Verifying Checksum 2024-06-26T07:04:48.7031430Z 789eff6c51aa: Download complete 2024-06-26T07:04:48.7706878Z de0f8a8c45c6: Verifying Checksum 2024-06-26T07:04:48.7707423Z de0f8a8c45c6: Download complete 2024-06-26T07:04:50.0288853Z 2060028f7488: Verifying Checksum 2024-06-26T07:04:50.0289383Z 2060028f7488: Download complete 2024-06-26T07:04:50.1088247Z e412c7f3b2de: Download complete 2024-06-26T07:04:54.1100421Z 95237e7311a1: Verifying Checksum 2024-06-26T07:04:54.1100897Z 95237e7311a1: Download complete 2024-06-26T07:05:16.7643685Z 84b2c7a89982: Verifying Checksum 2024-06-26T07:05:16.7644147Z 84b2c7a89982: Download complete 2024-06-26T07:05:34.3527123Z 7ed32bc8e469: Pull complete 2024-06-26T07:05:34.5612422Z ec1e7978c1fe: Pull complete 2024-06-26T07:05:34.7734610Z 309b872da805: Pull complete 2024-06-26T07:05:42.0218911Z e82932cb002c: Pull complete 2024-06-26T07:05:42.2201931Z ca9dd73e3e93: Pull complete 2024-06-26T07:05:42.4489231Z 16d76a79d5d6: Pull complete 2024-06-26T07:05:42.6591312Z 4bdf41c9ecbe: Pull complete 2024-06-26T07:05:45.0375595Z f4863801e30c: Pull complete 2024-06-26T07:05:45.2421379Z d37dd9ed8948: Pull complete 2024-06-26T07:05:45.3340059Z cf2f34f225db: Pull complete 2024-06-26T07:05:45.3801939Z df546d7b3e34: Pull complete 2024-06-26T07:06:34.8654190Z 3d914950f8ad: Pull complete 2024-06-26T07:06:35.0124103Z 96d0e2032184: Pull complete 2024-06-26T07:06:35.1629274Z 06e12ce59790: Pull complete 2024-06-26T07:06:35.2984507Z ed712f63cde7: Pull complete 2024-06-26T07:06:35.3572739Z a97a71ef8a4e: Pull complete 2024-06-26T07:06:35.4665198Z 25cd0c482ad6: Pull complete 2024-06-26T07:06:38.8493971Z 9bf94bc45658: Pull complete 2024-06-26T07:06:38.9705697Z de45a14c9b22: Pull complete 2024-06-26T07:06:39.0915797Z e99241e77209: Pull complete 2024-06-26T07:06:39.2784207Z 35e8f4e2a42f: Pull complete 2024-06-26T07:06:39.4952583Z a823ffa4c541: Pull complete 2024-06-26T07:06:39.6993818Z 44ba272eae20: Pull complete 2024-06-26T07:06:48.0895528Z da02d278e524: Pull complete 2024-06-26T07:06:48.3027097Z 73a0ce681c29: Pull complete 2024-06-26T07:06:48.4582852Z 49737bd5ae70: Pull complete 2024-06-26T07:06:49.3532854Z 9cd4c745f8e4: Pull complete 2024-06-26T07:06:49.4857340Z b2950a036534: Pull complete 2024-06-26T07:06:49.6115115Z 25e19d81d9fc: Pull complete 2024-06-26T07:06:50.0749975Z 60156b40fcbe: Pull complete 2024-06-26T07:06:50.2989289Z 7a868239227f: Pull complete 2024-06-26T07:06:50.6625483Z 83911d7c7ecf: Pull complete 2024-06-26T07:06:50.8649301Z b3c5d66c679f: Pull complete 2024-06-26T07:07:29.5116777Z 84b2c7a89982: Pull complete 2024-06-26T07:07:29.6482066Z c1068d03a811: Pull complete 2024-06-26T07:07:29.6767455Z 671b4a301d81: Pull complete 2024-06-26T07:07:31.6179527Z 67760cf5ec46: Pull complete 2024-06-26T07:07:31.6523847Z ddc9740bcea2: Pull complete 2024-06-26T07:07:31.6886084Z fb2e97768560: Pull complete 2024-06-26T07:07:31.7810419Z b599c5a8c7cb: Pull complete 2024-06-26T07:07:31.8166500Z e08e88c13668: Pull complete 2024-06-26T07:08:06.5449718Z 12ba7431ff38: Pull complete 2024-06-26T07:08:06.7700766Z cf7c2cb896cb: Pull complete 2024-06-26T07:08:06.9914761Z 3a778ddfac2a: Pull complete 2024-06-26T07:08:07.3388176Z 236c8f0245f4: Pull complete 2024-06-26T07:08:07.5746475Z 4f4fb700ef54: Pull complete 2024-06-26T07:08:07.8045535Z dda1ce302b9a: Pull complete 2024-06-26T07:08:08.0411437Z e797fe86caef: Pull complete 2024-06-26T07:08:08.4352489Z 967190076fc0: Pull complete 2024-06-26T07:08:08.6701670Z 4fb0a83f0685: Pull complete 2024-06-26T07:08:08.8967626Z 68af2ec4e30a: Pull complete 2024-06-26T07:08:09.1411277Z cc91933e985a: Pull complete 2024-06-26T07:08:09.3522331Z 9f3187916e7d: Pull complete 2024-06-26T07:08:10.4630287Z 526083994857: Pull complete 2024-06-26T07:08:10.6366768Z 05a6f6dcc1c0: Pull complete 2024-06-26T07:08:12.7335595Z 6bc9dc8315b0: Pull complete 2024-06-26T07:08:12.9479568Z 1b0e27c35f07: Pull complete 2024-06-26T07:08:26.2737145Z 95237e7311a1: Pull complete 2024-06-26T07:08:26.3420456Z 789eff6c51aa: Pull complete 2024-06-26T07:08:26.3724570Z de0f8a8c45c6: Pull complete 2024-06-26T07:08:27.0766849Z 2060028f7488: Pull complete 2024-06-26T07:08:27.1287701Z e412c7f3b2de: Pull complete 2024-06-26T07:08:27.3644145Z Digest: sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:08:27.3677397Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:08:27.3703185Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:08:27.3755878Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:08:27.3756787Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:08:27.3764762Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:27.3765265Z env: 2024-06-26T07:08:27.3765532Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:27.3765874Z ##[endgroup] 2024-06-26T07:08:27.3918372Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-26T07:08:27.3918916Z with: 2024-06-26T07:08:27.3919204Z driver-version: 550.54.15 2024-06-26T07:08:27.3919551Z env: 2024-06-26T07:08:27.3919828Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:27.3920183Z ##[endgroup] 2024-06-26T07:08:27.3959280Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:08:27.3959832Z with: 2024-06-26T07:08:27.3960124Z timeout_minutes: 10 2024-06-26T07:08:27.3960454Z max_attempts: 3 2024-06-26T07:08:27.3991618Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-26T07:08:27.4026171Z retry_wait_seconds: 10 2024-06-26T07:08:27.4026538Z polling_interval_seconds: 1 2024-06-26T07:08:27.4026912Z warning_on_retry: true 2024-06-26T07:08:27.4027268Z continue_on_error: false 2024-06-26T07:08:27.4027685Z env: 2024-06-26T07:08:27.4027963Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:27.4028320Z DRIVER_VERSION: 550.54.15 2024-06-26T07:08:27.4028673Z ##[endgroup] 2024-06-26T07:08:27.4548813Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-26T07:08:27.4549755Z + pre_install_nvidia_driver_amzn2 2024-06-26T07:08:27.4550437Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-26T07:08:27.7550161Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:08:27.7943539Z No Match for argument: nvidia-driver-latest-dkms 2024-06-26T07:08:27.8184394Z No Packages marked for removal 2024-06-26T07:08:27.8306577Z + install_nvidia_driver_common 2024-06-26T07:08:27.8308842Z + echo 'Before installing NVIDIA driver' 2024-06-26T07:08:27.8309366Z + lspci 2024-06-26T07:08:27.8309664Z Before installing NVIDIA driver 2024-06-26T07:08:27.8385228Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:08:27.8386226Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:08:27.8387160Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:08:27.8387955Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:08:27.8388715Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:08:27.8389457Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:08:27.8390178Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:08:27.8390950Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:08:27.8391534Z + lsmod 2024-06-26T07:08:27.8399603Z Module Size Used by 2024-06-26T07:08:27.8400015Z ib_core 266240 0 2024-06-26T07:08:27.8400394Z nvidia_modeset 1339392 0 2024-06-26T07:08:27.8400766Z veth 16384 0 2024-06-26T07:08:27.8401212Z nvidia_uvm 4599808 0 2024-06-26T07:08:27.8401668Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:08:27.8402321Z drm 421888 1 nvidia 2024-06-26T07:08:27.8402755Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:08:27.8403215Z backlight 16384 1 nvidia_modeset 2024-06-26T07:08:27.8403650Z xt_conntrack 16384 1 2024-06-26T07:08:27.8404023Z ipt_MASQUERADE 16384 1 2024-06-26T07:08:27.8404454Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:08:27.8404916Z nf_conntrack_netlink 49152 0 2024-06-26T07:08:27.8405379Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:08:27.8405892Z xfrm_user 45056 1 2024-06-26T07:08:27.8406281Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:08:27.8406685Z iptable_nat 16384 1 2024-06-26T07:08:27.8407131Z nf_conntrack_ipv4 16384 3 2024-06-26T07:08:27.8407556Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:08:27.8408037Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:08:27.8408561Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T07:08:27.8409500Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T07:08:27.8410332Z xt_addrtype 16384 2 2024-06-26T07:08:27.8410757Z iptable_filter 16384 1 2024-06-26T07:08:27.8411395Z br_netfilter 24576 0 2024-06-26T07:08:27.8411802Z bridge 172032 1 br_netfilter 2024-06-26T07:08:27.8412234Z stp 16384 1 bridge 2024-06-26T07:08:27.8412651Z llc 16384 2 bridge,stp 2024-06-26T07:08:27.8413063Z overlay 86016 0 2024-06-26T07:08:27.8413509Z sunrpc 393216 1 2024-06-26T07:08:27.8413878Z dm_mirror 28672 0 2024-06-26T07:08:27.8414262Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:08:27.8414746Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:08:27.8415396Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:08:27.8415850Z dax 69632 1 dm_mod 2024-06-26T07:08:27.8416247Z crc32_pclmul 16384 0 2024-06-26T07:08:27.8416628Z ghash_clmulni_intel 16384 0 2024-06-26T07:08:27.8417040Z pcbc 16384 0 2024-06-26T07:08:27.8417413Z aesni_intel 188416 0 2024-06-26T07:08:27.8417801Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:08:27.8418252Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:08:27.8418702Z glue_helper 16384 1 aesni_intel 2024-06-26T07:08:27.8419253Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:08:27.8419781Z mousedev 24576 0 2024-06-26T07:08:27.8420149Z evdev 20480 3 2024-06-26T07:08:27.8420518Z psmouse 32768 0 2024-06-26T07:08:27.8420884Z button 16384 0 2024-06-26T07:08:27.8421241Z ena 139264 0 2024-06-26T07:08:27.8421616Z ptp 20480 1 ena 2024-06-26T07:08:27.8422114Z pps_core 20480 1 ptp 2024-06-26T07:08:27.8422607Z crc32c_intel 24576 0 2024-06-26T07:08:27.8422985Z autofs4 49152 2 2024-06-26T07:08:27.8423419Z + modinfo nvidia 2024-06-26T07:08:27.8424059Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:08:27.8424789Z alias: char-major-195-* 2024-06-26T07:08:27.8425173Z version: 550.54.15 2024-06-26T07:08:27.8425512Z supported: external 2024-06-26T07:08:27.8425845Z license: NVIDIA 2024-06-26T07:08:27.8426214Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:08:27.8426683Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:08:27.8427139Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:08:27.8427596Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:08:27.8428099Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:08:27.8428584Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:08:27.8429172Z depends: i2c-core,drm 2024-06-26T07:08:27.8429534Z retpoline: Y 2024-06-26T07:08:27.8429832Z name: nvidia 2024-06-26T07:08:27.8430380Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:08:27.8431020Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:08:27.8431654Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:08:27.8432228Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:08:27.8432670Z parm: NVreg_RmLogonRC:int 2024-06-26T07:08:27.8433093Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:08:27.8433540Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:08:27.8433956Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:08:27.8434391Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:08:27.8434907Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:08:27.8435468Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:08:27.8435938Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:08:27.8436361Z parm: NVreg_EnableMSI:int 2024-06-26T07:08:27.8436770Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:08:27.8437222Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:08:27.8437733Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:08:27.8438301Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:08:27.8438852Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:08:27.8439439Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:08:27.8440110Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:08:27.8440715Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:08:27.8441300Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:08:27.8441774Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:08:27.8442395Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:08:27.8442939Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:08:27.8443423Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:08:27.8443876Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:08:27.8444351Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:08:27.8444816Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:08:27.8445251Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:08:27.8445734Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:08:27.8446247Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:08:27.8446717Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:08:27.8447192Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:08:27.8447679Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:08:27.8448155Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:08:27.8448654Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:08:27.8449130Z parm: NVreg_RmMsg:charp 2024-06-26T07:08:27.8449534Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:08:27.8449993Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:08:27.8450454Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:08:27.8451158Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:08:27.8451647Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:08:27.8452118Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:08:27.8452567Z parm: rm_firmware_active:charp 2024-06-26T07:08:27.8452970Z + HAS_NVIDIA_DRIVER=0 2024-06-26T07:08:27.8453345Z ++ command -v nvidia-smi 2024-06-26T07:08:27.8453730Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-26T07:08:27.8454091Z + set +e 2024-06-26T07:08:27.8454618Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-26T07:08:29.8411197Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-26T07:08:29.8411801Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:08:29.8412327Z + '[' 0 -ne 0 ']' 2024-06-26T07:08:29.8412975Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-26T07:08:29.8413357Z + HAS_NVIDIA_DRIVER=1 2024-06-26T07:08:29.8414071Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-26T07:08:29.8414774Z + set -e 2024-06-26T07:08:29.8415074Z + '[' 1 -eq 0 ']' 2024-06-26T07:08:29.8415637Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-26T07:08:29.8416318Z + post_install_nvidia_driver_common 2024-06-26T07:08:29.8416726Z + sudo modprobe nvidia 2024-06-26T07:08:29.8512264Z + echo 'After installing NVIDIA driver' 2024-06-26T07:08:29.8512752Z + lspci 2024-06-26T07:08:29.8513056Z After installing NVIDIA driver 2024-06-26T07:08:29.8593723Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:08:29.8594481Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:08:29.8595423Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:08:29.8596234Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:08:29.8596973Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:08:29.8597726Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:08:29.8598442Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:08:29.8599214Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:08:29.8599800Z + lsmod 2024-06-26T07:08:29.8607657Z Module Size Used by 2024-06-26T07:08:29.8608093Z ib_core 266240 0 2024-06-26T07:08:29.8608566Z nvidia_modeset 1339392 0 2024-06-26T07:08:29.8608983Z veth 16384 0 2024-06-26T07:08:29.8609369Z nvidia_uvm 4599808 0 2024-06-26T07:08:29.8609818Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:08:29.8610660Z drm 421888 1 nvidia 2024-06-26T07:08:29.8611459Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:08:29.8611932Z backlight 16384 1 nvidia_modeset 2024-06-26T07:08:29.8612379Z xt_conntrack 16384 1 2024-06-26T07:08:29.8612765Z ipt_MASQUERADE 16384 1 2024-06-26T07:08:29.8613203Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:08:29.8613676Z nf_conntrack_netlink 49152 0 2024-06-26T07:08:29.8614124Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:08:29.8614588Z xfrm_user 45056 1 2024-06-26T07:08:29.8614974Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:08:29.8615396Z iptable_nat 16384 1 2024-06-26T07:08:29.8615780Z nf_conntrack_ipv4 16384 3 2024-06-26T07:08:29.8616217Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:08:29.8616799Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:08:29.8617369Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T07:08:29.8618561Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T07:08:29.8619444Z xt_addrtype 16384 2 2024-06-26T07:08:29.8619829Z iptable_filter 16384 1 2024-06-26T07:08:29.8620199Z br_netfilter 24576 0 2024-06-26T07:08:29.8620606Z bridge 172032 1 br_netfilter 2024-06-26T07:08:29.8621044Z stp 16384 1 bridge 2024-06-26T07:08:29.8621459Z llc 16384 2 bridge,stp 2024-06-26T07:08:29.8621865Z overlay 86016 0 2024-06-26T07:08:29.8622236Z sunrpc 393216 1 2024-06-26T07:08:29.8622764Z dm_mirror 28672 0 2024-06-26T07:08:29.8623157Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:08:29.8623646Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:08:29.8624143Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:08:29.8624590Z dax 69632 1 dm_mod 2024-06-26T07:08:29.8625180Z crc32_pclmul 16384 0 2024-06-26T07:08:29.8625568Z ghash_clmulni_intel 16384 0 2024-06-26T07:08:29.8626038Z pcbc 16384 0 2024-06-26T07:08:29.8626402Z aesni_intel 188416 0 2024-06-26T07:08:29.8626783Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:08:29.8627227Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:08:29.8627674Z glue_helper 16384 1 aesni_intel 2024-06-26T07:08:29.8628228Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:08:29.8628754Z mousedev 24576 0 2024-06-26T07:08:29.8629119Z evdev 20480 3 2024-06-26T07:08:29.8629480Z psmouse 32768 0 2024-06-26T07:08:29.8629847Z button 16384 0 2024-06-26T07:08:29.8630200Z ena 139264 0 2024-06-26T07:08:29.8630572Z ptp 20480 1 ena 2024-06-26T07:08:29.8630971Z pps_core 20480 1 ptp 2024-06-26T07:08:29.8631362Z crc32c_intel 24576 0 2024-06-26T07:08:29.8631727Z autofs4 49152 2 2024-06-26T07:08:29.8632080Z + modinfo nvidia 2024-06-26T07:08:29.8632735Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:08:29.8633413Z alias: char-major-195-* 2024-06-26T07:08:29.8633795Z version: 550.54.15 2024-06-26T07:08:29.8634134Z supported: external 2024-06-26T07:08:29.8634466Z license: NVIDIA 2024-06-26T07:08:29.8634831Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:08:29.8635312Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:08:29.8635768Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:08:29.8636220Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:08:29.8636711Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:08:29.8637195Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:08:29.8637764Z depends: i2c-core,drm 2024-06-26T07:08:29.8638114Z retpoline: Y 2024-06-26T07:08:29.8638425Z name: nvidia 2024-06-26T07:08:29.8638970Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:08:29.8639615Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:08:29.8640242Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:08:29.8640829Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:08:29.8641269Z parm: NVreg_RmLogonRC:int 2024-06-26T07:08:29.8641694Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:08:29.8642132Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:08:29.8642560Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:08:29.8642990Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:08:29.8643509Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:08:29.8644060Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:08:29.8644542Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:08:29.8644974Z parm: NVreg_EnableMSI:int 2024-06-26T07:08:29.8645377Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:08:29.8645832Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:08:29.8646352Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:08:29.8646933Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:08:29.8647479Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:08:29.8648087Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:08:29.8648679Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:08:29.8649283Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:08:29.8649859Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:08:29.8650344Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:08:29.8651096Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:08:29.8651651Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:08:29.8652214Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:08:29.8652679Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:08:29.8653157Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:08:29.8653631Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:08:29.8654069Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:08:29.8654561Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:08:29.8655082Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:08:29.8655549Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:08:29.8656021Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:08:29.8656514Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:08:29.8656991Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:08:29.8657482Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:08:29.8657954Z parm: NVreg_RmMsg:charp 2024-06-26T07:08:29.8658361Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:08:29.8658833Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:08:29.8659291Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:08:29.8659740Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:08:29.8660213Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:08:29.8660683Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:08:29.8661126Z parm: rm_firmware_active:charp 2024-06-26T07:08:29.8661531Z + set +e 2024-06-26T07:08:29.8661833Z + nvidia-smi 2024-06-26T07:08:31.4233438Z Wed Jun 26 07:08:31 2024 2024-06-26T07:08:31.4234372Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:08:31.4235304Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-26T07:08:31.4236089Z |-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:08:31.4237120Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-26T07:08:31.4238003Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-26T07:08:31.4238685Z | | | MIG M. | 2024-06-26T07:08:31.4239226Z |=========================================+========================+======================| 2024-06-26T07:08:31.4327679Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-06-26T07:08:31.4328439Z | 0% 31C P0 58W / 300W | 0MiB / 23028MiB | 0% Default | 2024-06-26T07:08:31.4329362Z | | | N/A | 2024-06-26T07:08:31.4330180Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:08:31.4331109Z 2024-06-26T07:08:31.4332013Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:08:31.4332715Z | Processes: | 2024-06-26T07:08:31.4333419Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-26T07:08:31.4334114Z | ID ID Usage | 2024-06-26T07:08:31.4334677Z |=========================================================================================| 2024-06-26T07:08:31.4335323Z | No running processes found | 2024-06-26T07:08:31.4336078Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:08:32.0671937Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-26T07:08:33.6310486Z NVIDIA A10G 2024-06-26T07:08:34.0845852Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:08:34.0846901Z + '[' 0 -eq 0 ']' 2024-06-26T07:08:34.0847388Z + echo 'INFO: Ignoring allowed status 0' 2024-06-26T07:08:34.0847813Z + set -e 2024-06-26T07:08:34.0848105Z INFO: Ignoring allowed status 0 2024-06-26T07:08:34.0850049Z == Installing nvidia container toolkit for amzn2 == 2024-06-26T07:08:34.0852623Z + sudo yum install -y yum-utils 2024-06-26T07:08:34.3870684Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:08:35.8830714Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-26T07:08:35.8831407Z Nothing to do 2024-06-26T07:08:36.0324100Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-06-26T07:08:36.0325132Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:08:36.0326547Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:08:36.3555776Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:08:36.3808548Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:08:36.3809728Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:08:36.3810675Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:08:36.3923017Z + sudo yum install -y nvidia-docker2 2024-06-26T07:08:36.6934988Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:08:38.1681607Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-26T07:08:38.1682879Z Nothing to do 2024-06-26T07:08:38.3187876Z + sudo systemctl restart docker 2024-06-26T07:09:18.5017829Z Command completed after 1 attempt(s). 2024-06-26T07:09:18.5074696Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:09:18.5075452Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:09:18.5076261Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T07:09:18.5076908Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:09:18.5084643Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:18.5085137Z env: 2024-06-26T07:09:18.5085413Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:18.5085863Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:18.5086345Z ##[endgroup] 2024-06-26T07:09:18.7293724Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:09:18.7459328Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-06-26T07:09:18.7560330Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-06-26T07:09:18.8654033Z Prepare all required actions 2024-06-26T07:09:18.8654523Z Getting action download info 2024-06-26T07:09:19.0080994Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T07:09:19.2250196Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T07:09:19.3437956Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T07:09:19.3438438Z with: 2024-06-26T07:09:19.3438772Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:09:19.3439216Z s3-bucket: gha-artifacts 2024-06-26T07:09:19.3439556Z env: 2024-06-26T07:09:19.3439826Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:19.3440256Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:19.3440733Z ##[endgroup] 2024-06-26T07:09:19.3471831Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:09:19.3472267Z with: 2024-06-26T07:09:19.3472636Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:09:19.3473092Z s3-bucket: gha-artifacts 2024-06-26T07:09:19.3473446Z region: us-east-1 2024-06-26T07:09:19.3473745Z env: 2024-06-26T07:09:19.3474017Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:19.3474457Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:19.3474936Z ##[endgroup] 2024-06-26T07:09:19.7580001Z (node:10144) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:09:19.7580697Z 2024-06-26T07:09:19.7580955Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:09:19.7581680Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:09:19.7582769Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:09:19.8346261Z Found 1 objects with prefix pytorch/pytorch/9673646808/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-06-26T07:09:19.8347982Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:09:27.3269431Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:09:27.3275590Z Artifact download has finished successfully 2024-06-26T07:09:27.3435869Z ##[group]Run unzip -o artifacts.zip 2024-06-26T07:09:27.3436303Z unzip -o artifacts.zip 2024-06-26T07:09:27.3443872Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:27.3444375Z env: 2024-06-26T07:09:27.3444644Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:27.3445093Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:27.3445572Z ##[endgroup] 2024-06-26T07:09:27.3473928Z Archive: artifacts.zip 2024-06-26T07:09:27.3475643Z creating: dist/ 2024-06-26T07:09:29.3203011Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:09:29.3204322Z creating: build/custom_test_artifacts/ 2024-06-26T07:09:29.3205458Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T07:09:29.3206863Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T07:09:29.3208799Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:09:29.3210690Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:09:29.3212842Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T07:09:29.3213992Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:09:29.3214963Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:09:29.3215902Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:09:29.3216986Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:09:29.3218189Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:09:29.3219176Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:09:29.3220151Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:09:29.3221277Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:09:29.3222416Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:09:29.3223700Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:09:29.3224820Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:09:29.3225949Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:09:29.3227101Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:09:29.3228124Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:09:29.3229116Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:09:29.3262742Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:09:29.3301294Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:09:29.3303581Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:09:29.3344505Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:09:29.3346116Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:09:29.3347744Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:09:29.3349420Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:09:29.3351205Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:09:29.3352583Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:09:29.3353952Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:09:29.3355322Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:09:29.3356644Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:09:29.3358027Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:09:29.3359245Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:09:29.3360445Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:09:29.3361653Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:09:29.3362817Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:09:29.3364060Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:09:29.3411252Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:09:29.3469087Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:09:29.3470265Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:09:29.3471242Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:09:29.3472094Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:09:29.3472991Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T07:09:29.3473906Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T07:09:29.3474920Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T07:09:29.3476052Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T07:09:29.3477142Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T07:09:29.3478166Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T07:09:29.3479223Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T07:09:29.3480289Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T07:09:29.3481342Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T07:09:29.3482399Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T07:09:29.3483444Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T07:09:29.3497035Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T07:09:29.3624890Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T07:09:29.3625868Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T07:09:29.3626948Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T07:09:29.3628133Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T07:09:29.3629285Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T07:09:29.3630354Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T07:09:29.3631467Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T07:09:29.3632590Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T07:09:29.3633720Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T07:09:29.3634975Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T07:09:29.3636066Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T07:09:29.3650533Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T07:09:29.3730680Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T07:09:29.3733177Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:09:29.3734282Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:09:29.3735338Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T07:09:29.3736242Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T07:09:29.3737136Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T07:09:29.3738026Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-26T07:09:29.3738858Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T07:09:29.3739621Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T07:09:29.3740420Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T07:09:29.3847138Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T07:09:29.3906655Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T07:09:29.3907370Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T07:09:29.3908071Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T07:09:29.3908876Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:09:29.3914406Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:09:29.3915319Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T07:09:29.3916220Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:09:29.3917185Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:09:29.3918130Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:09:29.3919202Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:09:29.3920277Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:09:29.3921246Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:09:29.3922222Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:09:29.3923354Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:09:29.3924483Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:09:29.3925580Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:09:29.3926690Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:09:29.3928439Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:09:29.3930162Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:09:29.3931961Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:09:29.3933551Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:09:29.3969489Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:09:29.4008081Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:09:29.4010164Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:09:29.4052999Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:09:29.4055055Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:09:29.4057292Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:09:29.4059462Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:09:29.4061530Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:09:29.4063739Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:09:29.4065810Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:09:29.4067776Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:09:29.4069706Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:09:29.4071560Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:09:29.4073336Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:09:29.4075044Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:09:29.4076827Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:09:29.4078552Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:09:29.4080270Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:09:29.4125435Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:09:29.4184259Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:09:29.4186012Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:09:29.4187459Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:09:29.4188709Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:09:29.4190011Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T07:09:29.4191386Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T07:09:29.4192890Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T07:09:29.4194560Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T07:09:29.4196193Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T07:09:29.4197702Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T07:09:29.4199406Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T07:09:29.4200951Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T07:09:29.4202513Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T07:09:29.4204074Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T07:09:29.4205605Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T07:09:29.4213786Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T07:09:29.4277448Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T07:09:29.4279217Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:09:29.4280780Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:09:29.4282158Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T07:09:29.4283428Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T07:09:29.4284716Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T07:09:29.4286039Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-26T07:09:29.4287254Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T07:09:29.4288369Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T07:09:29.4289472Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T07:09:29.4334740Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T07:09:29.4335831Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T07:09:29.4336887Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T07:09:29.4338090Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:09:29.4344113Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:09:29.4345527Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T07:09:29.4346974Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:09:29.4348486Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:09:29.4349975Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:09:29.4351664Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:09:29.4353353Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:09:29.4354872Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:09:29.4356398Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:09:29.4358148Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:09:29.4359952Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:09:29.4361672Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:09:29.4363349Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:09:29.4365254Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:09:29.4366965Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:09:29.4368550Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:09:29.4370032Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:09:29.4400024Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:09:29.4437763Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:09:29.4439425Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:09:29.4481928Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:09:29.4483686Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:09:29.4486636Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:09:29.4489668Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:09:29.4492892Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:09:29.4503713Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:09:29.4505556Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:09:29.4507351Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:09:29.4508862Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:09:29.4510178Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:09:29.4511448Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:09:29.4512702Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:09:29.4535919Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:09:29.4537230Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:09:29.4538499Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:09:29.4567995Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:09:29.4625745Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:09:29.4627017Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:09:29.4628069Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:09:29.4628991Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:09:29.4629962Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T07:09:29.4631202Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T07:09:29.4632358Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T07:09:29.4633640Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T07:09:29.4634865Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T07:09:29.4636014Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T07:09:29.4637213Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T07:09:29.4638570Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T07:09:29.4639761Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T07:09:29.4640970Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T07:09:29.4642139Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T07:09:29.4643392Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T07:09:29.4755312Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T07:09:29.4756477Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T07:09:29.4757676Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T07:09:29.4758999Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T07:09:29.4760262Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T07:09:29.4761454Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T07:09:29.4762686Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T07:09:29.4763925Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T07:09:29.4765165Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T07:09:29.4766401Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T07:09:29.4767627Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T07:09:29.4778830Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T07:09:29.4831608Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T07:09:29.4833004Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:09:29.4835284Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:09:29.4837301Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T07:09:29.4839182Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T07:09:29.4841092Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T07:09:29.4842916Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-26T07:09:29.4843952Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T07:09:29.4844797Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T07:09:29.4845649Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T07:09:29.4936852Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T07:09:29.4977984Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T07:09:29.4978630Z creating: build/lib/ 2024-06-26T07:09:29.5063135Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T07:09:29.5501512Z inflating: build/lib/libprotobuf.a 2024-06-26T07:09:29.5511173Z inflating: build/lib/libpthreadpool.a 2024-06-26T07:09:29.5518649Z inflating: build/lib/libcpuinfo.a 2024-06-26T07:09:29.5526273Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T07:09:29.5526780Z inflating: build/lib/libclog.a 2024-06-26T07:09:29.5544307Z inflating: build/lib/libnnpack.a 2024-06-26T07:09:29.5546314Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T07:09:29.5608371Z inflating: build/lib/libgtest.a 2024-06-26T07:09:29.5680389Z inflating: build/lib/libbenchmark.a 2024-06-26T07:09:29.5742021Z inflating: build/lib/libasmjit.a 2024-06-26T07:09:29.5749167Z inflating: build/lib/libittnotify.a 2024-06-26T07:09:29.5775751Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T07:09:29.5898534Z inflating: build/lib/libgloo.a 2024-06-26T07:09:29.5918722Z inflating: build/lib/libfmt.a 2024-06-26T07:09:29.6008075Z inflating: build/lib/libc10.so 2024-06-26T07:09:29.6009198Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T07:09:29.6010279Z inflating: build/lib/libfoxi_loader.a 2024-06-26T07:09:29.6011843Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T07:09:29.6029554Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T07:09:29.6046703Z inflating: build/lib/libgmock.a 2024-06-26T07:09:29.6047457Z inflating: build/lib/libgtest_main.a 2024-06-26T07:09:29.6048045Z inflating: build/lib/libbenchmark_main.a 2024-06-26T07:09:29.6533880Z inflating: build/lib/libprotoc.a 2024-06-26T07:09:29.6903713Z inflating: build/lib/libgloo_cuda.a 2024-06-26T07:09:30.6629868Z inflating: build/lib/libdnnl.a 2024-06-26T07:09:30.7178535Z inflating: build/lib/libtensorpipe.a 2024-06-26T07:09:30.7231190Z inflating: build/lib/libc10_cuda.so 2024-06-26T07:09:30.7231645Z inflating: build/lib/libgmock_main.a 2024-06-26T07:09:30.8433941Z inflating: build/lib/libfbgemm.a 2024-06-26T07:09:30.8679273Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-26T07:09:30.9175300Z inflating: build/lib/libkineto.a 2024-06-26T07:09:30.9352990Z inflating: build/lib/libXNNPACK.a 2024-06-26T07:09:30.9393371Z inflating: build/lib/libonnx_proto.a 2024-06-26T07:09:31.0057185Z inflating: build/lib/libonnx.a 2024-06-26T07:09:33.3336223Z inflating: build/lib/libtorch_cpu.so 2024-06-26T07:09:33.3340118Z inflating: build/lib/libunbox_lib.a 2024-06-26T07:09:33.3344047Z inflating: build/lib/libshm.so 2024-06-26T07:09:35.3213243Z inflating: build/lib/libtorch_cuda.so 2024-06-26T07:09:35.3213750Z inflating: build/lib/libtorch.so 2024-06-26T07:09:35.3216340Z inflating: build/lib/libc10d_cuda_test.so 2024-06-26T07:09:36.1164222Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-26T07:09:36.3056564Z inflating: build/lib/libtorch_python.so 2024-06-26T07:09:36.3122358Z inflating: build/lib/libtorchbind_test.so 2024-06-26T07:09:36.3142472Z inflating: build/lib/libjitbackend_test.so 2024-06-26T07:09:36.3166766Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T07:09:36.3200097Z inflating: build/lib/libnnapi_backend.so 2024-06-26T07:09:36.3200572Z creating: build/bin/ 2024-06-26T07:09:36.3249427Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T07:09:36.3299045Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T07:09:36.3347806Z inflating: build/bin/c10_Device_test 2024-06-26T07:09:36.3405202Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T07:09:36.3457270Z inflating: build/bin/c10_Scalar_test 2024-06-26T07:09:36.3504757Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T07:09:36.3554189Z inflating: build/bin/c10_SymInt_test 2024-06-26T07:09:36.3606631Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T07:09:36.3660362Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T07:09:36.3714718Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T07:09:36.3783586Z inflating: build/bin/c10_cow_test 2024-06-26T07:09:36.3834661Z inflating: build/bin/c10_Bitset_test 2024-06-26T07:09:36.3881976Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T07:09:36.3930372Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T07:09:36.3979358Z inflating: build/bin/c10_Half_test 2024-06-26T07:09:36.4033737Z inflating: build/bin/c10_LeftRight_test 2024-06-26T07:09:36.4087318Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T07:09:36.4135358Z inflating: build/bin/c10_Synchronized_test 2024-06-26T07:09:36.4189116Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T07:09:36.4238944Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T07:09:36.4288549Z inflating: build/bin/c10_TypeList_test 2024-06-26T07:09:36.4336024Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T07:09:36.4386439Z inflating: build/bin/c10_accumulate_test 2024-06-26T07:09:36.4439948Z inflating: build/bin/c10_bfloat16_test 2024-06-26T07:09:36.4488898Z inflating: build/bin/c10_bit_cast_test 2024-06-26T07:09:36.4543591Z inflating: build/bin/c10_complex_math_test 2024-06-26T07:09:36.4597361Z inflating: build/bin/c10_complex_test 2024-06-26T07:09:36.4648345Z inflating: build/bin/c10_exception_test 2024-06-26T07:09:36.4697263Z inflating: build/bin/c10_flags_test 2024-06-26T07:09:36.4745776Z inflating: build/bin/c10_generic_math_test 2024-06-26T07:09:36.4904405Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T07:09:36.4954152Z inflating: build/bin/c10_irange_test 2024-06-26T07:09:36.5005453Z inflating: build/bin/c10_lazy_test 2024-06-26T07:09:36.5060916Z inflating: build/bin/c10_logging_test 2024-06-26T07:09:36.5133458Z inflating: build/bin/c10_optional_test 2024-06-26T07:09:36.5194333Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T07:09:36.5246336Z inflating: build/bin/c10_registry_test 2024-06-26T07:09:36.5392247Z inflating: build/bin/c10_small_vector_test 2024-06-26T07:09:36.5441797Z inflating: build/bin/c10_ssize_test 2024-06-26T07:09:36.5492494Z inflating: build/bin/c10_string_util_test 2024-06-26T07:09:36.5549680Z inflating: build/bin/c10_string_view_test 2024-06-26T07:09:36.5597750Z inflating: build/bin/c10_tempfile_test 2024-06-26T07:09:36.5644632Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T07:09:36.5698995Z inflating: build/bin/c10_typeid_test 2024-06-26T07:09:36.6132351Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T07:09:36.6564817Z inflating: build/bin/protoc 2024-06-26T07:09:36.6616022Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-26T07:09:36.6667459Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-26T07:09:36.6717978Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-26T07:09:36.6768287Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-26T07:09:36.6819316Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T07:09:36.6870605Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T07:09:36.6917769Z inflating: build/bin/c10_cuda_CUDATest 2024-06-26T07:09:36.6969503Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-26T07:09:36.7286889Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T07:09:36.7618884Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T07:09:36.7965599Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T07:09:36.8016777Z inflating: build/bin/FileStoreTest 2024-06-26T07:09:36.8071412Z inflating: build/bin/TCPStoreTest 2024-06-26T07:09:36.8123096Z inflating: build/bin/HashStoreTest 2024-06-26T07:09:36.8136145Z inflating: build/bin/ProcessGroupMPITest 2024-06-26T07:09:36.8188865Z inflating: build/bin/test_edge_op_registration 2024-06-26T07:09:36.8192770Z inflating: build/bin/torch_shm_manager 2024-06-26T07:09:36.8195296Z inflating: build/bin/example_allreduce 2024-06-26T07:09:36.8248152Z inflating: build/bin/test_dist_autograd 2024-06-26T07:09:36.8315554Z inflating: build/bin/test_cpp_rpc 2024-06-26T07:09:36.8317737Z inflating: build/bin/parallel_benchmark 2024-06-26T07:09:36.8383186Z inflating: build/bin/test_mobile_nnc 2024-06-26T07:09:36.8391925Z inflating: build/bin/aot_model_compiler_test 2024-06-26T07:09:36.8724181Z inflating: build/bin/test_lazy 2024-06-26T07:09:36.9868667Z inflating: build/bin/test_api 2024-06-26T07:09:36.9939747Z inflating: build/bin/Dict_test 2024-06-26T07:09:36.9990731Z inflating: build/bin/Dimname_test 2024-06-26T07:09:37.0052927Z inflating: build/bin/MaybeOwned_test 2024-06-26T07:09:37.0108670Z inflating: build/bin/NamedTensor_test 2024-06-26T07:09:37.0166814Z inflating: build/bin/apply_utils_test 2024-06-26T07:09:37.0223614Z inflating: build/bin/atest 2024-06-26T07:09:37.0284395Z inflating: build/bin/basic 2024-06-26T07:09:37.0337544Z inflating: build/bin/broadcast_test 2024-06-26T07:09:37.0386923Z inflating: build/bin/cpu_allocator_test 2024-06-26T07:09:37.0443281Z inflating: build/bin/cpu_generator_test 2024-06-26T07:09:37.0495136Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T07:09:37.0585309Z inflating: build/bin/cpu_rng_test 2024-06-26T07:09:37.0633642Z inflating: build/bin/dispatch_key_set_test 2024-06-26T07:09:37.0682335Z inflating: build/bin/dlconvertor_test 2024-06-26T07:09:37.0739267Z inflating: build/bin/extension_backend_test 2024-06-26T07:09:37.0792885Z inflating: build/bin/half_test 2024-06-26T07:09:37.0885472Z inflating: build/bin/ivalue_test 2024-06-26T07:09:37.0933442Z inflating: build/bin/lazy_tensor_test 2024-06-26T07:09:37.0986262Z inflating: build/bin/math_kernel_test 2024-06-26T07:09:37.1038711Z inflating: build/bin/memory_format_test 2024-06-26T07:09:37.1089930Z inflating: build/bin/memory_overlapping_test 2024-06-26T07:09:37.1141747Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T07:09:37.1195814Z inflating: build/bin/native_test 2024-06-26T07:09:37.1244650Z inflating: build/bin/operator_name_test 2024-06-26T07:09:37.1294765Z inflating: build/bin/operators_test 2024-06-26T07:09:37.1345005Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T07:09:37.1410158Z inflating: build/bin/pow_test 2024-06-26T07:09:37.1466108Z inflating: build/bin/quantized_test 2024-06-26T07:09:37.1514595Z inflating: build/bin/reduce_ops_test 2024-06-26T07:09:37.1563631Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T07:09:37.1618482Z inflating: build/bin/scalar_tensor_test 2024-06-26T07:09:37.1675049Z inflating: build/bin/scalar_test 2024-06-26T07:09:37.1725071Z inflating: build/bin/StorageUtils_test 2024-06-26T07:09:37.1776105Z inflating: build/bin/stride_properties_test 2024-06-26T07:09:37.1852529Z inflating: build/bin/tensor_iterator_test 2024-06-26T07:09:37.1904990Z inflating: build/bin/test_parallel 2024-06-26T07:09:37.1907675Z inflating: build/bin/thread_init_test 2024-06-26T07:09:37.1961846Z inflating: build/bin/type_ptr_test 2024-06-26T07:09:37.2019828Z inflating: build/bin/type_test 2024-06-26T07:09:37.2070676Z inflating: build/bin/undefined_tensor_test 2024-06-26T07:09:37.2071906Z inflating: build/bin/verify_api_visibility 2024-06-26T07:09:37.2139227Z inflating: build/bin/legacy_vmap_test 2024-06-26T07:09:37.2189820Z inflating: build/bin/weakref_test 2024-06-26T07:09:37.2239495Z inflating: build/bin/wrapdim_test 2024-06-26T07:09:37.2289524Z inflating: build/bin/xla_tensor_test 2024-06-26T07:09:37.2348138Z inflating: build/bin/IListRef_test 2024-06-26T07:09:37.2451562Z inflating: build/bin/List_test 2024-06-26T07:09:37.2515900Z inflating: build/bin/KernelFunction_test 2024-06-26T07:09:37.2634100Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T07:09:37.2727724Z inflating: build/bin/kernel_function_test 2024-06-26T07:09:37.2852853Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T07:09:37.2953718Z inflating: build/bin/kernel_lambda_test 2024-06-26T07:09:37.3013715Z inflating: build/bin/kernel_stackbased_test 2024-06-26T07:09:37.3106595Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T07:09:37.3156433Z inflating: build/bin/CppSignature_test 2024-06-26T07:09:37.3211529Z inflating: build/bin/backend_fallback_test 2024-06-26T07:09:37.3258277Z inflating: build/bin/op_allowlist_test 2024-06-26T07:09:37.3319901Z inflating: build/bin/inline_container_test 2024-06-26T07:09:37.3613048Z inflating: build/bin/op_registration_test 2024-06-26T07:09:37.3664548Z inflating: build/bin/cuda_apply_test 2024-06-26T07:09:37.3713890Z inflating: build/bin/cuda_allocator_test 2024-06-26T07:09:37.3771143Z inflating: build/bin/cuda_atomic_ops_test 2024-06-26T07:09:37.3823840Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-26T07:09:37.3892003Z inflating: build/bin/cuda_complex_math_test 2024-06-26T07:09:37.3948653Z inflating: build/bin/cuda_complex_test 2024-06-26T07:09:37.3996762Z inflating: build/bin/cuda_device_test 2024-06-26T07:09:37.4052179Z inflating: build/bin/cuda_cub_test 2024-06-26T07:09:37.4101487Z inflating: build/bin/cuda_dlconvertor_test 2024-06-26T07:09:37.4164200Z inflating: build/bin/cuda_distributions_test 2024-06-26T07:09:37.4219270Z inflating: build/bin/cuda_generator_test 2024-06-26T07:09:37.4267577Z inflating: build/bin/cuda_half_test 2024-06-26T07:09:37.4317135Z inflating: build/bin/cuda_integer_divider_test 2024-06-26T07:09:37.4364724Z inflating: build/bin/cuda_optional_test 2024-06-26T07:09:37.4414831Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-26T07:09:37.4465961Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-26T07:09:37.4514187Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-26T07:09:37.4572913Z inflating: build/bin/cuda_stream_test 2024-06-26T07:09:37.4620731Z inflating: build/bin/cuda_cudnn_test 2024-06-26T07:09:37.4671647Z inflating: build/bin/cuda_vectorized_test 2024-06-26T07:09:37.4685633Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T07:09:37.4749153Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T07:09:37.4804862Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-26T07:09:37.4866663Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-26T07:09:37.4926144Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-26T07:09:37.5731164Z inflating: build/bin/test_tensorexpr 2024-06-26T07:09:37.6290166Z inflating: build/bin/test_jit 2024-06-26T07:09:37.6290652Z creating: .additional_ci_files/ 2024-06-26T07:09:37.6343899Z inflating: .additional_ci_files/test-times.json 2024-06-26T07:09:37.6554786Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T07:09:37.6581795Z ##[group]Run rm artifacts.zip 2024-06-26T07:09:37.6582194Z rm artifacts.zip 2024-06-26T07:09:37.6589873Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:37.6590372Z env: 2024-06-26T07:09:37.6590656Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:37.6591112Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:37.6591733Z ##[endgroup] 2024-06-26T07:09:37.7348447Z ##[group]Run df -H 2024-06-26T07:09:37.7348775Z df -H 2024-06-26T07:09:37.7356432Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:37.7356938Z env: 2024-06-26T07:09:37.7357214Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:37.7357670Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:37.7358157Z ##[endgroup] 2024-06-26T07:09:37.7385072Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T07:09:37.7385613Z devtmpfs 34G 0 34G 0% /dev 2024-06-26T07:09:37.7386177Z tmpfs 34G 0 34G 0% /dev/shm 2024-06-26T07:09:37.7387034Z tmpfs 34G 455k 34G 1% /run 2024-06-26T07:09:37.7387611Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-06-26T07:09:37.7388094Z /dev/nvme0n1p1 162G 47G 115G 29% / 2024-06-26T07:09:37.7425332Z Prepare all required actions 2024-06-26T07:09:37.7425760Z Getting action download info 2024-06-26T07:09:37.8687614Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T07:09:37.8688082Z with: 2024-06-26T07:09:37.8688340Z env: 2024-06-26T07:09:37.8688618Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:37.8689071Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:37.8689555Z ##[endgroup] 2024-06-26T07:09:37.8719689Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:09:37.8720131Z with: 2024-06-26T07:09:37.8720392Z name: td_results 2024-06-26T07:09:37.8720705Z s3-bucket: gha-artifacts 2024-06-26T07:09:37.8721051Z region: us-east-1 2024-06-26T07:09:37.8721340Z env: 2024-06-26T07:09:37.8721606Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:37.8722053Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:37.8722530Z ##[endgroup] 2024-06-26T07:09:38.2871524Z (node:10166) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:09:38.2872870Z 2024-06-26T07:09:38.2873388Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:09:38.2874807Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:09:38.2875981Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:09:38.3844254Z Found 0 objects with prefix pytorch/pytorch/9673646808/td_results/ 2024-06-26T07:09:38.3850714Z Artifact download has finished successfully 2024-06-26T07:09:38.3980074Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T07:09:38.3980567Z mkdir -p .additional_ci_files 2024-06-26T07:09:38.3981120Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T07:09:38.3989175Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:38.3989668Z env: 2024-06-26T07:09:38.3989948Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:38.3990415Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:38.3990900Z ##[endgroup] 2024-06-26T07:09:38.4024856Z mv: cannot stat 'td_results.json': No such file or directory 2024-06-26T07:09:38.4034831Z ##[error]Process completed with exit code 1. 2024-06-26T07:09:38.4082812Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T07:09:38.4083287Z .github/scripts/parse_ref.py 2024-06-26T07:09:38.4091151Z shell: /usr/bin/bash -e {0} 2024-06-26T07:09:38.4091527Z env: 2024-06-26T07:09:38.4091797Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:38.4092247Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:38.4092727Z ##[endgroup] 2024-06-26T07:09:38.4319423Z Prepare all required actions 2024-06-26T07:09:38.4356053Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T07:09:38.4356506Z with: 2024-06-26T07:09:38.4357059Z github-token: *** 2024-06-26T07:09:38.4357372Z env: 2024-06-26T07:09:38.4357644Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:38.4358115Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:38.4358630Z ##[endgroup] 2024-06-26T07:09:38.4373627Z ##[group]Run set -eux 2024-06-26T07:09:38.4374093Z set -eux 2024-06-26T07:09:38.4374678Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T07:09:38.4382258Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:38.4382864Z env: 2024-06-26T07:09:38.4383141Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:38.4383592Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:38.4384221Z GITHUB_TOKEN: *** 2024-06-26T07:09:38.4384521Z ##[endgroup] 2024-06-26T07:09:38.4403647Z + python3 .github/scripts/get_workflow_job_id.py 9673646808 i-0bab72a3f8092178a 2024-06-26T07:09:39.7856757Z setting job-id=26688911008 2024-06-26T07:09:39.7857739Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:09:39.8049117Z Prepare all required actions 2024-06-26T07:09:39.8049554Z Getting action download info 2024-06-26T07:09:39.9101939Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T07:09:39.9102421Z with: 2024-06-26T07:09:39.9102959Z github-token: *** 2024-06-26T07:09:39.9110974Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:09:39.9119738Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:09:39.9120457Z env: 2024-06-26T07:09:39.9120726Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:39.9121173Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:39.9121644Z ##[endgroup] 2024-06-26T07:09:39.9161687Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:09:39.9162232Z with: 2024-06-26T07:09:39.9162494Z shell: bash 2024-06-26T07:09:39.9162787Z timeout_minutes: 10 2024-06-26T07:09:39.9163108Z max_attempts: 5 2024-06-26T07:09:39.9163410Z retry_wait_seconds: 30 2024-06-26T07:09:39.9164521Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:09:39.9165830Z polling_interval_seconds: 1 2024-06-26T07:09:39.9166194Z warning_on_retry: true 2024-06-26T07:09:39.9166532Z continue_on_error: false 2024-06-26T07:09:39.9166864Z env: 2024-06-26T07:09:39.9167131Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:39.9167569Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:39.9168184Z GITHUB_TOKEN: *** 2024-06-26T07:09:39.9168492Z ##[endgroup] 2024-06-26T07:09:39.9654764Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:09:40.1827257Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:09:40.1985711Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-06-26T07:09:40.2131597Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-06-26T07:09:40.2139252Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-06-26T07:09:40.2340528Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-06-26T07:09:40.2360550Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2) 2024-06-26T07:09:40.2368822Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-06-26T07:09:40.9657505Z Command completed after 1 attempt(s). 2024-06-26T07:09:40.9696389Z ##[group]Run set -x 2024-06-26T07:09:40.9696716Z set -x 2024-06-26T07:09:40.9697009Z  2024-06-26T07:09:40.9697566Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:09:40.9698236Z # in runner workspace 2024-06-26T07:09:40.9698759Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T07:09:40.9706579Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:40.9707077Z env: 2024-06-26T07:09:40.9707344Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:40.9707789Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:40.9708262Z ##[endgroup] 2024-06-26T07:09:40.9731211Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T07:09:40.9927132Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:09:40.9927672Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:09:40.9928128Z echo "Job name: ${JOB_NAME}" 2024-06-26T07:09:40.9928510Z  2024-06-26T07:09:40.9929061Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:09:40.9929721Z # in runner workspace 2024-06-26T07:09:40.9930283Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T07:09:40.9931138Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T07:09:40.9931587Z  --job-name "${JOB_NAME}" \ 2024-06-26T07:09:40.9939874Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-06-26T07:09:40.9948451Z  --selected-test-configs "" \ 2024-06-26T07:09:40.9948895Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T07:09:40.9949311Z  --tag "${TAG}" \ 2024-06-26T07:09:40.9949705Z  --event-name "${EVENT_NAME}" \ 2024-06-26T07:09:40.9950144Z  --schedule "${SCHEDULE}" \ 2024-06-26T07:09:40.9950560Z  --branch "${HEAD_BRANCH}" 2024-06-26T07:09:40.9958227Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:40.9958717Z env: 2024-06-26T07:09:40.9958990Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:40.9959441Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:40.9960101Z GITHUB_TOKEN: *** 2024-06-26T07:09:40.9960747Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:09:40.9961463Z PR_NUMBER: 2024-06-26T07:09:40.9961764Z TAG: ciflow/inductor/129470 2024-06-26T07:09:40.9962128Z EVENT_NAME: push 2024-06-26T07:09:40.9962434Z SCHEDULE: 2024-06-26T07:09:40.9962720Z HEAD_BRANCH: 2024-06-26T07:09:40.9963005Z ##[endgroup] 2024-06-26T07:09:40.9994739Z Workflow: inductor 2024-06-26T07:09:40.9995550Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:09:41.2153782Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T07:09:41.3538321Z ##[group]Run echo "Filtered matrix:" 2024-06-26T07:09:41.3538788Z echo "Filtered matrix:" 2024-06-26T07:09:41.3547045Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-06-26T07:09:41.3555689Z  2024-06-26T07:09:41.3555971Z echo 2024-06-26T07:09:41.3556349Z echo "Is the current job unstable? False" 2024-06-26T07:09:41.3556808Z  2024-06-26T07:09:41.3557081Z echo 2024-06-26T07:09:41.3557451Z echo "Is keep-going label set? False" 2024-06-26T07:09:41.3557889Z  2024-06-26T07:09:41.3558151Z echo 2024-06-26T07:09:41.3558463Z echo "Renabled issues? " 2024-06-26T07:09:41.3566547Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:41.3567071Z env: 2024-06-26T07:09:41.3567346Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:41.3567809Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:41.3568295Z ##[endgroup] 2024-06-26T07:09:41.3587958Z Filtered matrix: 2024-06-26T07:09:41.3598154Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-06-26T07:09:41.3606342Z 2024-06-26T07:09:41.3606497Z Is the current job unstable? False 2024-06-26T07:09:41.3606820Z 2024-06-26T07:09:41.3607055Z Is keep-going label set? False 2024-06-26T07:09:41.3607315Z 2024-06-26T07:09:41.3607438Z Renabled issues? 2024-06-26T07:09:41.3648631Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:09:41.3649340Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:09:41.3657027Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:41.3657517Z env: 2024-06-26T07:09:41.3657803Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:41.3658361Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:41.3658847Z JOB_TIMEOUT: 240 2024-06-26T07:09:41.3659147Z ##[endgroup] 2024-06-26T07:09:41.3726758Z ##[group]Run set -x 2024-06-26T07:09:41.3727149Z set -x 2024-06-26T07:09:41.3727441Z  2024-06-26T07:09:41.3727788Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T07:09:41.3728318Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T07:09:41.3728876Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T07:09:41.3729380Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-26T07:09:41.3729789Z else 2024-06-26T07:09:41.3730120Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:09:41.3730533Z fi 2024-06-26T07:09:41.3731292Z  2024-06-26T07:09:41.3731758Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T07:09:41.3732494Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T07:09:41.3733179Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T07:09:41.3733759Z # shellcheck disable=SC2086,SC2090 2024-06-26T07:09:41.3734212Z container_name=$(docker run \ 2024-06-26T07:09:41.3734627Z  ${GPU_FLAG:-} \ 2024-06-26T07:09:41.3734991Z  -e BUILD_ENVIRONMENT \ 2024-06-26T07:09:41.3735383Z  -e PR_NUMBER \ 2024-06-26T07:09:41.3735730Z  -e GITHUB_ACTIONS \ 2024-06-26T07:09:41.3736118Z  -e GITHUB_REPOSITORY \ 2024-06-26T07:09:41.3736515Z  -e GITHUB_WORKFLOW \ 2024-06-26T07:09:41.3736896Z  -e GITHUB_JOB \ 2024-06-26T07:09:41.3737248Z  -e GITHUB_RUN_ID \ 2024-06-26T07:09:41.3737628Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T07:09:41.3738028Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T07:09:41.3738410Z  -e JOB_ID \ 2024-06-26T07:09:41.3738749Z  -e JOB_NAME \ 2024-06-26T07:09:41.3739092Z  -e BASE_SHA \ 2024-06-26T07:09:41.3739436Z  -e BRANCH \ 2024-06-26T07:09:41.3739754Z  -e SHA1 \ 2024-06-26T07:09:41.3740096Z  -e AWS_DEFAULT_REGION \ 2024-06-26T07:09:41.3740495Z  -e IN_WHEEL_TEST \ 2024-06-26T07:09:41.3740872Z  -e SHARD_NUMBER \ 2024-06-26T07:09:41.3741230Z  -e TEST_CONFIG \ 2024-06-26T07:09:41.3741594Z  -e NUM_TEST_SHARDS \ 2024-06-26T07:09:41.3741979Z  -e REENABLED_ISSUES \ 2024-06-26T07:09:41.3742385Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T07:09:41.3742871Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T07:09:41.3743265Z  -e NO_TEST_TIMEOUT \ 2024-06-26T07:09:41.3743636Z  -e NO_TD \ 2024-06-26T07:09:41.3743966Z  -e TD_DISTRIBUTED \ 2024-06-26T07:09:41.3744341Z  -e PR_LABELS \ 2024-06-26T07:09:41.3744741Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T07:09:41.3745192Z  -e SCCACHE_BUCKET \ 2024-06-26T07:09:41.3745588Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-26T07:09:41.3745995Z  -e XLA_CUDA \ 2024-06-26T07:09:41.3746398Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T07:09:41.3746908Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T07:09:41.3747425Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T07:09:41.3747934Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-26T07:09:41.3748396Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-26T07:09:41.3748813Z  -e DASHBOARD_TAG \ 2024-06-26T07:09:41.3749276Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T07:09:41.3749823Z  --security-opt seccomp=unconfined \ 2024-06-26T07:09:41.3750280Z  --cap-add=SYS_PTRACE \ 2024-06-26T07:09:41.3750667Z  --ipc=host \ 2024-06-26T07:09:41.3751015Z  --shm-size="${SHM_SIZE}" \ 2024-06-26T07:09:41.3751406Z  --tty \ 2024-06-26T07:09:41.3751719Z  --detach \ 2024-06-26T07:09:41.3752181Z  --name="${container_name}" \ 2024-06-26T07:09:41.3752590Z  --user jenkins \ 2024-06-26T07:09:41.3753132Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T07:09:41.3753694Z  -w /var/lib/jenkins/workspace \ 2024-06-26T07:09:41.3754120Z  "${DOCKER_IMAGE}" 2024-06-26T07:09:41.3754461Z ) 2024-06-26T07:09:41.3754857Z # Propagate download.pytorch.org IP to container 2024-06-26T07:09:41.3755765Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-26T07:09:41.3756719Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-26T07:09:41.3757614Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-26T07:09:41.3764896Z shell: /usr/bin/bash -e {0} 2024-06-26T07:09:41.3765246Z env: 2024-06-26T07:09:41.3765523Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:41.3765977Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:41.3766595Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:09:41.3767123Z PR_NUMBER: 2024-06-26T07:09:41.3767442Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T07:09:41.3767844Z GITHUB_WORKFLOW: inductor 2024-06-26T07:09:41.3768191Z GITHUB_JOB: test 2024-06-26T07:09:41.3768499Z GITHUB_RUN_ID: 9673646808 2024-06-26T07:09:41.3768853Z GITHUB_RUN_NUMBER: 76977 2024-06-26T07:09:41.3769191Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T07:09:41.3769516Z JOB_ID: 26688911008 2024-06-26T07:09:41.3770164Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:09:41.3770991Z BRANCH: 2024-06-26T07:09:41.3771318Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:09:41.3771828Z BASE_SHA: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:09:41.3772300Z TEST_CONFIG: inductor_huggingface 2024-06-26T07:09:41.3772694Z SHARD_NUMBER: 1 2024-06-26T07:09:41.3772994Z NUM_TEST_SHARDS: 1 2024-06-26T07:09:41.3773313Z REENABLED_ISSUES: 2024-06-26T07:09:41.3773653Z CONTINUE_THROUGH_ERROR: False 2024-06-26T07:09:41.3774025Z VERBOSE_TEST_LOGS: False 2024-06-26T07:09:41.3774374Z NO_TEST_TIMEOUT: False 2024-06-26T07:09:41.3774702Z NO_TD: False 2024-06-26T07:09:41.3774995Z TD_DISTRIBUTED: False 2024-06-26T07:09:41.3775398Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T07:09:41.3775881Z SCCACHE_S3_KEY_PREFIX: inductor 2024-06-26T07:09:41.3776248Z SHM_SIZE: 2g 2024-06-26T07:09:41.3777370Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:09:41.3778509Z XLA_CUDA: 2024-06-26T07:09:41.3778987Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:09:41.3779596Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T07:09:41.3780025Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T07:09:41.3780426Z DASHBOARD_TAG: 2024-06-26T07:09:41.3780924Z HUGGING_FACE_HUB_TOKEN: *** 2024-06-26T07:09:41.3781283Z ##[endgroup] 2024-06-26T07:09:41.3800315Z + [[ inductor_huggingface == \m\u\l\t\i\g\p\u ]] 2024-06-26T07:09:41.3801014Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-06-26T07:09:41.3801531Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:09:41.3807584Z +++ nproc --ignore=2 2024-06-26T07:09:41.3820511Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9673646808 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:09:52.5823680Z + container_name=7ad6c3d35e4b88f659cb3bbc7d1ca4daf91f6fffa067a386e14c96b578954061 2024-06-26T07:09:52.5824680Z + grep download.pytorch.org /etc/hosts 2024-06-26T07:09:52.5827178Z + docker exec -i 7ad6c3d35e4b88f659cb3bbc7d1ca4daf91f6fffa067a386e14c96b578954061 sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-26T07:09:52.6456648Z + echo DOCKER_CONTAINER_ID=7ad6c3d35e4b88f659cb3bbc7d1ca4daf91f6fffa067a386e14c96b578954061 2024-06-26T07:09:52.6459487Z ++ echo dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:09:52.6461283Z + docker exec -t 7ad6c3d35e4b88f659cb3bbc7d1ca4daf91f6fffa067a386e14c96b578954061 sh -c 'pip install dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-26T07:09:53.0170441Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb8c4c54) 2024-06-26T07:09:53.3294630Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T07:09:53.3299080Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T07:09:53.3301543Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T07:09:53.3305212Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T07:09:53.3308052Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T07:09:53.3310720Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T07:09:53.3337074Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.3.0) 2024-06-26T07:09:53.3395131Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.21.2) 2024-06-26T07:09:53.3827873Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T07:09:53.3987041Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T07:09:54.7066505Z Installing collected packages: torch 2024-06-26T07:10:03.3696306Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-06-26T07:10:03.3697654Z timm 0.9.7 requires torchvision, which is not installed. 2024-06-26T07:10:03.3700796Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T07:10:03.4491353Z ++ dirname .ci/pytorch/test.sh 2024-06-26T07:10:03.4495801Z + source .ci/pytorch/common.sh 2024-06-26T07:10:03.4498499Z +++ dirname .ci/pytorch/common.sh 2024-06-26T07:10:03.4504988Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T07:10:03.4507167Z +++ declare -f -t trap_add 2024-06-26T07:10:03.4512074Z ++ set -ex 2024-06-26T07:10:03.4513514Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:10:03.4514120Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T07:10:03.4514628Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-06-26T07:10:03.4515622Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-26T07:10:03.4523529Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-26T07:10:03.4524098Z + trap_add cleanup_workspace EXIT 2024-06-26T07:10:03.4524651Z + trap_add_cmd=cleanup_workspace 2024-06-26T07:10:03.4525149Z + shift 2024-06-26T07:10:03.4525528Z + for trap_add_name in "$@" 2024-06-26T07:10:03.4528906Z +++ trap -p EXIT 2024-06-26T07:10:03.4530695Z ++ eval 'extract_trap_cmd ' 2024-06-26T07:10:03.4531384Z +++ extract_trap_cmd 2024-06-26T07:10:03.4531789Z +++ printf '%s\n' '' 2024-06-26T07:10:03.4532198Z ++ printf '%s\n' cleanup_workspace 2024-06-26T07:10:03.4533014Z + trap -- ' 2024-06-26T07:10:03.4533493Z cleanup_workspace' EXIT 2024-06-26T07:10:03.4534021Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-26T07:10:03.8645236Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-26T07:10:03.8657504Z + echo 'Environment variables:' 2024-06-26T07:10:03.8657965Z Environment variables: 2024-06-26T07:10:03.8658283Z + env 2024-06-26T07:10:03.8663215Z INSTALLED_DB=yes 2024-06-26T07:10:03.8664007Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:10:03.8664450Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:10:03.8664862Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:10:03.8665451Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:10:03.8666129Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:10:03.8666652Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:10:03.8667152Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:10:03.8667930Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:03.8668563Z HOSTNAME=7ad6c3d35e4b 2024-06-26T07:10:03.8669818Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_4038b1e2-8144-495d-bdbf-9ef0cb501b85 2024-06-26T07:10:03.8670994Z GITHUB_ACTION=__self 2024-06-26T07:10:03.8671436Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:10:03.8676594Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:10:03.8680533Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:10:03.8681052Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:10:03.8681411Z GITHUB_RUN_NUMBER=76977 2024-06-26T07:10:03.8681779Z TEST_CONFIG=inductor_huggingface 2024-06-26T07:10:03.8682180Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:10:03.8682655Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:10:03.8683121Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:10:03.8683566Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:10:03.8683982Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:10:03.8684428Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:10:03.8684931Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:10:03.8685395Z GITHUB_REF_TYPE=tag 2024-06-26T07:10:03.8685728Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:10:03.8686354Z NCCL_VERSION=2.17.1-1 2024-06-26T07:10:03.8686783Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:03.8687230Z XLA_CUDA= 2024-06-26T07:10:03.8687825Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:10:03.8689583Z *** 2024-06-26T07:10:03.8689877Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:10:03.8690265Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:10:03.8690620Z GITHUB_ACTIONS=true 2024-06-26T07:10:03.8691266Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:03.8691770Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:10:03.8692293Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:10:03.8692740Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:03.8693233Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:10:03.8693701Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:03.8694468Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:10:03.8695164Z UCC_HOME=/usr 2024-06-26T07:10:03.8695501Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:10:03.8695890Z VERBOSE_TEST_LOGS=False 2024-06-26T07:10:03.8696233Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:10:03.8696677Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:10:03.8697163Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:10:03.8697617Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:10:03.8697987Z SHARD_NUMBER=1 2024-06-26T07:10:03.8698284Z GITHUB_REF_PROTECTED=false 2024-06-26T07:10:03.8698625Z HOME=/var/lib/jenkins 2024-06-26T07:10:03.8698982Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:10:03.8699403Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:10:03.8699865Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:10:03.8700338Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-26T07:10:03.8700699Z CUDA_VERSION=12.1.1 2024-06-26T07:10:03.8701095Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:10:03.8701530Z NUM_TEST_SHARDS=1 2024-06-26T07:10:03.8701824Z UCX_HOME=/usr 2024-06-26T07:10:03.8702324Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:10:03.8703605Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_4038b1e2-8144-495d-bdbf-9ef0cb501b85 2024-06-26T07:10:03.8704811Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:03.8705993Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_4038b1e2-8144-495d-bdbf-9ef0cb501b85 2024-06-26T07:10:03.8707069Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:10:03.8707699Z GITHUB_EVENT_NAME=push 2024-06-26T07:10:03.8708025Z DASHBOARD_TAG= 2024-06-26T07:10:03.8708318Z GITHUB_RUN_ID=9673646808 2024-06-26T07:10:03.8708760Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:10:03.8709270Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:10:03.8710247Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_4038b1e2-8144-495d-bdbf-9ef0cb501b85 2024-06-26T07:10:03.8711134Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:10:03.8711536Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:10:03.8711897Z PR_NUMBER= 2024-06-26T07:10:03.8712172Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:10:03.8712503Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:10:03.8712926Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:10:03.8713373Z TERM=xterm 2024-06-26T07:10:03.8713695Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:10:03.8714099Z INSTALLED_VISION=yes 2024-06-26T07:10:03.8714396Z BRANCH= 2024-06-26T07:10:03.8714673Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:10:03.8715062Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:10:03.8715475Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:10:03.8716191Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:10:03.8716922Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:10:03.8717535Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:10:03.8717993Z REENABLED_ISSUES= 2024-06-26T07:10:03.8718284Z SHLVL=1 2024-06-26T07:10:03.8718607Z MAX_JOBS=14 2024-06-26T07:10:03.8718924Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:10:03.8719260Z NVARCH=x86_64 2024-06-26T07:10:03.8719551Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:10:03.8719991Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:03.8720510Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:10:03.8720981Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:10:03.8721620Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:10:03.8722180Z GITHUB_JOB=test 2024-06-26T07:10:03.8722555Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:10:03.8723098Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:10:03.8723602Z NO_TEST_TIMEOUT=False 2024-06-26T07:10:03.8723971Z TD_DISTRIBUTED=False 2024-06-26T07:10:03.8724330Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:10:03.8724765Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:10:03.8725169Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:10:03.8725532Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:10:03.8725865Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:10:03.8726212Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:10:03.8727230Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:10:03.8728264Z GITHUB_BASE_REF= 2024-06-26T07:10:03.8728572Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:10:03.8728941Z CI=true 2024-06-26T07:10:03.8729246Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:10:03.8729642Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:10:03.8730008Z JOB_ID=26688911008 2024-06-26T07:10:03.8730317Z INSTALLED_PROTOBUF=yes 2024-06-26T07:10:03.8730637Z GITHUB_HEAD_REF= 2024-06-26T07:10:03.8731251Z GITHUB_ACTION_REF= 2024-06-26T07:10:03.8731683Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:10:03.8732145Z GITHUB_WORKFLOW=inductor 2024-06-26T07:10:03.8732498Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:10:03.8733377Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_4038b1e2-8144-495d-bdbf-9ef0cb501b85 2024-06-26T07:10:03.8734172Z NO_TD=False 2024-06-26T07:10:03.8734468Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:10:03.8734826Z _=/usr/bin/env 2024-06-26T07:10:03.8735281Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T07:10:03.8812300Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-26T07:10:03.8813488Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T07:10:03.8814437Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-26T07:10:03.8815503Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-26T07:10:03.8816101Z + BUILD_DIR=build 2024-06-26T07:10:03.8816435Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T07:10:03.8816817Z + BUILD_BIN_DIR=build/bin 2024-06-26T07:10:03.8817171Z + SHARD_NUMBER=1 2024-06-26T07:10:03.8817589Z + NUM_TEST_SHARDS=1 2024-06-26T07:10:03.8817984Z + export VALGRIND=ON 2024-06-26T07:10:03.8818291Z + VALGRIND=ON 2024-06-26T07:10:03.8818754Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-06-26T07:10:03.8819235Z + [[ 0 == \1 ]] 2024-06-26T07:10:03.8819525Z + [[ False == \1 ]] 2024-06-26T07:10:03.8819974Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-06-26T07:10:03.8820486Z ++ realpath build/custom_test_artifacts 2024-06-26T07:10:03.8826199Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-26T07:10:03.8827072Z + [[ -n '' ]] 2024-06-26T07:10:03.8827406Z + echo 'Environment variables' 2024-06-26T07:10:03.8827778Z Environment variables 2024-06-26T07:10:03.8828091Z + env 2024-06-26T07:10:03.8832303Z INSTALLED_DB=yes 2024-06-26T07:10:03.8832806Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:10:03.8833541Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:10:03.8834062Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:10:03.8835020Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:10:03.8835786Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:10:03.8836382Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:10:03.8836918Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:10:03.8837478Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:03.8838013Z HOSTNAME=7ad6c3d35e4b 2024-06-26T07:10:03.8839006Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_4038b1e2-8144-495d-bdbf-9ef0cb501b85 2024-06-26T07:10:03.8840084Z GITHUB_ACTION=__self 2024-06-26T07:10:03.8840531Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:10:03.8844567Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:10:03.8848397Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:10:03.8848889Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:10:03.8849234Z GITHUB_RUN_NUMBER=76977 2024-06-26T07:10:03.8849585Z TEST_CONFIG=inductor_huggingface 2024-06-26T07:10:03.8849979Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:10:03.8850432Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:10:03.8851160Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:10:03.8851606Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:10:03.8852026Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:10:03.8852454Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:10:03.8852943Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:10:03.8853403Z GITHUB_REF_TYPE=tag 2024-06-26T07:10:03.8853764Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:10:03.8854135Z NCCL_VERSION=2.17.1-1 2024-06-26T07:10:03.8854519Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:03.8854956Z XLA_CUDA= 2024-06-26T07:10:03.8855415Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:10:03.8855807Z *** 2024-06-26T07:10:03.8856086Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:10:03.8856485Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:10:03.8856838Z GITHUB_ACTIONS=true 2024-06-26T07:10:03.8857170Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:03.8857648Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:10:03.8858175Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:10:03.8858632Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:03.8859128Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:10:03.8859607Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:03.8860383Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:10:03.8861077Z UCC_HOME=/usr 2024-06-26T07:10:03.8861412Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:10:03.8861809Z VERBOSE_TEST_LOGS=False 2024-06-26T07:10:03.8862145Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:10:03.8862587Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:10:03.8863167Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:10:03.8863625Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:10:03.8863990Z SHARD_NUMBER=1 2024-06-26T07:10:03.8864300Z GITHUB_REF_PROTECTED=false 2024-06-26T07:10:03.8864651Z HOME=/var/lib/jenkins 2024-06-26T07:10:03.8865010Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:10:03.8865574Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:10:03.8866106Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:10:03.8866586Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-26T07:10:03.8866952Z CUDA_VERSION=12.1.1 2024-06-26T07:10:03.8867360Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:10:03.8867800Z NUM_TEST_SHARDS=1 2024-06-26T07:10:03.8868139Z UCX_HOME=/usr 2024-06-26T07:10:03.8868646Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:10:03.8869702Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_4038b1e2-8144-495d-bdbf-9ef0cb501b85 2024-06-26T07:10:03.8870890Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:03.8872076Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_4038b1e2-8144-495d-bdbf-9ef0cb501b85 2024-06-26T07:10:03.8873160Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:10:03.8873819Z GITHUB_EVENT_NAME=push 2024-06-26T07:10:03.8874148Z DASHBOARD_TAG= 2024-06-26T07:10:03.8874450Z GITHUB_RUN_ID=9673646808 2024-06-26T07:10:03.8874901Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:10:03.8875424Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:10:03.8876382Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_4038b1e2-8144-495d-bdbf-9ef0cb501b85 2024-06-26T07:10:03.8877276Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:10:03.8877676Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:10:03.8878047Z PR_NUMBER= 2024-06-26T07:10:03.8878320Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:10:03.8878634Z VALGRIND=ON 2024-06-26T07:10:03.8878929Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:10:03.8879360Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:10:03.8879818Z TERM=xterm 2024-06-26T07:10:03.8880147Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:10:03.8880561Z INSTALLED_VISION=yes 2024-06-26T07:10:03.8880860Z BRANCH= 2024-06-26T07:10:03.8881140Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:10:03.8881542Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:10:03.8881967Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:10:03.8882710Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:10:03.8883466Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:10:03.8883989Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:10:03.8884444Z REENABLED_ISSUES= 2024-06-26T07:10:03.8884740Z SHLVL=1 2024-06-26T07:10:03.8884995Z MAX_JOBS=14 2024-06-26T07:10:03.8885306Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:10:03.8885646Z NVARCH=x86_64 2024-06-26T07:10:03.8885941Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:10:03.8886384Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:03.8886904Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:10:03.8887373Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:10:03.8888017Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:10:03.8888582Z GITHUB_JOB=test 2024-06-26T07:10:03.8888958Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:10:03.8889504Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:10:03.8890004Z NO_TEST_TIMEOUT=False 2024-06-26T07:10:03.8890325Z TD_DISTRIBUTED=False 2024-06-26T07:10:03.8890683Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:10:03.8891420Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:10:03.8891838Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:10:03.8892195Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:10:03.8892532Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:10:03.8892876Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:10:03.8893943Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:10:03.8895094Z GITHUB_BASE_REF= 2024-06-26T07:10:03.8895404Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:10:03.8895763Z CI=true 2024-06-26T07:10:03.8896130Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:10:03.8896531Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:10:03.8896905Z JOB_ID=26688911008 2024-06-26T07:10:03.8897217Z INSTALLED_PROTOBUF=yes 2024-06-26T07:10:03.8897539Z GITHUB_HEAD_REF= 2024-06-26T07:10:03.8916910Z GITHUB_ACTION_REF= 2024-06-26T07:10:03.8917531Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:10:03.8918163Z GITHUB_WORKFLOW=inductor 2024-06-26T07:10:03.8918537Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:10:03.8919419Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_4038b1e2-8144-495d-bdbf-9ef0cb501b85 2024-06-26T07:10:03.8920212Z NO_TD=False 2024-06-26T07:10:03.8920499Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:10:03.8920855Z _=/usr/bin/env 2024-06-26T07:10:03.8921181Z + echo 'Testing pytorch' 2024-06-26T07:10:03.8921513Z Testing pytorch 2024-06-26T07:10:03.8921841Z + export LANG=C.UTF-8 2024-06-26T07:10:03.8922163Z + LANG=C.UTF-8 2024-06-26T07:10:03.8922456Z + PR_NUMBER= 2024-06-26T07:10:03.8922797Z + [[ inductor_huggingface == \d\e\f\a\u\l\t ]] 2024-06-26T07:10:03.8923303Z + [[ inductor_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T07:10:03.8923848Z + [[ inductor_huggingface == \s\l\o\w ]] 2024-06-26T07:10:03.8924464Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-06-26T07:10:03.8925133Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-06-26T07:10:03.8925670Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:10:03.8926137Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:10:03.8926575Z + [[ inductor_huggingface == *crossref* ]] 2024-06-26T07:10:03.8927126Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:10:03.8927744Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-06-26T07:10:03.8928379Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:10:03.8928936Z + pip_install --user ninja==1.10.2 2024-06-26T07:10:03.8929451Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T07:10:04.2498561Z Collecting ninja==1.10.2 2024-06-26T07:10:04.2642054Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T07:10:04.2772391Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T07:10:05.5735016Z Installing collected packages: ninja 2024-06-26T07:10:05.5800855Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T07:10:05.5803573Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T07:10:05.5839697Z Successfully installed ninja-1.10.2 2024-06-26T07:10:05.6682619Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:10:05.6684775Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:10:05.6686282Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:10:05.6686788Z + install_tlparse 2024-06-26T07:10:05.6687157Z + pip_install --user tlparse==0.3.7 2024-06-26T07:10:05.6687710Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T07:10:06.0205392Z Collecting tlparse==0.3.7 2024-06-26T07:10:06.0341314Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T07:10:06.0431646Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T07:10:07.3546809Z Installing collected packages: tlparse 2024-06-26T07:10:07.3901371Z Successfully installed tlparse-0.3.7 2024-06-26T07:10:07.4734032Z ++ python -m site --user-base 2024-06-26T07:10:07.4888496Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:10:07.4890634Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-06-26T07:10:07.4892083Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-06-26T07:10:07.4892939Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:10:07.4893881Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-06-26T07:10:07.4895034Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-06-26T07:10:07.4895724Z + cd test 2024-06-26T07:10:07.4896239Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T07:10:08.9701382Z + [[ inductor_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T07:10:08.9702082Z + [[ inductor_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T07:10:08.9703009Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:10:08.9703509Z + pushd test 2024-06-26T07:10:08.9703935Z ~/workspace/test ~/workspace 2024-06-26T07:10:08.9704580Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T07:10:10.4583510Z + CUDA_VERSION=12.1 2024-06-26T07:10:10.4584088Z + '[' 12.1 == 12.4 ']' 2024-06-26T07:10:10.4584475Z + ISCUDA124= 2024-06-26T07:10:10.4584771Z + popd 2024-06-26T07:10:10.4585040Z ~/workspace 2024-06-26T07:10:10.4588744Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T07:10:10.4589328Z + [[ inductor_huggingface == *dynamo_eager* ]] 2024-06-26T07:10:10.4589986Z + [[ inductor_huggingface == *aot_eager* ]] 2024-06-26T07:10:10.4590629Z + [[ inductor_huggingface == *aot_inductor* ]] 2024-06-26T07:10:10.4591260Z + [[ inductor_huggingface == *inductor* ]] 2024-06-26T07:10:10.4591734Z + [[ inductor_huggingface != *perf* ]] 2024-06-26T07:10:10.4592264Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-06-26T07:10:10.4592705Z + [[ inductor_huggingface == *dynamic* ]] 2024-06-26T07:10:10.4593189Z + [[ inductor_huggingface == *cpu_inductor* ]] 2024-06-26T07:10:10.4593697Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T07:10:10.4620262Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:10:10.4621073Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-06-26T07:10:10.4622537Z + cd test 2024-06-26T07:10:10.4623096Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T07:10:11.7949134Z PyTorch built with: 2024-06-26T07:10:11.7949836Z - GCC 9.4 2024-06-26T07:10:11.7950310Z - C++ Version: 201703 2024-06-26T07:10:11.7951621Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:10:11.7953271Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:10:11.7954305Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:10:11.7955071Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T07:10:11.7955795Z - NNPACK is enabled 2024-06-26T07:10:11.7956363Z - CPU capability usage: AVX2 2024-06-26T07:10:11.7956959Z - CUDA Runtime 12.1 2024-06-26T07:10:11.7957691Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-06-26T07:10:11.7958571Z - CuDNN 90.1 (built against CUDA 12.4) 2024-06-26T07:10:11.7959200Z - Magma 2.6.1 2024-06-26T07:10:11.7969720Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T07:10:11.7979517Z 2024-06-26T07:10:12.0675914Z + cd test 2024-06-26T07:10:12.0676610Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T07:10:13.2731450Z ATen/Parallel: 2024-06-26T07:10:13.2732049Z at::get_num_threads() : 8 2024-06-26T07:10:13.2732600Z at::get_num_interop_threads() : 16 2024-06-26T07:10:13.2733222Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:10:13.2733780Z omp_get_max_threads() : 8 2024-06-26T07:10:13.2735144Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:10:13.2736333Z mkl_get_max_threads() : 8 2024-06-26T07:10:13.2737232Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:10:13.2738141Z std::thread::hardware_concurrency() : 16 2024-06-26T07:10:13.2738768Z Environment variables: 2024-06-26T07:10:13.2739264Z OMP_NUM_THREADS : [not set] 2024-06-26T07:10:13.2739782Z MKL_NUM_THREADS : [not set] 2024-06-26T07:10:13.2740345Z ATen parallel backend: OpenMP 2024-06-26T07:10:13.2740742Z 2024-06-26T07:10:13.5170195Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:10:13.5171221Z + [[ inductor_huggingface == *backward* ]] 2024-06-26T07:10:13.5171815Z + [[ inductor_huggingface == *xla* ]] 2024-06-26T07:10:13.5172350Z + [[ inductor_huggingface == *executorch* ]] 2024-06-26T07:10:13.5172871Z + [[ inductor_huggingface == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T07:10:13.5173546Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:10:13.5174102Z + [[ inductor_huggingface == distributed ]] 2024-06-26T07:10:13.5174659Z + [[ inductor_huggingface == *inductor_distributed* ]] 2024-06-26T07:10:13.5175290Z + [[ inductor_huggingface == *inductor-halide* ]] 2024-06-26T07:10:13.5175889Z + [[ inductor_huggingface == *inductor-micro-benchmark* ]] 2024-06-26T07:10:13.5176502Z + [[ inductor_huggingface == *huggingface* ]] 2024-06-26T07:10:13.5177013Z + install_torchvision 2024-06-26T07:10:13.5177343Z + local orig_preload 2024-06-26T07:10:13.5177698Z + local commit 2024-06-26T07:10:13.5178021Z ++ get_pinned_commit vision 2024-06-26T07:10:13.5178402Z ++ cat .github/ci_commit_pins/vision.txt 2024-06-26T07:10:13.5184718Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:10:13.5185341Z + orig_preload= 2024-06-26T07:10:13.5185763Z + '[' -n '' ']' 2024-06-26T07:10:13.5186630Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:10:13.5188033Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:10:13.8302382Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:10:13.8306746Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-o_2xxj44 2024-06-26T07:10:13.8321951Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-o_2xxj44 2024-06-26T07:10:15.2357148Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-06-26T07:10:15.2376140Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:10:16.4340992Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:10:16.6583532Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:10:18.9796249Z Preparing metadata (setup.py) ... [?25l- \ done 2024-06-26T07:10:18.9844242Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-06-26T07:10:18.9847394Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+gitb8c4c54) 2024-06-26T07:10:18.9853764Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-06-26T07:10:19.0059444Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-06-26T07:10:19.0066274Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-06-26T07:10:19.0069043Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1) 2024-06-26T07:10:19.0072400Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-06-26T07:10:19.0075366Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-06-26T07:10:19.0078615Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-06-26T07:10:19.0507199Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-06-26T07:10:19.0662556Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-06-26T07:10:19.0739198Z Building wheels for collected packages: torchvision 2024-06-26T07:11:32.4513495Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-06-26T07:11:32.4540128Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2113008 sha256=24288a26b07624a683ff626347922a6aec8dde4a1895b7043fe505f8a61aee60 2024-06-26T07:11:32.4542649Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-06-26T07:11:32.4576814Z Successfully built torchvision 2024-06-26T07:11:33.6588741Z Installing collected packages: torchvision 2024-06-26T07:11:34.0383177Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-06-26T07:11:34.1498488Z + '[' -n '' ']' 2024-06-26T07:11:34.1498815Z + id=0 2024-06-26T07:11:34.1499132Z + test_dynamo_benchmark huggingface 0 2024-06-26T07:11:34.1502869Z ++ pwd 2024-06-26T07:11:34.1503696Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:11:34.1504241Z + local suite=huggingface 2024-06-26T07:11:34.1506524Z + shift 2024-06-26T07:11:34.1506870Z + local shard_id=0 2024-06-26T07:11:34.1507186Z + shift 2024-06-26T07:11:34.1509505Z + [[ inductor_huggingface == *perf_compare* ]] 2024-06-26T07:11:34.1510071Z + [[ inductor_huggingface == *perf* ]] 2024-06-26T07:11:34.1510598Z + [[ inductor_huggingface == *cpu_inductor* ]] 2024-06-26T07:11:34.1511101Z + [[ inductor_huggingface == *aot_inductor* ]] 2024-06-26T07:11:34.1512244Z + test_single_dynamo_benchmark inference huggingface 0 --inference --bfloat16 2024-06-26T07:11:34.1512827Z ++ pwd 2024-06-26T07:11:34.1513387Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:11:34.1514046Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:11:34.1522736Z + local name=inference 2024-06-26T07:11:34.1523083Z + shift 2024-06-26T07:11:34.1523439Z + local suite=huggingface 2024-06-26T07:11:34.1523826Z + shift 2024-06-26T07:11:34.1524091Z + local shard_id=0 2024-06-26T07:11:34.1524398Z + shift 2024-06-26T07:11:34.1524658Z + partition_flags=() 2024-06-26T07:11:34.1524986Z + local partition_flags 2024-06-26T07:11:34.1525369Z + [[ -n 1 ]] 2024-06-26T07:11:34.1525672Z + [[ -n 0 ]] 2024-06-26T07:11:34.1526257Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-06-26T07:11:34.1557622Z + [[ inductor_huggingface == *perf_compare* ]] 2024-06-26T07:11:34.1558122Z + [[ inductor_huggingface == *perf* ]] 2024-06-26T07:11:34.1558609Z + [[ inductor_huggingface == *aot_inductor* ]] 2024-06-26T07:11:34.1560226Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --inductor --device cuda --inference --bfloat16 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv 2024-06-26T07:11:36.8361979Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:11:36.8363735Z warnings.warn( 2024-06-26T07:11:36.8920435Z 2024-06-26T07:11:36.8920871Z config.json: 0% 0.00/694 [00:00 will be ignored 2024-06-26T07:38:03.4601330Z pass 2024-06-26T07:38:03.4898606Z TIMING: entire_frame_compile:27.0325 code_gen:13.85337 inductor_compile:23.16457 backend_compile:23.67548 2024-06-26T07:38:03.4899886Z STATS: call_* op count: 534 | FakeTensorMode.__torch_dispatch__:60169 | FakeTensor.__torch_dispatch__:9058 | ProxyTorchDispatchMode.__torch_dispatch__:19976 2024-06-26T07:38:03.4901012Z Dynamo produced 2 graphs covering 534 ops with 5 graph breaks (4 unique) 2024-06-26T07:38:08.2671382Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:38:08.2673074Z warnings.warn( 2024-06-26T07:38:08.5045120Z 2024-06-26T07:38:11.1868723Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:38:11.1869283Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:38:11.1869859Z cuda train AllenaiLongformerBase 2024-06-26T07:38:19.3841388Z W0626 07:38:19.383000 140013065077376 torch/_dynamo/variables/tensor.py:714] [3/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T07:38:19.3842781Z W0626 07:38:19.383000 140013065077376 torch/_dynamo/variables/tensor.py:714] [3/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T07:38:19.3843867Z W0626 07:38:19.383000 140013065077376 torch/_dynamo/variables/tensor.py:714] [3/0] or: 2024-06-26T07:38:19.3844899Z W0626 07:38:19.383000 140013065077376 torch/_dynamo/variables/tensor.py:714] [3/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T07:38:19.3846190Z W0626 07:38:19.383000 140013065077376 torch/_dynamo/variables/tensor.py:714] [3/0] to include these operations in the captured graph. 2024-06-26T07:38:19.3847259Z W0626 07:38:19.383000 140013065077376 torch/_dynamo/variables/tensor.py:714] [3/0] 2024-06-26T07:39:41.4294134Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:39:41.4296636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1318, in torch_dynamo_resume_in_forward_at_1291 2024-06-26T07:39:41.4297772Z layer_outputs = layer_module( 2024-06-26T07:39:41.4298662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:39:41.4299481Z return forward_call(*args, **kwargs) 2024-06-26T07:39:41.4300533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1246, in forward 2024-06-26T07:39:41.4302002Z self_attn_outputs = self.attention( 2024-06-26T07:39:41.4303234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:39:41.4304139Z return forward_call(*args, **kwargs) 2024-06-26T07:39:41.4305453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1182, in forward 2024-06-26T07:39:41.4306408Z self_outputs = self.self( 2024-06-26T07:39:41.4307273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:39:41.4308090Z return forward_call(*args, **kwargs) 2024-06-26T07:39:41.4309134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 571, in forward 2024-06-26T07:39:41.4310147Z attn_scores = self._sliding_chunks_query_key_matmul( 2024-06-26T07:39:41.4311386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 836, in _sliding_chunks_query_key_matmul 2024-06-26T07:39:41.4312680Z query = self._chunk(query, window_overlap, getattr(self.config, "onnx_export", False)) 2024-06-26T07:39:41.4315417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 778, in _chunk 2024-06-26T07:39:41.4316526Z return hidden_states.as_strided(size=chunk_size, stride=chunk_stride) 2024-06-26T07:39:41.4316967Z 2024-06-26T07:39:41.9539341Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:39:41.9540833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1734, in forward 2024-06-26T07:39:41.9541931Z embedding_output = self.embeddings( 2024-06-26T07:39:41.9542991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:39:41.9543927Z return forward_call(*args, **kwargs) 2024-06-26T07:39:41.9545065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 470, in forward 2024-06-26T07:39:41.9546188Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T07:39:41.9546537Z 2024-06-26T07:39:41.9611524Z W0626 07:39:41.960000 140013065077376 torch/_logging/_internal.py:1040] [16/0] Profiler function will be ignored 2024-06-26T07:40:39.9211543Z pass 2024-06-26T07:40:39.9697210Z TIMING: entire_frame_compile:107.8651 inductor_compile:74.87563 backend_compile:91.67233 code_gen:39.26172 2024-06-26T07:40:39.9698659Z STATS: call_* op count: 2367 | FakeTensorMode.__torch_dispatch__:186807 | FakeTensor.__torch_dispatch__:21002 | ProxyTorchDispatchMode.__torch_dispatch__:58074 2024-06-26T07:40:39.9699799Z Dynamo produced 7 graphs covering 2367 ops with 9 graph breaks (5 unique) 2024-06-26T07:40:48.8108442Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:40:48.8110121Z warnings.warn( 2024-06-26T07:40:49.0653938Z 2024-06-26T07:40:52.7513517Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:40:52.7514467Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:40:52.7515050Z cuda train BartForCausalLM 2024-06-26T07:41:15.6809202Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:41:15.6810502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-06-26T07:41:15.6811607Z return super().forward(positions + self.offset) 2024-06-26T07:41:15.6812837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T07:41:15.6813944Z return F.embedding( 2024-06-26T07:41:15.6814171Z 2024-06-26T07:41:22.3940540Z pass 2024-06-26T07:41:22.5184079Z TIMING: entire_frame_compile:12.61604 inductor_compile:4.35694 backend_compile:11.00457 code_gen:2.09229 2024-06-26T07:41:22.5185369Z STATS: call_* op count: 325 | FakeTensorMode.__torch_dispatch__:32936 | FakeTensor.__torch_dispatch__:2534 | ProxyTorchDispatchMode.__torch_dispatch__:15974 2024-06-26T07:41:22.5186461Z Dynamo produced 16 graphs covering 325 ops with 6 graph breaks (5 unique) 2024-06-26T07:41:26.3484053Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:41:26.3485773Z warnings.warn( 2024-06-26T07:41:26.5373172Z 2024-06-26T07:41:33.0897925Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:41:33.0898435Z loading model: 0it [00:06, ?it/s] 2024-06-26T07:41:33.0898889Z cuda train BartForConditionalGeneration 2024-06-26T07:42:19.3808170Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:42:19.3812389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-06-26T07:42:19.3813673Z return super().forward(positions + self.offset) 2024-06-26T07:42:19.3814840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T07:42:19.3815703Z return F.embedding( 2024-06-26T07:42:19.3816005Z 2024-06-26T07:42:20.7926391Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:42:20.7928017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-06-26T07:42:20.7929009Z return super().forward(positions + self.offset) 2024-06-26T07:42:20.7929949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T07:42:20.7930744Z return F.embedding( 2024-06-26T07:42:20.7931234Z 2024-06-26T07:42:41.2537304Z pass 2024-06-26T07:42:41.4535624Z TIMING: entire_frame_compile:27.25021 inductor_compile:7.707 backend_compile:23.36986 code_gen:3.00197 2024-06-26T07:42:41.4538156Z STATS: call_* op count: 878 | FakeTensorMode.__torch_dispatch__:83431 | FakeTensor.__torch_dispatch__:6224 | ProxyTorchDispatchMode.__torch_dispatch__:41429 2024-06-26T07:42:41.4539289Z Dynamo produced 30 graphs covering 878 ops with 8 graph breaks (5 unique) 2024-06-26T07:42:45.7533747Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:42:45.7535483Z warnings.warn( 2024-06-26T07:42:45.9393912Z 2024-06-26T07:42:47.7408653Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:42:47.7409122Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:42:47.7409590Z cuda train BertForMaskedLM 2024-06-26T07:43:23.5713048Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:43:23.5715641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:43:23.5716697Z pred = mod(**cloned_inputs) 2024-06-26T07:43:23.5718060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:43:23.5719239Z return forward_call(*args, **kwargs) 2024-06-26T07:43:23.5720384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward 2024-06-26T07:43:23.5721548Z outputs = self.bert( 2024-06-26T07:43:23.5722405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:43:23.5723729Z return forward_call(*args, **kwargs) 2024-06-26T07:43:23.5724831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-06-26T07:43:23.5725748Z embedding_output = self.embeddings( 2024-06-26T07:43:23.5726656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:43:23.5727490Z return forward_call(*args, **kwargs) 2024-06-26T07:43:23.5728467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-06-26T07:43:23.5729391Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T07:43:23.5729733Z 2024-06-26T07:43:23.6560781Z W0626 07:43:23.655000 140094290969216 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:44:13.3727069Z pass 2024-06-26T07:44:13.4343386Z TIMING: entire_frame_compile:69.34175 code_gen:23.59344 inductor_compile:43.94832 backend_compile:57.36998 2024-06-26T07:44:13.4346184Z STATS: call_* op count: 996 | FakeTensorMode.__torch_dispatch__:109856 | FakeTensor.__torch_dispatch__:18352 | ProxyTorchDispatchMode.__torch_dispatch__:26951 2024-06-26T07:44:13.4347350Z Dynamo produced 2 graphs covering 996 ops with 5 graph breaks (4 unique) 2024-06-26T07:44:20.4891247Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:44:20.4893033Z warnings.warn( 2024-06-26T07:44:20.7727176Z 2024-06-26T07:44:22.4170425Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:44:22.4171077Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:44:22.4173583Z cuda train BertForQuestionAnswering 2024-06-26T07:44:56.8481510Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:44:56.8482748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:44:56.8483984Z pred = mod(**cloned_inputs) 2024-06-26T07:44:56.8485178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:44:56.8486029Z return forward_call(*args, **kwargs) 2024-06-26T07:44:56.8487360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1846, in forward 2024-06-26T07:44:56.8488490Z outputs = self.bert( 2024-06-26T07:44:56.8489537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:44:56.8490696Z return forward_call(*args, **kwargs) 2024-06-26T07:44:56.8492055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-06-26T07:44:56.8493176Z embedding_output = self.embeddings( 2024-06-26T07:44:56.8494303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:44:56.8495144Z return forward_call(*args, **kwargs) 2024-06-26T07:44:56.8496141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-06-26T07:44:56.8497086Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T07:44:56.8497433Z 2024-06-26T07:44:56.9243231Z W0626 07:44:56.923000 140229724111488 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:45:46.3317247Z pass 2024-06-26T07:45:46.3960761Z TIMING: entire_frame_compile:68.53031 code_gen:22.72823 inductor_compile:43.44254 backend_compile:56.81272 2024-06-26T07:45:46.3963915Z STATS: call_* op count: 994 | FakeTensorMode.__torch_dispatch__:108909 | FakeTensor.__torch_dispatch__:18133 | ProxyTorchDispatchMode.__torch_dispatch__:26764 2024-06-26T07:45:46.3965739Z Dynamo produced 2 graphs covering 994 ops with 5 graph breaks (4 unique) 2024-06-26T07:45:53.5189107Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:45:53.5190794Z warnings.warn( 2024-06-26T07:45:53.7467351Z 2024-06-26T07:46:20.8282407Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:46:20.8283948Z loading model: 0it [00:27, ?it/s] 2024-06-26T07:46:20.8284484Z cuda train BlenderbotForCausalLM 2024-06-26T07:46:20.8289329Z Traceback (most recent call last): 2024-06-26T07:46:20.8290553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T07:46:20.8291986Z self.model_iter_fn(model, example_inputs) 2024-06-26T07:46:20.8293322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in forward_and_backward_pass 2024-06-26T07:46:20.8294605Z pred = mod(**cloned_inputs) 2024-06-26T07:46:20.8296130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:46:20.8297469Z return self._call_impl(*args, **kwargs) 2024-06-26T07:46:20.8298859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:46:20.8300101Z return forward_call(*args, **kwargs) 2024-06-26T07:46:20.8301693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 1531, in forward 2024-06-26T07:46:20.8303281Z outputs = self.model.decoder( 2024-06-26T07:46:20.8304742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:46:20.8306077Z return self._call_impl(*args, **kwargs) 2024-06-26T07:46:20.8307442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:46:20.8308682Z return forward_call(*args, **kwargs) 2024-06-26T07:46:20.8310233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 997, in forward 2024-06-26T07:46:20.8311646Z layer_outputs = decoder_layer( 2024-06-26T07:46:20.8313030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:46:20.8314394Z return self._call_impl(*args, **kwargs) 2024-06-26T07:46:20.8315751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:46:20.8316976Z return forward_call(*args, **kwargs) 2024-06-26T07:46:20.8318547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 397, in forward 2024-06-26T07:46:20.8320209Z hidden_states, self_attn_weights, present_key_value = self.self_attn( 2024-06-26T07:46:20.8321867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:46:20.8323183Z return self._call_impl(*args, **kwargs) 2024-06-26T07:46:20.8324560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:46:20.8325808Z return forward_call(*args, **kwargs) 2024-06-26T07:46:20.8327395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 152, in forward 2024-06-26T07:46:20.8328933Z query_states = self.q_proj(hidden_states) * self.scaling 2024-06-26T07:46:20.8331232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:46:20.8332688Z return self._call_impl(*args, **kwargs) 2024-06-26T07:46:20.8333967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:46:20.8335153Z return forward_call(*args, **kwargs) 2024-06-26T07:46:20.8336450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 125, in forward 2024-06-26T07:46:20.8337705Z return F.linear(input, self.weight, self.bias) 2024-06-26T07:46:20.8342423Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 14.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 6.44 MiB is free. Process 22916 has 21.96 GiB memory in use. Of the allocated memory 21.61 GiB is allocated by PyTorch, and 27.77 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T07:46:20.8346320Z 2024-06-26T07:46:20.8346766Z The above exception was the direct cause of the following exception: 2024-06-26T07:46:20.8347439Z 2024-06-26T07:46:20.8347647Z Traceback (most recent call last): 2024-06-26T07:46:20.8348598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T07:46:20.8349562Z ) = runner.load_model( 2024-06-26T07:46:20.8350519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 522, in load_model 2024-06-26T07:46:20.8351686Z self.validate_model(model, example_inputs) 2024-06-26T07:46:20.8352789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T07:46:20.8353925Z raise RuntimeError("Eager run failed") from e 2024-06-26T07:46:20.8354645Z RuntimeError: Eager run failed 2024-06-26T07:46:20.8355023Z 2024-06-26T07:46:20.8355183Z eager_fail_to_run 2024-06-26T07:46:24.1268170Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:46:24.1269914Z warnings.warn( 2024-06-26T07:46:24.3285294Z 2024-06-26T07:46:25.6252761Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:46:25.6253236Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:46:25.6253697Z cuda train BlenderbotSmallForCausalLM 2024-06-26T07:46:25.6469938Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForCausalLM. Setting accuracy check to cosine 2024-06-26T07:46:40.0387733Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:46:40.0389209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 90, in forward 2024-06-26T07:46:40.0390250Z return super().forward(positions) 2024-06-26T07:46:40.0391131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T07:46:40.0391909Z return F.embedding( 2024-06-26T07:46:40.0392125Z 2024-06-26T07:46:44.5685158Z pass 2024-06-26T07:46:44.5964643Z TIMING: entire_frame_compile:10.5917 inductor_compile:4.72127 backend_compile:9.06771 code_gen:2.67269 2024-06-26T07:46:44.5966000Z STATS: call_* op count: 324 | FakeTensorMode.__torch_dispatch__:25848 | FakeTensor.__torch_dispatch__:1921 | ProxyTorchDispatchMode.__torch_dispatch__:12278 2024-06-26T07:46:44.5967127Z Dynamo produced 13 graphs covering 324 ops with 6 graph breaks (5 unique) 2024-06-26T07:46:48.3812811Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:46:48.3814915Z warnings.warn( 2024-06-26T07:46:48.5845052Z 2024-06-26T07:46:50.2328902Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:46:50.2329385Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:46:50.2329880Z cuda train BlenderbotSmallForConditionalGeneration 2024-06-26T07:46:50.3733609Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:47:16.0214400Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:47:16.0217411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 90, in forward 2024-06-26T07:47:16.0219965Z return super().forward(positions) 2024-06-26T07:47:16.0221178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T07:47:16.0221979Z return F.embedding( 2024-06-26T07:47:16.0222216Z 2024-06-26T07:47:17.1696185Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:47:17.1697807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 90, in forward 2024-06-26T07:47:17.1698829Z return super().forward(positions) 2024-06-26T07:47:17.1699710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T07:47:17.1700492Z return F.embedding( 2024-06-26T07:47:17.1700711Z 2024-06-26T07:47:29.3963717Z pass 2024-06-26T07:47:29.4262422Z TIMING: entire_frame_compile:21.53742 inductor_compile:7.1113 backend_compile:18.11766 code_gen:2.64477 2024-06-26T07:47:29.4263824Z STATS: call_* op count: 830 | FakeTensorMode.__torch_dispatch__:64219 | FakeTensor.__torch_dispatch__:4523 | ProxyTorchDispatchMode.__torch_dispatch__:31624 2024-06-26T07:47:29.4264980Z Dynamo produced 22 graphs covering 830 ops with 8 graph breaks (5 unique) 2024-06-26T07:47:33.5374909Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:47:33.5376574Z warnings.warn( 2024-06-26T07:47:33.7160206Z 2024-06-26T07:47:35.5655493Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:47:35.5656050Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:47:35.5656485Z cuda train CamemBert 2024-06-26T07:48:11.3144538Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:48:11.3146564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:48:11.3148059Z pred = mod(**cloned_inputs) 2024-06-26T07:48:11.3149255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:48:11.3150115Z return forward_call(*args, **kwargs) 2024-06-26T07:48:11.3151201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/camembert/modeling_camembert.py", line 979, in forward 2024-06-26T07:48:11.3152127Z outputs = self.roberta( 2024-06-26T07:48:11.3153125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:48:11.3153970Z return forward_call(*args, **kwargs) 2024-06-26T07:48:11.3155014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/camembert/modeling_camembert.py", line 881, in forward 2024-06-26T07:48:11.3155943Z embedding_output = self.embeddings( 2024-06-26T07:48:11.3156839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:48:11.3157983Z return forward_call(*args, **kwargs) 2024-06-26T07:48:11.3159130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/camembert/modeling_camembert.py", line 139, in forward 2024-06-26T07:48:11.3160111Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T07:48:11.3160452Z 2024-06-26T07:48:11.6134219Z W0626 07:48:11.612000 140197759009408 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:49:01.6537112Z pass 2024-06-26T07:49:01.7169784Z TIMING: entire_frame_compile:69.55668 code_gen:23.46858 inductor_compile:44.01576 backend_compile:57.5869 2024-06-26T07:49:01.7172067Z STATS: call_* op count: 1004 | FakeTensorMode.__torch_dispatch__:110009 | FakeTensor.__torch_dispatch__:18376 | ProxyTorchDispatchMode.__torch_dispatch__:26999 2024-06-26T07:49:01.7173226Z Dynamo produced 2 graphs covering 1004 ops with 5 graph breaks (4 unique) 2024-06-26T07:49:08.8816181Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:49:08.8817934Z warnings.warn( 2024-06-26T07:49:09.1370178Z 2024-06-26T07:49:11.3364075Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:49:11.3364556Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:49:11.3365103Z cuda train DebertaForMaskedLM 2024-06-26T07:49:49.9419800Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:49:49.9420973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:49:49.9421931Z pred = mod(**cloned_inputs) 2024-06-26T07:49:49.9423036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:49:49.9423897Z return forward_call(*args, **kwargs) 2024-06-26T07:49:49.9424914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 1062, in forward 2024-06-26T07:49:49.9425828Z outputs = self.deberta( 2024-06-26T07:49:49.9426665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:49:49.9427723Z return forward_call(*args, **kwargs) 2024-06-26T07:49:49.9428991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 966, in forward 2024-06-26T07:49:49.9429901Z embedding_output = self.embeddings( 2024-06-26T07:49:49.9430803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:49:49.9431614Z return forward_call(*args, **kwargs) 2024-06-26T07:49:49.9432609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 780, in forward 2024-06-26T07:49:49.9433558Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T07:49:49.9433902Z 2024-06-26T07:49:50.2506534Z W0626 07:49:50.249000 139733538955904 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:50:32.4099054Z pass 2024-06-26T07:50:32.4687162Z TIMING: entire_frame_compile:63.88411 code_gen:23.09153 inductor_compile:42.29989 backend_compile:53.00991 2024-06-26T07:50:32.4689756Z STATS: call_* op count: 1319 | FakeTensorMode.__torch_dispatch__:103763 | FakeTensor.__torch_dispatch__:16054 | ProxyTorchDispatchMode.__torch_dispatch__:28168 2024-06-26T07:50:32.4691399Z Dynamo produced 2 graphs covering 1319 ops with 5 graph breaks (4 unique) 2024-06-26T07:50:39.1994686Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:50:39.1996784Z warnings.warn( 2024-06-26T07:50:39.4574462Z 2024-06-26T07:50:41.2690499Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:50:41.2691582Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:50:41.2693005Z cuda train DebertaForQuestionAnswering 2024-06-26T07:51:17.1748429Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:51:17.1751885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:51:17.1753149Z pred = mod(**cloned_inputs) 2024-06-26T07:51:17.1754292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:51:17.1755137Z return forward_call(*args, **kwargs) 2024-06-26T07:51:17.1756371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 1388, in forward 2024-06-26T07:51:17.1757381Z outputs = self.deberta( 2024-06-26T07:51:17.1758256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:51:17.1759099Z return forward_call(*args, **kwargs) 2024-06-26T07:51:17.1760178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 966, in forward 2024-06-26T07:51:17.1761114Z embedding_output = self.embeddings( 2024-06-26T07:51:17.1762038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:51:17.1762871Z return forward_call(*args, **kwargs) 2024-06-26T07:51:17.1763907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 780, in forward 2024-06-26T07:51:17.1764871Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T07:51:17.1765221Z 2024-06-26T07:51:17.2447437Z W0626 07:51:17.243000 140598778253952 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:51:56.8533651Z pass 2024-06-26T07:51:56.9188140Z TIMING: entire_frame_compile:60.12607 code_gen:19.67866 inductor_compile:38.85265 backend_compile:49.36672 2024-06-26T07:51:56.9190405Z STATS: call_* op count: 1317 | FakeTensorMode.__torch_dispatch__:102777 | FakeTensor.__torch_dispatch__:15836 | ProxyTorchDispatchMode.__torch_dispatch__:27972 2024-06-26T07:51:56.9191571Z Dynamo produced 2 graphs covering 1317 ops with 5 graph breaks (4 unique) 2024-06-26T07:52:03.6823588Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:52:03.6825279Z warnings.warn( 2024-06-26T07:52:03.9047207Z 2024-06-26T07:52:15.8276655Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:52:15.8277165Z loading model: 0it [00:11, ?it/s] 2024-06-26T07:52:15.8280138Z cuda train DebertaV2ForMaskedLM 2024-06-26T07:52:15.8280703Z pass_due_to_skip 2024-06-26T07:52:16.0482635Z TIMING: 2024-06-26T07:52:16.0483414Z STATS: call_* op count: 0 2024-06-26T07:52:16.0484661Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T07:52:19.3339388Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:52:19.3342073Z warnings.warn( 2024-06-26T07:52:19.5453302Z 2024-06-26T07:52:29.5023983Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:52:29.5024454Z loading model: 0it [00:09, ?it/s] 2024-06-26T07:52:29.5025255Z cuda train DebertaV2ForQuestionAnswering 2024-06-26T07:52:40.1365637Z ERROR:common: 2024-06-26T07:52:40.1366523Z Traceback (most recent call last): 2024-06-26T07:52:40.1367462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2543, in check_accuracy 2024-06-26T07:52:40.1368402Z correct_result = self.run_n_iterations( 2024-06-26T07:52:40.1369357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2359, in run_n_iterations 2024-06-26T07:52:40.1370355Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-06-26T07:52:40.1371912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in forward_and_backward_pass 2024-06-26T07:52:40.1372918Z pred = mod(**cloned_inputs) 2024-06-26T07:52:40.1374258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:52:40.1375280Z return self._call_impl(*args, **kwargs) 2024-06-26T07:52:40.1376234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:52:40.1377372Z return forward_call(*args, **kwargs) 2024-06-26T07:52:40.1378684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1486, in forward 2024-06-26T07:52:40.1379795Z outputs = self.deberta( 2024-06-26T07:52:40.1380713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:52:40.1381622Z return self._call_impl(*args, **kwargs) 2024-06-26T07:52:40.1382553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:52:40.1383511Z return forward_call(*args, **kwargs) 2024-06-26T07:52:40.1384568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1070, in forward 2024-06-26T07:52:40.1385540Z encoder_outputs = self.encoder( 2024-06-26T07:52:40.1386517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:52:40.1387416Z return self._call_impl(*args, **kwargs) 2024-06-26T07:52:40.1388344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:52:40.1389179Z return forward_call(*args, **kwargs) 2024-06-26T07:52:40.1390233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 514, in forward 2024-06-26T07:52:40.1391174Z output_states = layer_module( 2024-06-26T07:52:40.1392123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:52:40.1393027Z return self._call_impl(*args, **kwargs) 2024-06-26T07:52:40.1393959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:52:40.1394803Z return forward_call(*args, **kwargs) 2024-06-26T07:52:40.1395863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 362, in forward 2024-06-26T07:52:40.1396817Z attention_output = self.attention( 2024-06-26T07:52:40.1397780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:52:40.1398678Z return self._call_impl(*args, **kwargs) 2024-06-26T07:52:40.1399660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:52:40.1400498Z return forward_call(*args, **kwargs) 2024-06-26T07:52:40.1401543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 293, in forward 2024-06-26T07:52:40.1404935Z self_output = self.self( 2024-06-26T07:52:40.1406826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:52:40.1407743Z return self._call_impl(*args, **kwargs) 2024-06-26T07:52:40.1408663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:52:40.1409551Z return forward_call(*args, **kwargs) 2024-06-26T07:52:40.1410577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 733, in forward 2024-06-26T07:52:40.1411979Z attention_probs = XSoftmax.apply(attention_scores, attention_mask, -1) 2024-06-26T07:52:40.1413011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 574, in apply 2024-06-26T07:52:40.1413894Z return super().apply(*args, **kwargs) # type: ignore[misc] 2024-06-26T07:52:40.1415024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 113, in forward 2024-06-26T07:52:40.1416010Z output = torch.softmax(output, self.dim) 2024-06-26T07:52:40.1419220Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 10.44 MiB is free. Process 24450 has 21.96 GiB memory in use. Of the allocated memory 21.29 GiB is allocated by PyTorch, and 318.25 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T07:52:40.1684221Z eager_1st_run_OOM 2024-06-26T07:52:40.1684543Z TIMING: 2024-06-26T07:52:40.1684819Z STATS: call_* op count: 0 2024-06-26T07:52:40.1685355Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T07:52:43.4792884Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:52:43.4794544Z warnings.warn( 2024-06-26T07:52:43.6654712Z 2024-06-26T07:52:44.9693891Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:52:44.9694784Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:52:44.9695671Z cuda train DistilBertForMaskedLM 2024-06-26T07:53:03.5934354Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:53:03.5935536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:53:03.5936656Z pred = mod(**cloned_inputs) 2024-06-26T07:53:03.5937683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:53:03.5938627Z return forward_call(*args, **kwargs) 2024-06-26T07:53:03.5939973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 905, in forward 2024-06-26T07:53:03.5941178Z dlbrt_output = self.distilbert( 2024-06-26T07:53:03.5942081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:53:03.5953542Z return forward_call(*args, **kwargs) 2024-06-26T07:53:03.5954834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2024-06-26T07:53:03.5955987Z embeddings = self.embeddings(input_ids, inputs_embeds) # (bs, seq_length, dim) 2024-06-26T07:53:03.5957092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:53:03.5957927Z return forward_call(*args, **kwargs) 2024-06-26T07:53:03.5959324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 141, in forward 2024-06-26T07:53:03.5960445Z input_embeds = self.word_embeddings(input_ids) # (bs, max_seq_length, dim) 2024-06-26T07:53:03.5960923Z 2024-06-26T07:53:03.7549011Z W0626 07:53:03.754000 140266578395776 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:53:31.1310392Z pass 2024-06-26T07:53:31.1699973Z TIMING: entire_frame_compile:37.892 code_gen:13.94666 inductor_compile:24.85002 backend_compile:32.40688 2024-06-26T07:53:31.1702203Z STATS: call_* op count: 541 | FakeTensor.__torch_dispatch__:9534 | FakeTensorMode.__torch_dispatch__:58009 | ProxyTorchDispatchMode.__torch_dispatch__:14211 2024-06-26T07:53:31.1703620Z Dynamo produced 2 graphs covering 541 ops with 5 graph breaks (4 unique) 2024-06-26T07:53:36.4438332Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:53:36.4439990Z warnings.warn( 2024-06-26T07:53:36.6643079Z 2024-06-26T07:53:37.7464817Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:53:37.7465292Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:53:37.7465757Z cuda train DistilBertForQuestionAnswering 2024-06-26T07:53:55.3542628Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:53:55.3543884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:53:55.3545959Z pred = mod(**cloned_inputs) 2024-06-26T07:53:55.3547328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:53:55.3548518Z return forward_call(*args, **kwargs) 2024-06-26T07:53:55.3549657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1124, in forward 2024-06-26T07:53:55.3550708Z distilbert_output = self.distilbert( 2024-06-26T07:53:55.3551902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:53:55.3552733Z return forward_call(*args, **kwargs) 2024-06-26T07:53:55.3553781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2024-06-26T07:53:55.3554918Z embeddings = self.embeddings(input_ids, inputs_embeds) # (bs, seq_length, dim) 2024-06-26T07:53:55.3556012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:53:55.3556834Z return forward_call(*args, **kwargs) 2024-06-26T07:53:55.3557882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 141, in forward 2024-06-26T07:53:55.3559007Z input_embeds = self.word_embeddings(input_ids) # (bs, max_seq_length, dim) 2024-06-26T07:53:55.3559474Z 2024-06-26T07:53:55.4179851Z W0626 07:53:55.417000 139795075003008 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:54:19.1042365Z pass 2024-06-26T07:54:19.1416719Z TIMING: entire_frame_compile:33.68234 code_gen:9.84753 inductor_compile:20.56818 backend_compile:28.16787 2024-06-26T07:54:19.1418076Z STATS: call_* op count: 540 | FakeTensor.__torch_dispatch__:9327 | FakeTensorMode.__torch_dispatch__:57126 | ProxyTorchDispatchMode.__torch_dispatch__:14037 2024-06-26T07:54:19.1419213Z Dynamo produced 2 graphs covering 540 ops with 5 graph breaks (4 unique) 2024-06-26T07:54:24.3573689Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:54:24.3575532Z warnings.warn( 2024-06-26T07:54:24.5447836Z 2024-06-26T07:54:26.3193838Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:54:26.3194525Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:54:26.3195842Z cuda train DistillGPT2 2024-06-26T07:54:45.7566490Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:54:45.7568058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:54:45.7569164Z pred = mod(**cloned_inputs) 2024-06-26T07:54:45.7570466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:54:45.7571547Z return forward_call(*args, **kwargs) 2024-06-26T07:54:45.7572720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1074, in forward 2024-06-26T07:54:45.7573823Z transformer_outputs = self.transformer( 2024-06-26T07:54:45.7574761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:54:45.7575583Z return forward_call(*args, **kwargs) 2024-06-26T07:54:45.7576553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 837, in forward 2024-06-26T07:54:45.7577479Z inputs_embeds = self.wte(input_ids) 2024-06-26T07:54:45.7577769Z 2024-06-26T07:54:45.8164623Z W0626 07:54:45.815000 139813089370752 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:55:07.7163250Z pass 2024-06-26T07:55:07.7516531Z TIMING: entire_frame_compile:32.18105 code_gen:12.85042 inductor_compile:22.13355 backend_compile:26.98758 2024-06-26T07:55:07.7517854Z STATS: call_* op count: 572 | FakeTensorMode.__torch_dispatch__:46229 | FakeTensor.__torch_dispatch__:7295 | ProxyTorchDispatchMode.__torch_dispatch__:11477 2024-06-26T07:55:07.7518965Z Dynamo produced 2 graphs covering 572 ops with 5 graph breaks (4 unique) 2024-06-26T07:55:12.6516773Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:55:12.6518612Z warnings.warn( 2024-06-26T07:55:12.8330092Z 2024-06-26T07:55:12.8338121Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2024-06-26T07:55:13.4280945Z 2024-06-26T07:55:13.4281594Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:55:13.4282097Z cuda train ElectraForCausalLM 2024-06-26T07:55:47.1686073Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:55:47.1687369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:55:47.1688435Z pred = mod(**cloned_inputs) 2024-06-26T07:55:47.1689565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:55:47.1690462Z return forward_call(*args, **kwargs) 2024-06-26T07:55:47.1691900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1617, in forward 2024-06-26T07:55:47.1692929Z outputs = self.electra( 2024-06-26T07:55:47.1693847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:55:47.1694764Z return forward_call(*args, **kwargs) 2024-06-26T07:55:47.1696130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 902, in forward 2024-06-26T07:55:47.1697366Z hidden_states = self.embeddings( 2024-06-26T07:55:47.1698382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:55:47.1699220Z return forward_call(*args, **kwargs) 2024-06-26T07:55:47.1700211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 203, in forward 2024-06-26T07:55:47.1701170Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T07:55:47.1701517Z 2024-06-26T07:55:47.8810770Z pass 2024-06-26T07:55:47.8992469Z TIMING: entire_frame_compile:22.0351 code_gen:8.5879 inductor_compile:16.79081 backend_compile:17.26832 2024-06-26T07:55:47.8993757Z STATS: call_* op count: 377 | FakeTensorMode.__torch_dispatch__:56448 | FakeTensor.__torch_dispatch__:8074 | ProxyTorchDispatchMode.__torch_dispatch__:17988 2024-06-26T07:55:47.8994872Z Dynamo produced 1 graphs covering 377 ops with 4 graph breaks (4 unique) 2024-06-26T07:55:52.7354758Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:55:52.7357375Z warnings.warn( 2024-06-26T07:55:52.9240127Z 2024-06-26T07:55:53.5047483Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:55:53.5047953Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:55:53.5048422Z cuda train ElectraForQuestionAnswering 2024-06-26T07:56:26.7573800Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:56:26.7575125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:56:26.7579160Z pred = mod(**cloned_inputs) 2024-06-26T07:56:26.7580503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:56:26.7581740Z return forward_call(*args, **kwargs) 2024-06-26T07:56:26.7583238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1385, in forward 2024-06-26T07:56:26.7584223Z discriminator_hidden_states = self.electra( 2024-06-26T07:56:26.7585171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:56:26.7585999Z return forward_call(*args, **kwargs) 2024-06-26T07:56:26.7587013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 902, in forward 2024-06-26T07:56:26.7587916Z hidden_states = self.embeddings( 2024-06-26T07:56:26.7588811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:56:26.7589640Z return forward_call(*args, **kwargs) 2024-06-26T07:56:26.7590649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 203, in forward 2024-06-26T07:56:26.7591593Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T07:56:26.7591939Z 2024-06-26T07:56:26.8341085Z W0626 07:56:26.833000 140369091822208 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:57:16.6006382Z pass 2024-06-26T07:57:16.6146549Z TIMING: entire_frame_compile:69.1381 code_gen:22.98167 inductor_compile:43.33279 backend_compile:57.38355 2024-06-26T07:57:16.6148180Z STATS: call_* op count: 1001 | FakeTensorMode.__torch_dispatch__:109856 | FakeTensor.__torch_dispatch__:18301 | ProxyTorchDispatchMode.__torch_dispatch__:26961 2024-06-26T07:57:16.6149335Z Dynamo produced 2 graphs covering 1001 ops with 5 graph breaks (4 unique) 2024-06-26T07:57:23.7165478Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:57:23.7167465Z warnings.warn( 2024-06-26T07:57:23.9917614Z 2024-06-26T07:57:26.0481679Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:57:26.0482395Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:57:26.0483140Z cuda train GPT2ForSequenceClassification 2024-06-26T07:58:02.0886859Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:58:02.0888280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1426, in forward 2024-06-26T07:58:02.0889326Z transformer_outputs = self.transformer( 2024-06-26T07:58:02.0890431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:58:02.0891610Z return forward_call(*args, **kwargs) 2024-06-26T07:58:02.0892739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 837, in forward 2024-06-26T07:58:02.0893704Z inputs_embeds = self.wte(input_ids) 2024-06-26T07:58:02.0894004Z 2024-06-26T07:58:02.1294461Z W0626 07:58:02.128000 140624392716928 torch/_logging/_internal.py:1040] [7/0] Profiler function will be ignored 2024-06-26T07:58:40.8937234Z pass 2024-06-26T07:58:40.9719218Z TIMING: entire_frame_compile:56.63702 inductor_compile:36.76579 backend_compile:45.73901 code_gen:19.71912 2024-06-26T07:58:40.9720514Z STATS: call_* op count: 1102 | FakeTensorMode.__torch_dispatch__:93384 | FakeTensor.__torch_dispatch__:14173 | ProxyTorchDispatchMode.__torch_dispatch__:22505 2024-06-26T07:58:40.9721788Z Dynamo produced 5 graphs covering 1102 ops with 7 graph breaks (5 unique) 2024-06-26T07:58:47.2427840Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:58:47.2429517Z warnings.warn( 2024-06-26T07:58:47.4445897Z 2024-06-26T07:58:48.7408246Z loading model: 0it [00:00, ?it/s]WARNING:common:Model GoogleFnet supports float32 only 2024-06-26T07:58:48.8934004Z 2024-06-26T07:58:48.8934608Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:58:48.8935302Z WARNING:common:Model GoogleFnet supports float32 only 2024-06-26T07:58:48.8946386Z cuda train GoogleFnet 2024-06-26T07:58:50.2232660Z WARNING:common:Model GoogleFnet supports float32 only 2024-06-26T07:59:07.5929029Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:59:07.5930372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:59:07.5931714Z pred = mod(**cloned_inputs) 2024-06-26T07:59:07.5932792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:59:07.5933899Z return forward_call(*args, **kwargs) 2024-06-26T07:59:07.5935087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 749, in forward 2024-06-26T07:59:07.5936164Z outputs = self.fnet( 2024-06-26T07:59:07.5937018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:59:07.5937854Z return forward_call(*args, **kwargs) 2024-06-26T07:59:07.5938827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 581, in forward 2024-06-26T07:59:07.5939719Z embedding_output = self.embeddings( 2024-06-26T07:59:07.5940627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:59:07.5941675Z return forward_call(*args, **kwargs) 2024-06-26T07:59:07.5942937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 148, in forward 2024-06-26T07:59:07.5943886Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T07:59:07.5944226Z 2024-06-26T07:59:07.6602168Z W0626 07:59:07.659000 139913490797184 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:59:35.5771547Z pass 2024-06-26T07:59:35.6187694Z TIMING: entire_frame_compile:36.77504 code_gen:14.74154 inductor_compile:26.05126 backend_compile:30.56341 2024-06-26T07:59:35.6188997Z STATS: call_* op count: 574 | FakeTensorMode.__torch_dispatch__:52210 | FakeTensor.__torch_dispatch__:9591 | ProxyTorchDispatchMode.__torch_dispatch__:12282 2024-06-26T07:59:35.6190105Z Dynamo produced 2 graphs covering 574 ops with 5 graph breaks (4 unique) 2024-06-26T07:59:40.8922328Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:59:40.8923979Z warnings.warn( 2024-06-26T07:59:41.1596927Z 2024-06-26T07:59:43.0362710Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:59:43.0363340Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:59:43.0363921Z cuda train LayoutLMForMaskedLM 2024-06-26T08:00:19.2705700Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:00:19.2706902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:00:19.2707880Z pred = mod(**cloned_inputs) 2024-06-26T08:00:19.2709088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:00:19.2709983Z return forward_call(*args, **kwargs) 2024-06-26T08:00:19.2711067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 938, in forward 2024-06-26T08:00:19.2711985Z outputs = self.layoutlm( 2024-06-26T08:00:19.2712863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:00:19.2713691Z return forward_call(*args, **kwargs) 2024-06-26T08:00:19.2714708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 821, in forward 2024-06-26T08:00:19.2715636Z embedding_output = self.embeddings( 2024-06-26T08:00:19.2716542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:00:19.2717370Z return forward_call(*args, **kwargs) 2024-06-26T08:00:19.2718440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 99, in forward 2024-06-26T08:00:19.2719412Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T08:00:19.2719757Z 2024-06-26T08:00:19.3549977Z W0626 08:00:19.354000 140408094245504 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T08:01:10.7831154Z pass 2024-06-26T08:01:10.8443570Z TIMING: entire_frame_compile:71.69174 code_gen:24.57957 inductor_compile:45.29335 backend_compile:59.32022 2024-06-26T08:01:10.8444975Z STATS: call_* op count: 1034 | FakeTensorMode.__torch_dispatch__:112112 | FakeTensor.__torch_dispatch__:18659 | ProxyTorchDispatchMode.__torch_dispatch__:27517 2024-06-26T08:01:10.8446109Z Dynamo produced 2 graphs covering 1034 ops with 5 graph breaks (4 unique) 2024-06-26T08:01:18.0199780Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:01:18.0201698Z warnings.warn( 2024-06-26T08:01:18.3074657Z 2024-06-26T08:01:19.9724944Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:01:19.9725419Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:01:19.9725890Z cuda train LayoutLMForSequenceClassification 2024-06-26T08:01:57.1384493Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:01:57.1386452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 1060, in forward 2024-06-26T08:01:57.1387882Z outputs = self.layoutlm( 2024-06-26T08:01:57.1389040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:01:57.1389967Z return forward_call(*args, **kwargs) 2024-06-26T08:01:57.1391146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 821, in forward 2024-06-26T08:01:57.1392244Z embedding_output = self.embeddings( 2024-06-26T08:01:57.1393262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:01:57.1394176Z return forward_call(*args, **kwargs) 2024-06-26T08:01:57.1395322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 99, in forward 2024-06-26T08:01:57.1396413Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T08:01:57.1396789Z 2024-06-26T08:01:57.2205297Z W0626 08:01:57.219000 140615386452608 torch/_logging/_internal.py:1040] [7/0] Profiler function will be ignored 2024-06-26T08:02:47.0505909Z pass 2024-06-26T08:02:47.1187179Z TIMING: entire_frame_compile:69.90423 inductor_compile:40.98454 backend_compile:55.4295 code_gen:20.12042 2024-06-26T08:02:47.1189193Z STATS: call_* op count: 1031 | FakeTensorMode.__torch_dispatch__:117000 | FakeTensor.__torch_dispatch__:18852 | ProxyTorchDispatchMode.__torch_dispatch__:27791 2024-06-26T08:02:47.1190809Z Dynamo produced 5 graphs covering 1031 ops with 7 graph breaks (5 unique) 2024-06-26T08:02:54.3199054Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:02:54.3200711Z warnings.warn( 2024-06-26T08:02:54.5288263Z 2024-06-26T08:03:04.4407029Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:03:04.4407512Z loading model: 0it [00:09, ?it/s] 2024-06-26T08:03:04.4408049Z cuda train M2M100ForConditionalGeneration 2024-06-26T08:03:04.5193884Z WARNING:common:fp64 golden ref were not generated for M2M100ForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T08:04:13.9801285Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:04:13.9803942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:04:13.9806477Z pred = mod(**cloned_inputs) 2024-06-26T08:04:13.9808568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:04:13.9809770Z return forward_call(*args, **kwargs) 2024-06-26T08:04:13.9811333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1275, in forward 2024-06-26T08:04:13.9812346Z outputs = self.model( 2024-06-26T08:04:13.9813224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:04:13.9814040Z return forward_call(*args, **kwargs) 2024-06-26T08:04:13.9815057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1162, in forward 2024-06-26T08:04:13.9816306Z encoder_outputs = self.encoder( 2024-06-26T08:04:13.9817201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:04:13.9818035Z return forward_call(*args, **kwargs) 2024-06-26T08:04:13.9819067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 763, in forward 2024-06-26T08:04:13.9820078Z inputs_embeds = self.embed_tokens(input_ids) * self.embed_scale 2024-06-26T08:04:13.9820494Z 2024-06-26T08:04:15.6274790Z pass 2024-06-26T08:04:15.7215335Z TIMING: entire_frame_compile:45.99666 code_gen:19.65365 inductor_compile:40.02889 backend_compile:40.40226 2024-06-26T08:04:15.7216750Z STATS: call_* op count: 1296 | FakeTensorMode.__torch_dispatch__:140959 | FakeTensor.__torch_dispatch__:17728 | ProxyTorchDispatchMode.__torch_dispatch__:47005 2024-06-26T08:04:15.7218030Z Dynamo produced 1 graphs covering 1296 ops with 4 graph breaks (4 unique) 2024-06-26T08:04:22.3065676Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:04:22.3068265Z warnings.warn( 2024-06-26T08:04:22.7813739Z 2024-06-26T08:04:26.4249577Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:04:26.4250056Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:04:26.4250510Z cuda train MBartForCausalLM 2024-06-26T08:04:26.4567086Z WARNING:common:fp64 golden ref were not generated for MBartForCausalLM. Setting accuracy check to cosine 2024-06-26T08:04:43.2883437Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:04:43.2884869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-06-26T08:04:43.2886108Z return super().forward(positions + self.offset) 2024-06-26T08:04:43.2887208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T08:04:43.2887993Z return F.embedding( 2024-06-26T08:04:43.2888220Z 2024-06-26T08:04:50.5289200Z pass 2024-06-26T08:04:50.5955907Z TIMING: entire_frame_compile:13.10073 inductor_compile:4.25062 backend_compile:11.23323 code_gen:1.77529 2024-06-26T08:04:50.5957203Z STATS: call_* op count: 478 | FakeTensorMode.__torch_dispatch__:36729 | FakeTensor.__torch_dispatch__:2464 | ProxyTorchDispatchMode.__torch_dispatch__:18211 2024-06-26T08:04:50.5958321Z Dynamo produced 17 graphs covering 478 ops with 6 graph breaks (5 unique) 2024-06-26T08:04:54.4659553Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:04:54.4661482Z warnings.warn( 2024-06-26T08:04:54.7299878Z 2024-06-26T08:05:01.5467598Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:05:01.5468085Z loading model: 0it [00:06, ?it/s] 2024-06-26T08:05:01.5468544Z cuda train MBartForConditionalGeneration 2024-06-26T08:05:01.6470886Z WARNING:common:fp64 golden ref were not generated for MBartForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T08:05:38.1135558Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:05:38.1137248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-06-26T08:05:38.1138374Z return super().forward(positions + self.offset) 2024-06-26T08:05:38.1139492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T08:05:38.1140890Z return F.embedding( 2024-06-26T08:05:38.1141121Z 2024-06-26T08:05:40.1100526Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:05:40.1101822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-06-26T08:05:40.1102858Z return super().forward(positions + self.offset) 2024-06-26T08:05:40.1103800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T08:05:40.1104596Z return F.embedding( 2024-06-26T08:05:40.1104817Z 2024-06-26T08:06:01.8782109Z pass 2024-06-26T08:06:01.9798859Z TIMING: entire_frame_compile:30.20383 inductor_compile:8.58119 backend_compile:25.91122 code_gen:3.07738 2024-06-26T08:06:01.9800289Z STATS: call_* op count: 1251 | FakeTensorMode.__torch_dispatch__:93404 | FakeTensor.__torch_dispatch__:6086 | ProxyTorchDispatchMode.__torch_dispatch__:47299 2024-06-26T08:06:01.9801800Z Dynamo produced 31 graphs covering 1251 ops with 8 graph breaks (5 unique) 2024-06-26T08:06:06.4000893Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:06:06.4002638Z warnings.warn( 2024-06-26T08:06:06.6325783Z 2024-06-26T08:06:10.5163700Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:06:10.5164340Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:06:10.5166668Z cuda train MT5ForConditionalGeneration 2024-06-26T08:07:06.1948835Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:07:06.1950619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:07:06.1951666Z pred = mod(**cloned_inputs) 2024-06-26T08:07:06.1952748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:07:06.1953688Z return forward_call(*args, **kwargs) 2024-06-26T08:07:06.1954775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1722, in forward 2024-06-26T08:07:06.1955739Z encoder_outputs = self.encoder( 2024-06-26T08:07:06.1956698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:07:06.1957763Z return forward_call(*args, **kwargs) 2024-06-26T08:07:06.1958981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 989, in forward 2024-06-26T08:07:06.1959899Z inputs_embeds = self.embed_tokens(input_ids) 2024-06-26T08:07:06.1960231Z 2024-06-26T08:07:06.2845390Z W0626 08:07:06.283000 140469690872448 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T08:07:58.9075404Z pass 2024-06-26T08:07:59.0517388Z TIMING: entire_frame_compile:84.89655 code_gen:27.22372 inductor_compile:53.85825 backend_compile:67.7078 2024-06-26T08:07:59.0518681Z STATS: call_* op count: 1764 | FakeTensorMode.__torch_dispatch__:155529 | FakeTensor.__torch_dispatch__:22500 | ProxyTorchDispatchMode.__torch_dispatch__:44560 2024-06-26T08:07:59.0522130Z Dynamo produced 2 graphs covering 1764 ops with 5 graph breaks (4 unique) 2024-06-26T08:08:06.8216916Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:08:06.8219610Z warnings.warn( 2024-06-26T08:08:07.1124930Z 2024-06-26T08:08:07.1175393Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2024-06-26T08:08:11.5488166Z 2024-06-26T08:08:11.5488696Z loading model: 0it [00:04, ?it/s] 2024-06-26T08:08:11.5489745Z cuda train MegatronBertForCausalLM 2024-06-26T08:09:19.5200927Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:09:19.5203386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:09:19.5204701Z pred = mod(**cloned_inputs) 2024-06-26T08:09:19.5205991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:09:19.5207057Z return forward_call(*args, **kwargs) 2024-06-26T08:09:19.5208270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1199, in forward 2024-06-26T08:09:19.5209645Z outputs = self.bert( 2024-06-26T08:09:19.5210688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:09:19.5211837Z return forward_call(*args, **kwargs) 2024-06-26T08:09:19.5212985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 967, in forward 2024-06-26T08:09:19.5214126Z embedding_output = self.embeddings( 2024-06-26T08:09:19.5215133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:09:19.5216022Z return forward_call(*args, **kwargs) 2024-06-26T08:09:19.5217161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 179, in forward 2024-06-26T08:09:19.5218228Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T08:09:19.5218629Z 2024-06-26T08:09:19.6318436Z W0626 08:09:19.630000 140088559649408 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T08:10:55.7824425Z pass 2024-06-26T08:10:55.9788797Z TIMING: entire_frame_compile:132.15726 code_gen:41.12312 inductor_compile:82.25574 backend_compile:107.85609 2024-06-26T08:10:55.9790763Z STATS: call_* op count: 1923 | FakeTensorMode.__torch_dispatch__:213841 | FakeTensor.__torch_dispatch__:35697 | ProxyTorchDispatchMode.__torch_dispatch__:52645 2024-06-26T08:10:55.9791919Z Dynamo produced 2 graphs covering 1923 ops with 5 graph breaks (4 unique) 2024-06-26T08:11:06.6931306Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:11:06.6934084Z warnings.warn( 2024-06-26T08:11:06.9395548Z 2024-06-26T08:11:11.0683914Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:11:11.0684762Z loading model: 0it [00:04, ?it/s] 2024-06-26T08:11:11.0685245Z cuda train MegatronBertForQuestionAnswering 2024-06-26T08:12:18.8793868Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:12:18.8795075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:12:18.8796260Z pred = mod(**cloned_inputs) 2024-06-26T08:12:18.8797734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:12:18.8798693Z return forward_call(*args, **kwargs) 2024-06-26T08:12:18.8799984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1792, in forward 2024-06-26T08:12:18.8801196Z outputs = self.bert( 2024-06-26T08:12:18.8802424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:12:18.8803492Z return forward_call(*args, **kwargs) 2024-06-26T08:12:18.8805062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 967, in forward 2024-06-26T08:12:18.8806129Z embedding_output = self.embeddings( 2024-06-26T08:12:18.8807104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:12:18.8807984Z return forward_call(*args, **kwargs) 2024-06-26T08:12:18.8809155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 179, in forward 2024-06-26T08:12:18.8810202Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T08:12:18.8810552Z 2024-06-26T08:12:18.9820712Z W0626 08:12:18.981000 140020792754816 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T08:13:52.0921111Z pass 2024-06-26T08:13:52.2892673Z TIMING: entire_frame_compile:129.49652 code_gen:38.34616 inductor_compile:78.97715 backend_compile:105.3615 2024-06-26T08:13:52.2894614Z STATS: call_* op count: 1917 | FakeTensorMode.__torch_dispatch__:212755 | FakeTensor.__torch_dispatch__:35470 | ProxyTorchDispatchMode.__torch_dispatch__:52414 2024-06-26T08:13:52.2895858Z Dynamo produced 2 graphs covering 1917 ops with 5 graph breaks (4 unique) 2024-06-26T08:14:03.0063120Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:14:03.0064778Z warnings.warn( 2024-06-26T08:14:03.2038701Z 2024-06-26T08:14:04.5235063Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:04.5235657Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:14:04.5236114Z cuda train MobileBertForMaskedLM 2024-06-26T08:16:08.0167265Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:16:08.0168464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:16:08.0170151Z pred = mod(**cloned_inputs) 2024-06-26T08:16:08.0171984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:16:08.0173285Z return forward_call(*args, **kwargs) 2024-06-26T08:16:08.0175003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1089, in forward 2024-06-26T08:16:08.0176344Z outputs = self.mobilebert( 2024-06-26T08:16:08.0177541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:16:08.0178587Z return forward_call(*args, **kwargs) 2024-06-26T08:16:08.0180153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 895, in forward 2024-06-26T08:16:08.0181614Z embedding_output = self.embeddings( 2024-06-26T08:16:08.0183270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:16:08.0184574Z return forward_call(*args, **kwargs) 2024-06-26T08:16:08.0186060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 218, in forward 2024-06-26T08:16:08.0187549Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T08:16:08.0188026Z 2024-06-26T08:16:10.6503704Z pass 2024-06-26T08:16:10.6836711Z TIMING: entire_frame_compile:86.09762 code_gen:27.77009 inductor_compile:57.73925 backend_compile:62.33375 2024-06-26T08:16:10.6838032Z STATS: call_* op count: 1449 | FakeTensorMode.__torch_dispatch__:216936 | FakeTensor.__torch_dispatch__:27386 | ProxyTorchDispatchMode.__torch_dispatch__:74460 2024-06-26T08:16:10.6839574Z Dynamo produced 1 graphs covering 1449 ops with 3 graph breaks (3 unique) 2024-06-26T08:16:19.3808605Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:16:19.3810425Z warnings.warn( 2024-06-26T08:16:19.5951109Z 2024-06-26T08:16:20.7615933Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:16:20.7616559Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:16:20.7617143Z cuda train MobileBertForQuestionAnswering 2024-06-26T08:18:23.6257144Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:18:23.6259615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:18:23.6260665Z pred = mod(**cloned_inputs) 2024-06-26T08:18:23.6261666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:18:23.6262627Z return forward_call(*args, **kwargs) 2024-06-26T08:18:23.6263693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1390, in forward 2024-06-26T08:18:23.6271136Z outputs = self.mobilebert( 2024-06-26T08:18:23.6272263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:18:23.6273168Z return forward_call(*args, **kwargs) 2024-06-26T08:18:23.6274331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 895, in forward 2024-06-26T08:18:23.6275365Z embedding_output = self.embeddings( 2024-06-26T08:18:23.6276355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:18:23.6277302Z return forward_call(*args, **kwargs) 2024-06-26T08:18:23.6278441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 218, in forward 2024-06-26T08:18:23.6279503Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T08:18:23.6279914Z 2024-06-26T08:18:26.1253414Z pass 2024-06-26T08:18:26.1469886Z TIMING: entire_frame_compile:85.57736 code_gen:27.09509 inductor_compile:57.17174 backend_compile:61.76413 2024-06-26T08:18:26.1471170Z STATS: call_* op count: 1453 | FakeTensorMode.__torch_dispatch__:216724 | FakeTensor.__torch_dispatch__:27353 | ProxyTorchDispatchMode.__torch_dispatch__:74366 2024-06-26T08:18:26.1472302Z Dynamo produced 1 graphs covering 1453 ops with 3 graph breaks (3 unique) 2024-06-26T08:18:34.8033403Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:18:34.8036063Z warnings.warn( 2024-06-26T08:18:35.0384286Z 2024-06-26T08:18:37.3248837Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:18:37.3249327Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:18:37.3249771Z cuda train OPTForCausalLM 2024-06-26T08:18:37.3451887Z WARNING:common:fp64 golden ref were not generated for OPTForCausalLM. Setting accuracy check to cosine 2024-06-26T08:18:55.7541739Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:18:55.7543175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 109, in forward 2024-06-26T08:18:55.7544131Z return super().forward(positions + self.offset) 2024-06-26T08:18:55.7545192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T08:18:55.7545967Z return F.embedding( 2024-06-26T08:18:55.7546395Z 2024-06-26T08:19:03.0516789Z pass 2024-06-26T08:19:03.0752449Z TIMING: entire_frame_compile:13.36327 inductor_compile:5.30692 backend_compile:11.5476 code_gen:2.83034 2024-06-26T08:19:03.0753771Z STATS: call_* op count: 533 | FakeTensorMode.__torch_dispatch__:37247 | FakeTensor.__torch_dispatch__:2544 | ProxyTorchDispatchMode.__torch_dispatch__:18427 2024-06-26T08:19:03.0754904Z Dynamo produced 17 graphs covering 533 ops with 6 graph breaks (5 unique) 2024-06-26T08:19:06.9807060Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:19:06.9808790Z warnings.warn( 2024-06-26T08:19:07.1533923Z 2024-06-26T08:19:08.8969790Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:19:08.8970263Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:19:08.8970717Z cuda train PLBartForCausalLM 2024-06-26T08:19:08.9142926Z WARNING:common:fp64 golden ref were not generated for PLBartForCausalLM. Setting accuracy check to cosine 2024-06-26T08:19:21.6734041Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:19:21.6735378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 106, in forward 2024-06-26T08:19:21.6736340Z return super().forward(positions + self.offset) 2024-06-26T08:19:21.6737264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T08:19:21.6738036Z return F.embedding( 2024-06-26T08:19:21.6738264Z 2024-06-26T08:19:25.1545394Z pass 2024-06-26T08:19:25.1859563Z TIMING: entire_frame_compile:9.23359 inductor_compile:4.41325 backend_compile:7.89935 code_gen:2.05124 2024-06-26T08:19:25.1862071Z STATS: call_* op count: 250 | FakeTensorMode.__torch_dispatch__:20561 | FakeTensor.__torch_dispatch__:1633 | ProxyTorchDispatchMode.__torch_dispatch__:9427 2024-06-26T08:19:25.1864138Z Dynamo produced 11 graphs covering 250 ops with 6 graph breaks (5 unique) 2024-06-26T08:19:28.9419746Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:19:28.9421380Z warnings.warn( 2024-06-26T08:19:29.1670544Z 2024-06-26T08:19:32.2547683Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:19:32.2548322Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:19:32.2548922Z cuda train PLBartForConditionalGeneration 2024-06-26T08:19:32.3091612Z WARNING:common:fp64 golden ref were not generated for PLBartForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T08:19:55.0431974Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:19:55.0434140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 106, in forward 2024-06-26T08:19:55.0435226Z return super().forward(positions + self.offset) 2024-06-26T08:19:55.0436490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T08:19:55.0437414Z return F.embedding( 2024-06-26T08:19:55.0437649Z 2024-06-26T08:19:56.2322188Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:19:56.2323784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 106, in forward 2024-06-26T08:19:56.2325063Z return super().forward(positions + self.offset) 2024-06-26T08:19:56.2325999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T08:19:56.2326778Z return F.embedding( 2024-06-26T08:19:56.2326994Z 2024-06-26T08:20:05.3570109Z pass 2024-06-26T08:20:05.3932367Z TIMING: entire_frame_compile:18.02719 inductor_compile:7.5664 backend_compile:15.35462 code_gen:3.21065 2024-06-26T08:20:05.3935131Z STATS: call_* op count: 646 | FakeTensorMode.__torch_dispatch__:51723 | FakeTensor.__torch_dispatch__:3926 | ProxyTorchDispatchMode.__torch_dispatch__:24571 2024-06-26T08:20:05.3936671Z Dynamo produced 19 graphs covering 646 ops with 8 graph breaks (5 unique) 2024-06-26T08:20:09.5455116Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:20:09.5456830Z warnings.warn( 2024-06-26T08:20:09.7294563Z 2024-06-26T08:20:15.7520654Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:20:15.7521123Z loading model: 0it [00:06, ?it/s] 2024-06-26T08:20:15.7521577Z cuda train PegasusForCausalLM 2024-06-26T08:20:15.7838011Z WARNING:common:fp64 golden ref were not generated for PegasusForCausalLM. Setting accuracy check to cosine 2024-06-26T08:20:40.9129517Z pass 2024-06-26T08:20:40.9936509Z TIMING: entire_frame_compile:14.27063 inductor_compile:4.70523 backend_compile:11.98732 code_gen:1.67027 2024-06-26T08:20:40.9937946Z STATS: call_* op count: 476 | FakeTensorMode.__torch_dispatch__:36892 | FakeTensor.__torch_dispatch__:2548 | ProxyTorchDispatchMode.__torch_dispatch__:18176 2024-06-26T08:20:40.9939080Z Dynamo produced 17 graphs covering 476 ops with 6 graph breaks (5 unique) 2024-06-26T08:20:44.8918122Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:20:44.8919853Z warnings.warn( 2024-06-26T08:20:45.1123678Z 2024-06-26T08:20:55.3910311Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:20:55.3910802Z loading model: 0it [00:10, ?it/s] 2024-06-26T08:20:55.3911263Z cuda train PegasusForConditionalGeneration 2024-06-26T08:20:55.4685611Z WARNING:common:fp64 golden ref were not generated for PegasusForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T08:21:54.0048536Z pass 2024-06-26T08:21:54.0892923Z TIMING: entire_frame_compile:30.0714 inductor_compile:8.32959 backend_compile:25.81766 code_gen:2.75854 2024-06-26T08:21:54.0896248Z STATS: call_* op count: 1234 | FakeTensorMode.__torch_dispatch__:93171 | FakeTensor.__torch_dispatch__:6119 | ProxyTorchDispatchMode.__torch_dispatch__:47120 2024-06-26T08:21:54.0897433Z Dynamo produced 30 graphs covering 1234 ops with 7 graph breaks (4 unique) 2024-06-26T08:21:58.4261395Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:21:58.4263253Z warnings.warn( 2024-06-26T08:21:58.6086463Z 2024-06-26T08:21:58.6122241Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2024-06-26T08:22:00.5216830Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-06-26T08:22:00.5218750Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-06-26T08:22:00.7353694Z 2024-06-26T08:22:00.7354351Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:22:00.7354877Z cuda train RobertaForCausalLM 2024-06-26T08:22:35.6261537Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:22:35.6262981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:22:35.6264334Z pred = mod(**cloned_inputs) 2024-06-26T08:22:35.6265657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:22:35.6266697Z return forward_call(*args, **kwargs) 2024-06-26T08:22:35.6268058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 953, in forward 2024-06-26T08:22:35.6269098Z outputs = self.roberta( 2024-06-26T08:22:35.6269972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:22:35.6270811Z return forward_call(*args, **kwargs) 2024-06-26T08:22:35.6271835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 828, in forward 2024-06-26T08:22:35.6272767Z embedding_output = self.embeddings( 2024-06-26T08:22:35.6273691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:22:35.6274535Z return forward_call(*args, **kwargs) 2024-06-26T08:22:35.6275555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 125, in forward 2024-06-26T08:22:35.6276515Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T08:22:35.6279132Z 2024-06-26T08:22:35.8100984Z W0626 08:22:35.809000 139968275088000 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T08:23:25.5470908Z pass 2024-06-26T08:23:25.6003875Z TIMING: entire_frame_compile:68.48934 code_gen:22.75398 inductor_compile:43.1831 backend_compile:56.64165 2024-06-26T08:23:25.6005329Z STATS: call_* op count: 1008 | FakeTensorMode.__torch_dispatch__:110136 | FakeTensor.__torch_dispatch__:18383 | ProxyTorchDispatchMode.__torch_dispatch__:27039 2024-06-26T08:23:25.6007551Z Dynamo produced 2 graphs covering 1008 ops with 5 graph breaks (4 unique) 2024-06-26T08:23:32.7518936Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:23:32.7520602Z warnings.warn( 2024-06-26T08:23:32.9881149Z 2024-06-26T08:23:34.5136372Z loading model: 0it [00:00, ?it/s]We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-06-26T08:23:34.5138465Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-06-26T08:23:34.7548187Z 2024-06-26T08:23:34.7549334Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:23:34.7550118Z cuda train RobertaForQuestionAnswering 2024-06-26T08:24:08.5082406Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:24:08.5083632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:24:08.5084607Z pred = mod(**cloned_inputs) 2024-06-26T08:24:08.5085768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:24:08.5086608Z return forward_call(*args, **kwargs) 2024-06-26T08:24:08.5087616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1500, in forward 2024-06-26T08:24:08.5088791Z outputs = self.roberta( 2024-06-26T08:24:08.5089980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:24:08.5091662Z return forward_call(*args, **kwargs) 2024-06-26T08:24:08.5092830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 828, in forward 2024-06-26T08:24:08.5093769Z embedding_output = self.embeddings( 2024-06-26T08:24:08.5094684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:24:08.5095510Z return forward_call(*args, **kwargs) 2024-06-26T08:24:08.5096506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 125, in forward 2024-06-26T08:24:08.5097467Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T08:24:08.5097815Z 2024-06-26T08:24:08.5843762Z W0626 08:24:08.583000 140106450502272 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T08:24:54.4133435Z pass 2024-06-26T08:24:54.4759643Z TIMING: entire_frame_compile:64.30006 code_gen:19.08312 inductor_compile:39.73281 backend_compile:52.63682 2024-06-26T08:24:54.4763630Z STATS: call_* op count: 1001 | FakeTensorMode.__torch_dispatch__:109017 | FakeTensor.__torch_dispatch__:18152 | ProxyTorchDispatchMode.__torch_dispatch__:26799 2024-06-26T08:24:54.4764823Z Dynamo produced 2 graphs covering 1001 ops with 5 graph breaks (4 unique) 2024-06-26T08:25:01.6521888Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:25:01.6523637Z warnings.warn( 2024-06-26T08:25:01.8321913Z 2024-06-26T08:25:02.3478994Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:25:02.3479611Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:25:02.3480162Z cuda train Speech2Text2ForCausalLM 2024-06-26T08:25:02.3640275Z WARNING:common:fp64 golden ref were not generated for Speech2Text2ForCausalLM. Setting accuracy check to cosine 2024-06-26T08:25:16.9683017Z pass 2024-06-26T08:25:16.9730390Z TIMING: entire_frame_compile:8.41423 inductor_compile:4.28728 backend_compile:7.2559 code_gen:1.85116 2024-06-26T08:25:16.9732078Z STATS: call_* op count: 258 | FakeTensorMode.__torch_dispatch__:20223 | FakeTensor.__torch_dispatch__:1555 | ProxyTorchDispatchMode.__torch_dispatch__:9305 2024-06-26T08:25:16.9733355Z Dynamo produced 11 graphs covering 258 ops with 6 graph breaks (5 unique) 2024-06-26T08:25:20.7074403Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:25:20.7076060Z warnings.warn( 2024-06-26T08:25:20.9437225Z 2024-06-26T08:25:22.1932283Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:25:22.1932766Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:25:22.1933223Z cuda train T5ForConditionalGeneration 2024-06-26T08:26:01.9291602Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:26:01.9293336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:26:01.9294683Z pred = mod(**cloned_inputs) 2024-06-26T08:26:01.9299910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:26:01.9301100Z return forward_call(*args, **kwargs) 2024-06-26T08:26:01.9302434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1706, in forward 2024-06-26T08:26:01.9303701Z encoder_outputs = self.encoder( 2024-06-26T08:26:01.9304880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:26:01.9306287Z return forward_call(*args, **kwargs) 2024-06-26T08:26:01.9307656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1016, in forward 2024-06-26T08:26:01.9308844Z inputs_embeds = self.embed_tokens(input_ids) 2024-06-26T08:26:01.9309287Z 2024-06-26T08:26:02.0011959Z W0626 08:26:02.000000 140682824913536 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T08:26:37.9620386Z pass 2024-06-26T08:26:38.0210122Z TIMING: entire_frame_compile:58.51579 code_gen:19.06011 inductor_compile:37.24424 backend_compile:47.87564 2024-06-26T08:26:38.0211977Z STATS: call_* op count: 1224 | FakeTensorMode.__torch_dispatch__:107355 | FakeTensor.__torch_dispatch__:14731 | ProxyTorchDispatchMode.__torch_dispatch__:30407 2024-06-26T08:26:38.0215054Z Dynamo produced 2 graphs covering 1224 ops with 5 graph breaks (4 unique) 2024-06-26T08:26:44.3138613Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:26:44.3140282Z warnings.warn( 2024-06-26T08:26:44.5168267Z 2024-06-26T08:26:45.7818464Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:26:45.7818946Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:26:45.7819368Z cuda train T5Small 2024-06-26T08:27:09.0234620Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:27:09.0236010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:27:09.0237413Z pred = mod(**cloned_inputs) 2024-06-26T08:27:09.0238784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:27:09.0239741Z return forward_call(*args, **kwargs) 2024-06-26T08:27:09.0240921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1706, in forward 2024-06-26T08:27:09.0242029Z encoder_outputs = self.encoder( 2024-06-26T08:27:09.0243012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:27:09.0243887Z return forward_call(*args, **kwargs) 2024-06-26T08:27:09.0244908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1016, in forward 2024-06-26T08:27:09.0245901Z inputs_embeds = self.embed_tokens(input_ids) 2024-06-26T08:27:09.0246231Z 2024-06-26T08:27:09.0945514Z W0626 08:27:09.093000 139917855277696 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T08:27:26.6764869Z pass 2024-06-26T08:27:26.7353243Z TIMING: entire_frame_compile:33.79112 inductor_compile:3.85684 backend_compile:23.2738 2024-06-26T08:27:26.7354486Z STATS: call_* op count: 1224 | FakeTensorMode.__torch_dispatch__:71205 | FakeTensor.__torch_dispatch__:4747 | ProxyTorchDispatchMode.__torch_dispatch__:30407 2024-06-26T08:27:26.7355610Z Dynamo produced 2 graphs covering 1224 ops with 5 graph breaks (4 unique) 2024-06-26T08:27:30.9582723Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:27:30.9585415Z warnings.warn( 2024-06-26T08:27:31.1323208Z 2024-06-26T08:27:34.7287152Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:27:34.7287623Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:27:34.7288065Z cuda train TrOCRForCausalLM 2024-06-26T08:27:34.7596997Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine 2024-06-26T08:27:52.6863567Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:27:52.6865465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 66, in forward 2024-06-26T08:27:52.6866753Z return super().forward(positions + self.offset) 2024-06-26T08:27:52.6867721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T08:27:52.6868505Z return F.embedding( 2024-06-26T08:27:52.6868726Z 2024-06-26T08:27:59.9200755Z pass 2024-06-26T08:28:00.0023612Z TIMING: entire_frame_compile:13.98684 inductor_compile:4.94666 backend_compile:11.94664 code_gen:2.5339 2024-06-26T08:28:00.0025015Z STATS: call_* op count: 477 | FakeTensorMode.__torch_dispatch__:37042 | FakeTensor.__torch_dispatch__:2529 | ProxyTorchDispatchMode.__torch_dispatch__:18211 2024-06-26T08:28:00.0026131Z Dynamo produced 17 graphs covering 477 ops with 6 graph breaks (5 unique) 2024-06-26T08:28:03.8949316Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:28:03.8951628Z warnings.warn( 2024-06-26T08:28:04.0823223Z 2024-06-26T08:28:13.2583505Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:28:13.2584051Z loading model: 0it [00:09, ?it/s] 2024-06-26T08:28:13.2584498Z cuda train XGLMForCausalLM 2024-06-26T08:28:13.4028652Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine 2024-06-26T08:28:17.3169562Z skipping cudagraphs due to mutated inputs (1 instances). Found from : 2024-06-26T08:28:17.3171094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 174, in forward 2024-06-26T08:28:17.3172013Z position_ids += self.offset 2024-06-26T08:28:17.3172276Z 2024-06-26T08:29:00.6827248Z pass 2024-06-26T08:29:00.7651531Z TIMING: entire_frame_compile:24.17264 inductor_compile:6.2132 backend_compile:20.94682 code_gen:1.97676 2024-06-26T08:29:00.7654526Z STATS: call_* op count: 987 | FakeTensorMode.__torch_dispatch__:72969 | FakeTensor.__torch_dispatch__:4507 | ProxyTorchDispatchMode.__torch_dispatch__:37344 2024-06-26T08:29:00.7655724Z Dynamo produced 29 graphs covering 987 ops with 6 graph breaks (5 unique) 2024-06-26T08:29:04.8849060Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:29:04.8850710Z warnings.warn( 2024-06-26T08:29:05.0779323Z 2024-06-26T08:29:09.2396124Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:29:09.2396600Z loading model: 0it [00:04, ?it/s] 2024-06-26T08:29:09.2401249Z cuda train XLNetLMHeadModel 2024-06-26T08:30:04.6490554Z W0626 08:30:04.648000 139717843051136 torch/_inductor/utils.py:1234] [3/0_1] DeviceCopy in input program 2024-06-26T08:30:18.3113127Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat). Found from : 2024-06-26T08:30:18.3115393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:30:18.3116460Z pred = mod(**cloned_inputs) 2024-06-26T08:30:18.3117801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:30:18.3118828Z return forward_call(*args, **kwargs) 2024-06-26T08:30:18.3120117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1446, in forward 2024-06-26T08:30:18.3121416Z transformer_outputs = self.transformer( 2024-06-26T08:30:18.3122751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:30:18.3124000Z return forward_call(*args, **kwargs) 2024-06-26T08:30:18.3125051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1203, in forward 2024-06-26T08:30:18.3126062Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2024-06-26T08:30:18.3127497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1055, in relative_positional_encoding 2024-06-26T08:30:18.3128949Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2024-06-26T08:30:18.3130167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1014, in positional_embedding 2024-06-26T08:30:18.3131520Z pos_emb = pos_emb[:, None, :] 2024-06-26T08:30:18.3131812Z 2024-06-26T08:30:42.1387429Z W0626 08:30:42.137000 139717843051136 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T08:32:16.3659336Z pass 2024-06-26T08:32:16.5719309Z TIMING: entire_frame_compile:150.14036 code_gen:43.18387 inductor_compile:89.97788 backend_compile:123.55166 2024-06-26T08:32:16.5721737Z STATS: call_* op count: 1874 | FakeTensorMode.__torch_dispatch__:262068 | FakeTensor.__torch_dispatch__:35162 | ProxyTorchDispatchMode.__torch_dispatch__:66724 2024-06-26T08:32:16.5722875Z Dynamo produced 2 graphs covering 1874 ops with 5 graph breaks (4 unique) 2024-06-26T08:32:27.5209198Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:32:27.5211115Z warnings.warn( 2024-06-26T08:32:27.7223918Z 2024-06-26T08:32:29.7748167Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:29.7748799Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:32:29.7751212Z cuda train YituTechConvBert 2024-06-26T08:33:23.4585672Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:33:23.4587083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:33:23.4588336Z pred = mod(**cloned_inputs) 2024-06-26T08:33:23.4589428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:33:23.4590389Z return forward_call(*args, **kwargs) 2024-06-26T08:33:23.4591543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 918, in forward 2024-06-26T08:33:23.4592891Z generator_hidden_states = self.convbert( 2024-06-26T08:33:23.4593846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:33:23.4594711Z return forward_call(*args, **kwargs) 2024-06-26T08:33:23.4595735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 834, in forward 2024-06-26T08:33:23.4596664Z hidden_states = self.embeddings( 2024-06-26T08:33:23.4597616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:33:23.4598441Z return forward_call(*args, **kwargs) 2024-06-26T08:33:23.4599455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 230, in forward 2024-06-26T08:33:23.4600428Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T08:33:23.4600772Z 2024-06-26T08:33:23.5586269Z W0626 08:33:23.557000 140468974535296 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T08:34:37.4484785Z pass 2024-06-26T08:34:37.5010534Z TIMING: entire_frame_compile:104.0594 code_gen:36.33446 inductor_compile:66.64759 backend_compile:85.57868 2024-06-26T08:34:37.5012862Z STATS: call_* op count: 1512 | FakeTensorMode.__torch_dispatch__:161410 | FakeTensor.__torch_dispatch__:25959 | ProxyTorchDispatchMode.__torch_dispatch__:39878 2024-06-26T08:34:37.5014607Z Dynamo produced 2 graphs covering 1512 ops with 5 graph breaks (4 unique) 2024-06-26T08:34:43.8424876Z accuracy pass_rate=93.48% 2024-06-26T08:34:43.8428243Z calls_captured gmean=0.00x mean=920.348x 2024-06-26T08:34:43.8431572Z unique_graphs gmean=0.00x mean=7.478x 2024-06-26T08:34:43.8435106Z graph_breaks gmean=0.00x mean=5.196x 2024-06-26T08:34:43.8438491Z unique_graph_breaks gmean=0.00x mean=4.043x 2024-06-26T08:34:43.8441824Z autograd_captures gmean=0.00x mean=0.000x 2024-06-26T08:34:43.8445064Z autograd_compiles gmean=0.00x mean=0.000x 2024-06-26T08:34:43.8448445Z cudagraph_skips gmean=0.00x mean=0.978x 2024-06-26T08:34:44.6834207Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy//inductor_huggingface_training.csv 2024-06-26T08:34:44.9383090Z AlbertForMaskedLM PASS 2024-06-26T08:34:44.9387260Z AlbertForQuestionAnswering PASS 2024-06-26T08:34:44.9391808Z AllenaiLongformerBase PASS 2024-06-26T08:34:44.9396338Z BartForCausalLM PASS 2024-06-26T08:34:44.9400885Z BartForConditionalGeneration PASS 2024-06-26T08:34:44.9405446Z BertForMaskedLM PASS 2024-06-26T08:34:44.9409703Z BertForQuestionAnswering PASS 2024-06-26T08:34:44.9414473Z BlenderbotForCausalLM XFAIL 2024-06-26T08:34:44.9418705Z BlenderbotSmallForCausalLM PASS 2024-06-26T08:34:44.9423315Z BlenderbotSmallForConditionalGeneration PASS 2024-06-26T08:34:44.9427795Z CamemBert PASS 2024-06-26T08:34:44.9432140Z DebertaForMaskedLM PASS 2024-06-26T08:34:44.9436469Z DebertaForQuestionAnswering PASS 2024-06-26T08:34:44.9440822Z DebertaV2ForMaskedLM XFAIL 2024-06-26T08:34:44.9445157Z DebertaV2ForQuestionAnswering XFAIL 2024-06-26T08:34:44.9449514Z DistilBertForMaskedLM PASS 2024-06-26T08:34:44.9454167Z DistilBertForQuestionAnswering PASS 2024-06-26T08:34:44.9458433Z DistillGPT2 PASS 2024-06-26T08:34:44.9463047Z ElectraForCausalLM PASS 2024-06-26T08:34:44.9467555Z ElectraForQuestionAnswering PASS 2024-06-26T08:34:44.9471840Z GPT2ForSequenceClassification PASS 2024-06-26T08:34:44.9476322Z GoogleFnet PASS 2024-06-26T08:34:44.9480663Z LayoutLMForMaskedLM PASS 2024-06-26T08:34:44.9485178Z LayoutLMForSequenceClassification PASS 2024-06-26T08:34:44.9489469Z M2M100ForConditionalGeneration PASS 2024-06-26T08:34:44.9494080Z MBartForCausalLM PASS 2024-06-26T08:34:44.9498467Z MBartForConditionalGeneration PASS 2024-06-26T08:34:44.9502751Z MT5ForConditionalGeneration PASS 2024-06-26T08:34:44.9507378Z MegatronBertForCausalLM PASS 2024-06-26T08:34:44.9511505Z MegatronBertForQuestionAnswering PASS 2024-06-26T08:34:44.9516026Z MobileBertForMaskedLM PASS 2024-06-26T08:34:44.9520224Z MobileBertForQuestionAnswering PASS 2024-06-26T08:34:44.9524580Z OPTForCausalLM PASS 2024-06-26T08:34:44.9528967Z PLBartForCausalLM PASS 2024-06-26T08:34:44.9533697Z PLBartForConditionalGeneration PASS 2024-06-26T08:34:44.9538031Z PegasusForCausalLM PASS 2024-06-26T08:34:44.9542370Z PegasusForConditionalGeneration PASS 2024-06-26T08:34:44.9546995Z RobertaForCausalLM PASS 2024-06-26T08:34:44.9551223Z RobertaForQuestionAnswering PASS 2024-06-26T08:34:44.9555409Z Speech2Text2ForCausalLM PASS 2024-06-26T08:34:44.9559906Z T5ForConditionalGeneration PASS 2024-06-26T08:34:44.9564364Z T5Small PASS 2024-06-26T08:34:44.9568469Z TrOCRForCausalLM PASS 2024-06-26T08:34:44.9572969Z XGLMForCausalLM PASS 2024-06-26T08:34:44.9577475Z XLNetLMHeadModel PASS 2024-06-26T08:34:44.9581750Z YituTechConvBert PASS 2024-06-26T08:34:44.9954375Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy//inductor_huggingface_training.csv 2024-06-26T08:34:45.2358302Z AlbertForMaskedLM PASS 2024-06-26T08:34:45.2362683Z AlbertForQuestionAnswering PASS 2024-06-26T08:34:45.2367602Z AllenaiLongformerBase PASS 2024-06-26T08:34:45.2372892Z BartForCausalLM PASS 2024-06-26T08:34:45.2378191Z BartForConditionalGeneration PASS 2024-06-26T08:34:45.2383291Z BertForMaskedLM PASS 2024-06-26T08:34:45.2388345Z BertForQuestionAnswering PASS 2024-06-26T08:34:45.2393533Z BlenderbotForCausalLM PASS 2024-06-26T08:34:45.2398796Z BlenderbotSmallForCausalLM PASS 2024-06-26T08:34:45.2403850Z BlenderbotSmallForConditionalGeneration PASS 2024-06-26T08:34:45.2408683Z CamemBert PASS 2024-06-26T08:34:45.2413894Z DebertaForMaskedLM PASS 2024-06-26T08:34:45.2418998Z DebertaForQuestionAnswering PASS 2024-06-26T08:34:45.2424025Z DebertaV2ForMaskedLM PASS 2024-06-26T08:34:45.2429174Z DebertaV2ForQuestionAnswering PASS 2024-06-26T08:34:45.2434173Z DistilBertForMaskedLM PASS 2024-06-26T08:34:45.2439243Z DistilBertForQuestionAnswering PASS 2024-06-26T08:34:45.2444156Z DistillGPT2 PASS 2024-06-26T08:34:45.2449121Z ElectraForCausalLM PASS 2024-06-26T08:34:45.2454318Z ElectraForQuestionAnswering PASS 2024-06-26T08:34:45.2459395Z GPT2ForSequenceClassification PASS 2024-06-26T08:34:45.2464530Z GoogleFnet PASS 2024-06-26T08:34:45.2469491Z LayoutLMForMaskedLM PASS 2024-06-26T08:34:45.2474665Z LayoutLMForSequenceClassification PASS 2024-06-26T08:34:45.2479364Z M2M100ForConditionalGeneration PASS 2024-06-26T08:34:45.2484364Z MBartForCausalLM PASS 2024-06-26T08:34:45.2489300Z MBartForConditionalGeneration PASS 2024-06-26T08:34:45.2494646Z MT5ForConditionalGeneration PASS 2024-06-26T08:34:45.2499391Z MegatronBertForCausalLM PASS 2024-06-26T08:34:45.2504608Z MegatronBertForQuestionAnswering PASS 2024-06-26T08:34:45.2509548Z MobileBertForMaskedLM PASS 2024-06-26T08:34:45.2514575Z MobileBertForQuestionAnswering PASS 2024-06-26T08:34:45.2519497Z OPTForCausalLM PASS 2024-06-26T08:34:45.2524423Z PLBartForCausalLM PASS 2024-06-26T08:34:45.2529320Z PLBartForConditionalGeneration PASS 2024-06-26T08:34:45.2534865Z PegasusForCausalLM PASS 2024-06-26T08:34:45.2539639Z PegasusForConditionalGeneration PASS 2024-06-26T08:34:45.2544831Z RobertaForCausalLM PASS 2024-06-26T08:34:45.2549811Z RobertaForQuestionAnswering PASS 2024-06-26T08:34:45.2554856Z Speech2Text2ForCausalLM PASS 2024-06-26T08:34:45.2559800Z T5ForConditionalGeneration PASS 2024-06-26T08:34:45.2564594Z T5Small PASS 2024-06-26T08:34:45.2569576Z TrOCRForCausalLM PASS 2024-06-26T08:34:45.2574831Z XGLMForCausalLM PASS 2024-06-26T08:34:45.2579769Z XLNetLMHeadModel PASS 2024-06-26T08:34:45.2585096Z YituTechConvBert PASS 2024-06-26T08:34:45.2972342Z + cleanup_workspace 2024-06-26T08:34:45.2973281Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-26T08:34:45.2975283Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-26T08:34:45.2976546Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-26T08:34:45.2977264Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T08:34:45.2978122Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-26T08:34:45.2979060Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-26T08:34:45.2979811Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-26T08:34:45.9438016Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T08:34:45.9438523Z cat test/**/*_toprint.log || true 2024-06-26T08:34:45.9446033Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:34:45.9446525Z env: 2024-06-26T08:34:45.9446795Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:34:45.9447242Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:34:45.9447990Z DOCKER_CONTAINER_ID: 7ad6c3d35e4b88f659cb3bbc7d1ca4daf91f6fffa067a386e14c96b578954061 2024-06-26T08:34:45.9448632Z ##[endgroup] 2024-06-26T08:34:45.9512118Z cat: test/**/*_toprint.log: No such file or directory 2024-06-26T08:34:45.9540554Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:34:45.9541020Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:34:45.9548705Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:34:45.9549201Z env: 2024-06-26T08:34:45.9549477Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:34:45.9549925Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:34:45.9550662Z DOCKER_CONTAINER_ID: 7ad6c3d35e4b88f659cb3bbc7d1ca4daf91f6fffa067a386e14c96b578954061 2024-06-26T08:34:45.9551329Z MONITOR_SCRIPT_PID: 10115 2024-06-26T08:34:45.9551679Z ##[endgroup] 2024-06-26T08:34:45.9748142Z Prepare all required actions 2024-06-26T08:34:45.9748581Z Getting action download info 2024-06-26T08:34:46.1800477Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T08:34:46.3180941Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T08:34:46.3181408Z with: 2024-06-26T08:34:46.3181948Z file-suffix: test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911008 2024-06-26T08:34:46.3182732Z s3-bucket: gha-artifacts 2024-06-26T08:34:46.3183074Z env: 2024-06-26T08:34:46.3183337Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:34:46.3183783Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:34:46.3184528Z DOCKER_CONTAINER_ID: 7ad6c3d35e4b88f659cb3bbc7d1ca4daf91f6fffa067a386e14c96b578954061 2024-06-26T08:34:46.3185171Z ##[endgroup] 2024-06-26T08:34:46.3217015Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-26T08:34:46.3217670Z # Remove any previous test jsons if they exist 2024-06-26T08:34:46.3218181Z rm -f test-jsons-*.zip 2024-06-26T08:34:46.3218698Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-26T08:34:46.3226674Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:34:46.3227167Z env: 2024-06-26T08:34:46.3227439Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:34:46.3227874Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:34:46.3228609Z DOCKER_CONTAINER_ID: 7ad6c3d35e4b88f659cb3bbc7d1ca4daf91f6fffa067a386e14c96b578954061 2024-06-26T08:34:46.3229513Z FILE_SUFFIX: test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911008 2024-06-26T08:34:46.3230154Z ##[endgroup] 2024-06-26T08:34:46.3412175Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-26T08:34:46.3441807Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-26T08:34:46.3442807Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-26T08:34:46.3449199Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-26T08:34:46.3474595Z ##[group]Run # Remove any previous test reports if they exist 2024-06-26T08:34:46.3475369Z # Remove any previous test reports if they exist 2024-06-26T08:34:46.3475884Z rm -f test-reports-*.zip 2024-06-26T08:34:46.3476464Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-26T08:34:46.3483886Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:34:46.3484389Z env: 2024-06-26T08:34:46.3484664Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:34:46.3485114Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:34:46.3485850Z DOCKER_CONTAINER_ID: 7ad6c3d35e4b88f659cb3bbc7d1ca4daf91f6fffa067a386e14c96b578954061 2024-06-26T08:34:46.3486754Z FILE_SUFFIX: test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911008 2024-06-26T08:34:46.3487414Z ##[endgroup] 2024-06-26T08:34:46.3645441Z adding: test/test-reports/inference_huggingface.csv (deflated 76%) 2024-06-26T08:34:46.3646325Z adding: test/test-reports/inference_huggingface_graph_breaks.csv (deflated 89%) 2024-06-26T08:34:46.3647292Z adding: test/test-reports/inference_huggingface_graph_break_deduped.csv (deflated 78%) 2024-06-26T08:34:46.3648160Z adding: test/test-reports/training_huggingface.csv (deflated 73%) 2024-06-26T08:34:46.3654955Z adding: test/test-reports/training_huggingface_graph_breaks.csv (deflated 97%) 2024-06-26T08:34:46.3655903Z adding: test/test-reports/training_huggingface_graph_break_deduped.csv (deflated 77%) 2024-06-26T08:34:46.3681302Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-26T08:34:46.3681925Z # Remove any previous usage logs if they exist 2024-06-26T08:34:46.3682418Z rm -f logs-*.zip 2024-06-26T08:34:46.3683085Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-26T08:34:46.3683975Z # so check to see if the file exists first 2024-06-26T08:34:46.3684481Z if [ -f 'usage_log.txt' ]; then 2024-06-26T08:34:46.3685013Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-26T08:34:46.3685497Z fi 2024-06-26T08:34:46.3685855Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-26T08:34:46.3686417Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-26T08:34:46.3686900Z fi 2024-06-26T08:34:46.3694636Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:34:46.3695125Z env: 2024-06-26T08:34:46.3695390Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:34:46.3695838Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:34:46.3696569Z DOCKER_CONTAINER_ID: 7ad6c3d35e4b88f659cb3bbc7d1ca4daf91f6fffa067a386e14c96b578954061 2024-06-26T08:34:46.3697461Z FILE_SUFFIX: test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911008 2024-06-26T08:34:46.3698096Z ##[endgroup] 2024-06-26T08:34:46.3768415Z adding: usage_log.txt (deflated 92%) 2024-06-26T08:34:46.3846058Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-26T08:34:46.3846771Z # Remove any previous debugging artifacts if they exist 2024-06-26T08:34:46.3847320Z rm -f debug-*.zip 2024-06-26T08:34:46.3847697Z if [ -d 'test/debug' ]; then 2024-06-26T08:34:46.3848200Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-26T08:34:46.3848666Z fi 2024-06-26T08:34:46.3856015Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:34:46.3856506Z env: 2024-06-26T08:34:46.3856777Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:34:46.3857215Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:34:46.3857952Z DOCKER_CONTAINER_ID: 7ad6c3d35e4b88f659cb3bbc7d1ca4daf91f6fffa067a386e14c96b578954061 2024-06-26T08:34:46.3858850Z FILE_SUFFIX: test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911008 2024-06-26T08:34:46.3859494Z ##[endgroup] 2024-06-26T08:34:46.3933431Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:34:46.3933860Z with: 2024-06-26T08:34:46.3934142Z s3-bucket: gha-artifacts 2024-06-26T08:34:46.3934688Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:34:46.3935154Z retention-days: 14 2024-06-26T08:34:46.3935479Z if-no-files-found: warn 2024-06-26T08:34:46.3935844Z path: test-jsons-*.zip 2024-06-26T08:34:46.3936180Z name: artifact 2024-06-26T08:34:46.3936483Z region: us-east-1 2024-06-26T08:34:46.3936771Z env: 2024-06-26T08:34:46.3937043Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:34:46.3937486Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:34:46.3938218Z DOCKER_CONTAINER_ID: 7ad6c3d35e4b88f659cb3bbc7d1ca4daf91f6fffa067a386e14c96b578954061 2024-06-26T08:34:46.3938862Z ##[endgroup] 2024-06-26T08:34:46.6964086Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:34:46.6964692Z With the provided path, there will be 1 file uploaded 2024-06-26T08:34:46.6965328Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:34:46.6991526Z Starting upload of test-jsons-test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911008.zip 2024-06-26T08:34:46.8076561Z Finished upload of test-jsons-test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911008.zip 2024-06-26T08:34:46.8204492Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:34:46.8204926Z with: 2024-06-26T08:34:46.8205206Z s3-bucket: gha-artifacts 2024-06-26T08:34:46.8205631Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:34:46.8206093Z retention-days: 14 2024-06-26T08:34:46.8206420Z if-no-files-found: error 2024-06-26T08:34:46.8206781Z path: test-reports-*.zip 2024-06-26T08:34:46.8207124Z name: artifact 2024-06-26T08:34:46.8207417Z region: us-east-1 2024-06-26T08:34:46.8207715Z env: 2024-06-26T08:34:46.8207984Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:34:46.8208432Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:34:46.8209306Z DOCKER_CONTAINER_ID: 7ad6c3d35e4b88f659cb3bbc7d1ca4daf91f6fffa067a386e14c96b578954061 2024-06-26T08:34:46.8209958Z ##[endgroup] 2024-06-26T08:34:47.1275854Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:34:47.1276459Z With the provided path, there will be 1 file uploaded 2024-06-26T08:34:47.1277205Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:34:47.1303072Z Starting upload of test-reports-test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911008.zip 2024-06-26T08:34:47.2436202Z Finished upload of test-reports-test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911008.zip 2024-06-26T08:34:47.2565922Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:34:47.2566352Z with: 2024-06-26T08:34:47.2566630Z s3-bucket: gha-artifacts 2024-06-26T08:34:47.2567048Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:34:47.2567515Z retention-days: 14 2024-06-26T08:34:47.2567841Z if-no-files-found: ignore 2024-06-26T08:34:47.2568211Z path: logs-*.zip 2024-06-26T08:34:47.2568517Z name: artifact 2024-06-26T08:34:47.2568810Z region: us-east-1 2024-06-26T08:34:47.2569115Z env: 2024-06-26T08:34:47.2569388Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:34:47.2569834Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:34:47.2570565Z DOCKER_CONTAINER_ID: 7ad6c3d35e4b88f659cb3bbc7d1ca4daf91f6fffa067a386e14c96b578954061 2024-06-26T08:34:47.2571470Z ##[endgroup] 2024-06-26T08:34:47.5643248Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:34:47.5643981Z With the provided path, there will be 1 file uploaded 2024-06-26T08:34:47.5644601Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:34:47.5670259Z Starting upload of logs-test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911008.zip 2024-06-26T08:34:47.7031824Z Finished upload of logs-test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911008.zip 2024-06-26T08:34:47.7158877Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:34:47.7159317Z with: 2024-06-26T08:34:47.7159600Z s3-bucket: gha-artifacts 2024-06-26T08:34:47.7160028Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:34:47.7160622Z retention-days: 14 2024-06-26T08:34:47.7160950Z if-no-files-found: ignore 2024-06-26T08:34:47.7161310Z path: debug-*.zip 2024-06-26T08:34:47.7161620Z name: artifact 2024-06-26T08:34:47.7161912Z region: us-east-1 2024-06-26T08:34:47.7162210Z env: 2024-06-26T08:34:47.7162476Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:34:47.7162921Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:34:47.7163651Z DOCKER_CONTAINER_ID: 7ad6c3d35e4b88f659cb3bbc7d1ca4daf91f6fffa067a386e14c96b578954061 2024-06-26T08:34:47.7164304Z ##[endgroup] 2024-06-26T08:34:48.0109739Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-26T08:34:48.0236923Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T08:34:48.0237419Z # shellcheck disable=SC2156 2024-06-26T08:34:48.0238312Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T08:34:48.0246327Z shell: /usr/bin/bash -e {0} 2024-06-26T08:34:48.0246671Z env: 2024-06-26T08:34:48.0246942Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:34:48.0247383Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:34:48.0248112Z DOCKER_CONTAINER_ID: 7ad6c3d35e4b88f659cb3bbc7d1ca4daf91f6fffa067a386e14c96b578954061 2024-06-26T08:34:48.0248752Z ##[endgroup] 2024-06-26T08:34:48.1852051Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-26T08:34:48.1852609Z with: 2024-06-26T08:34:48.1852868Z env: 2024-06-26T08:34:48.1853144Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:34:48.1853586Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:34:48.1854328Z DOCKER_CONTAINER_ID: 7ad6c3d35e4b88f659cb3bbc7d1ca4daf91f6fffa067a386e14c96b578954061 2024-06-26T08:34:48.1854978Z ##[endgroup] 2024-06-26T08:34:48.1884563Z ##[group]Run set -eou pipefail 2024-06-26T08:34:48.1884961Z set -eou pipefail 2024-06-26T08:34:48.1885321Z  2024-06-26T08:34:48.1885827Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-26T08:34:48.1886466Z for _ in $(seq 1440); do 2024-06-26T08:34:48.1886923Z  # Break if no ssh session exists anymore 2024-06-26T08:34:48.1887416Z  if [ "$(who)" = "" ]; then 2024-06-26T08:34:48.1887822Z  break 2024-06-26T08:34:48.1888156Z  fi 2024-06-26T08:34:48.1888455Z  echo "." 2024-06-26T08:34:48.1888780Z  sleep 5 2024-06-26T08:34:48.1889096Z done 2024-06-26T08:34:48.1896792Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:34:48.1897292Z env: 2024-06-26T08:34:48.1897573Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:34:48.1898023Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:34:48.1898757Z DOCKER_CONTAINER_ID: 7ad6c3d35e4b88f659cb3bbc7d1ca4daf91f6fffa067a386e14c96b578954061 2024-06-26T08:34:48.1899401Z ##[endgroup] 2024-06-26T08:34:48.1918542Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-26T08:34:48.1957590Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:34:48.1958358Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:34:48.1958941Z # shellcheck disable=SC2046 2024-06-26T08:34:48.1959418Z docker stop $(docker ps -q) || true 2024-06-26T08:34:48.1959890Z # Prune all of the docker images 2024-06-26T08:34:48.1960338Z docker system prune -af 2024-06-26T08:34:48.1967911Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:34:48.1968406Z env: 2024-06-26T08:34:48.1968675Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:34:48.1969132Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:34:48.1969872Z DOCKER_CONTAINER_ID: 7ad6c3d35e4b88f659cb3bbc7d1ca4daf91f6fffa067a386e14c96b578954061 2024-06-26T08:34:48.1970512Z ##[endgroup] 2024-06-26T08:34:48.5002713Z 7ad6c3d35e4b 2024-06-26T08:34:48.8821964Z Deleted Containers: 2024-06-26T08:34:48.8822789Z 7ad6c3d35e4b88f659cb3bbc7d1ca4daf91f6fffa067a386e14c96b578954061 2024-06-26T08:34:48.8823313Z 2024-06-26T08:34:53.8983683Z Deleted Images: 2024-06-26T08:34:53.8985547Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T08:34:53.8988444Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T08:34:53.8990289Z deleted: sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd 2024-06-26T08:34:53.8991203Z deleted: sha256:754ead7012f0b7691885a9660f02e80be6cee8c5665486fb995c1fb06adb6e92 2024-06-26T08:34:53.8992018Z deleted: sha256:fbe4496d8830e11be86d8f222522e7956f32f85bfc06b05ea53961f34a58af74 2024-06-26T08:34:53.8992825Z deleted: sha256:1c83814aa1a1a7e0cc998e3479bdab8e8766b975560cbf00690acf39ed899a80 2024-06-26T08:34:53.8993656Z deleted: sha256:e9ea8e61942c7f8ae4aab97b8d102b01d6bb4b6cf2a605808cd34130a17d8a0f 2024-06-26T08:34:53.8994541Z deleted: sha256:acc5fc32588cd95e082a6ce35e48d0f303ae8f96e3ab65c4e375b85e1f3c4a92 2024-06-26T08:34:53.8995632Z deleted: sha256:a93e53400904d29e4185d3be942ea34f19572ea438962788ae4e9b3d9bf4c498 2024-06-26T08:34:53.8996617Z deleted: sha256:c97092b88b5e4c77c68bbc4b51afab64e38719af1ff58013ec49a696cdd40a6e 2024-06-26T08:34:53.8997432Z deleted: sha256:b132889c90d67ffa53655846603e3fcc2372c23f04dbdbc54690a39d304eb89b 2024-06-26T08:34:53.8998227Z deleted: sha256:915d54081ed607710624abe3f2490114c745858831f4fc13396bd3b4b9a121bc 2024-06-26T08:34:53.8999028Z deleted: sha256:bfa8d446c498c87f535aef102695564de1b22c755e6c63944ae2f5073a2b31ec 2024-06-26T08:34:53.8999884Z deleted: sha256:e71460fb363c29e70bbe357d680aaa7ae694f094ef2f2713609158ad90611744 2024-06-26T08:34:53.9000686Z deleted: sha256:84079bb1b4ee1d0119ae3d3e701de460a1324912297c675425c9a36d30fcdb0f 2024-06-26T08:34:53.9001500Z deleted: sha256:d86ac3ca148d4758cb03b493fdd49109d8f1a2c7c85c4b755493c3c4529553e5 2024-06-26T08:34:53.9002455Z deleted: sha256:532b3ab97c44a673a0d5f62ee46fa6a83248a84de8877754a612355be56ed0a3 2024-06-26T08:34:53.9003264Z deleted: sha256:617e2a35f25ae24500c88c0c9b8475f0687fbcbed3d200013b4c99c6d8941aff 2024-06-26T08:34:53.9004167Z deleted: sha256:b88120bcd5a26a660c09c5df13176e0ef098fb2c8cdb57d74ab0796a3d1194d6 2024-06-26T08:34:53.9005197Z deleted: sha256:c016b2e7764735243205fb76fa9a273bb3dd4bf05532288e3a613ce022131ac3 2024-06-26T08:34:53.9006019Z deleted: sha256:bcaca462c390deddd2995de2d245a7495e7d2e1e94f0edad30a573c83d0588fd 2024-06-26T08:34:53.9006838Z deleted: sha256:9e260eba003d5c25d7b410e5c4c3093845d9e46611896840901c689d7fbf8ee6 2024-06-26T08:34:53.9007631Z deleted: sha256:3e62b86f8b911a9b59c62d55f063e71683ae6a01c22418db5d27930712adaad6 2024-06-26T08:34:53.9008432Z deleted: sha256:905920ccb1d88449b8dde686c34a4b2a8b6f59e78ac44916b189ea3593b9fbbb 2024-06-26T08:34:53.9009258Z deleted: sha256:961201237e03acd5061406238dd7a3c1343ac888d7583212cd0ab5ae1b4f0540 2024-06-26T08:34:53.9010066Z deleted: sha256:70047a2aac62159fa28fb4dbd4567ec2ffbab68fe046967e09acb0eb3e5cb9cb 2024-06-26T08:34:53.9011309Z deleted: sha256:1ec680ad781436ae65fa987f5fa1ed032ecd5c82a6e35eea6707de501f62c0f4 2024-06-26T08:34:53.9012154Z deleted: sha256:ef859623a50e85abf6e6c1dbd6188aa765a66cc7b2bd618a1cbe4825b7b2bd91 2024-06-26T08:34:53.9012983Z deleted: sha256:404ade5cacff59773b93ab438bfa64cd3a171d07f6f168bdc4a5d4bfa0b43526 2024-06-26T08:34:53.9013811Z deleted: sha256:5b27de6ee3d9f2f18217bf18dec725f7b4954c0e48c85d35752634c22d62209b 2024-06-26T08:34:53.9014642Z deleted: sha256:498e39c575b3fa069acaac7e46b91b4e54ae3a05f2ada5c4347091f346f25d4c 2024-06-26T08:34:53.9015447Z deleted: sha256:1405919d2694d65669b1f4c842282021afe89208ce86570e96464903733da2d0 2024-06-26T08:34:53.9016245Z deleted: sha256:d5b41e5d70896c0472fc3e6da0516a17d89cd6635dc75010ecddc31ce9690fad 2024-06-26T08:34:53.9017199Z deleted: sha256:18e9379b62b9d0bb40293d28a652eafaee66a499f66f0776bf14e37a513101b7 2024-06-26T08:34:53.9018019Z deleted: sha256:a781f80ed00ff2926a699fd01a309bb0b4fb7d5655eb652c205bb9c7f811035f 2024-06-26T08:34:53.9018824Z deleted: sha256:c0430973b7cf759075330ee29c79bfd1312c91e6033d9dd883b4b1c29743a171 2024-06-26T08:34:53.9019638Z deleted: sha256:e733b3d66efb9342e5a283376c0e1ddffa47a42fecc26e61ecaf0a3472885d97 2024-06-26T08:34:53.9020449Z deleted: sha256:8929c0b2047f74b84c82c39eff9077a92ae1a828cde00c9ab78fb3e0ed5e0283 2024-06-26T08:34:53.9021254Z deleted: sha256:f5b56155fb58a0a60fe6391d3581cf91610ed69d4f0a321257c19944da296a52 2024-06-26T08:34:53.9022058Z deleted: sha256:b54d2cb355238d1aa91e58df93064447b80f53706d1ca62823acc834f5ac54fe 2024-06-26T08:34:53.9023001Z deleted: sha256:e7311285af9eeddb51aa5629aa98386e89f1456ff32843b6aee6443247edf66c 2024-06-26T08:34:53.9023812Z deleted: sha256:67657ceec347da8dc8085d5731633efeac115ba677fa4b7c5966c4260f7abab0 2024-06-26T08:34:53.9024633Z deleted: sha256:b496e50409dbf414930fc9b4437aa0aa4ce27bc7840e050ae459aaffcf9aca86 2024-06-26T08:34:53.9025451Z deleted: sha256:8f5276170cc4a60d36ad7b964a502306b29546254cedcbcb101ea9123497d2c0 2024-06-26T08:34:53.9026253Z deleted: sha256:53d524d1564745e96a7e515462171b73184d9f1ff5454b32fcfbbca39d361467 2024-06-26T08:34:53.9027075Z deleted: sha256:2acbadb00d69b6aca575f04baba4ea26ae2b5c2d85f09945004ad93a78a8eb75 2024-06-26T08:34:53.9028002Z deleted: sha256:02665efeff878aeb603c1edb2e21cb700c171c025d64f5aae1b2b20b12f3964a 2024-06-26T08:34:53.9028838Z deleted: sha256:19bd23e9a51dfe883a8b8d2ff6d66ff1b3476b8912856406bb79b81762aa6d4b 2024-06-26T08:34:53.9029659Z deleted: sha256:73aa8637bf9299d6298dabd858480c274a62117388f2ca1d5e9eb15c6d1e4155 2024-06-26T08:34:53.9030467Z deleted: sha256:5f8968406305f348b1caebc0e035187d6e9965ace7b9d5f2f21114d37c9fe2e1 2024-06-26T08:34:53.9031275Z deleted: sha256:d827104914db0c503cf5c81f13e2456bf4efce42d4cf1f8d5d70838207278bee 2024-06-26T08:34:53.9032088Z deleted: sha256:867b219724f2f1aaf56f42e3e9f03a5d64bdca8a6a72407efd10328aeea5d510 2024-06-26T08:34:53.9032922Z deleted: sha256:b9ff2a7cceb9237b26bcb17728edf8a558eec5a76f63626e84f80541dfc0f1d4 2024-06-26T08:34:53.9033742Z deleted: sha256:be73f509106819007ce2b0f5a1301ac56ae583cb6c1fb69bee93efa0840368f1 2024-06-26T08:34:53.9034602Z deleted: sha256:2a11c07c915c3fdbb4e73096dea73ea950c970d50fe1a5764060bd62b7b7cf2c 2024-06-26T08:34:53.9035426Z deleted: sha256:5e9989622e2abc1c72f8184343507b45d75e4ff971edb68b96cbcd75ae484ff5 2024-06-26T08:34:53.9036250Z deleted: sha256:05ce642bd9cf5a9205bff107df28b1d6cef7d5f6ab3ea944d28c032ec61270df 2024-06-26T08:34:53.9037059Z deleted: sha256:017702617f6c07b73f2088678f4710b794cfe897a213a5cf122a0e2519eae0ac 2024-06-26T08:34:53.9037851Z deleted: sha256:44b3b37bb1ca39dc5cf356de41a23a5c12f605a1992813e408087fb2b0ad6d1c 2024-06-26T08:34:53.9038663Z deleted: sha256:8c387982e9f27db357a1d149dc149cabc0bd04668ad6956387a5072682dada67 2024-06-26T08:34:53.9039474Z deleted: sha256:10d415dca4e3fbb88a747b40317f3c6333c041ba8f73855f67fb9fe981fae5ee 2024-06-26T08:34:53.9040293Z deleted: sha256:6c65469bf0f53dcd625faefe363b8f3136c08388a6cf64d9158e8576e4ee89b1 2024-06-26T08:34:53.9041110Z deleted: sha256:027db5d8b76deeefbd8c34b485ab7ff14ed2a3e6800425c1ce0d77c2e9fdd4ab 2024-06-26T08:34:53.9041931Z deleted: sha256:24941234ad28cf5c8174d7cf18326c10b4e17f2c2137672ea0322df461597dd7 2024-06-26T08:34:53.9042755Z deleted: sha256:240ea2eebfa8519d6c2d88f4c99e66ce4e038d000f93ce8d03726c01d81180ef 2024-06-26T08:34:53.9043582Z deleted: sha256:09e823cb391a69f6a195e3ca8a41624b5f8b34eee14c6eb36277de814a91966e 2024-06-26T08:34:53.9044391Z deleted: sha256:730479d695169a6593bf0ac16b377491dce65175a6d158e012422367d04577fd 2024-06-26T08:34:53.9045190Z deleted: sha256:b0a30f9f7bd5459ff662453b9346a7ba8ea8db1191528345c1953eab4aa2f54e 2024-06-26T08:34:53.9046011Z deleted: sha256:75cefa87effaa3ad361c74642839923101f0f9982f814f3c9e2ed4ad52f2a008 2024-06-26T08:34:53.9046820Z deleted: sha256:6f97db87ba9512e615e871c6982000602982e1e5a5be334a56e7292251e84e00 2024-06-26T08:34:53.9047634Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-06-26T08:34:53.9048506Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-06-26T08:34:53.9049307Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-06-26T08:34:53.9050119Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-06-26T08:34:53.9052219Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-06-26T08:34:53.9053039Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-06-26T08:34:53.9053862Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-06-26T08:34:53.9054689Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-06-26T08:34:53.9055518Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-06-26T08:34:53.9056337Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-06-26T08:34:53.9057159Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-06-26T08:34:53.9057648Z 2024-06-26T08:34:53.9057794Z Total reclaimed space: 31.35GB 2024-06-26T08:34:53.9119706Z Post job cleanup. 2024-06-26T08:34:53.9168105Z Post job cleanup. 2024-06-26T08:34:53.9975338Z [command]/usr/bin/git version 2024-06-26T08:34:54.0016642Z git version 2.40.1 2024-06-26T08:34:54.0055792Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/81d8f656-2703-4dbd-8f5d-287b0d211ad9' before making global git config changes 2024-06-26T08:34:54.0057013Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T08:34:54.0061556Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T08:34:54.0088439Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T08:34:54.0111334Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T08:34:54.0322896Z Entering 'android/libs/fbjni' 2024-06-26T08:34:54.0359589Z Entering 'third_party/FP16' 2024-06-26T08:34:54.0398928Z Entering 'third_party/FXdiv' 2024-06-26T08:34:54.0436445Z Entering 'third_party/NNPACK' 2024-06-26T08:34:54.0475562Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:34:54.0511654Z Entering 'third_party/XNNPACK' 2024-06-26T08:34:54.0559999Z Entering 'third_party/benchmark' 2024-06-26T08:34:54.0598834Z Entering 'third_party/cpp-httplib' 2024-06-26T08:34:54.0634984Z Entering 'third_party/cpuinfo' 2024-06-26T08:34:54.0672508Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:34:54.0710199Z Entering 'third_party/cutlass' 2024-06-26T08:34:54.0755870Z Entering 'third_party/eigen' 2024-06-26T08:34:54.0795959Z Entering 'third_party/fbgemm' 2024-06-26T08:34:54.0833657Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:34:54.0868417Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:34:54.0905050Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:34:54.0944219Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:34:54.0979864Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:34:54.1018357Z Entering 'third_party/flatbuffers' 2024-06-26T08:34:54.1057101Z Entering 'third_party/fmt' 2024-06-26T08:34:54.1093606Z Entering 'third_party/foxi' 2024-06-26T08:34:54.1127476Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:34:54.1162245Z Entering 'third_party/gloo' 2024-06-26T08:34:54.1199399Z Entering 'third_party/googletest' 2024-06-26T08:34:54.1237390Z Entering 'third_party/ideep' 2024-06-26T08:34:54.1273660Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:34:54.1312925Z Entering 'third_party/ittapi' 2024-06-26T08:34:54.1350236Z Entering 'third_party/kineto' 2024-06-26T08:34:54.1383968Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:34:54.1418211Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:34:54.1453840Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:34:54.1487512Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:34:54.1521665Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:34:54.1553882Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:34:54.1589864Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:34:54.1625779Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:34:54.1658462Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:34:54.1692499Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:34:54.1724948Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:34:54.1760158Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:34:54.1796139Z Entering 'third_party/mimalloc' 2024-06-26T08:34:54.1833383Z Entering 'third_party/nccl/nccl' 2024-06-26T08:34:54.1867198Z Entering 'third_party/nlohmann' 2024-06-26T08:34:54.1902126Z Entering 'third_party/onnx' 2024-06-26T08:34:54.1949397Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:34:54.1984337Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:34:54.2021239Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:34:54.2058089Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:34:54.2094340Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:34:54.2129388Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:34:54.2163011Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:34:54.2197909Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:34:54.2233438Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:34:54.2270013Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:34:54.2303419Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:34:54.2338266Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:34:54.2373241Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:34:54.2423632Z Entering 'third_party/pocketfft' 2024-06-26T08:34:54.2457816Z Entering 'third_party/protobuf' 2024-06-26T08:34:54.2495330Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:34:54.2529703Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:34:54.2563145Z Entering 'third_party/psimd' 2024-06-26T08:34:54.2599708Z Entering 'third_party/pthreadpool' 2024-06-26T08:34:54.2634774Z Entering 'third_party/pybind11' 2024-06-26T08:34:54.2670481Z Entering 'third_party/python-peachpy' 2024-06-26T08:34:54.2705442Z Entering 'third_party/sleef' 2024-06-26T08:34:54.2739521Z Entering 'third_party/tensorpipe' 2024-06-26T08:34:54.2775456Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:34:54.2809492Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:34:54.2844103Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:34:54.2878174Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:34:54.2913205Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:34:54.2963192Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T08:34:54.2982797Z http.https://github.com/.extraheader 2024-06-26T08:34:54.2990584Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T08:34:54.3019683Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T08:34:54.3221862Z Entering 'android/libs/fbjni' 2024-06-26T08:34:54.3244651Z http.https://github.com/.extraheader 2024-06-26T08:34:54.3270209Z Entering 'third_party/FP16' 2024-06-26T08:34:54.3291280Z http.https://github.com/.extraheader 2024-06-26T08:34:54.3315166Z Entering 'third_party/FXdiv' 2024-06-26T08:34:54.3337816Z http.https://github.com/.extraheader 2024-06-26T08:34:54.3362031Z Entering 'third_party/NNPACK' 2024-06-26T08:34:54.3385428Z http.https://github.com/.extraheader 2024-06-26T08:34:54.3408240Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:34:54.3430862Z http.https://github.com/.extraheader 2024-06-26T08:34:54.3455216Z Entering 'third_party/XNNPACK' 2024-06-26T08:34:54.3476568Z http.https://github.com/.extraheader 2024-06-26T08:34:54.3509303Z Entering 'third_party/benchmark' 2024-06-26T08:34:54.3531361Z http.https://github.com/.extraheader 2024-06-26T08:34:54.3553076Z Entering 'third_party/cpp-httplib' 2024-06-26T08:34:54.3575432Z http.https://github.com/.extraheader 2024-06-26T08:34:54.3598546Z Entering 'third_party/cpuinfo' 2024-06-26T08:34:54.3620823Z http.https://github.com/.extraheader 2024-06-26T08:34:54.3644678Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:34:54.3666990Z http.https://github.com/.extraheader 2024-06-26T08:34:54.3691293Z Entering 'third_party/cutlass' 2024-06-26T08:34:54.3714269Z http.https://github.com/.extraheader 2024-06-26T08:34:54.3742031Z Entering 'third_party/eigen' 2024-06-26T08:34:54.3763142Z http.https://github.com/.extraheader 2024-06-26T08:34:54.3788920Z Entering 'third_party/fbgemm' 2024-06-26T08:34:54.3809354Z http.https://github.com/.extraheader 2024-06-26T08:34:54.3832879Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:34:54.3854249Z http.https://github.com/.extraheader 2024-06-26T08:34:54.3877524Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:34:54.3900495Z http.https://github.com/.extraheader 2024-06-26T08:34:54.3923322Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:34:54.3945481Z http.https://github.com/.extraheader 2024-06-26T08:34:54.3972968Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:34:54.3994202Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4016983Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:34:54.4038548Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4060824Z Entering 'third_party/flatbuffers' 2024-06-26T08:34:54.4082842Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4107487Z Entering 'third_party/fmt' 2024-06-26T08:34:54.4127271Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4152508Z Entering 'third_party/foxi' 2024-06-26T08:34:54.4174164Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4197582Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:34:54.4220347Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4244554Z Entering 'third_party/gloo' 2024-06-26T08:34:54.4267880Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4292289Z Entering 'third_party/googletest' 2024-06-26T08:34:54.4314263Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4338336Z Entering 'third_party/ideep' 2024-06-26T08:34:54.4360178Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4383953Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:34:54.4405241Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4435381Z Entering 'third_party/ittapi' 2024-06-26T08:34:54.4456865Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4482102Z Entering 'third_party/kineto' 2024-06-26T08:34:54.4504245Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4527621Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:34:54.4549531Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4573158Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:34:54.4594415Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4619540Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:34:54.4640689Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4665658Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:34:54.4686565Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4709776Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:34:54.4731028Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4753643Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:34:54.4775427Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4800701Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:34:54.4822257Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4845028Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:34:54.4866727Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4891280Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:34:54.4912275Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4937220Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:34:54.4957805Z http.https://github.com/.extraheader 2024-06-26T08:34:54.4981184Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:34:54.5002589Z http.https://github.com/.extraheader 2024-06-26T08:34:54.5024901Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:34:54.5045710Z http.https://github.com/.extraheader 2024-06-26T08:34:54.5070512Z Entering 'third_party/mimalloc' 2024-06-26T08:34:54.5092658Z http.https://github.com/.extraheader 2024-06-26T08:34:54.5116883Z Entering 'third_party/nccl/nccl' 2024-06-26T08:34:54.5137805Z http.https://github.com/.extraheader 2024-06-26T08:34:54.5161641Z Entering 'third_party/nlohmann' 2024-06-26T08:34:54.5184550Z http.https://github.com/.extraheader 2024-06-26T08:34:54.5209323Z Entering 'third_party/onnx' 2024-06-26T08:34:54.5231447Z http.https://github.com/.extraheader 2024-06-26T08:34:54.5268891Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:34:54.5290196Z http.https://github.com/.extraheader 2024-06-26T08:34:54.5314930Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:34:54.5336072Z http.https://github.com/.extraheader 2024-06-26T08:34:54.5361042Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:34:54.5384354Z http.https://github.com/.extraheader 2024-06-26T08:34:54.5408547Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:34:54.5431609Z http.https://github.com/.extraheader 2024-06-26T08:34:54.5456610Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:34:54.5478711Z http.https://github.com/.extraheader 2024-06-26T08:34:54.5503686Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:34:54.5524028Z http.https://github.com/.extraheader 2024-06-26T08:34:54.5546240Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:34:54.5566789Z http.https://github.com/.extraheader 2024-06-26T08:34:54.5592934Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:34:54.5614024Z http.https://github.com/.extraheader 2024-06-26T08:34:54.5637385Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:34:54.5658138Z http.https://github.com/.extraheader 2024-06-26T08:34:54.5681009Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:34:54.5702779Z http.https://github.com/.extraheader 2024-06-26T08:34:54.5725430Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:34:54.5746437Z http.https://github.com/.extraheader 2024-06-26T08:34:54.5770621Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:34:54.5792880Z http.https://github.com/.extraheader 2024-06-26T08:34:54.5817512Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:34:54.5838891Z http.https://github.com/.extraheader 2024-06-26T08:34:54.5880447Z Entering 'third_party/pocketfft' 2024-06-26T08:34:54.5903919Z http.https://github.com/.extraheader 2024-06-26T08:34:54.5928351Z Entering 'third_party/protobuf' 2024-06-26T08:34:54.5950870Z http.https://github.com/.extraheader 2024-06-26T08:34:54.5977275Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:34:54.5998528Z http.https://github.com/.extraheader 2024-06-26T08:34:54.6021880Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:34:54.6043755Z http.https://github.com/.extraheader 2024-06-26T08:34:54.6069590Z Entering 'third_party/psimd' 2024-06-26T08:34:54.6090482Z http.https://github.com/.extraheader 2024-06-26T08:34:54.6115640Z Entering 'third_party/pthreadpool' 2024-06-26T08:34:54.6137811Z http.https://github.com/.extraheader 2024-06-26T08:34:54.6161910Z Entering 'third_party/pybind11' 2024-06-26T08:34:54.6183372Z http.https://github.com/.extraheader 2024-06-26T08:34:54.6207196Z Entering 'third_party/python-peachpy' 2024-06-26T08:34:54.6228104Z http.https://github.com/.extraheader 2024-06-26T08:34:54.6249601Z Entering 'third_party/sleef' 2024-06-26T08:34:54.6273228Z http.https://github.com/.extraheader 2024-06-26T08:34:54.6298415Z Entering 'third_party/tensorpipe' 2024-06-26T08:34:54.6318993Z http.https://github.com/.extraheader 2024-06-26T08:34:54.6339833Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:34:54.6360638Z http.https://github.com/.extraheader 2024-06-26T08:34:54.6385210Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:34:54.6406033Z http.https://github.com/.extraheader 2024-06-26T08:34:54.6429891Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:34:54.6450778Z http.https://github.com/.extraheader 2024-06-26T08:34:54.6475642Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:34:54.6496004Z http.https://github.com/.extraheader 2024-06-26T08:34:54.6518285Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:34:54.6539658Z http.https://github.com/.extraheader 2024-06-26T08:34:54.6639153Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-26T08:34:54.6670940Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T08:34:54.6677683Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:34:54.6678186Z ##[endgroup] 2024-06-26T08:34:55.5565026Z Cleaning up orphan processes