2024-06-26T07:02:13.1175326Z Current runner version: '2.317.0' 2024-06-26T07:02:13.1181552Z Runner name: 'i-09875a2d32470f475' 2024-06-26T07:02:13.1182405Z Runner group name: 'Default' 2024-06-26T07:02:13.1183364Z Machine name: 'ip-10-0-30-116' 2024-06-26T07:02:13.1188516Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T07:02:13.1190457Z Actions: read 2024-06-26T07:02:13.1191060Z Attestations: read 2024-06-26T07:02:13.1191600Z Checks: read 2024-06-26T07:02:13.1192133Z Contents: read 2024-06-26T07:02:13.1192691Z Deployments: read 2024-06-26T07:02:13.1193238Z Discussions: read 2024-06-26T07:02:13.1193786Z Issues: read 2024-06-26T07:02:13.1194295Z Metadata: read 2024-06-26T07:02:13.1194862Z Packages: read 2024-06-26T07:02:13.1195389Z Pages: read 2024-06-26T07:02:13.1195909Z PullRequests: read 2024-06-26T07:02:13.1196497Z RepositoryProjects: read 2024-06-26T07:02:13.1197131Z SecurityEvents: read 2024-06-26T07:02:13.1197716Z Statuses: read 2024-06-26T07:02:13.1198267Z ##[endgroup] 2024-06-26T07:02:13.1201492Z Secret source: Actions 2024-06-26T07:02:13.1202265Z Prepare workflow directory 2024-06-26T07:02:13.4741926Z Prepare all required actions 2024-06-26T07:02:13.4929636Z Getting action download info 2024-06-26T07:02:13.7093513Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T07:02:14.0734503Z Download action repository 'pytorch/pytorch@main' (SHA:c422a9549d2b320b47a9391f6cbcf592b2876c88) 2024-06-26T07:02:16.8173993Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-26T07:02:16.9560728Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T07:02:17.2246891Z Getting action download info 2024-06-26T07:02:17.3217082Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T07:02:17.5086923Z Getting action download info 2024-06-26T07:02:17.6055152Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T07:02:17.7530621Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/129470 (b8c4c54d347aa776934c60784e35936878ef18dc) 2024-06-26T07:02:17.7532769Z ##[group] Inputs 2024-06-26T07:02:17.7533253Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:02:17.7540195Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:02:17.7548047Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:17.7549214Z sync-tag: 2024-06-26T07:02:17.7550069Z timeout-minutes: 240 2024-06-26T07:02:17.7550415Z use-gha: 2024-06-26T07:02:17.7550717Z dashboard-tag: 2024-06-26T07:02:17.7551055Z s3-bucket: gha-artifacts 2024-06-26T07:02:17.7551415Z aws-role-to-assume: 2024-06-26T07:02:17.7551761Z ##[endgroup] 2024-06-26T07:02:17.7552850Z Complete job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:02:17.8138928Z A job started hook has been configured by the self-hosted runner administrator 2024-06-26T07:02:17.8284190Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T07:02:17.8294747Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:17.8295308Z ##[endgroup] 2024-06-26T07:02:18.7867162Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-26T07:02:18.7867737Z with: 2024-06-26T07:02:18.7868453Z github-secret: *** 2024-06-26T07:02:18.7869367Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-26T07:02:18.7870352Z activate-with-label: false 2024-06-26T07:02:18.7870723Z label: with-ssh 2024-06-26T07:02:18.7871055Z remove-existing-keys: true 2024-06-26T07:02:18.7871443Z fail-silently: true 2024-06-26T07:02:18.7871763Z env: 2024-06-26T07:02:18.7872036Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:18.7872382Z ##[endgroup] 2024-06-26T07:02:18.8741512Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-26T07:02:18.8742527Z ciflow reference detected, attempting to extract PR number 2024-06-26T07:02:19.1301997Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-06-26T07:02:19.2031292Z No SSH keys found for user pytorch-bot[bot] 2024-06-26T07:02:19.2032033Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T07:02:19.2766044Z No SSH keys found for user leslie-fang-intel 2024-06-26T07:02:19.2876502Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T07:02:19.2877193Z with: 2024-06-26T07:02:19.2877635Z submodules: recursive 2024-06-26T07:02:19.2878095Z fetch-depth: 0 2024-06-26T07:02:19.2878524Z env: 2024-06-26T07:02:19.2878941Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:19.2879403Z ##[endgroup] 2024-06-26T07:02:19.3106026Z ##[group]Run retry () { 2024-06-26T07:02:19.3106512Z retry () { 2024-06-26T07:02:19.3107278Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T07:02:19.3107926Z } 2024-06-26T07:02:19.3108334Z echo "${GITHUB_WORKSPACE}" 2024-06-26T07:02:19.3108842Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T07:02:19.3109427Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:02:19.3109977Z else 2024-06-26T07:02:19.3110417Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:02:19.3110940Z fi 2024-06-26T07:02:19.3111382Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T07:02:19.3119517Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:19.3120100Z env: 2024-06-26T07:02:19.3120466Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:19.3120897Z NO_SUDO: 2024-06-26T07:02:19.3121247Z ##[endgroup] 2024-06-26T07:02:19.3143659Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:02:22.3208400Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T07:02:22.3208831Z with: 2024-06-26T07:02:22.3209155Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:02:22.3209599Z fetch-depth: 0 2024-06-26T07:02:22.3209911Z submodules: recursive 2024-06-26T07:02:22.3210245Z quiet-checkout: true 2024-06-26T07:02:22.3210732Z repository: pytorch/pytorch 2024-06-26T07:02:22.3211217Z token: *** 2024-06-26T07:02:22.3211509Z ssh-strict: true 2024-06-26T07:02:22.3211830Z persist-credentials: true 2024-06-26T07:02:22.3212183Z clean: true 2024-06-26T07:02:22.3212513Z sparse-checkout-cone-mode: true 2024-06-26T07:02:22.3212893Z lfs: false 2024-06-26T07:02:22.3213193Z set-safe-directory: true 2024-06-26T07:02:22.3213528Z env: 2024-06-26T07:02:22.3213799Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:22.3214130Z ##[endgroup] 2024-06-26T07:02:22.4139975Z Syncing repository: pytorch/pytorch 2024-06-26T07:02:22.4141543Z ##[group]Getting Git version info 2024-06-26T07:02:22.4142295Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:02:22.4143165Z [command]/usr/bin/git version 2024-06-26T07:02:22.4143545Z git version 2.40.1 2024-06-26T07:02:22.4144974Z ##[endgroup] 2024-06-26T07:02:22.4156488Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/e8748a68-fe19-4115-a0d8-c1ed3a230d1c/.gitconfig' 2024-06-26T07:02:22.4160823Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/e8748a68-fe19-4115-a0d8-c1ed3a230d1c' before making global git config changes 2024-06-26T07:02:22.4162124Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T07:02:22.4165860Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:02:22.4201107Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:02:22.4205123Z ##[group]Initializing the repository 2024-06-26T07:02:22.4208316Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:02:22.4232890Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T07:02:22.4233899Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T07:02:22.4234721Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T07:02:22.4235465Z hint: 2024-06-26T07:02:22.4235926Z hint: git config --global init.defaultBranch 2024-06-26T07:02:22.4236407Z hint: 2024-06-26T07:02:22.4237112Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T07:02:22.4237991Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T07:02:22.4238591Z hint: 2024-06-26T07:02:22.4238906Z hint: git branch -m 2024-06-26T07:02:22.4239654Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T07:02:22.4242300Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T07:02:22.4267010Z ##[endgroup] 2024-06-26T07:02:22.4267644Z ##[group]Disabling automatic garbage collection 2024-06-26T07:02:22.4270199Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T07:02:22.4289346Z ##[endgroup] 2024-06-26T07:02:22.4289882Z ##[group]Setting up auth 2024-06-26T07:02:22.4294983Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T07:02:22.4320894Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T07:02:22.4524379Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T07:02:22.4546523Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T07:02:22.4742216Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:02:22.4782853Z ##[endgroup] 2024-06-26T07:02:22.4783417Z ##[group]Fetching the repository 2024-06-26T07:02:22.4789450Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T07:02:24.9080843Z remote: Enumerating objects: 987619 2024-06-26T07:02:24.9081423Z remote: Enumerating objects: 990209, done. 2024-06-26T07:02:24.9088308Z remote: Counting objects: 0% (1/2590) 2024-06-26T07:02:24.9088847Z remote: Counting objects: 1% (26/2590) 2024-06-26T07:02:24.9090446Z remote: Counting objects: 2% (52/2590) 2024-06-26T07:02:24.9093024Z remote: Counting objects: 3% (78/2590) 2024-06-26T07:02:24.9093556Z remote: Counting objects: 4% (104/2590) 2024-06-26T07:02:24.9095968Z remote: Counting objects: 5% (130/2590) 2024-06-26T07:02:24.9097998Z remote: Counting objects: 6% (156/2590) 2024-06-26T07:02:24.9099543Z remote: Counting objects: 7% (182/2590) 2024-06-26T07:02:24.9100075Z remote: Counting objects: 8% (208/2590) 2024-06-26T07:02:24.9100600Z remote: Counting objects: 9% (234/2590) 2024-06-26T07:02:24.9101131Z remote: Counting objects: 10% (259/2590) 2024-06-26T07:02:24.9108637Z remote: Counting objects: 11% (285/2590) 2024-06-26T07:02:24.9109170Z remote: Counting objects: 12% (311/2590) 2024-06-26T07:02:24.9109695Z remote: Counting objects: 13% (337/2590) 2024-06-26T07:02:24.9110210Z remote: Counting objects: 14% (363/2590) 2024-06-26T07:02:24.9110764Z remote: Counting objects: 15% (389/2590) 2024-06-26T07:02:24.9111423Z remote: Counting objects: 16% (415/2590) 2024-06-26T07:02:24.9112041Z remote: Counting objects: 17% (441/2590) 2024-06-26T07:02:24.9112566Z remote: Counting objects: 18% (467/2590) 2024-06-26T07:02:24.9113174Z remote: Counting objects: 19% (493/2590) 2024-06-26T07:02:24.9113734Z remote: Counting objects: 20% (518/2590) 2024-06-26T07:02:24.9114286Z remote: Counting objects: 21% (544/2590) 2024-06-26T07:02:24.9114831Z remote: Counting objects: 22% (570/2590) 2024-06-26T07:02:24.9115414Z remote: Counting objects: 23% (596/2590) 2024-06-26T07:02:24.9116015Z remote: Counting objects: 24% (622/2590) 2024-06-26T07:02:24.9116623Z remote: Counting objects: 25% (648/2590) 2024-06-26T07:02:24.9117538Z remote: Counting objects: 26% (674/2590) 2024-06-26T07:02:24.9118136Z remote: Counting objects: 27% (700/2590) 2024-06-26T07:02:24.9118701Z remote: Counting objects: 28% (726/2590) 2024-06-26T07:02:24.9119225Z remote: Counting objects: 29% (752/2590) 2024-06-26T07:02:24.9119831Z remote: Counting objects: 30% (777/2590) 2024-06-26T07:02:24.9120416Z remote: Counting objects: 31% (803/2590) 2024-06-26T07:02:24.9121014Z remote: Counting objects: 32% (829/2590) 2024-06-26T07:02:24.9121602Z remote: Counting objects: 33% (855/2590) 2024-06-26T07:02:24.9122187Z remote: Counting objects: 34% (881/2590) 2024-06-26T07:02:24.9122778Z remote: Counting objects: 35% (907/2590) 2024-06-26T07:02:24.9123317Z remote: Counting objects: 36% (933/2590) 2024-06-26T07:02:24.9123932Z remote: Counting objects: 37% (959/2590) 2024-06-26T07:02:24.9124491Z remote: Counting objects: 38% (985/2590) 2024-06-26T07:02:24.9125194Z remote: Counting objects: 39% (1011/2590) 2024-06-26T07:02:24.9125855Z remote: Counting objects: 40% (1036/2590) 2024-06-26T07:02:24.9126399Z remote: Counting objects: 41% (1062/2590) 2024-06-26T07:02:24.9126978Z remote: Counting objects: 42% (1088/2590) 2024-06-26T07:02:24.9127693Z remote: Counting objects: 43% (1114/2590) 2024-06-26T07:02:24.9128233Z remote: Counting objects: 44% (1140/2590) 2024-06-26T07:02:24.9128838Z remote: Counting objects: 45% (1166/2590) 2024-06-26T07:02:24.9129506Z remote: Counting objects: 46% (1192/2590) 2024-06-26T07:02:24.9130209Z remote: Counting objects: 47% (1218/2590) 2024-06-26T07:02:24.9131084Z remote: Counting objects: 48% (1244/2590) 2024-06-26T07:02:24.9131795Z remote: Counting objects: 49% (1270/2590) 2024-06-26T07:02:24.9132489Z remote: Counting objects: 50% (1295/2590) 2024-06-26T07:02:24.9133212Z remote: Counting objects: 51% (1321/2590) 2024-06-26T07:02:24.9133907Z remote: Counting objects: 52% (1347/2590) 2024-06-26T07:02:24.9134603Z remote: Counting objects: 53% (1373/2590) 2024-06-26T07:02:24.9135310Z remote: Counting objects: 54% (1399/2590) 2024-06-26T07:02:24.9136008Z remote: Counting objects: 55% (1425/2590) 2024-06-26T07:02:24.9136731Z remote: Counting objects: 56% (1451/2590) 2024-06-26T07:02:24.9137526Z remote: Counting objects: 57% (1477/2590) 2024-06-26T07:02:24.9138235Z remote: Counting objects: 58% (1503/2590) 2024-06-26T07:02:24.9138913Z remote: Counting objects: 59% (1529/2590) 2024-06-26T07:02:24.9139620Z remote: Counting objects: 60% (1554/2590) 2024-06-26T07:02:24.9140172Z remote: Counting objects: 61% (1580/2590) 2024-06-26T07:02:24.9140708Z remote: Counting objects: 62% (1606/2590) 2024-06-26T07:02:24.9141415Z remote: Counting objects: 63% (1632/2590) 2024-06-26T07:02:24.9142049Z remote: Counting objects: 64% (1658/2590) 2024-06-26T07:02:24.9142582Z remote: Counting objects: 65% (1684/2590) 2024-06-26T07:02:24.9143122Z remote: Counting objects: 66% (1710/2590) 2024-06-26T07:02:24.9143655Z remote: Counting objects: 67% (1736/2590) 2024-06-26T07:02:24.9144191Z remote: Counting objects: 68% (1762/2590) 2024-06-26T07:02:24.9144740Z remote: Counting objects: 69% (1788/2590) 2024-06-26T07:02:24.9145446Z remote: Counting objects: 70% (1813/2590) 2024-06-26T07:02:24.9146039Z remote: Counting objects: 71% (1839/2590) 2024-06-26T07:02:24.9146591Z remote: Counting objects: 72% (1865/2590) 2024-06-26T07:02:24.9147290Z remote: Counting objects: 73% (1891/2590) 2024-06-26T07:02:24.9147831Z remote: Counting objects: 74% (1917/2590) 2024-06-26T07:02:24.9148365Z remote: Counting objects: 75% (1943/2590) 2024-06-26T07:02:24.9148897Z remote: Counting objects: 76% (1969/2590) 2024-06-26T07:02:24.9149535Z remote: Counting objects: 77% (1995/2590) 2024-06-26T07:02:24.9150073Z remote: Counting objects: 78% (2021/2590) 2024-06-26T07:02:24.9150609Z remote: Counting objects: 79% (2047/2590) 2024-06-26T07:02:24.9151139Z remote: Counting objects: 80% (2072/2590) 2024-06-26T07:02:24.9151670Z remote: Counting objects: 81% (2098/2590) 2024-06-26T07:02:24.9152201Z remote: Counting objects: 82% (2124/2590) 2024-06-26T07:02:24.9152738Z remote: Counting objects: 83% (2150/2590) 2024-06-26T07:02:24.9153269Z remote: Counting objects: 84% (2176/2590) 2024-06-26T07:02:24.9153796Z remote: Counting objects: 85% (2202/2590) 2024-06-26T07:02:24.9154330Z remote: Counting objects: 86% (2228/2590) 2024-06-26T07:02:24.9154868Z remote: Counting objects: 87% (2254/2590) 2024-06-26T07:02:24.9155404Z remote: Counting objects: 88% (2280/2590) 2024-06-26T07:02:24.9155930Z remote: Counting objects: 89% (2306/2590) 2024-06-26T07:02:24.9156475Z remote: Counting objects: 90% (2331/2590) 2024-06-26T07:02:24.9157013Z remote: Counting objects: 91% (2357/2590) 2024-06-26T07:02:24.9157547Z remote: Counting objects: 92% (2383/2590) 2024-06-26T07:02:24.9158072Z remote: Counting objects: 93% (2409/2590) 2024-06-26T07:02:24.9158607Z remote: Counting objects: 94% (2435/2590) 2024-06-26T07:02:24.9159142Z remote: Counting objects: 95% (2461/2590) 2024-06-26T07:02:24.9159675Z remote: Counting objects: 96% (2487/2590) 2024-06-26T07:02:24.9160200Z remote: Counting objects: 97% (2513/2590) 2024-06-26T07:02:24.9160741Z remote: Counting objects: 98% (2539/2590) 2024-06-26T07:02:24.9161277Z remote: Counting objects: 99% (2565/2590) 2024-06-26T07:02:24.9161882Z remote: Counting objects: 100% (2590/2590) 2024-06-26T07:02:24.9162446Z remote: Counting objects: 100% (2590/2590), done. 2024-06-26T07:02:24.9339164Z remote: Compressing objects: 0% (1/1068) 2024-06-26T07:02:24.9883762Z remote: Compressing objects: 1% (11/1068) 2024-06-26T07:02:25.0349555Z remote: Compressing objects: 2% (22/1068) 2024-06-26T07:02:25.0755343Z remote: Compressing objects: 3% (33/1068) 2024-06-26T07:02:25.1001240Z remote: Compressing objects: 4% (43/1068) 2024-06-26T07:02:25.1297474Z remote: Compressing objects: 5% (54/1068) 2024-06-26T07:02:25.1492311Z remote: Compressing objects: 6% (65/1068) 2024-06-26T07:02:25.1887783Z remote: Compressing objects: 7% (75/1068) 2024-06-26T07:02:25.2990288Z remote: Compressing objects: 8% (86/1068) 2024-06-26T07:02:25.3827042Z remote: Compressing objects: 9% (97/1068) 2024-06-26T07:02:25.4087100Z remote: Compressing objects: 10% (107/1068) 2024-06-26T07:02:25.4274394Z remote: Compressing objects: 11% (118/1068) 2024-06-26T07:02:25.4388983Z remote: Compressing objects: 12% (129/1068) 2024-06-26T07:02:25.4390220Z remote: Compressing objects: 13% (139/1068) 2024-06-26T07:02:25.4541378Z remote: Compressing objects: 14% (150/1068) 2024-06-26T07:02:25.4541969Z remote: Compressing objects: 15% (161/1068) 2024-06-26T07:02:25.4637470Z remote: Compressing objects: 16% (171/1068) 2024-06-26T07:02:25.4638657Z remote: Compressing objects: 17% (182/1068) 2024-06-26T07:02:25.4639663Z remote: Compressing objects: 18% (193/1068) 2024-06-26T07:02:25.4660298Z remote: Compressing objects: 19% (203/1068) 2024-06-26T07:02:25.4695994Z remote: Compressing objects: 20% (214/1068) 2024-06-26T07:02:25.4715837Z remote: Compressing objects: 21% (225/1068) 2024-06-26T07:02:25.4753476Z remote: Compressing objects: 22% (235/1068) 2024-06-26T07:02:25.4774908Z remote: Compressing objects: 23% (246/1068) 2024-06-26T07:02:25.4844834Z remote: Compressing objects: 24% (257/1068) 2024-06-26T07:02:25.4860781Z remote: Compressing objects: 25% (267/1068) 2024-06-26T07:02:25.4887915Z remote: Compressing objects: 26% (278/1068) 2024-06-26T07:02:25.4898911Z remote: Compressing objects: 27% (289/1068) 2024-06-26T07:02:25.4907320Z remote: Compressing objects: 28% (300/1068) 2024-06-26T07:02:25.4922884Z remote: Compressing objects: 29% (310/1068) 2024-06-26T07:02:25.4940411Z remote: Compressing objects: 30% (321/1068) 2024-06-26T07:02:25.4950554Z remote: Compressing objects: 31% (332/1068) 2024-06-26T07:02:25.4963642Z remote: Compressing objects: 32% (342/1068) 2024-06-26T07:02:25.4986633Z remote: Compressing objects: 33% (353/1068) 2024-06-26T07:02:25.5009834Z remote: Compressing objects: 34% (364/1068) 2024-06-26T07:02:25.5045733Z remote: Compressing objects: 35% (374/1068) 2024-06-26T07:02:25.5054216Z remote: Compressing objects: 36% (385/1068) 2024-06-26T07:02:25.5070587Z remote: Compressing objects: 37% (396/1068) 2024-06-26T07:02:25.5096979Z remote: Compressing objects: 38% (406/1068) 2024-06-26T07:02:25.5106200Z remote: Compressing objects: 39% (417/1068) 2024-06-26T07:02:25.5123069Z remote: Compressing objects: 40% (428/1068) 2024-06-26T07:02:25.5143504Z remote: Compressing objects: 41% (438/1068) 2024-06-26T07:02:25.5158863Z remote: Compressing objects: 42% (449/1068) 2024-06-26T07:02:25.5174297Z remote: Compressing objects: 43% (460/1068) 2024-06-26T07:02:25.5182845Z remote: Compressing objects: 44% (470/1068) 2024-06-26T07:02:25.5189882Z remote: Compressing objects: 45% (481/1068) 2024-06-26T07:02:25.5200251Z remote: Compressing objects: 46% (492/1068) 2024-06-26T07:02:25.5221253Z remote: Compressing objects: 47% (502/1068) 2024-06-26T07:02:25.5238696Z remote: Compressing objects: 48% (513/1068) 2024-06-26T07:02:25.5247317Z remote: Compressing objects: 49% (524/1068) 2024-06-26T07:02:25.5255230Z remote: Compressing objects: 50% (534/1068) 2024-06-26T07:02:25.5265409Z remote: Compressing objects: 51% (545/1068) 2024-06-26T07:02:25.5283524Z remote: Compressing objects: 52% (556/1068) 2024-06-26T07:02:25.5296521Z remote: Compressing objects: 53% (567/1068) 2024-06-26T07:02:25.5312467Z remote: Compressing objects: 54% (577/1068) 2024-06-26T07:02:25.5321440Z remote: Compressing objects: 55% (588/1068) 2024-06-26T07:02:25.5332060Z remote: Compressing objects: 56% (599/1068) 2024-06-26T07:02:25.5347606Z remote: Compressing objects: 57% (609/1068) 2024-06-26T07:02:25.5361527Z remote: Compressing objects: 58% (620/1068) 2024-06-26T07:02:25.5370224Z remote: Compressing objects: 59% (631/1068) 2024-06-26T07:02:25.5382051Z remote: Compressing objects: 60% (641/1068) 2024-06-26T07:02:25.5396569Z remote: Compressing objects: 61% (652/1068) 2024-06-26T07:02:25.5405338Z remote: Compressing objects: 62% (663/1068) 2024-06-26T07:02:25.5415244Z remote: Compressing objects: 63% (673/1068) 2024-06-26T07:02:25.5424233Z remote: Compressing objects: 64% (684/1068) 2024-06-26T07:02:25.5434146Z remote: Compressing objects: 65% (695/1068) 2024-06-26T07:02:25.5441602Z remote: Compressing objects: 66% (705/1068) 2024-06-26T07:02:25.5451699Z remote: Compressing objects: 67% (716/1068) 2024-06-26T07:02:25.5452797Z remote: Compressing objects: 68% (727/1068) 2024-06-26T07:02:25.5456802Z remote: Compressing objects: 69% (737/1068) 2024-06-26T07:02:25.5460866Z remote: Compressing objects: 70% (748/1068) 2024-06-26T07:02:25.5468588Z remote: Compressing objects: 71% (759/1068) 2024-06-26T07:02:25.5469405Z remote: Compressing objects: 72% (769/1068) 2024-06-26T07:02:25.5470096Z remote: Compressing objects: 73% (780/1068) 2024-06-26T07:02:25.5471334Z remote: Compressing objects: 74% (791/1068) 2024-06-26T07:02:25.5472263Z remote: Compressing objects: 75% (801/1068) 2024-06-26T07:02:25.5472862Z remote: Compressing objects: 76% (812/1068) 2024-06-26T07:02:25.5473413Z remote: Compressing objects: 77% (823/1068) 2024-06-26T07:02:25.5473960Z remote: Compressing objects: 78% (834/1068) 2024-06-26T07:02:25.5477766Z remote: Compressing objects: 79% (844/1068) 2024-06-26T07:02:25.5478544Z remote: Compressing objects: 80% (855/1068) 2024-06-26T07:02:25.5480837Z remote: Compressing objects: 81% (866/1068) 2024-06-26T07:02:25.5488326Z remote: Compressing objects: 82% (876/1068) 2024-06-26T07:02:25.5489144Z remote: Compressing objects: 83% (887/1068) 2024-06-26T07:02:25.5489700Z remote: Compressing objects: 84% (898/1068) 2024-06-26T07:02:25.5490249Z remote: Compressing objects: 85% (908/1068) 2024-06-26T07:02:25.5492494Z remote: Compressing objects: 86% (919/1068) 2024-06-26T07:02:25.5494965Z remote: Compressing objects: 87% (930/1068) 2024-06-26T07:02:25.5496924Z remote: Compressing objects: 88% (940/1068) 2024-06-26T07:02:25.5498385Z remote: Compressing objects: 89% (951/1068) 2024-06-26T07:02:25.5501918Z remote: Compressing objects: 90% (962/1068) 2024-06-26T07:02:25.5502476Z remote: Compressing objects: 91% (972/1068) 2024-06-26T07:02:25.5504744Z remote: Compressing objects: 92% (983/1068) 2024-06-26T07:02:25.5505515Z remote: Compressing objects: 93% (994/1068) 2024-06-26T07:02:25.5507090Z remote: Compressing objects: 94% (1004/1068) 2024-06-26T07:02:25.5509775Z remote: Compressing objects: 95% (1015/1068) 2024-06-26T07:02:25.5510358Z remote: Compressing objects: 96% (1026/1068) 2024-06-26T07:02:25.5511804Z remote: Compressing objects: 97% (1036/1068) 2024-06-26T07:02:25.5514233Z remote: Compressing objects: 98% (1047/1068) 2024-06-26T07:02:25.5516304Z remote: Compressing objects: 99% (1058/1068) 2024-06-26T07:02:25.5516889Z remote: Compressing objects: 100% (1068/1068) 2024-06-26T07:02:25.5517504Z remote: Compressing objects: 100% (1068/1068), done. 2024-06-26T07:02:44.5368714Z remote: Total 990209 (delta 2055), reused 1861 (delta 1521), pack-reused 987619 2024-06-26T07:03:09.0436786Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T07:03:09.0456670Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:03:09.0460077Z ##[endgroup] 2024-06-26T07:03:09.0460743Z ##[group]Determining the checkout info 2024-06-26T07:03:09.0463408Z ##[endgroup] 2024-06-26T07:03:09.0464094Z ##[group]Checking out the ref 2024-06-26T07:03:09.0464946Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:03:10.2713315Z ##[endgroup] 2024-06-26T07:03:10.2714055Z ##[group]Setting up auth for fetching submodules 2024-06-26T07:03:10.2717673Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:03:10.2757017Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T07:03:10.2776722Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T07:03:10.2796290Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T07:03:10.2812584Z ##[endgroup] 2024-06-26T07:03:10.2813141Z ##[group]Fetching submodules 2024-06-26T07:03:10.2816049Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T07:03:10.3017643Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T07:03:10.3223512Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T07:03:10.3225231Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T07:03:10.3227270Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T07:03:10.3228669Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T07:03:10.3230646Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T07:03:10.3232597Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T07:03:10.3234243Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T07:03:10.3236358Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T07:03:10.3238512Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T07:03:10.3241214Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T07:03:10.3243280Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T07:03:10.3246090Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T07:03:10.3248668Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T07:03:10.3251315Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T07:03:10.3253868Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T07:03:10.3256614Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T07:03:10.3259570Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:03:10.3262284Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T07:03:10.3265234Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T07:03:10.3268451Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T07:03:10.3271674Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T07:03:10.3274804Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T07:03:10.3278140Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T07:03:10.3281383Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T07:03:10.3284681Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T07:03:10.3288145Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T07:03:10.3291908Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T07:03:10.3295154Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T07:03:10.3298732Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T07:03:10.3302397Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T07:03:10.3306349Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T07:03:10.3310479Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T07:03:10.3314177Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T07:03:10.3317939Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T07:03:10.3322027Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T07:03:10.3341028Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T07:03:10.5873741Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T07:03:10.7822208Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T07:03:10.9428352Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T07:03:11.1728721Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T07:03:13.2445646Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T07:03:22.9297568Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T07:03:23.3075423Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T07:03:23.7998287Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T07:03:24.4163385Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T07:03:25.6633869Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T07:03:27.3906075Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T07:03:33.2270444Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T07:03:34.4910455Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T07:03:36.1341002Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T07:03:37.3607958Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T07:03:37.5635273Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T07:03:37.9748268Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T07:03:38.3024400Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T07:03:39.2854391Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T07:03:39.6074607Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T07:03:39.8755186Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T07:03:41.3130942Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T07:03:42.0542518Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T07:03:42.7504943Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T07:03:48.2819929Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T07:03:50.7910981Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T07:03:56.3469213Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T07:03:56.6082806Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T07:04:05.0287251Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T07:04:05.2322574Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T07:04:05.5250587Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T07:04:06.5408497Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T07:04:06.8598573Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T07:04:07.4608543Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T07:04:07.9099350Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T07:04:07.9184253Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T07:04:07.9244529Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T07:04:07.9424656Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T07:04:07.9733969Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T07:04:08.7473126Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T07:04:08.7645448Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T07:04:08.8001920Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T07:04:08.8844834Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T07:04:08.9116167Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T07:04:09.3420353Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T07:04:09.5555164Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T07:04:09.6194005Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T07:04:09.6205200Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:09.6207015Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:09.6208731Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:09.6210644Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:09.6212988Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:09.6230147Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T07:04:10.6636853Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T07:04:11.2545528Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T07:04:12.9707761Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T07:04:13.9532043Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T07:04:14.2850443Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T07:04:14.3688701Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T07:04:14.7193378Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T07:04:14.7732006Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T07:04:14.7818678Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T07:04:14.8729192Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T07:04:14.9020444Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T07:04:14.9086179Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T07:04:14.9392789Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T07:04:14.9588627Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T07:04:14.9956288Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:04:15.0053658Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T07:04:15.0064756Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:15.0080301Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T07:04:26.8556901Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T07:04:26.8700965Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T07:04:26.9484591Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T07:04:26.9496159Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:26.9497975Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:26.9499808Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:26.9517117Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T07:04:27.5045039Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T07:04:28.7323248Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T07:04:29.7537273Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T07:04:29.7547112Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:29.7549134Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:29.7551062Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:29.7553019Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:29.7555082Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:29.7557129Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:29.7559122Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:29.7561286Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:29.7580818Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T07:04:30.8428556Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T07:04:31.2107813Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T07:04:32.4368700Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T07:04:32.7263358Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T07:04:33.2369064Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T07:04:34.2305152Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T07:04:39.7980533Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T07:04:40.1609690Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T07:04:40.1744901Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T07:04:40.2038906Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T07:04:40.2137372Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T07:04:40.2147960Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:40.2165566Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T07:04:40.5195586Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T07:04:40.5333585Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T07:04:40.5670512Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T07:04:40.6462252Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T07:04:40.6582789Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T07:04:40.6881824Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T07:04:40.7374341Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T07:04:40.7676377Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T07:04:40.7873640Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T07:04:40.8751868Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T07:04:41.1595225Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T07:04:41.1622483Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:41.1624325Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:41.1643841Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T07:04:41.5940458Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T07:04:42.7583094Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T07:04:42.7859347Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T07:04:42.8405409Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T07:04:42.8417848Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:42.8424256Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:42.8426112Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:42.8427970Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:42.8429882Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:42.8431937Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:42.8433831Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:42.8435478Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:42.8448392Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T07:04:43.3170174Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T07:04:44.3130496Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T07:04:44.6765736Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T07:04:50.6013952Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T07:04:50.8811661Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T07:04:51.0806749Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T07:04:51.3547957Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T07:04:58.8835758Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T07:04:58.9168843Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T07:04:58.9295931Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T07:04:59.0134448Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T07:04:59.0233833Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T07:04:59.0341739Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T07:04:59.0456867Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T07:04:59.0468015Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:59.0470131Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:59.0487555Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T07:05:00.7759419Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T07:05:01.9610407Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T07:05:01.9982337Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:05:02.3600238Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T07:05:02.3687717Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T07:05:02.5861742Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T07:05:02.5876195Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:05:02.5877944Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T07:05:02.5893456Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T07:05:02.9762735Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T07:05:03.9615028Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T07:05:04.0241000Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T07:05:04.0304093Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T07:05:04.0387954Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T07:05:04.0662180Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T07:05:04.0884916Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T07:05:04.1216715Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T07:05:04.1419087Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T07:05:04.1429991Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:05:04.1431598Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:05:04.1433205Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:05:04.1435037Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:05:04.1453208Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T07:05:05.1718967Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T07:05:05.3933079Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T07:05:06.4763674Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T07:05:07.5424682Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T07:05:07.5539117Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T07:05:07.6101745Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T07:05:07.6332228Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T07:05:07.6343315Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:05:07.6360254Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T07:05:07.8786447Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T07:05:07.8809002Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T07:05:07.9020443Z Entering 'android/libs/fbjni' 2024-06-26T07:05:07.9048567Z Entering 'third_party/FP16' 2024-06-26T07:05:07.9074309Z Entering 'third_party/FXdiv' 2024-06-26T07:05:07.9103539Z Entering 'third_party/NNPACK' 2024-06-26T07:05:07.9130147Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:05:07.9157828Z Entering 'third_party/XNNPACK' 2024-06-26T07:05:07.9198492Z Entering 'third_party/benchmark' 2024-06-26T07:05:07.9226030Z Entering 'third_party/cpp-httplib' 2024-06-26T07:05:07.9254864Z Entering 'third_party/cpuinfo' 2024-06-26T07:05:07.9282025Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:05:07.9308209Z Entering 'third_party/cutlass' 2024-06-26T07:05:07.9342476Z Entering 'third_party/eigen' 2024-06-26T07:05:07.9372302Z Entering 'third_party/fbgemm' 2024-06-26T07:05:07.9399905Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:05:07.9428339Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:05:07.9455564Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:05:07.9488250Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:05:07.9516297Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:05:07.9544508Z Entering 'third_party/flatbuffers' 2024-06-26T07:05:07.9572568Z Entering 'third_party/fmt' 2024-06-26T07:05:07.9599416Z Entering 'third_party/foxi' 2024-06-26T07:05:07.9626377Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:05:07.9655096Z Entering 'third_party/gloo' 2024-06-26T07:05:07.9682544Z Entering 'third_party/googletest' 2024-06-26T07:05:07.9710294Z Entering 'third_party/ideep' 2024-06-26T07:05:07.9738929Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:05:07.9772322Z Entering 'third_party/ittapi' 2024-06-26T07:05:07.9800131Z Entering 'third_party/kineto' 2024-06-26T07:05:07.9829122Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:05:07.9860626Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:05:07.9888748Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:05:07.9914105Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:05:07.9940841Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:05:07.9968677Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:05:07.9996856Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:05:08.0026525Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:05:08.0055936Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:05:08.0084098Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:05:08.0110657Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:05:08.0137753Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:05:08.0167607Z Entering 'third_party/mimalloc' 2024-06-26T07:05:08.0194253Z Entering 'third_party/nccl/nccl' 2024-06-26T07:05:08.0222881Z Entering 'third_party/nlohmann' 2024-06-26T07:05:08.0251661Z Entering 'third_party/onnx' 2024-06-26T07:05:08.0292444Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:05:08.0322386Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:05:08.0350573Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:05:08.0379194Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:05:08.0407683Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:05:08.0433013Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:05:08.0461165Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:05:08.0490191Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:05:08.0516676Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:05:08.0543889Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:05:08.0568934Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:05:08.0596588Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:05:08.0626231Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:05:08.0668566Z Entering 'third_party/pocketfft' 2024-06-26T07:05:08.0695643Z Entering 'third_party/protobuf' 2024-06-26T07:05:08.0725274Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:05:08.0752467Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:05:08.0781971Z Entering 'third_party/psimd' 2024-06-26T07:05:08.0812254Z Entering 'third_party/pthreadpool' 2024-06-26T07:05:08.0839458Z Entering 'third_party/pybind11' 2024-06-26T07:05:08.0867970Z Entering 'third_party/python-peachpy' 2024-06-26T07:05:08.0896494Z Entering 'third_party/sleef' 2024-06-26T07:05:08.0923329Z Entering 'third_party/tensorpipe' 2024-06-26T07:05:08.0951911Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:05:08.0978959Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:05:08.1006191Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:05:08.1032262Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:05:08.1060077Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:05:08.1100086Z ##[endgroup] 2024-06-26T07:05:08.1101702Z ##[group]Persisting credentials for submodules 2024-06-26T07:05:08.1105232Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T07:05:08.1318925Z Entering 'android/libs/fbjni' 2024-06-26T07:05:08.1354990Z Entering 'third_party/FP16' 2024-06-26T07:05:08.1391713Z Entering 'third_party/FXdiv' 2024-06-26T07:05:08.1428957Z Entering 'third_party/NNPACK' 2024-06-26T07:05:08.1462647Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:05:08.1498461Z Entering 'third_party/XNNPACK' 2024-06-26T07:05:08.1545719Z Entering 'third_party/benchmark' 2024-06-26T07:05:08.1581651Z Entering 'third_party/cpp-httplib' 2024-06-26T07:05:08.1616999Z Entering 'third_party/cpuinfo' 2024-06-26T07:05:08.1651478Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:05:08.1686922Z Entering 'third_party/cutlass' 2024-06-26T07:05:08.1729207Z Entering 'third_party/eigen' 2024-06-26T07:05:08.1765241Z Entering 'third_party/fbgemm' 2024-06-26T07:05:08.1800206Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:05:08.1834592Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:05:08.1867307Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:05:08.1906094Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:05:08.1941995Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:05:08.1979022Z Entering 'third_party/flatbuffers' 2024-06-26T07:05:08.2018319Z Entering 'third_party/fmt' 2024-06-26T07:05:08.2054557Z Entering 'third_party/foxi' 2024-06-26T07:05:08.2091845Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:05:08.2125368Z Entering 'third_party/gloo' 2024-06-26T07:05:08.2160056Z Entering 'third_party/googletest' 2024-06-26T07:05:08.2193588Z Entering 'third_party/ideep' 2024-06-26T07:05:08.2229593Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:05:08.2266381Z Entering 'third_party/ittapi' 2024-06-26T07:05:08.2301816Z Entering 'third_party/kineto' 2024-06-26T07:05:08.2337927Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:05:08.2373366Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:05:08.2408282Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:05:08.2443307Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:05:08.2478359Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:05:08.2509985Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:05:08.2546845Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:05:08.2582123Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:05:08.2618728Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:05:08.2656077Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:05:08.2693602Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:05:08.2726889Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:05:08.2762880Z Entering 'third_party/mimalloc' 2024-06-26T07:05:08.2799533Z Entering 'third_party/nccl/nccl' 2024-06-26T07:05:08.2834958Z Entering 'third_party/nlohmann' 2024-06-26T07:05:08.2872067Z Entering 'third_party/onnx' 2024-06-26T07:05:08.2917485Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:05:08.2953757Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:05:08.2987705Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:05:08.3024610Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:05:08.3060862Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:05:08.3095089Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:05:08.3129079Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:05:08.3163187Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:05:08.3197278Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:05:08.3228976Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:05:08.3262229Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:05:08.3300221Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:05:08.3335372Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:05:08.3384035Z Entering 'third_party/pocketfft' 2024-06-26T07:05:08.3421463Z Entering 'third_party/protobuf' 2024-06-26T07:05:08.3460313Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:05:08.3494095Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:05:08.3528068Z Entering 'third_party/psimd' 2024-06-26T07:05:08.3563866Z Entering 'third_party/pthreadpool' 2024-06-26T07:05:08.3598993Z Entering 'third_party/pybind11' 2024-06-26T07:05:08.3634571Z Entering 'third_party/python-peachpy' 2024-06-26T07:05:08.3668447Z Entering 'third_party/sleef' 2024-06-26T07:05:08.3702480Z Entering 'third_party/tensorpipe' 2024-06-26T07:05:08.3733867Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:05:08.3768614Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:05:08.3802648Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:05:08.3836268Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:05:08.3869846Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:05:08.3918216Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T07:05:08.4131697Z Entering 'android/libs/fbjni' 2024-06-26T07:05:08.4164181Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T07:05:08.4176451Z Entering 'third_party/FP16' 2024-06-26T07:05:08.4208469Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T07:05:08.4221161Z Entering 'third_party/FXdiv' 2024-06-26T07:05:08.4253646Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T07:05:08.4268389Z Entering 'third_party/NNPACK' 2024-06-26T07:05:08.4302081Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T07:05:08.4316811Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:05:08.4346568Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T07:05:08.4359548Z Entering 'third_party/XNNPACK' 2024-06-26T07:05:08.4391511Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T07:05:08.4416784Z Entering 'third_party/benchmark' 2024-06-26T07:05:08.4449292Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:05:08.4460614Z Entering 'third_party/cpp-httplib' 2024-06-26T07:05:08.4493706Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T07:05:08.4507875Z Entering 'third_party/cpuinfo' 2024-06-26T07:05:08.4539222Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:05:08.4551975Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:05:08.4583878Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T07:05:08.4596016Z Entering 'third_party/cutlass' 2024-06-26T07:05:08.4627561Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:05:08.4646488Z Entering 'third_party/eigen' 2024-06-26T07:05:08.4679273Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T07:05:08.4693840Z Entering 'third_party/fbgemm' 2024-06-26T07:05:08.4726760Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T07:05:08.4740109Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:05:08.4771994Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T07:05:08.4785332Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:05:08.4818418Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:05:08.4831592Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:05:08.4863124Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:05:08.4880333Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:05:08.4911466Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:05:08.4923225Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:05:08.4954929Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T07:05:08.4968281Z Entering 'third_party/flatbuffers' 2024-06-26T07:05:08.4999716Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T07:05:08.5016407Z Entering 'third_party/fmt' 2024-06-26T07:05:08.5049143Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:05:08.5061633Z Entering 'third_party/foxi' 2024-06-26T07:05:08.5094426Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T07:05:08.5107879Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:05:08.5137432Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T07:05:08.5151819Z Entering 'third_party/gloo' 2024-06-26T07:05:08.5184317Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T07:05:08.5196518Z Entering 'third_party/googletest' 2024-06-26T07:05:08.5227776Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:05:08.5241817Z Entering 'third_party/ideep' 2024-06-26T07:05:08.5274105Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T07:05:08.5287032Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:05:08.5318649Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T07:05:08.5338188Z Entering 'third_party/ittapi' 2024-06-26T07:05:08.5368412Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T07:05:08.5379387Z Entering 'third_party/kineto' 2024-06-26T07:05:08.5412278Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T07:05:08.5423436Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:05:08.5456189Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T07:05:08.5469725Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:05:08.5501820Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T07:05:08.5515470Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:05:08.5547903Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T07:05:08.5560944Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:05:08.5593035Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:05:08.5606025Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:05:08.5638978Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T07:05:08.5651644Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:05:08.5683578Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T07:05:08.5699036Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:05:08.5731551Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T07:05:08.5743889Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:05:08.5776690Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:05:08.5789036Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:05:08.5820589Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T07:05:08.5834749Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:05:08.5866819Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T07:05:08.5881695Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:05:08.5913037Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T07:05:08.5925849Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:05:08.5956703Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T07:05:08.5970103Z Entering 'third_party/mimalloc' 2024-06-26T07:05:08.6001466Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T07:05:08.6015904Z Entering 'third_party/nccl/nccl' 2024-06-26T07:05:08.6049258Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T07:05:08.6062695Z Entering 'third_party/nlohmann' 2024-06-26T07:05:08.6094252Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T07:05:08.6109993Z Entering 'third_party/onnx' 2024-06-26T07:05:08.6140128Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T07:05:08.6163668Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:05:08.6195041Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:05:08.6208445Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:05:08.6240210Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:05:08.6256082Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:05:08.6288454Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T07:05:08.6302506Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:05:08.6335461Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:05:08.6346995Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:05:08.6378406Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:05:08.6390345Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:05:08.6420997Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T07:05:08.6433084Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:05:08.6463916Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T07:05:08.6477968Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:05:08.6509438Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T07:05:08.6521918Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:05:08.6552546Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T07:05:08.6564389Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:05:08.6595751Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T07:05:08.6605982Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:05:08.6638037Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T07:05:08.6653105Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:05:08.6684692Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T07:05:08.6699702Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:05:08.6730351Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T07:05:08.6758539Z Entering 'third_party/pocketfft' 2024-06-26T07:05:08.6790807Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T07:05:08.6803133Z Entering 'third_party/protobuf' 2024-06-26T07:05:08.6834272Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T07:05:08.6849381Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:05:08.6881940Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:05:08.6895133Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:05:08.6926456Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:05:08.6939731Z Entering 'third_party/psimd' 2024-06-26T07:05:08.6974646Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T07:05:08.6987114Z Entering 'third_party/pthreadpool' 2024-06-26T07:05:08.7018362Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T07:05:08.7032427Z Entering 'third_party/pybind11' 2024-06-26T07:05:08.7062401Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:05:08.7076000Z Entering 'third_party/python-peachpy' 2024-06-26T07:05:08.7109304Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T07:05:08.7122601Z Entering 'third_party/sleef' 2024-06-26T07:05:08.7155643Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T07:05:08.7168025Z Entering 'third_party/tensorpipe' 2024-06-26T07:05:08.7200700Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T07:05:08.7213677Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:05:08.7246112Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:05:08.7258102Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:05:08.7290383Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T07:05:08.7302670Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:05:08.7335956Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T07:05:08.7348621Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:05:08.7379950Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:05:08.7390939Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:05:08.7422760Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T07:05:08.8445568Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T07:05:08.8651264Z Entering 'android/libs/fbjni' 2024-06-26T07:05:08.8676983Z Entering 'third_party/FP16' 2024-06-26T07:05:08.8702147Z Entering 'third_party/FXdiv' 2024-06-26T07:05:08.8727345Z Entering 'third_party/NNPACK' 2024-06-26T07:05:08.8753032Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:05:08.8778055Z Entering 'third_party/XNNPACK' 2024-06-26T07:05:08.8818658Z Entering 'third_party/benchmark' 2024-06-26T07:05:08.8845480Z Entering 'third_party/cpp-httplib' 2024-06-26T07:05:08.8871366Z Entering 'third_party/cpuinfo' 2024-06-26T07:05:08.8896327Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:05:08.8921979Z Entering 'third_party/cutlass' 2024-06-26T07:05:08.8956108Z Entering 'third_party/eigen' 2024-06-26T07:05:08.8987167Z Entering 'third_party/fbgemm' 2024-06-26T07:05:08.9015855Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:05:08.9044301Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:05:08.9073406Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:05:08.9105734Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:05:08.9133739Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:05:08.9162409Z Entering 'third_party/flatbuffers' 2024-06-26T07:05:08.9193635Z Entering 'third_party/fmt' 2024-06-26T07:05:08.9222642Z Entering 'third_party/foxi' 2024-06-26T07:05:08.9251023Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:05:08.9278200Z Entering 'third_party/gloo' 2024-06-26T07:05:08.9306897Z Entering 'third_party/googletest' 2024-06-26T07:05:08.9336748Z Entering 'third_party/ideep' 2024-06-26T07:05:08.9364295Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:05:08.9398410Z Entering 'third_party/ittapi' 2024-06-26T07:05:08.9428705Z Entering 'third_party/kineto' 2024-06-26T07:05:08.9458490Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:05:08.9485787Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:05:08.9515760Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:05:08.9542920Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:05:08.9571687Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:05:08.9599782Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:05:08.9628629Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:05:08.9656398Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:05:08.9681991Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:05:08.9710430Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:05:08.9739851Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:05:08.9768555Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:05:08.9797133Z Entering 'third_party/mimalloc' 2024-06-26T07:05:08.9827199Z Entering 'third_party/nccl/nccl' 2024-06-26T07:05:08.9857027Z Entering 'third_party/nlohmann' 2024-06-26T07:05:08.9884200Z Entering 'third_party/onnx' 2024-06-26T07:05:08.9922210Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:05:08.9949990Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:05:08.9979133Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:05:09.0008345Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:05:09.0036202Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:05:09.0063663Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:05:09.0090736Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:05:09.0118934Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:05:09.0147519Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:05:09.0175872Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:05:09.0201924Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:05:09.0230887Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:05:09.0260889Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:05:09.0303297Z Entering 'third_party/pocketfft' 2024-06-26T07:05:09.0333218Z Entering 'third_party/protobuf' 2024-06-26T07:05:09.0362874Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:05:09.0390097Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:05:09.0419221Z Entering 'third_party/psimd' 2024-06-26T07:05:09.0447241Z Entering 'third_party/pthreadpool' 2024-06-26T07:05:09.0476321Z Entering 'third_party/pybind11' 2024-06-26T07:05:09.0503738Z Entering 'third_party/python-peachpy' 2024-06-26T07:05:09.0532686Z Entering 'third_party/sleef' 2024-06-26T07:05:09.0559980Z Entering 'third_party/tensorpipe' 2024-06-26T07:05:09.0589651Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:05:09.0617070Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:05:09.0644500Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:05:09.0672271Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:05:09.0699104Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:05:09.0741292Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T07:05:09.0953059Z Entering 'android/libs/fbjni' 2024-06-26T07:05:09.0979933Z Entering 'third_party/FP16' 2024-06-26T07:05:09.1010498Z Entering 'third_party/FXdiv' 2024-06-26T07:05:09.1036300Z Entering 'third_party/NNPACK' 2024-06-26T07:05:09.1062187Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:05:09.1092321Z Entering 'third_party/XNNPACK' 2024-06-26T07:05:09.1129734Z Entering 'third_party/benchmark' 2024-06-26T07:05:09.1159386Z Entering 'third_party/cpp-httplib' 2024-06-26T07:05:09.1190695Z Entering 'third_party/cpuinfo' 2024-06-26T07:05:09.1220976Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:05:09.1248725Z Entering 'third_party/cutlass' 2024-06-26T07:05:09.1282226Z Entering 'third_party/eigen' 2024-06-26T07:05:09.1311007Z Entering 'third_party/fbgemm' 2024-06-26T07:05:09.1338924Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:05:09.1366366Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:05:09.1394636Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:05:09.1428145Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:05:09.1459895Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:05:09.1487823Z Entering 'third_party/flatbuffers' 2024-06-26T07:05:09.1516117Z Entering 'third_party/fmt' 2024-06-26T07:05:09.1543751Z Entering 'third_party/foxi' 2024-06-26T07:05:09.1573054Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:05:09.1600846Z Entering 'third_party/gloo' 2024-06-26T07:05:09.1629475Z Entering 'third_party/googletest' 2024-06-26T07:05:09.1658655Z Entering 'third_party/ideep' 2024-06-26T07:05:09.1684942Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:05:09.1717543Z Entering 'third_party/ittapi' 2024-06-26T07:05:09.1745090Z Entering 'third_party/kineto' 2024-06-26T07:05:09.1773743Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:05:09.1800572Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:05:09.1828230Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:05:09.1855569Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:05:09.1884089Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:05:09.1911688Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:05:09.1939445Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:05:09.1966674Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:05:09.1992263Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:05:09.2020861Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:05:09.2048264Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:05:09.2073460Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:05:09.2102727Z Entering 'third_party/mimalloc' 2024-06-26T07:05:09.2133051Z Entering 'third_party/nccl/nccl' 2024-06-26T07:05:09.2158375Z Entering 'third_party/nlohmann' 2024-06-26T07:05:09.2187968Z Entering 'third_party/onnx' 2024-06-26T07:05:09.2228336Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:05:09.2257047Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:05:09.2287003Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:05:09.2315609Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:05:09.2344215Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:05:09.2372995Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:05:09.2399780Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:05:09.2425940Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:05:09.2453377Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:05:09.2482590Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:05:09.2507198Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:05:09.2537290Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:05:09.2565182Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:05:09.2607735Z Entering 'third_party/pocketfft' 2024-06-26T07:05:09.2636963Z Entering 'third_party/protobuf' 2024-06-26T07:05:09.2667867Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:05:09.2694680Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:05:09.2725885Z Entering 'third_party/psimd' 2024-06-26T07:05:09.2753754Z Entering 'third_party/pthreadpool' 2024-06-26T07:05:09.2782933Z Entering 'third_party/pybind11' 2024-06-26T07:05:09.2812974Z Entering 'third_party/python-peachpy' 2024-06-26T07:05:09.2839932Z Entering 'third_party/sleef' 2024-06-26T07:05:09.2868075Z Entering 'third_party/tensorpipe' 2024-06-26T07:05:09.2894419Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:05:09.2924685Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:05:09.2952864Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:05:09.2981338Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:05:09.3006809Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:05:09.3046188Z ##[endgroup] 2024-06-26T07:05:09.3075684Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T07:05:09.3097361Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T07:05:09.3267243Z Prepare all required actions 2024-06-26T07:05:09.3267714Z Getting action download info 2024-06-26T07:05:09.4943423Z ##[group]Run ./.github/actions/setup-linux 2024-06-26T07:05:09.4943846Z env: 2024-06-26T07:05:09.4944125Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:09.4944473Z ##[endgroup] 2024-06-26T07:05:09.5002665Z ##[group]Run set -euo pipefail 2024-06-26T07:05:09.5003107Z set -euo pipefail 2024-06-26T07:05:09.5003512Z function get_ec2_metadata() { 2024-06-26T07:05:09.5004047Z  # Pulled from instance metadata endpoint for EC2 2024-06-26T07:05:09.5004933Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-26T07:05:09.5005779Z  category=$1 2024-06-26T07:05:09.5006311Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-26T07:05:09.5006952Z  runner_name_str=i-09875a2d32470f475 2024-06-26T07:05:09.5007467Z  if [[ -f /.inarc ]]; then 2024-06-26T07:05:09.5007964Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-26T07:05:09.5008541Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-26T07:05:09.5009226Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-26T07:05:09.5009836Z  else 2024-06-26T07:05:09.5010330Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-26T07:05:09.5010906Z  fi 2024-06-26T07:05:09.5011201Z } 2024-06-26T07:05:09.5011570Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-26T07:05:09.5012148Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-26T07:05:09.5012805Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-26T07:05:09.5013380Z echo "system info $(uname -a)" 2024-06-26T07:05:09.5021135Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:09.5021631Z env: 2024-06-26T07:05:09.5021919Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:09.5022267Z ##[endgroup] 2024-06-26T07:05:09.5094508Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-26T07:05:09.5134937Z instance-id: i-09875a2d32470f475 2024-06-26T07:05:09.5174832Z instance-type: g5.4xlarge 2024-06-26T07:05:09.5181137Z system info Linux ip-10-0-30-116.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-26T07:05:09.5201290Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:05:09.5202205Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:05:09.5210180Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:09.5210681Z env: 2024-06-26T07:05:09.5210978Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:09.5211338Z ##[endgroup] 2024-06-26T07:05:09.5272280Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-26T07:05:09.5273029Z if systemctl is-active --quiet docker; then 2024-06-26T07:05:09.5273569Z  echo "Docker daemon is running..."; 2024-06-26T07:05:09.5274028Z else 2024-06-26T07:05:09.5274522Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-26T07:05:09.5275100Z fi 2024-06-26T07:05:09.5282590Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:09.5283094Z env: 2024-06-26T07:05:09.5283369Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:09.5283714Z ##[endgroup] 2024-06-26T07:05:09.5320353Z Docker daemon is running... 2024-06-26T07:05:09.5363621Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:05:09.5364183Z with: 2024-06-26T07:05:09.5364461Z shell: bash 2024-06-26T07:05:09.5364761Z timeout_minutes: 5 2024-06-26T07:05:09.5365089Z max_attempts: 3 2024-06-26T07:05:09.5365410Z retry_wait_seconds: 30 2024-06-26T07:05:09.5366976Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-26T07:05:09.5368450Z polling_interval_seconds: 1 2024-06-26T07:05:09.5368836Z warning_on_retry: true 2024-06-26T07:05:09.5369194Z continue_on_error: false 2024-06-26T07:05:09.5369538Z env: 2024-06-26T07:05:09.5369813Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:09.5370177Z AWS_RETRY_MODE: standard 2024-06-26T07:05:09.5370536Z AWS_MAX_ATTEMPTS: 5 2024-06-26T07:05:09.5370878Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:05:09.5371259Z ##[endgroup] 2024-06-26T07:05:10.3909514Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:05:10.3910400Z Configure a credential helper to remove this warning. See 2024-06-26T07:05:10.3911245Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:05:10.3911783Z 2024-06-26T07:05:10.3911932Z Login Succeeded 2024-06-26T07:05:10.5866224Z Command completed after 1 attempt(s). 2024-06-26T07:05:10.5908840Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:05:10.5909561Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:05:10.5910207Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:05:10.5917878Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:10.5918378Z env: 2024-06-26T07:05:10.5918659Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:10.5919010Z ##[endgroup] 2024-06-26T07:05:10.5982789Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:05:10.5983558Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:05:10.5984150Z # shellcheck disable=SC2046 2024-06-26T07:05:10.5984620Z docker stop $(docker ps -q) || true 2024-06-26T07:05:10.5985105Z # Prune all of the docker images 2024-06-26T07:05:10.5985568Z docker system prune -af 2024-06-26T07:05:10.5993436Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:10.5993934Z env: 2024-06-26T07:05:10.5994212Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:10.5994563Z ##[endgroup] 2024-06-26T07:05:10.6241445Z "docker stop" requires at least 1 argument. 2024-06-26T07:05:10.6242283Z See 'docker stop --help'. 2024-06-26T07:05:10.6242531Z 2024-06-26T07:05:10.6242761Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T07:05:10.6243143Z 2024-06-26T07:05:10.6243299Z Stop one or more running containers 2024-06-26T07:05:10.6392818Z Total reclaimed space: 0B 2024-06-26T07:05:10.6422251Z ##[group]Run set +e 2024-06-26T07:05:10.6422621Z set +e 2024-06-26T07:05:10.6422934Z set -x 2024-06-26T07:05:10.6423235Z  2024-06-26T07:05:10.6423561Z PT_DOMAIN=download.pytorch.org 2024-06-26T07:05:10.6424363Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-26T07:05:10.6425563Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-26T07:05:10.6426309Z # one is returned at random 2024-06-26T07:05:10.6427219Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-26T07:05:10.6427738Z  2024-06-26T07:05:10.6428073Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:05:10.6428750Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-26T07:05:10.6429509Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-26T07:05:10.6430065Z  2024-06-26T07:05:10.6430402Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:05:10.6430955Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-26T07:05:10.6431472Z  exit 1 2024-06-26T07:05:10.6431787Z  fi 2024-06-26T07:05:10.6432089Z fi 2024-06-26T07:05:10.6432506Z  2024-06-26T07:05:10.6432884Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-26T07:05:10.6433401Z  # Clean up any old records first 2024-06-26T07:05:10.6433912Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-26T07:05:10.6434371Z fi 2024-06-26T07:05:10.6434654Z  2024-06-26T07:05:10.6435090Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-26T07:05:10.6435635Z cat /etc/hosts 2024-06-26T07:05:10.6443310Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:10.6443811Z env: 2024-06-26T07:05:10.6444103Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:10.6444468Z ##[endgroup] 2024-06-26T07:05:10.6464120Z + PT_DOMAIN=download.pytorch.org 2024-06-26T07:05:10.6468309Z ++ dig -4 +short download.pytorch.org 2024-06-26T07:05:10.6468802Z ++ tail -n1 2024-06-26T07:05:10.6558527Z + RESOLVED_IP=18.160.10.76 2024-06-26T07:05:10.6558967Z + '[' -z 18.160.10.76 ']' 2024-06-26T07:05:10.6559448Z + grep -r download.pytorch.org /etc/hosts 2024-06-26T07:05:10.6564592Z 18.67.76.37 download.pytorch.org 2024-06-26T07:05:10.6565158Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-06-26T07:05:10.6661108Z + echo '18.160.10.76 download.pytorch.org' 2024-06-26T07:05:10.6661748Z + sudo tee -a /etc/hosts 2024-06-26T07:05:10.7143082Z 18.160.10.76 download.pytorch.org 2024-06-26T07:05:10.7156684Z + cat /etc/hosts 2024-06-26T07:05:10.7160543Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-26T07:05:10.7173614Z ::1 localhost6 localhost6.localdomain6 2024-06-26T07:05:10.7174131Z 18.160.10.76 download.pytorch.org 2024-06-26T07:05:10.7384577Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T07:05:10.7385183Z with: 2024-06-26T07:05:10.7386269Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:10.7387724Z docker-build-dir: .ci/docker 2024-06-26T07:05:10.7388109Z working-directory: . 2024-06-26T07:05:10.7388615Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:10.7389161Z force-push: false 2024-06-26T07:05:10.7389468Z env: 2024-06-26T07:05:10.7389750Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:10.7390099Z ##[endgroup] 2024-06-26T07:05:10.7407339Z ##[group]Run set -ex 2024-06-26T07:05:10.7407822Z set -ex 2024-06-26T07:05:10.7408127Z  2024-06-26T07:05:10.7408689Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T07:05:10.7409661Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T07:05:10.7410452Z # job could then download the pre-built image as usual 2024-06-26T07:05:10.7411193Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T07:05:10.7411986Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:10.7412626Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:10.7413191Z  2024-06-26T07:05:10.7413704Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T07:05:10.7414334Z  exit 0 2024-06-26T07:05:10.7414644Z else 2024-06-26T07:05:10.7415019Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:10.7415470Z fi 2024-06-26T07:05:10.7415756Z  2024-06-26T07:05:10.7416233Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T07:05:10.7417080Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T07:05:10.7417856Z  # use it as it is, but first let's extract the tag 2024-06-26T07:05:10.7418567Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T07:05:10.7419296Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:10.7419990Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:10.7420546Z else 2024-06-26T07:05:10.7420985Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T07:05:10.7421647Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:10.7422533Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:10.7423271Z fi 2024-06-26T07:05:10.7431789Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:10.7432287Z env: 2024-06-26T07:05:10.7432564Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:10.7432915Z REPO_NAME: pytorch 2024-06-26T07:05:10.7434028Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:10.7435225Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:05:10.7435729Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:10.7436247Z ##[endgroup] 2024-06-26T07:05:10.7456014Z + [[ ! -d .ci/docker ]] 2024-06-26T07:05:10.7456453Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T07:05:10.7456850Z + echo skip=false 2024-06-26T07:05:10.7458855Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T07:05:10.7461394Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:10.7462578Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T07:05:10.7468297Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:10.7469193Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:10.7470628Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:10.7497292Z ##[group]Run set +e 2024-06-26T07:05:10.7497679Z set +e 2024-06-26T07:05:10.7497987Z set -x 2024-06-26T07:05:10.7498283Z  2024-06-26T07:05:10.7498604Z login() { 2024-06-26T07:05:10.7499291Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:05:10.7500013Z } 2024-06-26T07:05:10.7500291Z  2024-06-26T07:05:10.7500579Z retry () { 2024-06-26T07:05:10.7500978Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:05:10.7501446Z } 2024-06-26T07:05:10.7501721Z  2024-06-26T07:05:10.7502163Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:05:10.7502592Z  2024-06-26T07:05:10.7503060Z # Check if image already exists, if it does then skip building it 2024-06-26T07:05:10.7503770Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T07:05:10.7504275Z  exit 0 2024-06-26T07:05:10.7504589Z fi 2024-06-26T07:05:10.7504868Z  2024-06-26T07:05:10.7505383Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T07:05:10.7506209Z # be empty. The default action would be to continue rebuild the image 2024-06-26T07:05:10.7507132Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T07:05:10.7507793Z  # if we're on the base branch then use the parent commit 2024-06-26T07:05:10.7508384Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T07:05:10.7508827Z else 2024-06-26T07:05:10.7509298Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T07:05:10.7509986Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T07:05:10.7510499Z fi 2024-06-26T07:05:10.7510785Z  2024-06-26T07:05:10.7511114Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T07:05:10.7511616Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:10.7512081Z  2024-06-26T07:05:10.7512733Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T07:05:10.7513489Z  exit 0 2024-06-26T07:05:10.7513796Z fi 2024-06-26T07:05:10.7514084Z  2024-06-26T07:05:10.7514540Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T07:05:10.7515510Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T07:05:10.7516301Z  exit 1 2024-06-26T07:05:10.7516619Z fi 2024-06-26T07:05:10.7516907Z  2024-06-26T07:05:10.7517400Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T07:05:10.7518344Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T07:05:10.7519214Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T07:05:10.7520169Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T07:05:10.7521258Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T07:05:10.7521985Z fi 2024-06-26T07:05:10.7522273Z  2024-06-26T07:05:10.7522646Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:10.7530442Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:10.7530957Z env: 2024-06-26T07:05:10.7531253Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:10.7531632Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:05:10.7532109Z BASE_REVISION: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:05:10.7533372Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:10.7534609Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:10.7535209Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:10.7535736Z ##[endgroup] 2024-06-26T07:05:10.7555126Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:10.7555790Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:10.7557467Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:05:10.7558244Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:11.1654210Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:05:11.1656221Z Configure a credential helper to remove this warning. See 2024-06-26T07:05:11.1657861Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:05:11.1658689Z 2024-06-26T07:05:11.1658819Z Login Succeeded 2024-06-26T07:05:11.1664609Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:11.3694613Z { 2024-06-26T07:05:11.3695026Z "schemaVersion": 2, 2024-06-26T07:05:11.3695833Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T07:05:11.3696547Z "config": { 2024-06-26T07:05:11.3697012Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T07:05:11.3697565Z "size": 48681, 2024-06-26T07:05:11.3698121Z "digest": "sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd" 2024-06-26T07:05:11.3698945Z }, 2024-06-26T07:05:11.3699307Z "layers": [ 2024-06-26T07:05:11.3699666Z { 2024-06-26T07:05:11.3700116Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3700691Z "size": 28580681, 2024-06-26T07:05:11.3701253Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-06-26T07:05:11.3701965Z }, 2024-06-26T07:05:11.3702236Z { 2024-06-26T07:05:11.3702675Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3703235Z "size": 7943451, 2024-06-26T07:05:11.3703905Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-06-26T07:05:11.3704606Z }, 2024-06-26T07:05:11.3704866Z { 2024-06-26T07:05:11.3705453Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3706200Z "size": 55728572, 2024-06-26T07:05:11.3708059Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-06-26T07:05:11.3708921Z }, 2024-06-26T07:05:11.3709258Z { 2024-06-26T07:05:11.3709819Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3710638Z "size": 186, 2024-06-26T07:05:11.3711180Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-06-26T07:05:11.3711986Z }, 2024-06-26T07:05:11.3712335Z { 2024-06-26T07:05:11.3712910Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3713713Z + exit 0 2024-06-26T07:05:11.3714000Z "size": 6886, 2024-06-26T07:05:11.3714547Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-06-26T07:05:11.3715390Z }, 2024-06-26T07:05:11.3715655Z { 2024-06-26T07:05:11.3716101Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3716668Z "size": 1329001756, 2024-06-26T07:05:11.3717246Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-06-26T07:05:11.3717882Z }, 2024-06-26T07:05:11.3718138Z { 2024-06-26T07:05:11.3718574Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3719120Z "size": 62501, 2024-06-26T07:05:11.3719666Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-06-26T07:05:11.3720278Z }, 2024-06-26T07:05:11.3720535Z { 2024-06-26T07:05:11.3720963Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3721518Z "size": 1684, 2024-06-26T07:05:11.3722056Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-06-26T07:05:11.3722656Z }, 2024-06-26T07:05:11.3722922Z { 2024-06-26T07:05:11.3723357Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3723910Z "size": 1523, 2024-06-26T07:05:11.3724459Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-06-26T07:05:11.3725086Z }, 2024-06-26T07:05:11.3725486Z { 2024-06-26T07:05:11.3726030Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3726581Z "size": 2528295403, 2024-06-26T07:05:11.3727149Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-06-26T07:05:11.3727767Z }, 2024-06-26T07:05:11.3728017Z { 2024-06-26T07:05:11.3728456Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3729059Z "size": 86016, 2024-06-26T07:05:11.3729610Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-06-26T07:05:11.3730223Z }, 2024-06-26T07:05:11.3730480Z { 2024-06-26T07:05:11.3730921Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3731476Z "size": 1823, 2024-06-26T07:05:11.3732003Z "digest": "sha256:309b872da805fcdf74137a887464876fa0615dc8b24199307119eb65fc2bfb49" 2024-06-26T07:05:11.3732608Z }, 2024-06-26T07:05:11.3732864Z { 2024-06-26T07:05:11.3733300Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3733857Z "size": 246737542, 2024-06-26T07:05:11.3734417Z "digest": "sha256:e82932cb002c06a782077b8e5965f8733eb8563406a23cab260d9d8c4a994f7d" 2024-06-26T07:05:11.3735029Z }, 2024-06-26T07:05:11.3735276Z { 2024-06-26T07:05:11.3735713Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3736270Z "size": 545, 2024-06-26T07:05:11.3736816Z "digest": "sha256:ca9dd73e3e9378225ebdcfd12199ef44649e34fe55003cfbc8afe2e8a4527c5b" 2024-06-26T07:05:11.3737428Z }, 2024-06-26T07:05:11.3737687Z { 2024-06-26T07:05:11.3738124Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3738680Z "size": 1283, 2024-06-26T07:05:11.3739205Z "digest": "sha256:16d76a79d5d6616e228537785ec83a869f68a402520bbc6e826201621ca6cb98" 2024-06-26T07:05:11.3739805Z }, 2024-06-26T07:05:11.3740061Z { 2024-06-26T07:05:11.3740491Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3741053Z "size": 485, 2024-06-26T07:05:11.3741604Z "digest": "sha256:4bdf41c9ecbe1651e0986ccbea4663a0556af27b7145a679cfa45c35dc96b891" 2024-06-26T07:05:11.3742231Z }, 2024-06-26T07:05:11.3742481Z { 2024-06-26T07:05:11.3742916Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3743475Z "size": 91710937, 2024-06-26T07:05:11.3744050Z "digest": "sha256:f4863801e30cdbcd1d4e6cb1cbbcde7dbcaf6ae317ca80429295741ce60ac31d" 2024-06-26T07:05:11.3744671Z }, 2024-06-26T07:05:11.3744927Z { 2024-06-26T07:05:11.3745364Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3745983Z "size": 3233, 2024-06-26T07:05:11.3746533Z "digest": "sha256:d37dd9ed8948a0d02e1198bbedadbca36477a0980dd553196385aab0c025581d" 2024-06-26T07:05:11.3747405Z }, 2024-06-26T07:05:11.3747665Z { 2024-06-26T07:05:11.3748093Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3748652Z "size": 1905, 2024-06-26T07:05:11.3749208Z "digest": "sha256:cf2f34f225dbed17c1a9a8b9ea4ad030aba797774a716dc0a82fe212324aa751" 2024-06-26T07:05:11.3749830Z }, 2024-06-26T07:05:11.3750079Z { 2024-06-26T07:05:11.3750515Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3751066Z "size": 702, 2024-06-26T07:05:11.3751613Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:05:11.3752225Z }, 2024-06-26T07:05:11.3752479Z { 2024-06-26T07:05:11.3752914Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3753456Z "size": 2784789002, 2024-06-26T07:05:11.3754026Z "digest": "sha256:3d914950f8addfc4d657a2365c76e1143eb80b710cd3f6e9ae05640217fc6347" 2024-06-26T07:05:11.3754643Z }, 2024-06-26T07:05:11.3754899Z { 2024-06-26T07:05:11.3755326Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3755994Z "size": 381, 2024-06-26T07:05:11.3756533Z "digest": "sha256:96d0e2032184c57e62a652aaffcf0241f0d0c7a23e894ca0ad192c4a64a36583" 2024-06-26T07:05:11.3757146Z }, 2024-06-26T07:05:11.3757395Z { 2024-06-26T07:05:11.3757834Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3758394Z "size": 11879, 2024-06-26T07:05:11.3758982Z "digest": "sha256:06e12ce59790cb8b091e5d619c2a62a28329008ee184a8b9a562e3db12471f30" 2024-06-26T07:05:11.3759577Z }, 2024-06-26T07:05:11.3759831Z { 2024-06-26T07:05:11.3760265Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3760811Z "size": 803, 2024-06-26T07:05:11.3761366Z "digest": "sha256:ed712f63cde7070bdec2a6179c4c35e752c6991ddbc69fe9f2724a9903250ceb" 2024-06-26T07:05:11.3761988Z }, 2024-06-26T07:05:11.3762244Z { 2024-06-26T07:05:11.3762671Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3763223Z "size": 106, 2024-06-26T07:05:11.3763768Z "digest": "sha256:a97a71ef8a4e66234dfc2098834f8395ac5630240e60164d1a251fdb0115ba85" 2024-06-26T07:05:11.3764377Z }, 2024-06-26T07:05:11.3764629Z { 2024-06-26T07:05:11.3765064Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3765614Z "size": 503, 2024-06-26T07:05:11.3766251Z "digest": "sha256:25cd0c482ad612f936b35b4f220c580e16be982226aac8801a6a849a6bfdc221" 2024-06-26T07:05:11.3766867Z }, 2024-06-26T07:05:11.3767126Z { 2024-06-26T07:05:11.3767559Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3768103Z "size": 121477177, 2024-06-26T07:05:11.3768675Z "digest": "sha256:9bf94bc456584365fda21d0ed83b7cc690b08a71727b71c4c1e667e15028a386" 2024-06-26T07:05:11.3769283Z }, 2024-06-26T07:05:11.3769539Z { 2024-06-26T07:05:11.3769965Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3770512Z "size": 109, 2024-06-26T07:05:11.3771052Z "digest": "sha256:de45a14c9b227e4b0c9d45d42223b50a71c03522ddd9e2ca0d9517835206c823" 2024-06-26T07:05:11.3771672Z }, 2024-06-26T07:05:11.3771922Z { 2024-06-26T07:05:11.3772355Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3772904Z "size": 491, 2024-06-26T07:05:11.3773430Z "digest": "sha256:e99241e77209b0b35a0fb59a61c0821fc161190178af7e4af45e6eeaa043375e" 2024-06-26T07:05:11.3774036Z }, 2024-06-26T07:05:11.3774295Z { 2024-06-26T07:05:11.3774727Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3775269Z "size": 291, 2024-06-26T07:05:11.3775818Z "digest": "sha256:35e8f4e2a42f9df0c969dedd2199cf71719cf5ecfa676dfa014bbdeb4fdb295f" 2024-06-26T07:05:11.3776531Z }, 2024-06-26T07:05:11.3776793Z { 2024-06-26T07:05:11.3777221Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3777775Z "size": 101, 2024-06-26T07:05:11.3778313Z "digest": "sha256:a823ffa4c541394b52953391dc0a6867e26c37fe4a71fa022a2cc3f07f0ba70d" 2024-06-26T07:05:11.3778929Z }, 2024-06-26T07:05:11.3779179Z { 2024-06-26T07:05:11.3779614Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3780164Z "size": 1473, 2024-06-26T07:05:11.3780681Z "digest": "sha256:44ba272eae2064603605a6238b3f35cc6578859d660068e702a047a9cd8ab2fd" 2024-06-26T07:05:11.3781282Z }, 2024-06-26T07:05:11.3781539Z { 2024-06-26T07:05:11.3781975Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3782521Z "size": 446710781, 2024-06-26T07:05:11.3783080Z "digest": "sha256:da02d278e5240c285bc855ce4e42cd6121d5b903787f5c14f03e575ad1268650" 2024-06-26T07:05:11.3783693Z }, 2024-06-26T07:05:11.3783960Z { 2024-06-26T07:05:11.3784388Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3784938Z "size": 161, 2024-06-26T07:05:11.3785462Z "digest": "sha256:73a0ce681c295135456e190578733273276b38b4dd5babbb39298f384d43b1c6" 2024-06-26T07:05:11.3786055Z }, 2024-06-26T07:05:11.3786383Z { 2024-06-26T07:05:11.3787004Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3787698Z "size": 564, 2024-06-26T07:05:11.3788317Z "digest": "sha256:49737bd5ae709dbf7956e21fc8b05ba009e9e6c9b130b0126a73bd3d3f5efc78" 2024-06-26T07:05:11.3788945Z }, 2024-06-26T07:05:11.3789204Z { 2024-06-26T07:05:11.3789641Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3790191Z "size": 35874600, 2024-06-26T07:05:11.3790752Z "digest": "sha256:9cd4c745f8e4f9d5c9bb8050923575afad21f5b7852975a4a1981bc9eff282c5" 2024-06-26T07:05:11.3791364Z }, 2024-06-26T07:05:11.3791612Z { 2024-06-26T07:05:11.3792060Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3792619Z "size": 104, 2024-06-26T07:05:11.3793150Z "digest": "sha256:b2950a0365345d4c17a48627c905fd8930b3b3154c727f0526eb9ca547102bb7" 2024-06-26T07:05:11.3793744Z }, 2024-06-26T07:05:11.3794005Z { 2024-06-26T07:05:11.3794451Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3795010Z "size": 425, 2024-06-26T07:05:11.3795536Z "digest": "sha256:25e19d81d9fc081962ff4cbc09e098f4035419822712c19a410f5282300dfb9e" 2024-06-26T07:05:11.3796152Z }, 2024-06-26T07:05:11.3796409Z { 2024-06-26T07:05:11.3796845Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3797390Z "size": 20262084, 2024-06-26T07:05:11.3797955Z "digest": "sha256:60156b40fcbec2c25f0170b0a82fa8cd9303e0daae8bb1811fd0fc6562e88e8e" 2024-06-26T07:05:11.3798578Z }, 2024-06-26T07:05:11.3798828Z { 2024-06-26T07:05:11.3799271Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3799832Z "size": 439, 2024-06-26T07:05:11.3800364Z "digest": "sha256:7a868239227fd19b30a19927bbc8f058d247403fe5b343c55f4b4b8987918911" 2024-06-26T07:05:11.3800958Z }, 2024-06-26T07:05:11.3801216Z { 2024-06-26T07:05:11.3801655Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3802215Z "size": 702, 2024-06-26T07:05:11.3802751Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:05:11.3803372Z }, 2024-06-26T07:05:11.3803629Z { 2024-06-26T07:05:11.3804057Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3804609Z "size": 143, 2024-06-26T07:05:11.3805148Z "digest": "sha256:83911d7c7ecf11b5932bb7449975c52e7f737a5b867691935e1ccb85ceb25b52" 2024-06-26T07:05:11.3805843Z }, 2024-06-26T07:05:11.3806093Z { 2024-06-26T07:05:11.3806529Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3807182Z "size": 135, 2024-06-26T07:05:11.3807735Z "digest": "sha256:b3c5d66c679f58c71febdd502889bd8cd3d36e5001754cdf1e30eabe21135cce" 2024-06-26T07:05:11.3808353Z }, 2024-06-26T07:05:11.3808610Z { 2024-06-26T07:05:11.3809049Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3809610Z "size": 4911495247, 2024-06-26T07:05:11.3810160Z "digest": "sha256:84b2c7a899820f6acdb22148c5e8346d8b26c33269fd01842c9573515251e1e1" 2024-06-26T07:05:11.3810764Z }, 2024-06-26T07:05:11.3811020Z { 2024-06-26T07:05:11.3811450Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3811998Z "size": 190, 2024-06-26T07:05:11.3812550Z "digest": "sha256:c1068d03a811d8dead5da9e82fdc8869205fc4ee9cb92d4f7a4acfbb5ba83950" 2024-06-26T07:05:11.3813173Z }, 2024-06-26T07:05:11.3813422Z { 2024-06-26T07:05:11.3813856Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3814406Z "size": 565, 2024-06-26T07:05:11.3814943Z "digest": "sha256:671b4a301d81074fca17d36851f81801552c4ecb2f90ff3412aa9ab6a1a91371" 2024-06-26T07:05:11.3815539Z }, 2024-06-26T07:05:11.3815796Z { 2024-06-26T07:05:11.3816231Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3816778Z "size": 43163095, 2024-06-26T07:05:11.3817418Z "digest": "sha256:67760cf5ec46ab58d3ca98637fd9b26cb47a67446bfd0db176fe5e023606aa0d" 2024-06-26T07:05:11.3818041Z }, 2024-06-26T07:05:11.3818300Z { 2024-06-26T07:05:11.3818728Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3819286Z "size": 106, 2024-06-26T07:05:11.3819832Z "digest": "sha256:ddc9740bcea2b301e99b50525059aea8d9c7121a1c8eab566561a8968438a8c2" 2024-06-26T07:05:11.3820442Z }, 2024-06-26T07:05:11.3820692Z { 2024-06-26T07:05:11.3821134Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3821687Z "size": 1213, 2024-06-26T07:05:11.3822231Z "digest": "sha256:fb2e977685609fcc01de170ba7f50a573e73e6520e47253717e92a28952c5ee8" 2024-06-26T07:05:11.3822829Z }, 2024-06-26T07:05:11.3823087Z { 2024-06-26T07:05:11.3823526Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3824098Z "size": 702, 2024-06-26T07:05:11.3824638Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:05:11.3825258Z }, 2024-06-26T07:05:11.3825515Z { 2024-06-26T07:05:11.3825941Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3826494Z "size": 138, 2024-06-26T07:05:11.3827232Z "digest": "sha256:b599c5a8c7cb461cbd7b3c98b99960b8eb04506e05e79db1cc0cfadb14a47e91" 2024-06-26T07:05:11.3827858Z }, 2024-06-26T07:05:11.3828112Z { 2024-06-26T07:05:11.3828550Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3829101Z "size": 120, 2024-06-26T07:05:11.3829654Z "digest": "sha256:e08e88c1366827455ee7ddbf92b6ae615ceb3cbd3fb11cd03ddd31bd3fa94a9c" 2024-06-26T07:05:11.3830267Z }, 2024-06-26T07:05:11.3830527Z { 2024-06-26T07:05:11.3830967Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3831515Z "size": 1904157495, 2024-06-26T07:05:11.3832092Z "digest": "sha256:12ba7431ff38de8cf6af3052a2a52e79f4e712b8c99f5bafde3c20e63b1f570f" 2024-06-26T07:05:11.3832724Z }, 2024-06-26T07:05:11.3832979Z { 2024-06-26T07:05:11.3833410Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3833962Z "size": 172, 2024-06-26T07:05:11.3834508Z "digest": "sha256:cf7c2cb896cba6a594028cc0fd246e127a2f4481d48905a3aa12c5f5872332d4" 2024-06-26T07:05:11.3835133Z }, 2024-06-26T07:05:11.3835385Z { 2024-06-26T07:05:11.3835821Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3836376Z "size": 908, 2024-06-26T07:05:11.3836925Z "digest": "sha256:3a778ddfac2af5b3a11f49ddf2f3f1ad1731f74ba691f434ab69966d1e2a189f" 2024-06-26T07:05:11.3837658Z }, 2024-06-26T07:05:11.3837923Z { 2024-06-26T07:05:11.3838361Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3838903Z "size": 702, 2024-06-26T07:05:11.3839448Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:05:11.3840068Z }, 2024-06-26T07:05:11.3840327Z { 2024-06-26T07:05:11.3840759Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3841313Z "size": 134, 2024-06-26T07:05:11.3841849Z "digest": "sha256:236c8f0245f4b40616a4322eaf1e1b3e178d5990db1fcaba39622ea4e79c06a1" 2024-06-26T07:05:11.3842462Z }, 2024-06-26T07:05:11.3842716Z { 2024-06-26T07:05:11.3843150Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3843701Z "size": 32, 2024-06-26T07:05:11.3844238Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:05:11.3844851Z }, 2024-06-26T07:05:11.3845116Z { 2024-06-26T07:05:11.3845558Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3846166Z "size": 156, 2024-06-26T07:05:11.3846717Z "digest": "sha256:dda1ce302b9a9df0c8f1145d2e3eaa03c96909bb45a792451f1fcb555b06738d" 2024-06-26T07:05:11.3847343Z }, 2024-06-26T07:05:11.3847683Z { 2024-06-26T07:05:11.3848112Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3848666Z "size": 1841, 2024-06-26T07:05:11.3849216Z "digest": "sha256:e797fe86caef19319e03f8dca92cf96838ab3bb3644f4447c6dfc04230118afa" 2024-06-26T07:05:11.3849827Z }, 2024-06-26T07:05:11.3850090Z { 2024-06-26T07:05:11.3850528Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3851082Z "size": 7529784, 2024-06-26T07:05:11.3851630Z "digest": "sha256:967190076fc0ab2bb2fc0d74e58edd3d662a537d702ff3f8b6e876230154a16f" 2024-06-26T07:05:11.3852243Z }, 2024-06-26T07:05:11.3852497Z { 2024-06-26T07:05:11.3852941Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3853489Z "size": 165, 2024-06-26T07:05:11.3854029Z "digest": "sha256:4fb0a83f06855534ef82b80dcf7c685303edb2bcbd446196066dc2ad2dd71281" 2024-06-26T07:05:11.3854649Z }, 2024-06-26T07:05:11.3854908Z { 2024-06-26T07:05:11.3855346Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3855898Z "size": 7944, 2024-06-26T07:05:11.3856449Z "digest": "sha256:68af2ec4e30a33f26947ece9f3a31b7d3b04c2bca5865d4b48618457832a1da2" 2024-06-26T07:05:11.3857056Z }, 2024-06-26T07:05:11.3857313Z { 2024-06-26T07:05:11.3857750Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3858305Z "size": 8070, 2024-06-26T07:05:11.3858842Z "digest": "sha256:cc91933e985a044c6df53caa15488d2285896f5e991f38ec068c253ceec53f14" 2024-06-26T07:05:11.3859452Z }, 2024-06-26T07:05:11.3859708Z { 2024-06-26T07:05:11.3860146Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3860688Z "size": 303, 2024-06-26T07:05:11.3861232Z "digest": "sha256:9f3187916e7dcad33cc572f8f0e949d2f507b887f99cfd6acefd14826b44960d" 2024-06-26T07:05:11.3861850Z }, 2024-06-26T07:05:11.3862103Z { 2024-06-26T07:05:11.3862541Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3863116Z "size": 7629071, 2024-06-26T07:05:11.3863654Z "digest": "sha256:52608399485738d10d779254946c9e1d536010dd45593c69bf1eb9f62f85dc24" 2024-06-26T07:05:11.3864242Z }, 2024-06-26T07:05:11.3864505Z { 2024-06-26T07:05:11.3864941Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3865488Z "size": 108, 2024-06-26T07:05:11.3866025Z "digest": "sha256:05a6f6dcc1c00bac6660923a22adc7d7c34416718f8c89daf9b3351d9365f694" 2024-06-26T07:05:11.3866642Z }, 2024-06-26T07:05:11.3867104Z { 2024-06-26T07:05:11.3867550Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3868204Z "size": 54145775, 2024-06-26T07:05:11.3868771Z "digest": "sha256:6bc9dc8315b09b27c976bfba9700767ba412d72fd0528b18bbe8f8e62005ec22" 2024-06-26T07:05:11.3869391Z }, 2024-06-26T07:05:11.3869642Z { 2024-06-26T07:05:11.3870076Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3870630Z "size": 474, 2024-06-26T07:05:11.3871175Z "digest": "sha256:1b0e27c35f07b3fbf0aeb548d1949bf1d60a420136da21789d94d196c891d981" 2024-06-26T07:05:11.3871782Z }, 2024-06-26T07:05:11.3872043Z { 2024-06-26T07:05:11.3872480Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3873032Z "size": 1374858747, 2024-06-26T07:05:11.3873583Z "digest": "sha256:95237e7311a1c5ce6937db7196c866632f44e8788514a5c0eb4a3f04da4017e6" 2024-06-26T07:05:11.3874186Z }, 2024-06-26T07:05:11.3874442Z { 2024-06-26T07:05:11.3874870Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3875424Z "size": 106, 2024-06-26T07:05:11.3875986Z "digest": "sha256:789eff6c51aada70bf75eb1acafed63c0d839b768638001500616ff6e53b854a" 2024-06-26T07:05:11.3876612Z }, 2024-06-26T07:05:11.3876863Z { 2024-06-26T07:05:11.3877299Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3877853Z "size": 558, 2024-06-26T07:05:11.3878478Z "digest": "sha256:de0f8a8c45c6077b3c5ba3b84d7f07de98353fd618dad9a1e2f368ffe00493c7" 2024-06-26T07:05:11.3879095Z }, 2024-06-26T07:05:11.3879352Z { 2024-06-26T07:05:11.3879789Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3880351Z "size": 46248513, 2024-06-26T07:05:11.3880895Z "digest": "sha256:2060028f74888d218bebdd10d723f66c33ee1c708935c4ffd4654d34912ac4dc" 2024-06-26T07:05:11.3881507Z }, 2024-06-26T07:05:11.3881766Z { 2024-06-26T07:05:11.3882199Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3882756Z "size": 111, 2024-06-26T07:05:11.3883298Z "digest": "sha256:e412c7f3b2ded440992955bd92512345cbece8a8894cd3592f714ebc758ad1a0" 2024-06-26T07:05:11.3883909Z }, 2024-06-26T07:05:11.3884162Z { 2024-06-26T07:05:11.3884602Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3885160Z "size": 32, 2024-06-26T07:05:11.3885775Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:05:11.3886391Z }, 2024-06-26T07:05:11.3886649Z { 2024-06-26T07:05:11.3887089Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3887638Z "size": 32, 2024-06-26T07:05:11.3888179Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:05:11.3888800Z }, 2024-06-26T07:05:11.3889057Z { 2024-06-26T07:05:11.3889484Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3890040Z "size": 32, 2024-06-26T07:05:11.3890589Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:05:11.3891209Z }, 2024-06-26T07:05:11.3891459Z { 2024-06-26T07:05:11.3891895Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:11.3892447Z "size": 32, 2024-06-26T07:05:11.3892987Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:05:11.3893602Z } 2024-06-26T07:05:11.3893857Z ] 2024-06-26T07:05:11.3894108Z } 2024-06-26T07:05:11.4011381Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:05:11.4011987Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:05:11.4012542Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-26T07:05:11.4020501Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:11.4021013Z env: 2024-06-26T07:05:11.4021306Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:11.4022468Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:11.4023639Z ##[endgroup] 2024-06-26T07:05:11.4046815Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:11.4093809Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T07:05:11.4094399Z with: 2024-06-26T07:05:11.4095599Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:11.4096911Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:11.4097435Z env: 2024-06-26T07:05:11.4097724Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:11.4098082Z ##[endgroup] 2024-06-26T07:05:11.4113483Z ##[group]Run set -x 2024-06-26T07:05:11.4113843Z set -x 2024-06-26T07:05:11.4114264Z set +e 2024-06-26T07:05:11.4114585Z  2024-06-26T07:05:11.4114869Z login() { 2024-06-26T07:05:11.4115534Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:05:11.4116253Z } 2024-06-26T07:05:11.4116543Z  2024-06-26T07:05:11.4116862Z retry () { 2024-06-26T07:05:11.4117262Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:05:11.4117832Z } 2024-06-26T07:05:11.4118121Z  2024-06-26T07:05:11.4118455Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:05:11.4118875Z  2024-06-26T07:05:11.4119161Z set -e 2024-06-26T07:05:11.4119642Z # ignore output since only exit code is used for conditional 2024-06-26T07:05:11.4120351Z # only pull docker image if it's not available locally 2024-06-26T07:05:11.4121113Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T07:05:11.4121818Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T07:05:11.4122269Z fi 2024-06-26T07:05:11.4130575Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:11.4131255Z env: 2024-06-26T07:05:11.4131639Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:11.4132779Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:11.4134070Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:11.4134588Z ##[endgroup] 2024-06-26T07:05:11.4154787Z + set +e 2024-06-26T07:05:11.4155415Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:11.4156056Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:11.4157009Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:05:11.4157788Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:11.8256790Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:05:11.8257600Z Configure a credential helper to remove this warning. See 2024-06-26T07:05:11.8258633Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:05:11.8259229Z 2024-06-26T07:05:11.8259369Z Login Succeeded 2024-06-26T07:05:11.8268134Z + set -e 2024-06-26T07:05:11.8269478Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:11.8390953Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:11.8393066Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:12.0595156Z 91382da70d5719cd7007b6b80b71d2f48398f6b7: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-06-26T07:05:12.0599089Z 7a2c55901189: Pulling fs layer 2024-06-26T07:05:12.0599679Z 224fe954d725: Pulling fs layer 2024-06-26T07:05:12.0600583Z 75722010b82e: Pulling fs layer 2024-06-26T07:05:12.0601101Z d527cbbb87e3: Pulling fs layer 2024-06-26T07:05:12.0601675Z b57676e46aee: Pulling fs layer 2024-06-26T07:05:12.0602218Z a8c1e85b5e14: Pulling fs layer 2024-06-26T07:05:12.0602755Z a41a8d1c11c8: Pulling fs layer 2024-06-26T07:05:12.0603431Z 0c1227890755: Pulling fs layer 2024-06-26T07:05:12.0603984Z d8d1234baab3: Pulling fs layer 2024-06-26T07:05:12.0604526Z 7ed32bc8e469: Pulling fs layer 2024-06-26T07:05:12.0605061Z ec1e7978c1fe: Pulling fs layer 2024-06-26T07:05:12.0605609Z 309b872da805: Pulling fs layer 2024-06-26T07:05:12.0606246Z e82932cb002c: Pulling fs layer 2024-06-26T07:05:12.0606803Z ca9dd73e3e93: Pulling fs layer 2024-06-26T07:05:12.0607344Z 16d76a79d5d6: Pulling fs layer 2024-06-26T07:05:12.0607818Z 4bdf41c9ecbe: Pulling fs layer 2024-06-26T07:05:12.0608204Z f4863801e30c: Pulling fs layer 2024-06-26T07:05:12.0608575Z b57676e46aee: Waiting 2024-06-26T07:05:12.0608993Z d37dd9ed8948: Pulling fs layer 2024-06-26T07:05:12.0609409Z cf2f34f225db: Pulling fs layer 2024-06-26T07:05:12.0609852Z df546d7b3e34: Pulling fs layer 2024-06-26T07:05:12.0610413Z a8c1e85b5e14: Waiting 2024-06-26T07:05:12.0610766Z 3d914950f8ad: Pulling fs layer 2024-06-26T07:05:12.0611139Z 96d0e2032184: Pulling fs layer 2024-06-26T07:05:12.0611519Z 06e12ce59790: Pulling fs layer 2024-06-26T07:05:12.0611887Z a41a8d1c11c8: Waiting 2024-06-26T07:05:12.0612254Z 0c1227890755: Waiting 2024-06-26T07:05:12.0612580Z d8d1234baab3: Waiting 2024-06-26T07:05:12.0612897Z 7ed32bc8e469: Waiting 2024-06-26T07:05:12.0613217Z e82932cb002c: Waiting 2024-06-26T07:05:12.0613558Z ed712f63cde7: Pulling fs layer 2024-06-26T07:05:12.0613922Z ec1e7978c1fe: Waiting 2024-06-26T07:05:12.0614263Z a97a71ef8a4e: Pulling fs layer 2024-06-26T07:05:12.0614633Z ca9dd73e3e93: Waiting 2024-06-26T07:05:12.0614956Z 309b872da805: Waiting 2024-06-26T07:05:12.0615287Z 25cd0c482ad6: Pulling fs layer 2024-06-26T07:05:12.0615671Z 9bf94bc45658: Pulling fs layer 2024-06-26T07:05:12.0616037Z 16d76a79d5d6: Waiting 2024-06-26T07:05:12.0616372Z de45a14c9b22: Pulling fs layer 2024-06-26T07:05:12.0616752Z 4bdf41c9ecbe: Waiting 2024-06-26T07:05:12.0617094Z e99241e77209: Pulling fs layer 2024-06-26T07:05:12.0617485Z 35e8f4e2a42f: Pulling fs layer 2024-06-26T07:05:12.0617866Z a823ffa4c541: Pulling fs layer 2024-06-26T07:05:12.0618249Z 44ba272eae20: Pulling fs layer 2024-06-26T07:05:12.0618629Z da02d278e524: Pulling fs layer 2024-06-26T07:05:12.0619009Z 73a0ce681c29: Pulling fs layer 2024-06-26T07:05:12.0619381Z 49737bd5ae70: Pulling fs layer 2024-06-26T07:05:12.0619760Z 9cd4c745f8e4: Pulling fs layer 2024-06-26T07:05:12.0620137Z b2950a036534: Pulling fs layer 2024-06-26T07:05:12.0620492Z d37dd9ed8948: Waiting 2024-06-26T07:05:12.0620817Z 96d0e2032184: Waiting 2024-06-26T07:05:12.0621152Z 25e19d81d9fc: Pulling fs layer 2024-06-26T07:05:12.0621519Z cf2f34f225db: Waiting 2024-06-26T07:05:12.0621854Z 60156b40fcbe: Pulling fs layer 2024-06-26T07:05:12.0622231Z 7a868239227f: Pulling fs layer 2024-06-26T07:05:12.0622597Z 25cd0c482ad6: Waiting 2024-06-26T07:05:12.0622922Z a97a71ef8a4e: Waiting 2024-06-26T07:05:12.0623342Z 06e12ce59790: Waiting 2024-06-26T07:05:12.0623867Z 9bf94bc45658: Waiting 2024-06-26T07:05:12.0624317Z ed712f63cde7: Waiting 2024-06-26T07:05:12.0624797Z 83911d7c7ecf: Pulling fs layer 2024-06-26T07:05:12.0625428Z b3c5d66c679f: Pulling fs layer 2024-06-26T07:05:12.0625811Z a823ffa4c541: Waiting 2024-06-26T07:05:12.0626137Z de45a14c9b22: Waiting 2024-06-26T07:05:12.0626459Z 60156b40fcbe: Waiting 2024-06-26T07:05:12.0627101Z 7a868239227f: Waiting 2024-06-26T07:05:12.0627452Z 84b2c7a89982: Pulling fs layer 2024-06-26T07:05:12.0627829Z c1068d03a811: Pulling fs layer 2024-06-26T07:05:12.0628197Z 73a0ce681c29: Waiting 2024-06-26T07:05:12.0628534Z 671b4a301d81: Pulling fs layer 2024-06-26T07:05:12.0628930Z 49737bd5ae70: Waiting 2024-06-26T07:05:12.0629262Z b2950a036534: Waiting 2024-06-26T07:05:12.0629598Z 67760cf5ec46: Pulling fs layer 2024-06-26T07:05:12.0630007Z e99241e77209: Waiting 2024-06-26T07:05:12.0630514Z ddc9740bcea2: Pulling fs layer 2024-06-26T07:05:12.0630884Z f4863801e30c: Waiting 2024-06-26T07:05:12.0631211Z b3c5d66c679f: Waiting 2024-06-26T07:05:12.0631533Z 25e19d81d9fc: Waiting 2024-06-26T07:05:12.0631866Z fb2e97768560: Pulling fs layer 2024-06-26T07:05:12.0632228Z 67760cf5ec46: Waiting 2024-06-26T07:05:12.0632546Z ddc9740bcea2: Waiting 2024-06-26T07:05:12.0632871Z 44ba272eae20: Waiting 2024-06-26T07:05:12.0633202Z b599c5a8c7cb: Pulling fs layer 2024-06-26T07:05:12.0633584Z e08e88c13668: Pulling fs layer 2024-06-26T07:05:12.0633969Z 12ba7431ff38: Pulling fs layer 2024-06-26T07:05:12.0634339Z 9cd4c745f8e4: Waiting 2024-06-26T07:05:12.0634676Z cf7c2cb896cb: Pulling fs layer 2024-06-26T07:05:12.0635060Z 3a778ddfac2a: Pulling fs layer 2024-06-26T07:05:12.0635430Z 83911d7c7ecf: Waiting 2024-06-26T07:05:12.0635743Z 12ba7431ff38: Waiting 2024-06-26T07:05:12.0636061Z e08e88c13668: Waiting 2024-06-26T07:05:12.0636393Z 236c8f0245f4: Pulling fs layer 2024-06-26T07:05:12.0636775Z 4f4fb700ef54: Pulling fs layer 2024-06-26T07:05:12.0637142Z 236c8f0245f4: Waiting 2024-06-26T07:05:12.0637483Z dda1ce302b9a: Pulling fs layer 2024-06-26T07:05:12.0637927Z 84b2c7a89982: Waiting 2024-06-26T07:05:12.0638330Z e797fe86caef: Pulling fs layer 2024-06-26T07:05:12.0638707Z 4f4fb700ef54: Waiting 2024-06-26T07:05:12.0639048Z 967190076fc0: Pulling fs layer 2024-06-26T07:05:12.0639426Z 4fb0a83f0685: Pulling fs layer 2024-06-26T07:05:12.0639789Z dda1ce302b9a: Waiting 2024-06-26T07:05:12.0640135Z 68af2ec4e30a: Pulling fs layer 2024-06-26T07:05:12.0640503Z e797fe86caef: Waiting 2024-06-26T07:05:12.0640994Z 967190076fc0: Waiting 2024-06-26T07:05:12.0641547Z cc91933e985a: Pulling fs layer 2024-06-26T07:05:12.0642144Z c1068d03a811: Waiting 2024-06-26T07:05:12.0642634Z 9f3187916e7d: Pulling fs layer 2024-06-26T07:05:12.0643033Z 526083994857: Pulling fs layer 2024-06-26T07:05:12.0643400Z 68af2ec4e30a: Waiting 2024-06-26T07:05:12.0643741Z 05a6f6dcc1c0: Pulling fs layer 2024-06-26T07:05:12.0644110Z cc91933e985a: Waiting 2024-06-26T07:05:12.0644454Z 6bc9dc8315b0: Pulling fs layer 2024-06-26T07:05:12.0644836Z 1b0e27c35f07: Pulling fs layer 2024-06-26T07:05:12.0645215Z 9f3187916e7d: Waiting 2024-06-26T07:05:12.0645536Z 05a6f6dcc1c0: Waiting 2024-06-26T07:05:12.0645984Z 95237e7311a1: Pulling fs layer 2024-06-26T07:05:12.0646353Z 6bc9dc8315b0: Waiting 2024-06-26T07:05:12.0646697Z 789eff6c51aa: Pulling fs layer 2024-06-26T07:05:12.0647058Z 1b0e27c35f07: Waiting 2024-06-26T07:05:12.0647398Z de0f8a8c45c6: Pulling fs layer 2024-06-26T07:05:12.0647779Z 2060028f7488: Pulling fs layer 2024-06-26T07:05:12.0648158Z e412c7f3b2de: Pulling fs layer 2024-06-26T07:05:12.0648520Z de0f8a8c45c6: Waiting 2024-06-26T07:05:12.0648845Z 671b4a301d81: Waiting 2024-06-26T07:05:12.0649175Z e412c7f3b2de: Waiting 2024-06-26T07:05:12.0649495Z 789eff6c51aa: Waiting 2024-06-26T07:05:12.0649822Z d527cbbb87e3: Waiting 2024-06-26T07:05:12.0650148Z 4fb0a83f0685: Waiting 2024-06-26T07:05:12.1959245Z 224fe954d725: Verifying Checksum 2024-06-26T07:05:12.1959683Z 224fe954d725: Download complete 2024-06-26T07:05:12.3081516Z d527cbbb87e3: Verifying Checksum 2024-06-26T07:05:12.3082126Z d527cbbb87e3: Download complete 2024-06-26T07:05:12.3992875Z b57676e46aee: Verifying Checksum 2024-06-26T07:05:12.4086939Z 7a2c55901189: Verifying Checksum 2024-06-26T07:05:12.4087435Z 7a2c55901189: Download complete 2024-06-26T07:05:12.5284045Z a41a8d1c11c8: Download complete 2024-06-26T07:05:12.6268700Z 0c1227890755: Verifying Checksum 2024-06-26T07:05:12.6269305Z 0c1227890755: Download complete 2024-06-26T07:05:12.6714292Z 75722010b82e: Verifying Checksum 2024-06-26T07:05:12.6714945Z 75722010b82e: Download complete 2024-06-26T07:05:12.7058730Z d8d1234baab3: Download complete 2024-06-26T07:05:12.8239621Z ec1e7978c1fe: Download complete 2024-06-26T07:05:12.8988632Z 309b872da805: Verifying Checksum 2024-06-26T07:05:12.8989221Z 309b872da805: Download complete 2024-06-26T07:05:13.4237524Z 7a2c55901189: Pull complete 2024-06-26T07:05:13.7118462Z 224fe954d725: Pull complete 2024-06-26T07:05:14.6488934Z 75722010b82e: Pull complete 2024-06-26T07:05:14.6907211Z d527cbbb87e3: Pull complete 2024-06-26T07:05:14.7495907Z b57676e46aee: Pull complete 2024-06-26T07:05:15.4219389Z e82932cb002c: Verifying Checksum 2024-06-26T07:05:15.4220052Z e82932cb002c: Download complete 2024-06-26T07:05:15.4904085Z ca9dd73e3e93: Verifying Checksum 2024-06-26T07:05:15.4904546Z ca9dd73e3e93: Download complete 2024-06-26T07:05:15.5625050Z 16d76a79d5d6: Verifying Checksum 2024-06-26T07:05:15.5625519Z 16d76a79d5d6: Download complete 2024-06-26T07:05:15.6549676Z 4bdf41c9ecbe: Verifying Checksum 2024-06-26T07:05:15.6550148Z 4bdf41c9ecbe: Download complete 2024-06-26T07:05:16.6329042Z f4863801e30c: Verifying Checksum 2024-06-26T07:05:16.6329592Z f4863801e30c: Download complete 2024-06-26T07:05:16.7000983Z d37dd9ed8948: Download complete 2024-06-26T07:05:16.7744966Z cf2f34f225db: Verifying Checksum 2024-06-26T07:05:16.7745433Z cf2f34f225db: Download complete 2024-06-26T07:05:16.8909166Z df546d7b3e34: Verifying Checksum 2024-06-26T07:05:16.8909726Z df546d7b3e34: Download complete 2024-06-26T07:05:25.7491558Z a8c1e85b5e14: Verifying Checksum 2024-06-26T07:05:25.7492242Z a8c1e85b5e14: Download complete 2024-06-26T07:05:25.8337481Z 96d0e2032184: Verifying Checksum 2024-06-26T07:05:25.8338109Z 96d0e2032184: Download complete 2024-06-26T07:05:25.9414516Z 06e12ce59790: Verifying Checksum 2024-06-26T07:05:25.9415109Z 06e12ce59790: Download complete 2024-06-26T07:05:26.0188439Z ed712f63cde7: Download complete 2024-06-26T07:05:26.0877209Z a97a71ef8a4e: Verifying Checksum 2024-06-26T07:05:26.0877848Z a97a71ef8a4e: Download complete 2024-06-26T07:05:26.1569407Z 25cd0c482ad6: Verifying Checksum 2024-06-26T07:05:26.1570016Z 25cd0c482ad6: Download complete 2024-06-26T07:05:27.4265171Z 9bf94bc45658: Verifying Checksum 2024-06-26T07:05:27.4265804Z 9bf94bc45658: Download complete 2024-06-26T07:05:27.5057388Z de45a14c9b22: Verifying Checksum 2024-06-26T07:05:27.5057880Z de45a14c9b22: Download complete 2024-06-26T07:05:27.5864331Z e99241e77209: Verifying Checksum 2024-06-26T07:05:27.5865327Z e99241e77209: Download complete 2024-06-26T07:05:27.6708787Z 35e8f4e2a42f: Verifying Checksum 2024-06-26T07:05:27.6709348Z 35e8f4e2a42f: Download complete 2024-06-26T07:05:27.7543689Z a823ffa4c541: Download complete 2024-06-26T07:05:27.9052774Z 44ba272eae20: Verifying Checksum 2024-06-26T07:05:27.9053368Z 44ba272eae20: Download complete 2024-06-26T07:05:32.4472591Z da02d278e524: Verifying Checksum 2024-06-26T07:05:32.4473211Z da02d278e524: Download complete 2024-06-26T07:05:32.5403175Z 73a0ce681c29: Verifying Checksum 2024-06-26T07:05:32.5403655Z 73a0ce681c29: Download complete 2024-06-26T07:05:32.6172651Z 49737bd5ae70: Verifying Checksum 2024-06-26T07:05:32.6173359Z 49737bd5ae70: Download complete 2024-06-26T07:05:33.0307108Z 9cd4c745f8e4: Verifying Checksum 2024-06-26T07:05:33.0307823Z 9cd4c745f8e4: Download complete 2024-06-26T07:05:33.1749203Z 25e19d81d9fc: Download complete 2024-06-26T07:05:33.4351919Z 60156b40fcbe: Verifying Checksum 2024-06-26T07:05:33.4352481Z 60156b40fcbe: Download complete 2024-06-26T07:05:33.5120385Z 7a868239227f: Verifying Checksum 2024-06-26T07:05:33.5120958Z 7a868239227f: Download complete 2024-06-26T07:05:33.5855439Z 83911d7c7ecf: Download complete 2024-06-26T07:05:33.6582648Z b3c5d66c679f: Verifying Checksum 2024-06-26T07:05:33.6583283Z b3c5d66c679f: Download complete 2024-06-26T07:05:38.0065168Z 7ed32bc8e469: Verifying Checksum 2024-06-26T07:05:38.0065670Z 7ed32bc8e469: Download complete 2024-06-26T07:05:38.0853106Z c1068d03a811: Verifying Checksum 2024-06-26T07:05:38.0853643Z c1068d03a811: Download complete 2024-06-26T07:05:38.1784213Z 671b4a301d81: Verifying Checksum 2024-06-26T07:05:38.1784653Z 671b4a301d81: Download complete 2024-06-26T07:05:38.5351062Z a8c1e85b5e14: Pull complete 2024-06-26T07:05:38.7280067Z 67760cf5ec46: Verifying Checksum 2024-06-26T07:05:38.7280501Z 67760cf5ec46: Download complete 2024-06-26T07:05:38.7651653Z a41a8d1c11c8: Pull complete 2024-06-26T07:05:38.8109772Z ddc9740bcea2: Download complete 2024-06-26T07:05:38.9070579Z fb2e97768560: Verifying Checksum 2024-06-26T07:05:38.9071680Z fb2e97768560: Download complete 2024-06-26T07:05:38.9738012Z 0c1227890755: Pull complete 2024-06-26T07:05:39.0057211Z b599c5a8c7cb: Download complete 2024-06-26T07:05:39.0813315Z e08e88c13668: Verifying Checksum 2024-06-26T07:05:39.1790667Z d8d1234baab3: Pull complete 2024-06-26T07:05:44.7853835Z 3d914950f8ad: Verifying Checksum 2024-06-26T07:05:44.7854313Z 3d914950f8ad: Download complete 2024-06-26T07:05:44.8722511Z cf7c2cb896cb: Verifying Checksum 2024-06-26T07:05:44.8723016Z cf7c2cb896cb: Download complete 2024-06-26T07:05:44.9394732Z 3a778ddfac2a: Verifying Checksum 2024-06-26T07:05:44.9395294Z 3a778ddfac2a: Download complete 2024-06-26T07:05:45.0259140Z 236c8f0245f4: Verifying Checksum 2024-06-26T07:05:45.0259616Z 236c8f0245f4: Download complete 2024-06-26T07:05:45.0344134Z 4f4fb700ef54: Download complete 2024-06-26T07:05:45.1071396Z dda1ce302b9a: Verifying Checksum 2024-06-26T07:05:45.1813813Z dda1ce302b9a: Download complete 2024-06-26T07:05:45.1814266Z e797fe86caef: Download complete 2024-06-26T07:05:45.3152113Z 967190076fc0: Verifying Checksum 2024-06-26T07:05:45.3152934Z 967190076fc0: Download complete 2024-06-26T07:05:45.3978638Z 4fb0a83f0685: Verifying Checksum 2024-06-26T07:05:45.3979072Z 4fb0a83f0685: Download complete 2024-06-26T07:05:45.4884040Z 68af2ec4e30a: Verifying Checksum 2024-06-26T07:05:45.4884472Z 68af2ec4e30a: Download complete 2024-06-26T07:05:45.5696361Z cc91933e985a: Download complete 2024-06-26T07:05:45.6567360Z 9f3187916e7d: Verifying Checksum 2024-06-26T07:05:45.6567849Z 9f3187916e7d: Download complete 2024-06-26T07:05:45.7882995Z 526083994857: Verifying Checksum 2024-06-26T07:05:45.7883553Z 526083994857: Download complete 2024-06-26T07:05:45.8788196Z 05a6f6dcc1c0: Verifying Checksum 2024-06-26T07:05:45.8788715Z 05a6f6dcc1c0: Download complete 2024-06-26T07:05:46.4793219Z 6bc9dc8315b0: Verifying Checksum 2024-06-26T07:05:46.4793796Z 6bc9dc8315b0: Download complete 2024-06-26T07:05:46.5494124Z 1b0e27c35f07: Verifying Checksum 2024-06-26T07:05:46.5494590Z 1b0e27c35f07: Download complete 2024-06-26T07:06:09.6111154Z 12ba7431ff38: Verifying Checksum 2024-06-26T07:06:09.6111648Z 12ba7431ff38: Download complete 2024-06-26T07:06:09.6932250Z 789eff6c51aa: Verifying Checksum 2024-06-26T07:06:09.6932785Z 789eff6c51aa: Download complete 2024-06-26T07:06:09.7725356Z de0f8a8c45c6: Verifying Checksum 2024-06-26T07:06:09.7726423Z de0f8a8c45c6: Download complete 2024-06-26T07:06:11.2524515Z 2060028f7488: Verifying Checksum 2024-06-26T07:06:11.2525133Z 2060028f7488: Download complete 2024-06-26T07:06:11.3313785Z e412c7f3b2de: Verifying Checksum 2024-06-26T07:06:11.3314261Z e412c7f3b2de: Download complete 2024-06-26T07:06:13.5859027Z 95237e7311a1: Verifying Checksum 2024-06-26T07:06:13.5859577Z 95237e7311a1: Download complete 2024-06-26T07:06:37.4992378Z 84b2c7a89982: Verifying Checksum 2024-06-26T07:06:37.4993470Z 84b2c7a89982: Download complete 2024-06-26T07:06:54.5258327Z 7ed32bc8e469: Pull complete 2024-06-26T07:06:54.7205752Z ec1e7978c1fe: Pull complete 2024-06-26T07:06:54.9167146Z 309b872da805: Pull complete 2024-06-26T07:07:02.2495679Z e82932cb002c: Pull complete 2024-06-26T07:07:02.4091037Z ca9dd73e3e93: Pull complete 2024-06-26T07:07:02.6417134Z 16d76a79d5d6: Pull complete 2024-06-26T07:07:02.8588532Z 4bdf41c9ecbe: Pull complete 2024-06-26T07:07:05.2517512Z f4863801e30c: Pull complete 2024-06-26T07:07:05.4643445Z d37dd9ed8948: Pull complete 2024-06-26T07:07:05.6667402Z cf2f34f225db: Pull complete 2024-06-26T07:07:05.8675991Z df546d7b3e34: Pull complete 2024-06-26T07:07:55.8189844Z 3d914950f8ad: Pull complete 2024-06-26T07:07:55.9770774Z 96d0e2032184: Pull complete 2024-06-26T07:07:56.1171594Z 06e12ce59790: Pull complete 2024-06-26T07:07:56.1920312Z ed712f63cde7: Pull complete 2024-06-26T07:07:56.3156312Z a97a71ef8a4e: Pull complete 2024-06-26T07:07:56.4162428Z 25cd0c482ad6: Pull complete 2024-06-26T07:07:59.9105631Z 9bf94bc45658: Pull complete 2024-06-26T07:08:00.1031187Z de45a14c9b22: Pull complete 2024-06-26T07:08:00.3328188Z e99241e77209: Pull complete 2024-06-26T07:08:00.5180989Z 35e8f4e2a42f: Pull complete 2024-06-26T07:08:00.7269378Z a823ffa4c541: Pull complete 2024-06-26T07:08:00.9362373Z 44ba272eae20: Pull complete 2024-06-26T07:08:09.2427430Z da02d278e524: Pull complete 2024-06-26T07:08:09.4642513Z 73a0ce681c29: Pull complete 2024-06-26T07:08:09.6555055Z 49737bd5ae70: Pull complete 2024-06-26T07:08:10.5839695Z 9cd4c745f8e4: Pull complete 2024-06-26T07:08:10.7199743Z b2950a036534: Pull complete 2024-06-26T07:08:10.9154693Z 25e19d81d9fc: Pull complete 2024-06-26T07:08:11.4137430Z 60156b40fcbe: Pull complete 2024-06-26T07:08:11.6050137Z 7a868239227f: Pull complete 2024-06-26T07:08:11.9898645Z 83911d7c7ecf: Pull complete 2024-06-26T07:08:12.1738103Z b3c5d66c679f: Pull complete 2024-06-26T07:08:51.2306570Z 84b2c7a89982: Pull complete 2024-06-26T07:08:51.4351582Z c1068d03a811: Pull complete 2024-06-26T07:08:51.6470184Z 671b4a301d81: Pull complete 2024-06-26T07:08:53.6689340Z 67760cf5ec46: Pull complete 2024-06-26T07:08:53.6983855Z ddc9740bcea2: Pull complete 2024-06-26T07:08:53.7367428Z fb2e97768560: Pull complete 2024-06-26T07:08:53.8448792Z b599c5a8c7cb: Pull complete 2024-06-26T07:08:53.8879919Z e08e88c13668: Pull complete 2024-06-26T07:09:27.7474145Z 12ba7431ff38: Pull complete 2024-06-26T07:09:27.9472971Z cf7c2cb896cb: Pull complete 2024-06-26T07:09:28.0609788Z 3a778ddfac2a: Pull complete 2024-06-26T07:09:28.2686546Z 236c8f0245f4: Pull complete 2024-06-26T07:09:28.4716263Z 4f4fb700ef54: Pull complete 2024-06-26T07:09:28.6976600Z dda1ce302b9a: Pull complete 2024-06-26T07:09:28.9073766Z e797fe86caef: Pull complete 2024-06-26T07:09:29.2701018Z 967190076fc0: Pull complete 2024-06-26T07:09:29.4750270Z 4fb0a83f0685: Pull complete 2024-06-26T07:09:29.6708116Z 68af2ec4e30a: Pull complete 2024-06-26T07:09:29.8953102Z cc91933e985a: Pull complete 2024-06-26T07:09:30.1013562Z 9f3187916e7d: Pull complete 2024-06-26T07:09:31.1921231Z 526083994857: Pull complete 2024-06-26T07:09:31.3781259Z 05a6f6dcc1c0: Pull complete 2024-06-26T07:09:33.3698707Z 6bc9dc8315b0: Pull complete 2024-06-26T07:09:33.5407639Z 1b0e27c35f07: Pull complete 2024-06-26T07:09:47.0471316Z 95237e7311a1: Pull complete 2024-06-26T07:09:47.1007980Z 789eff6c51aa: Pull complete 2024-06-26T07:09:47.1360095Z de0f8a8c45c6: Pull complete 2024-06-26T07:09:47.8515495Z 2060028f7488: Pull complete 2024-06-26T07:09:47.9175991Z e412c7f3b2de: Pull complete 2024-06-26T07:09:48.3622742Z Digest: sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:09:48.3745949Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:09:48.3827798Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:09:48.3869339Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:09:48.3870255Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:09:48.3878583Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:48.3879081Z env: 2024-06-26T07:09:48.3879356Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:48.3879712Z ##[endgroup] 2024-06-26T07:09:48.4034787Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-26T07:09:48.4035338Z with: 2024-06-26T07:09:48.4035636Z driver-version: 550.54.15 2024-06-26T07:09:48.4035990Z env: 2024-06-26T07:09:48.4036275Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:48.4036624Z ##[endgroup] 2024-06-26T07:09:48.4076796Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:09:48.4077362Z with: 2024-06-26T07:09:48.4077655Z timeout_minutes: 10 2024-06-26T07:09:48.4077987Z max_attempts: 3 2024-06-26T07:09:48.4109518Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-26T07:09:48.4140594Z retry_wait_seconds: 10 2024-06-26T07:09:48.4140970Z polling_interval_seconds: 1 2024-06-26T07:09:48.4141414Z warning_on_retry: true 2024-06-26T07:09:48.4141780Z continue_on_error: false 2024-06-26T07:09:48.4142127Z env: 2024-06-26T07:09:48.4142416Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:48.4142780Z DRIVER_VERSION: 550.54.15 2024-06-26T07:09:48.4143144Z ##[endgroup] 2024-06-26T07:09:48.4668101Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-26T07:09:48.4668995Z + pre_install_nvidia_driver_amzn2 2024-06-26T07:09:48.4669513Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-26T07:09:48.7819968Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:09:48.8204475Z No Match for argument: nvidia-driver-latest-dkms 2024-06-26T07:09:48.8439032Z No Packages marked for removal 2024-06-26T07:09:48.8555739Z + install_nvidia_driver_common 2024-06-26T07:09:48.8558534Z + echo 'Before installing NVIDIA driver' 2024-06-26T07:09:48.8559148Z + lspci 2024-06-26T07:09:48.8559504Z Before installing NVIDIA driver 2024-06-26T07:09:48.8636805Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:09:48.8637573Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:09:48.8638514Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:09:48.8639305Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:09:48.8640068Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:09:48.8640838Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:09:48.8641568Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:09:48.8642353Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:09:48.8642948Z + lsmod 2024-06-26T07:09:48.8651433Z Module Size Used by 2024-06-26T07:09:48.8651858Z xt_nat 16384 0 2024-06-26T07:09:48.8652228Z ib_core 266240 0 2024-06-26T07:09:48.8652623Z nvidia_modeset 1339392 0 2024-06-26T07:09:48.8653014Z veth 16384 0 2024-06-26T07:09:48.8653398Z nvidia_uvm 4599808 0 2024-06-26T07:09:48.8654029Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:09:48.8654532Z drm 421888 1 nvidia 2024-06-26T07:09:48.8654965Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:09:48.8655443Z backlight 16384 1 nvidia_modeset 2024-06-26T07:09:48.8655890Z xt_conntrack 16384 1 2024-06-26T07:09:48.8656284Z ipt_MASQUERADE 16384 1 2024-06-26T07:09:48.8656729Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:09:48.8657209Z nf_conntrack_netlink 49152 0 2024-06-26T07:09:48.8657654Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:09:48.8658128Z xfrm_user 45056 1 2024-06-26T07:09:48.8658531Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:09:48.8658945Z iptable_nat 16384 1 2024-06-26T07:09:48.8659339Z nf_conntrack_ipv4 16384 3 2024-06-26T07:09:48.8659779Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:09:48.8660292Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:09:48.8660860Z nf_nat 32768 3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat 2024-06-26T07:09:48.8661854Z nf_conntrack 155648 8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink 2024-06-26T07:09:48.8662719Z xt_addrtype 16384 2 2024-06-26T07:09:48.8663110Z iptable_filter 16384 1 2024-06-26T07:09:48.8663498Z br_netfilter 24576 0 2024-06-26T07:09:48.8663906Z bridge 172032 1 br_netfilter 2024-06-26T07:09:48.8664368Z stp 16384 1 bridge 2024-06-26T07:09:48.8664798Z llc 16384 2 bridge,stp 2024-06-26T07:09:48.8665216Z overlay 86016 0 2024-06-26T07:09:48.8665593Z sunrpc 393216 1 2024-06-26T07:09:48.8666119Z dm_mirror 28672 0 2024-06-26T07:09:48.8666535Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:09:48.8667368Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:09:48.8667984Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:09:48.8668435Z dax 69632 1 dm_mod 2024-06-26T07:09:48.8668843Z crc32_pclmul 16384 0 2024-06-26T07:09:48.8669221Z ghash_clmulni_intel 16384 0 2024-06-26T07:09:48.8669604Z pcbc 16384 0 2024-06-26T07:09:48.8669975Z aesni_intel 188416 0 2024-06-26T07:09:48.8670367Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:09:48.8670807Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:09:48.8671255Z glue_helper 16384 1 aesni_intel 2024-06-26T07:09:48.8671810Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:09:48.8672349Z mousedev 24576 0 2024-06-26T07:09:48.8672739Z evdev 20480 3 2024-06-26T07:09:48.8673123Z psmouse 32768 0 2024-06-26T07:09:48.8673497Z button 16384 0 2024-06-26T07:09:48.8673957Z ena 139264 0 2024-06-26T07:09:48.8674334Z ptp 20480 1 ena 2024-06-26T07:09:48.8674775Z pps_core 20480 1 ptp 2024-06-26T07:09:48.8675188Z crc32c_intel 24576 0 2024-06-26T07:09:48.8675561Z autofs4 49152 2 2024-06-26T07:09:48.8675929Z + modinfo nvidia 2024-06-26T07:09:48.8676574Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:09:48.8677264Z alias: char-major-195-* 2024-06-26T07:09:48.8677640Z version: 550.54.15 2024-06-26T07:09:48.8677988Z supported: external 2024-06-26T07:09:48.8678332Z license: NVIDIA 2024-06-26T07:09:48.8678714Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:09:48.8679193Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:09:48.8679665Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:09:48.8680139Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:09:48.8680631Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:09:48.8681225Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:09:48.8681724Z depends: i2c-core,drm 2024-06-26T07:09:48.8682093Z retpoline: Y 2024-06-26T07:09:48.8682401Z name: nvidia 2024-06-26T07:09:48.8682955Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:09:48.8683615Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:09:48.8684259Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:09:48.8708124Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:09:48.8708823Z parm: NVreg_RmLogonRC:int 2024-06-26T07:09:48.8709448Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:09:48.8709902Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:09:48.8710374Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:09:48.8710815Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:09:48.8711340Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:09:48.8711911Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:09:48.8712390Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:09:48.8712821Z parm: NVreg_EnableMSI:int 2024-06-26T07:09:48.8713234Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:09:48.8713686Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:09:48.8714212Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:09:48.8714790Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:09:48.8715343Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:09:48.8715934Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:09:48.8716524Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:09:48.8717130Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:09:48.8718057Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:09:48.8718533Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:09:48.8719063Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:09:48.8719606Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:09:48.8720090Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:09:48.8720549Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:09:48.8721026Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:09:48.8721499Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:09:48.8721935Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:09:48.8722430Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:09:48.8722945Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:09:48.8723417Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:09:48.8723893Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:09:48.8724399Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:09:48.8724879Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:09:48.8725378Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:09:48.8725844Z parm: NVreg_RmMsg:charp 2024-06-26T07:09:48.8726384Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:09:48.8726848Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:09:48.8727310Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:09:48.8727753Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:09:48.8728226Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:09:48.8728698Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:09:48.8729135Z parm: rm_firmware_active:charp 2024-06-26T07:09:48.8729547Z + HAS_NVIDIA_DRIVER=0 2024-06-26T07:09:48.8730026Z ++ command -v nvidia-smi 2024-06-26T07:09:48.8730417Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-26T07:09:48.8730771Z + set +e 2024-06-26T07:09:48.8731284Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-26T07:09:50.8602212Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-26T07:09:50.8602766Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:09:50.8603643Z + '[' 0 -ne 0 ']' 2024-06-26T07:09:50.8604023Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-26T07:09:50.8604414Z + HAS_NVIDIA_DRIVER=1 2024-06-26T07:09:50.8605141Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-26T07:09:50.8605847Z + set -e 2024-06-26T07:09:50.8606244Z + '[' 1 -eq 0 ']' 2024-06-26T07:09:50.8606818Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-26T07:09:50.8607503Z + post_install_nvidia_driver_common 2024-06-26T07:09:50.8607918Z + sudo modprobe nvidia 2024-06-26T07:09:50.8703715Z + echo 'After installing NVIDIA driver' 2024-06-26T07:09:50.8704370Z + lspci 2024-06-26T07:09:50.8704803Z After installing NVIDIA driver 2024-06-26T07:09:50.8784237Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:09:50.8785246Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:09:50.8786184Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:09:50.8787373Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:09:50.8788144Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:09:50.8788939Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:09:50.8789652Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:09:50.8790434Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:09:50.8791024Z + lsmod 2024-06-26T07:09:50.8799114Z Module Size Used by 2024-06-26T07:09:50.8799618Z xt_nat 16384 0 2024-06-26T07:09:50.8800128Z ib_core 266240 0 2024-06-26T07:09:50.8800923Z nvidia_modeset 1339392 0 2024-06-26T07:09:50.8801418Z veth 16384 0 2024-06-26T07:09:50.8801935Z nvidia_uvm 4599808 0 2024-06-26T07:09:50.8802565Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:09:50.8803237Z drm 421888 1 nvidia 2024-06-26T07:09:50.8803810Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:09:50.8804403Z backlight 16384 1 nvidia_modeset 2024-06-26T07:09:50.8804860Z xt_conntrack 16384 1 2024-06-26T07:09:50.8805255Z ipt_MASQUERADE 16384 1 2024-06-26T07:09:50.8805708Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:09:50.8806478Z nf_conntrack_netlink 49152 0 2024-06-26T07:09:50.8807129Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:09:50.8807707Z xfrm_user 45056 1 2024-06-26T07:09:50.8808151Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:09:50.8808718Z iptable_nat 16384 1 2024-06-26T07:09:50.8809128Z nf_conntrack_ipv4 16384 3 2024-06-26T07:09:50.8809646Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:09:50.8810234Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:09:50.8810918Z nf_nat 32768 3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat 2024-06-26T07:09:50.8811913Z nf_conntrack 155648 8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink 2024-06-26T07:09:50.8812776Z xt_addrtype 16384 2 2024-06-26T07:09:50.8813159Z iptable_filter 16384 1 2024-06-26T07:09:50.8813553Z br_netfilter 24576 0 2024-06-26T07:09:50.8813971Z bridge 172032 1 br_netfilter 2024-06-26T07:09:50.8814464Z stp 16384 1 bridge 2024-06-26T07:09:50.8814895Z llc 16384 2 bridge,stp 2024-06-26T07:09:50.8815317Z overlay 86016 0 2024-06-26T07:09:50.8815695Z sunrpc 393216 1 2024-06-26T07:09:50.8816169Z dm_mirror 28672 0 2024-06-26T07:09:50.8816585Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:09:50.8817110Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:09:50.8817773Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:09:50.8818268Z dax 69632 1 dm_mod 2024-06-26T07:09:50.8818702Z crc32_pclmul 16384 0 2024-06-26T07:09:50.8819103Z ghash_clmulni_intel 16384 0 2024-06-26T07:09:50.8819496Z pcbc 16384 0 2024-06-26T07:09:50.8819873Z aesni_intel 188416 0 2024-06-26T07:09:50.8820268Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:09:50.8820726Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:09:50.8821186Z glue_helper 16384 1 aesni_intel 2024-06-26T07:09:50.8821750Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:09:50.8822288Z mousedev 24576 0 2024-06-26T07:09:50.8822670Z evdev 20480 3 2024-06-26T07:09:50.8823052Z psmouse 32768 0 2024-06-26T07:09:50.8823431Z button 16384 0 2024-06-26T07:09:50.8823795Z ena 139264 0 2024-06-26T07:09:50.8824182Z ptp 20480 1 ena 2024-06-26T07:09:50.8824588Z pps_core 20480 1 ptp 2024-06-26T07:09:50.8824986Z crc32c_intel 24576 0 2024-06-26T07:09:50.8825363Z autofs4 49152 2 2024-06-26T07:09:50.8825731Z + modinfo nvidia 2024-06-26T07:09:50.8826395Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:09:50.8827455Z alias: char-major-195-* 2024-06-26T07:09:50.8827852Z version: 550.54.15 2024-06-26T07:09:50.8828209Z supported: external 2024-06-26T07:09:50.8828562Z license: NVIDIA 2024-06-26T07:09:50.8828944Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:09:50.8829433Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:09:50.8829899Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:09:50.8830483Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:09:50.8830993Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:09:50.8831500Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:09:50.8831995Z depends: i2c-core,drm 2024-06-26T07:09:50.8832359Z retpoline: Y 2024-06-26T07:09:50.8832681Z name: nvidia 2024-06-26T07:09:50.8833239Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:09:50.8833896Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:09:50.8834540Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:09:50.8835137Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:09:50.8835593Z parm: NVreg_RmLogonRC:int 2024-06-26T07:09:50.8836041Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:09:50.8836501Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:09:50.8836954Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:09:50.8837409Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:09:50.8837944Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:09:50.8838516Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:09:50.8839005Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:09:50.8839452Z parm: NVreg_EnableMSI:int 2024-06-26T07:09:50.8839881Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:09:50.8840346Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:09:50.8840890Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:09:50.8841479Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:09:50.8842043Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:09:50.8842665Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:09:50.8843269Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:09:50.8843890Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:09:50.8844486Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:09:50.8844986Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:09:50.8845610Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:09:50.8846246Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:09:50.8846744Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:09:50.8847216Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:09:50.8847705Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:09:50.8848183Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:09:50.8848655Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:09:50.8849192Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:09:50.8849726Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:09:50.8850216Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:09:50.8850705Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:09:50.8851219Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:09:50.8851714Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:09:50.8852227Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:09:50.8852708Z parm: NVreg_RmMsg:charp 2024-06-26T07:09:50.8853127Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:09:50.8853608Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:09:50.8854077Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:09:50.8854543Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:09:50.8855026Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:09:50.8855507Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:09:50.8855959Z parm: rm_firmware_active:charp 2024-06-26T07:09:50.8856369Z + set +e 2024-06-26T07:09:50.8871404Z + nvidia-smi 2024-06-26T07:09:52.4468669Z Wed Jun 26 07:09:52 2024 2024-06-26T07:09:52.4469662Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:09:52.4470805Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-26T07:09:52.4471599Z |-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:09:52.4472390Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-26T07:09:52.4473278Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-26T07:09:52.4473982Z | | | MIG M. | 2024-06-26T07:09:52.4474538Z |=========================================+========================+======================| 2024-06-26T07:09:52.4601321Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-06-26T07:09:52.4602277Z | 0% 31C P0 56W / 300W | 0MiB / 23028MiB | 0% Default | 2024-06-26T07:09:52.4602970Z | | | N/A | 2024-06-26T07:09:52.4603930Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:09:52.4604744Z 2024-06-26T07:09:52.4605555Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:09:52.4606450Z | Processes: | 2024-06-26T07:09:52.4607317Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-26T07:09:52.4608010Z | ID ID Usage | 2024-06-26T07:09:52.4608569Z |=========================================================================================| 2024-06-26T07:09:52.4609212Z | No running processes found | 2024-06-26T07:09:52.4609979Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:09:53.0663216Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-26T07:09:54.6326910Z NVIDIA A10G 2024-06-26T07:09:55.0863706Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:09:55.0864407Z + '[' 0 -eq 0 ']' 2024-06-26T07:09:55.0864912Z + echo 'INFO: Ignoring allowed status 0' 2024-06-26T07:09:55.0865440Z + set -e 2024-06-26T07:09:55.0865740Z INFO: Ignoring allowed status 0 2024-06-26T07:09:55.0868120Z == Installing nvidia container toolkit for amzn2 == 2024-06-26T07:09:55.0870492Z + sudo yum install -y yum-utils 2024-06-26T07:09:55.3873027Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:09:56.8879810Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-26T07:09:56.8880524Z Nothing to do 2024-06-26T07:09:57.0396780Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-06-26T07:09:57.0397866Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:09:57.0398984Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:09:57.3602655Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:09:57.3865417Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:09:57.3866603Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:09:57.3867817Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:09:57.3991976Z + sudo yum install -y nvidia-docker2 2024-06-26T07:09:57.7099997Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:09:59.1966753Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-26T07:09:59.1967434Z Nothing to do 2024-06-26T07:09:59.3464536Z + sudo systemctl restart docker 2024-06-26T07:10:35.5128562Z Command completed after 1 attempt(s). 2024-06-26T07:10:35.5186257Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:10:35.5187335Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:10:35.5188030Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T07:10:35.5188691Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:10:35.5196445Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:35.5196936Z env: 2024-06-26T07:10:35.5197217Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:35.5197670Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:35.5198159Z ##[endgroup] 2024-06-26T07:10:35.7370235Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:10:35.7528744Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-06-26T07:10:35.7626530Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-06-26T07:10:35.8707981Z Prepare all required actions 2024-06-26T07:10:35.8708496Z Getting action download info 2024-06-26T07:10:35.9743267Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T07:10:36.1628021Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T07:10:36.2994658Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T07:10:36.2995136Z with: 2024-06-26T07:10:36.2995479Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:36.2995927Z s3-bucket: gha-artifacts 2024-06-26T07:10:36.2996269Z env: 2024-06-26T07:10:36.2996546Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:36.2996993Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:36.2997478Z ##[endgroup] 2024-06-26T07:10:36.3029768Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:10:36.3030230Z with: 2024-06-26T07:10:36.3030614Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:36.3031085Z s3-bucket: gha-artifacts 2024-06-26T07:10:36.3031456Z region: us-east-1 2024-06-26T07:10:36.3031808Z env: 2024-06-26T07:10:36.3032122Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:36.3032595Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:36.3033086Z ##[endgroup] 2024-06-26T07:10:36.7295777Z (node:2134) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:10:36.7296559Z 2024-06-26T07:10:36.7296831Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:10:36.7297554Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:10:36.7298574Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:10:36.8062280Z Found 1 objects with prefix pytorch/pytorch/9673646806/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-06-26T07:10:36.8063380Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:10:57.9563347Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:10:57.9571206Z Artifact download has finished successfully 2024-06-26T07:10:57.9730863Z ##[group]Run unzip -o artifacts.zip 2024-06-26T07:10:57.9731306Z unzip -o artifacts.zip 2024-06-26T07:10:57.9738928Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:57.9739431Z env: 2024-06-26T07:10:57.9739706Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:57.9740162Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:57.9740648Z ##[endgroup] 2024-06-26T07:10:57.9766889Z Archive: artifacts.zip 2024-06-26T07:10:57.9768443Z creating: dist/ 2024-06-26T07:10:59.9466013Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:10:59.9468508Z creating: build/custom_test_artifacts/ 2024-06-26T07:10:59.9470136Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T07:10:59.9471640Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T07:10:59.9473598Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:10:59.9475707Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:10:59.9476978Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T07:10:59.9478088Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:10:59.9479458Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:10:59.9480426Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:10:59.9481525Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:10:59.9482794Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:10:59.9483792Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:10:59.9484784Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:10:59.9486027Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:10:59.9487188Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:10:59.9488313Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:10:59.9489441Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:10:59.9490589Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:10:59.9491755Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:10:59.9492776Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:10:59.9493759Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:10:59.9525351Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:10:59.9563252Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:10:59.9564667Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:10:59.9608533Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:10:59.9611368Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:10:59.9614149Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:10:59.9616152Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:10:59.9617529Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:10:59.9618892Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:10:59.9620265Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:10:59.9621625Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:10:59.9623086Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:10:59.9624350Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:10:59.9625575Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:10:59.9627023Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:10:59.9628244Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:10:59.9629431Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:10:59.9630631Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:10:59.9676592Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:10:59.9734528Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:10:59.9735771Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:10:59.9736895Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:10:59.9737873Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:10:59.9738897Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T07:10:59.9739936Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T07:10:59.9741101Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T07:10:59.9742433Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T07:10:59.9743695Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T07:10:59.9744877Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T07:10:59.9746094Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T07:10:59.9747465Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T07:10:59.9748535Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T07:10:59.9749600Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T07:10:59.9750644Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T07:10:59.9762571Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T07:10:59.9890256Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T07:10:59.9891253Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T07:10:59.9892340Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T07:10:59.9893530Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T07:10:59.9894690Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T07:10:59.9895777Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T07:10:59.9896894Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T07:10:59.9898202Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T07:10:59.9899328Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T07:10:59.9900457Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T07:10:59.9901572Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T07:10:59.9915789Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T07:10:59.9995317Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T07:10:59.9996653Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:10:59.9997725Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:10:59.9998821Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T07:10:59.9999742Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T07:11:00.0000637Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T07:11:00.0001535Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-26T07:11:00.0002387Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T07:11:00.0003168Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T07:11:00.0003980Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T07:11:00.0111204Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T07:11:00.0172427Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T07:11:00.0174128Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T07:11:00.0175493Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T07:11:00.0176358Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:11:00.0179564Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:11:00.0180603Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T07:11:00.0181524Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:11:00.0182497Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:11:00.0183450Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:11:00.0184525Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:11:00.0185720Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:11:00.0186935Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:11:00.0187923Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:11:00.0189052Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:11:00.0190173Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:11:00.0191284Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:11:00.0192399Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:11:00.0193539Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:11:00.0194809Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:11:00.0195826Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:11:00.0196809Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:11:00.0234242Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:11:00.0272150Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:11:00.0273583Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:11:00.0316432Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:11:00.0317846Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:11:00.0319391Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:11:00.0320849Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:11:00.0322224Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:11:00.0323585Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:11:00.0324949Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:11:00.0326399Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:11:00.0327731Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:11:00.0328985Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:11:00.0330197Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:11:00.0331395Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:11:00.0332602Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:11:00.0333767Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:11:00.0334951Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:11:00.0386125Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:11:00.0445450Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:11:00.0447236Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:11:00.0448666Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:11:00.0449897Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:11:00.0451189Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T07:11:00.0452562Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T07:11:00.0454107Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T07:11:00.0455744Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T07:11:00.0457547Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T07:11:00.0459035Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T07:11:00.0460591Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T07:11:00.0462120Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T07:11:00.0463685Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T07:11:00.0465244Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T07:11:00.0467061Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T07:11:00.0475280Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T07:11:00.0536770Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T07:11:00.0537962Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:11:00.0539051Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:11:00.0540032Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T07:11:00.0540942Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T07:11:00.0541840Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T07:11:00.0542744Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-26T07:11:00.0543590Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T07:11:00.0544384Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T07:11:00.0545210Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T07:11:00.0590227Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T07:11:00.0591007Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T07:11:00.0591769Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T07:11:00.0592658Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:11:00.0597667Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:11:00.0598667Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T07:11:00.0599675Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:11:00.0600737Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:11:00.0601772Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:11:00.0602929Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:11:00.0604094Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:11:00.0605146Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:11:00.0606280Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:11:00.0607500Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:11:00.0608704Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:11:00.0610048Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:11:00.0611235Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:11:00.0612453Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:11:00.0613647Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:11:00.0614733Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:11:00.0615781Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:11:00.0648836Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:11:00.0686626Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:11:00.0688242Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:11:00.0730754Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:11:00.0732218Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:11:00.0733701Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:11:00.0735212Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:11:00.0736715Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:11:00.0738168Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:11:00.0739609Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:11:00.0741034Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:11:00.0742418Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:11:00.0743745Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:11:00.0745015Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:11:00.0746316Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:11:00.0747811Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:11:00.0749051Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:11:00.0750311Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:11:00.0796131Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:11:00.0854000Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:11:00.0855242Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:11:00.0856340Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:11:00.0857370Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:11:00.0858320Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T07:11:00.0859323Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T07:11:00.0860443Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T07:11:00.0861679Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T07:11:00.0862872Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T07:11:00.0863989Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T07:11:00.0865151Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T07:11:00.0866412Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T07:11:00.0867737Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T07:11:00.0868909Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T07:11:00.0870063Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T07:11:00.0871308Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T07:11:00.0983602Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T07:11:00.0984767Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T07:11:00.0986013Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T07:11:00.0987425Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T07:11:00.0988676Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T07:11:00.0989863Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T07:11:00.0991085Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T07:11:00.0992311Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T07:11:00.0993534Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T07:11:00.0994765Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T07:11:00.0995976Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T07:11:00.1007766Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T07:11:00.1060475Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T07:11:00.1061753Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:11:00.1062889Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:11:00.1063913Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T07:11:00.1064870Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T07:11:00.1065951Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T07:11:00.1067044Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-26T07:11:00.1067949Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T07:11:00.1068778Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T07:11:00.1069633Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T07:11:00.1165643Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T07:11:00.1206583Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T07:11:00.1207223Z creating: build/lib/ 2024-06-26T07:11:00.1291877Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T07:11:00.1730688Z inflating: build/lib/libprotobuf.a 2024-06-26T07:11:00.1739881Z inflating: build/lib/libpthreadpool.a 2024-06-26T07:11:00.1747785Z inflating: build/lib/libcpuinfo.a 2024-06-26T07:11:00.1755547Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T07:11:00.1756068Z inflating: build/lib/libclog.a 2024-06-26T07:11:00.1773459Z inflating: build/lib/libnnpack.a 2024-06-26T07:11:00.1775394Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T07:11:00.1837707Z inflating: build/lib/libgtest.a 2024-06-26T07:11:00.1909946Z inflating: build/lib/libbenchmark.a 2024-06-26T07:11:00.1970912Z inflating: build/lib/libasmjit.a 2024-06-26T07:11:00.1978030Z inflating: build/lib/libittnotify.a 2024-06-26T07:11:00.2004637Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T07:11:00.2126794Z inflating: build/lib/libgloo.a 2024-06-26T07:11:00.2147938Z inflating: build/lib/libfmt.a 2024-06-26T07:11:00.2236515Z inflating: build/lib/libc10.so 2024-06-26T07:11:00.2237943Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T07:11:00.2238455Z inflating: build/lib/libfoxi_loader.a 2024-06-26T07:11:00.2239358Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T07:11:00.2258274Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T07:11:00.2275528Z inflating: build/lib/libgmock.a 2024-06-26T07:11:00.2276036Z inflating: build/lib/libgtest_main.a 2024-06-26T07:11:00.2276527Z inflating: build/lib/libbenchmark_main.a 2024-06-26T07:11:00.2760683Z inflating: build/lib/libprotoc.a 2024-06-26T07:11:00.3132045Z inflating: build/lib/libgloo_cuda.a 2024-06-26T07:11:01.2848730Z inflating: build/lib/libdnnl.a 2024-06-26T07:11:01.3399300Z inflating: build/lib/libtensorpipe.a 2024-06-26T07:11:01.3451918Z inflating: build/lib/libc10_cuda.so 2024-06-26T07:11:01.3452382Z inflating: build/lib/libgmock_main.a 2024-06-26T07:11:01.4654104Z inflating: build/lib/libfbgemm.a 2024-06-26T07:11:01.4900249Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-26T07:11:01.5397290Z inflating: build/lib/libkineto.a 2024-06-26T07:11:01.5576610Z inflating: build/lib/libXNNPACK.a 2024-06-26T07:11:01.5617553Z inflating: build/lib/libonnx_proto.a 2024-06-26T07:11:01.6281519Z inflating: build/lib/libonnx.a 2024-06-26T07:11:03.9537827Z inflating: build/lib/libtorch_cpu.so 2024-06-26T07:11:03.9542159Z inflating: build/lib/libunbox_lib.a 2024-06-26T07:11:03.9545995Z inflating: build/lib/libshm.so 2024-06-26T07:11:05.9384026Z inflating: build/lib/libtorch_cuda.so 2024-06-26T07:11:05.9384873Z inflating: build/lib/libtorch.so 2024-06-26T07:11:06.7322484Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-26T07:11:06.7323826Z inflating: build/lib/libc10d_cuda_test.so 2024-06-26T07:11:06.9215712Z inflating: build/lib/libtorch_python.so 2024-06-26T07:11:06.9281206Z inflating: build/lib/libtorchbind_test.so 2024-06-26T07:11:06.9301161Z inflating: build/lib/libjitbackend_test.so 2024-06-26T07:11:06.9325596Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T07:11:06.9359102Z inflating: build/lib/libnnapi_backend.so 2024-06-26T07:11:06.9359576Z creating: build/bin/ 2024-06-26T07:11:06.9408751Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T07:11:06.9457831Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T07:11:06.9506434Z inflating: build/bin/c10_Device_test 2024-06-26T07:11:06.9563868Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T07:11:06.9615644Z inflating: build/bin/c10_Scalar_test 2024-06-26T07:11:06.9663047Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T07:11:06.9716248Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T07:11:06.9764802Z inflating: build/bin/c10_SymInt_test 2024-06-26T07:11:06.9818588Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T07:11:06.9872940Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T07:11:06.9924031Z inflating: build/bin/c10_Bitset_test 2024-06-26T07:11:06.9992952Z inflating: build/bin/c10_cow_test 2024-06-26T07:11:07.0040276Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T07:11:07.0088615Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T07:11:07.0137642Z inflating: build/bin/c10_Half_test 2024-06-26T07:11:07.0192724Z inflating: build/bin/c10_LeftRight_test 2024-06-26T07:11:07.0245340Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T07:11:07.0293581Z inflating: build/bin/c10_Synchronized_test 2024-06-26T07:11:07.0347329Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T07:11:07.0397790Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T07:11:07.0446778Z inflating: build/bin/c10_TypeList_test 2024-06-26T07:11:07.0494041Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T07:11:07.0544021Z inflating: build/bin/c10_accumulate_test 2024-06-26T07:11:07.0599017Z inflating: build/bin/c10_bfloat16_test 2024-06-26T07:11:07.0647874Z inflating: build/bin/c10_bit_cast_test 2024-06-26T07:11:07.0702489Z inflating: build/bin/c10_complex_math_test 2024-06-26T07:11:07.0757106Z inflating: build/bin/c10_complex_test 2024-06-26T07:11:07.0807324Z inflating: build/bin/c10_exception_test 2024-06-26T07:11:07.0857174Z inflating: build/bin/c10_flags_test 2024-06-26T07:11:07.0904818Z inflating: build/bin/c10_generic_math_test 2024-06-26T07:11:07.1064596Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T07:11:07.1114761Z inflating: build/bin/c10_irange_test 2024-06-26T07:11:07.1165740Z inflating: build/bin/c10_lazy_test 2024-06-26T07:11:07.1220931Z inflating: build/bin/c10_logging_test 2024-06-26T07:11:07.1294153Z inflating: build/bin/c10_optional_test 2024-06-26T07:11:07.1353693Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T07:11:07.1406023Z inflating: build/bin/c10_registry_test 2024-06-26T07:11:07.1552390Z inflating: build/bin/c10_small_vector_test 2024-06-26T07:11:07.1602309Z inflating: build/bin/c10_ssize_test 2024-06-26T07:11:07.1652617Z inflating: build/bin/c10_string_util_test 2024-06-26T07:11:07.1710158Z inflating: build/bin/c10_string_view_test 2024-06-26T07:11:07.1758140Z inflating: build/bin/c10_tempfile_test 2024-06-26T07:11:07.1805066Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T07:11:07.1859317Z inflating: build/bin/c10_typeid_test 2024-06-26T07:11:07.2291387Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T07:11:07.2723330Z inflating: build/bin/protoc 2024-06-26T07:11:07.2774473Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-26T07:11:07.2825750Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-26T07:11:07.2876894Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-26T07:11:07.2927058Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-26T07:11:07.2978241Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T07:11:07.3029085Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T07:11:07.3077768Z inflating: build/bin/c10_cuda_CUDATest 2024-06-26T07:11:07.3128799Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-26T07:11:07.3447105Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T07:11:07.3780590Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T07:11:07.4126045Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T07:11:07.4177407Z inflating: build/bin/FileStoreTest 2024-06-26T07:11:07.4231890Z inflating: build/bin/TCPStoreTest 2024-06-26T07:11:07.4282755Z inflating: build/bin/HashStoreTest 2024-06-26T07:11:07.4295796Z inflating: build/bin/ProcessGroupMPITest 2024-06-26T07:11:07.4348399Z inflating: build/bin/test_edge_op_registration 2024-06-26T07:11:07.4352479Z inflating: build/bin/torch_shm_manager 2024-06-26T07:11:07.4354467Z inflating: build/bin/example_allreduce 2024-06-26T07:11:07.4407602Z inflating: build/bin/test_dist_autograd 2024-06-26T07:11:07.4475889Z inflating: build/bin/test_cpp_rpc 2024-06-26T07:11:07.4477461Z inflating: build/bin/parallel_benchmark 2024-06-26T07:11:07.4543124Z inflating: build/bin/test_mobile_nnc 2024-06-26T07:11:07.4551813Z inflating: build/bin/aot_model_compiler_test 2024-06-26T07:11:07.4884142Z inflating: build/bin/test_lazy 2024-06-26T07:11:07.6031251Z inflating: build/bin/test_api 2024-06-26T07:11:07.6102694Z inflating: build/bin/Dict_test 2024-06-26T07:11:07.6153181Z inflating: build/bin/Dimname_test 2024-06-26T07:11:07.6216184Z inflating: build/bin/MaybeOwned_test 2024-06-26T07:11:07.6270947Z inflating: build/bin/NamedTensor_test 2024-06-26T07:11:07.6328896Z inflating: build/bin/apply_utils_test 2024-06-26T07:11:07.6385188Z inflating: build/bin/atest 2024-06-26T07:11:07.6446237Z inflating: build/bin/basic 2024-06-26T07:11:07.6499174Z inflating: build/bin/broadcast_test 2024-06-26T07:11:07.6548297Z inflating: build/bin/cpu_allocator_test 2024-06-26T07:11:07.6604500Z inflating: build/bin/cpu_generator_test 2024-06-26T07:11:07.6656090Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T07:11:07.6746090Z inflating: build/bin/cpu_rng_test 2024-06-26T07:11:07.6794683Z inflating: build/bin/dispatch_key_set_test 2024-06-26T07:11:07.6843139Z inflating: build/bin/dlconvertor_test 2024-06-26T07:11:07.6900024Z inflating: build/bin/extension_backend_test 2024-06-26T07:11:07.6952559Z inflating: build/bin/half_test 2024-06-26T07:11:07.7044977Z inflating: build/bin/ivalue_test 2024-06-26T07:11:07.7093045Z inflating: build/bin/lazy_tensor_test 2024-06-26T07:11:07.7145571Z inflating: build/bin/math_kernel_test 2024-06-26T07:11:07.7197971Z inflating: build/bin/memory_format_test 2024-06-26T07:11:07.7248983Z inflating: build/bin/memory_overlapping_test 2024-06-26T07:11:07.7300271Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T07:11:07.7354230Z inflating: build/bin/native_test 2024-06-26T07:11:07.7403436Z inflating: build/bin/operator_name_test 2024-06-26T07:11:07.7453127Z inflating: build/bin/operators_test 2024-06-26T07:11:07.7502965Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T07:11:07.7568620Z inflating: build/bin/pow_test 2024-06-26T07:11:07.7623160Z inflating: build/bin/quantized_test 2024-06-26T07:11:07.7671489Z inflating: build/bin/reduce_ops_test 2024-06-26T07:11:07.7721222Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T07:11:07.7775719Z inflating: build/bin/scalar_tensor_test 2024-06-26T07:11:07.7832175Z inflating: build/bin/scalar_test 2024-06-26T07:11:07.7882062Z inflating: build/bin/StorageUtils_test 2024-06-26T07:11:07.7932799Z inflating: build/bin/stride_properties_test 2024-06-26T07:11:07.8008529Z inflating: build/bin/tensor_iterator_test 2024-06-26T07:11:07.8061150Z inflating: build/bin/test_parallel 2024-06-26T07:11:07.8063985Z inflating: build/bin/thread_init_test 2024-06-26T07:11:07.8118227Z inflating: build/bin/type_ptr_test 2024-06-26T07:11:07.8176015Z inflating: build/bin/type_test 2024-06-26T07:11:07.8226375Z inflating: build/bin/undefined_tensor_test 2024-06-26T07:11:07.8227563Z inflating: build/bin/verify_api_visibility 2024-06-26T07:11:07.8295070Z inflating: build/bin/legacy_vmap_test 2024-06-26T07:11:07.8345034Z inflating: build/bin/weakref_test 2024-06-26T07:11:07.8394960Z inflating: build/bin/wrapdim_test 2024-06-26T07:11:07.8444940Z inflating: build/bin/xla_tensor_test 2024-06-26T07:11:07.8503016Z inflating: build/bin/IListRef_test 2024-06-26T07:11:07.8605771Z inflating: build/bin/List_test 2024-06-26T07:11:07.8671768Z inflating: build/bin/KernelFunction_test 2024-06-26T07:11:07.8789520Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T07:11:07.8882873Z inflating: build/bin/kernel_function_test 2024-06-26T07:11:07.9007633Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T07:11:07.9107919Z inflating: build/bin/kernel_lambda_test 2024-06-26T07:11:07.9166723Z inflating: build/bin/kernel_stackbased_test 2024-06-26T07:11:07.9260001Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T07:11:07.9309326Z inflating: build/bin/CppSignature_test 2024-06-26T07:11:07.9363090Z inflating: build/bin/backend_fallback_test 2024-06-26T07:11:07.9410498Z inflating: build/bin/op_allowlist_test 2024-06-26T07:11:07.9701668Z inflating: build/bin/op_registration_test 2024-06-26T07:11:07.9763315Z inflating: build/bin/inline_container_test 2024-06-26T07:11:07.9815102Z inflating: build/bin/cuda_apply_test 2024-06-26T07:11:07.9863606Z inflating: build/bin/cuda_allocator_test 2024-06-26T07:11:07.9920668Z inflating: build/bin/cuda_atomic_ops_test 2024-06-26T07:11:07.9973223Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-26T07:11:08.0041573Z inflating: build/bin/cuda_complex_math_test 2024-06-26T07:11:08.0098270Z inflating: build/bin/cuda_complex_test 2024-06-26T07:11:08.0146507Z inflating: build/bin/cuda_device_test 2024-06-26T07:11:08.0202247Z inflating: build/bin/cuda_cub_test 2024-06-26T07:11:08.0251402Z inflating: build/bin/cuda_dlconvertor_test 2024-06-26T07:11:08.0314699Z inflating: build/bin/cuda_distributions_test 2024-06-26T07:11:08.0369091Z inflating: build/bin/cuda_generator_test 2024-06-26T07:11:08.0417242Z inflating: build/bin/cuda_half_test 2024-06-26T07:11:08.0467700Z inflating: build/bin/cuda_integer_divider_test 2024-06-26T07:11:08.0516124Z inflating: build/bin/cuda_optional_test 2024-06-26T07:11:08.0565302Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-26T07:11:08.0617124Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-26T07:11:08.0664433Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-26T07:11:08.0723101Z inflating: build/bin/cuda_stream_test 2024-06-26T07:11:08.0771197Z inflating: build/bin/cuda_cudnn_test 2024-06-26T07:11:08.0822729Z inflating: build/bin/cuda_vectorized_test 2024-06-26T07:11:08.0837247Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T07:11:08.0900690Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T07:11:08.0957072Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-26T07:11:08.1017867Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-26T07:11:08.1078015Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-26T07:11:08.1880827Z inflating: build/bin/test_tensorexpr 2024-06-26T07:11:08.2438827Z inflating: build/bin/test_jit 2024-06-26T07:11:08.2439361Z creating: .additional_ci_files/ 2024-06-26T07:11:08.2492638Z inflating: .additional_ci_files/test-times.json 2024-06-26T07:11:08.2704667Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T07:11:08.2729235Z ##[group]Run rm artifacts.zip 2024-06-26T07:11:08.2729624Z rm artifacts.zip 2024-06-26T07:11:08.2739172Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:08.2739674Z env: 2024-06-26T07:11:08.2739960Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:08.2740415Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:08.2740900Z ##[endgroup] 2024-06-26T07:11:08.3453816Z ##[group]Run df -H 2024-06-26T07:11:08.3454320Z df -H 2024-06-26T07:11:08.3464281Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:08.3464988Z env: 2024-06-26T07:11:08.3465390Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:08.3466054Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:08.3466951Z ##[endgroup] 2024-06-26T07:11:08.3498433Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T07:11:08.3498974Z devtmpfs 34G 0 34G 0% /dev 2024-06-26T07:11:08.3499453Z tmpfs 34G 0 34G 0% /dev/shm 2024-06-26T07:11:08.3500208Z tmpfs 34G 455k 34G 1% /run 2024-06-26T07:11:08.3500718Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-06-26T07:11:08.3501211Z /dev/nvme0n1p1 162G 66G 96G 41% / 2024-06-26T07:11:08.3539348Z Prepare all required actions 2024-06-26T07:11:08.3539789Z Getting action download info 2024-06-26T07:11:08.4879832Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T07:11:08.4880294Z with: 2024-06-26T07:11:08.4880556Z env: 2024-06-26T07:11:08.4880834Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:08.4881289Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:08.4881766Z ##[endgroup] 2024-06-26T07:11:08.4912499Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:11:08.4912940Z with: 2024-06-26T07:11:08.4913206Z name: td_results 2024-06-26T07:11:08.4913523Z s3-bucket: gha-artifacts 2024-06-26T07:11:08.4913871Z region: us-east-1 2024-06-26T07:11:08.4914171Z env: 2024-06-26T07:11:08.4914438Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:08.4914887Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:08.4915378Z ##[endgroup] 2024-06-26T07:11:08.9082475Z (node:2184) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:11:08.9083549Z 2024-06-26T07:11:08.9083980Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:11:08.9085102Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:11:08.9086545Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:11:08.9981599Z Found 0 objects with prefix pytorch/pytorch/9673646806/td_results/ 2024-06-26T07:11:08.9989190Z Artifact download has finished successfully 2024-06-26T07:11:09.0133532Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T07:11:09.0134022Z mkdir -p .additional_ci_files 2024-06-26T07:11:09.0134588Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T07:11:09.0142275Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:09.0142792Z env: 2024-06-26T07:11:09.0143070Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:09.0143533Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:09.0144015Z ##[endgroup] 2024-06-26T07:11:09.0179540Z mv: cannot stat 'td_results.json': No such file or directory 2024-06-26T07:11:09.0190480Z ##[error]Process completed with exit code 1. 2024-06-26T07:11:09.0222246Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T07:11:09.0222736Z .github/scripts/parse_ref.py 2024-06-26T07:11:09.0230522Z shell: /usr/bin/bash -e {0} 2024-06-26T07:11:09.0230880Z env: 2024-06-26T07:11:09.0231161Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:09.0231615Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:09.0232094Z ##[endgroup] 2024-06-26T07:11:09.0456852Z Prepare all required actions 2024-06-26T07:11:09.0493253Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T07:11:09.0493715Z with: 2024-06-26T07:11:09.0494270Z github-token: *** 2024-06-26T07:11:09.0494719Z env: 2024-06-26T07:11:09.0495002Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:09.0495468Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:09.0495964Z ##[endgroup] 2024-06-26T07:11:09.0511054Z ##[group]Run set -eux 2024-06-26T07:11:09.0511398Z set -eux 2024-06-26T07:11:09.0511982Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T07:11:09.0519265Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:09.0519995Z env: 2024-06-26T07:11:09.0520281Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:09.0520742Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:09.0521398Z GITHUB_TOKEN: *** 2024-06-26T07:11:09.0521735Z ##[endgroup] 2024-06-26T07:11:09.0542918Z + python3 .github/scripts/get_workflow_job_id.py 9673646806 i-09875a2d32470f475 2024-06-26T07:11:09.6337494Z setting job-id=26688917205 2024-06-26T07:11:09.6338565Z setting job-name=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:09.6526511Z Prepare all required actions 2024-06-26T07:11:09.6526963Z Getting action download info 2024-06-26T07:11:09.7566307Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T07:11:09.7566764Z with: 2024-06-26T07:11:09.7567228Z github-token: *** 2024-06-26T07:11:09.7573712Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:11:09.7581316Z job-name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:09.7582242Z env: 2024-06-26T07:11:09.7582515Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:09.7582968Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:09.7583451Z ##[endgroup] 2024-06-26T07:11:09.7623392Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:11:09.7623941Z with: 2024-06-26T07:11:09.7624209Z shell: bash 2024-06-26T07:11:09.7624506Z timeout_minutes: 10 2024-06-26T07:11:09.7624831Z max_attempts: 5 2024-06-26T07:11:09.7625135Z retry_wait_seconds: 30 2024-06-26T07:11:09.7626231Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:11:09.7627514Z polling_interval_seconds: 1 2024-06-26T07:11:09.7627884Z warning_on_retry: true 2024-06-26T07:11:09.7628380Z continue_on_error: false 2024-06-26T07:11:09.7628720Z env: 2024-06-26T07:11:09.7628995Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:09.7629441Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:09.7630070Z GITHUB_TOKEN: *** 2024-06-26T07:11:09.7630392Z ##[endgroup] 2024-06-26T07:11:09.8122021Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:11:10.0285705Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:11:10.0439745Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-06-26T07:11:10.0583850Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-06-26T07:11:10.0593753Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-06-26T07:11:10.0615047Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2) 2024-06-26T07:11:10.0624721Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-06-26T07:11:10.0638236Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-06-26T07:11:10.8124413Z Command completed after 1 attempt(s). 2024-06-26T07:11:10.8166564Z ##[group]Run set -x 2024-06-26T07:11:10.8166923Z set -x 2024-06-26T07:11:10.8167237Z  2024-06-26T07:11:10.8167787Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:11:10.8168463Z # in runner workspace 2024-06-26T07:11:10.8168993Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T07:11:10.8176757Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:10.8177358Z env: 2024-06-26T07:11:10.8177656Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:10.8178114Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:10.8178587Z ##[endgroup] 2024-06-26T07:11:10.8214037Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T07:11:10.8415497Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:11:10.8416056Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:11:10.8416528Z echo "Job name: ${JOB_NAME}" 2024-06-26T07:11:10.8416936Z  2024-06-26T07:11:10.8417493Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:11:10.8418172Z # in runner workspace 2024-06-26T07:11:10.8418761Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T07:11:10.8419405Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T07:11:10.8419896Z  --job-name "${JOB_NAME}" \ 2024-06-26T07:11:10.8426985Z  --test-matrix "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-06-26T07:11:10.8433972Z  --selected-test-configs "" \ 2024-06-26T07:11:10.8434443Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T07:11:10.8434879Z  --tag "${TAG}" \ 2024-06-26T07:11:10.8435280Z  --event-name "${EVENT_NAME}" \ 2024-06-26T07:11:10.8435737Z  --schedule "${SCHEDULE}" \ 2024-06-26T07:11:10.8436274Z  --branch "${HEAD_BRANCH}" 2024-06-26T07:11:10.8443709Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:10.8444218Z env: 2024-06-26T07:11:10.8444505Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:10.8444972Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:10.8445649Z GITHUB_TOKEN: *** 2024-06-26T07:11:10.8446585Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:10.8447513Z PR_NUMBER: 2024-06-26T07:11:10.8447826Z TAG: ciflow/inductor/129470 2024-06-26T07:11:10.8448198Z EVENT_NAME: push 2024-06-26T07:11:10.8448520Z SCHEDULE: 2024-06-26T07:11:10.8448813Z HEAD_BRANCH: 2024-06-26T07:11:10.8449117Z ##[endgroup] 2024-06-26T07:11:10.8468187Z Workflow: inductor-periodic 2024-06-26T07:11:10.8469272Z Job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:11.0561004Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T07:11:11.1800793Z ##[group]Run echo "Filtered matrix:" 2024-06-26T07:11:11.1801246Z echo "Filtered matrix:" 2024-06-26T07:11:11.1807857Z echo "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-06-26T07:11:11.1814639Z  2024-06-26T07:11:11.1815040Z echo 2024-06-26T07:11:11.1815434Z echo "Is the current job unstable? False" 2024-06-26T07:11:11.1815885Z  2024-06-26T07:11:11.1816160Z echo 2024-06-26T07:11:11.1816521Z echo "Is keep-going label set? False" 2024-06-26T07:11:11.1816952Z  2024-06-26T07:11:11.1817224Z echo 2024-06-26T07:11:11.1817543Z echo "Renabled issues? " 2024-06-26T07:11:11.1825277Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:11.1825780Z env: 2024-06-26T07:11:11.1826065Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:11.1826524Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:11.1827238Z ##[endgroup] 2024-06-26T07:11:11.1847720Z Filtered matrix: 2024-06-26T07:11:11.1855470Z {include: [{config: dynamo_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-06-26T07:11:11.1862047Z 2024-06-26T07:11:11.1862201Z Is the current job unstable? False 2024-06-26T07:11:11.1862487Z 2024-06-26T07:11:11.1862765Z Is keep-going label set? False 2024-06-26T07:11:11.1863023Z 2024-06-26T07:11:11.1863159Z Renabled issues? 2024-06-26T07:11:11.1903041Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:11:11.1903741Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:11:11.1911676Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:11.1912175Z env: 2024-06-26T07:11:11.1912472Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:11.1912945Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:11.1913442Z JOB_TIMEOUT: 240 2024-06-26T07:11:11.1913770Z ##[endgroup] 2024-06-26T07:11:11.1984781Z ##[group]Run set -x 2024-06-26T07:11:11.1985194Z set -x 2024-06-26T07:11:11.1985502Z  2024-06-26T07:11:11.1985867Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T07:11:11.1986413Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T07:11:11.1987140Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T07:11:11.1987658Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-26T07:11:11.1988079Z else 2024-06-26T07:11:11.1988423Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:11:11.1988851Z fi 2024-06-26T07:11:11.1989138Z  2024-06-26T07:11:11.1989611Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T07:11:11.1990371Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T07:11:11.1991056Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T07:11:11.1991651Z # shellcheck disable=SC2086,SC2090 2024-06-26T07:11:11.1992250Z container_name=$(docker run \ 2024-06-26T07:11:11.1992680Z  ${GPU_FLAG:-} \ 2024-06-26T07:11:11.1993066Z  -e BUILD_ENVIRONMENT \ 2024-06-26T07:11:11.1993485Z  -e PR_NUMBER \ 2024-06-26T07:11:11.1993852Z  -e GITHUB_ACTIONS \ 2024-06-26T07:11:11.1994257Z  -e GITHUB_REPOSITORY \ 2024-06-26T07:11:11.1994673Z  -e GITHUB_WORKFLOW \ 2024-06-26T07:11:11.1995070Z  -e GITHUB_JOB \ 2024-06-26T07:11:11.1995440Z  -e GITHUB_RUN_ID \ 2024-06-26T07:11:11.1995835Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T07:11:11.1996252Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T07:11:11.1996653Z  -e JOB_ID \ 2024-06-26T07:11:11.1997007Z  -e JOB_NAME \ 2024-06-26T07:11:11.1997364Z  -e BASE_SHA \ 2024-06-26T07:11:11.1997716Z  -e BRANCH \ 2024-06-26T07:11:11.1998057Z  -e SHA1 \ 2024-06-26T07:11:11.1998414Z  -e AWS_DEFAULT_REGION \ 2024-06-26T07:11:11.1998837Z  -e IN_WHEEL_TEST \ 2024-06-26T07:11:11.1999228Z  -e SHARD_NUMBER \ 2024-06-26T07:11:11.1999605Z  -e TEST_CONFIG \ 2024-06-26T07:11:11.1999989Z  -e NUM_TEST_SHARDS \ 2024-06-26T07:11:11.2000396Z  -e REENABLED_ISSUES \ 2024-06-26T07:11:11.2000822Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T07:11:11.2001250Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T07:11:11.2001660Z  -e NO_TEST_TIMEOUT \ 2024-06-26T07:11:11.2002049Z  -e NO_TD \ 2024-06-26T07:11:11.2002394Z  -e TD_DISTRIBUTED \ 2024-06-26T07:11:11.2002784Z  -e PR_LABELS \ 2024-06-26T07:11:11.2003202Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T07:11:11.2003668Z  -e SCCACHE_BUCKET \ 2024-06-26T07:11:11.2004069Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-26T07:11:11.2004490Z  -e XLA_CUDA \ 2024-06-26T07:11:11.2004913Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T07:11:11.2005441Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T07:11:11.2006022Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T07:11:11.2006551Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-26T07:11:11.2007027Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-26T07:11:11.2007460Z  -e DASHBOARD_TAG \ 2024-06-26T07:11:11.2007939Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T07:11:11.2008502Z  --security-opt seccomp=unconfined \ 2024-06-26T07:11:11.2008979Z  --cap-add=SYS_PTRACE \ 2024-06-26T07:11:11.2009384Z  --ipc=host \ 2024-06-26T07:11:11.2009749Z  --shm-size="${SHM_SIZE}" \ 2024-06-26T07:11:11.2010164Z  --tty \ 2024-06-26T07:11:11.2010492Z  --detach \ 2024-06-26T07:11:11.2010867Z  --name="${container_name}" \ 2024-06-26T07:11:11.2011289Z  --user jenkins \ 2024-06-26T07:11:11.2011868Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T07:11:11.2012451Z  -w /var/lib/jenkins/workspace \ 2024-06-26T07:11:11.2012902Z  "${DOCKER_IMAGE}" 2024-06-26T07:11:11.2013260Z ) 2024-06-26T07:11:11.2013676Z # Propagate download.pytorch.org IP to container 2024-06-26T07:11:11.2014595Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-26T07:11:11.2015559Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-26T07:11:11.2016455Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-26T07:11:11.2023799Z shell: /usr/bin/bash -e {0} 2024-06-26T07:11:11.2024156Z env: 2024-06-26T07:11:11.2024438Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:11.2024886Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:11.2025500Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:11:11.2026068Z PR_NUMBER: 2024-06-26T07:11:11.2026395Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T07:11:11.2026957Z GITHUB_WORKFLOW: inductor-periodic 2024-06-26T07:11:11.2027373Z GITHUB_JOB: test 2024-06-26T07:11:11.2027698Z GITHUB_RUN_ID: 9673646806 2024-06-26T07:11:11.2028060Z GITHUB_RUN_NUMBER: 45567 2024-06-26T07:11:11.2028413Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T07:11:11.2028750Z JOB_ID: 26688917205 2024-06-26T07:11:11.2029619Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:11.2030538Z BRANCH: 2024-06-26T07:11:11.2030872Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:11.2031386Z BASE_SHA: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:11.2031898Z TEST_CONFIG: dynamic_aot_eager_huggingface 2024-06-26T07:11:11.2032340Z SHARD_NUMBER: 1 2024-06-26T07:11:11.2032650Z NUM_TEST_SHARDS: 1 2024-06-26T07:11:11.2032994Z REENABLED_ISSUES: 2024-06-26T07:11:11.2033340Z CONTINUE_THROUGH_ERROR: False 2024-06-26T07:11:11.2033726Z VERBOSE_TEST_LOGS: False 2024-06-26T07:11:11.2034091Z NO_TEST_TIMEOUT: False 2024-06-26T07:11:11.2034432Z NO_TD: False 2024-06-26T07:11:11.2034737Z TD_DISTRIBUTED: False 2024-06-26T07:11:11.2035152Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T07:11:11.2035674Z SCCACHE_S3_KEY_PREFIX: inductor-periodic 2024-06-26T07:11:11.2036106Z SHM_SIZE: 2g 2024-06-26T07:11:11.2037186Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:11:11.2038343Z XLA_CUDA: 2024-06-26T07:11:11.2038829Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:11:11.2039443Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T07:11:11.2039884Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T07:11:11.2040298Z DASHBOARD_TAG: 2024-06-26T07:11:11.2040825Z HUGGING_FACE_HUB_TOKEN: *** 2024-06-26T07:11:11.2041200Z ##[endgroup] 2024-06-26T07:11:11.2062989Z + [[ dynamic_aot_eager_huggingface == \m\u\l\t\i\g\p\u ]] 2024-06-26T07:11:11.2063894Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-06-26T07:11:11.2064415Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:11:11.2069972Z +++ nproc --ignore=2 2024-06-26T07:11:11.2082314Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9673646806 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:11:22.4259544Z + container_name=240afe0ee980a530f2b3b6c166f41eee90a2a7b5c98112b59afbe2b10b264efa 2024-06-26T07:11:22.4260646Z + grep download.pytorch.org /etc/hosts 2024-06-26T07:11:22.4268253Z + docker exec -i 240afe0ee980a530f2b3b6c166f41eee90a2a7b5c98112b59afbe2b10b264efa sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-26T07:11:22.4827487Z + echo DOCKER_CONTAINER_ID=240afe0ee980a530f2b3b6c166f41eee90a2a7b5c98112b59afbe2b10b264efa 2024-06-26T07:11:22.4829934Z ++ echo dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:11:22.4832545Z + docker exec -t 240afe0ee980a530f2b3b6c166f41eee90a2a7b5c98112b59afbe2b10b264efa sh -c 'pip install dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-26T07:11:22.8561831Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb8c4c54) 2024-06-26T07:11:23.1708711Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T07:11:23.1712616Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T07:11:23.1714731Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T07:11:23.1716966Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T07:11:23.1719828Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T07:11:23.1722620Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T07:11:23.1749092Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.3.0) 2024-06-26T07:11:23.1809393Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.21.2) 2024-06-26T07:11:23.2239520Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T07:11:23.2399879Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T07:11:24.5481174Z Installing collected packages: torch 2024-06-26T07:11:33.2633020Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-06-26T07:11:33.2634604Z timm 0.9.7 requires torchvision, which is not installed. 2024-06-26T07:11:33.2635284Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T07:11:33.3489013Z ++ dirname .ci/pytorch/test.sh 2024-06-26T07:11:33.3493208Z + source .ci/pytorch/common.sh 2024-06-26T07:11:33.3495364Z +++ dirname .ci/pytorch/common.sh 2024-06-26T07:11:33.3500733Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T07:11:33.3501856Z +++ declare -f -t trap_add 2024-06-26T07:11:33.3507913Z ++ set -ex 2024-06-26T07:11:33.3508588Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:11:33.3509131Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T07:11:33.3509630Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-06-26T07:11:33.3511897Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-26T07:11:33.3519885Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-26T07:11:33.3520400Z + trap_add cleanup_workspace EXIT 2024-06-26T07:11:33.3520922Z + trap_add_cmd=cleanup_workspace 2024-06-26T07:11:33.3521366Z + shift 2024-06-26T07:11:33.3521679Z + for trap_add_name in "$@" 2024-06-26T07:11:33.3526127Z +++ trap -p EXIT 2024-06-26T07:11:33.3527629Z ++ eval 'extract_trap_cmd ' 2024-06-26T07:11:33.3528024Z +++ extract_trap_cmd 2024-06-26T07:11:33.3528400Z +++ printf '%s\n' '' 2024-06-26T07:11:33.3528805Z ++ printf '%s\n' cleanup_workspace 2024-06-26T07:11:33.3530386Z + trap -- ' 2024-06-26T07:11:33.3530727Z cleanup_workspace' EXIT 2024-06-26T07:11:33.3531204Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-26T07:11:33.9260367Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-26T07:11:33.9271086Z + echo 'Environment variables:' 2024-06-26T07:11:33.9271501Z Environment variables: 2024-06-26T07:11:33.9271838Z + env 2024-06-26T07:11:33.9276788Z INSTALLED_DB=yes 2024-06-26T07:11:33.9277186Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:11:33.9277588Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:11:33.9277987Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:11:33.9278594Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:11:33.9279216Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:11:33.9279885Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:11:33.9280537Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:11:33.9281188Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:11:33.9281737Z HOSTNAME=240afe0ee980 2024-06-26T07:11:33.9282692Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_9ba538fc-c80a-4cec-ade8-eb5af453966f 2024-06-26T07:11:33.9283514Z GITHUB_ACTION=__self 2024-06-26T07:11:33.9283928Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:11:33.9288625Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:11:33.9293310Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:11:33.9293827Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:11:33.9294184Z GITHUB_RUN_NUMBER=45567 2024-06-26T07:11:33.9294572Z TEST_CONFIG=dynamic_aot_eager_huggingface 2024-06-26T07:11:33.9295028Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:11:33.9295545Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:11:33.9296014Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:11:33.9296461Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:11:33.9296891Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:11:33.9297342Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:11:33.9297845Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:11:33.9298312Z GITHUB_REF_TYPE=tag 2024-06-26T07:11:33.9298646Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:11:33.9299035Z NCCL_VERSION=2.17.1-1 2024-06-26T07:11:33.9299732Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:33.9300196Z XLA_CUDA= 2024-06-26T07:11:33.9300708Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:11:33.9302770Z *** 2024-06-26T07:11:33.9303083Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:11:33.9303486Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:11:33.9303861Z GITHUB_ACTIONS=true 2024-06-26T07:11:33.9304206Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:33.9304758Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:11:33.9305301Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:11:33.9305772Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:33.9306274Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:11:33.9306959Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:33.9307934Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:11:33.9308700Z UCC_HOME=/usr 2024-06-26T07:11:33.9309045Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:11:33.9309611Z VERBOSE_TEST_LOGS=False 2024-06-26T07:11:33.9309963Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:11:33.9310425Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:11:33.9310930Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:11:33.9311394Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:11:33.9311778Z SHARD_NUMBER=1 2024-06-26T07:11:33.9312100Z GITHUB_REF_PROTECTED=false 2024-06-26T07:11:33.9312453Z HOME=/var/lib/jenkins 2024-06-26T07:11:33.9312829Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:11:33.9313277Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:11:33.9313758Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:11:33.9314354Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-06-26T07:11:33.9314782Z CUDA_VERSION=12.1.1 2024-06-26T07:11:33.9315206Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:11:33.9315649Z NUM_TEST_SHARDS=1 2024-06-26T07:11:33.9315953Z UCX_HOME=/usr 2024-06-26T07:11:33.9316473Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:11:33.9317556Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_9ba538fc-c80a-4cec-ade8-eb5af453966f 2024-06-26T07:11:33.9319010Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:33.9320415Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_9ba538fc-c80a-4cec-ade8-eb5af453966f 2024-06-26T07:11:33.9321532Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:11:33.9322189Z GITHUB_EVENT_NAME=push 2024-06-26T07:11:33.9322528Z DASHBOARD_TAG= 2024-06-26T07:11:33.9322831Z GITHUB_RUN_ID=9673646806 2024-06-26T07:11:33.9323291Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:11:33.9323829Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:11:33.9324823Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_9ba538fc-c80a-4cec-ade8-eb5af453966f 2024-06-26T07:11:33.9325732Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:11:33.9326252Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:11:33.9326637Z PR_NUMBER= 2024-06-26T07:11:33.9326928Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:11:33.9327272Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:11:33.9327722Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:11:33.9328190Z TERM=xterm 2024-06-26T07:11:33.9328525Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:11:33.9328945Z INSTALLED_VISION=yes 2024-06-26T07:11:33.9329271Z BRANCH= 2024-06-26T07:11:33.9329565Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:11:33.9329965Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:11:33.9330395Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:11:33.9331145Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:11:33.9331903Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:11:33.9332495Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:11:33.9332977Z REENABLED_ISSUES= 2024-06-26T07:11:33.9333283Z SHLVL=1 2024-06-26T07:11:33.9333547Z MAX_JOBS=14 2024-06-26T07:11:33.9333873Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:11:33.9334281Z NVARCH=x86_64 2024-06-26T07:11:33.9334587Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:11:33.9335043Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:33.9335581Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:11:33.9336071Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:11:33.9336730Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:11:33.9337302Z GITHUB_JOB=test 2024-06-26T07:11:33.9337705Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:11:33.9338266Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:11:33.9338772Z NO_TEST_TIMEOUT=False 2024-06-26T07:11:33.9339106Z TD_DISTRIBUTED=False 2024-06-26T07:11:33.9339491Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:11:33.9340005Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:11:33.9340425Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:11:33.9340801Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:11:33.9341158Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:11:33.9341515Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:11:33.9342551Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:33.9343621Z GITHUB_BASE_REF= 2024-06-26T07:11:33.9343964Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:11:33.9344387Z CI=true 2024-06-26T07:11:33.9344699Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:11:33.9345124Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:11:33.9345508Z JOB_ID=26688917205 2024-06-26T07:11:33.9345821Z INSTALLED_PROTOBUF=yes 2024-06-26T07:11:33.9346157Z GITHUB_HEAD_REF= 2024-06-26T07:11:33.9346468Z GITHUB_ACTION_REF= 2024-06-26T07:11:33.9347144Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:11:33.9347679Z GITHUB_WORKFLOW=inductor-periodic 2024-06-26T07:11:33.9348095Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:11:33.9348997Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_9ba538fc-c80a-4cec-ade8-eb5af453966f 2024-06-26T07:11:33.9349817Z NO_TD=False 2024-06-26T07:11:33.9350122Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:11:33.9350497Z _=/usr/bin/env 2024-06-26T07:11:33.9350975Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T07:11:33.9421955Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-26T07:11:33.9423187Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T07:11:33.9424240Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-26T07:11:33.9425119Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-26T07:11:33.9425945Z + BUILD_DIR=build 2024-06-26T07:11:33.9426357Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T07:11:33.9427051Z + BUILD_BIN_DIR=build/bin 2024-06-26T07:11:33.9427413Z + SHARD_NUMBER=1 2024-06-26T07:11:33.9427731Z + NUM_TEST_SHARDS=1 2024-06-26T07:11:33.9428065Z + export VALGRIND=ON 2024-06-26T07:11:33.9428394Z + VALGRIND=ON 2024-06-26T07:11:33.9428875Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-06-26T07:11:33.9429374Z + [[ 0 == \1 ]] 2024-06-26T07:11:33.9429681Z + [[ False == \1 ]] 2024-06-26T07:11:33.9430159Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-06-26T07:11:33.9430700Z ++ realpath build/custom_test_artifacts 2024-06-26T07:11:33.9432844Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-26T07:11:33.9433603Z + [[ -n '' ]] 2024-06-26T07:11:33.9433960Z + echo 'Environment variables' 2024-06-26T07:11:33.9434399Z Environment variables 2024-06-26T07:11:33.9434717Z + env 2024-06-26T07:11:33.9437551Z INSTALLED_DB=yes 2024-06-26T07:11:33.9438302Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:11:33.9438863Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:11:33.9439380Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:11:33.9440190Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:11:33.9440817Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:11:33.9441314Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:11:33.9441831Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:11:33.9442527Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:11:33.9443122Z HOSTNAME=240afe0ee980 2024-06-26T07:11:33.9444301Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_9ba538fc-c80a-4cec-ade8-eb5af453966f 2024-06-26T07:11:33.9445385Z GITHUB_ACTION=__self 2024-06-26T07:11:33.9445857Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:11:33.9449832Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:11:33.9453789Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:11:33.9454308Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:11:33.9454679Z GITHUB_RUN_NUMBER=45567 2024-06-26T07:11:33.9455062Z TEST_CONFIG=dynamic_aot_eager_huggingface 2024-06-26T07:11:33.9455517Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:11:33.9455996Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:11:33.9456481Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:11:33.9456927Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:11:33.9457361Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:11:33.9457812Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:11:33.9458325Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:11:33.9458796Z GITHUB_REF_TYPE=tag 2024-06-26T07:11:33.9459133Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:11:33.9478206Z NCCL_VERSION=2.17.1-1 2024-06-26T07:11:33.9478642Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:33.9479098Z XLA_CUDA= 2024-06-26T07:11:33.9479567Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:11:33.9479956Z *** 2024-06-26T07:11:33.9480239Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:11:33.9480628Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:11:33.9480999Z GITHUB_ACTIONS=true 2024-06-26T07:11:33.9481337Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:33.9481821Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:11:33.9482364Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:11:33.9482834Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:33.9483338Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:11:33.9483822Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:33.9484747Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:11:33.9485507Z UCC_HOME=/usr 2024-06-26T07:11:33.9485843Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:11:33.9486336Z VERBOSE_TEST_LOGS=False 2024-06-26T07:11:33.9486668Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:11:33.9487120Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:11:33.9487621Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:11:33.9488082Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:11:33.9488466Z SHARD_NUMBER=1 2024-06-26T07:11:33.9488783Z GITHUB_REF_PROTECTED=false 2024-06-26T07:11:33.9489139Z HOME=/var/lib/jenkins 2024-06-26T07:11:33.9489664Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:11:33.9490107Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:11:33.9490571Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:11:33.9491110Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-06-26T07:11:33.9491516Z CUDA_VERSION=12.1.1 2024-06-26T07:11:33.9491932Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:11:33.9492373Z NUM_TEST_SHARDS=1 2024-06-26T07:11:33.9492674Z UCX_HOME=/usr 2024-06-26T07:11:33.9493210Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:11:33.9494275Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_9ba538fc-c80a-4cec-ade8-eb5af453966f 2024-06-26T07:11:33.9495720Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:33.9497134Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_9ba538fc-c80a-4cec-ade8-eb5af453966f 2024-06-26T07:11:33.9498335Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:11:33.9498979Z GITHUB_EVENT_NAME=push 2024-06-26T07:11:33.9499314Z DASHBOARD_TAG= 2024-06-26T07:11:33.9499615Z GITHUB_RUN_ID=9673646806 2024-06-26T07:11:33.9500071Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:11:33.9500596Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:11:33.9501584Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_9ba538fc-c80a-4cec-ade8-eb5af453966f 2024-06-26T07:11:33.9502484Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:11:33.9502889Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:11:33.9503259Z PR_NUMBER= 2024-06-26T07:11:33.9503543Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:11:33.9503864Z VALGRIND=ON 2024-06-26T07:11:33.9504160Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:11:33.9504603Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:11:33.9505062Z TERM=xterm 2024-06-26T07:11:33.9505411Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:11:33.9505818Z INSTALLED_VISION=yes 2024-06-26T07:11:33.9506132Z BRANCH= 2024-06-26T07:11:33.9506421Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:11:33.9507065Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:11:33.9507494Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:11:33.9508244Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:11:33.9508993Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:11:33.9509475Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:11:33.9509942Z REENABLED_ISSUES= 2024-06-26T07:11:33.9510240Z SHLVL=1 2024-06-26T07:11:33.9510504Z MAX_JOBS=14 2024-06-26T07:11:33.9510810Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:11:33.9511167Z NVARCH=x86_64 2024-06-26T07:11:33.9511467Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:11:33.9511925Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:33.9512458Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:11:33.9512948Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:11:33.9513603Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:11:33.9514164Z GITHUB_JOB=test 2024-06-26T07:11:33.9514558Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:11:33.9515113Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:11:33.9515625Z NO_TEST_TIMEOUT=False 2024-06-26T07:11:33.9515946Z TD_DISTRIBUTED=False 2024-06-26T07:11:33.9516323Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:11:33.9516762Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:11:33.9517187Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:11:33.9517546Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:11:33.9517897Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:11:33.9518251Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:11:33.9519390Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:33.9520449Z GITHUB_BASE_REF= 2024-06-26T07:11:33.9520779Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:11:33.9521149Z CI=true 2024-06-26T07:11:33.9521458Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:11:33.9521877Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:11:33.9522252Z JOB_ID=26688917205 2024-06-26T07:11:33.9522571Z INSTALLED_PROTOBUF=yes 2024-06-26T07:11:33.9522896Z GITHUB_HEAD_REF= 2024-06-26T07:11:33.9523207Z GITHUB_ACTION_REF= 2024-06-26T07:11:33.9523635Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:11:33.9524155Z GITHUB_WORKFLOW=inductor-periodic 2024-06-26T07:11:33.9524558Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:11:33.9525460Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_9ba538fc-c80a-4cec-ade8-eb5af453966f 2024-06-26T07:11:33.9526326Z NO_TD=False 2024-06-26T07:11:33.9526636Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:11:33.9526994Z _=/usr/bin/env 2024-06-26T07:11:33.9527429Z + echo 'Testing pytorch' 2024-06-26T07:11:33.9527776Z Testing pytorch 2024-06-26T07:11:33.9528094Z + export LANG=C.UTF-8 2024-06-26T07:11:33.9528435Z + LANG=C.UTF-8 2024-06-26T07:11:33.9528730Z + PR_NUMBER= 2024-06-26T07:11:33.9529110Z + [[ dynamic_aot_eager_huggingface == \d\e\f\a\u\l\t ]] 2024-06-26T07:11:33.9529697Z + [[ dynamic_aot_eager_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T07:11:33.9530266Z + [[ dynamic_aot_eager_huggingface == \s\l\o\w ]] 2024-06-26T07:11:33.9530927Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-06-26T07:11:33.9531612Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-06-26T07:11:33.9532153Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:11:33.9532630Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:11:33.9533110Z + [[ dynamic_aot_eager_huggingface == *crossref* ]] 2024-06-26T07:11:33.9533713Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:11:33.9534394Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-06-26T07:11:33.9535044Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:11:33.9535600Z + pip_install --user ninja==1.10.2 2024-06-26T07:11:33.9536121Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T07:11:34.3077531Z Collecting ninja==1.10.2 2024-06-26T07:11:34.3252300Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T07:11:34.3400811Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T07:11:35.6255191Z Installing collected packages: ninja 2024-06-26T07:11:35.6321359Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T07:11:35.6322611Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T07:11:35.6374279Z Successfully installed ninja-1.10.2 2024-06-26T07:11:35.7257995Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:35.7260501Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:35.7261909Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:11:35.7262421Z + install_tlparse 2024-06-26T07:11:35.7262807Z + pip_install --user tlparse==0.3.7 2024-06-26T07:11:35.7263356Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T07:11:36.0811267Z Collecting tlparse==0.3.7 2024-06-26T07:11:36.0978975Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T07:11:36.1106020Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T07:11:37.4396516Z Installing collected packages: tlparse 2024-06-26T07:11:37.4756761Z Successfully installed tlparse-0.3.7 2024-06-26T07:11:37.5634118Z ++ python -m site --user-base 2024-06-26T07:11:37.5784734Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:37.5786568Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-06-26T07:11:37.5787474Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-06-26T07:11:37.5788323Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:11:37.5789319Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-06-26T07:11:37.5790549Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-06-26T07:11:37.5791248Z + cd test 2024-06-26T07:11:37.5791794Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T07:11:39.0564118Z + [[ dynamic_aot_eager_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T07:11:39.0564959Z + [[ dynamic_aot_eager_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T07:11:39.0565861Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:11:39.0566505Z + pushd test 2024-06-26T07:11:39.0566838Z ~/workspace/test ~/workspace 2024-06-26T07:11:39.0567366Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T07:11:40.5532198Z + CUDA_VERSION=12.1 2024-06-26T07:11:40.5532830Z + '[' 12.1 == 12.4 ']' 2024-06-26T07:11:40.5533214Z + ISCUDA124= 2024-06-26T07:11:40.5533516Z + popd 2024-06-26T07:11:40.5533841Z ~/workspace 2024-06-26T07:11:40.5537253Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T07:11:40.5537887Z + [[ dynamic_aot_eager_huggingface == *dynamo_eager* ]] 2024-06-26T07:11:40.5538535Z + [[ dynamic_aot_eager_huggingface == *aot_eager* ]] 2024-06-26T07:11:40.5539193Z + DYNAMO_BENCHMARK_FLAGS+=(--backend aot_eager) 2024-06-26T07:11:40.5539769Z + [[ dynamic_aot_eager_huggingface == *dynamic* ]] 2024-06-26T07:11:40.5540496Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2024-06-26T07:11:40.5541166Z + [[ dynamic_aot_eager_huggingface == *cpu_inductor* ]] 2024-06-26T07:11:40.5541785Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T07:11:40.5565716Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:11:40.5566606Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-06-26T07:11:40.5568174Z + cd test 2024-06-26T07:11:40.5568647Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T07:11:41.9051200Z PyTorch built with: 2024-06-26T07:11:41.9051896Z - GCC 9.4 2024-06-26T07:11:41.9052383Z - C++ Version: 201703 2024-06-26T07:11:41.9053254Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:11:41.9054355Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:11:41.9055024Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:11:41.9055534Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T07:11:41.9056025Z - NNPACK is enabled 2024-06-26T07:11:41.9056448Z - CPU capability usage: AVX2 2024-06-26T07:11:41.9056843Z - CUDA Runtime 12.1 2024-06-26T07:11:41.9057361Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-06-26T07:11:41.9057958Z - CuDNN 90.1 (built against CUDA 12.4) 2024-06-26T07:11:41.9058405Z - Magma 2.6.1 2024-06-26T07:11:41.9065511Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T07:11:41.9072244Z 2024-06-26T07:11:42.1796939Z + cd test 2024-06-26T07:11:42.1797715Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T07:11:43.3947515Z ATen/Parallel: 2024-06-26T07:11:43.3948185Z at::get_num_threads() : 8 2024-06-26T07:11:43.3948653Z at::get_num_interop_threads() : 16 2024-06-26T07:11:43.3949100Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:11:43.3949495Z omp_get_max_threads() : 8 2024-06-26T07:11:43.3950461Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:11:43.3951256Z mkl_get_max_threads() : 8 2024-06-26T07:11:43.3951882Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:11:43.3952509Z std::thread::hardware_concurrency() : 16 2024-06-26T07:11:43.3952951Z Environment variables: 2024-06-26T07:11:43.3953308Z OMP_NUM_THREADS : [not set] 2024-06-26T07:11:43.3953693Z MKL_NUM_THREADS : [not set] 2024-06-26T07:11:43.3954074Z ATen parallel backend: OpenMP 2024-06-26T07:11:43.3954337Z 2024-06-26T07:11:43.6387988Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:11:43.6388710Z + [[ dynamic_aot_eager_huggingface == *backward* ]] 2024-06-26T07:11:43.6389247Z + [[ dynamic_aot_eager_huggingface == *xla* ]] 2024-06-26T07:11:43.6389774Z + [[ dynamic_aot_eager_huggingface == *executorch* ]] 2024-06-26T07:11:43.6390413Z + [[ dynamic_aot_eager_huggingface == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T07:11:43.6391092Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:11:43.6391668Z + [[ dynamic_aot_eager_huggingface == distributed ]] 2024-06-26T07:11:43.6392248Z + [[ dynamic_aot_eager_huggingface == *inductor_distributed* ]] 2024-06-26T07:11:43.6392906Z + [[ dynamic_aot_eager_huggingface == *inductor-halide* ]] 2024-06-26T07:11:43.6393594Z + [[ dynamic_aot_eager_huggingface == *inductor-micro-benchmark* ]] 2024-06-26T07:11:43.6394212Z + [[ dynamic_aot_eager_huggingface == *huggingface* ]] 2024-06-26T07:11:43.6394682Z + install_torchvision 2024-06-26T07:11:43.6395019Z + local orig_preload 2024-06-26T07:11:43.6395344Z + local commit 2024-06-26T07:11:43.6395659Z ++ get_pinned_commit vision 2024-06-26T07:11:43.6396056Z ++ cat .github/ci_commit_pins/vision.txt 2024-06-26T07:11:43.6402910Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:43.6403379Z + orig_preload= 2024-06-26T07:11:43.6403743Z + '[' -n '' ']' 2024-06-26T07:11:43.6404554Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:43.6406088Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:43.9551035Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:43.9555304Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-jk9sgges 2024-06-26T07:11:43.9573696Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-jk9sgges 2024-06-26T07:11:45.3616884Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-06-26T07:11:45.3637442Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:46.5899047Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:46.8109959Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:49.1220411Z Preparing metadata (setup.py) ... [?25l- \ done 2024-06-26T07:11:49.1268638Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-06-26T07:11:49.1272030Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+gitb8c4c54) 2024-06-26T07:11:49.1277507Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-06-26T07:11:49.1482276Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-06-26T07:11:49.1488680Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-06-26T07:11:49.1491549Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1) 2024-06-26T07:11:49.1494285Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-06-26T07:11:49.1496876Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-06-26T07:11:49.1499608Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-06-26T07:11:49.1933293Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-06-26T07:11:49.2085806Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-06-26T07:11:49.2160071Z Building wheels for collected packages: torchvision 2024-06-26T07:13:03.1496679Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-06-26T07:13:03.1522452Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2112844 sha256=d7bcfe2835777c0fa3da9551012ced0bc672d7d0f77ee0d85c6ad9e4ce6cf3f5 2024-06-26T07:13:03.1525213Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-06-26T07:13:03.1559779Z Successfully built torchvision 2024-06-26T07:13:04.3710533Z Installing collected packages: torchvision 2024-06-26T07:13:04.7548735Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-06-26T07:13:04.8679133Z + '[' -n '' ']' 2024-06-26T07:13:04.8680849Z + id=0 2024-06-26T07:13:04.8681614Z + test_dynamo_benchmark huggingface 0 2024-06-26T07:13:04.8682187Z ++ pwd 2024-06-26T07:13:04.8684752Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:13:04.8685582Z + local suite=huggingface 2024-06-26T07:13:04.8686178Z + shift 2024-06-26T07:13:04.8686580Z + local shard_id=0 2024-06-26T07:13:04.8686954Z + shift 2024-06-26T07:13:04.8688912Z + [[ dynamic_aot_eager_huggingface == *perf_compare* ]] 2024-06-26T07:13:04.8689727Z + [[ dynamic_aot_eager_huggingface == *perf* ]] 2024-06-26T07:13:04.8690710Z + [[ dynamic_aot_eager_huggingface == *cpu_inductor* ]] 2024-06-26T07:13:04.8691327Z + [[ dynamic_aot_eager_huggingface == *aot_inductor* ]] 2024-06-26T07:13:04.8692099Z + test_single_dynamo_benchmark inference huggingface 0 --inference --bfloat16 2024-06-26T07:13:04.8692693Z ++ pwd 2024-06-26T07:13:04.8693166Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:13:04.8693827Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:13:04.8701082Z + local name=inference 2024-06-26T07:13:04.8701564Z + shift 2024-06-26T07:13:04.8701947Z + local suite=huggingface 2024-06-26T07:13:04.8702413Z + shift 2024-06-26T07:13:04.8702735Z + local shard_id=0 2024-06-26T07:13:04.8704003Z + shift 2024-06-26T07:13:04.8704327Z + partition_flags=() 2024-06-26T07:13:04.8704707Z + local partition_flags 2024-06-26T07:13:04.8705122Z + [[ -n 1 ]] 2024-06-26T07:13:04.8705454Z + [[ -n 0 ]] 2024-06-26T07:13:04.8706069Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-06-26T07:13:04.8707196Z + [[ dynamic_aot_eager_huggingface == *perf_compare* ]] 2024-06-26T07:13:04.8707740Z + [[ dynamic_aot_eager_huggingface == *perf* ]] 2024-06-26T07:13:04.8708277Z + [[ dynamic_aot_eager_huggingface == *aot_inductor* ]] 2024-06-26T07:13:04.8710179Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --backend aot_eager --dynamic-shapes --dynamic-batch-only --device cuda --inference --bfloat16 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv 2024-06-26T07:13:07.5426326Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:13:07.5428082Z warnings.warn( 2024-06-26T07:13:07.5893397Z 2024-06-26T07:13:07.5893932Z config.json: 0% 0.00/694 [00:00 will be ignored 2024-06-26T07:29:42.2520678Z pass 2024-06-26T07:29:42.3065545Z TIMING: entire_frame_compile:10.8202 backend_compile:7.19421 2024-06-26T07:29:42.3068039Z STATS: call_* op count: 534 | FakeTensorMode.__torch_dispatch__:35947 | FakeTensor.__torch_dispatch__:1758 | ProxyTorchDispatchMode.__torch_dispatch__:14646 2024-06-26T07:29:42.3069182Z Dynamo produced 2 graphs covering 534 ops with 5 graph breaks (4 unique) 2024-06-26T07:29:45.8307303Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:29:45.8310151Z warnings.warn( 2024-06-26T07:29:46.0316320Z 2024-06-26T07:29:48.9312828Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:29:48.9313297Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:29:48.9313767Z cuda train AllenaiLongformerBase 2024-06-26T07:29:56.4189220Z W0626 07:29:56.418000 139878623429248 torch/_dynamo/variables/tensor.py:714] [3/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T07:29:56.4190598Z W0626 07:29:56.418000 139878623429248 torch/_dynamo/variables/tensor.py:714] [3/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T07:29:56.4192011Z W0626 07:29:56.418000 139878623429248 torch/_dynamo/variables/tensor.py:714] [3/0] or: 2024-06-26T07:29:56.4193397Z W0626 07:29:56.418000 139878623429248 torch/_dynamo/variables/tensor.py:714] [3/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T07:29:56.4195076Z W0626 07:29:56.418000 139878623429248 torch/_dynamo/variables/tensor.py:714] [3/0] to include these operations in the captured graph. 2024-06-26T07:29:56.4196152Z W0626 07:29:56.418000 139878623429248 torch/_dynamo/variables/tensor.py:714] [3/0] 2024-06-26T07:30:26.9717724Z W0626 07:30:26.970000 139878623429248 torch/_logging/_internal.py:1040] [16/0] Profiler function will be ignored 2024-06-26T07:30:48.3717045Z pass 2024-06-26T07:30:48.4351899Z TIMING: entire_frame_compile:52.40203 backend_compile:34.81838 2024-06-26T07:30:48.4354189Z STATS: call_* op count: 2367 | FakeTensorMode.__torch_dispatch__:116429 | FakeTensor.__torch_dispatch__:4625 | ProxyTorchDispatchMode.__torch_dispatch__:48592 2024-06-26T07:30:48.4355353Z Dynamo produced 7 graphs covering 2367 ops with 9 graph breaks (5 unique) 2024-06-26T07:30:53.0288464Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:30:53.0290518Z warnings.warn( 2024-06-26T07:30:53.2188648Z 2024-06-26T07:30:56.8307687Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:30:56.8308219Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:30:56.8308698Z cuda train BartForCausalLM 2024-06-26T07:31:12.0987714Z pass 2024-06-26T07:31:12.2453519Z TIMING: entire_frame_compile:7.47534 backend_compile:5.7513 2024-06-26T07:31:12.2454619Z STATS: call_* op count: 325 | FakeTensorMode.__torch_dispatch__:26169 | FakeTensor.__torch_dispatch__:847 | ProxyTorchDispatchMode.__torch_dispatch__:11407 2024-06-26T07:31:12.2456520Z Dynamo produced 16 graphs covering 325 ops with 6 graph breaks (5 unique) 2024-06-26T07:31:15.7635373Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:31:15.7637100Z warnings.warn( 2024-06-26T07:31:15.9796522Z 2024-06-26T07:31:22.4988962Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:31:22.4989542Z loading model: 0it [00:06, ?it/s] 2024-06-26T07:31:22.4990069Z cuda train BartForConditionalGeneration 2024-06-26T07:31:57.4246388Z pass 2024-06-26T07:31:57.6727033Z TIMING: entire_frame_compile:18.64018 backend_compile:14.40914 2024-06-26T07:31:57.6728194Z STATS: call_* op count: 878 | FakeTensorMode.__torch_dispatch__:67193 | FakeTensor.__torch_dispatch__:2172 | ProxyTorchDispatchMode.__torch_dispatch__:29836 2024-06-26T07:31:57.6731476Z Dynamo produced 30 graphs covering 878 ops with 8 graph breaks (5 unique) 2024-06-26T07:32:01.4758992Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:32:01.4760652Z warnings.warn( 2024-06-26T07:32:01.7224427Z 2024-06-26T07:32:03.5230054Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:32:03.5230543Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:32:03.5230999Z cuda train BertForMaskedLM 2024-06-26T07:32:17.8407993Z W0626 07:32:17.839000 140001780085376 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:32:37.6574709Z pass 2024-06-26T07:32:37.7774999Z TIMING: entire_frame_compile:30.95319 backend_compile:18.66477 2024-06-26T07:32:37.7780477Z STATS: call_* op count: 996 | FakeTensorMode.__torch_dispatch__:62011 | FakeTensor.__torch_dispatch__:5113 | ProxyTorchDispatchMode.__torch_dispatch__:21485 2024-06-26T07:32:37.7781627Z Dynamo produced 2 graphs covering 996 ops with 5 graph breaks (4 unique) 2024-06-26T07:32:41.7928451Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:32:41.7930104Z warnings.warn( 2024-06-26T07:32:41.9708416Z 2024-06-26T07:32:43.5445003Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:32:43.5446332Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:32:43.5447471Z cuda train BertForQuestionAnswering 2024-06-26T07:32:57.3790999Z W0626 07:32:57.378000 140028728476288 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:33:16.9941509Z pass 2024-06-26T07:33:17.1103108Z TIMING: entire_frame_compile:30.61288 backend_compile:18.1798 2024-06-26T07:33:17.1108762Z STATS: call_* op count: 994 | FakeTensorMode.__torch_dispatch__:61510 | FakeTensor.__torch_dispatch__:5037 | ProxyTorchDispatchMode.__torch_dispatch__:21302 2024-06-26T07:33:17.1110823Z Dynamo produced 2 graphs covering 994 ops with 5 graph breaks (4 unique) 2024-06-26T07:33:21.1149171Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:33:21.1151867Z warnings.warn( 2024-06-26T07:33:21.3073473Z 2024-06-26T07:33:48.3939723Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:33:48.3942225Z loading model: 0it [00:27, ?it/s] 2024-06-26T07:33:48.3943309Z cuda train BlenderbotForCausalLM 2024-06-26T07:33:48.3947188Z Traceback (most recent call last): 2024-06-26T07:33:48.3948560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T07:33:48.3949338Z self.model_iter_fn(model, example_inputs) 2024-06-26T07:33:48.3950309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in forward_and_backward_pass 2024-06-26T07:33:48.3951425Z pred = mod(**cloned_inputs) 2024-06-26T07:33:48.3952490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:33:48.3953548Z return self._call_impl(*args, **kwargs) 2024-06-26T07:33:48.3954835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:33:48.3955936Z return forward_call(*args, **kwargs) 2024-06-26T07:33:48.3957359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 1531, in forward 2024-06-26T07:33:48.3958322Z outputs = self.model.decoder( 2024-06-26T07:33:48.3959286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:33:48.3960193Z return self._call_impl(*args, **kwargs) 2024-06-26T07:33:48.3961115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:33:48.3961948Z return forward_call(*args, **kwargs) 2024-06-26T07:33:48.3962996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 997, in forward 2024-06-26T07:33:48.3963936Z layer_outputs = decoder_layer( 2024-06-26T07:33:48.3964860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:33:48.3965867Z return self._call_impl(*args, **kwargs) 2024-06-26T07:33:48.3968533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:33:48.3969671Z return forward_call(*args, **kwargs) 2024-06-26T07:33:48.3971136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 397, in forward 2024-06-26T07:33:48.3972267Z hidden_states, self_attn_weights, present_key_value = self.self_attn( 2024-06-26T07:33:48.3973404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:33:48.3974312Z return self._call_impl(*args, **kwargs) 2024-06-26T07:33:48.3975242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:33:48.3976083Z return forward_call(*args, **kwargs) 2024-06-26T07:33:48.3977143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 152, in forward 2024-06-26T07:33:48.3978189Z query_states = self.q_proj(hidden_states) * self.scaling 2024-06-26T07:33:48.3979246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:33:48.3980273Z return self._call_impl(*args, **kwargs) 2024-06-26T07:33:48.3981201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:33:48.3982214Z return forward_call(*args, **kwargs) 2024-06-26T07:33:48.3983105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 125, in forward 2024-06-26T07:33:48.3983966Z return F.linear(input, self.weight, self.bias) 2024-06-26T07:33:48.3987392Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 14.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 6.44 MiB is free. Process 9926 has 21.96 GiB memory in use. Of the allocated memory 21.61 GiB is allocated by PyTorch, and 27.77 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T07:33:48.3990098Z 2024-06-26T07:33:48.3990419Z The above exception was the direct cause of the following exception: 2024-06-26T07:33:48.3990875Z 2024-06-26T07:33:48.3991040Z Traceback (most recent call last): 2024-06-26T07:33:48.3991699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T07:33:48.3992381Z ) = runner.load_model( 2024-06-26T07:33:48.3993061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 522, in load_model 2024-06-26T07:33:48.3993843Z self.validate_model(model, example_inputs) 2024-06-26T07:33:48.3994676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T07:33:48.3995447Z raise RuntimeError("Eager run failed") from e 2024-06-26T07:33:48.3995934Z RuntimeError: Eager run failed 2024-06-26T07:33:48.3996195Z 2024-06-26T07:33:48.3996337Z eager_fail_to_run 2024-06-26T07:33:51.7267118Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:33:51.7268764Z warnings.warn( 2024-06-26T07:33:51.9307145Z 2024-06-26T07:33:53.1964470Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:33:53.1965226Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:33:53.1966058Z cuda train BlenderbotSmallForCausalLM 2024-06-26T07:33:53.2188393Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForCausalLM. Setting accuracy check to cosine 2024-06-26T07:33:59.9438207Z pass 2024-06-26T07:33:59.9791091Z TIMING: entire_frame_compile:5.92616 backend_compile:4.3475 2024-06-26T07:33:59.9792253Z STATS: call_* op count: 324 | FakeTensorMode.__torch_dispatch__:19676 | FakeTensor.__torch_dispatch__:535 | ProxyTorchDispatchMode.__torch_dispatch__:8650 2024-06-26T07:33:59.9793748Z Dynamo produced 13 graphs covering 324 ops with 6 graph breaks (5 unique) 2024-06-26T07:34:03.4565786Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:34:03.4567523Z warnings.warn( 2024-06-26T07:34:03.6497042Z 2024-06-26T07:34:05.2997000Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:34:05.2997479Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:34:05.2997994Z cuda train BlenderbotSmallForConditionalGeneration 2024-06-26T07:34:05.4436785Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:34:20.5910360Z pass 2024-06-26T07:34:20.6277265Z TIMING: entire_frame_compile:13.79284 backend_compile:10.28835 2024-06-26T07:34:20.6278462Z STATS: call_* op count: 830 | FakeTensorMode.__torch_dispatch__:49002 | FakeTensor.__torch_dispatch__:1303 | ProxyTorchDispatchMode.__torch_dispatch__:22166 2024-06-26T07:34:20.6279822Z Dynamo produced 22 graphs covering 830 ops with 8 graph breaks (5 unique) 2024-06-26T07:34:24.3007576Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:34:24.3009212Z warnings.warn( 2024-06-26T07:34:24.4998111Z 2024-06-26T07:34:26.3527131Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:34:26.3527833Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:34:26.3528514Z cuda train CamemBert 2024-06-26T07:34:40.6306011Z W0626 07:34:40.629000 140568761311872 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:35:00.5759853Z pass 2024-06-26T07:35:00.6936416Z TIMING: entire_frame_compile:31.00636 backend_compile:18.67015 2024-06-26T07:35:00.6938813Z STATS: call_* op count: 1004 | FakeTensorMode.__torch_dispatch__:62113 | FakeTensor.__torch_dispatch__:5113 | ProxyTorchDispatchMode.__torch_dispatch__:21521 2024-06-26T07:35:00.6939951Z Dynamo produced 2 graphs covering 1004 ops with 5 graph breaks (4 unique) 2024-06-26T07:35:04.7409318Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:35:04.7411000Z warnings.warn( 2024-06-26T07:35:04.9147516Z 2024-06-26T07:35:07.0842426Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:35:07.0842905Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:35:07.0843363Z cuda train DebertaForMaskedLM 2024-06-26T07:35:22.5762834Z W0626 07:35:22.575000 139871045964416 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:35:38.7447142Z pass 2024-06-26T07:35:38.8545729Z TIMING: entire_frame_compile:28.32641 backend_compile:17.18552 2024-06-26T07:35:38.8547052Z STATS: call_* op count: 1319 | FakeTensorMode.__torch_dispatch__:62189 | FakeTensor.__torch_dispatch__:3744 | ProxyTorchDispatchMode.__torch_dispatch__:23799 2024-06-26T07:35:38.8548269Z Dynamo produced 2 graphs covering 1319 ops with 5 graph breaks (4 unique) 2024-06-26T07:35:42.8635446Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:35:42.8637179Z warnings.warn( 2024-06-26T07:35:43.0593993Z 2024-06-26T07:35:44.8551887Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:35:44.8553244Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:35:44.8554187Z cuda train DebertaForQuestionAnswering 2024-06-26T07:35:59.7409356Z W0626 07:35:59.740000 140385641656960 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:36:15.7883405Z pass 2024-06-26T07:36:15.9089102Z TIMING: entire_frame_compile:28.14056 backend_compile:17.07702 2024-06-26T07:36:15.9091286Z STATS: call_* op count: 1317 | FakeTensorMode.__torch_dispatch__:61688 | FakeTensor.__torch_dispatch__:3673 | ProxyTorchDispatchMode.__torch_dispatch__:23616 2024-06-26T07:36:15.9093517Z Dynamo produced 2 graphs covering 1317 ops with 5 graph breaks (4 unique) 2024-06-26T07:36:19.9101777Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:36:19.9103742Z warnings.warn( 2024-06-26T07:36:20.1955839Z 2024-06-26T07:36:32.1109651Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:36:32.1110280Z loading model: 0it [00:11, ?it/s] 2024-06-26T07:36:32.1113855Z cuda train DebertaV2ForMaskedLM 2024-06-26T07:36:32.1114364Z pass_due_to_skip 2024-06-26T07:36:32.3316106Z TIMING: 2024-06-26T07:36:32.3316731Z STATS: call_* op count: 0 2024-06-26T07:36:32.3317834Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T07:36:35.6281404Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:36:35.6283128Z warnings.warn( 2024-06-26T07:36:35.8266658Z 2024-06-26T07:36:45.7620571Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:36:45.7621781Z loading model: 0it [00:09, ?it/s] 2024-06-26T07:36:45.7622630Z cuda train DebertaV2ForQuestionAnswering 2024-06-26T07:36:56.3952844Z ERROR:common: 2024-06-26T07:36:56.3953330Z Traceback (most recent call last): 2024-06-26T07:36:56.3956492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2543, in check_accuracy 2024-06-26T07:36:56.3957443Z correct_result = self.run_n_iterations( 2024-06-26T07:36:56.3958228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2359, in run_n_iterations 2024-06-26T07:36:56.3959395Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-06-26T07:36:56.3960688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in forward_and_backward_pass 2024-06-26T07:36:56.3961793Z pred = mod(**cloned_inputs) 2024-06-26T07:36:56.3963201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:36:56.3964243Z return self._call_impl(*args, **kwargs) 2024-06-26T07:36:56.3965332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:36:56.3966307Z return forward_call(*args, **kwargs) 2024-06-26T07:36:56.3967361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1486, in forward 2024-06-26T07:36:56.3968295Z outputs = self.deberta( 2024-06-26T07:36:56.3969186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:36:56.3970069Z return self._call_impl(*args, **kwargs) 2024-06-26T07:36:56.3970981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:36:56.3971803Z return forward_call(*args, **kwargs) 2024-06-26T07:36:56.3973122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1070, in forward 2024-06-26T07:36:56.3974067Z encoder_outputs = self.encoder( 2024-06-26T07:36:56.3975004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:36:56.3975885Z return self._call_impl(*args, **kwargs) 2024-06-26T07:36:56.3976789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:36:56.3977615Z return forward_call(*args, **kwargs) 2024-06-26T07:36:56.3980780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 514, in forward 2024-06-26T07:36:56.3981811Z output_states = layer_module( 2024-06-26T07:36:56.3982781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:36:56.3983667Z return self._call_impl(*args, **kwargs) 2024-06-26T07:36:56.3984600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:36:56.3985596Z return forward_call(*args, **kwargs) 2024-06-26T07:36:56.3986643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 362, in forward 2024-06-26T07:36:56.3987934Z attention_output = self.attention( 2024-06-26T07:36:56.3988903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:36:56.3989789Z return self._call_impl(*args, **kwargs) 2024-06-26T07:36:56.3990717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:36:56.3991544Z return forward_call(*args, **kwargs) 2024-06-26T07:36:56.3992574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 293, in forward 2024-06-26T07:36:56.3993510Z self_output = self.self( 2024-06-26T07:36:56.3994416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:36:56.3995327Z return self._call_impl(*args, **kwargs) 2024-06-26T07:36:56.3996258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:36:56.3997078Z return forward_call(*args, **kwargs) 2024-06-26T07:36:56.3998113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 733, in forward 2024-06-26T07:36:56.3999268Z attention_probs = XSoftmax.apply(attention_scores, attention_mask, -1) 2024-06-26T07:36:56.4000295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 574, in apply 2024-06-26T07:36:56.4001186Z return super().apply(*args, **kwargs) # type: ignore[misc] 2024-06-26T07:36:56.4002327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 113, in forward 2024-06-26T07:36:56.4003291Z output = torch.softmax(output, self.dim) 2024-06-26T07:36:56.4006549Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 10.44 MiB is free. Process 10606 has 21.96 GiB memory in use. Of the allocated memory 21.29 GiB is allocated by PyTorch, and 318.25 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T07:36:56.4270158Z eager_1st_run_OOM 2024-06-26T07:36:56.4270557Z TIMING: 2024-06-26T07:36:56.4270842Z STATS: call_* op count: 0 2024-06-26T07:36:56.4271379Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T07:36:59.7223919Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:36:59.7225634Z warnings.warn( 2024-06-26T07:36:59.9080792Z 2024-06-26T07:37:01.2218989Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:37:01.2219614Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:37:01.2220078Z cuda train DistilBertForMaskedLM 2024-06-26T07:37:06.8102759Z W0626 07:37:06.809000 140232446833280 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:37:17.1127724Z pass 2024-06-26T07:37:17.1733282Z TIMING: entire_frame_compile:14.73921 backend_compile:9.40342 2024-06-26T07:37:17.1734408Z STATS: call_* op count: 541 | FakeTensor.__torch_dispatch__:2683 | FakeTensorMode.__torch_dispatch__:32380 | ProxyTorchDispatchMode.__torch_dispatch__:11147 2024-06-26T07:37:17.1735847Z Dynamo produced 2 graphs covering 541 ops with 5 graph breaks (4 unique) 2024-06-26T07:37:20.8481562Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:37:20.8483205Z warnings.warn( 2024-06-26T07:37:21.0980723Z 2024-06-26T07:37:22.1686178Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:37:22.1686898Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:37:22.1687583Z cuda train DistilBertForQuestionAnswering 2024-06-26T07:37:27.6634725Z W0626 07:37:27.662000 140663116726912 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:37:37.8389549Z pass 2024-06-26T07:37:37.9009147Z TIMING: entire_frame_compile:14.62937 backend_compile:9.30965 2024-06-26T07:37:37.9010309Z STATS: call_* op count: 540 | FakeTensor.__torch_dispatch__:2612 | FakeTensorMode.__torch_dispatch__:31927 | ProxyTorchDispatchMode.__torch_dispatch__:10978 2024-06-26T07:37:37.9011432Z Dynamo produced 2 graphs covering 540 ops with 5 graph breaks (4 unique) 2024-06-26T07:37:41.5697538Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:37:41.5699235Z warnings.warn( 2024-06-26T07:37:41.8032187Z 2024-06-26T07:37:43.5608147Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:37:43.5608635Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:37:43.5610750Z cuda train DistillGPT2 2024-06-26T07:37:50.5976886Z W0626 07:37:50.596000 140097758319232 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:37:58.1636545Z pass 2024-06-26T07:37:58.2210263Z TIMING: entire_frame_compile:12.52243 backend_compile:7.53879 2024-06-26T07:37:58.2211335Z STATS: call_* op count: 572 | FakeTensorMode.__torch_dispatch__:26704 | FakeTensor.__torch_dispatch__:1681 | ProxyTorchDispatchMode.__torch_dispatch__:9304 2024-06-26T07:37:58.2212457Z Dynamo produced 2 graphs covering 572 ops with 5 graph breaks (4 unique) 2024-06-26T07:38:01.8571286Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:38:01.8572972Z warnings.warn( 2024-06-26T07:38:02.0707009Z 2024-06-26T07:38:02.0715008Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2024-06-26T07:38:02.6781332Z 2024-06-26T07:38:02.6781794Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:38:02.6782332Z cuda train ElectraForCausalLM 2024-06-26T07:38:15.6163921Z pass 2024-06-26T07:38:15.6379377Z TIMING: entire_frame_compile:11.70903 backend_compile:6.9333 2024-06-26T07:38:15.6380572Z STATS: call_* op count: 377 | FakeTensorMode.__torch_dispatch__:32441 | FakeTensor.__torch_dispatch__:1252 | ProxyTorchDispatchMode.__torch_dispatch__:12488 2024-06-26T07:38:15.6384487Z Dynamo produced 1 graphs covering 377 ops with 4 graph breaks (4 unique) 2024-06-26T07:38:19.2776305Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:38:19.2777992Z warnings.warn( 2024-06-26T07:38:19.5033663Z 2024-06-26T07:38:20.1403731Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:38:20.1404956Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:38:20.1405688Z cuda train ElectraForQuestionAnswering 2024-06-26T07:38:32.7612220Z W0626 07:38:32.760000 139654770975360 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:38:52.6202275Z pass 2024-06-26T07:38:52.6381006Z TIMING: entire_frame_compile:31.01289 backend_compile:18.52631 2024-06-26T07:38:52.6382123Z STATS: call_* op count: 1001 | FakeTensorMode.__torch_dispatch__:62040 | FakeTensor.__torch_dispatch__:5102 | ProxyTorchDispatchMode.__torch_dispatch__:21467 2024-06-26T07:38:52.6384777Z Dynamo produced 2 graphs covering 1001 ops with 5 graph breaks (4 unique) 2024-06-26T07:38:56.6524941Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:38:56.6526843Z warnings.warn( 2024-06-26T07:38:56.8776119Z 2024-06-26T07:38:58.9451686Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:38:58.9452338Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:38:58.9452939Z cuda train GPT2ForSequenceClassification 2024-06-26T07:39:14.6985292Z W0626 07:39:14.697000 140671304716928 torch/_logging/_internal.py:1040] [7/0] Profiler function will be ignored 2024-06-26T07:39:29.1673095Z pass 2024-06-26T07:39:29.2806020Z TIMING: entire_frame_compile:25.6615 backend_compile:14.53746 2024-06-26T07:39:29.2808095Z STATS: call_* op count: 1102 | FakeTensorMode.__torch_dispatch__:55081 | FakeTensor.__torch_dispatch__:3413 | ProxyTorchDispatchMode.__torch_dispatch__:18351 2024-06-26T07:39:29.2809246Z Dynamo produced 5 graphs covering 1102 ops with 7 graph breaks (5 unique) 2024-06-26T07:39:33.1636923Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:39:33.1638645Z warnings.warn( 2024-06-26T07:39:33.3548960Z 2024-06-26T07:39:34.6569472Z loading model: 0it [00:00, ?it/s]WARNING:common:Model GoogleFnet supports float32 only 2024-06-26T07:39:34.8096521Z 2024-06-26T07:39:34.8096792Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:39:34.8101100Z WARNING:common:Model GoogleFnet supports float32 only 2024-06-26T07:39:34.8108743Z cuda train GoogleFnet 2024-06-26T07:39:36.1396865Z WARNING:common:Model GoogleFnet supports float32 only 2024-06-26T07:39:41.4449759Z W0626 07:39:41.444000 140027960603264 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:39:51.9993595Z pass 2024-06-26T07:39:52.0612910Z TIMING: entire_frame_compile:15.10847 backend_compile:8.85321 2024-06-26T07:39:52.0614257Z STATS: call_* op count: 574 | FakeTensorMode.__torch_dispatch__:29842 | FakeTensor.__torch_dispatch__:2775 | ProxyTorchDispatchMode.__torch_dispatch__:10146 2024-06-26T07:39:52.0615372Z Dynamo produced 2 graphs covering 574 ops with 5 graph breaks (4 unique) 2024-06-26T07:39:55.7151440Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:39:55.7153085Z warnings.warn( 2024-06-26T07:39:55.9229131Z 2024-06-26T07:39:57.7793657Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:39:57.7794139Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:39:57.7794586Z cuda train LayoutLMForMaskedLM 2024-06-26T07:40:12.2735235Z W0626 07:40:12.272000 140532760949376 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:40:32.6463351Z pass 2024-06-26T07:40:32.7733020Z TIMING: entire_frame_compile:31.65904 backend_compile:19.02924 2024-06-26T07:40:32.7735801Z STATS: call_* op count: 1034 | FakeTensorMode.__torch_dispatch__:63598 | FakeTensor.__torch_dispatch__:5242 | ProxyTorchDispatchMode.__torch_dispatch__:21936 2024-06-26T07:40:32.7736984Z Dynamo produced 2 graphs covering 1034 ops with 5 graph breaks (4 unique) 2024-06-26T07:40:36.8393388Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:40:36.8395034Z warnings.warn( 2024-06-26T07:40:37.0568371Z 2024-06-26T07:40:38.6698308Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:40:38.6698837Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:40:38.6701618Z cuda train LayoutLMForSequenceClassification 2024-06-26T07:40:55.1464764Z W0626 07:40:55.145000 140620099797632 torch/_logging/_internal.py:1040] [7/0] Profiler function will be ignored 2024-06-26T07:41:15.5197842Z pass 2024-06-26T07:41:15.6407906Z TIMING: entire_frame_compile:33.96891 backend_compile:18.88902 2024-06-26T07:41:15.6410322Z STATS: call_* op count: 1031 | FakeTensorMode.__torch_dispatch__:68019 | FakeTensor.__torch_dispatch__:5850 | ProxyTorchDispatchMode.__torch_dispatch__:22181 2024-06-26T07:41:15.6411507Z Dynamo produced 5 graphs covering 1031 ops with 7 graph breaks (5 unique) 2024-06-26T07:41:19.7107675Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:41:19.7109325Z warnings.warn( 2024-06-26T07:41:21.4120904Z 2024-06-26T07:41:31.3121293Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:41:31.3122933Z loading model: 0it [00:09, ?it/s] 2024-06-26T07:41:31.3123599Z cuda train M2M100ForConditionalGeneration 2024-06-26T07:41:31.3913270Z WARNING:common:fp64 golden ref were not generated for M2M100ForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:41:55.0181528Z pass 2024-06-26T07:41:55.2315690Z TIMING: entire_frame_compile:20.99961 backend_compile:15.4111 2024-06-26T07:41:55.2318456Z STATS: call_* op count: 1296 | FakeTensorMode.__torch_dispatch__:81067 | FakeTensor.__torch_dispatch__:3135 | ProxyTorchDispatchMode.__torch_dispatch__:32498 2024-06-26T07:41:55.2319588Z Dynamo produced 1 graphs covering 1296 ops with 4 graph breaks (4 unique) 2024-06-26T07:41:58.9969093Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:41:58.9970725Z warnings.warn( 2024-06-26T07:41:59.2093615Z 2024-06-26T07:42:02.8200681Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:42:02.8201406Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:42:02.8202092Z cuda train MBartForCausalLM 2024-06-26T07:42:02.8524428Z WARNING:common:fp64 golden ref were not generated for MBartForCausalLM. Setting accuracy check to cosine 2024-06-26T07:42:12.6350843Z pass 2024-06-26T07:42:12.7422918Z TIMING: entire_frame_compile:8.44839 backend_compile:6.28695 2024-06-26T07:42:12.7424019Z STATS: call_* op count: 478 | FakeTensorMode.__torch_dispatch__:28954 | FakeTensor.__torch_dispatch__:763 | ProxyTorchDispatchMode.__torch_dispatch__:12867 2024-06-26T07:42:12.7425145Z Dynamo produced 17 graphs covering 478 ops with 6 graph breaks (5 unique) 2024-06-26T07:42:16.3431878Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:42:16.3433878Z warnings.warn( 2024-06-26T07:42:16.6196233Z 2024-06-26T07:42:23.1838957Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:42:23.1839429Z loading model: 0it [00:06, ?it/s] 2024-06-26T07:42:23.1841800Z cuda train MBartForConditionalGeneration 2024-06-26T07:42:23.2862772Z WARNING:common:fp64 golden ref were not generated for MBartForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:42:47.1794977Z pass 2024-06-26T07:42:47.4019517Z TIMING: entire_frame_compile:20.4159 backend_compile:15.71859 2024-06-26T07:42:47.4020869Z STATS: call_* op count: 1251 | FakeTensorMode.__torch_dispatch__:73174 | FakeTensor.__torch_dispatch__:1873 | ProxyTorchDispatchMode.__torch_dispatch__:33239 2024-06-26T07:42:47.4024830Z Dynamo produced 31 graphs covering 1251 ops with 8 graph breaks (5 unique) 2024-06-26T07:42:51.2673164Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:42:51.2674913Z warnings.warn( 2024-06-26T07:42:51.5667599Z 2024-06-26T07:42:55.4040247Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:42:55.4040943Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:42:55.4041642Z cuda train MT5ForConditionalGeneration 2024-06-26T07:43:18.6659589Z W0626 07:43:18.665000 140065581191808 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:43:39.5674508Z pass 2024-06-26T07:43:39.8795963Z TIMING: entire_frame_compile:40.92496 backend_compile:25.33666 2024-06-26T07:43:39.8798546Z STATS: call_* op count: 1764 | FakeTensorMode.__torch_dispatch__:92032 | FakeTensor.__torch_dispatch__:4589 | ProxyTorchDispatchMode.__torch_dispatch__:36442 2024-06-26T07:43:39.8799696Z Dynamo produced 2 graphs covering 1764 ops with 5 graph breaks (4 unique) 2024-06-26T07:43:44.1895744Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:43:44.1897906Z warnings.warn( 2024-06-26T07:43:44.3741755Z 2024-06-26T07:43:44.3788336Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2024-06-26T07:43:48.8232212Z 2024-06-26T07:43:48.8235737Z loading model: 0it [00:04, ?it/s] 2024-06-26T07:43:48.8236722Z cuda train MegatronBertForCausalLM 2024-06-26T07:44:18.9010337Z W0626 07:44:18.900000 139975226712704 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:44:57.8658770Z pass 2024-06-26T07:44:58.2147924Z TIMING: entire_frame_compile:60.99474 backend_compile:36.55184 2024-06-26T07:44:58.2150193Z STATS: call_* op count: 1923 | FakeTensorMode.__torch_dispatch__:120693 | FakeTensor.__torch_dispatch__:9913 | ProxyTorchDispatchMode.__torch_dispatch__:42045 2024-06-26T07:44:58.2151351Z Dynamo produced 2 graphs covering 1923 ops with 5 graph breaks (4 unique) 2024-06-26T07:45:03.0269159Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:45:03.0270862Z warnings.warn( 2024-06-26T07:45:03.2706397Z 2024-06-26T07:45:07.4228173Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:45:07.4228654Z loading model: 0it [00:04, ?it/s] 2024-06-26T07:45:07.4229121Z cuda train MegatronBertForQuestionAnswering 2024-06-26T07:45:36.8465719Z W0626 07:45:36.845000 140144264536704 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:46:15.5255239Z pass 2024-06-26T07:46:15.8646210Z TIMING: entire_frame_compile:60.50282 backend_compile:36.15779 2024-06-26T07:46:15.8648368Z STATS: call_* op count: 1917 | FakeTensorMode.__torch_dispatch__:120076 | FakeTensor.__torch_dispatch__:9837 | ProxyTorchDispatchMode.__torch_dispatch__:41827 2024-06-26T07:46:15.8650626Z Dynamo produced 2 graphs covering 1917 ops with 5 graph breaks (4 unique) 2024-06-26T07:46:20.6697391Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:46:20.6699023Z warnings.warn( 2024-06-26T07:46:20.8617815Z 2024-06-26T07:46:22.2000616Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:46:22.2001116Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:46:22.2004335Z cuda train MobileBertForMaskedLM 2024-06-26T07:47:20.6925087Z pass 2024-06-26T07:47:20.7254161Z TIMING: entire_frame_compile:54.70939 backend_compile:30.48844 2024-06-26T07:47:20.7256269Z STATS: call_* op count: 1449 | FakeTensorMode.__torch_dispatch__:130119 | FakeTensor.__torch_dispatch__:4395 | ProxyTorchDispatchMode.__torch_dispatch__:55808 2024-06-26T07:47:20.7257432Z Dynamo produced 1 graphs covering 1449 ops with 3 graph breaks (3 unique) 2024-06-26T07:47:25.4435410Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:47:25.4437041Z warnings.warn( 2024-06-26T07:47:25.6506125Z 2024-06-26T07:47:26.8231495Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:47:26.8232410Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:47:26.8235557Z cuda train MobileBertForQuestionAnswering 2024-06-26T07:48:25.2464995Z pass 2024-06-26T07:48:25.2686336Z TIMING: entire_frame_compile:54.67982 backend_compile:30.43904 2024-06-26T07:48:25.2688879Z STATS: call_* op count: 1453 | FakeTensorMode.__torch_dispatch__:129979 | FakeTensor.__torch_dispatch__:4387 | ProxyTorchDispatchMode.__torch_dispatch__:55709 2024-06-26T07:48:25.2690035Z Dynamo produced 1 graphs covering 1453 ops with 3 graph breaks (3 unique) 2024-06-26T07:48:29.9147207Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:48:29.9148899Z warnings.warn( 2024-06-26T07:48:30.1218641Z 2024-06-26T07:48:32.4564227Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:48:32.4565130Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:48:32.4565595Z cuda train OPTForCausalLM 2024-06-26T07:48:32.4784738Z WARNING:common:fp64 golden ref were not generated for OPTForCausalLM. Setting accuracy check to cosine 2024-06-26T07:48:42.9618214Z pass 2024-06-26T07:48:43.0129949Z TIMING: entire_frame_compile:8.38455 backend_compile:6.55628 2024-06-26T07:48:43.0131935Z STATS: call_* op count: 533 | FakeTensorMode.__torch_dispatch__:29523 | FakeTensor.__torch_dispatch__:721 | ProxyTorchDispatchMode.__torch_dispatch__:13208 2024-06-26T07:48:43.0133055Z Dynamo produced 17 graphs covering 533 ops with 6 graph breaks (5 unique) 2024-06-26T07:48:46.5845424Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:48:46.5847916Z warnings.warn( 2024-06-26T07:48:46.7797505Z 2024-06-26T07:48:48.5182621Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:48:48.5183279Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:48:48.5194241Z cuda train PLBartForCausalLM 2024-06-26T07:48:48.5358716Z WARNING:common:fp64 golden ref were not generated for PLBartForCausalLM. Setting accuracy check to cosine 2024-06-26T07:48:53.9961720Z pass 2024-06-26T07:48:54.0401907Z TIMING: entire_frame_compile:4.68231 backend_compile:3.32745 2024-06-26T07:48:54.0403235Z STATS: call_* op count: 250 | FakeTensorMode.__torch_dispatch__:15285 | FakeTensor.__torch_dispatch__:421 | ProxyTorchDispatchMode.__torch_dispatch__:6609 2024-06-26T07:48:54.0404353Z Dynamo produced 11 graphs covering 250 ops with 6 graph breaks (5 unique) 2024-06-26T07:48:57.4958821Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:48:57.4960545Z warnings.warn( 2024-06-26T07:48:57.7037975Z 2024-06-26T07:49:00.8199533Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:49:00.8200021Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:49:00.8200506Z cuda train PLBartForConditionalGeneration 2024-06-26T07:49:00.8735182Z WARNING:common:fp64 golden ref were not generated for PLBartForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:49:13.5128860Z pass 2024-06-26T07:49:13.5830198Z TIMING: entire_frame_compile:11.01184 backend_compile:8.39855 2024-06-26T07:49:13.5831290Z STATS: call_* op count: 646 | FakeTensorMode.__torch_dispatch__:38771 | FakeTensor.__torch_dispatch__:1045 | ProxyTorchDispatchMode.__torch_dispatch__:17249 2024-06-26T07:49:13.5832395Z Dynamo produced 19 graphs covering 646 ops with 8 graph breaks (5 unique) 2024-06-26T07:49:17.2277114Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:49:17.2278747Z warnings.warn( 2024-06-26T07:49:17.4399031Z 2024-06-26T07:49:23.4896292Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:49:23.4896942Z loading model: 0it [00:06, ?it/s] 2024-06-26T07:49:23.4897597Z cuda train PegasusForCausalLM 2024-06-26T07:49:23.6133858Z WARNING:common:fp64 golden ref were not generated for PegasusForCausalLM. Setting accuracy check to cosine 2024-06-26T07:49:33.1630595Z pass 2024-06-26T07:49:33.2450177Z TIMING: entire_frame_compile:8.50196 backend_compile:6.49786 2024-06-26T07:49:33.2452308Z STATS: call_* op count: 476 | FakeTensorMode.__torch_dispatch__:28829 | FakeTensor.__torch_dispatch__:744 | ProxyTorchDispatchMode.__torch_dispatch__:12848 2024-06-26T07:49:33.2454264Z Dynamo produced 17 graphs covering 476 ops with 6 graph breaks (5 unique) 2024-06-26T07:49:36.8337072Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:49:36.8338794Z warnings.warn( 2024-06-26T07:49:37.0634009Z 2024-06-26T07:49:47.2953765Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:49:47.2954845Z loading model: 0it [00:10, ?it/s] 2024-06-26T07:49:47.2955917Z cuda train PegasusForConditionalGeneration 2024-06-26T07:49:47.3727508Z WARNING:common:fp64 golden ref were not generated for PegasusForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:50:09.6137958Z pass 2024-06-26T07:50:09.7243334Z TIMING: entire_frame_compile:19.98235 backend_compile:15.53139 2024-06-26T07:50:09.7247086Z STATS: call_* op count: 1234 | FakeTensorMode.__torch_dispatch__:72609 | FakeTensor.__torch_dispatch__:1833 | ProxyTorchDispatchMode.__torch_dispatch__:33098 2024-06-26T07:50:09.7250146Z Dynamo produced 30 graphs covering 1234 ops with 7 graph breaks (4 unique) 2024-06-26T07:50:13.5865008Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:50:13.5866894Z warnings.warn( 2024-06-26T07:50:13.7951139Z 2024-06-26T07:50:13.7987789Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2024-06-26T07:50:15.6992166Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-06-26T07:50:15.6994098Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-06-26T07:50:15.9261657Z 2024-06-26T07:50:15.9262667Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:50:15.9263835Z cuda train RobertaForCausalLM 2024-06-26T07:50:30.4206575Z W0626 07:50:30.419000 139796746715776 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:50:50.3863881Z pass 2024-06-26T07:50:50.4990221Z TIMING: entire_frame_compile:31.02069 backend_compile:18.68841 2024-06-26T07:50:50.4994003Z STATS: call_* op count: 1008 | FakeTensorMode.__torch_dispatch__:62215 | FakeTensor.__torch_dispatch__:5113 | ProxyTorchDispatchMode.__torch_dispatch__:21552 2024-06-26T07:50:50.4995155Z Dynamo produced 2 graphs covering 1008 ops with 5 graph breaks (4 unique) 2024-06-26T07:50:54.5678682Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:50:54.5681624Z warnings.warn( 2024-06-26T07:50:54.7668361Z 2024-06-26T07:50:56.2772860Z loading model: 0it [00:00, ?it/s]We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-06-26T07:50:56.2774919Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-06-26T07:50:56.5164493Z 2024-06-26T07:50:56.5165455Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:50:56.5166191Z cuda train RobertaForQuestionAnswering 2024-06-26T07:51:10.3579766Z W0626 07:51:10.357000 140472793535104 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:51:29.8718180Z pass 2024-06-26T07:51:29.9974093Z TIMING: entire_frame_compile:30.48706 backend_compile:18.34668 2024-06-26T07:51:29.9978839Z STATS: call_* op count: 1001 | FakeTensorMode.__torch_dispatch__:61594 | FakeTensor.__torch_dispatch__:5037 | ProxyTorchDispatchMode.__torch_dispatch__:21334 2024-06-26T07:51:29.9981133Z Dynamo produced 2 graphs covering 1001 ops with 5 graph breaks (4 unique) 2024-06-26T07:51:34.0591440Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:51:34.0594421Z warnings.warn( 2024-06-26T07:51:34.2823733Z 2024-06-26T07:51:34.7957502Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:51:34.7958078Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:51:34.7958542Z cuda train Speech2Text2ForCausalLM 2024-06-26T07:51:34.8122178Z WARNING:common:fp64 golden ref were not generated for Speech2Text2ForCausalLM. Setting accuracy check to cosine 2024-06-26T07:51:40.0972021Z pass 2024-06-26T07:51:40.1025267Z TIMING: entire_frame_compile:4.6691 backend_compile:3.48401 2024-06-26T07:51:40.1026517Z STATS: call_* op count: 258 | FakeTensorMode.__torch_dispatch__:15309 | FakeTensor.__torch_dispatch__:404 | ProxyTorchDispatchMode.__torch_dispatch__:6652 2024-06-26T07:51:40.1028245Z Dynamo produced 11 graphs covering 258 ops with 6 graph breaks (5 unique) 2024-06-26T07:51:43.5858413Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:51:43.5860046Z warnings.warn( 2024-06-26T07:51:43.7848912Z 2024-06-26T07:51:45.0482327Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:51:45.0482814Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:51:45.0483292Z cuda train T5ForConditionalGeneration 2024-06-26T07:52:03.3481629Z W0626 07:52:03.347000 139713621746304 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:52:17.6905725Z pass 2024-06-26T07:52:17.7543569Z TIMING: entire_frame_compile:28.5505 backend_compile:17.63706 2024-06-26T07:52:17.7546131Z STATS: call_* op count: 1224 | FakeTensorMode.__torch_dispatch__:63325 | FakeTensor.__torch_dispatch__:3167 | ProxyTorchDispatchMode.__torch_dispatch__:24715 2024-06-26T07:52:17.7547479Z Dynamo produced 2 graphs covering 1224 ops with 5 graph breaks (4 unique) 2024-06-26T07:52:21.7038234Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:52:21.7039919Z warnings.warn( 2024-06-26T07:52:21.9109108Z 2024-06-26T07:52:23.2074389Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:52:23.2074982Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:52:23.2075434Z cuda train T5Small 2024-06-26T07:52:41.3396180Z W0626 07:52:41.338000 139905428902528 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:52:55.4858789Z pass 2024-06-26T07:52:55.5448033Z TIMING: entire_frame_compile:28.21351 backend_compile:17.38444 2024-06-26T07:52:55.5450562Z STATS: call_* op count: 1224 | FakeTensorMode.__torch_dispatch__:63325 | FakeTensor.__torch_dispatch__:3167 | ProxyTorchDispatchMode.__torch_dispatch__:24715 2024-06-26T07:52:55.5451714Z Dynamo produced 2 graphs covering 1224 ops with 5 graph breaks (4 unique) 2024-06-26T07:52:59.5287711Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:52:59.5289466Z warnings.warn( 2024-06-26T07:52:59.9508026Z 2024-06-26T07:53:03.5399321Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:53:03.5399817Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:53:03.5400269Z cuda train TrOCRForCausalLM 2024-06-26T07:53:03.5727001Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine 2024-06-26T07:53:13.0511135Z pass 2024-06-26T07:53:13.1718953Z TIMING: entire_frame_compile:8.34387 backend_compile:6.20675 2024-06-26T07:53:13.1720081Z STATS: call_* op count: 477 | FakeTensorMode.__torch_dispatch__:28948 | FakeTensor.__torch_dispatch__:763 | ProxyTorchDispatchMode.__torch_dispatch__:12867 2024-06-26T07:53:13.1721218Z Dynamo produced 17 graphs covering 477 ops with 6 graph breaks (5 unique) 2024-06-26T07:53:16.7192033Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:53:16.7195212Z warnings.warn( 2024-06-26T07:53:16.9283099Z 2024-06-26T07:53:26.1232819Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:53:26.1233307Z loading model: 0it [00:09, ?it/s] 2024-06-26T07:53:26.1233760Z cuda train XGLMForCausalLM 2024-06-26T07:53:26.2704517Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine 2024-06-26T07:53:44.9835523Z pass 2024-06-26T07:53:45.1459571Z TIMING: entire_frame_compile:16.65849 backend_compile:13.20282 2024-06-26T07:53:45.1461761Z STATS: call_* op count: 987 | FakeTensorMode.__torch_dispatch__:58729 | FakeTensor.__torch_dispatch__:1433 | ProxyTorchDispatchMode.__torch_dispatch__:26536 2024-06-26T07:53:45.1464044Z Dynamo produced 29 graphs covering 987 ops with 6 graph breaks (5 unique) 2024-06-26T07:53:48.8843064Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:53:48.8844666Z warnings.warn( 2024-06-26T07:53:49.0811648Z 2024-06-26T07:53:53.2289782Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:53:53.2290351Z loading model: 0it [00:04, ?it/s] 2024-06-26T07:53:53.2290806Z cuda train XLNetLMHeadModel 2024-06-26T07:54:36.7215936Z W0626 07:54:36.720000 139626001625728 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:55:18.7601906Z pass 2024-06-26T07:55:19.1290484Z TIMING: entire_frame_compile:75.7097 backend_compile:47.7569 2024-06-26T07:55:19.1293040Z STATS: call_* op count: 1874 | FakeTensorMode.__torch_dispatch__:152227 | FakeTensor.__torch_dispatch__:8098 | ProxyTorchDispatchMode.__torch_dispatch__:55611 2024-06-26T07:55:19.1294376Z Dynamo produced 2 graphs covering 1874 ops with 5 graph breaks (4 unique) 2024-06-26T07:55:24.1798095Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:55:24.1799729Z warnings.warn( 2024-06-26T07:55:24.3790223Z 2024-06-26T07:55:26.4407210Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:55:26.4407880Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:55:26.4408551Z cuda train YituTechConvBert 2024-06-26T07:55:46.6172030Z W0626 07:55:46.616000 140431745688192 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:56:16.2582734Z pass 2024-06-26T07:56:16.3553055Z TIMING: entire_frame_compile:46.38361 backend_compile:27.77464 2024-06-26T07:56:16.3554179Z STATS: call_* op count: 1512 | FakeTensorMode.__torch_dispatch__:89993 | FakeTensor.__torch_dispatch__:7175 | ProxyTorchDispatchMode.__torch_dispatch__:31640 2024-06-26T07:56:16.3555303Z Dynamo produced 2 graphs covering 1512 ops with 5 graph breaks (4 unique) 2024-06-26T07:56:18.1573246Z accuracy pass_rate=93.48% 2024-06-26T07:56:18.1587504Z calls_captured gmean=0.00x mean=920.348x 2024-06-26T07:56:18.1592790Z unique_graphs gmean=0.00x mean=7.478x 2024-06-26T07:56:18.1597828Z graph_breaks gmean=0.00x mean=5.196x 2024-06-26T07:56:18.1602413Z unique_graph_breaks gmean=0.00x mean=4.043x 2024-06-26T07:56:18.1607281Z autograd_captures gmean=0.00x mean=0.000x 2024-06-26T07:56:18.1612059Z autograd_compiles gmean=0.00x mean=0.000x 2024-06-26T07:56:18.1617117Z cudagraph_skips gmean=0.00x mean=0.000x 2024-06-26T07:56:18.7403285Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamic_aot_eager_huggingface_training.csv 2024-06-26T07:56:18.9949656Z AlbertForMaskedLM PASS 2024-06-26T07:56:18.9954203Z AlbertForQuestionAnswering PASS 2024-06-26T07:56:18.9958771Z AllenaiLongformerBase PASS 2024-06-26T07:56:18.9963021Z BartForCausalLM PASS 2024-06-26T07:56:18.9967811Z BartForConditionalGeneration PASS 2024-06-26T07:56:18.9972344Z BertForMaskedLM PASS 2024-06-26T07:56:18.9976876Z BertForQuestionAnswering PASS 2024-06-26T07:56:18.9981529Z BlenderbotForCausalLM XFAIL 2024-06-26T07:56:18.9985963Z BlenderbotSmallForCausalLM PASS 2024-06-26T07:56:18.9990872Z BlenderbotSmallForConditionalGeneration PASS 2024-06-26T07:56:18.9994926Z CamemBert PASS 2024-06-26T07:56:18.9999274Z DebertaForMaskedLM PASS 2024-06-26T07:56:19.0003730Z DebertaForQuestionAnswering PASS 2024-06-26T07:56:19.0008724Z DebertaV2ForMaskedLM XFAIL 2024-06-26T07:56:19.0013060Z DebertaV2ForQuestionAnswering XFAIL 2024-06-26T07:56:19.0017357Z DistilBertForMaskedLM PASS 2024-06-26T07:56:19.0021937Z DistilBertForQuestionAnswering PASS 2024-06-26T07:56:19.0026303Z DistillGPT2 PASS 2024-06-26T07:56:19.0031202Z ElectraForCausalLM PASS 2024-06-26T07:56:19.0035815Z ElectraForQuestionAnswering PASS 2024-06-26T07:56:19.0040129Z GPT2ForSequenceClassification PASS 2024-06-26T07:56:19.0044376Z GoogleFnet PASS 2024-06-26T07:56:19.0049266Z LayoutLMForMaskedLM PASS 2024-06-26T07:56:19.0053614Z LayoutLMForSequenceClassification PASS 2024-06-26T07:56:19.0057999Z M2M100ForConditionalGeneration PASS 2024-06-26T07:56:19.0062522Z MBartForCausalLM PASS 2024-06-26T07:56:19.0066972Z MBartForConditionalGeneration PASS 2024-06-26T07:56:19.0071691Z MT5ForConditionalGeneration PASS 2024-06-26T07:56:19.0075908Z MegatronBertForCausalLM PASS 2024-06-26T07:56:19.0080488Z MegatronBertForQuestionAnswering PASS 2024-06-26T07:56:19.0085004Z MobileBertForMaskedLM PASS 2024-06-26T07:56:19.0089573Z MobileBertForQuestionAnswering PASS 2024-06-26T07:56:19.0093859Z OPTForCausalLM PASS 2024-06-26T07:56:19.0098203Z PLBartForCausalLM PASS 2024-06-26T07:56:19.0102848Z PLBartForConditionalGeneration PASS 2024-06-26T07:56:19.0107344Z PegasusForCausalLM PASS 2024-06-26T07:56:19.0111988Z PegasusForConditionalGeneration PASS 2024-06-26T07:56:19.0116331Z RobertaForCausalLM PASS 2024-06-26T07:56:19.0120884Z RobertaForQuestionAnswering PASS 2024-06-26T07:56:19.0125203Z Speech2Text2ForCausalLM PASS 2024-06-26T07:56:19.0130326Z T5ForConditionalGeneration PASS 2024-06-26T07:56:19.0134355Z T5Small PASS 2024-06-26T07:56:19.0138746Z TrOCRForCausalLM PASS 2024-06-26T07:56:19.0143302Z XGLMForCausalLM PASS 2024-06-26T07:56:19.0147853Z XLNetLMHeadModel PASS 2024-06-26T07:56:19.0152472Z YituTechConvBert PASS 2024-06-26T07:56:19.0529961Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamic_aot_eager_huggingface_training.csv 2024-06-26T07:56:19.3086789Z AlbertForMaskedLM PASS 2024-06-26T07:56:19.3090060Z AlbertForQuestionAnswering PASS 2024-06-26T07:56:19.3095364Z AllenaiLongformerBase PASS 2024-06-26T07:56:19.3100304Z BartForCausalLM PASS 2024-06-26T07:56:19.3104816Z BartForConditionalGeneration PASS 2024-06-26T07:56:19.3110085Z BertForMaskedLM PASS 2024-06-26T07:56:19.3114873Z BertForQuestionAnswering PASS 2024-06-26T07:56:19.3120251Z BlenderbotForCausalLM PASS 2024-06-26T07:56:19.3125077Z BlenderbotSmallForCausalLM PASS 2024-06-26T07:56:19.3130378Z BlenderbotSmallForConditionalGeneration PASS 2024-06-26T07:56:19.3134984Z CamemBert PASS 2024-06-26T07:56:19.3139807Z DebertaForMaskedLM PASS 2024-06-26T07:56:19.3144649Z DebertaForQuestionAnswering PASS 2024-06-26T07:56:19.3149742Z DebertaV2ForMaskedLM PASS 2024-06-26T07:56:19.3154615Z DebertaV2ForQuestionAnswering PASS 2024-06-26T07:56:19.3159476Z DistilBertForMaskedLM PASS 2024-06-26T07:56:19.3164354Z DistilBertForQuestionAnswering PASS 2024-06-26T07:56:19.3169570Z DistillGPT2 PASS 2024-06-26T07:56:19.3174159Z ElectraForCausalLM PASS 2024-06-26T07:56:19.3179158Z ElectraForQuestionAnswering PASS 2024-06-26T07:56:19.3184316Z GPT2ForSequenceClassification PASS 2024-06-26T07:56:19.3189500Z GoogleFnet PASS 2024-06-26T07:56:19.3194537Z LayoutLMForMaskedLM PASS 2024-06-26T07:56:19.3199470Z LayoutLMForSequenceClassification PASS 2024-06-26T07:56:19.3204465Z M2M100ForConditionalGeneration PASS 2024-06-26T07:56:19.3209910Z MBartForCausalLM PASS 2024-06-26T07:56:19.3214760Z MBartForConditionalGeneration PASS 2024-06-26T07:56:19.3219819Z MT5ForConditionalGeneration PASS 2024-06-26T07:56:19.3224840Z MegatronBertForCausalLM PASS 2024-06-26T07:56:19.3230021Z MegatronBertForQuestionAnswering PASS 2024-06-26T07:56:19.3234994Z MobileBertForMaskedLM PASS 2024-06-26T07:56:19.3239869Z MobileBertForQuestionAnswering PASS 2024-06-26T07:56:19.3244714Z OPTForCausalLM PASS 2024-06-26T07:56:19.3250668Z PLBartForCausalLM PASS 2024-06-26T07:56:19.3255960Z PLBartForConditionalGeneration PASS 2024-06-26T07:56:19.3261055Z PegasusForCausalLM PASS 2024-06-26T07:56:19.3266343Z PegasusForConditionalGeneration PASS 2024-06-26T07:56:19.3271738Z RobertaForCausalLM PASS 2024-06-26T07:56:19.3276993Z RobertaForQuestionAnswering PASS 2024-06-26T07:56:19.3282150Z Speech2Text2ForCausalLM PASS 2024-06-26T07:56:19.3287566Z T5ForConditionalGeneration PASS 2024-06-26T07:56:19.3292850Z T5Small PASS 2024-06-26T07:56:19.3298118Z TrOCRForCausalLM PASS 2024-06-26T07:56:19.3303240Z XGLMForCausalLM PASS 2024-06-26T07:56:19.3308536Z XLNetLMHeadModel PASS 2024-06-26T07:56:19.3313637Z YituTechConvBert PASS 2024-06-26T07:56:19.3699201Z + cleanup_workspace 2024-06-26T07:56:19.3700167Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-26T07:56:19.3703531Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-26T07:56:19.3704536Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-26T07:56:19.3705184Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T07:56:19.3705989Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-26T07:56:19.3707100Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-26T07:56:19.3707806Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-26T07:56:19.8574447Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T07:56:19.8575161Z cat test/**/*_toprint.log || true 2024-06-26T07:56:19.8585019Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:56:19.8585697Z env: 2024-06-26T07:56:19.8586072Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:19.8586682Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:56:19.8587904Z DOCKER_CONTAINER_ID: 240afe0ee980a530f2b3b6c166f41eee90a2a7b5c98112b59afbe2b10b264efa 2024-06-26T07:56:19.8588984Z ##[endgroup] 2024-06-26T07:56:19.8655426Z cat: test/**/*_toprint.log: No such file or directory 2024-06-26T07:56:19.8683997Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T07:56:19.8684473Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T07:56:19.8692223Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:56:19.8692726Z env: 2024-06-26T07:56:19.8693008Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:19.8693466Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:56:19.8694198Z DOCKER_CONTAINER_ID: 240afe0ee980a530f2b3b6c166f41eee90a2a7b5c98112b59afbe2b10b264efa 2024-06-26T07:56:19.8694864Z MONITOR_SCRIPT_PID: 2102 2024-06-26T07:56:19.8695217Z ##[endgroup] 2024-06-26T07:56:19.8880541Z Prepare all required actions 2024-06-26T07:56:19.8880999Z Getting action download info 2024-06-26T07:56:20.0385185Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T07:56:20.1936693Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T07:56:20.1937172Z with: 2024-06-26T07:56:20.1937761Z file-suffix: test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688917205 2024-06-26T07:56:20.1938479Z s3-bucket: gha-artifacts 2024-06-26T07:56:20.1938830Z env: 2024-06-26T07:56:20.1939112Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:20.1939573Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:56:20.1940328Z DOCKER_CONTAINER_ID: 240afe0ee980a530f2b3b6c166f41eee90a2a7b5c98112b59afbe2b10b264efa 2024-06-26T07:56:20.1940978Z ##[endgroup] 2024-06-26T07:56:20.1966232Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-26T07:56:20.1966876Z # Remove any previous test jsons if they exist 2024-06-26T07:56:20.1967389Z rm -f test-jsons-*.zip 2024-06-26T07:56:20.1967908Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-26T07:56:20.1975651Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:56:20.1976163Z env: 2024-06-26T07:56:20.1976454Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:20.1976906Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:56:20.1977647Z DOCKER_CONTAINER_ID: 240afe0ee980a530f2b3b6c166f41eee90a2a7b5c98112b59afbe2b10b264efa 2024-06-26T07:56:20.1978593Z FILE_SUFFIX: test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688917205 2024-06-26T07:56:20.1979288Z ##[endgroup] 2024-06-26T07:56:20.2158615Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-26T07:56:20.2187613Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-26T07:56:20.2188293Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-26T07:56:20.2194159Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-26T07:56:20.2219684Z ##[group]Run # Remove any previous test reports if they exist 2024-06-26T07:56:20.2220345Z # Remove any previous test reports if they exist 2024-06-26T07:56:20.2220899Z rm -f test-reports-*.zip 2024-06-26T07:56:20.2221498Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-26T07:56:20.2229208Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:56:20.2229705Z env: 2024-06-26T07:56:20.2229998Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:20.2230465Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:56:20.2231210Z DOCKER_CONTAINER_ID: 240afe0ee980a530f2b3b6c166f41eee90a2a7b5c98112b59afbe2b10b264efa 2024-06-26T07:56:20.2232152Z FILE_SUFFIX: test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688917205 2024-06-26T07:56:20.2232839Z ##[endgroup] 2024-06-26T07:56:20.2395595Z adding: test/test-reports/inference_huggingface.csv (deflated 76%) 2024-06-26T07:56:20.2396598Z adding: test/test-reports/inference_huggingface_graph_breaks.csv (deflated 89%) 2024-06-26T07:56:20.2397548Z adding: test/test-reports/inference_huggingface_graph_break_deduped.csv (deflated 78%) 2024-06-26T07:56:20.2398569Z adding: test/test-reports/training_huggingface.csv (deflated 73%) 2024-06-26T07:56:20.2404717Z adding: test/test-reports/training_huggingface_graph_breaks.csv (deflated 97%) 2024-06-26T07:56:20.2405721Z adding: test/test-reports/training_huggingface_graph_break_deduped.csv (deflated 77%) 2024-06-26T07:56:20.2431421Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-26T07:56:20.2432059Z # Remove any previous usage logs if they exist 2024-06-26T07:56:20.2432565Z rm -f logs-*.zip 2024-06-26T07:56:20.2433236Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-26T07:56:20.2434134Z # so check to see if the file exists first 2024-06-26T07:56:20.2434644Z if [ -f 'usage_log.txt' ]; then 2024-06-26T07:56:20.2435176Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-26T07:56:20.2435691Z fi 2024-06-26T07:56:20.2436091Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-26T07:56:20.2436660Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-26T07:56:20.2437151Z fi 2024-06-26T07:56:20.2444533Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:56:20.2445031Z env: 2024-06-26T07:56:20.2445303Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:20.2445826Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:56:20.2446569Z DOCKER_CONTAINER_ID: 240afe0ee980a530f2b3b6c166f41eee90a2a7b5c98112b59afbe2b10b264efa 2024-06-26T07:56:20.2447504Z FILE_SUFFIX: test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688917205 2024-06-26T07:56:20.2448178Z ##[endgroup] 2024-06-26T07:56:20.2500933Z adding: usage_log.txt (deflated 92%) 2024-06-26T07:56:20.2565744Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-26T07:56:20.2566491Z # Remove any previous debugging artifacts if they exist 2024-06-26T07:56:20.2567061Z rm -f debug-*.zip 2024-06-26T07:56:20.2567454Z if [ -d 'test/debug' ]; then 2024-06-26T07:56:20.2567962Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-26T07:56:20.2568449Z fi 2024-06-26T07:56:20.2575313Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:56:20.2575865Z env: 2024-06-26T07:56:20.2576143Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:20.2576603Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:56:20.2577350Z DOCKER_CONTAINER_ID: 240afe0ee980a530f2b3b6c166f41eee90a2a7b5c98112b59afbe2b10b264efa 2024-06-26T07:56:20.2578297Z FILE_SUFFIX: test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688917205 2024-06-26T07:56:20.2578987Z ##[endgroup] 2024-06-26T07:56:20.2674558Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T07:56:20.2674999Z with: 2024-06-26T07:56:20.2675280Z s3-bucket: gha-artifacts 2024-06-26T07:56:20.2675726Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T07:56:20.2676245Z retention-days: 14 2024-06-26T07:56:20.2676577Z if-no-files-found: warn 2024-06-26T07:56:20.2676933Z path: test-jsons-*.zip 2024-06-26T07:56:20.2677271Z name: artifact 2024-06-26T07:56:20.2677573Z region: us-east-1 2024-06-26T07:56:20.2677862Z env: 2024-06-26T07:56:20.2678136Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:20.2678586Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:56:20.2679319Z DOCKER_CONTAINER_ID: 240afe0ee980a530f2b3b6c166f41eee90a2a7b5c98112b59afbe2b10b264efa 2024-06-26T07:56:20.2679950Z ##[endgroup] 2024-06-26T07:56:20.5734260Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T07:56:20.5735169Z With the provided path, there will be 1 file uploaded 2024-06-26T07:56:20.5735982Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T07:56:20.5763312Z Starting upload of test-jsons-test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688917205.zip 2024-06-26T07:56:20.7002876Z Finished upload of test-jsons-test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688917205.zip 2024-06-26T07:56:20.7132202Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T07:56:20.7132639Z with: 2024-06-26T07:56:20.7132924Z s3-bucket: gha-artifacts 2024-06-26T07:56:20.7133362Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T07:56:20.7133833Z retention-days: 14 2024-06-26T07:56:20.7134168Z if-no-files-found: error 2024-06-26T07:56:20.7134535Z path: test-reports-*.zip 2024-06-26T07:56:20.7134886Z name: artifact 2024-06-26T07:56:20.7135183Z region: us-east-1 2024-06-26T07:56:20.7135482Z env: 2024-06-26T07:56:20.7135760Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:20.7136332Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:56:20.7137071Z DOCKER_CONTAINER_ID: 240afe0ee980a530f2b3b6c166f41eee90a2a7b5c98112b59afbe2b10b264efa 2024-06-26T07:56:20.7137716Z ##[endgroup] 2024-06-26T07:56:21.0274120Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T07:56:21.0274911Z With the provided path, there will be 1 file uploaded 2024-06-26T07:56:21.0275659Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T07:56:21.0301251Z Starting upload of test-reports-test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688917205.zip 2024-06-26T07:56:21.1216548Z Finished upload of test-reports-test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688917205.zip 2024-06-26T07:56:21.1351686Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T07:56:21.1352140Z with: 2024-06-26T07:56:21.1352442Z s3-bucket: gha-artifacts 2024-06-26T07:56:21.1352880Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T07:56:21.1353377Z retention-days: 14 2024-06-26T07:56:21.1353719Z if-no-files-found: ignore 2024-06-26T07:56:21.1354088Z path: logs-*.zip 2024-06-26T07:56:21.1354407Z name: artifact 2024-06-26T07:56:21.1354713Z region: us-east-1 2024-06-26T07:56:21.1355041Z env: 2024-06-26T07:56:21.1355329Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:21.1355799Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:56:21.1356540Z DOCKER_CONTAINER_ID: 240afe0ee980a530f2b3b6c166f41eee90a2a7b5c98112b59afbe2b10b264efa 2024-06-26T07:56:21.1357191Z ##[endgroup] 2024-06-26T07:56:21.4423210Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T07:56:21.4424043Z With the provided path, there will be 1 file uploaded 2024-06-26T07:56:21.4424802Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T07:56:21.4450939Z Starting upload of logs-test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688917205.zip 2024-06-26T07:56:21.5640813Z Finished upload of logs-test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688917205.zip 2024-06-26T07:56:21.5770748Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T07:56:21.5771185Z with: 2024-06-26T07:56:21.5771475Z s3-bucket: gha-artifacts 2024-06-26T07:56:21.5771922Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T07:56:21.5772390Z retention-days: 14 2024-06-26T07:56:21.5772725Z if-no-files-found: ignore 2024-06-26T07:56:21.5773082Z path: debug-*.zip 2024-06-26T07:56:21.5773393Z name: artifact 2024-06-26T07:56:21.5773687Z region: us-east-1 2024-06-26T07:56:21.5773985Z env: 2024-06-26T07:56:21.5774255Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:21.5774706Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:56:21.5775440Z DOCKER_CONTAINER_ID: 240afe0ee980a530f2b3b6c166f41eee90a2a7b5c98112b59afbe2b10b264efa 2024-06-26T07:56:21.5776085Z ##[endgroup] 2024-06-26T07:56:21.8764028Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-26T07:56:21.8895432Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T07:56:21.8895934Z # shellcheck disable=SC2156 2024-06-26T07:56:21.8896784Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T07:56:21.8904862Z shell: /usr/bin/bash -e {0} 2024-06-26T07:56:21.8905215Z env: 2024-06-26T07:56:21.8905498Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:21.8905953Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:56:21.8906685Z DOCKER_CONTAINER_ID: 240afe0ee980a530f2b3b6c166f41eee90a2a7b5c98112b59afbe2b10b264efa 2024-06-26T07:56:21.8907595Z ##[endgroup] 2024-06-26T07:56:22.0524766Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-26T07:56:22.0525334Z with: 2024-06-26T07:56:22.0525671Z env: 2024-06-26T07:56:22.0525960Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:22.0526421Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:56:22.0527177Z DOCKER_CONTAINER_ID: 240afe0ee980a530f2b3b6c166f41eee90a2a7b5c98112b59afbe2b10b264efa 2024-06-26T07:56:22.0527829Z ##[endgroup] 2024-06-26T07:56:22.0545621Z ##[group]Run set -eou pipefail 2024-06-26T07:56:22.0546030Z set -eou pipefail 2024-06-26T07:56:22.0546399Z  2024-06-26T07:56:22.0547034Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-26T07:56:22.0547660Z for _ in $(seq 1440); do 2024-06-26T07:56:22.0548135Z  # Break if no ssh session exists anymore 2024-06-26T07:56:22.0548630Z  if [ "$(who)" = "" ]; then 2024-06-26T07:56:22.0549038Z  break 2024-06-26T07:56:22.0549370Z  fi 2024-06-26T07:56:22.0549665Z  echo "." 2024-06-26T07:56:22.0549991Z  sleep 5 2024-06-26T07:56:22.0550304Z done 2024-06-26T07:56:22.0557722Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:56:22.0558213Z env: 2024-06-26T07:56:22.0558490Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:22.0558942Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:56:22.0559675Z DOCKER_CONTAINER_ID: 240afe0ee980a530f2b3b6c166f41eee90a2a7b5c98112b59afbe2b10b264efa 2024-06-26T07:56:22.0560301Z ##[endgroup] 2024-06-26T07:56:22.0579914Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-26T07:56:22.0616542Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:56:22.0617313Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:56:22.0617910Z # shellcheck disable=SC2046 2024-06-26T07:56:22.0618387Z docker stop $(docker ps -q) || true 2024-06-26T07:56:22.0618870Z # Prune all of the docker images 2024-06-26T07:56:22.0619333Z docker system prune -af 2024-06-26T07:56:22.0626130Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:56:22.0626908Z env: 2024-06-26T07:56:22.0627234Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:22.0627690Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:56:22.0628425Z DOCKER_CONTAINER_ID: 240afe0ee980a530f2b3b6c166f41eee90a2a7b5c98112b59afbe2b10b264efa 2024-06-26T07:56:22.0641315Z ##[endgroup] 2024-06-26T07:56:22.3906542Z 240afe0ee980 2024-06-26T07:56:22.7762128Z Deleted Containers: 2024-06-26T07:56:22.7762731Z 240afe0ee980a530f2b3b6c166f41eee90a2a7b5c98112b59afbe2b10b264efa 2024-06-26T07:56:22.7763174Z 2024-06-26T07:56:27.7663595Z Deleted Images: 2024-06-26T07:56:27.7665376Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:56:27.7668508Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:56:27.7670463Z deleted: sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd 2024-06-26T07:56:27.7671308Z deleted: sha256:754ead7012f0b7691885a9660f02e80be6cee8c5665486fb995c1fb06adb6e92 2024-06-26T07:56:27.7672146Z deleted: sha256:fbe4496d8830e11be86d8f222522e7956f32f85bfc06b05ea53961f34a58af74 2024-06-26T07:56:27.7673192Z deleted: sha256:1c83814aa1a1a7e0cc998e3479bdab8e8766b975560cbf00690acf39ed899a80 2024-06-26T07:56:27.7674052Z deleted: sha256:e9ea8e61942c7f8ae4aab97b8d102b01d6bb4b6cf2a605808cd34130a17d8a0f 2024-06-26T07:56:27.7674908Z deleted: sha256:acc5fc32588cd95e082a6ce35e48d0f303ae8f96e3ab65c4e375b85e1f3c4a92 2024-06-26T07:56:27.7675740Z deleted: sha256:a93e53400904d29e4185d3be942ea34f19572ea438962788ae4e9b3d9bf4c498 2024-06-26T07:56:27.7676672Z deleted: sha256:c97092b88b5e4c77c68bbc4b51afab64e38719af1ff58013ec49a696cdd40a6e 2024-06-26T07:56:27.7677509Z deleted: sha256:b132889c90d67ffa53655846603e3fcc2372c23f04dbdbc54690a39d304eb89b 2024-06-26T07:56:27.7678376Z deleted: sha256:915d54081ed607710624abe3f2490114c745858831f4fc13396bd3b4b9a121bc 2024-06-26T07:56:27.7679193Z deleted: sha256:bfa8d446c498c87f535aef102695564de1b22c755e6c63944ae2f5073a2b31ec 2024-06-26T07:56:27.7680011Z deleted: sha256:e71460fb363c29e70bbe357d680aaa7ae694f094ef2f2713609158ad90611744 2024-06-26T07:56:27.7680830Z deleted: sha256:84079bb1b4ee1d0119ae3d3e701de460a1324912297c675425c9a36d30fcdb0f 2024-06-26T07:56:27.7681682Z deleted: sha256:d86ac3ca148d4758cb03b493fdd49109d8f1a2c7c85c4b755493c3c4529553e5 2024-06-26T07:56:27.7682518Z deleted: sha256:532b3ab97c44a673a0d5f62ee46fa6a83248a84de8877754a612355be56ed0a3 2024-06-26T07:56:27.7683340Z deleted: sha256:617e2a35f25ae24500c88c0c9b8475f0687fbcbed3d200013b4c99c6d8941aff 2024-06-26T07:56:27.7684492Z deleted: sha256:b88120bcd5a26a660c09c5df13176e0ef098fb2c8cdb57d74ab0796a3d1194d6 2024-06-26T07:56:27.7685778Z deleted: sha256:c016b2e7764735243205fb76fa9a273bb3dd4bf05532288e3a613ce022131ac3 2024-06-26T07:56:27.7686711Z deleted: sha256:bcaca462c390deddd2995de2d245a7495e7d2e1e94f0edad30a573c83d0588fd 2024-06-26T07:56:27.7687546Z deleted: sha256:9e260eba003d5c25d7b410e5c4c3093845d9e46611896840901c689d7fbf8ee6 2024-06-26T07:56:27.7688355Z deleted: sha256:3e62b86f8b911a9b59c62d55f063e71683ae6a01c22418db5d27930712adaad6 2024-06-26T07:56:27.7689175Z deleted: sha256:905920ccb1d88449b8dde686c34a4b2a8b6f59e78ac44916b189ea3593b9fbbb 2024-06-26T07:56:27.7690008Z deleted: sha256:961201237e03acd5061406238dd7a3c1343ac888d7583212cd0ab5ae1b4f0540 2024-06-26T07:56:27.7690833Z deleted: sha256:70047a2aac62159fa28fb4dbd4567ec2ffbab68fe046967e09acb0eb3e5cb9cb 2024-06-26T07:56:27.7691666Z deleted: sha256:1ec680ad781436ae65fa987f5fa1ed032ecd5c82a6e35eea6707de501f62c0f4 2024-06-26T07:56:27.7692515Z deleted: sha256:ef859623a50e85abf6e6c1dbd6188aa765a66cc7b2bd618a1cbe4825b7b2bd91 2024-06-26T07:56:27.7693368Z deleted: sha256:404ade5cacff59773b93ab438bfa64cd3a171d07f6f168bdc4a5d4bfa0b43526 2024-06-26T07:56:27.7694210Z deleted: sha256:5b27de6ee3d9f2f18217bf18dec725f7b4954c0e48c85d35752634c22d62209b 2024-06-26T07:56:27.7695035Z deleted: sha256:498e39c575b3fa069acaac7e46b91b4e54ae3a05f2ada5c4347091f346f25d4c 2024-06-26T07:56:27.7695844Z deleted: sha256:1405919d2694d65669b1f4c842282021afe89208ce86570e96464903733da2d0 2024-06-26T07:56:27.7696647Z deleted: sha256:d5b41e5d70896c0472fc3e6da0516a17d89cd6635dc75010ecddc31ce9690fad 2024-06-26T07:56:27.7697473Z deleted: sha256:18e9379b62b9d0bb40293d28a652eafaee66a499f66f0776bf14e37a513101b7 2024-06-26T07:56:27.7698297Z deleted: sha256:a781f80ed00ff2926a699fd01a309bb0b4fb7d5655eb652c205bb9c7f811035f 2024-06-26T07:56:27.7699118Z deleted: sha256:c0430973b7cf759075330ee29c79bfd1312c91e6033d9dd883b4b1c29743a171 2024-06-26T07:56:27.7699952Z deleted: sha256:e733b3d66efb9342e5a283376c0e1ddffa47a42fecc26e61ecaf0a3472885d97 2024-06-26T07:56:27.7700792Z deleted: sha256:8929c0b2047f74b84c82c39eff9077a92ae1a828cde00c9ab78fb3e0ed5e0283 2024-06-26T07:56:27.7701641Z deleted: sha256:f5b56155fb58a0a60fe6391d3581cf91610ed69d4f0a321257c19944da296a52 2024-06-26T07:56:27.7702454Z deleted: sha256:b54d2cb355238d1aa91e58df93064447b80f53706d1ca62823acc834f5ac54fe 2024-06-26T07:56:27.7703276Z deleted: sha256:e7311285af9eeddb51aa5629aa98386e89f1456ff32843b6aee6443247edf66c 2024-06-26T07:56:27.7704101Z deleted: sha256:67657ceec347da8dc8085d5731633efeac115ba677fa4b7c5966c4260f7abab0 2024-06-26T07:56:27.7704935Z deleted: sha256:b496e50409dbf414930fc9b4437aa0aa4ce27bc7840e050ae459aaffcf9aca86 2024-06-26T07:56:27.7705857Z deleted: sha256:8f5276170cc4a60d36ad7b964a502306b29546254cedcbcb101ea9123497d2c0 2024-06-26T07:56:27.7706661Z deleted: sha256:53d524d1564745e96a7e515462171b73184d9f1ff5454b32fcfbbca39d361467 2024-06-26T07:56:27.7707740Z deleted: sha256:2acbadb00d69b6aca575f04baba4ea26ae2b5c2d85f09945004ad93a78a8eb75 2024-06-26T07:56:27.7708701Z deleted: sha256:02665efeff878aeb603c1edb2e21cb700c171c025d64f5aae1b2b20b12f3964a 2024-06-26T07:56:27.7709540Z deleted: sha256:19bd23e9a51dfe883a8b8d2ff6d66ff1b3476b8912856406bb79b81762aa6d4b 2024-06-26T07:56:27.7710361Z deleted: sha256:73aa8637bf9299d6298dabd858480c274a62117388f2ca1d5e9eb15c6d1e4155 2024-06-26T07:56:27.7711209Z deleted: sha256:5f8968406305f348b1caebc0e035187d6e9965ace7b9d5f2f21114d37c9fe2e1 2024-06-26T07:56:27.7712033Z deleted: sha256:d827104914db0c503cf5c81f13e2456bf4efce42d4cf1f8d5d70838207278bee 2024-06-26T07:56:27.7712850Z deleted: sha256:867b219724f2f1aaf56f42e3e9f03a5d64bdca8a6a72407efd10328aeea5d510 2024-06-26T07:56:27.7713683Z deleted: sha256:b9ff2a7cceb9237b26bcb17728edf8a558eec5a76f63626e84f80541dfc0f1d4 2024-06-26T07:56:27.7714514Z deleted: sha256:be73f509106819007ce2b0f5a1301ac56ae583cb6c1fb69bee93efa0840368f1 2024-06-26T07:56:27.7715335Z deleted: sha256:2a11c07c915c3fdbb4e73096dea73ea950c970d50fe1a5764060bd62b7b7cf2c 2024-06-26T07:56:27.7716160Z deleted: sha256:5e9989622e2abc1c72f8184343507b45d75e4ff971edb68b96cbcd75ae484ff5 2024-06-26T07:56:27.7716982Z deleted: sha256:05ce642bd9cf5a9205bff107df28b1d6cef7d5f6ab3ea944d28c032ec61270df 2024-06-26T07:56:27.7717799Z deleted: sha256:017702617f6c07b73f2088678f4710b794cfe897a213a5cf122a0e2519eae0ac 2024-06-26T07:56:27.7718605Z deleted: sha256:44b3b37bb1ca39dc5cf356de41a23a5c12f605a1992813e408087fb2b0ad6d1c 2024-06-26T07:56:27.7719420Z deleted: sha256:8c387982e9f27db357a1d149dc149cabc0bd04668ad6956387a5072682dada67 2024-06-26T07:56:27.7720230Z deleted: sha256:10d415dca4e3fbb88a747b40317f3c6333c041ba8f73855f67fb9fe981fae5ee 2024-06-26T07:56:27.7721056Z deleted: sha256:6c65469bf0f53dcd625faefe363b8f3136c08388a6cf64d9158e8576e4ee89b1 2024-06-26T07:56:27.7721888Z deleted: sha256:027db5d8b76deeefbd8c34b485ab7ff14ed2a3e6800425c1ce0d77c2e9fdd4ab 2024-06-26T07:56:27.7722708Z deleted: sha256:24941234ad28cf5c8174d7cf18326c10b4e17f2c2137672ea0322df461597dd7 2024-06-26T07:56:27.7723523Z deleted: sha256:240ea2eebfa8519d6c2d88f4c99e66ce4e038d000f93ce8d03726c01d81180ef 2024-06-26T07:56:27.7724333Z deleted: sha256:09e823cb391a69f6a195e3ca8a41624b5f8b34eee14c6eb36277de814a91966e 2024-06-26T07:56:27.7725133Z deleted: sha256:730479d695169a6593bf0ac16b377491dce65175a6d158e012422367d04577fd 2024-06-26T07:56:27.7726026Z deleted: sha256:b0a30f9f7bd5459ff662453b9346a7ba8ea8db1191528345c1953eab4aa2f54e 2024-06-26T07:56:27.7726846Z deleted: sha256:75cefa87effaa3ad361c74642839923101f0f9982f814f3c9e2ed4ad52f2a008 2024-06-26T07:56:27.7727646Z deleted: sha256:6f97db87ba9512e615e871c6982000602982e1e5a5be334a56e7292251e84e00 2024-06-26T07:56:27.7728454Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-06-26T07:56:27.7729276Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-06-26T07:56:27.7730071Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-06-26T07:56:27.7730872Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-06-26T07:56:27.7731695Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-06-26T07:56:27.7732509Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-06-26T07:56:27.7733338Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-06-26T07:56:27.7734162Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-06-26T07:56:27.7734994Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-06-26T07:56:27.7735825Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-06-26T07:56:27.7736735Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-06-26T07:56:27.7737221Z 2024-06-26T07:56:27.7737374Z Total reclaimed space: 31.35GB 2024-06-26T07:56:27.7800376Z Post job cleanup. 2024-06-26T07:56:27.7848340Z Post job cleanup. 2024-06-26T07:56:27.8673527Z [command]/usr/bin/git version 2024-06-26T07:56:27.8715612Z git version 2.40.1 2024-06-26T07:56:27.8748280Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/8c2873c2-9cc9-49a7-a06f-d7a52d2e2471/.gitconfig' 2024-06-26T07:56:27.8759097Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/8c2873c2-9cc9-49a7-a06f-d7a52d2e2471' before making global git config changes 2024-06-26T07:56:27.8760329Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T07:56:27.8764119Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:56:27.8793275Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T07:56:27.8815517Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T07:56:27.9035853Z Entering 'android/libs/fbjni' 2024-06-26T07:56:27.9073725Z Entering 'third_party/FP16' 2024-06-26T07:56:27.9110207Z Entering 'third_party/FXdiv' 2024-06-26T07:56:27.9147586Z Entering 'third_party/NNPACK' 2024-06-26T07:56:27.9185232Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:56:27.9221510Z Entering 'third_party/XNNPACK' 2024-06-26T07:56:27.9269945Z Entering 'third_party/benchmark' 2024-06-26T07:56:27.9305710Z Entering 'third_party/cpp-httplib' 2024-06-26T07:56:27.9343434Z Entering 'third_party/cpuinfo' 2024-06-26T07:56:27.9380061Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:56:27.9416247Z Entering 'third_party/cutlass' 2024-06-26T07:56:27.9459789Z Entering 'third_party/eigen' 2024-06-26T07:56:27.9498622Z Entering 'third_party/fbgemm' 2024-06-26T07:56:27.9535360Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:56:27.9571524Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:56:27.9607310Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:56:27.9647396Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:56:27.9683737Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:56:27.9721554Z Entering 'third_party/flatbuffers' 2024-06-26T07:56:27.9760389Z Entering 'third_party/fmt' 2024-06-26T07:56:27.9797507Z Entering 'third_party/foxi' 2024-06-26T07:56:27.9833672Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:56:27.9870460Z Entering 'third_party/gloo' 2024-06-26T07:56:27.9906119Z Entering 'third_party/googletest' 2024-06-26T07:56:27.9942879Z Entering 'third_party/ideep' 2024-06-26T07:56:27.9979217Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:56:28.0019664Z Entering 'third_party/ittapi' 2024-06-26T07:56:28.0055126Z Entering 'third_party/kineto' 2024-06-26T07:56:28.0090606Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:56:28.0124957Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:56:28.0161352Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:56:28.0196505Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:56:28.0232915Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:56:28.0267077Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:56:28.0303350Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:56:28.0338436Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:56:28.0373078Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:56:28.0409355Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:56:28.0445116Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:56:28.0478435Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:56:28.0514147Z Entering 'third_party/mimalloc' 2024-06-26T07:56:28.0548364Z Entering 'third_party/nccl/nccl' 2024-06-26T07:56:28.0582631Z Entering 'third_party/nlohmann' 2024-06-26T07:56:28.0620658Z Entering 'third_party/onnx' 2024-06-26T07:56:28.0669927Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:56:28.0706164Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:56:28.0744549Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:56:28.0778987Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:56:28.0815485Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:56:28.0849428Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:56:28.0884848Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:56:28.0919567Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:56:28.0953350Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:56:28.0986443Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:56:28.1020499Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:56:28.1056563Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:56:28.1092482Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:56:28.1142067Z Entering 'third_party/pocketfft' 2024-06-26T07:56:28.1178678Z Entering 'third_party/protobuf' 2024-06-26T07:56:28.1216981Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:56:28.1250314Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:56:28.1286609Z Entering 'third_party/psimd' 2024-06-26T07:56:28.1321973Z Entering 'third_party/pthreadpool' 2024-06-26T07:56:28.1357570Z Entering 'third_party/pybind11' 2024-06-26T07:56:28.1393010Z Entering 'third_party/python-peachpy' 2024-06-26T07:56:28.1427401Z Entering 'third_party/sleef' 2024-06-26T07:56:28.1462103Z Entering 'third_party/tensorpipe' 2024-06-26T07:56:28.1495748Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:56:28.1530319Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:56:28.1564550Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:56:28.1598155Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:56:28.1631741Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:56:28.1681340Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T07:56:28.1702815Z http.https://github.com/.extraheader 2024-06-26T07:56:28.1712281Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T07:56:28.1735616Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T07:56:28.1945011Z Entering 'android/libs/fbjni' 2024-06-26T07:56:28.1966011Z http.https://github.com/.extraheader 2024-06-26T07:56:28.1989651Z Entering 'third_party/FP16' 2024-06-26T07:56:28.2010013Z http.https://github.com/.extraheader 2024-06-26T07:56:28.2032589Z Entering 'third_party/FXdiv' 2024-06-26T07:56:28.2052568Z http.https://github.com/.extraheader 2024-06-26T07:56:28.2074960Z Entering 'third_party/NNPACK' 2024-06-26T07:56:28.2097120Z http.https://github.com/.extraheader 2024-06-26T07:56:28.2121314Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:56:28.2143148Z http.https://github.com/.extraheader 2024-06-26T07:56:28.2166346Z Entering 'third_party/XNNPACK' 2024-06-26T07:56:28.2189007Z http.https://github.com/.extraheader 2024-06-26T07:56:28.2223423Z Entering 'third_party/benchmark' 2024-06-26T07:56:28.2244680Z http.https://github.com/.extraheader 2024-06-26T07:56:28.2268213Z Entering 'third_party/cpp-httplib' 2024-06-26T07:56:28.2289387Z http.https://github.com/.extraheader 2024-06-26T07:56:28.2314272Z Entering 'third_party/cpuinfo' 2024-06-26T07:56:28.2336048Z http.https://github.com/.extraheader 2024-06-26T07:56:28.2359839Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:56:28.2381881Z http.https://github.com/.extraheader 2024-06-26T07:56:28.2404623Z Entering 'third_party/cutlass' 2024-06-26T07:56:28.2424918Z http.https://github.com/.extraheader 2024-06-26T07:56:28.2455406Z Entering 'third_party/eigen' 2024-06-26T07:56:28.2477769Z http.https://github.com/.extraheader 2024-06-26T07:56:28.2501972Z Entering 'third_party/fbgemm' 2024-06-26T07:56:28.2524655Z http.https://github.com/.extraheader 2024-06-26T07:56:28.2549300Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:56:28.2570622Z http.https://github.com/.extraheader 2024-06-26T07:56:28.2593820Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:56:28.2615243Z http.https://github.com/.extraheader 2024-06-26T07:56:28.2640062Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:56:28.2661090Z http.https://github.com/.extraheader 2024-06-26T07:56:28.2688664Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:56:28.2711169Z http.https://github.com/.extraheader 2024-06-26T07:56:28.2735198Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:56:28.2757184Z http.https://github.com/.extraheader 2024-06-26T07:56:28.2782949Z Entering 'third_party/flatbuffers' 2024-06-26T07:56:28.2804868Z http.https://github.com/.extraheader 2024-06-26T07:56:28.2830590Z Entering 'third_party/fmt' 2024-06-26T07:56:28.2852699Z http.https://github.com/.extraheader 2024-06-26T07:56:28.2875363Z Entering 'third_party/foxi' 2024-06-26T07:56:28.2896625Z http.https://github.com/.extraheader 2024-06-26T07:56:28.2919197Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:56:28.2940764Z http.https://github.com/.extraheader 2024-06-26T07:56:28.2966037Z Entering 'third_party/gloo' 2024-06-26T07:56:28.2988425Z http.https://github.com/.extraheader 2024-06-26T07:56:28.3011798Z Entering 'third_party/googletest' 2024-06-26T07:56:28.3034419Z http.https://github.com/.extraheader 2024-06-26T07:56:28.3058674Z Entering 'third_party/ideep' 2024-06-26T07:56:28.3080848Z http.https://github.com/.extraheader 2024-06-26T07:56:28.3104241Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:56:28.3126077Z http.https://github.com/.extraheader 2024-06-26T07:56:28.3156102Z Entering 'third_party/ittapi' 2024-06-26T07:56:28.3178183Z http.https://github.com/.extraheader 2024-06-26T07:56:28.3201091Z Entering 'third_party/kineto' 2024-06-26T07:56:28.3222978Z http.https://github.com/.extraheader 2024-06-26T07:56:28.3246715Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:56:28.3268951Z http.https://github.com/.extraheader 2024-06-26T07:56:28.3294108Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:56:28.3315526Z http.https://github.com/.extraheader 2024-06-26T07:56:28.3341695Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:56:28.3364325Z http.https://github.com/.extraheader 2024-06-26T07:56:28.3388864Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:56:28.3409711Z http.https://github.com/.extraheader 2024-06-26T07:56:28.3433359Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:56:28.3455699Z http.https://github.com/.extraheader 2024-06-26T07:56:28.3479552Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:56:28.3501481Z http.https://github.com/.extraheader 2024-06-26T07:56:28.3526718Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:56:28.3548370Z http.https://github.com/.extraheader 2024-06-26T07:56:28.3572045Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:56:28.3593981Z http.https://github.com/.extraheader 2024-06-26T07:56:28.3619060Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:56:28.3641332Z http.https://github.com/.extraheader 2024-06-26T07:56:28.3665833Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:56:28.3688562Z http.https://github.com/.extraheader 2024-06-26T07:56:28.3713748Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:56:28.3735948Z http.https://github.com/.extraheader 2024-06-26T07:56:28.3758025Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:56:28.3779428Z http.https://github.com/.extraheader 2024-06-26T07:56:28.3804982Z Entering 'third_party/mimalloc' 2024-06-26T07:56:28.3827499Z http.https://github.com/.extraheader 2024-06-26T07:56:28.3850270Z Entering 'third_party/nccl/nccl' 2024-06-26T07:56:28.3872044Z http.https://github.com/.extraheader 2024-06-26T07:56:28.3895699Z Entering 'third_party/nlohmann' 2024-06-26T07:56:28.3918076Z http.https://github.com/.extraheader 2024-06-26T07:56:28.3943428Z Entering 'third_party/onnx' 2024-06-26T07:56:28.3965675Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4001116Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:56:28.4022865Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4047851Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:56:28.4069972Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4096664Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:56:28.4118145Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4144552Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:56:28.4166843Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4191725Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:56:28.4213522Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4237952Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:56:28.4258664Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4282090Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:56:28.4303712Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4328003Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:56:28.4349609Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4372813Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:56:28.4394064Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4417860Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:56:28.4439572Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4463659Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:56:28.4485803Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4512609Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:56:28.4534084Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4558703Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:56:28.4579634Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4618346Z Entering 'third_party/pocketfft' 2024-06-26T07:56:28.4640689Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4664270Z Entering 'third_party/protobuf' 2024-06-26T07:56:28.4686834Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4713564Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:56:28.4734706Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4758890Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:56:28.4780624Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4806196Z Entering 'third_party/psimd' 2024-06-26T07:56:28.4826482Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4851317Z Entering 'third_party/pthreadpool' 2024-06-26T07:56:28.4872856Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4896621Z Entering 'third_party/pybind11' 2024-06-26T07:56:28.4918052Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4941376Z Entering 'third_party/python-peachpy' 2024-06-26T07:56:28.4963989Z http.https://github.com/.extraheader 2024-06-26T07:56:28.4988106Z Entering 'third_party/sleef' 2024-06-26T07:56:28.5010642Z http.https://github.com/.extraheader 2024-06-26T07:56:28.5035095Z Entering 'third_party/tensorpipe' 2024-06-26T07:56:28.5055206Z http.https://github.com/.extraheader 2024-06-26T07:56:28.5076227Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:56:28.5098016Z http.https://github.com/.extraheader 2024-06-26T07:56:28.5121466Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:56:28.5141397Z http.https://github.com/.extraheader 2024-06-26T07:56:28.5165650Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:56:28.5188343Z http.https://github.com/.extraheader 2024-06-26T07:56:28.5212442Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:56:28.5234061Z http.https://github.com/.extraheader 2024-06-26T07:56:28.5257407Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:56:28.5278819Z http.https://github.com/.extraheader 2024-06-26T07:56:28.5381308Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-26T07:56:28.5398855Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T07:56:28.5405750Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:56:28.5406280Z ##[endgroup] 2024-06-26T07:56:29.3791923Z Cleaning up orphan processes