2024-06-26T07:03:25.3361557Z Current runner version: '2.317.0' 2024-06-26T07:03:25.3368347Z Runner name: 'i-0262ed88bcec9993d' 2024-06-26T07:03:25.3369243Z Runner group name: 'Default' 2024-06-26T07:03:25.3370199Z Machine name: 'ip-10-0-42-220' 2024-06-26T07:03:25.3375370Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T07:03:25.3377463Z Actions: read 2024-06-26T07:03:25.3378082Z Attestations: read 2024-06-26T07:03:25.3378646Z Checks: read 2024-06-26T07:03:25.3379179Z Contents: read 2024-06-26T07:03:25.3379706Z Deployments: read 2024-06-26T07:03:25.3380302Z Discussions: read 2024-06-26T07:03:25.3380867Z Issues: read 2024-06-26T07:03:25.3381390Z Metadata: read 2024-06-26T07:03:25.3381899Z Packages: read 2024-06-26T07:03:25.3382451Z Pages: read 2024-06-26T07:03:25.3383021Z PullRequests: read 2024-06-26T07:03:25.3383586Z RepositoryProjects: read 2024-06-26T07:03:25.3384245Z SecurityEvents: read 2024-06-26T07:03:25.3384819Z Statuses: read 2024-06-26T07:03:25.3385353Z ##[endgroup] 2024-06-26T07:03:25.3388746Z Secret source: Actions 2024-06-26T07:03:25.3389539Z Prepare workflow directory 2024-06-26T07:03:25.7261278Z Prepare all required actions 2024-06-26T07:03:25.7461047Z Getting action download info 2024-06-26T07:03:25.9291380Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T07:03:26.2651856Z Download action repository 'pytorch/pytorch@main' (SHA:c422a9549d2b320b47a9391f6cbcf592b2876c88) 2024-06-26T07:03:28.9383091Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-26T07:03:29.0683480Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T07:03:29.3204726Z Getting action download info 2024-06-26T07:03:29.4218924Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T07:03:29.5891687Z Getting action download info 2024-06-26T07:03:29.7255079Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T07:03:29.8743469Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/129470 (b8c4c54d347aa776934c60784e35936878ef18dc) 2024-06-26T07:03:29.8745729Z ##[group] Inputs 2024-06-26T07:03:29.8746277Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:03:29.8753407Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:03:29.8761229Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:29.8762491Z sync-tag: 2024-06-26T07:03:29.8763369Z timeout-minutes: 240 2024-06-26T07:03:29.8763915Z use-gha: 2024-06-26T07:03:29.8764274Z dashboard-tag: 2024-06-26T07:03:29.8764669Z s3-bucket: gha-artifacts 2024-06-26T07:03:29.8765091Z aws-role-to-assume: 2024-06-26T07:03:29.8765502Z ##[endgroup] 2024-06-26T07:03:29.8766693Z Complete job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:03:29.9348418Z A job started hook has been configured by the self-hosted runner administrator 2024-06-26T07:03:29.9490604Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T07:03:29.9501441Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:29.9502048Z ##[endgroup] 2024-06-26T07:03:31.0731307Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-26T07:03:31.0731862Z with: 2024-06-26T07:03:31.0732574Z github-secret: *** 2024-06-26T07:03:31.0733723Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-26T07:03:31.0734711Z activate-with-label: false 2024-06-26T07:03:31.0735072Z label: with-ssh 2024-06-26T07:03:31.0735393Z remove-existing-keys: true 2024-06-26T07:03:31.0735753Z fail-silently: true 2024-06-26T07:03:31.0736066Z env: 2024-06-26T07:03:31.0736343Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:31.0736822Z ##[endgroup] 2024-06-26T07:03:31.1614549Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-26T07:03:31.1615584Z ciflow reference detected, attempting to extract PR number 2024-06-26T07:03:31.4530527Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-06-26T07:03:31.5148686Z No SSH keys found for user pytorch-bot[bot] 2024-06-26T07:03:31.5149417Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T07:03:31.5723192Z No SSH keys found for user leslie-fang-intel 2024-06-26T07:03:31.5826137Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T07:03:31.5826738Z with: 2024-06-26T07:03:31.5827104Z submodules: recursive 2024-06-26T07:03:31.5827483Z fetch-depth: 0 2024-06-26T07:03:31.5827822Z env: 2024-06-26T07:03:31.5828134Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:31.5828532Z ##[endgroup] 2024-06-26T07:03:31.6025198Z ##[group]Run retry () { 2024-06-26T07:03:31.6025603Z retry () { 2024-06-26T07:03:31.6026168Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T07:03:31.6026800Z } 2024-06-26T07:03:31.6027190Z echo "${GITHUB_WORKSPACE}" 2024-06-26T07:03:31.6027641Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T07:03:31.6028209Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:03:31.6028722Z else 2024-06-26T07:03:31.6029130Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:03:31.6058568Z fi 2024-06-26T07:03:31.6058938Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T07:03:31.6067192Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:31.6067714Z env: 2024-06-26T07:03:31.6067994Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:31.6068334Z NO_SUDO: 2024-06-26T07:03:31.6068609Z ##[endgroup] 2024-06-26T07:03:31.6090288Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:03:34.5797675Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T07:03:34.5798121Z with: 2024-06-26T07:03:34.5798441Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:03:34.5798888Z fetch-depth: 0 2024-06-26T07:03:34.5799209Z submodules: recursive 2024-06-26T07:03:34.5799555Z quiet-checkout: true 2024-06-26T07:03:34.5800027Z repository: pytorch/pytorch 2024-06-26T07:03:34.5800516Z token: *** 2024-06-26T07:03:34.5800813Z ssh-strict: true 2024-06-26T07:03:34.5801141Z persist-credentials: true 2024-06-26T07:03:34.5801495Z clean: true 2024-06-26T07:03:34.5801832Z sparse-checkout-cone-mode: true 2024-06-26T07:03:34.5802224Z lfs: false 2024-06-26T07:03:34.5802525Z set-safe-directory: true 2024-06-26T07:03:34.5802866Z env: 2024-06-26T07:03:34.5803145Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:34.5803494Z ##[endgroup] 2024-06-26T07:03:34.6728783Z Syncing repository: pytorch/pytorch 2024-06-26T07:03:34.6730332Z ##[group]Getting Git version info 2024-06-26T07:03:34.6731078Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:03:34.6731941Z [command]/usr/bin/git version 2024-06-26T07:03:34.6732316Z git version 2.40.1 2024-06-26T07:03:34.6733650Z ##[endgroup] 2024-06-26T07:03:34.6747707Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/5ca3ae28-a735-4945-958d-e3fe6e43a6c8' before making global git config changes 2024-06-26T07:03:34.6748987Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T07:03:34.6753094Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:03:34.6788691Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:03:34.6793216Z ##[group]Initializing the repository 2024-06-26T07:03:34.6796838Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:03:34.6817768Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T07:03:34.6818889Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T07:03:34.6819705Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T07:03:34.6820280Z hint: 2024-06-26T07:03:34.6820923Z hint: git config --global init.defaultBranch 2024-06-26T07:03:34.6821427Z hint: 2024-06-26T07:03:34.6821949Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T07:03:34.6822795Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T07:03:34.6823611Z hint: 2024-06-26T07:03:34.6823947Z hint: git branch -m 2024-06-26T07:03:34.6824694Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T07:03:34.6826584Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T07:03:34.6846790Z ##[endgroup] 2024-06-26T07:03:34.6847367Z ##[group]Disabling automatic garbage collection 2024-06-26T07:03:34.6849523Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T07:03:34.6873837Z ##[endgroup] 2024-06-26T07:03:34.6874382Z ##[group]Setting up auth 2024-06-26T07:03:34.6879466Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T07:03:34.6905779Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T07:03:34.7093883Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T07:03:34.7114922Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T07:03:34.7311468Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:03:34.7348075Z ##[endgroup] 2024-06-26T07:03:34.7348777Z ##[group]Fetching the repository 2024-06-26T07:03:34.7354015Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T07:03:36.8901969Z remote: Enumerating objects: 987612 2024-06-26T07:03:36.8902710Z remote: Enumerating objects: 990209, done. 2024-06-26T07:03:36.8904753Z remote: Counting objects: 0% (1/2597) 2024-06-26T07:03:36.8907845Z remote: Counting objects: 1% (26/2597) 2024-06-26T07:03:36.8909064Z remote: Counting objects: 2% (52/2597) 2024-06-26T07:03:36.8909626Z remote: Counting objects: 3% (78/2597) 2024-06-26T07:03:36.8910126Z remote: Counting objects: 4% (104/2597) 2024-06-26T07:03:36.8910759Z remote: Counting objects: 5% (130/2597) 2024-06-26T07:03:36.8911547Z remote: Counting objects: 6% (156/2597) 2024-06-26T07:03:36.8912066Z remote: Counting objects: 7% (182/2597) 2024-06-26T07:03:36.8912560Z remote: Counting objects: 8% (208/2597) 2024-06-26T07:03:36.8913186Z remote: Counting objects: 9% (234/2597) 2024-06-26T07:03:36.8913908Z remote: Counting objects: 10% (260/2597) 2024-06-26T07:03:36.8914567Z remote: Counting objects: 11% (286/2597) 2024-06-26T07:03:36.8915077Z remote: Counting objects: 12% (312/2597) 2024-06-26T07:03:36.8915708Z remote: Counting objects: 13% (338/2597) 2024-06-26T07:03:36.8916422Z remote: Counting objects: 14% (364/2597) 2024-06-26T07:03:36.8917025Z remote: Counting objects: 15% (390/2597) 2024-06-26T07:03:36.8917539Z remote: Counting objects: 16% (416/2597) 2024-06-26T07:03:36.8918160Z remote: Counting objects: 17% (442/2597) 2024-06-26T07:03:36.8918668Z remote: Counting objects: 18% (468/2597) 2024-06-26T07:03:36.8919247Z remote: Counting objects: 19% (494/2597) 2024-06-26T07:03:36.8919926Z remote: Counting objects: 20% (520/2597) 2024-06-26T07:03:36.8920495Z remote: Counting objects: 21% (546/2597) 2024-06-26T07:03:36.8921004Z remote: Counting objects: 22% (572/2597) 2024-06-26T07:03:36.8921509Z remote: Counting objects: 23% (598/2597) 2024-06-26T07:03:36.8922001Z remote: Counting objects: 24% (624/2597) 2024-06-26T07:03:36.8922497Z remote: Counting objects: 25% (650/2597) 2024-06-26T07:03:36.8923079Z remote: Counting objects: 26% (676/2597) 2024-06-26T07:03:36.8924047Z remote: Counting objects: 27% (702/2597) 2024-06-26T07:03:36.8924741Z remote: Counting objects: 28% (728/2597) 2024-06-26T07:03:36.8925289Z remote: Counting objects: 29% (754/2597) 2024-06-26T07:03:36.8925885Z remote: Counting objects: 30% (780/2597) 2024-06-26T07:03:36.8926389Z remote: Counting objects: 31% (806/2597) 2024-06-26T07:03:36.8926899Z remote: Counting objects: 32% (832/2597) 2024-06-26T07:03:36.8927395Z remote: Counting objects: 33% (858/2597) 2024-06-26T07:03:36.8927975Z remote: Counting objects: 34% (883/2597) 2024-06-26T07:03:36.8928694Z remote: Counting objects: 35% (909/2597) 2024-06-26T07:03:36.8929282Z remote: Counting objects: 36% (935/2597) 2024-06-26T07:03:36.8929774Z remote: Counting objects: 37% (961/2597) 2024-06-26T07:03:36.8931557Z remote: Counting objects: 38% (987/2597) 2024-06-26T07:03:36.8932241Z remote: Counting objects: 39% (1013/2597) 2024-06-26T07:03:36.8932820Z remote: Counting objects: 40% (1039/2597) 2024-06-26T07:03:36.8933480Z remote: Counting objects: 41% (1065/2597) 2024-06-26T07:03:36.8934103Z remote: Counting objects: 42% (1091/2597) 2024-06-26T07:03:36.8934821Z remote: Counting objects: 43% (1117/2597) 2024-06-26T07:03:36.8935369Z remote: Counting objects: 44% (1143/2597) 2024-06-26T07:03:36.8936000Z remote: Counting objects: 45% (1169/2597) 2024-06-26T07:03:36.8936637Z remote: Counting objects: 46% (1195/2597) 2024-06-26T07:03:36.8937336Z remote: Counting objects: 47% (1221/2597) 2024-06-26T07:03:36.8937888Z remote: Counting objects: 48% (1247/2597) 2024-06-26T07:03:36.8938519Z remote: Counting objects: 49% (1273/2597) 2024-06-26T07:03:36.8939326Z remote: Counting objects: 50% (1299/2597) 2024-06-26T07:03:36.8940003Z remote: Counting objects: 51% (1325/2597) 2024-06-26T07:03:36.8940525Z remote: Counting objects: 52% (1351/2597) 2024-06-26T07:03:36.8941122Z remote: Counting objects: 53% (1377/2597) 2024-06-26T07:03:36.8941732Z remote: Counting objects: 54% (1403/2597) 2024-06-26T07:03:36.8942383Z remote: Counting objects: 55% (1429/2597) 2024-06-26T07:03:36.8942913Z remote: Counting objects: 56% (1455/2597) 2024-06-26T07:03:36.8943422Z remote: Counting objects: 57% (1481/2597) 2024-06-26T07:03:36.8944063Z remote: Counting objects: 58% (1507/2597) 2024-06-26T07:03:36.8944589Z remote: Counting objects: 59% (1533/2597) 2024-06-26T07:03:36.8945138Z remote: Counting objects: 60% (1559/2597) 2024-06-26T07:03:36.8945693Z remote: Counting objects: 61% (1585/2597) 2024-06-26T07:03:36.8946206Z remote: Counting objects: 62% (1611/2597) 2024-06-26T07:03:36.8946875Z remote: Counting objects: 63% (1637/2597) 2024-06-26T07:03:36.8947417Z remote: Counting objects: 64% (1663/2597) 2024-06-26T07:03:36.8948103Z remote: Counting objects: 65% (1689/2597) 2024-06-26T07:03:36.8948624Z remote: Counting objects: 66% (1715/2597) 2024-06-26T07:03:36.8949257Z remote: Counting objects: 67% (1740/2597) 2024-06-26T07:03:36.8949805Z remote: Counting objects: 68% (1766/2597) 2024-06-26T07:03:36.8950388Z remote: Counting objects: 69% (1792/2597) 2024-06-26T07:03:36.8951070Z remote: Counting objects: 70% (1818/2597) 2024-06-26T07:03:36.8951665Z remote: Counting objects: 71% (1844/2597) 2024-06-26T07:03:36.8952218Z remote: Counting objects: 72% (1870/2597) 2024-06-26T07:03:36.8952813Z remote: Counting objects: 73% (1896/2597) 2024-06-26T07:03:36.8953382Z remote: Counting objects: 74% (1922/2597) 2024-06-26T07:03:36.8953976Z remote: Counting objects: 75% (1948/2597) 2024-06-26T07:03:36.8954502Z remote: Counting objects: 76% (1974/2597) 2024-06-26T07:03:36.8955013Z remote: Counting objects: 77% (2000/2597) 2024-06-26T07:03:36.8955527Z remote: Counting objects: 78% (2026/2597) 2024-06-26T07:03:36.8956129Z remote: Counting objects: 79% (2052/2597) 2024-06-26T07:03:36.8956657Z remote: Counting objects: 80% (2078/2597) 2024-06-26T07:03:36.8957179Z remote: Counting objects: 81% (2104/2597) 2024-06-26T07:03:36.8957709Z remote: Counting objects: 82% (2130/2597) 2024-06-26T07:03:36.8958226Z remote: Counting objects: 83% (2156/2597) 2024-06-26T07:03:36.8958734Z remote: Counting objects: 84% (2182/2597) 2024-06-26T07:03:36.8959259Z remote: Counting objects: 85% (2208/2597) 2024-06-26T07:03:36.8959779Z remote: Counting objects: 86% (2234/2597) 2024-06-26T07:03:36.8960295Z remote: Counting objects: 87% (2260/2597) 2024-06-26T07:03:36.8960805Z remote: Counting objects: 88% (2286/2597) 2024-06-26T07:03:36.8961331Z remote: Counting objects: 89% (2312/2597) 2024-06-26T07:03:36.8961852Z remote: Counting objects: 90% (2338/2597) 2024-06-26T07:03:36.8962368Z remote: Counting objects: 91% (2364/2597) 2024-06-26T07:03:36.8962883Z remote: Counting objects: 92% (2390/2597) 2024-06-26T07:03:36.8963411Z remote: Counting objects: 93% (2416/2597) 2024-06-26T07:03:36.8964113Z remote: Counting objects: 94% (2442/2597) 2024-06-26T07:03:36.8964630Z remote: Counting objects: 95% (2468/2597) 2024-06-26T07:03:36.8965145Z remote: Counting objects: 96% (2494/2597) 2024-06-26T07:03:36.8965665Z remote: Counting objects: 97% (2520/2597) 2024-06-26T07:03:36.8966180Z remote: Counting objects: 98% (2546/2597) 2024-06-26T07:03:36.8966692Z remote: Counting objects: 99% (2572/2597) 2024-06-26T07:03:36.8967203Z remote: Counting objects: 100% (2597/2597) 2024-06-26T07:03:36.8967754Z remote: Counting objects: 100% (2597/2597), done. 2024-06-26T07:03:36.9071269Z remote: Compressing objects: 0% (1/1074) 2024-06-26T07:03:36.9441970Z remote: Compressing objects: 1% (11/1074) 2024-06-26T07:03:36.9827518Z remote: Compressing objects: 2% (22/1074) 2024-06-26T07:03:37.0164660Z remote: Compressing objects: 3% (33/1074) 2024-06-26T07:03:37.0355041Z remote: Compressing objects: 4% (43/1074) 2024-06-26T07:03:37.0592488Z remote: Compressing objects: 5% (54/1074) 2024-06-26T07:03:37.0772373Z remote: Compressing objects: 6% (65/1074) 2024-06-26T07:03:37.1109111Z remote: Compressing objects: 7% (76/1074) 2024-06-26T07:03:37.1862379Z remote: Compressing objects: 8% (86/1074) 2024-06-26T07:03:37.2549921Z remote: Compressing objects: 9% (97/1074) 2024-06-26T07:03:37.2904170Z remote: Compressing objects: 10% (108/1074) 2024-06-26T07:03:37.2929008Z remote: Compressing objects: 11% (119/1074) 2024-06-26T07:03:37.3021207Z remote: Compressing objects: 12% (129/1074) 2024-06-26T07:03:37.3088313Z remote: Compressing objects: 13% (140/1074) 2024-06-26T07:03:37.3130147Z remote: Compressing objects: 14% (151/1074) 2024-06-26T07:03:37.3130722Z remote: Compressing objects: 15% (162/1074) 2024-06-26T07:03:37.3189701Z remote: Compressing objects: 16% (172/1074) 2024-06-26T07:03:37.3190249Z remote: Compressing objects: 17% (183/1074) 2024-06-26T07:03:37.3190782Z remote: Compressing objects: 18% (194/1074) 2024-06-26T07:03:37.3206568Z remote: Compressing objects: 19% (205/1074) 2024-06-26T07:03:37.3224632Z remote: Compressing objects: 20% (215/1074) 2024-06-26T07:03:37.3240141Z remote: Compressing objects: 21% (226/1074) 2024-06-26T07:03:37.3253323Z remote: Compressing objects: 22% (237/1074) 2024-06-26T07:03:37.3273648Z remote: Compressing objects: 23% (248/1074) 2024-06-26T07:03:37.3282875Z remote: Compressing objects: 24% (258/1074) 2024-06-26T07:03:37.3291271Z remote: Compressing objects: 25% (269/1074) 2024-06-26T07:03:37.3298749Z remote: Compressing objects: 26% (280/1074) 2024-06-26T07:03:37.3303353Z remote: Compressing objects: 27% (290/1074) 2024-06-26T07:03:37.3307696Z remote: Compressing objects: 28% (301/1074) 2024-06-26T07:03:37.3316324Z remote: Compressing objects: 29% (312/1074) 2024-06-26T07:03:37.3322355Z remote: Compressing objects: 30% (323/1074) 2024-06-26T07:03:37.3327733Z remote: Compressing objects: 31% (333/1074) 2024-06-26T07:03:37.3333330Z remote: Compressing objects: 32% (344/1074) 2024-06-26T07:03:37.3341473Z remote: Compressing objects: 33% (355/1074) 2024-06-26T07:03:37.3352858Z remote: Compressing objects: 34% (366/1074) 2024-06-26T07:03:37.3368475Z remote: Compressing objects: 35% (376/1074) 2024-06-26T07:03:37.3385560Z remote: Compressing objects: 36% (387/1074) 2024-06-26T07:03:37.3399922Z remote: Compressing objects: 37% (398/1074) 2024-06-26T07:03:37.3415078Z remote: Compressing objects: 38% (409/1074) 2024-06-26T07:03:37.3429452Z remote: Compressing objects: 39% (419/1074) 2024-06-26T07:03:37.3444132Z remote: Compressing objects: 40% (430/1074) 2024-06-26T07:03:37.3461191Z remote: Compressing objects: 41% (441/1074) 2024-06-26T07:03:37.3473640Z remote: Compressing objects: 42% (452/1074) 2024-06-26T07:03:37.3488260Z remote: Compressing objects: 43% (462/1074) 2024-06-26T07:03:37.3490737Z remote: Compressing objects: 44% (473/1074) 2024-06-26T07:03:37.3494365Z remote: Compressing objects: 45% (484/1074) 2024-06-26T07:03:37.3502884Z remote: Compressing objects: 46% (495/1074) 2024-06-26T07:03:37.3515339Z remote: Compressing objects: 47% (505/1074) 2024-06-26T07:03:37.3525638Z remote: Compressing objects: 48% (516/1074) 2024-06-26T07:03:37.3528138Z remote: Compressing objects: 49% (527/1074) 2024-06-26T07:03:37.3535277Z remote: Compressing objects: 50% (537/1074) 2024-06-26T07:03:37.3542846Z remote: Compressing objects: 51% (548/1074) 2024-06-26T07:03:37.3551930Z remote: Compressing objects: 52% (559/1074) 2024-06-26T07:03:37.3566176Z remote: Compressing objects: 53% (570/1074) 2024-06-26T07:03:37.3583298Z remote: Compressing objects: 54% (580/1074) 2024-06-26T07:03:37.3592973Z remote: Compressing objects: 55% (591/1074) 2024-06-26T07:03:37.3602613Z remote: Compressing objects: 56% (602/1074) 2024-06-26T07:03:37.3613784Z remote: Compressing objects: 57% (613/1074) 2024-06-26T07:03:37.3625470Z remote: Compressing objects: 58% (623/1074) 2024-06-26T07:03:37.3635729Z remote: Compressing objects: 59% (634/1074) 2024-06-26T07:03:37.3643671Z remote: Compressing objects: 60% (645/1074) 2024-06-26T07:03:37.3653183Z remote: Compressing objects: 61% (656/1074) 2024-06-26T07:03:37.3663240Z remote: Compressing objects: 62% (666/1074) 2024-06-26T07:03:37.3671774Z remote: Compressing objects: 63% (677/1074) 2024-06-26T07:03:37.3680749Z remote: Compressing objects: 64% (688/1074) 2024-06-26T07:03:37.3687181Z remote: Compressing objects: 65% (699/1074) 2024-06-26T07:03:37.3688999Z remote: Compressing objects: 66% (709/1074) 2024-06-26T07:03:37.3696807Z remote: Compressing objects: 67% (720/1074) 2024-06-26T07:03:37.3701348Z remote: Compressing objects: 68% (731/1074) 2024-06-26T07:03:37.3704352Z remote: Compressing objects: 69% (742/1074) 2024-06-26T07:03:37.3708276Z remote: Compressing objects: 70% (752/1074) 2024-06-26T07:03:37.3711980Z remote: Compressing objects: 71% (763/1074) 2024-06-26T07:03:37.3714449Z remote: Compressing objects: 72% (774/1074) 2024-06-26T07:03:37.3717048Z remote: Compressing objects: 73% (785/1074) 2024-06-26T07:03:37.3717581Z remote: Compressing objects: 74% (795/1074) 2024-06-26T07:03:37.3719650Z remote: Compressing objects: 75% (806/1074) 2024-06-26T07:03:37.3720195Z remote: Compressing objects: 76% (817/1074) 2024-06-26T07:03:37.3720845Z remote: Compressing objects: 77% (827/1074) 2024-06-26T07:03:37.3721418Z remote: Compressing objects: 78% (838/1074) 2024-06-26T07:03:37.3723009Z remote: Compressing objects: 79% (849/1074) 2024-06-26T07:03:37.3725678Z remote: Compressing objects: 80% (860/1074) 2024-06-26T07:03:37.3726221Z remote: Compressing objects: 81% (870/1074) 2024-06-26T07:03:37.3728175Z remote: Compressing objects: 82% (881/1074) 2024-06-26T07:03:37.3728711Z remote: Compressing objects: 83% (892/1074) 2024-06-26T07:03:37.3729306Z remote: Compressing objects: 84% (903/1074) 2024-06-26T07:03:37.3733172Z remote: Compressing objects: 85% (913/1074) 2024-06-26T07:03:37.3733714Z remote: Compressing objects: 86% (924/1074) 2024-06-26T07:03:37.3735778Z remote: Compressing objects: 87% (935/1074) 2024-06-26T07:03:37.3738434Z remote: Compressing objects: 88% (946/1074) 2024-06-26T07:03:37.3738978Z remote: Compressing objects: 89% (956/1074) 2024-06-26T07:03:37.3739578Z remote: Compressing objects: 90% (967/1074) 2024-06-26T07:03:37.3740153Z remote: Compressing objects: 91% (978/1074) 2024-06-26T07:03:37.3743369Z remote: Compressing objects: 92% (989/1074) 2024-06-26T07:03:37.3743913Z remote: Compressing objects: 93% (999/1074) 2024-06-26T07:03:37.3744557Z remote: Compressing objects: 94% (1010/1074) 2024-06-26T07:03:37.3745791Z remote: Compressing objects: 95% (1021/1074) 2024-06-26T07:03:37.3746344Z remote: Compressing objects: 96% (1032/1074) 2024-06-26T07:03:37.3748561Z remote: Compressing objects: 97% (1042/1074) 2024-06-26T07:03:37.3749125Z remote: Compressing objects: 98% (1053/1074) 2024-06-26T07:03:37.3749787Z remote: Compressing objects: 99% (1064/1074) 2024-06-26T07:03:37.3750859Z remote: Compressing objects: 100% (1074/1074) 2024-06-26T07:03:37.3751452Z remote: Compressing objects: 100% (1074/1074), done. 2024-06-26T07:03:57.4840462Z remote: Total 990209 (delta 2062), reused 1863 (delta 1522), pack-reused 987612 2024-06-26T07:04:21.2967176Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T07:04:21.2984488Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:04:21.2987784Z ##[endgroup] 2024-06-26T07:04:21.2988320Z ##[group]Determining the checkout info 2024-06-26T07:04:21.2990918Z ##[endgroup] 2024-06-26T07:04:21.2991521Z ##[group]Checking out the ref 2024-06-26T07:04:21.2992714Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:04:22.5177025Z ##[endgroup] 2024-06-26T07:04:22.5177629Z ##[group]Setting up auth for fetching submodules 2024-06-26T07:04:22.5181112Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:04:22.5226521Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T07:04:22.5247400Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T07:04:22.5272541Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T07:04:22.5294346Z ##[endgroup] 2024-06-26T07:04:22.5294878Z ##[group]Fetching submodules 2024-06-26T07:04:22.5298553Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T07:04:22.5513896Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T07:04:22.5723046Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T07:04:22.5724692Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T07:04:22.5726340Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T07:04:22.5728561Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T07:04:22.5731426Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:22.5733350Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T07:04:22.5735780Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T07:04:22.5738588Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T07:04:22.5741103Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T07:04:22.5743952Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T07:04:22.5746634Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T07:04:22.5749451Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T07:04:22.5752389Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T07:04:22.5755485Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T07:04:22.5758337Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T07:04:22.5761455Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T07:04:22.5765868Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:22.5768928Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T07:04:22.5772220Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T07:04:22.5775431Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T07:04:22.5779151Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T07:04:22.5782581Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T07:04:22.5786349Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T07:04:22.5789904Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T07:04:22.5793605Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T07:04:22.5797297Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T07:04:22.5801453Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T07:04:22.5805354Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T07:04:22.5809347Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T07:04:22.5813423Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T07:04:22.5817728Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T07:04:22.5821880Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T07:04:22.5826025Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T07:04:22.5830144Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T07:04:22.5834614Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T07:04:22.5855485Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T07:04:22.8359673Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T07:04:22.9979331Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T07:04:23.1819807Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T07:04:23.4343182Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T07:04:25.4267790Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T07:04:35.5546592Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T07:04:35.9276711Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T07:04:36.4510757Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T07:04:37.0732701Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T07:04:38.2276769Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T07:04:40.0239063Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T07:04:45.9278339Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T07:04:47.2278004Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T07:04:48.9663487Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T07:04:50.1704625Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T07:04:50.3187424Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T07:04:50.7048340Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T07:04:51.0425237Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T07:04:52.0213653Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T07:04:52.3509785Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T07:04:52.5959584Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T07:04:54.1590276Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T07:04:54.8955506Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T07:04:55.5481498Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T07:05:01.3925402Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T07:05:03.8971648Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T07:05:09.4447919Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T07:05:09.6678692Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T07:05:17.9481927Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T07:05:18.1034081Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T07:05:18.2931696Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T07:05:19.3542398Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T07:05:19.6593641Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T07:05:20.2720483Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T07:05:20.6902810Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T07:05:20.6989463Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T07:05:20.7048563Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T07:05:20.7232616Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T07:05:20.7537170Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T07:05:21.5310205Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T07:05:21.5481129Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T07:05:21.5830211Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T07:05:21.6688797Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T07:05:21.6959127Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T07:05:22.1298217Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T07:05:22.3422632Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T07:05:22.4056775Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T07:05:22.4068930Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:05:22.4072137Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:05:22.4073707Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:05:22.4075240Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:05:22.4076939Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:05:22.4091786Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T07:05:23.4066993Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T07:05:23.9808613Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T07:05:25.7089460Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T07:05:26.7078726Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T07:05:27.0555246Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T07:05:27.1389648Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T07:05:27.4804187Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T07:05:27.5347837Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T07:05:27.5435643Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T07:05:27.6377495Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T07:05:27.6659752Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T07:05:27.6722333Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T07:05:27.7025379Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T07:05:27.7216203Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T07:05:27.7578552Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:05:27.7674075Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T07:05:27.7684901Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T07:05:27.7700357Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T07:05:39.8001389Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T07:05:39.8142093Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T07:05:39.8920639Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T07:05:39.8932304Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:05:39.8934006Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:05:39.8935701Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:05:39.8953227Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T07:05:40.4378034Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T07:05:41.6978490Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T07:05:42.7214727Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T07:05:42.7225249Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:05:42.7227011Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:05:42.7228761Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:05:42.7230805Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:05:42.7232674Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:05:42.7234980Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:05:42.7237204Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:05:42.7239333Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:05:42.7258319Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T07:05:43.7526081Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T07:05:44.1064201Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T07:05:45.2892827Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T07:05:45.5600607Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T07:05:46.0434886Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T07:05:47.0199652Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T07:05:52.7702562Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T07:05:53.1134540Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T07:05:53.1271087Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T07:05:53.1564933Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T07:05:53.1659757Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T07:05:53.1668846Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:05:53.1684479Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T07:05:53.4632922Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T07:05:53.4768316Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T07:05:53.5106662Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T07:05:53.5925957Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T07:05:53.6042181Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T07:05:53.6366019Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T07:05:53.6822604Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T07:05:53.7124138Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T07:05:53.7315038Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T07:05:53.8190101Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T07:05:54.0920454Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T07:05:54.0943674Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T07:05:54.0945416Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T07:05:54.0971135Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T07:05:54.5404162Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T07:05:55.6270875Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T07:05:55.6565329Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T07:05:55.7104929Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T07:05:55.7116764Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:05:55.7125596Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:05:55.7127245Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:05:55.7129160Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:05:55.7131104Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:05:55.7133144Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:05:55.7135211Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:05:55.7136975Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:05:55.7151196Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T07:05:57.0985179Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T07:05:58.0640681Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T07:05:58.3867605Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T07:06:04.1445849Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T07:06:04.3985781Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T07:06:04.5728012Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T07:06:04.8484158Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T07:06:12.1331795Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T07:06:12.1663441Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T07:06:12.1787729Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T07:06:12.2639029Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T07:06:12.2729969Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T07:06:12.2832003Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T07:06:12.2946159Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T07:06:12.2956626Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:06:12.2958467Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:06:12.2974591Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T07:06:14.0086649Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T07:06:15.1609269Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T07:06:15.1973286Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:06:15.5581231Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T07:06:15.5664110Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T07:06:15.7817836Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T07:06:15.7830946Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:06:15.7833012Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T07:06:15.7849780Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T07:06:16.1925813Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T07:06:17.1357670Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T07:06:17.1984383Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T07:06:17.2044857Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T07:06:17.2131728Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T07:06:17.2411156Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T07:06:17.2631460Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T07:06:17.2966214Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T07:06:17.3169308Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T07:06:17.3182767Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:06:17.3184676Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:06:17.3186762Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:06:17.3189092Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:06:17.3204896Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T07:06:18.2753016Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T07:06:18.4970435Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T07:06:19.5461899Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T07:06:20.5678603Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T07:06:20.5793295Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T07:06:20.6356163Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T07:06:20.6585030Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T07:06:20.6595549Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:06:20.6611947Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T07:06:20.8698179Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T07:06:20.8720586Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T07:06:20.8919935Z Entering 'android/libs/fbjni' 2024-06-26T07:06:20.8945500Z Entering 'third_party/FP16' 2024-06-26T07:06:20.8970651Z Entering 'third_party/FXdiv' 2024-06-26T07:06:20.8997770Z Entering 'third_party/NNPACK' 2024-06-26T07:06:20.9026767Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:06:20.9052550Z Entering 'third_party/XNNPACK' 2024-06-26T07:06:20.9089337Z Entering 'third_party/benchmark' 2024-06-26T07:06:20.9116200Z Entering 'third_party/cpp-httplib' 2024-06-26T07:06:20.9145149Z Entering 'third_party/cpuinfo' 2024-06-26T07:06:20.9170802Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:06:20.9196978Z Entering 'third_party/cutlass' 2024-06-26T07:06:20.9232507Z Entering 'third_party/eigen' 2024-06-26T07:06:20.9263394Z Entering 'third_party/fbgemm' 2024-06-26T07:06:20.9289794Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:06:20.9318600Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:06:20.9344753Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:06:20.9377810Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:06:20.9405540Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:06:20.9435349Z Entering 'third_party/flatbuffers' 2024-06-26T07:06:20.9465074Z Entering 'third_party/fmt' 2024-06-26T07:06:20.9493733Z Entering 'third_party/foxi' 2024-06-26T07:06:20.9521000Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:06:20.9549918Z Entering 'third_party/gloo' 2024-06-26T07:06:20.9577300Z Entering 'third_party/googletest' 2024-06-26T07:06:20.9604688Z Entering 'third_party/ideep' 2024-06-26T07:06:20.9630631Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:06:20.9662993Z Entering 'third_party/ittapi' 2024-06-26T07:06:20.9688871Z Entering 'third_party/kineto' 2024-06-26T07:06:20.9717975Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:06:20.9748538Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:06:20.9776307Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:06:20.9803769Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:06:20.9831939Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:06:20.9858079Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:06:20.9886102Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:06:20.9914336Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:06:20.9941615Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:06:20.9969937Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:06:20.9999265Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:06:21.0027859Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:06:21.0054887Z Entering 'third_party/mimalloc' 2024-06-26T07:06:21.0083226Z Entering 'third_party/nccl/nccl' 2024-06-26T07:06:21.0114036Z Entering 'third_party/nlohmann' 2024-06-26T07:06:21.0144140Z Entering 'third_party/onnx' 2024-06-26T07:06:21.0183908Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:06:21.0212994Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:06:21.0241914Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:06:21.0273223Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:06:21.0299826Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:06:21.0328001Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:06:21.0360166Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:06:21.0390564Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:06:21.0417670Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:06:21.0446560Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:06:21.0472247Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:06:21.0500543Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:06:21.0529567Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:06:21.0571468Z Entering 'third_party/pocketfft' 2024-06-26T07:06:21.0601248Z Entering 'third_party/protobuf' 2024-06-26T07:06:21.0629215Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:06:21.0657547Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:06:21.0683506Z Entering 'third_party/psimd' 2024-06-26T07:06:21.0713495Z Entering 'third_party/pthreadpool' 2024-06-26T07:06:21.0740562Z Entering 'third_party/pybind11' 2024-06-26T07:06:21.0767910Z Entering 'third_party/python-peachpy' 2024-06-26T07:06:21.0793700Z Entering 'third_party/sleef' 2024-06-26T07:06:21.0824581Z Entering 'third_party/tensorpipe' 2024-06-26T07:06:21.0854014Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:06:21.0881083Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:06:21.0908933Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:06:21.0936206Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:06:21.0961349Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:06:21.0999948Z ##[endgroup] 2024-06-26T07:06:21.1001968Z ##[group]Persisting credentials for submodules 2024-06-26T07:06:21.1005160Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T07:06:21.1219841Z Entering 'android/libs/fbjni' 2024-06-26T07:06:21.1254887Z Entering 'third_party/FP16' 2024-06-26T07:06:21.1291359Z Entering 'third_party/FXdiv' 2024-06-26T07:06:21.1324598Z Entering 'third_party/NNPACK' 2024-06-26T07:06:21.1359248Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:06:21.1394342Z Entering 'third_party/XNNPACK' 2024-06-26T07:06:21.1439563Z Entering 'third_party/benchmark' 2024-06-26T07:06:21.1474848Z Entering 'third_party/cpp-httplib' 2024-06-26T07:06:21.1510824Z Entering 'third_party/cpuinfo' 2024-06-26T07:06:21.1547151Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:06:21.1582962Z Entering 'third_party/cutlass' 2024-06-26T07:06:21.1623723Z Entering 'third_party/eigen' 2024-06-26T07:06:21.1659406Z Entering 'third_party/fbgemm' 2024-06-26T07:06:21.1695839Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:06:21.1730566Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:06:21.1766621Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:06:21.1805714Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:06:21.1839285Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:06:21.1875475Z Entering 'third_party/flatbuffers' 2024-06-26T07:06:21.1914047Z Entering 'third_party/fmt' 2024-06-26T07:06:21.1951400Z Entering 'third_party/foxi' 2024-06-26T07:06:21.1987920Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:06:21.2020906Z Entering 'third_party/gloo' 2024-06-26T07:06:21.2056434Z Entering 'third_party/googletest' 2024-06-26T07:06:21.2092441Z Entering 'third_party/ideep' 2024-06-26T07:06:21.2126399Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:06:21.2166246Z Entering 'third_party/ittapi' 2024-06-26T07:06:21.2199695Z Entering 'third_party/kineto' 2024-06-26T07:06:21.2234986Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:06:21.2269191Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:06:21.2303175Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:06:21.2335877Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:06:21.2371174Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:06:21.2405346Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:06:21.2440786Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:06:21.2475464Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:06:21.2509145Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:06:21.2546715Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:06:21.2582063Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:06:21.2618150Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:06:21.2652377Z Entering 'third_party/mimalloc' 2024-06-26T07:06:21.2687476Z Entering 'third_party/nccl/nccl' 2024-06-26T07:06:21.2724066Z Entering 'third_party/nlohmann' 2024-06-26T07:06:21.2760166Z Entering 'third_party/onnx' 2024-06-26T07:06:21.2808991Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:06:21.2844562Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:06:21.2881195Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:06:21.2918185Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:06:21.2953754Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:06:21.2986355Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:06:21.3022246Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:06:21.3058727Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:06:21.3091463Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:06:21.3127396Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:06:21.3160401Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:06:21.3197857Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:06:21.3234296Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:06:21.3287040Z Entering 'third_party/pocketfft' 2024-06-26T07:06:21.3321669Z Entering 'third_party/protobuf' 2024-06-26T07:06:21.3358722Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:06:21.3392935Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:06:21.3429636Z Entering 'third_party/psimd' 2024-06-26T07:06:21.3464446Z Entering 'third_party/pthreadpool' 2024-06-26T07:06:21.3501310Z Entering 'third_party/pybind11' 2024-06-26T07:06:21.3539934Z Entering 'third_party/python-peachpy' 2024-06-26T07:06:21.3573685Z Entering 'third_party/sleef' 2024-06-26T07:06:21.3609408Z Entering 'third_party/tensorpipe' 2024-06-26T07:06:21.3645223Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:06:21.3679185Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:06:21.3713589Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:06:21.3744932Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:06:21.3779590Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:06:21.3827751Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T07:06:21.4041110Z Entering 'android/libs/fbjni' 2024-06-26T07:06:21.4073733Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T07:06:21.4086093Z Entering 'third_party/FP16' 2024-06-26T07:06:21.4117399Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T07:06:21.4130553Z Entering 'third_party/FXdiv' 2024-06-26T07:06:21.4163001Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T07:06:21.4176922Z Entering 'third_party/NNPACK' 2024-06-26T07:06:21.4207783Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T07:06:21.4222032Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:06:21.4251717Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T07:06:21.4265596Z Entering 'third_party/XNNPACK' 2024-06-26T07:06:21.4298108Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T07:06:21.4321858Z Entering 'third_party/benchmark' 2024-06-26T07:06:21.4355940Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:06:21.4369572Z Entering 'third_party/cpp-httplib' 2024-06-26T07:06:21.4402729Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T07:06:21.4415912Z Entering 'third_party/cpuinfo' 2024-06-26T07:06:21.4447785Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:06:21.4461751Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:06:21.4492018Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T07:06:21.4504708Z Entering 'third_party/cutlass' 2024-06-26T07:06:21.4536968Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:06:21.4555839Z Entering 'third_party/eigen' 2024-06-26T07:06:21.4588472Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T07:06:21.4603166Z Entering 'third_party/fbgemm' 2024-06-26T07:06:21.4636526Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T07:06:21.4650024Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:06:21.4682222Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T07:06:21.4694745Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:06:21.4725539Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:06:21.4738947Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:06:21.4769572Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:06:21.4787172Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:06:21.4818380Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:06:21.4831131Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:06:21.4862684Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T07:06:21.4877002Z Entering 'third_party/flatbuffers' 2024-06-26T07:06:21.4910285Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T07:06:21.4925919Z Entering 'third_party/fmt' 2024-06-26T07:06:21.4956142Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:06:21.4968213Z Entering 'third_party/foxi' 2024-06-26T07:06:21.4999710Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T07:06:21.5013723Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:06:21.5044234Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T07:06:21.5057010Z Entering 'third_party/gloo' 2024-06-26T07:06:21.5088010Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T07:06:21.5101175Z Entering 'third_party/googletest' 2024-06-26T07:06:21.5131983Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:06:21.5144516Z Entering 'third_party/ideep' 2024-06-26T07:06:21.5176950Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T07:06:21.5187862Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:06:21.5218570Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T07:06:21.5237769Z Entering 'third_party/ittapi' 2024-06-26T07:06:21.5270670Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T07:06:21.5284685Z Entering 'third_party/kineto' 2024-06-26T07:06:21.5317773Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T07:06:21.5329642Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:06:21.5362339Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T07:06:21.5374932Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:06:21.5407322Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T07:06:21.5421050Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:06:21.5453540Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T07:06:21.5466417Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:06:21.5498431Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:06:21.5512231Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:06:21.5543405Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T07:06:21.5556098Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:06:21.5589053Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T07:06:21.5603339Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:06:21.5636310Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T07:06:21.5649488Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:06:21.5681438Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:06:21.5693724Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:06:21.5725889Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T07:06:21.5739255Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:06:21.5774067Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T07:06:21.5788238Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:06:21.5820374Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T07:06:21.5833279Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:06:21.5865959Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T07:06:21.5880352Z Entering 'third_party/mimalloc' 2024-06-26T07:06:21.5913459Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T07:06:21.5926944Z Entering 'third_party/nccl/nccl' 2024-06-26T07:06:21.5959835Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T07:06:21.5972654Z Entering 'third_party/nlohmann' 2024-06-26T07:06:21.6003317Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T07:06:21.6016940Z Entering 'third_party/onnx' 2024-06-26T07:06:21.6047981Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T07:06:21.6074055Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:06:21.6106245Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:06:21.6118565Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:06:21.6152848Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:06:21.6167651Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:06:21.6199900Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T07:06:21.6215133Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:06:21.6248021Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:06:21.6260498Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:06:21.6291087Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:06:21.6303905Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:06:21.6336257Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T07:06:21.6348703Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:06:21.6379483Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T07:06:21.6393007Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:06:21.6425448Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T07:06:21.6437282Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:06:21.6471000Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T07:06:21.6482495Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:06:21.6515483Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T07:06:21.6527178Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:06:21.6558084Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T07:06:21.6572356Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:06:21.6604388Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T07:06:21.6618346Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:06:21.6651057Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T07:06:21.6680654Z Entering 'third_party/pocketfft' 2024-06-26T07:06:21.6714459Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T07:06:21.6728827Z Entering 'third_party/protobuf' 2024-06-26T07:06:21.6760833Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T07:06:21.6777511Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:06:21.6807827Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:06:21.6820673Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:06:21.6853500Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:06:21.6867480Z Entering 'third_party/psimd' 2024-06-26T07:06:21.6898728Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T07:06:21.6912891Z Entering 'third_party/pthreadpool' 2024-06-26T07:06:21.6945401Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T07:06:21.6958791Z Entering 'third_party/pybind11' 2024-06-26T07:06:21.6991350Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:06:21.7004982Z Entering 'third_party/python-peachpy' 2024-06-26T07:06:21.7036622Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T07:06:21.7050875Z Entering 'third_party/sleef' 2024-06-26T07:06:21.7081797Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T07:06:21.7095386Z Entering 'third_party/tensorpipe' 2024-06-26T07:06:21.7126979Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T07:06:21.7139709Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:06:21.7171523Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:06:21.7184637Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:06:21.7218047Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T07:06:21.7230718Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:06:21.7263319Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T07:06:21.7275695Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:06:21.7307774Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:06:21.7319255Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:06:21.7352522Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T07:06:21.8244431Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T07:06:21.8459448Z Entering 'android/libs/fbjni' 2024-06-26T07:06:21.8489480Z Entering 'third_party/FP16' 2024-06-26T07:06:21.8517989Z Entering 'third_party/FXdiv' 2024-06-26T07:06:21.8545102Z Entering 'third_party/NNPACK' 2024-06-26T07:06:21.8578421Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:06:21.8606301Z Entering 'third_party/XNNPACK' 2024-06-26T07:06:21.8644476Z Entering 'third_party/benchmark' 2024-06-26T07:06:21.8674651Z Entering 'third_party/cpp-httplib' 2024-06-26T07:06:21.8701976Z Entering 'third_party/cpuinfo' 2024-06-26T07:06:21.8729947Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:06:21.8759751Z Entering 'third_party/cutlass' 2024-06-26T07:06:21.8794326Z Entering 'third_party/eigen' 2024-06-26T07:06:21.8823676Z Entering 'third_party/fbgemm' 2024-06-26T07:06:21.8851021Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:06:21.8877493Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:06:21.8905658Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:06:21.8936374Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:06:21.8963410Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:06:21.8992607Z Entering 'third_party/flatbuffers' 2024-06-26T07:06:21.9023697Z Entering 'third_party/fmt' 2024-06-26T07:06:21.9053500Z Entering 'third_party/foxi' 2024-06-26T07:06:21.9080817Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:06:21.9109785Z Entering 'third_party/gloo' 2024-06-26T07:06:21.9141974Z Entering 'third_party/googletest' 2024-06-26T07:06:21.9170226Z Entering 'third_party/ideep' 2024-06-26T07:06:21.9196692Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:06:21.9231275Z Entering 'third_party/ittapi' 2024-06-26T07:06:21.9259081Z Entering 'third_party/kineto' 2024-06-26T07:06:21.9285889Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:06:21.9314096Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:06:21.9345207Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:06:21.9372116Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:06:21.9400111Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:06:21.9427385Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:06:21.9454999Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:06:21.9481237Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:06:21.9509833Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:06:21.9539423Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:06:21.9569784Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:06:21.9597283Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:06:21.9624792Z Entering 'third_party/mimalloc' 2024-06-26T07:06:21.9653984Z Entering 'third_party/nccl/nccl' 2024-06-26T07:06:21.9681143Z Entering 'third_party/nlohmann' 2024-06-26T07:06:21.9711050Z Entering 'third_party/onnx' 2024-06-26T07:06:21.9749240Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:06:21.9779090Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:06:21.9809504Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:06:21.9839871Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:06:21.9868731Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:06:21.9899486Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:06:21.9925964Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:06:21.9954378Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:06:21.9983538Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:06:22.0010734Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:06:22.0038980Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:06:22.0069194Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:06:22.0099504Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:06:22.0145170Z Entering 'third_party/pocketfft' 2024-06-26T07:06:22.0175016Z Entering 'third_party/protobuf' 2024-06-26T07:06:22.0207496Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:06:22.0234959Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:06:22.0264079Z Entering 'third_party/psimd' 2024-06-26T07:06:22.0290842Z Entering 'third_party/pthreadpool' 2024-06-26T07:06:22.0317265Z Entering 'third_party/pybind11' 2024-06-26T07:06:22.0346003Z Entering 'third_party/python-peachpy' 2024-06-26T07:06:22.0372150Z Entering 'third_party/sleef' 2024-06-26T07:06:22.0401177Z Entering 'third_party/tensorpipe' 2024-06-26T07:06:22.0431267Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:06:22.0458831Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:06:22.0488388Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:06:22.0517175Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:06:22.0544093Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:06:22.0582249Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T07:06:22.0794430Z Entering 'android/libs/fbjni' 2024-06-26T07:06:22.0823504Z Entering 'third_party/FP16' 2024-06-26T07:06:22.0849668Z Entering 'third_party/FXdiv' 2024-06-26T07:06:22.0874998Z Entering 'third_party/NNPACK' 2024-06-26T07:06:22.0903526Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:06:22.0933822Z Entering 'third_party/XNNPACK' 2024-06-26T07:06:22.0971796Z Entering 'third_party/benchmark' 2024-06-26T07:06:22.1000941Z Entering 'third_party/cpp-httplib' 2024-06-26T07:06:22.1026016Z Entering 'third_party/cpuinfo' 2024-06-26T07:06:22.1055157Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:06:22.1084993Z Entering 'third_party/cutlass' 2024-06-26T07:06:22.1120190Z Entering 'third_party/eigen' 2024-06-26T07:06:22.1151641Z Entering 'third_party/fbgemm' 2024-06-26T07:06:22.1182662Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:06:22.1210067Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:06:22.1238852Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:06:22.1272518Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:06:22.1300205Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:06:22.1329890Z Entering 'third_party/flatbuffers' 2024-06-26T07:06:22.1360101Z Entering 'third_party/fmt' 2024-06-26T07:06:22.1389469Z Entering 'third_party/foxi' 2024-06-26T07:06:22.1416749Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:06:22.1442017Z Entering 'third_party/gloo' 2024-06-26T07:06:22.1471303Z Entering 'third_party/googletest' 2024-06-26T07:06:22.1498483Z Entering 'third_party/ideep' 2024-06-26T07:06:22.1526272Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:06:22.1559182Z Entering 'third_party/ittapi' 2024-06-26T07:06:22.1588542Z Entering 'third_party/kineto' 2024-06-26T07:06:22.1614377Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:06:22.1639384Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:06:22.1670063Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:06:22.1697514Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:06:22.1722853Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:06:22.1751494Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:06:22.1780868Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:06:22.1810476Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:06:22.1836580Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:06:22.1865652Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:06:22.1895125Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:06:22.1922590Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:06:22.1952429Z Entering 'third_party/mimalloc' 2024-06-26T07:06:22.1980594Z Entering 'third_party/nccl/nccl' 2024-06-26T07:06:22.2009455Z Entering 'third_party/nlohmann' 2024-06-26T07:06:22.2038385Z Entering 'third_party/onnx' 2024-06-26T07:06:22.2080480Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:06:22.2109952Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:06:22.2140046Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:06:22.2169885Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:06:22.2197008Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:06:22.2224814Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:06:22.2252262Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:06:22.2279960Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:06:22.2307815Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:06:22.2336291Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:06:22.2362767Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:06:22.2393559Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:06:22.2421197Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:06:22.2464486Z Entering 'third_party/pocketfft' 2024-06-26T07:06:22.2493980Z Entering 'third_party/protobuf' 2024-06-26T07:06:22.2526105Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:06:22.2553548Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:06:22.2582608Z Entering 'third_party/psimd' 2024-06-26T07:06:22.2612183Z Entering 'third_party/pthreadpool' 2024-06-26T07:06:22.2640214Z Entering 'third_party/pybind11' 2024-06-26T07:06:22.2669545Z Entering 'third_party/python-peachpy' 2024-06-26T07:06:22.2698171Z Entering 'third_party/sleef' 2024-06-26T07:06:22.2727921Z Entering 'third_party/tensorpipe' 2024-06-26T07:06:22.2756988Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:06:22.2784039Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:06:22.2811431Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:06:22.2838650Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:06:22.2866701Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:06:22.2902952Z ##[endgroup] 2024-06-26T07:06:22.2936160Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T07:06:22.2953538Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T07:06:22.3126018Z Prepare all required actions 2024-06-26T07:06:22.3126551Z Getting action download info 2024-06-26T07:06:22.4512785Z ##[group]Run ./.github/actions/setup-linux 2024-06-26T07:06:22.4513215Z env: 2024-06-26T07:06:22.4513493Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:22.4513841Z ##[endgroup] 2024-06-26T07:06:22.4571947Z ##[group]Run set -euo pipefail 2024-06-26T07:06:22.4572387Z set -euo pipefail 2024-06-26T07:06:22.4572773Z function get_ec2_metadata() { 2024-06-26T07:06:22.4573334Z  # Pulled from instance metadata endpoint for EC2 2024-06-26T07:06:22.4574243Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-26T07:06:22.4575007Z  category=$1 2024-06-26T07:06:22.4575530Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-26T07:06:22.4576162Z  runner_name_str=i-0262ed88bcec9993d 2024-06-26T07:06:22.4576765Z  if [[ -f /.inarc ]]; then 2024-06-26T07:06:22.4577255Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-26T07:06:22.4577820Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-26T07:06:22.4578520Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-26T07:06:22.4579126Z  else 2024-06-26T07:06:22.4579611Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-26T07:06:22.4580181Z  fi 2024-06-26T07:06:22.4580466Z } 2024-06-26T07:06:22.4580833Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-26T07:06:22.4581408Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-26T07:06:22.4582058Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-26T07:06:22.4582622Z echo "system info $(uname -a)" 2024-06-26T07:06:22.4590403Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:22.4590890Z env: 2024-06-26T07:06:22.4591168Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:22.4591515Z ##[endgroup] 2024-06-26T07:06:22.4662713Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-26T07:06:22.4700747Z instance-id: i-0262ed88bcec9993d 2024-06-26T07:06:22.4744111Z instance-type: g5.4xlarge 2024-06-26T07:06:22.4751925Z system info Linux ip-10-0-42-220.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-26T07:06:22.4771944Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:06:22.4772858Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:06:22.4780914Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:22.4781409Z env: 2024-06-26T07:06:22.4781692Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:22.4782043Z ##[endgroup] 2024-06-26T07:06:22.4845754Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-26T07:06:22.4846457Z if systemctl is-active --quiet docker; then 2024-06-26T07:06:22.4847273Z  echo "Docker daemon is running..."; 2024-06-26T07:06:22.4847942Z else 2024-06-26T07:06:22.4848517Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-26T07:06:22.4849185Z fi 2024-06-26T07:06:22.4857030Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:22.4857607Z env: 2024-06-26T07:06:22.4857969Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:22.4858392Z ##[endgroup] 2024-06-26T07:06:22.4896537Z Docker daemon is running... 2024-06-26T07:06:22.4943143Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:06:22.4943777Z with: 2024-06-26T07:06:22.4944118Z shell: bash 2024-06-26T07:06:22.4944488Z timeout_minutes: 5 2024-06-26T07:06:22.4944885Z max_attempts: 3 2024-06-26T07:06:22.4945281Z retry_wait_seconds: 30 2024-06-26T07:06:22.4946862Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-26T07:06:22.4948373Z polling_interval_seconds: 1 2024-06-26T07:06:22.4948833Z warning_on_retry: true 2024-06-26T07:06:22.4949276Z continue_on_error: false 2024-06-26T07:06:22.4949695Z env: 2024-06-26T07:06:22.4950041Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:22.4950480Z AWS_RETRY_MODE: standard 2024-06-26T07:06:22.4950921Z AWS_MAX_ATTEMPTS: 5 2024-06-26T07:06:22.4951350Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:06:22.4951802Z ##[endgroup] 2024-06-26T07:06:23.3711038Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:06:23.3711892Z Configure a credential helper to remove this warning. See 2024-06-26T07:06:23.3712737Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:06:23.3713270Z 2024-06-26T07:06:23.3713437Z Login Succeeded 2024-06-26T07:06:23.5447579Z Command completed after 1 attempt(s). 2024-06-26T07:06:23.5488936Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:06:23.5489670Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:06:23.5490332Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:06:23.5498331Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:23.5498822Z env: 2024-06-26T07:06:23.5499099Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:23.5499447Z ##[endgroup] 2024-06-26T07:06:23.5558177Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:06:23.5558938Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:06:23.5559536Z # shellcheck disable=SC2046 2024-06-26T07:06:23.5559999Z docker stop $(docker ps -q) || true 2024-06-26T07:06:23.5560493Z # Prune all of the docker images 2024-06-26T07:06:23.5560945Z docker system prune -af 2024-06-26T07:06:23.5568585Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:23.5569101Z env: 2024-06-26T07:06:23.5569376Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:23.5569729Z ##[endgroup] 2024-06-26T07:06:23.5812087Z "docker stop" requires at least 1 argument. 2024-06-26T07:06:23.5812763Z See 'docker stop --help'. 2024-06-26T07:06:23.5812998Z 2024-06-26T07:06:23.5813223Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T07:06:23.5813605Z 2024-06-26T07:06:23.5813756Z Stop one or more running containers 2024-06-26T07:06:23.6006955Z Total reclaimed space: 0B 2024-06-26T07:06:23.6036386Z ##[group]Run set +e 2024-06-26T07:06:23.6036742Z set +e 2024-06-26T07:06:23.6037043Z set -x 2024-06-26T07:06:23.6037340Z  2024-06-26T07:06:23.6037648Z PT_DOMAIN=download.pytorch.org 2024-06-26T07:06:23.6038433Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-26T07:06:23.6039563Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-26T07:06:23.6040418Z # one is returned at random 2024-06-26T07:06:23.6040955Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-26T07:06:23.6041451Z  2024-06-26T07:06:23.6041761Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:06:23.6042367Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-26T07:06:23.6043117Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-26T07:06:23.6043917Z  2024-06-26T07:06:23.6044241Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:06:23.6044785Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-26T07:06:23.6045286Z  exit 1 2024-06-26T07:06:23.6045587Z  fi 2024-06-26T07:06:23.6045867Z fi 2024-06-26T07:06:23.6046268Z  2024-06-26T07:06:23.6046629Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-26T07:06:23.6047134Z  # Clean up any old records first 2024-06-26T07:06:23.6047634Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-26T07:06:23.6048075Z fi 2024-06-26T07:06:23.6048344Z  2024-06-26T07:06:23.6048765Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-26T07:06:23.6049304Z cat /etc/hosts 2024-06-26T07:06:23.6057110Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:23.6057596Z env: 2024-06-26T07:06:23.6057870Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:23.6058220Z ##[endgroup] 2024-06-26T07:06:23.6078262Z + PT_DOMAIN=download.pytorch.org 2024-06-26T07:06:23.6081642Z ++ dig -4 +short download.pytorch.org 2024-06-26T07:06:23.6082325Z ++ tail -n1 2024-06-26T07:06:23.6207142Z + RESOLVED_IP=18.160.10.36 2024-06-26T07:06:23.6207612Z + '[' -z 18.160.10.36 ']' 2024-06-26T07:06:23.6208069Z + grep -r download.pytorch.org /etc/hosts 2024-06-26T07:06:23.6213016Z 18.67.76.99 download.pytorch.org 2024-06-26T07:06:23.6213870Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-06-26T07:06:23.6316475Z + echo '18.160.10.36 download.pytorch.org' 2024-06-26T07:06:23.6316970Z + sudo tee -a /etc/hosts 2024-06-26T07:06:23.6638456Z 18.160.10.36 download.pytorch.org 2024-06-26T07:06:23.6655651Z + cat /etc/hosts 2024-06-26T07:06:23.6660058Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-26T07:06:23.6673390Z ::1 localhost6 localhost6.localdomain6 2024-06-26T07:06:23.6673879Z 18.160.10.36 download.pytorch.org 2024-06-26T07:06:23.6879839Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T07:06:23.6880434Z with: 2024-06-26T07:06:23.6881541Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:23.6882774Z docker-build-dir: .ci/docker 2024-06-26T07:06:23.6883142Z working-directory: . 2024-06-26T07:06:23.6883811Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:23.6884393Z force-push: false 2024-06-26T07:06:23.6884684Z env: 2024-06-26T07:06:23.6884946Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:23.6885271Z ##[endgroup] 2024-06-26T07:06:23.6903924Z ##[group]Run set -ex 2024-06-26T07:06:23.6904284Z set -ex 2024-06-26T07:06:23.6904581Z  2024-06-26T07:06:23.6905127Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T07:06:23.6906094Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T07:06:23.6906901Z # job could then download the pre-built image as usual 2024-06-26T07:06:23.6907636Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T07:06:23.6908295Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:23.6908921Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:23.6909597Z  2024-06-26T07:06:23.6910094Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T07:06:23.6910709Z  exit 0 2024-06-26T07:06:23.6911002Z else 2024-06-26T07:06:23.6911363Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:23.6911803Z fi 2024-06-26T07:06:23.6912071Z  2024-06-26T07:06:23.6912543Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T07:06:23.6913389Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T07:06:23.6914155Z  # use it as it is, but first let's extract the tag 2024-06-26T07:06:23.6914854Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T07:06:23.6915580Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:23.6916279Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:23.6916840Z else 2024-06-26T07:06:23.6917263Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T07:06:23.6917928Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:23.6918821Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:23.6919573Z fi 2024-06-26T07:06:23.6927481Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:23.6927983Z env: 2024-06-26T07:06:23.6928257Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:23.6928611Z REPO_NAME: pytorch 2024-06-26T07:06:23.6929717Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:23.6930897Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:06:23.6931405Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:23.6931915Z ##[endgroup] 2024-06-26T07:06:23.6952972Z + [[ ! -d .ci/docker ]] 2024-06-26T07:06:23.6953516Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T07:06:23.6953997Z + echo skip=false 2024-06-26T07:06:23.6956096Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T07:06:23.6958995Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:23.6960174Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T07:06:23.6965241Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:23.6966078Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:23.6968256Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:23.6994244Z ##[group]Run set +e 2024-06-26T07:06:23.6994621Z set +e 2024-06-26T07:06:23.6994922Z set -x 2024-06-26T07:06:23.6995209Z  2024-06-26T07:06:23.6995485Z login() { 2024-06-26T07:06:23.6996162Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:06:23.6996882Z } 2024-06-26T07:06:23.6997153Z  2024-06-26T07:06:23.6997433Z retry () { 2024-06-26T07:06:23.6997829Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:06:23.6998280Z } 2024-06-26T07:06:23.6998550Z  2024-06-26T07:06:23.6998869Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:06:23.6999286Z  2024-06-26T07:06:23.6999749Z # Check if image already exists, if it does then skip building it 2024-06-26T07:06:23.7000631Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T07:06:23.7001140Z  exit 0 2024-06-26T07:06:23.7001440Z fi 2024-06-26T07:06:23.7001712Z  2024-06-26T07:06:23.7002210Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T07:06:23.7003034Z # be empty. The default action would be to continue rebuild the image 2024-06-26T07:06:23.7004090Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T07:06:23.7004766Z  # if we're on the base branch then use the parent commit 2024-06-26T07:06:23.7005358Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T07:06:23.7005790Z else 2024-06-26T07:06:23.7006260Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T07:06:23.7006944Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T07:06:23.7007466Z fi 2024-06-26T07:06:23.7007747Z  2024-06-26T07:06:23.7008067Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T07:06:23.7008564Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:23.7009020Z  2024-06-26T07:06:23.7009670Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T07:06:23.7010425Z  exit 0 2024-06-26T07:06:23.7010719Z fi 2024-06-26T07:06:23.7011008Z  2024-06-26T07:06:23.7011452Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T07:06:23.7012422Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T07:06:23.7013219Z  exit 1 2024-06-26T07:06:23.7013525Z fi 2024-06-26T07:06:23.7013807Z  2024-06-26T07:06:23.7014291Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T07:06:23.7015236Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T07:06:23.7016088Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T07:06:23.7017183Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T07:06:23.7018282Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T07:06:23.7019059Z fi 2024-06-26T07:06:23.7019338Z  2024-06-26T07:06:23.7019699Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:23.7026978Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:23.7027476Z env: 2024-06-26T07:06:23.7027746Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:23.7028108Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:06:23.7028569Z BASE_REVISION: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:06:23.7029835Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:23.7031067Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:23.7031660Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:23.7032178Z ##[endgroup] 2024-06-26T07:06:23.7051588Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:23.7052381Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:23.7053140Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:06:23.7054159Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:24.1273620Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:06:24.1274558Z Configure a credential helper to remove this warning. See 2024-06-26T07:06:24.1275754Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:06:24.1276311Z 2024-06-26T07:06:24.1276466Z Login Succeeded 2024-06-26T07:06:24.1285008Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:24.3462509Z { 2024-06-26T07:06:24.3462984Z "schemaVersion": 2, 2024-06-26T07:06:24.3463768Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T07:06:24.3464562Z "config": { 2024-06-26T07:06:24.3465222Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T07:06:24.3466100Z "size": 48681, 2024-06-26T07:06:24.3467224Z "digest": "sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd" 2024-06-26T07:06:24.3468127Z }, 2024-06-26T07:06:24.3468460Z "layers": [ 2024-06-26T07:06:24.3468835Z { 2024-06-26T07:06:24.3469446Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3470236Z "size": 28580681, 2024-06-26T07:06:24.3471071Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-06-26T07:06:24.3471995Z }, 2024-06-26T07:06:24.3472344Z { 2024-06-26T07:06:24.3472994Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3473881Z "size": 7943451, 2024-06-26T07:06:24.3474652Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-06-26T07:06:24.3475484Z }, 2024-06-26T07:06:24.3475730Z { 2024-06-26T07:06:24.3476169Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3476723Z "size": 55728572, 2024-06-26T07:06:24.3477748Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-06-26T07:06:24.3478580Z }, 2024-06-26T07:06:24.3478866Z { 2024-06-26T07:06:24.3479367Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3480081Z "size": 186, 2024-06-26T07:06:24.3480793Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-06-26T07:06:24.3481622Z }, 2024-06-26T07:06:24.3481939Z { 2024-06-26T07:06:24.3482476Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3483198Z "size": 6886, 2024-06-26T07:06:24.3484225Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-06-26T07:06:24.3484836Z }, 2024-06-26T07:06:24.3485261Z { 2024-06-26T07:06:24.3485693Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3486245Z "size": 1329001756, 2024-06-26T07:06:24.3486816Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-06-26T07:06:24.3487432Z }, 2024-06-26T07:06:24.3487675Z { 2024-06-26T07:06:24.3488095Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3488635Z "size": 62501, 2024-06-26T07:06:24.3489178Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-06-26T07:06:24.3489784Z }, 2024-06-26T07:06:24.3490030Z { 2024-06-26T07:06:24.3490443Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3490986Z "size": 1684, 2024-06-26T07:06:24.3491504Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-06-26T07:06:24.3492302Z + exit 0 2024-06-26T07:06:24.3492555Z }, 2024-06-26T07:06:24.3492813Z { 2024-06-26T07:06:24.3493241Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3493792Z "size": 1523, 2024-06-26T07:06:24.3494380Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-06-26T07:06:24.3495004Z }, 2024-06-26T07:06:24.3495244Z { 2024-06-26T07:06:24.3495659Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3496201Z "size": 2528295403, 2024-06-26T07:06:24.3497038Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-06-26T07:06:24.3497645Z }, 2024-06-26T07:06:24.3497882Z { 2024-06-26T07:06:24.3498303Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3498848Z "size": 86016, 2024-06-26T07:06:24.3499397Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-06-26T07:06:24.3500003Z }, 2024-06-26T07:06:24.3500243Z { 2024-06-26T07:06:24.3500673Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3501208Z "size": 1823, 2024-06-26T07:06:24.3501733Z "digest": "sha256:309b872da805fcdf74137a887464876fa0615dc8b24199307119eb65fc2bfb49" 2024-06-26T07:06:24.3502336Z }, 2024-06-26T07:06:24.3502582Z { 2024-06-26T07:06:24.3503002Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3503551Z "size": 246737542, 2024-06-26T07:06:24.3504107Z "digest": "sha256:e82932cb002c06a782077b8e5965f8733eb8563406a23cab260d9d8c4a994f7d" 2024-06-26T07:06:24.3504712Z }, 2024-06-26T07:06:24.3504952Z { 2024-06-26T07:06:24.3505379Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3505926Z "size": 545, 2024-06-26T07:06:24.3506469Z "digest": "sha256:ca9dd73e3e9378225ebdcfd12199ef44649e34fe55003cfbc8afe2e8a4527c5b" 2024-06-26T07:06:24.3507079Z }, 2024-06-26T07:06:24.3507324Z { 2024-06-26T07:06:24.3507759Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3508295Z "size": 1283, 2024-06-26T07:06:24.3508817Z "digest": "sha256:16d76a79d5d6616e228537785ec83a869f68a402520bbc6e826201621ca6cb98" 2024-06-26T07:06:24.3509411Z }, 2024-06-26T07:06:24.3509653Z { 2024-06-26T07:06:24.3510072Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3510620Z "size": 485, 2024-06-26T07:06:24.3511172Z "digest": "sha256:4bdf41c9ecbe1651e0986ccbea4663a0556af27b7145a679cfa45c35dc96b891" 2024-06-26T07:06:24.3511798Z }, 2024-06-26T07:06:24.3512033Z { 2024-06-26T07:06:24.3512461Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3513015Z "size": 91710937, 2024-06-26T07:06:24.3513582Z "digest": "sha256:f4863801e30cdbcd1d4e6cb1cbbcde7dbcaf6ae317ca80429295741ce60ac31d" 2024-06-26T07:06:24.3514215Z }, 2024-06-26T07:06:24.3514470Z { 2024-06-26T07:06:24.3514894Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3515520Z "size": 3233, 2024-06-26T07:06:24.3516068Z "digest": "sha256:d37dd9ed8948a0d02e1198bbedadbca36477a0980dd553196385aab0c025581d" 2024-06-26T07:06:24.3516681Z }, 2024-06-26T07:06:24.3516926Z { 2024-06-26T07:06:24.3517348Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3517908Z "size": 1905, 2024-06-26T07:06:24.3518462Z "digest": "sha256:cf2f34f225dbed17c1a9a8b9ea4ad030aba797774a716dc0a82fe212324aa751" 2024-06-26T07:06:24.3519094Z }, 2024-06-26T07:06:24.3519336Z { 2024-06-26T07:06:24.3519762Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3520309Z "size": 702, 2024-06-26T07:06:24.3520837Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:06:24.3521447Z }, 2024-06-26T07:06:24.3521691Z { 2024-06-26T07:06:24.3522118Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3522661Z "size": 2784789002, 2024-06-26T07:06:24.3523229Z "digest": "sha256:3d914950f8addfc4d657a2365c76e1143eb80b710cd3f6e9ae05640217fc6347" 2024-06-26T07:06:24.3524048Z }, 2024-06-26T07:06:24.3524299Z { 2024-06-26T07:06:24.3524719Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3525267Z "size": 381, 2024-06-26T07:06:24.3525804Z "digest": "sha256:96d0e2032184c57e62a652aaffcf0241f0d0c7a23e894ca0ad192c4a64a36583" 2024-06-26T07:06:24.3526509Z }, 2024-06-26T07:06:24.3526749Z { 2024-06-26T07:06:24.3527175Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3527722Z "size": 11879, 2024-06-26T07:06:24.3528248Z "digest": "sha256:06e12ce59790cb8b091e5d619c2a62a28329008ee184a8b9a562e3db12471f30" 2024-06-26T07:06:24.3528851Z }, 2024-06-26T07:06:24.3529099Z { 2024-06-26T07:06:24.3529524Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3530070Z "size": 803, 2024-06-26T07:06:24.3530606Z "digest": "sha256:ed712f63cde7070bdec2a6179c4c35e752c6991ddbc69fe9f2724a9903250ceb" 2024-06-26T07:06:24.3531226Z }, 2024-06-26T07:06:24.3531467Z { 2024-06-26T07:06:24.3531883Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3532432Z "size": 106, 2024-06-26T07:06:24.3532959Z "digest": "sha256:a97a71ef8a4e66234dfc2098834f8395ac5630240e60164d1a251fdb0115ba85" 2024-06-26T07:06:24.3533558Z }, 2024-06-26T07:06:24.3533805Z { 2024-06-26T07:06:24.3534241Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3534788Z "size": 503, 2024-06-26T07:06:24.3535306Z "digest": "sha256:25cd0c482ad612f936b35b4f220c580e16be982226aac8801a6a849a6bfdc221" 2024-06-26T07:06:24.3535911Z }, 2024-06-26T07:06:24.3536156Z { 2024-06-26T07:06:24.3536686Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3537244Z "size": 121477177, 2024-06-26T07:06:24.3537812Z "digest": "sha256:9bf94bc456584365fda21d0ed83b7cc690b08a71727b71c4c1e667e15028a386" 2024-06-26T07:06:24.3538429Z }, 2024-06-26T07:06:24.3538675Z { 2024-06-26T07:06:24.3539106Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3539647Z "size": 109, 2024-06-26T07:06:24.3540182Z "digest": "sha256:de45a14c9b227e4b0c9d45d42223b50a71c03522ddd9e2ca0d9517835206c823" 2024-06-26T07:06:24.3540791Z }, 2024-06-26T07:06:24.3541038Z { 2024-06-26T07:06:24.3541473Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3542045Z "size": 491, 2024-06-26T07:06:24.3542569Z "digest": "sha256:e99241e77209b0b35a0fb59a61c0821fc161190178af7e4af45e6eeaa043375e" 2024-06-26T07:06:24.3543183Z }, 2024-06-26T07:06:24.3543432Z { 2024-06-26T07:06:24.3543868Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3544470Z "size": 291, 2024-06-26T07:06:24.3545035Z "digest": "sha256:35e8f4e2a42f9df0c969dedd2199cf71719cf5ecfa676dfa014bbdeb4fdb295f" 2024-06-26T07:06:24.3545741Z }, 2024-06-26T07:06:24.3545985Z { 2024-06-26T07:06:24.3546421Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3546969Z "size": 101, 2024-06-26T07:06:24.3547500Z "digest": "sha256:a823ffa4c541394b52953391dc0a6867e26c37fe4a71fa022a2cc3f07f0ba70d" 2024-06-26T07:06:24.3548109Z }, 2024-06-26T07:06:24.3548358Z { 2024-06-26T07:06:24.3548787Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3549357Z "size": 1473, 2024-06-26T07:06:24.3549879Z "digest": "sha256:44ba272eae2064603605a6238b3f35cc6578859d660068e702a047a9cd8ab2fd" 2024-06-26T07:06:24.3550482Z }, 2024-06-26T07:06:24.3550730Z { 2024-06-26T07:06:24.3551155Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3551702Z "size": 446710781, 2024-06-26T07:06:24.3552255Z "digest": "sha256:da02d278e5240c285bc855ce4e42cd6121d5b903787f5c14f03e575ad1268650" 2024-06-26T07:06:24.3552865Z }, 2024-06-26T07:06:24.3553107Z { 2024-06-26T07:06:24.3553531Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3554067Z "size": 161, 2024-06-26T07:06:24.3554583Z "digest": "sha256:73a0ce681c295135456e190578733273276b38b4dd5babbb39298f384d43b1c6" 2024-06-26T07:06:24.3555163Z }, 2024-06-26T07:06:24.3555404Z { 2024-06-26T07:06:24.3555829Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3556374Z "size": 564, 2024-06-26T07:06:24.3556965Z "digest": "sha256:49737bd5ae709dbf7956e21fc8b05ba009e9e6c9b130b0126a73bd3d3f5efc78" 2024-06-26T07:06:24.3557572Z }, 2024-06-26T07:06:24.3557814Z { 2024-06-26T07:06:24.3558232Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3558776Z "size": 35874600, 2024-06-26T07:06:24.3559334Z "digest": "sha256:9cd4c745f8e4f9d5c9bb8050923575afad21f5b7852975a4a1981bc9eff282c5" 2024-06-26T07:06:24.3559942Z }, 2024-06-26T07:06:24.3560177Z { 2024-06-26T07:06:24.3560605Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3561153Z "size": 104, 2024-06-26T07:06:24.3561674Z "digest": "sha256:b2950a0365345d4c17a48627c905fd8930b3b3154c727f0526eb9ca547102bb7" 2024-06-26T07:06:24.3562265Z }, 2024-06-26T07:06:24.3562508Z { 2024-06-26T07:06:24.3562932Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3563471Z "size": 425, 2024-06-26T07:06:24.3564175Z "digest": "sha256:25e19d81d9fc081962ff4cbc09e098f4035419822712c19a410f5282300dfb9e" 2024-06-26T07:06:24.3564772Z }, 2024-06-26T07:06:24.3565014Z { 2024-06-26T07:06:24.3565428Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3565971Z "size": 20262084, 2024-06-26T07:06:24.3566529Z "digest": "sha256:60156b40fcbec2c25f0170b0a82fa8cd9303e0daae8bb1811fd0fc6562e88e8e" 2024-06-26T07:06:24.3567150Z }, 2024-06-26T07:06:24.3567386Z { 2024-06-26T07:06:24.3567815Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3568360Z "size": 439, 2024-06-26T07:06:24.3568882Z "digest": "sha256:7a868239227fd19b30a19927bbc8f058d247403fe5b343c55f4b4b8987918911" 2024-06-26T07:06:24.3569475Z }, 2024-06-26T07:06:24.3569716Z { 2024-06-26T07:06:24.3570137Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3570678Z "size": 702, 2024-06-26T07:06:24.3571213Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:06:24.3571832Z }, 2024-06-26T07:06:24.3572075Z { 2024-06-26T07:06:24.3572491Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3573033Z "size": 143, 2024-06-26T07:06:24.3573582Z "digest": "sha256:83911d7c7ecf11b5932bb7449975c52e7f737a5b867691935e1ccb85ceb25b52" 2024-06-26T07:06:24.3574211Z }, 2024-06-26T07:06:24.3574447Z { 2024-06-26T07:06:24.3574870Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3575509Z "size": 135, 2024-06-26T07:06:24.3576053Z "digest": "sha256:b3c5d66c679f58c71febdd502889bd8cd3d36e5001754cdf1e30eabe21135cce" 2024-06-26T07:06:24.3576789Z }, 2024-06-26T07:06:24.3577031Z { 2024-06-26T07:06:24.3577452Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3577995Z "size": 4911495247, 2024-06-26T07:06:24.3578543Z "digest": "sha256:84b2c7a899820f6acdb22148c5e8346d8b26c33269fd01842c9573515251e1e1" 2024-06-26T07:06:24.3579151Z }, 2024-06-26T07:06:24.3579394Z { 2024-06-26T07:06:24.3579808Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3580352Z "size": 190, 2024-06-26T07:06:24.3580894Z "digest": "sha256:c1068d03a811d8dead5da9e82fdc8869205fc4ee9cb92d4f7a4acfbb5ba83950" 2024-06-26T07:06:24.3581522Z }, 2024-06-26T07:06:24.3581753Z { 2024-06-26T07:06:24.3582177Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3582722Z "size": 565, 2024-06-26T07:06:24.3583247Z "digest": "sha256:671b4a301d81074fca17d36851f81801552c4ecb2f90ff3412aa9ab6a1a91371" 2024-06-26T07:06:24.3583845Z }, 2024-06-26T07:06:24.3584091Z { 2024-06-26T07:06:24.3584514Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3585056Z "size": 43163095, 2024-06-26T07:06:24.3585613Z "digest": "sha256:67760cf5ec46ab58d3ca98637fd9b26cb47a67446bfd0db176fe5e023606aa0d" 2024-06-26T07:06:24.3586226Z }, 2024-06-26T07:06:24.3586542Z { 2024-06-26T07:06:24.3586962Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3587505Z "size": 106, 2024-06-26T07:06:24.3588036Z "digest": "sha256:ddc9740bcea2b301e99b50525059aea8d9c7121a1c8eab566561a8968438a8c2" 2024-06-26T07:06:24.3588637Z }, 2024-06-26T07:06:24.3588875Z { 2024-06-26T07:06:24.3589296Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3589837Z "size": 1213, 2024-06-26T07:06:24.3590357Z "digest": "sha256:fb2e977685609fcc01de170ba7f50a573e73e6520e47253717e92a28952c5ee8" 2024-06-26T07:06:24.3590957Z }, 2024-06-26T07:06:24.3591208Z { 2024-06-26T07:06:24.3591637Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3592195Z "size": 702, 2024-06-26T07:06:24.3592738Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:06:24.3593342Z }, 2024-06-26T07:06:24.3593595Z { 2024-06-26T07:06:24.3594020Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3594568Z "size": 138, 2024-06-26T07:06:24.3595115Z "digest": "sha256:b599c5a8c7cb461cbd7b3c98b99960b8eb04506e05e79db1cc0cfadb14a47e91" 2024-06-26T07:06:24.3595735Z }, 2024-06-26T07:06:24.3595976Z { 2024-06-26T07:06:24.3596402Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3596949Z "size": 120, 2024-06-26T07:06:24.3597489Z "digest": "sha256:e08e88c1366827455ee7ddbf92b6ae615ceb3cbd3fb11cd03ddd31bd3fa94a9c" 2024-06-26T07:06:24.3598106Z }, 2024-06-26T07:06:24.3598350Z { 2024-06-26T07:06:24.3598783Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3599321Z "size": 1904157495, 2024-06-26T07:06:24.3599891Z "digest": "sha256:12ba7431ff38de8cf6af3052a2a52e79f4e712b8c99f5bafde3c20e63b1f570f" 2024-06-26T07:06:24.3600504Z }, 2024-06-26T07:06:24.3600746Z { 2024-06-26T07:06:24.3601168Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3601732Z "size": 172, 2024-06-26T07:06:24.3602274Z "digest": "sha256:cf7c2cb896cba6a594028cc0fd246e127a2f4481d48905a3aa12c5f5872332d4" 2024-06-26T07:06:24.3602877Z }, 2024-06-26T07:06:24.3603123Z { 2024-06-26T07:06:24.3603552Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3604275Z "size": 908, 2024-06-26T07:06:24.3604814Z "digest": "sha256:3a778ddfac2af5b3a11f49ddf2f3f1ad1731f74ba691f434ab69966d1e2a189f" 2024-06-26T07:06:24.3605535Z }, 2024-06-26T07:06:24.3605788Z { 2024-06-26T07:06:24.3606222Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3606769Z "size": 702, 2024-06-26T07:06:24.3607312Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:06:24.3607933Z }, 2024-06-26T07:06:24.3608168Z { 2024-06-26T07:06:24.3608593Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3609155Z "size": 134, 2024-06-26T07:06:24.3609688Z "digest": "sha256:236c8f0245f4b40616a4322eaf1e1b3e178d5990db1fcaba39622ea4e79c06a1" 2024-06-26T07:06:24.3610288Z }, 2024-06-26T07:06:24.3610536Z { 2024-06-26T07:06:24.3610972Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3611533Z "size": 32, 2024-06-26T07:06:24.3612065Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:06:24.3612680Z }, 2024-06-26T07:06:24.3612941Z { 2024-06-26T07:06:24.3613371Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3613914Z "size": 156, 2024-06-26T07:06:24.3614509Z "digest": "sha256:dda1ce302b9a9df0c8f1145d2e3eaa03c96909bb45a792451f1fcb555b06738d" 2024-06-26T07:06:24.3615134Z }, 2024-06-26T07:06:24.3615373Z { 2024-06-26T07:06:24.3615802Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3616367Z "size": 1841, 2024-06-26T07:06:24.3617121Z "digest": "sha256:e797fe86caef19319e03f8dca92cf96838ab3bb3644f4447c6dfc04230118afa" 2024-06-26T07:06:24.3617722Z }, 2024-06-26T07:06:24.3617971Z { 2024-06-26T07:06:24.3618401Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3618955Z "size": 7529784, 2024-06-26T07:06:24.3619498Z "digest": "sha256:967190076fc0ab2bb2fc0d74e58edd3d662a537d702ff3f8b6e876230154a16f" 2024-06-26T07:06:24.3620117Z }, 2024-06-26T07:06:24.3620369Z { 2024-06-26T07:06:24.3620795Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3621346Z "size": 165, 2024-06-26T07:06:24.3621877Z "digest": "sha256:4fb0a83f06855534ef82b80dcf7c685303edb2bcbd446196066dc2ad2dd71281" 2024-06-26T07:06:24.3622488Z }, 2024-06-26T07:06:24.3622729Z { 2024-06-26T07:06:24.3623154Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3623709Z "size": 7944, 2024-06-26T07:06:24.3624251Z "digest": "sha256:68af2ec4e30a33f26947ece9f3a31b7d3b04c2bca5865d4b48618457832a1da2" 2024-06-26T07:06:24.3624867Z }, 2024-06-26T07:06:24.3625117Z { 2024-06-26T07:06:24.3625543Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3626093Z "size": 8070, 2024-06-26T07:06:24.3626627Z "digest": "sha256:cc91933e985a044c6df53caa15488d2285896f5e991f38ec068c253ceec53f14" 2024-06-26T07:06:24.3627235Z }, 2024-06-26T07:06:24.3627478Z { 2024-06-26T07:06:24.3627903Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3628440Z "size": 303, 2024-06-26T07:06:24.3628968Z "digest": "sha256:9f3187916e7dcad33cc572f8f0e949d2f507b887f99cfd6acefd14826b44960d" 2024-06-26T07:06:24.3629578Z }, 2024-06-26T07:06:24.3629815Z { 2024-06-26T07:06:24.3630234Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3630776Z "size": 7629071, 2024-06-26T07:06:24.3631296Z "digest": "sha256:52608399485738d10d779254946c9e1d536010dd45593c69bf1eb9f62f85dc24" 2024-06-26T07:06:24.3631878Z }, 2024-06-26T07:06:24.3632118Z { 2024-06-26T07:06:24.3632541Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3633074Z "size": 108, 2024-06-26T07:06:24.3633614Z "digest": "sha256:05a6f6dcc1c00bac6660923a22adc7d7c34416718f8c89daf9b3351d9365f694" 2024-06-26T07:06:24.3634259Z }, 2024-06-26T07:06:24.3634498Z { 2024-06-26T07:06:24.3634916Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3635515Z "size": 54145775, 2024-06-26T07:06:24.3636064Z "digest": "sha256:6bc9dc8315b09b27c976bfba9700767ba412d72fd0528b18bbe8f8e62005ec22" 2024-06-26T07:06:24.3636675Z }, 2024-06-26T07:06:24.3636910Z { 2024-06-26T07:06:24.3637329Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3637871Z "size": 474, 2024-06-26T07:06:24.3638399Z "digest": "sha256:1b0e27c35f07b3fbf0aeb548d1949bf1d60a420136da21789d94d196c891d981" 2024-06-26T07:06:24.3639012Z }, 2024-06-26T07:06:24.3639253Z { 2024-06-26T07:06:24.3639675Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3640215Z "size": 1374858747, 2024-06-26T07:06:24.3640754Z "digest": "sha256:95237e7311a1c5ce6937db7196c866632f44e8788514a5c0eb4a3f04da4017e6" 2024-06-26T07:06:24.3641351Z }, 2024-06-26T07:06:24.3641592Z { 2024-06-26T07:06:24.3642007Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3642544Z "size": 106, 2024-06-26T07:06:24.3643088Z "digest": "sha256:789eff6c51aada70bf75eb1acafed63c0d839b768638001500616ff6e53b854a" 2024-06-26T07:06:24.3643880Z }, 2024-06-26T07:06:24.3644138Z { 2024-06-26T07:06:24.3644561Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3645106Z "size": 558, 2024-06-26T07:06:24.3645646Z "digest": "sha256:de0f8a8c45c6077b3c5ba3b84d7f07de98353fd618dad9a1e2f368ffe00493c7" 2024-06-26T07:06:24.3646260Z }, 2024-06-26T07:06:24.3646612Z { 2024-06-26T07:06:24.3647037Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3647575Z "size": 46248513, 2024-06-26T07:06:24.3648116Z "digest": "sha256:2060028f74888d218bebdd10d723f66c33ee1c708935c4ffd4654d34912ac4dc" 2024-06-26T07:06:24.3648720Z }, 2024-06-26T07:06:24.3648962Z { 2024-06-26T07:06:24.3649380Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3649929Z "size": 111, 2024-06-26T07:06:24.3650467Z "digest": "sha256:e412c7f3b2ded440992955bd92512345cbece8a8894cd3592f714ebc758ad1a0" 2024-06-26T07:06:24.3651076Z }, 2024-06-26T07:06:24.3651313Z { 2024-06-26T07:06:24.3651736Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3652276Z "size": 32, 2024-06-26T07:06:24.3652803Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:06:24.3653415Z }, 2024-06-26T07:06:24.3653661Z { 2024-06-26T07:06:24.3654092Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3654627Z "size": 32, 2024-06-26T07:06:24.3655164Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:06:24.3655778Z }, 2024-06-26T07:06:24.3656017Z { 2024-06-26T07:06:24.3656439Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3657118Z "size": 32, 2024-06-26T07:06:24.3657654Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:06:24.3658260Z }, 2024-06-26T07:06:24.3658495Z { 2024-06-26T07:06:24.3658917Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:24.3659467Z "size": 32, 2024-06-26T07:06:24.3659989Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:06:24.3660592Z } 2024-06-26T07:06:24.3660838Z ] 2024-06-26T07:06:24.3661079Z } 2024-06-26T07:06:24.3778096Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:06:24.3778593Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:06:24.3779141Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-26T07:06:24.3787139Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:24.3787626Z env: 2024-06-26T07:06:24.3787901Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:24.3789043Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:24.3790213Z ##[endgroup] 2024-06-26T07:06:24.3816448Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:24.3928403Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T07:06:24.3928982Z with: 2024-06-26T07:06:24.3930036Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:24.3931366Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:24.3931872Z env: 2024-06-26T07:06:24.3932154Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:24.3932497Z ##[endgroup] 2024-06-26T07:06:24.4093395Z ##[group]Run set -x 2024-06-26T07:06:24.4093753Z set -x 2024-06-26T07:06:24.4094067Z set +e 2024-06-26T07:06:24.4094369Z  2024-06-26T07:06:24.4094650Z login() { 2024-06-26T07:06:24.4095358Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:06:24.4096147Z } 2024-06-26T07:06:24.4096436Z  2024-06-26T07:06:24.4096836Z retry () { 2024-06-26T07:06:24.4097227Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:06:24.4097672Z } 2024-06-26T07:06:24.4097950Z  2024-06-26T07:06:24.4098367Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:06:24.4098776Z  2024-06-26T07:06:24.4099049Z set -e 2024-06-26T07:06:24.4099520Z # ignore output since only exit code is used for conditional 2024-06-26T07:06:24.4100236Z # only pull docker image if it's not available locally 2024-06-26T07:06:24.4100995Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T07:06:24.4101682Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T07:06:24.4102127Z fi 2024-06-26T07:06:24.4109612Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:24.4110103Z env: 2024-06-26T07:06:24.4110382Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:24.4111518Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:24.4112807Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:24.4113328Z ##[endgroup] 2024-06-26T07:06:24.4133358Z + set +e 2024-06-26T07:06:24.4133953Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:24.4134589Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:24.4135620Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:06:24.4137187Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:24.8258391Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:06:24.8259221Z Configure a credential helper to remove this warning. See 2024-06-26T07:06:24.8260025Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:06:24.8260568Z 2024-06-26T07:06:24.8260694Z Login Succeeded 2024-06-26T07:06:24.8269820Z + set -e 2024-06-26T07:06:24.8271128Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:24.8396566Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:24.8398680Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:25.0557979Z 91382da70d5719cd7007b6b80b71d2f48398f6b7: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-06-26T07:06:25.0564376Z 7a2c55901189: Pulling fs layer 2024-06-26T07:06:25.0564946Z 224fe954d725: Pulling fs layer 2024-06-26T07:06:25.0565681Z 75722010b82e: Pulling fs layer 2024-06-26T07:06:25.0566299Z d527cbbb87e3: Pulling fs layer 2024-06-26T07:06:25.0566929Z b57676e46aee: Pulling fs layer 2024-06-26T07:06:25.0567456Z a8c1e85b5e14: Pulling fs layer 2024-06-26T07:06:25.0567993Z a41a8d1c11c8: Pulling fs layer 2024-06-26T07:06:25.0568541Z 0c1227890755: Pulling fs layer 2024-06-26T07:06:25.0569115Z d8d1234baab3: Pulling fs layer 2024-06-26T07:06:25.0569636Z 7ed32bc8e469: Pulling fs layer 2024-06-26T07:06:25.0570091Z ec1e7978c1fe: Pulling fs layer 2024-06-26T07:06:25.0570616Z 309b872da805: Pulling fs layer 2024-06-26T07:06:25.0571140Z e82932cb002c: Pulling fs layer 2024-06-26T07:06:25.0571589Z ca9dd73e3e93: Pulling fs layer 2024-06-26T07:06:25.0572050Z d527cbbb87e3: Waiting 2024-06-26T07:06:25.0572481Z b57676e46aee: Waiting 2024-06-26T07:06:25.0572816Z 16d76a79d5d6: Pulling fs layer 2024-06-26T07:06:25.0573300Z 4bdf41c9ecbe: Pulling fs layer 2024-06-26T07:06:25.0573683Z f4863801e30c: Pulling fs layer 2024-06-26T07:06:25.0574113Z d37dd9ed8948: Pulling fs layer 2024-06-26T07:06:25.0574491Z cf2f34f225db: Pulling fs layer 2024-06-26T07:06:25.0574908Z 0c1227890755: Waiting 2024-06-26T07:06:25.0575311Z df546d7b3e34: Pulling fs layer 2024-06-26T07:06:25.0575683Z 3d914950f8ad: Pulling fs layer 2024-06-26T07:06:25.0576186Z 96d0e2032184: Pulling fs layer 2024-06-26T07:06:25.0576551Z 06e12ce59790: Pulling fs layer 2024-06-26T07:06:25.0577086Z ed712f63cde7: Pulling fs layer 2024-06-26T07:06:25.0577437Z 7ed32bc8e469: Waiting 2024-06-26T07:06:25.0577771Z a97a71ef8a4e: Pulling fs layer 2024-06-26T07:06:25.0578134Z ec1e7978c1fe: Waiting 2024-06-26T07:06:25.0578466Z 25cd0c482ad6: Pulling fs layer 2024-06-26T07:06:25.0578819Z 309b872da805: Waiting 2024-06-26T07:06:25.0579139Z a41a8d1c11c8: Waiting 2024-06-26T07:06:25.0579459Z d8d1234baab3: Waiting 2024-06-26T07:06:25.0579770Z f4863801e30c: Waiting 2024-06-26T07:06:25.0580100Z 9bf94bc45658: Pulling fs layer 2024-06-26T07:06:25.0580457Z e82932cb002c: Waiting 2024-06-26T07:06:25.0580766Z 96d0e2032184: Waiting 2024-06-26T07:06:25.0581094Z de45a14c9b22: Pulling fs layer 2024-06-26T07:06:25.0581466Z e99241e77209: Pulling fs layer 2024-06-26T07:06:25.0581820Z cf2f34f225db: Waiting 2024-06-26T07:06:25.0582159Z 35e8f4e2a42f: Pulling fs layer 2024-06-26T07:06:25.0582610Z 4bdf41c9ecbe: Waiting 2024-06-26T07:06:25.0582935Z 16d76a79d5d6: Waiting 2024-06-26T07:06:25.0583265Z a823ffa4c541: Pulling fs layer 2024-06-26T07:06:25.0583619Z df546d7b3e34: Waiting 2024-06-26T07:06:25.0584005Z a97a71ef8a4e: Waiting 2024-06-26T07:06:25.0584324Z ed712f63cde7: Waiting 2024-06-26T07:06:25.0584638Z de45a14c9b22: Waiting 2024-06-26T07:06:25.0584941Z 35e8f4e2a42f: Waiting 2024-06-26T07:06:25.0585248Z e99241e77209: Waiting 2024-06-26T07:06:25.0585559Z 25cd0c482ad6: Waiting 2024-06-26T07:06:25.0585871Z ca9dd73e3e93: Waiting 2024-06-26T07:06:25.0586181Z d37dd9ed8948: Waiting 2024-06-26T07:06:25.0586490Z 3d914950f8ad: Waiting 2024-06-26T07:06:25.0586816Z 44ba272eae20: Pulling fs layer 2024-06-26T07:06:25.0587184Z da02d278e524: Pulling fs layer 2024-06-26T07:06:25.0587562Z 73a0ce681c29: Pulling fs layer 2024-06-26T07:06:25.0587935Z 49737bd5ae70: Pulling fs layer 2024-06-26T07:06:25.0588292Z 44ba272eae20: Waiting 2024-06-26T07:06:25.0588615Z 9cd4c745f8e4: Pulling fs layer 2024-06-26T07:06:25.0588991Z b2950a036534: Pulling fs layer 2024-06-26T07:06:25.0589344Z 06e12ce59790: Waiting 2024-06-26T07:06:25.0589664Z 25e19d81d9fc: Pulling fs layer 2024-06-26T07:06:25.0590020Z da02d278e524: Waiting 2024-06-26T07:06:25.0590347Z 60156b40fcbe: Pulling fs layer 2024-06-26T07:06:25.0590710Z 73a0ce681c29: Waiting 2024-06-26T07:06:25.0591030Z 7a868239227f: Pulling fs layer 2024-06-26T07:06:25.0591386Z 49737bd5ae70: Waiting 2024-06-26T07:06:25.0591696Z 9cd4c745f8e4: Waiting 2024-06-26T07:06:25.0592007Z b2950a036534: Waiting 2024-06-26T07:06:25.0592325Z 83911d7c7ecf: Pulling fs layer 2024-06-26T07:06:25.0592702Z b3c5d66c679f: Pulling fs layer 2024-06-26T07:06:25.0593075Z 84b2c7a89982: Pulling fs layer 2024-06-26T07:06:25.0593437Z c1068d03a811: Pulling fs layer 2024-06-26T07:06:25.0593900Z 671b4a301d81: Pulling fs layer 2024-06-26T07:06:25.0594279Z 67760cf5ec46: Pulling fs layer 2024-06-26T07:06:25.0594632Z 7a868239227f: Waiting 2024-06-26T07:06:25.0594954Z ddc9740bcea2: Pulling fs layer 2024-06-26T07:06:25.0595332Z fb2e97768560: Pulling fs layer 2024-06-26T07:06:25.0595688Z ddc9740bcea2: Waiting 2024-06-26T07:06:25.0596007Z 84b2c7a89982: Waiting 2024-06-26T07:06:25.0596313Z a823ffa4c541: Waiting 2024-06-26T07:06:25.0596645Z b599c5a8c7cb: Pulling fs layer 2024-06-26T07:06:25.0597018Z e08e88c13668: Pulling fs layer 2024-06-26T07:06:25.0597383Z 12ba7431ff38: Pulling fs layer 2024-06-26T07:06:25.0597756Z cf7c2cb896cb: Pulling fs layer 2024-06-26T07:06:25.0598114Z fb2e97768560: Waiting 2024-06-26T07:06:25.0598489Z 3a778ddfac2a: Pulling fs layer 2024-06-26T07:06:25.0598838Z e08e88c13668: Waiting 2024-06-26T07:06:25.0599162Z 236c8f0245f4: Pulling fs layer 2024-06-26T07:06:25.0599518Z 12ba7431ff38: Waiting 2024-06-26T07:06:25.0599828Z cf7c2cb896cb: Waiting 2024-06-26T07:06:25.0600148Z b599c5a8c7cb: Waiting 2024-06-26T07:06:25.0600473Z 4f4fb700ef54: Pulling fs layer 2024-06-26T07:06:25.0600829Z 3a778ddfac2a: Waiting 2024-06-26T07:06:25.0601153Z dda1ce302b9a: Pulling fs layer 2024-06-26T07:06:25.0601529Z e797fe86caef: Pulling fs layer 2024-06-26T07:06:25.0601993Z 967190076fc0: Pulling fs layer 2024-06-26T07:06:25.0602361Z 4fb0a83f0685: Pulling fs layer 2024-06-26T07:06:25.0602711Z 4f4fb700ef54: Waiting 2024-06-26T07:06:25.0603040Z 68af2ec4e30a: Pulling fs layer 2024-06-26T07:06:25.0603397Z dda1ce302b9a: Waiting 2024-06-26T07:06:25.0603891Z cc91933e985a: Pulling fs layer 2024-06-26T07:06:25.0604256Z e797fe86caef: Waiting 2024-06-26T07:06:25.0604586Z 9f3187916e7d: Pulling fs layer 2024-06-26T07:06:25.0604954Z 526083994857: Pulling fs layer 2024-06-26T07:06:25.0605317Z 05a6f6dcc1c0: Pulling fs layer 2024-06-26T07:06:25.0605672Z 526083994857: Waiting 2024-06-26T07:06:25.0605994Z 6bc9dc8315b0: Pulling fs layer 2024-06-26T07:06:25.0606348Z 05a6f6dcc1c0: Waiting 2024-06-26T07:06:25.0606673Z 1b0e27c35f07: Pulling fs layer 2024-06-26T07:06:25.0607039Z 95237e7311a1: Pulling fs layer 2024-06-26T07:06:25.0607408Z 789eff6c51aa: Pulling fs layer 2024-06-26T07:06:25.0607775Z de0f8a8c45c6: Pulling fs layer 2024-06-26T07:06:25.0608143Z 2060028f7488: Pulling fs layer 2024-06-26T07:06:25.0608515Z e412c7f3b2de: Pulling fs layer 2024-06-26T07:06:25.0608869Z 95237e7311a1: Waiting 2024-06-26T07:06:25.0609173Z 789eff6c51aa: Waiting 2024-06-26T07:06:25.0609488Z e412c7f3b2de: Waiting 2024-06-26T07:06:25.0609795Z 2060028f7488: Waiting 2024-06-26T07:06:25.1902999Z 224fe954d725: Verifying Checksum 2024-06-26T07:06:25.1903514Z 224fe954d725: Download complete 2024-06-26T07:06:25.3449093Z d527cbbb87e3: Verifying Checksum 2024-06-26T07:06:25.3449511Z d527cbbb87e3: Download complete 2024-06-26T07:06:25.4077421Z 7a2c55901189: Download complete 2024-06-26T07:06:25.4282859Z b57676e46aee: Verifying Checksum 2024-06-26T07:06:25.4283280Z b57676e46aee: Download complete 2024-06-26T07:06:25.5176310Z a41a8d1c11c8: Download complete 2024-06-26T07:06:25.5978101Z 0c1227890755: Verifying Checksum 2024-06-26T07:06:25.5978719Z 0c1227890755: Download complete 2024-06-26T07:06:25.6705768Z d8d1234baab3: Verifying Checksum 2024-06-26T07:06:25.6706369Z d8d1234baab3: Download complete 2024-06-26T07:06:25.6900412Z 75722010b82e: Verifying Checksum 2024-06-26T07:06:25.6900951Z 75722010b82e: Download complete 2024-06-26T07:06:25.7758808Z ec1e7978c1fe: Verifying Checksum 2024-06-26T07:06:25.7759449Z ec1e7978c1fe: Download complete 2024-06-26T07:06:25.8575373Z 309b872da805: Download complete 2024-06-26T07:06:26.4267959Z 7a2c55901189: Pull complete 2024-06-26T07:06:26.7184919Z 224fe954d725: Pull complete 2024-06-26T07:06:27.6571215Z 75722010b82e: Pull complete 2024-06-26T07:06:27.7106192Z d527cbbb87e3: Pull complete 2024-06-26T07:06:27.7418217Z b57676e46aee: Pull complete 2024-06-26T07:06:28.3884650Z e82932cb002c: Verifying Checksum 2024-06-26T07:06:28.3885171Z e82932cb002c: Download complete 2024-06-26T07:06:28.4679786Z ca9dd73e3e93: Verifying Checksum 2024-06-26T07:06:28.4680485Z ca9dd73e3e93: Download complete 2024-06-26T07:06:28.5550906Z 16d76a79d5d6: Verifying Checksum 2024-06-26T07:06:28.5551486Z 16d76a79d5d6: Download complete 2024-06-26T07:06:28.6508166Z 4bdf41c9ecbe: Download complete 2024-06-26T07:06:29.6378348Z f4863801e30c: Verifying Checksum 2024-06-26T07:06:29.6378798Z f4863801e30c: Download complete 2024-06-26T07:06:29.7233018Z d37dd9ed8948: Download complete 2024-06-26T07:06:29.7905969Z cf2f34f225db: Verifying Checksum 2024-06-26T07:06:29.7906405Z cf2f34f225db: Download complete 2024-06-26T07:06:29.8677717Z df546d7b3e34: Verifying Checksum 2024-06-26T07:06:29.8678267Z df546d7b3e34: Download complete 2024-06-26T07:06:38.7587721Z a8c1e85b5e14: Verifying Checksum 2024-06-26T07:06:38.7588338Z a8c1e85b5e14: Download complete 2024-06-26T07:06:38.8448887Z 96d0e2032184: Verifying Checksum 2024-06-26T07:06:38.8449418Z 96d0e2032184: Download complete 2024-06-26T07:06:38.9323021Z 06e12ce59790: Download complete 2024-06-26T07:06:39.0066688Z ed712f63cde7: Verifying Checksum 2024-06-26T07:06:39.0067163Z ed712f63cde7: Download complete 2024-06-26T07:06:39.0774868Z a97a71ef8a4e: Verifying Checksum 2024-06-26T07:06:39.0775383Z a97a71ef8a4e: Download complete 2024-06-26T07:06:39.2120375Z 25cd0c482ad6: Verifying Checksum 2024-06-26T07:06:39.2121043Z 25cd0c482ad6: Download complete 2024-06-26T07:06:40.5191332Z 9bf94bc45658: Verifying Checksum 2024-06-26T07:06:40.5191817Z 9bf94bc45658: Download complete 2024-06-26T07:06:40.6137920Z de45a14c9b22: Verifying Checksum 2024-06-26T07:06:40.6138557Z de45a14c9b22: Download complete 2024-06-26T07:06:40.6930106Z e99241e77209: Verifying Checksum 2024-06-26T07:06:40.6930573Z e99241e77209: Download complete 2024-06-26T07:06:40.7846328Z 35e8f4e2a42f: Download complete 2024-06-26T07:06:40.8596143Z a823ffa4c541: Verifying Checksum 2024-06-26T07:06:40.8596667Z a823ffa4c541: Download complete 2024-06-26T07:06:40.9385393Z 44ba272eae20: Verifying Checksum 2024-06-26T07:06:40.9386026Z 44ba272eae20: Download complete 2024-06-26T07:06:45.4634543Z da02d278e524: Verifying Checksum 2024-06-26T07:06:45.4635179Z da02d278e524: Download complete 2024-06-26T07:06:45.5376910Z 73a0ce681c29: Verifying Checksum 2024-06-26T07:06:45.5377808Z 73a0ce681c29: Download complete 2024-06-26T07:06:45.6152882Z 49737bd5ae70: Verifying Checksum 2024-06-26T07:06:45.6153791Z 49737bd5ae70: Download complete 2024-06-26T07:06:46.0297033Z 9cd4c745f8e4: Verifying Checksum 2024-06-26T07:06:46.0297673Z 9cd4c745f8e4: Download complete 2024-06-26T07:06:46.1038153Z b2950a036534: Verifying Checksum 2024-06-26T07:06:46.1038627Z b2950a036534: Download complete 2024-06-26T07:06:46.1846107Z 25e19d81d9fc: Verifying Checksum 2024-06-26T07:06:46.1846539Z 25e19d81d9fc: Download complete 2024-06-26T07:06:46.4752742Z 60156b40fcbe: Verifying Checksum 2024-06-26T07:06:46.4753367Z 60156b40fcbe: Download complete 2024-06-26T07:06:46.5534813Z 7a868239227f: Download complete 2024-06-26T07:06:46.6269110Z 83911d7c7ecf: Verifying Checksum 2024-06-26T07:06:46.6269763Z 83911d7c7ecf: Download complete 2024-06-26T07:06:46.6907037Z b3c5d66c679f: Verifying Checksum 2024-06-26T07:06:46.6907633Z b3c5d66c679f: Download complete 2024-06-26T07:06:51.0195740Z 7ed32bc8e469: Verifying Checksum 2024-06-26T07:06:51.0196340Z 7ed32bc8e469: Download complete 2024-06-26T07:06:51.0853952Z c1068d03a811: Verifying Checksum 2024-06-26T07:06:51.0854434Z c1068d03a811: Download complete 2024-06-26T07:06:51.1948995Z 671b4a301d81: Download complete 2024-06-26T07:06:51.6961953Z 67760cf5ec46: Download complete 2024-06-26T07:06:51.7747513Z a8c1e85b5e14: Pull complete 2024-06-26T07:06:51.7800485Z ddc9740bcea2: Download complete 2024-06-26T07:06:51.8752147Z fb2e97768560: Verifying Checksum 2024-06-26T07:06:51.8752638Z fb2e97768560: Download complete 2024-06-26T07:06:51.9426377Z b599c5a8c7cb: Verifying Checksum 2024-06-26T07:06:51.9426862Z b599c5a8c7cb: Download complete 2024-06-26T07:06:51.9443343Z a41a8d1c11c8: Pull complete 2024-06-26T07:06:52.0350545Z e08e88c13668: Download complete 2024-06-26T07:06:52.1677139Z 0c1227890755: Pull complete 2024-06-26T07:06:52.3459478Z d8d1234baab3: Pull complete 2024-06-26T07:06:57.7762271Z 3d914950f8ad: Verifying Checksum 2024-06-26T07:06:57.7762815Z 3d914950f8ad: Download complete 2024-06-26T07:06:57.8543352Z cf7c2cb896cb: Verifying Checksum 2024-06-26T07:06:57.8543833Z cf7c2cb896cb: Download complete 2024-06-26T07:06:57.9439491Z 3a778ddfac2a: Verifying Checksum 2024-06-26T07:06:57.9440047Z 3a778ddfac2a: Download complete 2024-06-26T07:06:58.0458598Z 236c8f0245f4: Verifying Checksum 2024-06-26T07:06:58.0459070Z 236c8f0245f4: Download complete 2024-06-26T07:06:58.0549334Z 4f4fb700ef54: Download complete 2024-06-26T07:06:58.1232029Z dda1ce302b9a: Verifying Checksum 2024-06-26T07:06:58.1232528Z dda1ce302b9a: Download complete 2024-06-26T07:06:58.2181085Z e797fe86caef: Download complete 2024-06-26T07:06:58.3533519Z 967190076fc0: Verifying Checksum 2024-06-26T07:06:58.3534183Z 967190076fc0: Download complete 2024-06-26T07:06:58.4211963Z 4fb0a83f0685: Verifying Checksum 2024-06-26T07:06:58.4212627Z 4fb0a83f0685: Download complete 2024-06-26T07:06:58.4919955Z 68af2ec4e30a: Verifying Checksum 2024-06-26T07:06:58.4920605Z 68af2ec4e30a: Download complete 2024-06-26T07:06:58.5761790Z cc91933e985a: Download complete 2024-06-26T07:06:58.6568901Z 9f3187916e7d: Download complete 2024-06-26T07:06:58.7953217Z 526083994857: Verifying Checksum 2024-06-26T07:06:58.7953749Z 526083994857: Download complete 2024-06-26T07:06:58.8662609Z 05a6f6dcc1c0: Download complete 2024-06-26T07:06:59.4647287Z 6bc9dc8315b0: Verifying Checksum 2024-06-26T07:06:59.4647823Z 6bc9dc8315b0: Download complete 2024-06-26T07:06:59.5402054Z 1b0e27c35f07: Verifying Checksum 2024-06-26T07:06:59.5403214Z 1b0e27c35f07: Download complete 2024-06-26T07:07:22.8562353Z 12ba7431ff38: Verifying Checksum 2024-06-26T07:07:22.8564195Z 12ba7431ff38: Download complete 2024-06-26T07:07:22.9361170Z 789eff6c51aa: Verifying Checksum 2024-06-26T07:07:22.9364450Z 789eff6c51aa: Download complete 2024-06-26T07:07:23.1266082Z de0f8a8c45c6: Verifying Checksum 2024-06-26T07:07:23.1266740Z de0f8a8c45c6: Download complete 2024-06-26T07:07:24.5465654Z 2060028f7488: Verifying Checksum 2024-06-26T07:07:24.5466132Z 2060028f7488: Download complete 2024-06-26T07:07:24.6410904Z e412c7f3b2de: Verifying Checksum 2024-06-26T07:07:24.6411718Z e412c7f3b2de: Download complete 2024-06-26T07:07:26.6945696Z 95237e7311a1: Verifying Checksum 2024-06-26T07:07:26.6946171Z 95237e7311a1: Download complete 2024-06-26T07:07:50.2780274Z 84b2c7a89982: Verifying Checksum 2024-06-26T07:07:50.2780738Z 84b2c7a89982: Download complete 2024-06-26T07:08:07.6294907Z 7ed32bc8e469: Pull complete 2024-06-26T07:08:07.8339009Z ec1e7978c1fe: Pull complete 2024-06-26T07:08:08.0673074Z 309b872da805: Pull complete 2024-06-26T07:08:15.3400818Z e82932cb002c: Pull complete 2024-06-26T07:08:15.4919935Z ca9dd73e3e93: Pull complete 2024-06-26T07:08:15.6967138Z 16d76a79d5d6: Pull complete 2024-06-26T07:08:15.9010715Z 4bdf41c9ecbe: Pull complete 2024-06-26T07:08:18.2857577Z f4863801e30c: Pull complete 2024-06-26T07:08:18.4892589Z d37dd9ed8948: Pull complete 2024-06-26T07:08:18.7026199Z cf2f34f225db: Pull complete 2024-06-26T07:08:18.9032725Z df546d7b3e34: Pull complete 2024-06-26T07:09:08.2536338Z 3d914950f8ad: Pull complete 2024-06-26T07:09:08.4263828Z 96d0e2032184: Pull complete 2024-06-26T07:09:08.6625008Z 06e12ce59790: Pull complete 2024-06-26T07:09:08.8410602Z ed712f63cde7: Pull complete 2024-06-26T07:09:08.9089027Z a97a71ef8a4e: Pull complete 2024-06-26T07:09:09.0050371Z 25cd0c482ad6: Pull complete 2024-06-26T07:09:12.4909809Z 9bf94bc45658: Pull complete 2024-06-26T07:09:12.6847445Z de45a14c9b22: Pull complete 2024-06-26T07:09:12.8070519Z e99241e77209: Pull complete 2024-06-26T07:09:12.9331943Z 35e8f4e2a42f: Pull complete 2024-06-26T07:09:13.0631356Z a823ffa4c541: Pull complete 2024-06-26T07:09:13.2844575Z 44ba272eae20: Pull complete 2024-06-26T07:09:21.6285152Z da02d278e524: Pull complete 2024-06-26T07:09:21.8298240Z 73a0ce681c29: Pull complete 2024-06-26T07:09:21.9947372Z 49737bd5ae70: Pull complete 2024-06-26T07:09:22.9278126Z 9cd4c745f8e4: Pull complete 2024-06-26T07:09:23.1420882Z b2950a036534: Pull complete 2024-06-26T07:09:23.2645355Z 25e19d81d9fc: Pull complete 2024-06-26T07:09:23.7631050Z 60156b40fcbe: Pull complete 2024-06-26T07:09:23.9574340Z 7a868239227f: Pull complete 2024-06-26T07:09:24.2933995Z 83911d7c7ecf: Pull complete 2024-06-26T07:09:24.4197686Z b3c5d66c679f: Pull complete 2024-06-26T07:10:03.1099278Z 84b2c7a89982: Pull complete 2024-06-26T07:10:03.3119149Z c1068d03a811: Pull complete 2024-06-26T07:10:03.4584254Z 671b4a301d81: Pull complete 2024-06-26T07:10:05.4302000Z 67760cf5ec46: Pull complete 2024-06-26T07:10:05.5029382Z ddc9740bcea2: Pull complete 2024-06-26T07:10:05.5540432Z fb2e97768560: Pull complete 2024-06-26T07:10:05.6613487Z b599c5a8c7cb: Pull complete 2024-06-26T07:10:05.7014081Z e08e88c13668: Pull complete 2024-06-26T07:10:39.3728993Z 12ba7431ff38: Pull complete 2024-06-26T07:10:39.5815750Z cf7c2cb896cb: Pull complete 2024-06-26T07:10:39.7945555Z 3a778ddfac2a: Pull complete 2024-06-26T07:10:40.2590127Z 236c8f0245f4: Pull complete 2024-06-26T07:10:40.4738741Z 4f4fb700ef54: Pull complete 2024-06-26T07:10:40.6769273Z dda1ce302b9a: Pull complete 2024-06-26T07:10:40.8868765Z e797fe86caef: Pull complete 2024-06-26T07:10:41.2631489Z 967190076fc0: Pull complete 2024-06-26T07:10:41.4394098Z 4fb0a83f0685: Pull complete 2024-06-26T07:10:41.6465601Z 68af2ec4e30a: Pull complete 2024-06-26T07:10:41.8309993Z cc91933e985a: Pull complete 2024-06-26T07:10:42.0452313Z 9f3187916e7d: Pull complete 2024-06-26T07:10:43.1251427Z 526083994857: Pull complete 2024-06-26T07:10:43.3460226Z 05a6f6dcc1c0: Pull complete 2024-06-26T07:10:45.4202777Z 6bc9dc8315b0: Pull complete 2024-06-26T07:10:45.6366334Z 1b0e27c35f07: Pull complete 2024-06-26T07:10:58.8764809Z 95237e7311a1: Pull complete 2024-06-26T07:10:58.9308890Z 789eff6c51aa: Pull complete 2024-06-26T07:10:58.9754525Z de0f8a8c45c6: Pull complete 2024-06-26T07:10:59.6883268Z 2060028f7488: Pull complete 2024-06-26T07:10:59.7685395Z e412c7f3b2de: Pull complete 2024-06-26T07:11:00.2422900Z Digest: sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:11:00.2578084Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:11:00.2652589Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:11:00.2705277Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:11:00.2706200Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:11:00.2714183Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:00.2714679Z env: 2024-06-26T07:11:00.2714954Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:00.2715298Z ##[endgroup] 2024-06-26T07:11:00.2866587Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-26T07:11:00.2867153Z with: 2024-06-26T07:11:00.2867432Z driver-version: 550.54.15 2024-06-26T07:11:00.2867766Z env: 2024-06-26T07:11:00.2868033Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:00.2868394Z ##[endgroup] 2024-06-26T07:11:00.3026232Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:11:00.3026826Z with: 2024-06-26T07:11:00.3027107Z timeout_minutes: 10 2024-06-26T07:11:00.3027428Z max_attempts: 3 2024-06-26T07:11:00.3058967Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-26T07:11:00.3090920Z retry_wait_seconds: 10 2024-06-26T07:11:00.3091284Z polling_interval_seconds: 1 2024-06-26T07:11:00.3091649Z warning_on_retry: true 2024-06-26T07:11:00.3091999Z continue_on_error: false 2024-06-26T07:11:00.3092332Z env: 2024-06-26T07:11:00.3092600Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:00.3092950Z DRIVER_VERSION: 550.54.15 2024-06-26T07:11:00.3093293Z ##[endgroup] 2024-06-26T07:11:00.3625457Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-26T07:11:00.3626279Z + pre_install_nvidia_driver_amzn2 2024-06-26T07:11:00.6667641Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-26T07:11:00.6668538Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:11:00.7057356Z No Match for argument: nvidia-driver-latest-dkms 2024-06-26T07:11:00.7295195Z No Packages marked for removal 2024-06-26T07:11:00.7416643Z + install_nvidia_driver_common 2024-06-26T07:11:00.7418727Z + echo 'Before installing NVIDIA driver' 2024-06-26T07:11:00.7419668Z Before installing NVIDIA driver 2024-06-26T07:11:00.7421649Z + lspci 2024-06-26T07:11:00.7499222Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:11:00.7500230Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:11:00.7501533Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:11:00.7502658Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:11:00.7503734Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:11:00.7504842Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:11:00.7505894Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:11:00.7507083Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:11:00.7507915Z + lsmod 2024-06-26T07:11:00.7514308Z Module Size Used by 2024-06-26T07:11:00.7514758Z xt_nat 16384 0 2024-06-26T07:11:00.7515302Z ib_core 266240 0 2024-06-26T07:11:00.7515807Z nvidia_modeset 1339392 0 2024-06-26T07:11:00.7516342Z veth 16384 0 2024-06-26T07:11:00.7516831Z nvidia_uvm 4599808 0 2024-06-26T07:11:00.7517527Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:11:00.7518040Z drm 421888 1 nvidia 2024-06-26T07:11:00.7518467Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:11:00.7518933Z backlight 16384 1 nvidia_modeset 2024-06-26T07:11:00.7519415Z xt_conntrack 16384 1 2024-06-26T07:11:00.7519965Z ipt_MASQUERADE 16384 1 2024-06-26T07:11:00.7520559Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:11:00.7521026Z nf_conntrack_netlink 49152 0 2024-06-26T07:11:00.7521639Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:11:00.7522364Z xfrm_user 45056 1 2024-06-26T07:11:00.7522945Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:11:00.7523376Z iptable_nat 16384 1 2024-06-26T07:11:00.7524084Z nf_conntrack_ipv4 16384 3 2024-06-26T07:11:00.7524521Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:11:00.7525007Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:11:00.7525556Z nf_nat 32768 3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat 2024-06-26T07:11:00.7526549Z nf_conntrack 155648 8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink 2024-06-26T07:11:00.7527421Z xt_addrtype 16384 2 2024-06-26T07:11:00.7527803Z iptable_filter 16384 1 2024-06-26T07:11:00.7528178Z br_netfilter 24576 0 2024-06-26T07:11:00.7528591Z bridge 172032 1 br_netfilter 2024-06-26T07:11:00.7529037Z stp 16384 1 bridge 2024-06-26T07:11:00.7534286Z llc 16384 2 bridge,stp 2024-06-26T07:11:00.7534909Z overlay 86016 0 2024-06-26T07:11:00.7535468Z sunrpc 393216 1 2024-06-26T07:11:00.7535980Z dm_mirror 28672 0 2024-06-26T07:11:00.7536693Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:11:00.7537225Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:11:00.7537741Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:11:00.7538181Z dax 69632 1 dm_mod 2024-06-26T07:11:00.7538578Z crc32_pclmul 16384 0 2024-06-26T07:11:00.7538953Z ghash_clmulni_intel 16384 0 2024-06-26T07:11:00.7539335Z pcbc 16384 0 2024-06-26T07:11:00.7539894Z aesni_intel 188416 0 2024-06-26T07:11:00.7540285Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:11:00.7540731Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:11:00.7541191Z glue_helper 16384 1 aesni_intel 2024-06-26T07:11:00.7541743Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:11:00.7542285Z mousedev 24576 0 2024-06-26T07:11:00.7542703Z evdev 20480 3 2024-06-26T07:11:00.7543071Z psmouse 32768 0 2024-06-26T07:11:00.7543425Z button 16384 0 2024-06-26T07:11:00.7543869Z ena 139264 0 2024-06-26T07:11:00.7544237Z ptp 20480 1 ena 2024-06-26T07:11:00.7544620Z pps_core 20480 1 ptp 2024-06-26T07:11:00.7545010Z crc32c_intel 24576 0 2024-06-26T07:11:00.7545378Z autofs4 49152 2 2024-06-26T07:11:00.7545730Z + modinfo nvidia 2024-06-26T07:11:00.7546390Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:11:00.7547073Z alias: char-major-195-* 2024-06-26T07:11:00.7547449Z version: 550.54.15 2024-06-26T07:11:00.7547786Z supported: external 2024-06-26T07:11:00.7548113Z license: NVIDIA 2024-06-26T07:11:00.7548494Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:11:00.7548972Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:11:00.7549419Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:11:00.7549877Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:11:00.7550367Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:11:00.7550953Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:11:00.7551422Z depends: i2c-core,drm 2024-06-26T07:11:00.7551773Z retpoline: Y 2024-06-26T07:11:00.7552070Z name: nvidia 2024-06-26T07:11:00.7552610Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:11:00.7553253Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:11:00.7553888Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:11:00.7554483Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:11:00.7554921Z parm: NVreg_RmLogonRC:int 2024-06-26T07:11:00.7555340Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:11:00.7555789Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:11:00.7556215Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:11:00.7556641Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:11:00.7557152Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:11:00.7557712Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:11:00.7558181Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:11:00.7558604Z parm: NVreg_EnableMSI:int 2024-06-26T07:11:00.7559009Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:11:00.7559458Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:11:00.7559986Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:11:00.7560552Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:11:00.7561100Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:11:00.7561697Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:11:00.7562287Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:11:00.7562890Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:11:00.7563468Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:11:00.7564245Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:11:00.7564784Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:11:00.7565319Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:11:00.7565804Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:11:00.7566266Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:11:00.7566886Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:11:00.7567344Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:11:00.7567782Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:11:00.7568277Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:11:00.7568796Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:11:00.7569264Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:11:00.7569747Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:11:00.7570240Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:11:00.7570725Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:11:00.7571220Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:11:00.7571695Z parm: NVreg_RmMsg:charp 2024-06-26T07:11:00.7572104Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:11:00.7572561Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:11:00.7573027Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:11:00.7573486Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:11:00.7573963Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:11:00.7574424Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:11:00.7574872Z parm: rm_firmware_active:charp 2024-06-26T07:11:00.7575281Z + HAS_NVIDIA_DRIVER=0 2024-06-26T07:11:00.7575651Z ++ command -v nvidia-smi 2024-06-26T07:11:00.7576027Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-26T07:11:00.7576396Z + set +e 2024-06-26T07:11:00.7577008Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-26T07:11:02.8285064Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-26T07:11:02.8285568Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:11:02.8286380Z + '[' 0 -ne 0 ']' 2024-06-26T07:11:02.8286751Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-26T07:11:02.8287126Z + HAS_NVIDIA_DRIVER=1 2024-06-26T07:11:02.8287843Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-26T07:11:02.8288565Z + set -e 2024-06-26T07:11:02.8288860Z + '[' 1 -eq 0 ']' 2024-06-26T07:11:02.8289419Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-26T07:11:02.8290106Z + post_install_nvidia_driver_common 2024-06-26T07:11:02.8290506Z + sudo modprobe nvidia 2024-06-26T07:11:02.8390932Z + echo 'After installing NVIDIA driver' 2024-06-26T07:11:02.8391369Z + lspci 2024-06-26T07:11:02.8392022Z After installing NVIDIA driver 2024-06-26T07:11:02.8470316Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:11:02.8471079Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:11:02.8472001Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:11:02.8472790Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:11:02.8473540Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:11:02.8474285Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:11:02.8475009Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:11:02.8475797Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:11:02.8476382Z + lsmod 2024-06-26T07:11:02.8482807Z Module Size Used by 2024-06-26T07:11:02.8483215Z xt_nat 16384 0 2024-06-26T07:11:02.8483580Z ib_core 266240 0 2024-06-26T07:11:02.8484591Z nvidia_modeset 1339392 0 2024-06-26T07:11:02.8485052Z veth 16384 0 2024-06-26T07:11:02.8485455Z nvidia_uvm 4599808 0 2024-06-26T07:11:02.8485899Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:11:02.8486382Z drm 421888 1 nvidia 2024-06-26T07:11:02.8486803Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:11:02.8487270Z backlight 16384 1 nvidia_modeset 2024-06-26T07:11:02.8487710Z xt_conntrack 16384 1 2024-06-26T07:11:02.8488256Z ipt_MASQUERADE 16384 1 2024-06-26T07:11:02.8488701Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:11:02.8489171Z nf_conntrack_netlink 49152 0 2024-06-26T07:11:02.8489608Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:11:02.8490075Z xfrm_user 45056 1 2024-06-26T07:11:02.8490464Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:11:02.8490873Z iptable_nat 16384 1 2024-06-26T07:11:02.8491246Z nf_conntrack_ipv4 16384 3 2024-06-26T07:11:02.8491674Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:11:02.8492154Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:11:02.8492695Z nf_nat 32768 3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat 2024-06-26T07:11:02.8493677Z nf_conntrack 155648 8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink 2024-06-26T07:11:02.8494818Z xt_addrtype 16384 2 2024-06-26T07:11:02.8495326Z iptable_filter 16384 1 2024-06-26T07:11:02.8495700Z br_netfilter 24576 0 2024-06-26T07:11:02.8496107Z bridge 172032 1 br_netfilter 2024-06-26T07:11:02.8496733Z stp 16384 1 bridge 2024-06-26T07:11:02.8497153Z llc 16384 2 bridge,stp 2024-06-26T07:11:02.8497562Z overlay 86016 0 2024-06-26T07:11:02.8497931Z sunrpc 393216 1 2024-06-26T07:11:02.8498307Z dm_mirror 28672 0 2024-06-26T07:11:02.8498696Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:11:02.8499226Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:11:02.8499951Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:11:02.8500402Z dax 69632 1 dm_mod 2024-06-26T07:11:02.8500800Z crc32_pclmul 16384 0 2024-06-26T07:11:02.8501174Z ghash_clmulni_intel 16384 0 2024-06-26T07:11:02.8501551Z pcbc 16384 0 2024-06-26T07:11:02.8501922Z aesni_intel 188416 0 2024-06-26T07:11:02.8502303Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:11:02.8502746Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:11:02.8503196Z glue_helper 16384 1 aesni_intel 2024-06-26T07:11:02.8503753Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:11:02.8504284Z mousedev 24576 0 2024-06-26T07:11:02.8504654Z evdev 20480 3 2024-06-26T07:11:02.8505026Z psmouse 32768 0 2024-06-26T07:11:02.8505394Z button 16384 0 2024-06-26T07:11:02.8505747Z ena 139264 0 2024-06-26T07:11:02.8506122Z ptp 20480 1 ena 2024-06-26T07:11:02.8506513Z pps_core 20480 1 ptp 2024-06-26T07:11:02.8506900Z crc32c_intel 24576 0 2024-06-26T07:11:02.8507270Z autofs4 49152 2 2024-06-26T07:11:02.8507626Z + modinfo nvidia 2024-06-26T07:11:02.8508296Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:11:02.8508994Z alias: char-major-195-* 2024-06-26T07:11:02.8509363Z version: 550.54.15 2024-06-26T07:11:02.8509721Z supported: external 2024-06-26T07:11:02.8510058Z license: NVIDIA 2024-06-26T07:11:02.8510425Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:11:02.8510902Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:11:02.8511368Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:11:02.8511836Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:11:02.8512322Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:11:02.8512824Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:11:02.8513303Z depends: i2c-core,drm 2024-06-26T07:11:02.8513655Z retpoline: Y 2024-06-26T07:11:02.8513955Z name: nvidia 2024-06-26T07:11:02.8514509Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:11:02.8515231Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:11:02.8515996Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:11:02.8516583Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:11:02.8517024Z parm: NVreg_RmLogonRC:int 2024-06-26T07:11:02.8517455Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:11:02.8517909Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:11:02.8518344Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:11:02.8518783Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:11:02.8519305Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:11:02.8519866Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:11:02.8520348Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:11:02.8520776Z parm: NVreg_EnableMSI:int 2024-06-26T07:11:02.8521189Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:11:02.8521636Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:11:02.8522239Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:11:02.8522824Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:11:02.8523388Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:11:02.8524261Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:11:02.8524862Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:11:02.8525480Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:11:02.8526071Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:11:02.8526555Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:11:02.8527204Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:11:02.8527763Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:11:02.8528264Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:11:02.8528721Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:11:02.8529216Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:11:02.8529686Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:11:02.8530135Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:11:02.8530632Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:11:02.8531156Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:11:02.8531632Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:11:02.8532118Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:11:02.8532605Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:11:02.8533088Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:11:02.8533589Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:11:02.8534061Z parm: NVreg_RmMsg:charp 2024-06-26T07:11:02.8534477Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:11:02.8534945Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:11:02.8535414Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:11:02.8535962Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:11:02.8536444Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:11:02.8537062Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:11:02.8537514Z parm: rm_firmware_active:charp 2024-06-26T07:11:02.8537902Z + set +e 2024-06-26T07:11:02.8538205Z + nvidia-smi 2024-06-26T07:11:04.4223722Z Wed Jun 26 07:11:04 2024 2024-06-26T07:11:04.4224624Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:11:04.4225804Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-26T07:11:04.4226636Z |-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:11:04.4227431Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-26T07:11:04.4228318Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-26T07:11:04.4229231Z | | | MIG M. | 2024-06-26T07:11:04.4229782Z |=========================================+========================+======================| 2024-06-26T07:11:04.4353463Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-06-26T07:11:04.4354378Z | 0% 31C P0 58W / 300W | 0MiB / 23028MiB | 0% Default | 2024-06-26T07:11:04.4355233Z | | | N/A | 2024-06-26T07:11:04.4356132Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:11:04.4356774Z 2024-06-26T07:11:04.4357713Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:11:04.4358353Z | Processes: | 2024-06-26T07:11:04.4359074Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-26T07:11:04.4359777Z | ID ID Usage | 2024-06-26T07:11:04.4360350Z |=========================================================================================| 2024-06-26T07:11:04.4360998Z | No running processes found | 2024-06-26T07:11:04.4361754Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:11:05.0486043Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-26T07:11:06.6107281Z NVIDIA A10G 2024-06-26T07:11:07.0453860Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:11:07.0454557Z + '[' 0 -eq 0 ']' 2024-06-26T07:11:07.0455147Z + echo 'INFO: Ignoring allowed status 0' 2024-06-26T07:11:07.0455576Z + set -e 2024-06-26T07:11:07.0455889Z INFO: Ignoring allowed status 0 2024-06-26T07:11:07.0458189Z == Installing nvidia container toolkit for amzn2 == 2024-06-26T07:11:07.0460718Z + sudo yum install -y yum-utils 2024-06-26T07:11:07.3453364Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:11:08.8654944Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-26T07:11:08.8655657Z Nothing to do 2024-06-26T07:11:09.0145580Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-06-26T07:11:09.0146566Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:11:09.0147936Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:11:09.3302532Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:11:09.3544099Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:11:09.3546400Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:11:09.3548138Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:11:09.3652460Z + sudo yum install -y nvidia-docker2 2024-06-26T07:11:09.6752036Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:11:11.1497789Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-26T07:11:11.1498452Z Nothing to do 2024-06-26T07:11:11.3016833Z + sudo systemctl restart docker 2024-06-26T07:11:47.4085128Z Command completed after 1 attempt(s). 2024-06-26T07:11:47.4142787Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:11:47.4143564Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:11:47.4144256Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T07:11:47.4144915Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:11:47.4152633Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:47.4153253Z env: 2024-06-26T07:11:47.4153531Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:47.4153986Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:47.4154461Z ##[endgroup] 2024-06-26T07:11:47.6337710Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:11:47.6488343Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-06-26T07:11:47.6588282Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-06-26T07:11:47.7499523Z Prepare all required actions 2024-06-26T07:11:47.7500007Z Getting action download info 2024-06-26T07:11:47.8618126Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T07:11:48.0323096Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T07:11:48.1444866Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T07:11:48.1445345Z with: 2024-06-26T07:11:48.1445909Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:11:48.1446365Z s3-bucket: gha-artifacts 2024-06-26T07:11:48.1446703Z env: 2024-06-26T07:11:48.1446967Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:48.1447413Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:48.1447888Z ##[endgroup] 2024-06-26T07:11:48.1479100Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:11:48.1479543Z with: 2024-06-26T07:11:48.1479900Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:11:48.1480355Z s3-bucket: gha-artifacts 2024-06-26T07:11:48.1480705Z region: us-east-1 2024-06-26T07:11:48.1481009Z env: 2024-06-26T07:11:48.1481271Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:48.1481721Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:48.1482200Z ##[endgroup] 2024-06-26T07:11:48.5626920Z (node:16106) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:11:48.5627639Z 2024-06-26T07:11:48.5627899Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:11:48.5628625Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:11:48.5629630Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:11:48.6379532Z Found 1 objects with prefix pytorch/pytorch/9673646806/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-06-26T07:11:48.6380615Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:11:56.1709329Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:11:56.1715656Z Artifact download has finished successfully 2024-06-26T07:11:56.1879809Z ##[group]Run unzip -o artifacts.zip 2024-06-26T07:11:56.1880251Z unzip -o artifacts.zip 2024-06-26T07:11:56.1888720Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:56.1889222Z env: 2024-06-26T07:11:56.1889503Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:56.1889956Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:56.1890433Z ##[endgroup] 2024-06-26T07:11:56.1918753Z Archive: artifacts.zip 2024-06-26T07:11:56.1920558Z creating: dist/ 2024-06-26T07:11:58.1675884Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:11:58.1677284Z creating: build/custom_test_artifacts/ 2024-06-26T07:11:58.1678430Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T07:11:58.1679836Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T07:11:58.1681469Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:11:58.1683239Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:11:58.1684458Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T07:11:58.1685951Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:11:58.1687255Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:11:58.1688235Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:11:58.1689594Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:11:58.1691097Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:11:58.1692364Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:11:58.1693442Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:11:58.1695032Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:11:58.1696298Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:11:58.1697573Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:11:58.1698692Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:11:58.1699834Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:11:58.1700997Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:11:58.1702031Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:11:58.1703023Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:11:58.1734188Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:11:58.1772572Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:11:58.1774031Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:11:58.1816758Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:11:58.1818709Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:11:58.1820604Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:11:58.1822654Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:11:58.1824260Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:11:58.1825898Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:11:58.1827302Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:11:58.1828686Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:11:58.1830025Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:11:58.1831314Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:11:58.1832545Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:11:58.1833890Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:11:58.1835113Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:11:58.1836298Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:11:58.1837501Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:11:58.1883550Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:11:58.1941613Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:11:58.1943260Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:11:58.1944640Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:11:58.1945811Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:11:58.1947075Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T07:11:58.1948023Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T07:11:58.1949057Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T07:11:58.1950213Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T07:11:58.1951330Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T07:11:58.1952363Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T07:11:58.1953439Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T07:11:58.1954521Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T07:11:58.1955608Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T07:11:58.1956682Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T07:11:58.1957750Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T07:11:58.1967242Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T07:11:58.2094286Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T07:11:58.2095637Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T07:11:58.2097248Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T07:11:58.2098728Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T07:11:58.2099881Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T07:11:58.2100958Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T07:11:58.2102098Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T07:11:58.2103228Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T07:11:58.2104364Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T07:11:58.2105482Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T07:11:58.2106583Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T07:11:58.2118245Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T07:11:58.2197630Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T07:11:58.2198949Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:11:58.2200158Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:11:58.2201242Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T07:11:58.2202233Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T07:11:58.2203317Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T07:11:58.2204465Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-26T07:11:58.2205320Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T07:11:58.2206100Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T07:11:58.2206914Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T07:11:58.2311613Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T07:11:58.2372291Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T07:11:58.2373215Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T07:11:58.2373908Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T07:11:58.2374714Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:11:58.2379931Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:11:58.2380992Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T07:11:58.2382084Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:11:58.2383363Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:11:58.2384561Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:11:58.2385643Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:11:58.2387017Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:11:58.2388249Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:11:58.2389518Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:11:58.2390815Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:11:58.2392430Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:11:58.2393560Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:11:58.2394677Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:11:58.2395833Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:11:58.2396967Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:11:58.2397985Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:11:58.2398985Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:11:58.2431385Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:11:58.2469256Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:11:58.2470702Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:11:58.2514420Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:11:58.2518290Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:11:58.2522080Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:11:58.2523884Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:11:58.2525942Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:11:58.2527378Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:11:58.2528781Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:11:58.2530167Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:11:58.2531539Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:11:58.2532846Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:11:58.2534071Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:11:58.2535291Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:11:58.2536633Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:11:58.2537830Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:11:58.2539039Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:11:58.2579872Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:11:58.2638515Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:11:58.2640683Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:11:58.2642398Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:11:58.2643251Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:11:58.2644345Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T07:11:58.2645281Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T07:11:58.2646328Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T07:11:58.2647514Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T07:11:58.2648654Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T07:11:58.2649718Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T07:11:58.2650823Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T07:11:58.2651943Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T07:11:58.2653258Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T07:11:58.2654360Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T07:11:58.2655455Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T07:11:58.2662092Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T07:11:58.2724009Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T07:11:58.2725184Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:11:58.2726246Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:11:58.2727206Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T07:11:58.2728094Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T07:11:58.2728979Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T07:11:58.2729863Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-26T07:11:58.2730697Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T07:11:58.2731460Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T07:11:58.2732264Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T07:11:58.2777535Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T07:11:58.2778280Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T07:11:58.2779043Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T07:11:58.2779925Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:11:58.2784961Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:11:58.2786211Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T07:11:58.2787295Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:11:58.2788348Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:11:58.2789480Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:11:58.2790651Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:11:58.2791822Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:11:58.2792879Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:11:58.2793920Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:11:58.2795137Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:11:58.2796344Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:11:58.2797663Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:11:58.2798965Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:11:58.2800178Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:11:58.2801393Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:11:58.2802619Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:11:58.2803837Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:11:58.2837176Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:11:58.2876365Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:11:58.2879583Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:11:58.2935436Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:11:58.2937215Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:11:58.2938717Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:11:58.2940333Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:11:58.2942252Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:11:58.2944130Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:11:58.2946070Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:11:58.2947873Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:11:58.2949298Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:11:58.2950638Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:11:58.2951947Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:11:58.2953220Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:11:58.2954506Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:11:58.2955751Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:11:58.2957033Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:11:58.3006657Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:11:58.3065785Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:11:58.3067510Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:11:58.3068903Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:11:58.3070147Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:11:58.3071289Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T07:11:58.3072294Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T07:11:58.3073411Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T07:11:58.3075085Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T07:11:58.3076301Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T07:11:58.3077436Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T07:11:58.3078598Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T07:11:58.3079930Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T07:11:58.3081120Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T07:11:58.3082288Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T07:11:58.3083441Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T07:11:58.3084819Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T07:11:58.3196165Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T07:11:58.3197712Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T07:11:58.3199292Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T07:11:58.3200594Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T07:11:58.3201841Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T07:11:58.3203018Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T07:11:58.3204407Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T07:11:58.3205651Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T07:11:58.3206886Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T07:11:58.3208112Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T07:11:58.3209335Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T07:11:58.3219133Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T07:11:58.3272078Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T07:11:58.3273500Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:11:58.3274612Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:11:58.3275641Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T07:11:58.3276589Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T07:11:58.3277548Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T07:11:58.3278492Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-26T07:11:58.3279397Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T07:11:58.3280237Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T07:11:58.3281099Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T07:11:58.3377649Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T07:11:58.3418860Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T07:11:58.3419484Z creating: build/lib/ 2024-06-26T07:11:58.3504293Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T07:11:58.3943598Z inflating: build/lib/libprotobuf.a 2024-06-26T07:11:58.3952868Z inflating: build/lib/libpthreadpool.a 2024-06-26T07:11:58.3960606Z inflating: build/lib/libcpuinfo.a 2024-06-26T07:11:58.3968411Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T07:11:58.3969171Z inflating: build/lib/libclog.a 2024-06-26T07:11:58.3986474Z inflating: build/lib/libnnpack.a 2024-06-26T07:11:58.3988423Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T07:11:58.4052484Z inflating: build/lib/libgtest.a 2024-06-26T07:11:58.4124648Z inflating: build/lib/libbenchmark.a 2024-06-26T07:11:58.4186789Z inflating: build/lib/libasmjit.a 2024-06-26T07:11:58.4193806Z inflating: build/lib/libittnotify.a 2024-06-26T07:11:58.4220642Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T07:11:58.4343412Z inflating: build/lib/libgloo.a 2024-06-26T07:11:58.4364377Z inflating: build/lib/libfmt.a 2024-06-26T07:11:58.4454821Z inflating: build/lib/libc10.so 2024-06-26T07:11:58.4456225Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T07:11:58.4456775Z inflating: build/lib/libfoxi_loader.a 2024-06-26T07:11:58.4457960Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T07:11:58.4476607Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T07:11:58.4493956Z inflating: build/lib/libgmock.a 2024-06-26T07:11:58.4494406Z inflating: build/lib/libgtest_main.a 2024-06-26T07:11:58.4495074Z inflating: build/lib/libbenchmark_main.a 2024-06-26T07:11:58.4981224Z inflating: build/lib/libprotoc.a 2024-06-26T07:11:58.5352170Z inflating: build/lib/libgloo_cuda.a 2024-06-26T07:11:59.5109117Z inflating: build/lib/libdnnl.a 2024-06-26T07:11:59.5661573Z inflating: build/lib/libtensorpipe.a 2024-06-26T07:11:59.5714421Z inflating: build/lib/libc10_cuda.so 2024-06-26T07:11:59.5714921Z inflating: build/lib/libgmock_main.a 2024-06-26T07:11:59.6920788Z inflating: build/lib/libfbgemm.a 2024-06-26T07:11:59.7166609Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-26T07:11:59.7666041Z inflating: build/lib/libkineto.a 2024-06-26T07:11:59.7844783Z inflating: build/lib/libXNNPACK.a 2024-06-26T07:11:59.7884986Z inflating: build/lib/libonnx_proto.a 2024-06-26T07:11:59.8551892Z inflating: build/lib/libonnx.a 2024-06-26T07:12:02.1928731Z inflating: build/lib/libtorch_cpu.so 2024-06-26T07:12:02.1932817Z inflating: build/lib/libunbox_lib.a 2024-06-26T07:12:02.1936896Z inflating: build/lib/libshm.so 2024-06-26T07:12:04.1897050Z inflating: build/lib/libtorch_cuda.so 2024-06-26T07:12:04.1897779Z inflating: build/lib/libtorch.so 2024-06-26T07:12:04.9888687Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-26T07:12:04.9890150Z inflating: build/lib/libc10d_cuda_test.so 2024-06-26T07:12:05.1794498Z inflating: build/lib/libtorch_python.so 2024-06-26T07:12:05.1860337Z inflating: build/lib/libtorchbind_test.so 2024-06-26T07:12:05.1880292Z inflating: build/lib/libjitbackend_test.so 2024-06-26T07:12:05.1904906Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T07:12:05.1938297Z inflating: build/lib/libnnapi_backend.so 2024-06-26T07:12:05.1938775Z creating: build/bin/ 2024-06-26T07:12:05.1987485Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T07:12:05.2037969Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T07:12:05.2086445Z inflating: build/bin/c10_Device_test 2024-06-26T07:12:05.2144222Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T07:12:05.2196240Z inflating: build/bin/c10_Scalar_test 2024-06-26T07:12:05.2245376Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T07:12:05.2298317Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T07:12:05.2347715Z inflating: build/bin/c10_SymInt_test 2024-06-26T07:12:05.2402453Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T07:12:05.2456646Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T07:12:05.2508028Z inflating: build/bin/c10_Bitset_test 2024-06-26T07:12:05.2577901Z inflating: build/bin/c10_cow_test 2024-06-26T07:12:05.2625507Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T07:12:05.2674011Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T07:12:05.2724393Z inflating: build/bin/c10_Half_test 2024-06-26T07:12:05.2778008Z inflating: build/bin/c10_LeftRight_test 2024-06-26T07:12:05.2831527Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T07:12:05.2880126Z inflating: build/bin/c10_Synchronized_test 2024-06-26T07:12:05.2934476Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T07:12:05.2984552Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T07:12:05.3034038Z inflating: build/bin/c10_TypeList_test 2024-06-26T07:12:05.3082152Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T07:12:05.3132118Z inflating: build/bin/c10_accumulate_test 2024-06-26T07:12:05.3185951Z inflating: build/bin/c10_bfloat16_test 2024-06-26T07:12:05.3235051Z inflating: build/bin/c10_bit_cast_test 2024-06-26T07:12:05.3290414Z inflating: build/bin/c10_complex_math_test 2024-06-26T07:12:05.3344127Z inflating: build/bin/c10_complex_test 2024-06-26T07:12:05.3395219Z inflating: build/bin/c10_exception_test 2024-06-26T07:12:05.3446058Z inflating: build/bin/c10_flags_test 2024-06-26T07:12:05.3494431Z inflating: build/bin/c10_generic_math_test 2024-06-26T07:12:05.3653387Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T07:12:05.3702767Z inflating: build/bin/c10_irange_test 2024-06-26T07:12:05.3754972Z inflating: build/bin/c10_lazy_test 2024-06-26T07:12:05.3810451Z inflating: build/bin/c10_logging_test 2024-06-26T07:12:05.3883853Z inflating: build/bin/c10_optional_test 2024-06-26T07:12:05.3944187Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T07:12:05.3996730Z inflating: build/bin/c10_registry_test 2024-06-26T07:12:05.4142380Z inflating: build/bin/c10_small_vector_test 2024-06-26T07:12:05.4192440Z inflating: build/bin/c10_ssize_test 2024-06-26T07:12:05.4243452Z inflating: build/bin/c10_string_util_test 2024-06-26T07:12:05.4300115Z inflating: build/bin/c10_string_view_test 2024-06-26T07:12:05.4349321Z inflating: build/bin/c10_tempfile_test 2024-06-26T07:12:05.4397220Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T07:12:05.4452015Z inflating: build/bin/c10_typeid_test 2024-06-26T07:12:05.4886643Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T07:12:05.5322490Z inflating: build/bin/protoc 2024-06-26T07:12:05.5373971Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-26T07:12:05.5425145Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-26T07:12:05.5477644Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-26T07:12:05.5528529Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-26T07:12:05.5579078Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T07:12:05.5631530Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T07:12:05.5679374Z inflating: build/bin/c10_cuda_CUDATest 2024-06-26T07:12:05.5730955Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-26T07:12:05.6050520Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T07:12:05.6384636Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T07:12:05.6733776Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T07:12:05.6785483Z inflating: build/bin/FileStoreTest 2024-06-26T07:12:05.6840580Z inflating: build/bin/TCPStoreTest 2024-06-26T07:12:05.6892586Z inflating: build/bin/HashStoreTest 2024-06-26T07:12:05.6905272Z inflating: build/bin/ProcessGroupMPITest 2024-06-26T07:12:05.6958206Z inflating: build/bin/test_edge_op_registration 2024-06-26T07:12:05.6961858Z inflating: build/bin/torch_shm_manager 2024-06-26T07:12:05.6964721Z inflating: build/bin/example_allreduce 2024-06-26T07:12:05.7017745Z inflating: build/bin/test_dist_autograd 2024-06-26T07:12:05.7085379Z inflating: build/bin/test_cpp_rpc 2024-06-26T07:12:05.7087620Z inflating: build/bin/parallel_benchmark 2024-06-26T07:12:05.7153196Z inflating: build/bin/test_mobile_nnc 2024-06-26T07:12:05.7161405Z inflating: build/bin/aot_model_compiler_test 2024-06-26T07:12:05.7496275Z inflating: build/bin/test_lazy 2024-06-26T07:12:05.8649203Z inflating: build/bin/test_api 2024-06-26T07:12:05.8721407Z inflating: build/bin/Dict_test 2024-06-26T07:12:05.8771565Z inflating: build/bin/Dimname_test 2024-06-26T07:12:05.8834287Z inflating: build/bin/MaybeOwned_test 2024-06-26T07:12:05.8890096Z inflating: build/bin/NamedTensor_test 2024-06-26T07:12:05.8948080Z inflating: build/bin/apply_utils_test 2024-06-26T07:12:05.9005536Z inflating: build/bin/atest 2024-06-26T07:12:05.9067440Z inflating: build/bin/basic 2024-06-26T07:12:05.9120408Z inflating: build/bin/broadcast_test 2024-06-26T07:12:05.9169789Z inflating: build/bin/cpu_allocator_test 2024-06-26T07:12:05.9227055Z inflating: build/bin/cpu_generator_test 2024-06-26T07:12:05.9278480Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T07:12:05.9371151Z inflating: build/bin/cpu_rng_test 2024-06-26T07:12:05.9419570Z inflating: build/bin/dispatch_key_set_test 2024-06-26T07:12:05.9468805Z inflating: build/bin/dlconvertor_test 2024-06-26T07:12:05.9526133Z inflating: build/bin/extension_backend_test 2024-06-26T07:12:05.9579578Z inflating: build/bin/half_test 2024-06-26T07:12:05.9672707Z inflating: build/bin/ivalue_test 2024-06-26T07:12:05.9721201Z inflating: build/bin/lazy_tensor_test 2024-06-26T07:12:05.9774344Z inflating: build/bin/math_kernel_test 2024-06-26T07:12:05.9826881Z inflating: build/bin/memory_format_test 2024-06-26T07:12:05.9878513Z inflating: build/bin/memory_overlapping_test 2024-06-26T07:12:05.9930150Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T07:12:05.9984577Z inflating: build/bin/native_test 2024-06-26T07:12:06.0033718Z inflating: build/bin/operator_name_test 2024-06-26T07:12:06.0084187Z inflating: build/bin/operators_test 2024-06-26T07:12:06.0134598Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T07:12:06.0199763Z inflating: build/bin/pow_test 2024-06-26T07:12:06.0255454Z inflating: build/bin/quantized_test 2024-06-26T07:12:06.0304329Z inflating: build/bin/reduce_ops_test 2024-06-26T07:12:06.0354096Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T07:12:06.0409273Z inflating: build/bin/scalar_tensor_test 2024-06-26T07:12:06.0466390Z inflating: build/bin/scalar_test 2024-06-26T07:12:06.0516750Z inflating: build/bin/StorageUtils_test 2024-06-26T07:12:06.0567826Z inflating: build/bin/stride_properties_test 2024-06-26T07:12:06.0644629Z inflating: build/bin/tensor_iterator_test 2024-06-26T07:12:06.0697140Z inflating: build/bin/test_parallel 2024-06-26T07:12:06.0699866Z inflating: build/bin/thread_init_test 2024-06-26T07:12:06.0754212Z inflating: build/bin/type_ptr_test 2024-06-26T07:12:06.0813015Z inflating: build/bin/type_test 2024-06-26T07:12:06.0863671Z inflating: build/bin/undefined_tensor_test 2024-06-26T07:12:06.0864999Z inflating: build/bin/verify_api_visibility 2024-06-26T07:12:06.0932847Z inflating: build/bin/legacy_vmap_test 2024-06-26T07:12:06.0983134Z inflating: build/bin/weakref_test 2024-06-26T07:12:06.1033163Z inflating: build/bin/wrapdim_test 2024-06-26T07:12:06.1084178Z inflating: build/bin/xla_tensor_test 2024-06-26T07:12:06.1142419Z inflating: build/bin/IListRef_test 2024-06-26T07:12:06.1245805Z inflating: build/bin/List_test 2024-06-26T07:12:06.1310447Z inflating: build/bin/KernelFunction_test 2024-06-26T07:12:06.1429306Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T07:12:06.1523008Z inflating: build/bin/kernel_function_test 2024-06-26T07:12:06.1647964Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T07:12:06.1748798Z inflating: build/bin/kernel_lambda_test 2024-06-26T07:12:06.1808194Z inflating: build/bin/kernel_stackbased_test 2024-06-26T07:12:06.1902236Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T07:12:06.1951269Z inflating: build/bin/CppSignature_test 2024-06-26T07:12:06.2006067Z inflating: build/bin/backend_fallback_test 2024-06-26T07:12:06.2053296Z inflating: build/bin/op_allowlist_test 2024-06-26T07:12:06.2346501Z inflating: build/bin/op_registration_test 2024-06-26T07:12:06.2408480Z inflating: build/bin/inline_container_test 2024-06-26T07:12:06.2460306Z inflating: build/bin/cuda_apply_test 2024-06-26T07:12:06.2509710Z inflating: build/bin/cuda_allocator_test 2024-06-26T07:12:06.2567314Z inflating: build/bin/cuda_atomic_ops_test 2024-06-26T07:12:06.2620573Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-26T07:12:06.2689044Z inflating: build/bin/cuda_complex_math_test 2024-06-26T07:12:06.2746292Z inflating: build/bin/cuda_complex_test 2024-06-26T07:12:06.2794651Z inflating: build/bin/cuda_device_test 2024-06-26T07:12:06.2851018Z inflating: build/bin/cuda_cub_test 2024-06-26T07:12:06.2900112Z inflating: build/bin/cuda_dlconvertor_test 2024-06-26T07:12:06.2964163Z inflating: build/bin/cuda_distributions_test 2024-06-26T07:12:06.3018699Z inflating: build/bin/cuda_generator_test 2024-06-26T07:12:06.3067047Z inflating: build/bin/cuda_half_test 2024-06-26T07:12:06.3117506Z inflating: build/bin/cuda_integer_divider_test 2024-06-26T07:12:06.3166406Z inflating: build/bin/cuda_optional_test 2024-06-26T07:12:06.3216956Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-26T07:12:06.3268147Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-26T07:12:06.3316623Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-26T07:12:06.3375894Z inflating: build/bin/cuda_stream_test 2024-06-26T07:12:06.3424053Z inflating: build/bin/cuda_cudnn_test 2024-06-26T07:12:06.3475281Z inflating: build/bin/cuda_vectorized_test 2024-06-26T07:12:06.3489566Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T07:12:06.3553277Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T07:12:06.3609860Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-26T07:12:06.3671048Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-26T07:12:06.3732338Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-26T07:12:06.4542778Z inflating: build/bin/test_tensorexpr 2024-06-26T07:12:06.5103611Z inflating: build/bin/test_jit 2024-06-26T07:12:06.5104054Z creating: .additional_ci_files/ 2024-06-26T07:12:06.5157693Z inflating: .additional_ci_files/test-times.json 2024-06-26T07:12:06.5369529Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T07:12:06.5394877Z ##[group]Run rm artifacts.zip 2024-06-26T07:12:06.5395265Z rm artifacts.zip 2024-06-26T07:12:06.5404339Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:06.5404840Z env: 2024-06-26T07:12:06.5405121Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:06.5405575Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:06.5406056Z ##[endgroup] 2024-06-26T07:12:06.6101731Z ##[group]Run df -H 2024-06-26T07:12:06.6102058Z df -H 2024-06-26T07:12:06.6109727Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:06.6110225Z env: 2024-06-26T07:12:06.6110638Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:06.6111088Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:06.6111568Z ##[endgroup] 2024-06-26T07:12:06.6137373Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T07:12:06.6138044Z devtmpfs 34G 0 34G 0% /dev 2024-06-26T07:12:06.6138923Z tmpfs 34G 0 34G 0% /dev/shm 2024-06-26T07:12:06.6139517Z tmpfs 34G 455k 34G 1% /run 2024-06-26T07:12:06.6140169Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-06-26T07:12:06.6140756Z /dev/nvme0n1p1 162G 44G 118G 28% / 2024-06-26T07:12:06.6178341Z Prepare all required actions 2024-06-26T07:12:06.6178767Z Getting action download info 2024-06-26T07:12:06.7701748Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T07:12:06.7702203Z with: 2024-06-26T07:12:06.7702461Z env: 2024-06-26T07:12:06.7702789Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:06.7703340Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:06.7703832Z ##[endgroup] 2024-06-26T07:12:06.7734803Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:12:06.7735241Z with: 2024-06-26T07:12:06.7735499Z name: td_results 2024-06-26T07:12:06.7735814Z s3-bucket: gha-artifacts 2024-06-26T07:12:06.7736160Z region: us-east-1 2024-06-26T07:12:06.7736563Z env: 2024-06-26T07:12:06.7736829Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:06.7737286Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:06.7737763Z ##[endgroup] 2024-06-26T07:12:07.1801747Z (node:16127) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:12:07.1802457Z 2024-06-26T07:12:07.1802723Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:12:07.1803449Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:12:07.1804562Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:12:07.2510465Z Found 0 objects with prefix pytorch/pytorch/9673646806/td_results/ 2024-06-26T07:12:07.2517558Z Artifact download has finished successfully 2024-06-26T07:12:07.2651047Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T07:12:07.2651562Z mkdir -p .additional_ci_files 2024-06-26T07:12:07.2652110Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T07:12:07.2660010Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:07.2660493Z env: 2024-06-26T07:12:07.2660765Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:07.2661226Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:07.2661703Z ##[endgroup] 2024-06-26T07:12:07.2695806Z mv: cannot stat 'td_results.json': No such file or directory 2024-06-26T07:12:07.2705689Z ##[error]Process completed with exit code 1. 2024-06-26T07:12:07.2750539Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T07:12:07.2751016Z .github/scripts/parse_ref.py 2024-06-26T07:12:07.2758665Z shell: /usr/bin/bash -e {0} 2024-06-26T07:12:07.2759007Z env: 2024-06-26T07:12:07.2759288Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:07.2759741Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:07.2760228Z ##[endgroup] 2024-06-26T07:12:07.2985752Z Prepare all required actions 2024-06-26T07:12:07.3022533Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T07:12:07.3022984Z with: 2024-06-26T07:12:07.3023524Z github-token: *** 2024-06-26T07:12:07.3023833Z env: 2024-06-26T07:12:07.3024100Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:07.3024594Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:07.3025069Z ##[endgroup] 2024-06-26T07:12:07.3039739Z ##[group]Run set -eux 2024-06-26T07:12:07.3040233Z set -eux 2024-06-26T07:12:07.3040815Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T07:12:07.3048713Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:07.3049325Z env: 2024-06-26T07:12:07.3049594Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:07.3050040Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:07.3050673Z GITHUB_TOKEN: *** 2024-06-26T07:12:07.3050976Z ##[endgroup] 2024-06-26T07:12:07.3070327Z + python3 .github/scripts/get_workflow_job_id.py 9673646806 i-0262ed88bcec9993d 2024-06-26T07:12:07.9744131Z setting job-id=26688917069 2024-06-26T07:12:07.9745228Z setting job-name=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:07.9940068Z Prepare all required actions 2024-06-26T07:12:07.9940519Z Getting action download info 2024-06-26T07:12:08.0981267Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T07:12:08.0981725Z with: 2024-06-26T07:12:08.0982174Z github-token: *** 2024-06-26T07:12:08.0989958Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:12:08.0998754Z job-name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:08.0999790Z env: 2024-06-26T07:12:08.1000071Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:08.1000552Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:08.1001067Z ##[endgroup] 2024-06-26T07:12:08.1040966Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:12:08.1041510Z with: 2024-06-26T07:12:08.1041776Z shell: bash 2024-06-26T07:12:08.1042065Z timeout_minutes: 10 2024-06-26T07:12:08.1042382Z max_attempts: 5 2024-06-26T07:12:08.1042684Z retry_wait_seconds: 30 2024-06-26T07:12:08.1043934Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:12:08.1045093Z polling_interval_seconds: 1 2024-06-26T07:12:08.1045458Z warning_on_retry: true 2024-06-26T07:12:08.1045794Z continue_on_error: false 2024-06-26T07:12:08.1046128Z env: 2024-06-26T07:12:08.1046394Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:08.1046834Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:08.1047458Z GITHUB_TOKEN: *** 2024-06-26T07:12:08.1047766Z ##[endgroup] 2024-06-26T07:12:08.1544962Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:12:08.3705000Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:12:08.3860633Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-06-26T07:12:08.4007468Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-06-26T07:12:08.4016171Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-06-26T07:12:08.4028536Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2) 2024-06-26T07:12:08.4036670Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-06-26T07:12:08.4057057Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-06-26T07:12:09.1539983Z Command completed after 1 attempt(s). 2024-06-26T07:12:09.1579956Z ##[group]Run set -x 2024-06-26T07:12:09.1580293Z set -x 2024-06-26T07:12:09.1580587Z  2024-06-26T07:12:09.1581131Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:12:09.1581804Z # in runner workspace 2024-06-26T07:12:09.1582330Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T07:12:09.1590027Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:09.1590519Z env: 2024-06-26T07:12:09.1590807Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:09.1591261Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:09.1591745Z ##[endgroup] 2024-06-26T07:12:09.1612826Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T07:12:09.1808290Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:12:09.1808866Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:12:09.1809345Z echo "Job name: ${JOB_NAME}" 2024-06-26T07:12:09.1809755Z  2024-06-26T07:12:09.1810305Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:12:09.1810997Z # in runner workspace 2024-06-26T07:12:09.1811583Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T07:12:09.1812232Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T07:12:09.1812706Z  --job-name "${JOB_NAME}" \ 2024-06-26T07:12:09.1819499Z  --test-matrix "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-06-26T07:12:09.1826587Z  --selected-test-configs "" \ 2024-06-26T07:12:09.1827047Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T07:12:09.1827478Z  --tag "${TAG}" \ 2024-06-26T07:12:09.1827877Z  --event-name "${EVENT_NAME}" \ 2024-06-26T07:12:09.1828319Z  --schedule "${SCHEDULE}" \ 2024-06-26T07:12:09.1828843Z  --branch "${HEAD_BRANCH}" 2024-06-26T07:12:09.1836217Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:09.1836716Z env: 2024-06-26T07:12:09.1836994Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:09.1837449Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:09.1838116Z GITHUB_TOKEN: *** 2024-06-26T07:12:09.1838974Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:09.1839906Z PR_NUMBER: 2024-06-26T07:12:09.1840218Z TAG: ciflow/inductor/129470 2024-06-26T07:12:09.1840590Z EVENT_NAME: push 2024-06-26T07:12:09.1840901Z SCHEDULE: 2024-06-26T07:12:09.1841184Z HEAD_BRANCH: 2024-06-26T07:12:09.1858125Z ##[endgroup] 2024-06-26T07:12:09.1877320Z Workflow: inductor-periodic 2024-06-26T07:12:09.1878358Z Job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:09.4084664Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T07:12:09.5352093Z ##[group]Run echo "Filtered matrix:" 2024-06-26T07:12:09.5352630Z echo "Filtered matrix:" 2024-06-26T07:12:09.5359604Z echo "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-06-26T07:12:09.5366945Z  2024-06-26T07:12:09.5367222Z echo 2024-06-26T07:12:09.5367607Z echo "Is the current job unstable? False" 2024-06-26T07:12:09.5368068Z  2024-06-26T07:12:09.5368330Z echo 2024-06-26T07:12:09.5368685Z echo "Is keep-going label set? False" 2024-06-26T07:12:09.5369126Z  2024-06-26T07:12:09.5369393Z echo 2024-06-26T07:12:09.5369706Z echo "Renabled issues? " 2024-06-26T07:12:09.5377459Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:09.5378075Z env: 2024-06-26T07:12:09.5378346Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:09.5378801Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:09.5379288Z ##[endgroup] 2024-06-26T07:12:09.5401081Z Filtered matrix: 2024-06-26T07:12:09.5411267Z {include: [{config: dynamo_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-06-26T07:12:09.5418225Z 2024-06-26T07:12:09.5418373Z Is the current job unstable? False 2024-06-26T07:12:09.5418662Z 2024-06-26T07:12:09.5418902Z Is keep-going label set? False 2024-06-26T07:12:09.5419158Z 2024-06-26T07:12:09.5419286Z Renabled issues? 2024-06-26T07:12:09.5460515Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:12:09.5461219Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:12:09.5469003Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:09.5469490Z env: 2024-06-26T07:12:09.5469766Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:09.5470215Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:09.5470692Z JOB_TIMEOUT: 240 2024-06-26T07:12:09.5470987Z ##[endgroup] 2024-06-26T07:12:09.5541051Z ##[group]Run set -x 2024-06-26T07:12:09.5541446Z set -x 2024-06-26T07:12:09.5541740Z  2024-06-26T07:12:09.5542086Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T07:12:09.5542619Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T07:12:09.5543173Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T07:12:09.5543682Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-26T07:12:09.5544088Z else 2024-06-26T07:12:09.5544431Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:12:09.5544848Z fi 2024-06-26T07:12:09.5545123Z  2024-06-26T07:12:09.5545579Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T07:12:09.5546322Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T07:12:09.5546990Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T07:12:09.5547576Z # shellcheck disable=SC2086,SC2090 2024-06-26T07:12:09.5548026Z container_name=$(docker run \ 2024-06-26T07:12:09.5548438Z  ${GPU_FLAG:-} \ 2024-06-26T07:12:09.5548810Z  -e BUILD_ENVIRONMENT \ 2024-06-26T07:12:09.5549206Z  -e PR_NUMBER \ 2024-06-26T07:12:09.5549556Z  -e GITHUB_ACTIONS \ 2024-06-26T07:12:09.5549947Z  -e GITHUB_REPOSITORY \ 2024-06-26T07:12:09.5550346Z  -e GITHUB_WORKFLOW \ 2024-06-26T07:12:09.5550728Z  -e GITHUB_JOB \ 2024-06-26T07:12:09.5551198Z  -e GITHUB_RUN_ID \ 2024-06-26T07:12:09.5551577Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T07:12:09.5551976Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T07:12:09.5552360Z  -e JOB_ID \ 2024-06-26T07:12:09.5552698Z  -e JOB_NAME \ 2024-06-26T07:12:09.5553041Z  -e BASE_SHA \ 2024-06-26T07:12:09.5553380Z  -e BRANCH \ 2024-06-26T07:12:09.5553701Z  -e SHA1 \ 2024-06-26T07:12:09.5554040Z  -e AWS_DEFAULT_REGION \ 2024-06-26T07:12:09.5554444Z  -e IN_WHEEL_TEST \ 2024-06-26T07:12:09.5554817Z  -e SHARD_NUMBER \ 2024-06-26T07:12:09.5555172Z  -e TEST_CONFIG \ 2024-06-26T07:12:09.5555543Z  -e NUM_TEST_SHARDS \ 2024-06-26T07:12:09.5555934Z  -e REENABLED_ISSUES \ 2024-06-26T07:12:09.5556340Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T07:12:09.5556751Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T07:12:09.5557147Z  -e NO_TEST_TIMEOUT \ 2024-06-26T07:12:09.5557527Z  -e NO_TD \ 2024-06-26T07:12:09.5557855Z  -e TD_DISTRIBUTED \ 2024-06-26T07:12:09.5558229Z  -e PR_LABELS \ 2024-06-26T07:12:09.5558628Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T07:12:09.5559082Z  -e SCCACHE_BUCKET \ 2024-06-26T07:12:09.5559461Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-26T07:12:09.5559862Z  -e XLA_CUDA \ 2024-06-26T07:12:09.5560273Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T07:12:09.5560781Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T07:12:09.5561294Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T07:12:09.5561802Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-26T07:12:09.5562266Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-26T07:12:09.5562681Z  -e DASHBOARD_TAG \ 2024-06-26T07:12:09.5563141Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T07:12:09.5564197Z  --security-opt seccomp=unconfined \ 2024-06-26T07:12:09.5564675Z  --cap-add=SYS_PTRACE \ 2024-06-26T07:12:09.5565067Z  --ipc=host \ 2024-06-26T07:12:09.5565412Z  --shm-size="${SHM_SIZE}" \ 2024-06-26T07:12:09.5565802Z  --tty \ 2024-06-26T07:12:09.5566113Z  --detach \ 2024-06-26T07:12:09.5566464Z  --name="${container_name}" \ 2024-06-26T07:12:09.5566868Z  --user jenkins \ 2024-06-26T07:12:09.5567441Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T07:12:09.5568013Z  -w /var/lib/jenkins/workspace \ 2024-06-26T07:12:09.5568448Z  "${DOCKER_IMAGE}" 2024-06-26T07:12:09.5568788Z ) 2024-06-26T07:12:09.5569186Z # Propagate download.pytorch.org IP to container 2024-06-26T07:12:09.5570092Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-26T07:12:09.5571055Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-26T07:12:09.5571952Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-26T07:12:09.5579373Z shell: /usr/bin/bash -e {0} 2024-06-26T07:12:09.5579723Z env: 2024-06-26T07:12:09.5579995Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:09.5580436Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:09.5581041Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:12:09.5581540Z PR_NUMBER: 2024-06-26T07:12:09.5581863Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T07:12:09.5582288Z GITHUB_WORKFLOW: inductor-periodic 2024-06-26T07:12:09.5582687Z GITHUB_JOB: test 2024-06-26T07:12:09.5582999Z GITHUB_RUN_ID: 9673646806 2024-06-26T07:12:09.5583354Z GITHUB_RUN_NUMBER: 45567 2024-06-26T07:12:09.5583700Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T07:12:09.5584030Z JOB_ID: 26688917069 2024-06-26T07:12:09.5584886Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:09.5585896Z BRANCH: 2024-06-26T07:12:09.5586222Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:09.5586731Z BASE_SHA: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:09.5587229Z TEST_CONFIG: dynamic_aot_eager_torchbench 2024-06-26T07:12:09.5587654Z SHARD_NUMBER: 2 2024-06-26T07:12:09.5587953Z NUM_TEST_SHARDS: 2 2024-06-26T07:12:09.5588283Z REENABLED_ISSUES: 2024-06-26T07:12:09.5588619Z CONTINUE_THROUGH_ERROR: False 2024-06-26T07:12:09.5588995Z VERBOSE_TEST_LOGS: False 2024-06-26T07:12:09.5589346Z NO_TEST_TIMEOUT: False 2024-06-26T07:12:09.5589681Z NO_TD: False 2024-06-26T07:12:09.5589981Z TD_DISTRIBUTED: False 2024-06-26T07:12:09.5590383Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T07:12:09.5590891Z SCCACHE_S3_KEY_PREFIX: inductor-periodic 2024-06-26T07:12:09.5591309Z SHM_SIZE: 2g 2024-06-26T07:12:09.5592376Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:12:09.5593539Z XLA_CUDA: 2024-06-26T07:12:09.5594024Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:12:09.5594634Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T07:12:09.5595069Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T07:12:09.5595466Z DASHBOARD_TAG: 2024-06-26T07:12:09.5595980Z HUGGING_FACE_HUB_TOKEN: *** 2024-06-26T07:12:09.5596341Z ##[endgroup] 2024-06-26T07:12:09.5615973Z + [[ dynamic_aot_eager_torchbench == \m\u\l\t\i\g\p\u ]] 2024-06-26T07:12:09.5616787Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-06-26T07:12:09.5617309Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:12:09.5624301Z +++ nproc --ignore=2 2024-06-26T07:12:09.5642113Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9673646806 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:12:20.8139801Z + container_name=ba5a631ec45646ea607916b9626442abdafeeff36170f13db63963abace10ca4 2024-06-26T07:12:20.8140954Z + grep download.pytorch.org /etc/hosts 2024-06-26T07:12:20.8142595Z + docker exec -i ba5a631ec45646ea607916b9626442abdafeeff36170f13db63963abace10ca4 sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-26T07:12:20.8804720Z + echo DOCKER_CONTAINER_ID=ba5a631ec45646ea607916b9626442abdafeeff36170f13db63963abace10ca4 2024-06-26T07:12:20.8807858Z ++ echo dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:12:20.8809684Z + docker exec -t ba5a631ec45646ea607916b9626442abdafeeff36170f13db63963abace10ca4 sh -c 'pip install dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-26T07:12:21.2664173Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb8c4c54) 2024-06-26T07:12:21.5794540Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T07:12:21.5798615Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T07:12:21.5800934Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T07:12:21.5803363Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T07:12:21.5806366Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T07:12:21.5809637Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T07:12:21.5835820Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.3.0) 2024-06-26T07:12:21.5894578Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.21.2) 2024-06-26T07:12:21.6334999Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T07:12:21.6497173Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T07:12:22.9789031Z Installing collected packages: torch 2024-06-26T07:12:31.7260627Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-06-26T07:12:31.7262083Z timm 0.9.7 requires torchvision, which is not installed. 2024-06-26T07:12:31.7262759Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T07:12:31.8045076Z ++ dirname .ci/pytorch/test.sh 2024-06-26T07:12:31.8047984Z + source .ci/pytorch/common.sh 2024-06-26T07:12:31.8050587Z +++ dirname .ci/pytorch/common.sh 2024-06-26T07:12:31.8058241Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T07:12:31.8059419Z +++ declare -f -t trap_add 2024-06-26T07:12:31.8065107Z ++ set -ex 2024-06-26T07:12:31.8065646Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:12:31.8066319Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T07:12:31.8067354Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-06-26T07:12:31.8069776Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-26T07:12:31.8077005Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-26T07:12:31.8077460Z + trap_add cleanup_workspace EXIT 2024-06-26T07:12:31.8077880Z + trap_add_cmd=cleanup_workspace 2024-06-26T07:12:31.8078255Z + shift 2024-06-26T07:12:31.8078528Z + for trap_add_name in "$@" 2024-06-26T07:12:31.8083564Z +++ trap -p EXIT 2024-06-26T07:12:31.8085468Z ++ eval 'extract_trap_cmd ' 2024-06-26T07:12:31.8085843Z +++ extract_trap_cmd 2024-06-26T07:12:31.8086201Z +++ printf '%s\n' '' 2024-06-26T07:12:31.8086579Z ++ printf '%s\n' cleanup_workspace 2024-06-26T07:12:31.8087356Z + trap -- ' 2024-06-26T07:12:31.8087759Z cleanup_workspace' EXIT 2024-06-26T07:12:31.8088226Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-26T07:12:32.3930810Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-26T07:12:32.3943128Z + echo 'Environment variables:' 2024-06-26T07:12:32.3943546Z Environment variables: 2024-06-26T07:12:32.3943871Z + env 2024-06-26T07:12:32.3950116Z INSTALLED_DB=yes 2024-06-26T07:12:32.3950651Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:12:32.3951481Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:12:32.3952010Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:12:32.3952745Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:12:32.3953325Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:12:32.3955672Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:12:32.3956253Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:12:32.3956829Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:12:32.3957357Z HOSTNAME=ba5a631ec456 2024-06-26T07:12:32.3958314Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_a9cdd44a-2064-490a-9966-1d297419526f 2024-06-26T07:12:32.3959125Z GITHUB_ACTION=__self 2024-06-26T07:12:32.3959471Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:12:32.3963351Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:12:32.3967640Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:12:32.3968256Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:12:32.3968597Z GITHUB_RUN_NUMBER=45567 2024-06-26T07:12:32.3968966Z TEST_CONFIG=dynamic_aot_eager_torchbench 2024-06-26T07:12:32.3969456Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:12:32.3970003Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:12:32.3970490Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:12:32.3970921Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:12:32.3971377Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:12:32.3971914Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:12:32.3972393Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:12:32.3972852Z GITHUB_REF_TYPE=tag 2024-06-26T07:12:32.3973171Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:12:32.3973626Z NCCL_VERSION=2.17.1-1 2024-06-26T07:12:32.3974287Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:32.3974731Z XLA_CUDA= 2024-06-26T07:12:32.3975227Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:12:32.3977253Z *** 2024-06-26T07:12:32.3977623Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:12:32.3978024Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:12:32.3978460Z GITHUB_ACTIONS=true 2024-06-26T07:12:32.3978815Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:32.3979304Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:12:32.3979928Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:12:32.3980402Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:32.3980905Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:12:32.3981448Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:32.3982427Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:12:32.3983266Z UCC_HOME=/usr 2024-06-26T07:12:32.3983602Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:12:32.3984000Z VERBOSE_TEST_LOGS=False 2024-06-26T07:12:32.3984339Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:12:32.3984784Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:12:32.3985267Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:12:32.3985800Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:12:32.3986173Z SHARD_NUMBER=2 2024-06-26T07:12:32.3986479Z GITHUB_REF_PROTECTED=false 2024-06-26T07:12:32.3986816Z HOME=/var/lib/jenkins 2024-06-26T07:12:32.3987173Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:12:32.3987864Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:12:32.3988334Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:12:32.3988985Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-06-26T07:12:32.3989496Z CUDA_VERSION=12.1.1 2024-06-26T07:12:32.3989942Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:12:32.3990365Z NUM_TEST_SHARDS=2 2024-06-26T07:12:32.3990697Z UCX_HOME=/usr 2024-06-26T07:12:32.3991371Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:12:32.3992464Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_a9cdd44a-2064-490a-9966-1d297419526f 2024-06-26T07:12:32.3993988Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:32.3995512Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_a9cdd44a-2064-490a-9966-1d297419526f 2024-06-26T07:12:32.3996720Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:12:32.3997373Z GITHUB_EVENT_NAME=push 2024-06-26T07:12:32.3997702Z DASHBOARD_TAG= 2024-06-26T07:12:32.3998000Z GITHUB_RUN_ID=9673646806 2024-06-26T07:12:32.3998521Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:12:32.3999052Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:12:32.4000184Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_a9cdd44a-2064-490a-9966-1d297419526f 2024-06-26T07:12:32.4001118Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:12:32.4001520Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:12:32.4001964Z PR_NUMBER= 2024-06-26T07:12:32.4002235Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:12:32.4002564Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:12:32.4002998Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:12:32.4003465Z TERM=xterm 2024-06-26T07:12:32.4004113Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:12:32.4004578Z INSTALLED_VISION=yes 2024-06-26T07:12:32.4004882Z BRANCH= 2024-06-26T07:12:32.4005175Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:12:32.4005684Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:12:32.4006104Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:12:32.4006847Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:12:32.4007656Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:12:32.4008336Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:12:32.4008845Z REENABLED_ISSUES= 2024-06-26T07:12:32.4009146Z SHLVL=1 2024-06-26T07:12:32.4009404Z MAX_JOBS=14 2024-06-26T07:12:32.4009736Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:12:32.4010091Z NVARCH=x86_64 2024-06-26T07:12:32.4010395Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:12:32.4010939Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:32.4011468Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:12:32.4012030Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:12:32.4012749Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:12:32.4013309Z GITHUB_JOB=test 2024-06-26T07:12:32.4013695Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:12:32.4014262Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:12:32.4014857Z NO_TEST_TIMEOUT=False 2024-06-26T07:12:32.4015180Z TD_DISTRIBUTED=False 2024-06-26T07:12:32.4015560Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:12:32.4016019Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:12:32.4016695Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:12:32.4017124Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:12:32.4017471Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:12:32.4017811Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:12:32.4018914Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:12:32.4020273Z GITHUB_BASE_REF= 2024-06-26T07:12:32.4020597Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:12:32.4020951Z CI=true 2024-06-26T07:12:32.4021267Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:12:32.4021812Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:12:32.4022203Z JOB_ID=26688917069 2024-06-26T07:12:32.4022505Z INSTALLED_PROTOBUF=yes 2024-06-26T07:12:32.4022827Z GITHUB_HEAD_REF= 2024-06-26T07:12:32.4023123Z GITHUB_ACTION_REF= 2024-06-26T07:12:32.4023629Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:12:32.4024173Z GITHUB_WORKFLOW=inductor-periodic 2024-06-26T07:12:32.4024574Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:12:32.4025518Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_a9cdd44a-2064-490a-9966-1d297419526f 2024-06-26T07:12:32.4026408Z NO_TD=False 2024-06-26T07:12:32.4026695Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:12:32.4027052Z _=/usr/bin/env 2024-06-26T07:12:32.4027616Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T07:12:32.4095719Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-26T07:12:32.4097142Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T07:12:32.4098126Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-26T07:12:32.4099011Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-26T07:12:32.4099604Z + BUILD_DIR=build 2024-06-26T07:12:32.4099932Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T07:12:32.4100374Z + BUILD_BIN_DIR=build/bin 2024-06-26T07:12:32.4100707Z + SHARD_NUMBER=2 2024-06-26T07:12:32.4101057Z + NUM_TEST_SHARDS=2 2024-06-26T07:12:32.4101372Z + export VALGRIND=ON 2024-06-26T07:12:32.4101688Z + VALGRIND=ON 2024-06-26T07:12:32.4102148Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-06-26T07:12:32.4102652Z + [[ 0 == \1 ]] 2024-06-26T07:12:32.4102956Z + [[ False == \1 ]] 2024-06-26T07:12:32.4103418Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-06-26T07:12:32.4103946Z ++ realpath build/custom_test_artifacts 2024-06-26T07:12:32.4107598Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-26T07:12:32.4108278Z + [[ -n '' ]] 2024-06-26T07:12:32.4108618Z + echo 'Environment variables' 2024-06-26T07:12:32.4108995Z Environment variables 2024-06-26T07:12:32.4109301Z + env 2024-06-26T07:12:32.4113143Z INSTALLED_DB=yes 2024-06-26T07:12:32.4113703Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:12:32.4114345Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:12:32.4114830Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:12:32.4115497Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:12:32.4116166Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:12:32.4116671Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:12:32.4117358Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:12:32.4118113Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:12:32.4118633Z HOSTNAME=ba5a631ec456 2024-06-26T07:12:32.4119587Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_a9cdd44a-2064-490a-9966-1d297419526f 2024-06-26T07:12:32.4120618Z GITHUB_ACTION=__self 2024-06-26T07:12:32.4120986Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:12:32.4125153Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:12:32.4129485Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:12:32.4130004Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:12:32.4130359Z GITHUB_RUN_NUMBER=45567 2024-06-26T07:12:32.4130724Z TEST_CONFIG=dynamic_aot_eager_torchbench 2024-06-26T07:12:32.4131167Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:12:32.4131630Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:12:32.4132094Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:12:32.4132532Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:12:32.4132949Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:12:32.4133385Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:12:32.4133868Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:12:32.4134324Z GITHUB_REF_TYPE=tag 2024-06-26T07:12:32.4134645Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:12:32.4135015Z NCCL_VERSION=2.17.1-1 2024-06-26T07:12:32.4135391Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:32.4135845Z XLA_CUDA= 2024-06-26T07:12:32.4136305Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:12:32.4136846Z *** 2024-06-26T07:12:32.4137116Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:12:32.4137510Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:12:32.4137923Z GITHUB_ACTIONS=true 2024-06-26T07:12:32.4138250Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:32.4138731Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:12:32.4139254Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:12:32.4139714Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:32.4140209Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:12:32.4140682Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:32.4141591Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:12:32.4142364Z UCC_HOME=/usr 2024-06-26T07:12:32.4142698Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:12:32.4143098Z VERBOSE_TEST_LOGS=False 2024-06-26T07:12:32.4143428Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:12:32.4143881Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:12:32.4144374Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:12:32.4144825Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:12:32.4145199Z SHARD_NUMBER=2 2024-06-26T07:12:32.4145512Z GITHUB_REF_PROTECTED=false 2024-06-26T07:12:32.4145858Z HOME=/var/lib/jenkins 2024-06-26T07:12:32.4146318Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:12:32.4146752Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:12:32.4147224Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:12:32.4147762Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-06-26T07:12:32.4148166Z CUDA_VERSION=12.1.1 2024-06-26T07:12:32.4148574Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:12:32.4149014Z NUM_TEST_SHARDS=2 2024-06-26T07:12:32.4149303Z UCX_HOME=/usr 2024-06-26T07:12:32.4149798Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:12:32.4150866Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_a9cdd44a-2064-490a-9966-1d297419526f 2024-06-26T07:12:32.4152299Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:32.4153710Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_a9cdd44a-2064-490a-9966-1d297419526f 2024-06-26T07:12:32.4154820Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:12:32.4155466Z GITHUB_EVENT_NAME=push 2024-06-26T07:12:32.4155792Z DASHBOARD_TAG= 2024-06-26T07:12:32.4156090Z GITHUB_RUN_ID=9673646806 2024-06-26T07:12:32.4156529Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:12:32.4157059Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:12:32.4158037Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_a9cdd44a-2064-490a-9966-1d297419526f 2024-06-26T07:12:32.4159005Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:12:32.4159400Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:12:32.4159769Z PR_NUMBER= 2024-06-26T07:12:32.4160043Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:12:32.4160351Z VALGRIND=ON 2024-06-26T07:12:32.4160640Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:12:32.4161084Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:12:32.4161543Z TERM=xterm 2024-06-26T07:12:32.4161869Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:12:32.4162281Z INSTALLED_VISION=yes 2024-06-26T07:12:32.4162592Z BRANCH= 2024-06-26T07:12:32.4162865Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:12:32.4163262Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:12:32.4163915Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:12:32.4164703Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:12:32.4165449Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:12:32.4165935Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:12:32.4166401Z REENABLED_ISSUES= 2024-06-26T07:12:32.4166694Z SHLVL=1 2024-06-26T07:12:32.4166943Z MAX_JOBS=14 2024-06-26T07:12:32.4167261Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:12:32.4167609Z NVARCH=x86_64 2024-06-26T07:12:32.4167898Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:12:32.4168377Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:32.4168909Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:12:32.4169388Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:12:32.4170026Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:12:32.4170594Z GITHUB_JOB=test 2024-06-26T07:12:32.4170984Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:12:32.4171534Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:12:32.4185810Z NO_TEST_TIMEOUT=False 2024-06-26T07:12:32.4186199Z TD_DISTRIBUTED=False 2024-06-26T07:12:32.4186612Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:12:32.4187050Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:12:32.4187475Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:12:32.4187846Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:12:32.4188189Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:12:32.4188557Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:12:32.4189745Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:12:32.4190841Z GITHUB_BASE_REF= 2024-06-26T07:12:32.4191181Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:12:32.4191676Z CI=true 2024-06-26T07:12:32.4192005Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:12:32.4192627Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:12:32.4192988Z JOB_ID=26688917069 2024-06-26T07:12:32.4193297Z INSTALLED_PROTOBUF=yes 2024-06-26T07:12:32.4193631Z GITHUB_HEAD_REF= 2024-06-26T07:12:32.4193939Z GITHUB_ACTION_REF= 2024-06-26T07:12:32.4194369Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:12:32.4194890Z GITHUB_WORKFLOW=inductor-periodic 2024-06-26T07:12:32.4195293Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:12:32.4196174Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_a9cdd44a-2064-490a-9966-1d297419526f 2024-06-26T07:12:32.4196977Z NO_TD=False 2024-06-26T07:12:32.4197282Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:12:32.4197641Z _=/usr/bin/env 2024-06-26T07:12:32.4197973Z + echo 'Testing pytorch' 2024-06-26T07:12:32.4198312Z Testing pytorch 2024-06-26T07:12:32.4198630Z + export LANG=C.UTF-8 2024-06-26T07:12:32.4198959Z + LANG=C.UTF-8 2024-06-26T07:12:32.4199234Z + PR_NUMBER= 2024-06-26T07:12:32.4199609Z + [[ dynamic_aot_eager_torchbench == \d\e\f\a\u\l\t ]] 2024-06-26T07:12:32.4200191Z + [[ dynamic_aot_eager_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T07:12:32.4200734Z + [[ dynamic_aot_eager_torchbench == \s\l\o\w ]] 2024-06-26T07:12:32.4201377Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-06-26T07:12:32.4202241Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-06-26T07:12:32.4202917Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:12:32.4203378Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:12:32.4203975Z + [[ dynamic_aot_eager_torchbench == *crossref* ]] 2024-06-26T07:12:32.4204595Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:12:32.4205212Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-06-26T07:12:32.4205841Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:12:32.4206384Z + pip_install --user ninja==1.10.2 2024-06-26T07:12:32.4206911Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T07:12:32.7823994Z Collecting ninja==1.10.2 2024-06-26T07:12:32.7980386Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T07:12:32.8121241Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T07:12:34.1190785Z Installing collected packages: ninja 2024-06-26T07:12:34.1257699Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T07:12:34.1258968Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T07:12:34.1307932Z Successfully installed ninja-1.10.2 2024-06-26T07:12:34.2138128Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:12:34.2140223Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:12:34.2141645Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:12:34.2142148Z + install_tlparse 2024-06-26T07:12:34.2142530Z + pip_install --user tlparse==0.3.7 2024-06-26T07:12:34.2143069Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T07:12:34.5669242Z Collecting tlparse==0.3.7 2024-06-26T07:12:34.5816823Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T07:12:34.5925544Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T07:12:35.9294813Z Installing collected packages: tlparse 2024-06-26T07:12:35.9663753Z Successfully installed tlparse-0.3.7 2024-06-26T07:12:36.0480989Z ++ python -m site --user-base 2024-06-26T07:12:36.0638463Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:12:36.0642001Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-06-26T07:12:36.0642748Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-06-26T07:12:36.0643408Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:12:36.0644530Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-06-26T07:12:36.0645669Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-06-26T07:12:36.0646371Z + cd test 2024-06-26T07:12:36.0646901Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T07:12:37.5401726Z + [[ dynamic_aot_eager_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T07:12:37.5402469Z + [[ dynamic_aot_eager_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T07:12:37.5403313Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:12:37.5404227Z + pushd test 2024-06-26T07:12:37.5404548Z ~/workspace/test ~/workspace 2024-06-26T07:12:37.5405079Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T07:12:39.0552973Z + CUDA_VERSION=12.1 2024-06-26T07:12:39.0553570Z + '[' 12.1 == 12.4 ']' 2024-06-26T07:12:39.0553906Z + ISCUDA124= 2024-06-26T07:12:39.0554213Z + popd 2024-06-26T07:12:39.0554507Z ~/workspace 2024-06-26T07:12:39.0557896Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T07:12:39.0558393Z + [[ dynamic_aot_eager_torchbench == *dynamo_eager* ]] 2024-06-26T07:12:39.0558944Z + [[ dynamic_aot_eager_torchbench == *aot_eager* ]] 2024-06-26T07:12:39.0559520Z + DYNAMO_BENCHMARK_FLAGS+=(--backend aot_eager) 2024-06-26T07:12:39.0560057Z + [[ dynamic_aot_eager_torchbench == *dynamic* ]] 2024-06-26T07:12:39.0560700Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2024-06-26T07:12:39.0561310Z + [[ dynamic_aot_eager_torchbench == *cpu_inductor* ]] 2024-06-26T07:12:39.0561851Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T07:12:39.0585625Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:12:39.0586378Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-06-26T07:12:39.0587355Z + cd test 2024-06-26T07:12:39.0587938Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T07:12:40.4087782Z PyTorch built with: 2024-06-26T07:12:40.4088522Z - GCC 9.4 2024-06-26T07:12:40.4089008Z - C++ Version: 201703 2024-06-26T07:12:40.4090328Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:12:40.4092028Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:12:40.4093051Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:12:40.4093827Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T07:12:40.4094512Z - NNPACK is enabled 2024-06-26T07:12:40.4095045Z - CPU capability usage: AVX2 2024-06-26T07:12:40.4095621Z - CUDA Runtime 12.1 2024-06-26T07:12:40.4096345Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-06-26T07:12:40.4097391Z - CuDNN 90.1 (built against CUDA 12.4) 2024-06-26T07:12:40.4098017Z - Magma 2.6.1 2024-06-26T07:12:40.4108645Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T07:12:40.4118307Z 2024-06-26T07:12:40.6844375Z + cd test 2024-06-26T07:12:40.6845154Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T07:12:41.8963183Z ATen/Parallel: 2024-06-26T07:12:41.8964068Z at::get_num_threads() : 8 2024-06-26T07:12:41.8964579Z at::get_num_interop_threads() : 16 2024-06-26T07:12:41.8965014Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:12:41.8965415Z omp_get_max_threads() : 8 2024-06-26T07:12:41.8966385Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:12:41.8967170Z mkl_get_max_threads() : 8 2024-06-26T07:12:41.8967793Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:12:41.8968745Z std::thread::hardware_concurrency() : 16 2024-06-26T07:12:41.8969175Z Environment variables: 2024-06-26T07:12:41.8969516Z OMP_NUM_THREADS : [not set] 2024-06-26T07:12:41.8969887Z MKL_NUM_THREADS : [not set] 2024-06-26T07:12:41.8970264Z ATen parallel backend: OpenMP 2024-06-26T07:12:41.8970516Z 2024-06-26T07:12:42.1403428Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:12:42.1404650Z + [[ dynamic_aot_eager_torchbench == *backward* ]] 2024-06-26T07:12:42.1405176Z + [[ dynamic_aot_eager_torchbench == *xla* ]] 2024-06-26T07:12:42.1405693Z + [[ dynamic_aot_eager_torchbench == *executorch* ]] 2024-06-26T07:12:42.1406275Z + [[ dynamic_aot_eager_torchbench == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T07:12:42.1406999Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:12:42.1407570Z + [[ dynamic_aot_eager_torchbench == distributed ]] 2024-06-26T07:12:42.1408143Z + [[ dynamic_aot_eager_torchbench == *inductor_distributed* ]] 2024-06-26T07:12:42.1408806Z + [[ dynamic_aot_eager_torchbench == *inductor-halide* ]] 2024-06-26T07:12:42.1409473Z + [[ dynamic_aot_eager_torchbench == *inductor-micro-benchmark* ]] 2024-06-26T07:12:42.1410083Z + [[ dynamic_aot_eager_torchbench == *huggingface* ]] 2024-06-26T07:12:42.1410598Z + [[ dynamic_aot_eager_torchbench == *timm* ]] 2024-06-26T07:12:42.1411108Z + [[ dynamic_aot_eager_torchbench == *torchbench* ]] 2024-06-26T07:12:42.1411642Z + [[ dynamic_aot_eager_torchbench == *cpu_inductor* ]] 2024-06-26T07:12:42.1412119Z + install_torchaudio cuda 2024-06-26T07:12:42.1412466Z + local commit 2024-06-26T07:12:42.1412768Z ++ get_pinned_commit audio 2024-06-26T07:12:42.1413146Z ++ cat .github/ci_commit_pins/audio.txt 2024-06-26T07:12:42.1417870Z + commit=b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:12:42.1418414Z + [[ cuda == \c\u\d\a ]] 2024-06-26T07:12:42.1418819Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-06-26T07:12:42.1419721Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:12:42.1421155Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:12:42.4552183Z Collecting git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:12:42.4556709Z Cloning https://github.com/pytorch/audio.git (to revision b829e936f7cc61b48149f5f957a451a38bf2a178) to /tmp/pip-req-build-k35b3p33 2024-06-26T07:12:42.4574604Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-k35b3p33 2024-06-26T07:12:43.1121808Z Running command git rev-parse -q --verify 'sha^b829e936f7cc61b48149f5f957a451a38bf2a178' 2024-06-26T07:12:43.1141760Z Running command git fetch -q https://github.com/pytorch/audio.git b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:12:43.4879199Z Running command git checkout -q b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:12:43.5973401Z Resolved https://github.com/pytorch/audio.git to commit b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:12:43.5974487Z Running command git submodule update --init --recursive -q 2024-06-26T07:12:45.7647027Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:12:45.7677757Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.4.0a0+b829e93) (2.5.0a0+gitb8c4c54) 2024-06-26T07:12:45.7739449Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (3.13.1) 2024-06-26T07:12:45.7745466Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (4.12.2) 2024-06-26T07:12:45.7748020Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (1.12.1) 2024-06-26T07:12:45.7751351Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (2.8.8) 2024-06-26T07:12:45.7754486Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (3.1.4) 2024-06-26T07:12:45.7757668Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (2024.2.0) 2024-06-26T07:12:45.8187784Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.4.0a0+b829e93) (2.1.5) 2024-06-26T07:12:45.8343694Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchaudio==2.4.0a0+b829e93) (1.3.0) 2024-06-26T07:12:45.8416314Z Building wheels for collected packages: torchaudio 2024-06-26T07:15:12.3154427Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-06-26T07:15:12.3183153Z [?25h Created wheel for torchaudio: filename=torchaudio-2.4.0a0+b829e93-cp310-cp310-linux_x86_64.whl size=2308825 sha256=6b2a0645fcfe454466f92ec6fdec75e79401ec2fc3cc2515f7910211809c4169 2024-06-26T07:15:12.3184794Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/30/b3/9c/882f183f00af2e582825654a92a04400924519922db53ff522 2024-06-26T07:15:12.3211346Z Successfully built torchaudio 2024-06-26T07:15:13.5461317Z Installing collected packages: torchaudio 2024-06-26T07:15:13.7420883Z Successfully installed torchaudio-2.4.0a0+b829e93 2024-06-26T07:15:13.8728126Z + install_torchtext 2024-06-26T07:15:13.8728579Z + local data_commit 2024-06-26T07:15:13.8728896Z + local text_commit 2024-06-26T07:15:13.8731790Z ++ get_pinned_commit data 2024-06-26T07:15:13.8733728Z ++ cat .github/ci_commit_pins/data.txt 2024-06-26T07:15:13.8742970Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:13.8745354Z ++ get_pinned_commit text 2024-06-26T07:15:13.8745795Z ++ cat .github/ci_commit_pins/text.txt 2024-06-26T07:15:13.8756007Z + text_commit=b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:15:13.8757536Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:13.8759615Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:14.1883173Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:14.1889127Z Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-qznvm7h6 2024-06-26T07:15:14.1907097Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-qznvm7h6 2024-06-26T07:15:14.5984209Z Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f' 2024-06-26T07:15:14.6002221Z Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:14.8193926Z Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:14.9463662Z Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:14.9464705Z Running command git submodule update --init --recursive -q 2024-06-26T07:16:26.3417717Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:16:26.3468852Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.19) 2024-06-26T07:16:26.3472445Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.32.3) 2024-06-26T07:16:26.3477811Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.5.0a0+gitb8c4c54) 2024-06-26T07:16:26.3539919Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1) 2024-06-26T07:16:26.3545318Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.12.2) 2024-06-26T07:16:26.3548663Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.12.1) 2024-06-26T07:16:26.3552020Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8) 2024-06-26T07:16:26.3555276Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4) 2024-06-26T07:16:26.3558598Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.2.0) 2024-06-26T07:16:26.3743287Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2) 2024-06-26T07:16:26.3749924Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7) 2024-06-26T07:16:26.3756365Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.6.2) 2024-06-26T07:16:26.4214343Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5) 2024-06-26T07:16:26.4380005Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0) 2024-06-26T07:16:26.4459539Z Building wheels for collected packages: torchdata 2024-06-26T07:16:28.1665548Z Building wheel for torchdata (setup.py) ... [?25l- \ | done 2024-06-26T07:16:28.1671524Z [?25h Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182956 sha256=6159437b1541f88986d70f2190a936e8c00bec787e9f838d8498542f73e689da 2024-06-26T07:16:28.1675371Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce 2024-06-26T07:16:28.1699286Z Successfully built torchdata 2024-06-26T07:16:29.3810505Z Installing collected packages: torchdata 2024-06-26T07:16:29.4652846Z Successfully installed torchdata-0.7.0a0+11bb5b8 2024-06-26T07:16:31.9719198Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:16:31.9721248Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:16:32.2855713Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:16:32.2860088Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-tvcf03sa 2024-06-26T07:16:32.2877216Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-tvcf03sa 2024-06-26T07:16:33.0570478Z Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db' 2024-06-26T07:16:33.0584655Z Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:16:33.5057248Z Running command git checkout -q b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:16:33.6408066Z Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:16:33.6409351Z Running command git submodule update --init --recursive -q 2024-06-26T07:16:40.7145629Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:16:40.7196153Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.4) 2024-06-26T07:16:40.7198982Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.32.3) 2024-06-26T07:16:40.7202239Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.5.0a0+gitb8c4c54) 2024-06-26T07:16:40.7205977Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.21.2) 2024-06-26T07:16:40.7209256Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8) 2024-06-26T07:16:40.7270263Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2) 2024-06-26T07:16:40.7275433Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7) 2024-06-26T07:16:40.7282045Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.19) 2024-06-26T07:16:40.7287372Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.6.2) 2024-06-26T07:16:40.7344095Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1) 2024-06-26T07:16:40.7350623Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.12.2) 2024-06-26T07:16:40.7352925Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.12.1) 2024-06-26T07:16:40.7356105Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8) 2024-06-26T07:16:40.7359042Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4) 2024-06-26T07:16:40.7362110Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.2.0) 2024-06-26T07:16:40.8036449Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5) 2024-06-26T07:16:40.8199189Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0) 2024-06-26T07:16:40.8281581Z Building wheels for collected packages: torchtext 2024-06-26T07:17:17.5233297Z Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-06-26T07:17:17.5260393Z [?25h Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2045299 sha256=a102f87b810d7f619cdb188dec51a1222b0577a8040dd5633c08afd25f17e783 2024-06-26T07:17:17.5262558Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf 2024-06-26T07:17:17.5286319Z Successfully built torchtext 2024-06-26T07:17:18.7319151Z Installing collected packages: torchtext 2024-06-26T07:17:18.8325015Z Successfully installed torchtext-0.16.0a0+b0ebddc 2024-06-26T07:17:18.9544052Z + install_torchvision 2024-06-26T07:17:18.9544567Z + local orig_preload 2024-06-26T07:17:18.9544963Z + local commit 2024-06-26T07:17:18.9545615Z ++ get_pinned_commit vision 2024-06-26T07:17:18.9546018Z ++ cat .github/ci_commit_pins/vision.txt 2024-06-26T07:17:18.9554906Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:17:18.9555368Z + orig_preload= 2024-06-26T07:17:18.9555801Z + '[' -n '' ']' 2024-06-26T07:17:18.9556647Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:17:18.9558094Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:17:19.2693271Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:17:19.2696698Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-jyevyks6 2024-06-26T07:17:19.2712317Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-jyevyks6 2024-06-26T07:17:20.6741135Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-06-26T07:17:20.6756429Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:17:21.8867798Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:17:22.1202142Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:17:24.4207547Z Preparing metadata (setup.py) ... [?25l- \ done 2024-06-26T07:17:24.4257861Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-06-26T07:17:24.4260893Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+gitb8c4c54) 2024-06-26T07:17:24.4265794Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-06-26T07:17:24.4469814Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-06-26T07:17:24.4475954Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-06-26T07:17:24.4478726Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1) 2024-06-26T07:17:24.4481466Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-06-26T07:17:24.4484543Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-06-26T07:17:24.4487559Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-06-26T07:17:24.4913708Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-06-26T07:17:24.5071496Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-06-26T07:17:24.5144183Z Building wheels for collected packages: torchvision 2024-06-26T07:18:37.0500154Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-06-26T07:18:37.0526160Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2112985 sha256=cc4aa24179c622191e4cffbc5b591a97f1d972a85bd3a58fce6898bc1f4064b9 2024-06-26T07:18:37.0529002Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-06-26T07:18:37.0564605Z Successfully built torchvision 2024-06-26T07:18:38.2752864Z Installing collected packages: torchvision 2024-06-26T07:18:38.6592112Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-06-26T07:18:38.7690327Z + '[' -n '' ']' 2024-06-26T07:18:38.7690706Z + id=1 2024-06-26T07:18:38.7691073Z + pip_install opencv-python==4.8.0.74 2024-06-26T07:18:38.7691645Z + pip install --progress-bar off opencv-python==4.8.0.74 2024-06-26T07:18:39.1992138Z Collecting opencv-python==4.8.0.74 2024-06-26T07:18:39.2151883Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2024-06-26T07:18:39.2331537Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.21.2) 2024-06-26T07:18:39.2404181Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2024-06-26T07:18:41.0767409Z Installing collected packages: opencv-python 2024-06-26T07:18:41.8964033Z Successfully installed opencv-python-4.8.0.74 2024-06-26T07:18:41.9963409Z + [[ dynamic_aot_eager_torchbench == *inductor_torchbench_smoketest_perf* ]] 2024-06-26T07:18:41.9964497Z + [[ dynamic_aot_eager_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]] 2024-06-26T07:18:41.9965319Z + [[ dynamic_aot_eager_torchbench == *torchbench_gcp_smoketest* ]] 2024-06-26T07:18:41.9965951Z + checkout_install_torchbench 2024-06-26T07:18:41.9966310Z + local commit 2024-06-26T07:18:41.9966702Z ++ get_pinned_commit torchbench 2024-06-26T07:18:41.9967139Z ++ cat .github/ci_commit_pins/torchbench.txt 2024-06-26T07:18:41.9976799Z + commit=23512dbebd44a11eb84afbf53c3c071dd105297e 2024-06-26T07:18:41.9977495Z + git clone https://github.com/pytorch/benchmark torchbench 2024-06-26T07:18:41.9987293Z Cloning into 'torchbench'... 2024-06-26T07:18:42.1287488Z remote: Enumerating objects: 30187, done. 2024-06-26T07:18:42.1288735Z remote: Counting objects: 0% (1/546) 2024-06-26T07:18:42.1289940Z remote: Counting objects: 1% (6/546) 2024-06-26T07:18:42.1290879Z remote: Counting objects: 2% (11/546) 2024-06-26T07:18:42.1291443Z remote: Counting objects: 3% (17/546) 2024-06-26T07:18:42.1292230Z remote: Counting objects: 4% (22/546) 2024-06-26T07:18:42.1292869Z remote: Counting objects: 5% (28/546) 2024-06-26T07:18:42.1293614Z remote: Counting objects: 6% (33/546) 2024-06-26T07:18:42.1294179Z remote: Counting objects: 7% (39/546) 2024-06-26T07:18:42.1294676Z remote: Counting objects: 8% (44/546) 2024-06-26T07:18:42.1295160Z remote: Counting objects: 9% (50/546) 2024-06-26T07:18:42.1295659Z remote: Counting objects: 10% (55/546) 2024-06-26T07:18:42.1296304Z remote: Counting objects: 11% (61/546) 2024-06-26T07:18:42.1297013Z remote: Counting objects: 12% (66/546) 2024-06-26T07:18:42.1297497Z remote: Counting objects: 13% (71/546) 2024-06-26T07:18:42.1297994Z remote: Counting objects: 14% (77/546) 2024-06-26T07:18:42.1298676Z remote: Counting objects: 15% (82/546) 2024-06-26T07:18:42.1299361Z remote: Counting objects: 16% (88/546) 2024-06-26T07:18:42.1299948Z remote: Counting objects: 17% (93/546) 2024-06-26T07:18:42.1301030Z remote: Counting objects: 18% (99/546) 2024-06-26T07:18:42.1301714Z remote: Counting objects: 19% (104/546) 2024-06-26T07:18:42.1302391Z remote: Counting objects: 20% (110/546) 2024-06-26T07:18:42.1302896Z remote: Counting objects: 21% (115/546) 2024-06-26T07:18:42.1303396Z remote: Counting objects: 22% (121/546) 2024-06-26T07:18:42.1303902Z remote: Counting objects: 23% (126/546) 2024-06-26T07:18:42.1304404Z remote: Counting objects: 24% (132/546) 2024-06-26T07:18:42.1304903Z remote: Counting objects: 25% (137/546) 2024-06-26T07:18:42.1305515Z remote: Counting objects: 26% (142/546) 2024-06-26T07:18:42.1306143Z remote: Counting objects: 27% (148/546) 2024-06-26T07:18:42.1306784Z remote: Counting objects: 28% (153/546) 2024-06-26T07:18:42.1307282Z remote: Counting objects: 29% (159/546) 2024-06-26T07:18:42.1307779Z remote: Counting objects: 30% (164/546) 2024-06-26T07:18:42.1308282Z remote: Counting objects: 31% (170/546) 2024-06-26T07:18:42.1308796Z remote: Counting objects: 32% (175/546) 2024-06-26T07:18:42.1309337Z remote: Counting objects: 33% (181/546) 2024-06-26T07:18:42.1309834Z remote: Counting objects: 34% (186/546) 2024-06-26T07:18:42.1310331Z remote: Counting objects: 35% (192/546) 2024-06-26T07:18:42.1310915Z remote: Counting objects: 36% (197/546) 2024-06-26T07:18:42.1311410Z remote: Counting objects: 37% (203/546) 2024-06-26T07:18:42.1311966Z remote: Counting objects: 38% (208/546) 2024-06-26T07:18:42.1312469Z remote: Counting objects: 39% (213/546) 2024-06-26T07:18:42.1312968Z remote: Counting objects: 40% (219/546) 2024-06-26T07:18:42.1313460Z remote: Counting objects: 41% (224/546) 2024-06-26T07:18:42.1313953Z remote: Counting objects: 42% (230/546) 2024-06-26T07:18:42.1314451Z remote: Counting objects: 43% (235/546) 2024-06-26T07:18:42.1314944Z remote: Counting objects: 44% (241/546) 2024-06-26T07:18:42.1315432Z remote: Counting objects: 45% (246/546) 2024-06-26T07:18:42.1315929Z remote: Counting objects: 46% (252/546) 2024-06-26T07:18:42.1316424Z remote: Counting objects: 47% (257/546) 2024-06-26T07:18:42.1316919Z remote: Counting objects: 48% (263/546) 2024-06-26T07:18:42.1317409Z remote: Counting objects: 49% (268/546) 2024-06-26T07:18:42.1317897Z remote: Counting objects: 50% (273/546) 2024-06-26T07:18:42.1318392Z remote: Counting objects: 51% (279/546) 2024-06-26T07:18:42.1318887Z remote: Counting objects: 52% (284/546) 2024-06-26T07:18:42.1319553Z remote: Counting objects: 53% (290/546) 2024-06-26T07:18:42.1320064Z remote: Counting objects: 54% (295/546) 2024-06-26T07:18:42.1320556Z remote: Counting objects: 55% (301/546) 2024-06-26T07:18:42.1321051Z remote: Counting objects: 56% (306/546) 2024-06-26T07:18:42.1321538Z remote: Counting objects: 57% (312/546) 2024-06-26T07:18:42.1322036Z remote: Counting objects: 58% (317/546) 2024-06-26T07:18:42.1322534Z remote: Counting objects: 59% (323/546) 2024-06-26T07:18:42.1323031Z remote: Counting objects: 60% (328/546) 2024-06-26T07:18:42.1323527Z remote: Counting objects: 61% (334/546) 2024-06-26T07:18:42.1324293Z remote: Counting objects: 62% (339/546) 2024-06-26T07:18:42.1324793Z remote: Counting objects: 63% (344/546) 2024-06-26T07:18:42.1325278Z remote: Counting objects: 64% (350/546) 2024-06-26T07:18:42.1325779Z remote: Counting objects: 65% (355/546) 2024-06-26T07:18:42.1326275Z remote: Counting objects: 66% (361/546) 2024-06-26T07:18:42.1326778Z remote: Counting objects: 67% (366/546) 2024-06-26T07:18:42.1327266Z remote: Counting objects: 68% (372/546) 2024-06-26T07:18:42.1327761Z remote: Counting objects: 69% (377/546) 2024-06-26T07:18:42.1328263Z remote: Counting objects: 70% (383/546) 2024-06-26T07:18:42.1328759Z remote: Counting objects: 71% (388/546) 2024-06-26T07:18:42.1329246Z remote: Counting objects: 72% (394/546) 2024-06-26T07:18:42.1329743Z remote: Counting objects: 73% (399/546) 2024-06-26T07:18:42.1330352Z remote: Counting objects: 74% (405/546) 2024-06-26T07:18:42.1330850Z remote: Counting objects: 75% (410/546) 2024-06-26T07:18:42.1331337Z remote: Counting objects: 76% (415/546) 2024-06-26T07:18:42.1331832Z remote: Counting objects: 77% (421/546) 2024-06-26T07:18:42.1332327Z remote: Counting objects: 78% (426/546) 2024-06-26T07:18:42.1332813Z remote: Counting objects: 79% (432/546) 2024-06-26T07:18:42.1333306Z remote: Counting objects: 80% (437/546) 2024-06-26T07:18:42.1333803Z remote: Counting objects: 81% (443/546) 2024-06-26T07:18:42.1334297Z remote: Counting objects: 82% (448/546) 2024-06-26T07:18:42.1334784Z remote: Counting objects: 83% (454/546) 2024-06-26T07:18:42.1335277Z remote: Counting objects: 84% (459/546) 2024-06-26T07:18:42.1335770Z remote: Counting objects: 85% (465/546) 2024-06-26T07:18:42.1336265Z remote: Counting objects: 86% (470/546) 2024-06-26T07:18:42.1336876Z remote: Counting objects: 87% (476/546) 2024-06-26T07:18:42.1337378Z remote: Counting objects: 88% (481/546) 2024-06-26T07:18:42.1337874Z remote: Counting objects: 89% (486/546) 2024-06-26T07:18:42.1338368Z remote: Counting objects: 90% (492/546) 2024-06-26T07:18:42.1338855Z remote: Counting objects: 91% (497/546) 2024-06-26T07:18:42.1339530Z remote: Counting objects: 92% (503/546) 2024-06-26T07:18:42.1340030Z remote: Counting objects: 93% (508/546) 2024-06-26T07:18:42.1340525Z remote: Counting objects: 94% (514/546) 2024-06-26T07:18:42.1341018Z remote: Counting objects: 95% (519/546) 2024-06-26T07:18:42.1341524Z remote: Counting objects: 96% (525/546) 2024-06-26T07:18:42.1342019Z remote: Counting objects: 97% (530/546) 2024-06-26T07:18:42.1342504Z remote: Counting objects: 98% (536/546) 2024-06-26T07:18:42.1343053Z remote: Counting objects: 99% (541/546) 2024-06-26T07:18:42.1343552Z remote: Counting objects: 100% (546/546) 2024-06-26T07:18:42.1344080Z remote: Counting objects: 100% (546/546), done. 2024-06-26T07:18:42.1344631Z remote: Compressing objects: 0% (1/318) 2024-06-26T07:18:42.1345237Z remote: Compressing objects: 1% (4/318) 2024-06-26T07:18:42.1345755Z remote: Compressing objects: 2% (7/318) 2024-06-26T07:18:42.1346292Z remote: Compressing objects: 3% (10/318) 2024-06-26T07:18:42.1346813Z remote: Compressing objects: 4% (13/318) 2024-06-26T07:18:42.1347344Z remote: Compressing objects: 5% (16/318) 2024-06-26T07:18:42.1347878Z remote: Compressing objects: 6% (20/318) 2024-06-26T07:18:42.1348504Z remote: Compressing objects: 7% (23/318) 2024-06-26T07:18:42.1349030Z remote: Compressing objects: 8% (26/318) 2024-06-26T07:18:42.1349608Z remote: Compressing objects: 9% (29/318) 2024-06-26T07:18:42.1350128Z remote: Compressing objects: 10% (32/318) 2024-06-26T07:18:42.1350783Z remote: Compressing objects: 11% (35/318) 2024-06-26T07:18:42.1351517Z remote: Compressing objects: 12% (39/318) 2024-06-26T07:18:42.1354218Z remote: Compressing objects: 13% (42/318) 2024-06-26T07:18:42.1354900Z remote: Compressing objects: 14% (45/318) 2024-06-26T07:18:42.1355539Z remote: Compressing objects: 15% (48/318) 2024-06-26T07:18:42.1356109Z remote: Compressing objects: 16% (51/318) 2024-06-26T07:18:42.1356722Z remote: Compressing objects: 17% (55/318) 2024-06-26T07:18:42.1357447Z remote: Compressing objects: 18% (58/318) 2024-06-26T07:18:42.1358098Z remote: Compressing objects: 19% (61/318) 2024-06-26T07:18:42.1358616Z remote: Compressing objects: 20% (64/318) 2024-06-26T07:18:42.1359132Z remote: Compressing objects: 21% (67/318) 2024-06-26T07:18:42.1359651Z remote: Compressing objects: 22% (70/318) 2024-06-26T07:18:42.1360215Z remote: Compressing objects: 23% (74/318) 2024-06-26T07:18:42.1360901Z remote: Compressing objects: 24% (77/318) 2024-06-26T07:18:42.1361521Z remote: Compressing objects: 25% (80/318) 2024-06-26T07:18:42.1362148Z remote: Compressing objects: 26% (83/318) 2024-06-26T07:18:42.1362787Z remote: Compressing objects: 27% (86/318) 2024-06-26T07:18:42.1363457Z remote: Compressing objects: 28% (90/318) 2024-06-26T07:18:42.1364315Z remote: Compressing objects: 29% (93/318) 2024-06-26T07:18:42.1364949Z remote: Compressing objects: 30% (96/318) 2024-06-26T07:18:42.1365658Z remote: Compressing objects: 31% (99/318) 2024-06-26T07:18:42.1366185Z remote: Compressing objects: 32% (102/318) 2024-06-26T07:18:42.1366728Z remote: Compressing objects: 33% (105/318) 2024-06-26T07:18:42.1367268Z remote: Compressing objects: 34% (109/318) 2024-06-26T07:18:42.1367853Z remote: Compressing objects: 35% (112/318) 2024-06-26T07:18:42.1368378Z remote: Compressing objects: 36% (115/318) 2024-06-26T07:18:42.1369019Z remote: Compressing objects: 37% (118/318) 2024-06-26T07:18:42.1369707Z remote: Compressing objects: 38% (121/318) 2024-06-26T07:18:42.1370380Z remote: Compressing objects: 39% (125/318) 2024-06-26T07:18:42.1371099Z remote: Compressing objects: 40% (128/318) 2024-06-26T07:18:42.1371831Z remote: Compressing objects: 41% (131/318) 2024-06-26T07:18:42.1372475Z remote: Compressing objects: 42% (134/318) 2024-06-26T07:18:42.1373066Z remote: Compressing objects: 43% (137/318) 2024-06-26T07:18:42.1373632Z remote: Compressing objects: 44% (140/318) 2024-06-26T07:18:42.1374182Z remote: Compressing objects: 45% (144/318) 2024-06-26T07:18:42.1374712Z remote: Compressing objects: 46% (147/318) 2024-06-26T07:18:42.1375286Z remote: Compressing objects: 47% (150/318) 2024-06-26T07:18:42.1375951Z remote: Compressing objects: 48% (153/318) 2024-06-26T07:18:42.1376478Z remote: Compressing objects: 49% (156/318) 2024-06-26T07:18:42.1377136Z remote: Compressing objects: 50% (159/318) 2024-06-26T07:18:42.1377764Z remote: Compressing objects: 51% (163/318) 2024-06-26T07:18:42.1378436Z remote: Compressing objects: 52% (166/318) 2024-06-26T07:18:42.1379005Z remote: Compressing objects: 53% (169/318) 2024-06-26T07:18:42.1379531Z remote: Compressing objects: 54% (172/318) 2024-06-26T07:18:42.1380059Z remote: Compressing objects: 55% (175/318) 2024-06-26T07:18:42.1380796Z remote: Compressing objects: 56% (179/318) 2024-06-26T07:18:42.1381438Z remote: Compressing objects: 57% (182/318) 2024-06-26T07:18:42.1381968Z remote: Compressing objects: 58% (185/318) 2024-06-26T07:18:42.1382494Z remote: Compressing objects: 59% (188/318) 2024-06-26T07:18:42.1383118Z remote: Compressing objects: 60% (191/318) 2024-06-26T07:18:42.1383924Z remote: Compressing objects: 61% (194/318) 2024-06-26T07:18:42.1384473Z remote: Compressing objects: 62% (198/318) 2024-06-26T07:18:42.1384987Z remote: Compressing objects: 63% (201/318) 2024-06-26T07:18:42.1385546Z remote: Compressing objects: 64% (204/318) 2024-06-26T07:18:42.1386265Z remote: Compressing objects: 65% (207/318) 2024-06-26T07:18:42.1386842Z remote: Compressing objects: 66% (210/318) 2024-06-26T07:18:42.1387363Z remote: Compressing objects: 67% (214/318) 2024-06-26T07:18:42.1387899Z remote: Compressing objects: 68% (217/318) 2024-06-26T07:18:42.1388623Z remote: Compressing objects: 69% (220/318) 2024-06-26T07:18:42.1389221Z remote: Compressing objects: 70% (223/318) 2024-06-26T07:18:42.1389738Z remote: Compressing objects: 71% (226/318) 2024-06-26T07:18:42.1390263Z remote: Compressing objects: 72% (229/318) 2024-06-26T07:18:42.1390812Z remote: Compressing objects: 73% (233/318) 2024-06-26T07:18:42.1391535Z remote: Compressing objects: 74% (236/318) 2024-06-26T07:18:42.1392101Z remote: Compressing objects: 75% (239/318) 2024-06-26T07:18:42.1392666Z remote: Compressing objects: 76% (242/318) 2024-06-26T07:18:42.1393191Z remote: Compressing objects: 77% (245/318) 2024-06-26T07:18:42.1393715Z remote: Compressing objects: 78% (249/318) 2024-06-26T07:18:42.1394271Z remote: Compressing objects: 79% (252/318) 2024-06-26T07:18:42.1394823Z remote: Compressing objects: 80% (255/318) 2024-06-26T07:18:42.1395346Z remote: Compressing objects: 81% (258/318) 2024-06-26T07:18:42.1395993Z remote: Compressing objects: 82% (261/318) 2024-06-26T07:18:42.1396558Z remote: Compressing objects: 83% (264/318) 2024-06-26T07:18:42.1397083Z remote: Compressing objects: 84% (268/318) 2024-06-26T07:18:42.1397605Z remote: Compressing objects: 85% (271/318) 2024-06-26T07:18:42.1398130Z remote: Compressing objects: 86% (274/318) 2024-06-26T07:18:42.1398646Z remote: Compressing objects: 87% (277/318) 2024-06-26T07:18:42.1399213Z remote: Compressing objects: 88% (280/318) 2024-06-26T07:18:42.1399747Z remote: Compressing objects: 89% (284/318) 2024-06-26T07:18:42.1400268Z remote: Compressing objects: 90% (287/318) 2024-06-26T07:18:42.1400786Z remote: Compressing objects: 91% (290/318) 2024-06-26T07:18:42.1401309Z remote: Compressing objects: 92% (293/318) 2024-06-26T07:18:42.1401871Z remote: Compressing objects: 93% (296/318) 2024-06-26T07:18:42.1402386Z remote: Compressing objects: 94% (299/318) 2024-06-26T07:18:42.1402912Z remote: Compressing objects: 95% (303/318) 2024-06-26T07:18:42.1403433Z remote: Compressing objects: 96% (306/318) 2024-06-26T07:18:42.1404414Z remote: Compressing objects: 97% (309/318) 2024-06-26T07:18:42.1404934Z remote: Compressing objects: 98% (312/318) 2024-06-26T07:18:42.1405458Z remote: Compressing objects: 99% (315/318) 2024-06-26T07:18:42.1405980Z remote: Compressing objects: 100% (318/318) 2024-06-26T07:18:42.1406543Z remote: Compressing objects: 100% (318/318), done. 2024-06-26T07:18:42.1463766Z Receiving objects: 0% (1/30187) 2024-06-26T07:18:42.1505792Z Receiving objects: 1% (302/30187) 2024-06-26T07:18:42.1549995Z Receiving objects: 2% (604/30187) 2024-06-26T07:18:42.1589154Z Receiving objects: 3% (906/30187) 2024-06-26T07:18:42.1622697Z Receiving objects: 4% (1208/30187) 2024-06-26T07:18:42.1658836Z Receiving objects: 5% (1510/30187) 2024-06-26T07:18:42.1693366Z Receiving objects: 6% (1812/30187) 2024-06-26T07:18:42.1737105Z Receiving objects: 7% (2114/30187) 2024-06-26T07:18:42.1773450Z Receiving objects: 8% (2415/30187) 2024-06-26T07:18:42.1808348Z Receiving objects: 9% (2717/30187) 2024-06-26T07:18:42.1848513Z Receiving objects: 10% (3019/30187) 2024-06-26T07:18:42.1916888Z Receiving objects: 11% (3321/30187) 2024-06-26T07:18:42.1988602Z Receiving objects: 12% (3623/30187) 2024-06-26T07:18:42.2051947Z Receiving objects: 13% (3925/30187) 2024-06-26T07:18:42.2242644Z Receiving objects: 14% (4227/30187) 2024-06-26T07:18:42.2361856Z Receiving objects: 15% (4529/30187) 2024-06-26T07:18:42.2458553Z Receiving objects: 16% (4830/30187) 2024-06-26T07:18:42.2519063Z Receiving objects: 17% (5132/30187) 2024-06-26T07:18:42.2560713Z Receiving objects: 18% (5434/30187) 2024-06-26T07:18:42.2592095Z Receiving objects: 19% (5736/30187) 2024-06-26T07:18:42.2664507Z Receiving objects: 20% (6038/30187) 2024-06-26T07:18:42.2685896Z Receiving objects: 21% (6340/30187) 2024-06-26T07:18:42.2706930Z Receiving objects: 22% (6642/30187) 2024-06-26T07:18:42.3530744Z Receiving objects: 23% (6944/30187) 2024-06-26T07:18:43.1411165Z Receiving objects: 24% (7245/30187) 2024-06-26T07:18:43.3149451Z Receiving objects: 24% (7443/30187), 73.29 MiB | 73.29 MiB/s 2024-06-26T07:18:43.8181601Z Receiving objects: 25% (7547/30187), 73.29 MiB | 73.29 MiB/s 2024-06-26T07:18:43.8865174Z Receiving objects: 26% (7849/30187), 112.61 MiB | 75.07 MiB/s 2024-06-26T07:18:43.9538098Z Receiving objects: 27% (8151/30187), 112.61 MiB | 75.07 MiB/s 2024-06-26T07:18:44.0223784Z Receiving objects: 28% (8453/30187), 112.61 MiB | 75.07 MiB/s 2024-06-26T07:18:44.0891468Z Receiving objects: 29% (8755/30187), 112.61 MiB | 75.07 MiB/s 2024-06-26T07:18:44.1411268Z Receiving objects: 30% (9057/30187), 112.61 MiB | 75.07 MiB/s 2024-06-26T07:18:44.1572455Z Receiving objects: 30% (9286/30187), 152.28 MiB | 76.14 MiB/s 2024-06-26T07:18:44.2259154Z Receiving objects: 31% (9358/30187), 152.28 MiB | 76.14 MiB/s 2024-06-26T07:18:44.2934696Z Receiving objects: 32% (9660/30187), 152.28 MiB | 76.14 MiB/s 2024-06-26T07:18:44.5239133Z Receiving objects: 33% (9962/30187), 152.28 MiB | 76.14 MiB/s 2024-06-26T07:18:44.5975370Z Receiving objects: 34% (10264/30187), 152.28 MiB | 76.14 MiB/s 2024-06-26T07:18:44.6101930Z Receiving objects: 35% (10566/30187), 152.28 MiB | 76.14 MiB/s 2024-06-26T07:18:44.8120498Z Receiving objects: 36% (10868/30187), 152.28 MiB | 76.14 MiB/s 2024-06-26T07:18:45.1307946Z Receiving objects: 37% (11170/30187), 185.21 MiB | 74.08 MiB/s 2024-06-26T07:18:45.1413337Z Receiving objects: 38% (11472/30187), 185.21 MiB | 74.08 MiB/s 2024-06-26T07:18:45.1538633Z Receiving objects: 38% (11518/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.1553953Z Receiving objects: 39% (11773/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.1582173Z Receiving objects: 40% (12075/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.1595965Z Receiving objects: 41% (12377/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.1610916Z Receiving objects: 42% (12679/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.1622765Z Receiving objects: 43% (12981/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.1633302Z Receiving objects: 44% (13283/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.1654733Z Receiving objects: 45% (13585/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.1666979Z Receiving objects: 46% (13887/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.1671871Z Receiving objects: 47% (14188/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.1678827Z Receiving objects: 48% (14490/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.1701820Z Receiving objects: 49% (14792/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.1719419Z Receiving objects: 50% (15094/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.1730932Z Receiving objects: 51% (15396/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.1947631Z Receiving objects: 52% (15698/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.1948285Z Receiving objects: 53% (16000/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.1952740Z Receiving objects: 54% (16301/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.1959214Z Receiving objects: 55% (16603/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.1965163Z Receiving objects: 56% (16905/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.1983371Z Receiving objects: 57% (17207/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.1994330Z Receiving objects: 58% (17509/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.2005314Z Receiving objects: 59% (17811/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.2020157Z Receiving objects: 60% (18113/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.2029163Z Receiving objects: 61% (18415/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.2039973Z Receiving objects: 62% (18716/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.2194176Z Receiving objects: 63% (19018/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.2200857Z Receiving objects: 64% (19320/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.2207867Z Receiving objects: 65% (19622/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.2214790Z Receiving objects: 66% (19924/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.2224864Z Receiving objects: 67% (20226/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.2233222Z Receiving objects: 68% (20528/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.2506004Z Receiving objects: 69% (20830/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.2925107Z Receiving objects: 70% (21131/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.2931803Z Receiving objects: 71% (21433/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.2944223Z Receiving objects: 72% (21735/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.2957908Z Receiving objects: 73% (22037/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.2972052Z Receiving objects: 74% (22339/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.2983335Z Receiving objects: 75% (22641/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.2992830Z Receiving objects: 76% (22943/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.3012795Z Receiving objects: 77% (23244/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.3019893Z Receiving objects: 78% (23546/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.3028866Z Receiving objects: 79% (23848/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.3052761Z Receiving objects: 80% (24150/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.5186552Z Receiving objects: 81% (24452/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.5409093Z Receiving objects: 82% (24754/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.5443976Z Receiving objects: 83% (25056/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.5455126Z Receiving objects: 84% (25358/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.5467143Z Receiving objects: 85% (25659/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.5499242Z Receiving objects: 86% (25961/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.5530359Z Receiving objects: 87% (26263/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.5685760Z Receiving objects: 88% (26565/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.5692601Z Receiving objects: 89% (26867/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.5700623Z Receiving objects: 90% (27169/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.5706605Z Receiving objects: 91% (27471/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.5714039Z Receiving objects: 92% (27773/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.5720652Z Receiving objects: 93% (28074/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.5838616Z Receiving objects: 94% (28376/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.5854816Z Receiving objects: 95% (28678/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.5866063Z Receiving objects: 96% (28980/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.5885033Z Receiving objects: 97% (29282/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.5901761Z Receiving objects: 98% (29584/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.5921845Z Receiving objects: 99% (29886/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.5922745Z remote: Total 30187 (delta 256), reused 486 (delta 216), pack-reused 29641 2024-06-26T07:18:45.5961457Z Receiving objects: 100% (30187/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:18:45.5962731Z Receiving objects: 100% (30187/30187), 245.25 MiB | 70.98 MiB/s, done. 2024-06-26T07:18:45.5989965Z Resolving deltas: 0% (0/16205) 2024-06-26T07:18:45.5998558Z Resolving deltas: 1% (166/16205) 2024-06-26T07:18:45.6013313Z Resolving deltas: 2% (327/16205) 2024-06-26T07:18:45.6028242Z Resolving deltas: 3% (511/16205) 2024-06-26T07:18:45.6048910Z Resolving deltas: 4% (649/16205) 2024-06-26T07:18:45.6059406Z Resolving deltas: 5% (821/16205) 2024-06-26T07:18:45.6067854Z Resolving deltas: 6% (973/16205) 2024-06-26T07:18:45.6078731Z Resolving deltas: 7% (1140/16205) 2024-06-26T07:18:45.6088554Z Resolving deltas: 8% (1303/16205) 2024-06-26T07:18:45.6098642Z Resolving deltas: 9% (1477/16205) 2024-06-26T07:18:45.6108900Z Resolving deltas: 10% (1656/16205) 2024-06-26T07:18:45.6119837Z Resolving deltas: 11% (1786/16205) 2024-06-26T07:18:45.6129455Z Resolving deltas: 12% (1945/16205) 2024-06-26T07:18:45.6143674Z Resolving deltas: 13% (2108/16205) 2024-06-26T07:18:45.6152371Z Resolving deltas: 14% (2273/16205) 2024-06-26T07:18:45.6171995Z Resolving deltas: 15% (2431/16205) 2024-06-26T07:18:45.6188064Z Resolving deltas: 16% (2619/16205) 2024-06-26T07:18:45.6204609Z Resolving deltas: 17% (2761/16205) 2024-06-26T07:18:45.6207509Z Resolving deltas: 18% (3031/16205) 2024-06-26T07:18:45.6216407Z Resolving deltas: 19% (3081/16205) 2024-06-26T07:18:45.6233805Z Resolving deltas: 20% (3269/16205) 2024-06-26T07:18:45.6261742Z Resolving deltas: 21% (3416/16205) 2024-06-26T07:18:45.6287957Z Resolving deltas: 22% (3566/16205) 2024-06-26T07:18:45.6288602Z Resolving deltas: 23% (3884/16205) 2024-06-26T07:18:45.6297490Z Resolving deltas: 24% (3907/16205) 2024-06-26T07:18:45.6313571Z Resolving deltas: 25% (4053/16205) 2024-06-26T07:18:45.6318400Z Resolving deltas: 26% (4329/16205) 2024-06-26T07:18:45.6329206Z Resolving deltas: 27% (4409/16205) 2024-06-26T07:18:45.6365490Z Resolving deltas: 28% (4543/16205) 2024-06-26T07:18:45.6387346Z Resolving deltas: 29% (4700/16205) 2024-06-26T07:18:45.6400151Z Resolving deltas: 30% (4892/16205) 2024-06-26T07:18:45.6415812Z Resolving deltas: 31% (5027/16205) 2024-06-26T07:18:45.6432034Z Resolving deltas: 32% (5199/16205) 2024-06-26T07:18:45.6447490Z Resolving deltas: 33% (5348/16205) 2024-06-26T07:18:45.6456981Z Resolving deltas: 34% (5533/16205) 2024-06-26T07:18:45.6465504Z Resolving deltas: 35% (5707/16205) 2024-06-26T07:18:45.6476045Z Resolving deltas: 36% (5834/16205) 2024-06-26T07:18:45.6497194Z Resolving deltas: 37% (5998/16205) 2024-06-26T07:18:45.6510448Z Resolving deltas: 38% (6172/16205) 2024-06-26T07:18:45.6528381Z Resolving deltas: 39% (6324/16205) 2024-06-26T07:18:45.6535395Z Resolving deltas: 40% (6489/16205) 2024-06-26T07:18:45.6557703Z Resolving deltas: 41% (6660/16205) 2024-06-26T07:18:45.6568121Z Resolving deltas: 42% (6824/16205) 2024-06-26T07:18:45.6574295Z Resolving deltas: 43% (7008/16205) 2024-06-26T07:18:45.6596213Z Resolving deltas: 44% (7144/16205) 2024-06-26T07:18:45.6600637Z Resolving deltas: 45% (7356/16205) 2024-06-26T07:18:45.6609021Z Resolving deltas: 46% (7474/16205) 2024-06-26T07:18:45.6619721Z Resolving deltas: 47% (7626/16205) 2024-06-26T07:18:45.6640719Z Resolving deltas: 48% (7781/16205) 2024-06-26T07:18:45.6660200Z Resolving deltas: 49% (7941/16205) 2024-06-26T07:18:45.6672963Z Resolving deltas: 50% (8106/16205) 2024-06-26T07:18:45.6698597Z Resolving deltas: 51% (8266/16205) 2024-06-26T07:18:45.6709745Z Resolving deltas: 52% (8448/16205) 2024-06-26T07:18:45.6715979Z Resolving deltas: 53% (8589/16205) 2024-06-26T07:18:45.6727801Z Resolving deltas: 54% (8754/16205) 2024-06-26T07:18:45.6744562Z Resolving deltas: 55% (8920/16205) 2024-06-26T07:18:45.6752059Z Resolving deltas: 56% (9101/16205) 2024-06-26T07:18:45.6767893Z Resolving deltas: 57% (9239/16205) 2024-06-26T07:18:45.6776033Z Resolving deltas: 58% (9402/16205) 2024-06-26T07:18:45.6786031Z Resolving deltas: 59% (9613/16205) 2024-06-26T07:18:45.6787426Z Resolving deltas: 60% (9871/16205) 2024-06-26T07:18:45.6795554Z Resolving deltas: 61% (9890/16205) 2024-06-26T07:18:45.6809439Z Resolving deltas: 62% (10049/16205) 2024-06-26T07:18:45.6823253Z Resolving deltas: 63% (10290/16205) 2024-06-26T07:18:45.6824784Z Resolving deltas: 64% (10512/16205) 2024-06-26T07:18:45.6832723Z Resolving deltas: 65% (10550/16205) 2024-06-26T07:18:45.6840317Z Resolving deltas: 66% (10707/16205) 2024-06-26T07:18:45.6860380Z Resolving deltas: 67% (10877/16205) 2024-06-26T07:18:45.6867205Z Resolving deltas: 68% (11172/16205) 2024-06-26T07:18:45.6882980Z Resolving deltas: 69% (11274/16205) 2024-06-26T07:18:45.6892371Z Resolving deltas: 70% (11367/16205) 2024-06-26T07:18:45.6899551Z Resolving deltas: 71% (11506/16205) 2024-06-26T07:18:45.6913507Z Resolving deltas: 72% (11672/16205) 2024-06-26T07:18:45.6926401Z Resolving deltas: 73% (11857/16205) 2024-06-26T07:18:45.6940175Z Resolving deltas: 74% (11994/16205) 2024-06-26T07:18:45.6973547Z Resolving deltas: 75% (12177/16205) 2024-06-26T07:18:45.6977799Z Resolving deltas: 78% (12718/16205) 2024-06-26T07:18:45.6988354Z Resolving deltas: 79% (12804/16205) 2024-06-26T07:18:45.6995830Z Resolving deltas: 80% (12981/16205) 2024-06-26T07:18:45.7008458Z Resolving deltas: 81% (13132/16205) 2024-06-26T07:18:45.7016450Z Resolving deltas: 82% (13312/16205) 2024-06-26T07:18:45.7025812Z Resolving deltas: 83% (13451/16205) 2024-06-26T07:18:45.7036448Z Resolving deltas: 84% (13614/16205) 2024-06-26T07:18:45.7058047Z Resolving deltas: 85% (13775/16205) 2024-06-26T07:18:45.7067515Z Resolving deltas: 86% (13937/16205) 2024-06-26T07:18:45.7074743Z Resolving deltas: 87% (14105/16205) 2024-06-26T07:18:45.7082172Z Resolving deltas: 88% (14271/16205) 2024-06-26T07:18:45.7090082Z Resolving deltas: 89% (14434/16205) 2024-06-26T07:18:45.7101304Z Resolving deltas: 90% (14590/16205) 2024-06-26T07:18:45.7108524Z Resolving deltas: 91% (14766/16205) 2024-06-26T07:18:45.7120610Z Resolving deltas: 92% (14913/16205) 2024-06-26T07:18:45.7125156Z Resolving deltas: 93% (15135/16205) 2024-06-26T07:18:45.7133467Z Resolving deltas: 94% (15250/16205) 2024-06-26T07:18:45.7141576Z Resolving deltas: 95% (15406/16205) 2024-06-26T07:18:45.7147367Z Resolving deltas: 96% (15561/16205) 2024-06-26T07:18:45.7153134Z Resolving deltas: 97% (15721/16205) 2024-06-26T07:18:45.7162092Z Resolving deltas: 98% (15885/16205) 2024-06-26T07:18:45.7196024Z Resolving deltas: 99% (16043/16205) 2024-06-26T07:18:45.7196514Z Resolving deltas: 100% (16205/16205) 2024-06-26T07:18:45.7196946Z Resolving deltas: 100% (16205/16205), done. 2024-06-26T07:18:46.7781189Z + pushd torchbench 2024-06-26T07:18:46.7781710Z ~/workspace/torchbench ~/workspace 2024-06-26T07:18:46.7782398Z + git checkout 23512dbebd44a11eb84afbf53c3c071dd105297e 2024-06-26T07:18:46.8357804Z Note: switching to '23512dbebd44a11eb84afbf53c3c071dd105297e'. 2024-06-26T07:18:46.8358413Z 2024-06-26T07:18:46.8358895Z You are in 'detached HEAD' state. You can look around, make experimental 2024-06-26T07:18:46.8359685Z changes and commit them, and you can discard any commits you make in this 2024-06-26T07:18:46.8360433Z state without impacting any branches by switching back to a branch. 2024-06-26T07:18:46.8360884Z 2024-06-26T07:18:46.8361195Z If you want to create a new branch to retain commits you create, you may 2024-06-26T07:18:46.8361996Z do so (now or later) by using -c with the switch command. Example: 2024-06-26T07:18:46.8362506Z 2024-06-26T07:18:46.8362691Z git switch -c 2024-06-26T07:18:46.8362985Z 2024-06-26T07:18:46.8363182Z Or undo this operation with: 2024-06-26T07:18:46.8363436Z 2024-06-26T07:18:46.8363562Z git switch - 2024-06-26T07:18:46.8363981Z 2024-06-26T07:18:46.8364331Z Turn off this advice by setting config variable advice.detachedHead to false 2024-06-26T07:18:46.8364831Z 2024-06-26T07:18:46.8364995Z HEAD is now at 23512dbe Pin numpy version 2024-06-26T07:18:46.8365454Z + '[' '' ']' 2024-06-26T07:18:46.8365814Z + python install.py --continue_on_fail 2024-06-26T07:18:48.8224307Z checking packages torch, torchvision, torchaudio are installed...OK 2024-06-26T07:19:04.4566786Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2024-06-26T07:19:06.8547172Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2024-06-26T07:19:09.4868567Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2024-06-26T07:19:11.9174521Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2024-06-26T07:19:11.9329591Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2024-06-26T07:19:20.6367381Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2024-06-26T07:19:24.0609352Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2024-06-26T07:19:27.4364940Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2024-06-26T07:19:30.8386424Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2024-06-26T07:19:30.8389333Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2024-06-26T07:19:32.6844366Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2024-06-26T07:19:36.2743037Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2024-06-26T07:19:36.2888079Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2024-06-26T07:20:31.3515993Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2024-06-26T07:20:40.3178374Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2024-06-26T07:20:47.8873667Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2024-06-26T07:20:55.1496667Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2024-06-26T07:21:03.8318131Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2024-06-26T07:21:11.6903508Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2024-06-26T07:21:18.6767414Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2024-06-26T07:21:26.2189575Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2024-06-26T07:21:33.8401708Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2024-06-26T07:21:41.8950322Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2024-06-26T07:21:49.2480815Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2024-06-26T07:21:56.8059670Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2024-06-26T07:22:00.8765012Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2024-06-26T07:22:40.4605393Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2024-06-26T07:22:46.5291666Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2024-06-26T07:22:53.4009163Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2024-06-26T07:23:04.1971993Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2024-06-26T07:23:06.4140627Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2024-06-26T07:23:08.8844521Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2024-06-26T07:23:16.8325141Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2024-06-26T07:23:24.9508631Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2024-06-26T07:23:31.6598995Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2024-06-26T07:23:40.3238164Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2024-06-26T07:23:47.4363452Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2024-06-26T07:23:53.7395441Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2024-06-26T07:24:01.0927224Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2024-06-26T07:24:17.3515994Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2024-06-26T07:24:24.7321326Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2024-06-26T07:24:30.0776142Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2024-06-26T07:24:36.1564901Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2024-06-26T07:24:44.0358722Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2024-06-26T07:24:44.0365102Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2024-06-26T07:24:57.3774828Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2024-06-26T07:25:02.6986748Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2024-06-26T07:25:02.6989587Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2024-06-26T07:25:11.3022050Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2024-06-26T07:25:13.5977967Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2024-06-26T07:25:15.9267491Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2024-06-26T07:25:55.7808730Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2024-06-26T07:27:15.6053400Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2024-06-26T07:27:15.6211401Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2024-06-26T07:27:18.1986996Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2024-06-26T07:27:18.2146628Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2024-06-26T07:27:18.2308684Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2024-06-26T07:27:18.2461737Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2024-06-26T07:27:18.2613922Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2024-06-26T07:27:18.2768026Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2024-06-26T07:27:38.7429104Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2024-06-26T07:27:38.7432340Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2024-06-26T07:27:41.0920223Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2024-06-26T07:27:44.4655767Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2024-06-26T07:27:44.4813423Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2024-06-26T07:27:44.4970623Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2024-06-26T07:27:44.5124388Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2024-06-26T07:27:44.5271579Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2024-06-26T07:27:44.5427954Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2024-06-26T07:27:51.8104878Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2024-06-26T07:27:54.3484318Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2024-06-26T07:27:58.9389452Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2024-06-26T07:27:58.9547802Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2024-06-26T07:27:58.9708890Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2024-06-26T07:27:58.9861398Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2024-06-26T07:27:59.0014871Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2024-06-26T07:27:59.0169068Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2024-06-26T07:28:08.9777530Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2024-06-26T07:28:23.6864014Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2024-06-26T07:28:23.7024154Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2024-06-26T07:28:23.7027520Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2024-06-26T07:28:23.7030071Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2024-06-26T07:28:27.4082742Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2024-06-26T07:28:30.3550298Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2024-06-26T07:28:30.3704096Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2024-06-26T07:28:50.1625901Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2024-06-26T07:28:57.0463789Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2024-06-26T07:29:01.4722587Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2024-06-26T07:29:09.6350578Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2024-06-26T07:29:09.6511892Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2024-06-26T07:29:09.6673829Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2024-06-26T07:29:09.6831375Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2024-06-26T07:29:09.6986333Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2024-06-26T07:29:09.7133353Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2024-06-26T07:29:09.7284368Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2024-06-26T07:29:09.7432907Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2024-06-26T07:29:21.6525702Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2024-06-26T07:29:34.2006868Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2024-06-26T07:29:34.2161351Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2024-06-26T07:29:37.2245680Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2024-06-26T07:29:40.4754449Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2024-06-26T07:29:40.8961077Z + popd 2024-06-26T07:29:40.8961416Z ~/workspace 2024-06-26T07:29:40.8961863Z + [[ dynamic_aot_eager_torchbench != *cpu_inductor* ]] 2024-06-26T07:29:40.8966637Z + install_torchrec_and_fbgemm 2024-06-26T07:29:40.8967045Z + local torchrec_commit 2024-06-26T07:29:40.8967425Z ++ get_pinned_commit torchrec 2024-06-26T07:29:40.8967826Z ++ cat .github/ci_commit_pins/torchrec.txt 2024-06-26T07:29:40.9048098Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T07:29:40.9048612Z + local fbgemm_commit 2024-06-26T07:29:40.9050412Z ++ get_pinned_commit fbgemm 2024-06-26T07:29:40.9050879Z ++ cat .github/ci_commit_pins/fbgemm.txt 2024-06-26T07:29:40.9060590Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:29:40.9061303Z + pip_uninstall torchrec-nightly 2024-06-26T07:29:40.9062023Z + pip uninstall -y torchrec-nightly 2024-06-26T07:29:41.2613476Z WARNING: Skipping torchrec-nightly as it is not installed. 2024-06-26T07:29:41.3082513Z + pip_uninstall fbgemm-gpu-nightly 2024-06-26T07:29:41.3083079Z + pip uninstall -y fbgemm-gpu-nightly 2024-06-26T07:29:41.6567338Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed. 2024-06-26T07:29:41.6984162Z + pip_install setuptools-git-versioning scikit-build pyre-extensions 2024-06-26T07:29:41.6985300Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions 2024-06-26T07:29:42.1452516Z Collecting setuptools-git-versioning 2024-06-26T07:29:42.1608520Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB) 2024-06-26T07:29:42.2586665Z Collecting scikit-build 2024-06-26T07:29:42.2621749Z Downloading scikit_build-0.18.0-py3-none-any.whl.metadata (17 kB) 2024-06-26T07:29:42.3529236Z Collecting pyre-extensions 2024-06-26T07:29:42.3564141Z Downloading pyre_extensions-0.0.30-py3-none-any.whl.metadata (4.0 kB) 2024-06-26T07:29:42.3969385Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.1) 2024-06-26T07:29:42.3973457Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (69.5.1) 2024-06-26T07:29:42.4528191Z Collecting toml>=0.10.2 (from setuptools-git-versioning) 2024-06-26T07:29:42.4557104Z Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) 2024-06-26T07:29:42.5201062Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0) 2024-06-26T07:29:42.5209915Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1) 2024-06-26T07:29:42.5214773Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.43.0) 2024-06-26T07:29:42.5787559Z Collecting typing-inspect (from pyre-extensions) 2024-06-26T07:29:42.5816632Z Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB) 2024-06-26T07:29:42.6174646Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.2) 2024-06-26T07:29:42.6401669Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0) 2024-06-26T07:29:42.6454788Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB) 2024-06-26T07:29:42.7031588Z Downloading scikit_build-0.18.0-py3-none-any.whl (85 kB) 2024-06-26T07:29:42.7614644Z Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB) 2024-06-26T07:29:42.8134899Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2024-06-26T07:29:42.8737635Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2024-06-26T07:29:45.0788267Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions 2024-06-26T07:29:45.4050370Z Successfully installed pyre-extensions-0.0.30 scikit-build-0.18.0 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0 2024-06-26T07:29:45.5239474Z + CUDA_PATH=/usr/local/cuda-12.1 2024-06-26T07:29:45.5240638Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-06-26T07:29:45.5242495Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-06-26T07:29:45.8648546Z Collecting fbgemm-gpu 2024-06-26T07:29:45.8651491Z Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-rb85k22y/fbgemm-gpu_721c46dea30345c3ae5da2c42ae82988 2024-06-26T07:29:45.8667745Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-rb85k22y/fbgemm-gpu_721c46dea30345c3ae5da2c42ae82988 2024-06-26T07:29:46.7734829Z Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd' 2024-06-26T07:29:46.7748769Z Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:29:47.1620799Z Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:29:47.6118059Z Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:29:47.6119153Z Running command git submodule update --init --recursive -q 2024-06-26T07:29:54.9523189Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:29:54.9544392Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.21.2) 2024-06-26T07:29:54.9550976Z Building wheels for collected packages: fbgemm-gpu 2024-06-26T08:12:12.3884561Z Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-06-26T08:12:12.5679471Z [?25h Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=252628107 sha256=1d08965c36861fd6975998d935adbaeeeefad121b499ebfc27c479007ab3c5b4 2024-06-26T08:12:12.5681284Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39 2024-06-26T08:12:12.5699625Z Successfully built fbgemm-gpu 2024-06-26T08:12:14.7495600Z Installing collected packages: fbgemm-gpu 2024-06-26T08:12:17.8726864Z Successfully installed fbgemm-gpu-0.4.1rc0.post421 2024-06-26T08:12:18.4980604Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:18.4982154Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:18.8364331Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:18.8368808Z Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-8w36c5j8 2024-06-26T08:12:18.8387505Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-8w36c5j8 2024-06-26T08:12:19.4386393Z Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2' 2024-06-26T08:12:19.4404719Z Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:19.7204238Z Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:20.0232625Z Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:20.2533124Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T08:12:20.2582702Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9) 2024-06-26T08:12:20.2586040Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3) 2024-06-26T08:12:20.2590662Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0) 2024-06-26T08:12:20.3106223Z Collecting torchmetrics (from torchrec==0.3.2) 2024-06-26T08:12:20.3264332Z Downloading torchmetrics-1.4.0.post0-py3-none-any.whl.metadata (19 kB) 2024-06-26T08:12:20.3323722Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.4) 2024-06-26T08:12:20.3326916Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421) 2024-06-26T08:12:20.3339900Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.21.2) 2024-06-26T08:12:20.3360192Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.10.0) 2024-06-26T08:12:20.4020367Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0) 2024-06-26T08:12:20.4024809Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-06-26T08:12:20.4029445Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-06-26T08:12:20.4803721Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.1) 2024-06-26T08:12:20.4808399Z Requirement already satisfied: torch>=1.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.5.0a0+gitb8c4c54) 2024-06-26T08:12:20.4997362Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2) 2024-06-26T08:12:20.5026812Z Downloading lightning_utilities-0.11.2-py3-none-any.whl.metadata (4.7 kB) 2024-06-26T08:12:20.5207034Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (69.5.1) 2024-06-26T08:12:20.5209175Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.2) 2024-06-26T08:12:20.5246472Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0) 2024-06-26T08:12:20.5316439Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.13.1) 2024-06-26T08:12:20.5319875Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.12.1) 2024-06-26T08:12:20.5322690Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.8.8) 2024-06-26T08:12:20.5325256Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.1.4) 2024-06-26T08:12:20.5328568Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2024.2.0) 2024-06-26T08:12:20.6004011Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.1.5) 2024-06-26T08:12:20.6313317Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.3.0) 2024-06-26T08:12:20.6462017Z Downloading torchmetrics-1.4.0.post0-py3-none-any.whl (868 kB) 2024-06-26T08:12:20.6648063Z Downloading lightning_utilities-0.11.2-py3-none-any.whl (26 kB) 2024-06-26T08:12:20.6746501Z Building wheels for collected packages: torchrec 2024-06-26T08:12:21.1111053Z Building wheel for torchrec (setup.py) ... [?25l- \ | done 2024-06-26T08:12:21.1119793Z [?25h Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374488 sha256=73bc8118c0ad574799abb953bfab32afa9c72d39c03f57ae3c18832063e3ac3c 2024-06-26T08:12:21.1122031Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8 2024-06-26T08:12:21.1144966Z Successfully built torchrec 2024-06-26T08:12:23.2359695Z Installing collected packages: lightning-utilities, torchmetrics, torchrec 2024-06-26T08:12:23.7940135Z Successfully installed lightning-utilities-0.11.2 torchmetrics-1.4.0.post0 torchrec-0.3.2 2024-06-26T08:12:23.9271882Z ++ pwd 2024-06-26T08:12:23.9272626Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2024-06-26T08:12:23.9273140Z + test_dynamo_benchmark torchbench 1 2024-06-26T08:12:23.9275852Z ++ pwd 2024-06-26T08:12:23.9277222Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:12:23.9277770Z + local suite=torchbench 2024-06-26T08:12:23.9279478Z + shift 2024-06-26T08:12:23.9279893Z + local shard_id=1 2024-06-26T08:12:23.9280304Z + shift 2024-06-26T08:12:23.9280709Z + [[ dynamic_aot_eager_torchbench == *perf_compare* ]] 2024-06-26T08:12:23.9281245Z + [[ dynamic_aot_eager_torchbench == *perf* ]] 2024-06-26T08:12:23.9281968Z + [[ dynamic_aot_eager_torchbench == *cpu_inductor* ]] 2024-06-26T08:12:23.9282606Z + [[ dynamic_aot_eager_torchbench == *aot_inductor* ]] 2024-06-26T08:12:23.9284183Z + test_single_dynamo_benchmark inference torchbench 1 --inference --bfloat16 2024-06-26T08:12:23.9284800Z ++ pwd 2024-06-26T08:12:23.9285278Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:12:23.9285936Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:12:23.9381924Z + local name=inference 2024-06-26T08:12:23.9382263Z + shift 2024-06-26T08:12:23.9382595Z + local suite=torchbench 2024-06-26T08:12:23.9383053Z + shift 2024-06-26T08:12:23.9383400Z + local shard_id=1 2024-06-26T08:12:23.9383770Z + shift 2024-06-26T08:12:23.9384054Z + partition_flags=() 2024-06-26T08:12:23.9384378Z + local partition_flags 2024-06-26T08:12:23.9384749Z + [[ -n 2 ]] 2024-06-26T08:12:23.9385056Z + [[ -n 1 ]] 2024-06-26T08:12:23.9385630Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-06-26T08:12:23.9386342Z + [[ dynamic_aot_eager_torchbench == *perf_compare* ]] 2024-06-26T08:12:23.9386869Z + [[ dynamic_aot_eager_torchbench == *perf* ]] 2024-06-26T08:12:23.9387394Z + [[ dynamic_aot_eager_torchbench == *aot_inductor* ]] 2024-06-26T08:12:23.9389307Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --backend aot_eager --dynamic-shapes --dynamic-batch-only --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2024-06-26T08:12:30.0121486Z 2024-06-26T08:12:30.4934906Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:12:30.4935468Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:12:30.4973541Z cuda eval llama 2024-06-26T08:12:58.6082609Z pass 2024-06-26T08:12:58.6242784Z TIMING: backend_compile:21.32705 entire_frame_compile:27.46367 2024-06-26T08:12:58.6245716Z STATS: call_* op count: 1154 | FakeTensor.__torch_dispatch__:428 | FakeTensorMode.__torch_dispatch__:31450 | attempt fast:846 | fast is_contiguous:846 | ProxyTorchDispatchMode.__torch_dispatch__:8227 2024-06-26T08:12:58.6247937Z Dynamo produced 1 graphs covering 1154 ops with 0 graph breaks (0 unique) 2024-06-26T08:13:03.5553955Z 2024-06-26T08:13:03.8301157Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:03.8302100Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:03.8303014Z cuda eval llama_v2_7b_16h 2024-06-26T08:13:03.8308068Z Traceback (most recent call last): 2024-06-26T08:13:03.8309400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:13:03.8310155Z ) = runner.load_model( 2024-06-26T08:13:03.8310898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 327, in load_model 2024-06-26T08:13:03.8311706Z benchmark = benchmark_cls( 2024-06-26T08:13:03.8314070Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-26T08:13:03.8315078Z obj = type.__call__(cls, *args, **kwargs) 2024-06-26T08:13:03.8316105Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h/__init__.py", line 12, in __init__ 2024-06-26T08:13:03.8317321Z super().__init__(name="llama_v2_7b_16h", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:13:03.8319443Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-06-26T08:13:03.8321296Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:13:03.8322836Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-06-26T08:13:03.8324394Z self._skip_by_device_name() 2024-06-26T08:13:03.8325824Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-06-26T08:13:03.8327902Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-06-26T08:13:03.8329863Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llama_v2_7b_16h/metadata.yaml`. 2024-06-26T08:13:03.8331110Z 2024-06-26T08:13:03.8331323Z model_fail_to_load 2024-06-26T08:13:07.0052089Z 2024-06-26T08:13:07.2763854Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:07.2764432Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:07.2764858Z cuda eval llava 2024-06-26T08:13:07.2769163Z Traceback (most recent call last): 2024-06-26T08:13:07.2769882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:13:07.2770551Z ) = runner.load_model( 2024-06-26T08:13:07.2771232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 327, in load_model 2024-06-26T08:13:07.2771948Z benchmark = benchmark_cls( 2024-06-26T08:13:07.2772682Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-26T08:13:07.2773467Z obj = type.__call__(cls, *args, **kwargs) 2024-06-26T08:13:07.2774324Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 11, in __init__ 2024-06-26T08:13:07.2775421Z super().__init__(name="llava", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:13:07.2776678Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-06-26T08:13:07.2777941Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:13:07.2778919Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-06-26T08:13:07.2788177Z self._skip_by_device_name() 2024-06-26T08:13:07.2789003Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-06-26T08:13:07.2790291Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-06-26T08:13:07.2791390Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`. 2024-06-26T08:13:07.2791959Z 2024-06-26T08:13:07.2792084Z model_fail_to_load 2024-06-26T08:13:10.4651142Z 2024-06-26T08:13:11.0348058Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:11.0349244Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:11.0355459Z cuda eval maml 2024-06-26T08:13:11.0358455Z pass_due_to_skip 2024-06-26T08:13:11.0366873Z TIMING: 2024-06-26T08:13:11.0367724Z STATS: call_* op count: 0 2024-06-26T08:13:11.0368499Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T08:13:14.2722685Z 2024-06-26T08:13:14.7271850Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:14.7272356Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:14.7279104Z cuda eval maml_omniglot 2024-06-26T08:13:15.3450910Z pass 2024-06-26T08:13:15.3451573Z TIMING: backend_compile:0.27375 entire_frame_compile:0.44352 2024-06-26T08:13:15.3452876Z STATS: call_* op count: 28 | FakeTensor.__torch_dispatch__:89 | FakeTensorMode.__torch_dispatch__:1572 | ProxyTorchDispatchMode.__torch_dispatch__:340 2024-06-26T08:13:15.3453984Z Dynamo produced 1 graphs covering 28 ops with 0 graph breaks (0 unique) 2024-06-26T08:13:18.6526534Z 2024-06-26T08:13:18.8467230Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/mnasnet1.0_top1_73.512-f206786ef8.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/mnasnet1.0_top1_73.512-f206786ef8.pth 2024-06-26T08:13:18.8652868Z 2024-06-26T08:13:18.8653123Z 2024-06-26T08:13:18.9074078Z 0% 0.00/16.9M [00:00 will be ignored 2024-06-26T08:14:01.3657120Z [rank0]:W0626 08:14:01.364000 139716219425408 torch/_dynamo/backends/distributed.py:88] [2/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance. 2024-06-26T08:14:01.3847337Z [rank0]:W0626 08:14:01.384000 139716219425408 torch/_dynamo/backends/distributed.py:105] [2/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes: 2024-06-26T08:14:01.3853561Z [rank0]:W0626 08:14:01.384000 139716219425408 torch/_dynamo/backends/distributed.py:105] [2/0_1] ┌─────────┬─────────────┬────────────────────────┐ 2024-06-26T08:14:01.3855648Z [rank0]:W0626 08:14:01.384000 139716219425408 torch/_dynamo/backends/distributed.py:105] [2/0_1] │ Index │ Extra Ops │ Extra Param Size (b) │ 2024-06-26T08:14:01.3857539Z [rank0]:W0626 08:14:01.384000 139716219425408 torch/_dynamo/backends/distributed.py:105] [2/0_1] ├─────────┼─────────────┼────────────────────────┤ 2024-06-26T08:14:01.3859202Z [rank0]:W0626 08:14:01.384000 139716219425408 torch/_dynamo/backends/distributed.py:105] [2/0_1] │ 0 │ 157 │ 44910720 │ 2024-06-26T08:14:01.3860905Z [rank0]:W0626 08:14:01.384000 139716219425408 torch/_dynamo/backends/distributed.py:105] [2/0_1] └─────────┴─────────────┴────────────────────────┘ 2024-06-26T08:14:09.0271160Z [rank0]:W0626 08:14:09.026000 139716219425408 torch/_dynamo/variables/tensor.py:714] [5/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T08:14:09.0272637Z [rank0]:W0626 08:14:09.026000 139716219425408 torch/_dynamo/variables/tensor.py:714] [5/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T08:14:09.0273795Z [rank0]:W0626 08:14:09.026000 139716219425408 torch/_dynamo/variables/tensor.py:714] [5/0] or: 2024-06-26T08:14:09.0274910Z [rank0]:W0626 08:14:09.026000 139716219425408 torch/_dynamo/variables/tensor.py:714] [5/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T08:14:09.0276243Z [rank0]:W0626 08:14:09.026000 139716219425408 torch/_dynamo/variables/tensor.py:714] [5/0] to include these operations in the captured graph. 2024-06-26T08:14:09.0277353Z [rank0]:W0626 08:14:09.026000 139716219425408 torch/_dynamo/variables/tensor.py:714] [5/0] 2024-06-26T08:14:12.2729115Z pass 2024-06-26T08:14:12.2746068Z TIMING: entire_frame_compile:18.15013 backend_compile:8.72832 2024-06-26T08:14:12.2747356Z STATS: call_* op count: 877 | FakeTensor.__torch_dispatch__:2512 | FakeTensorMode.__torch_dispatch__:41738 | ProxyTorchDispatchMode.__torch_dispatch__:9290 2024-06-26T08:14:12.2748486Z Dynamo produced 7 graphs covering 877 ops with 5 graph breaks (3 unique) 2024-06-26T08:14:12.7080451Z [rank0]:[W626 08:14:12.759035822 ProcessGroupNCCL.cpp:1161] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) 2024-06-26T08:14:16.3195428Z 2024-06-26T08:14:16.5899200Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:16.5899723Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:16.5900164Z cuda eval moondream 2024-06-26T08:14:16.5904441Z Traceback (most recent call last): 2024-06-26T08:14:16.5905380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:14:16.5906272Z ) = runner.load_model( 2024-06-26T08:14:16.5906969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 327, in load_model 2024-06-26T08:14:16.5907709Z benchmark = benchmark_cls( 2024-06-26T08:14:16.5908434Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-26T08:14:16.5909270Z obj = type.__call__(cls, *args, **kwargs) 2024-06-26T08:14:16.5910348Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream/__init__.py", line 11, in __init__ 2024-06-26T08:14:16.5911669Z super().__init__(name="moondream", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:14:16.5913116Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-06-26T08:14:16.5914448Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:14:16.5915798Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-06-26T08:14:16.5916570Z self._skip_by_device_name() 2024-06-26T08:14:16.5917376Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-06-26T08:14:16.5918611Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-06-26T08:14:16.5919704Z NotImplementedError: The current device NVIDIA A10G is skipped by its `moondream/metadata.yaml`. 2024-06-26T08:14:16.5920294Z 2024-06-26T08:14:16.5920414Z model_fail_to_load 2024-06-26T08:14:19.7660752Z 2024-06-26T08:14:21.6502559Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M 2024-06-26T08:14:21.8909553Z num decayed parameter tensors: 50, with 124,354,560 parameters 2024-06-26T08:14:21.8910467Z num non-decayed parameter tensors: 98, with 121,344 parameters 2024-06-26T08:14:21.8912364Z using fused AdamW: True 2024-06-26T08:14:22.1486626Z 2024-06-26T08:14:22.1487009Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:14:22.1531683Z cuda eval nanogpt 2024-06-26T08:14:44.5334105Z pass 2024-06-26T08:14:44.5334573Z TIMING: backend_compile:16.04845 entire_frame_compile:21.84205 2024-06-26T08:14:44.5335837Z STATS: call_* op count: 834 | FakeTensorMode.__torch_dispatch__:31779 | FakeTensor.__torch_dispatch__:819 | attempt fast:1908 | fast is_contiguous:1908 | ProxyTorchDispatchMode.__torch_dispatch__:6111 2024-06-26T08:14:44.5337270Z Dynamo produced 1 graphs covering 834 ops with 0 graph breaks (0 unique) 2024-06-26T08:14:48.9185521Z 2024-06-26T08:14:51.2475796Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:51.2476262Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:14:51.2479308Z cuda eval nvidia_deeprecommender 2024-06-26T08:14:51.9032641Z pass 2024-06-26T08:14:51.9033429Z TIMING: backend_compile:0.19009 entire_frame_compile:0.37932 2024-06-26T08:14:51.9034640Z STATS: call_* op count: 26 | FakeTensorMode.__torch_dispatch__:815 | ProxyTorchDispatchMode.__torch_dispatch__:242 2024-06-26T08:14:51.9035590Z Dynamo produced 1 graphs covering 26 ops with 0 graph breaks (0 unique) 2024-06-26T08:14:55.1784839Z 2024-06-26T08:14:56.0096076Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:56.0096583Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:56.0115807Z cuda eval opacus_cifar10 2024-06-26T08:15:00.4494495Z pass 2024-06-26T08:15:00.4495434Z TIMING: backend_compile:1.05706 entire_frame_compile:4.11446 2024-06-26T08:15:00.4496400Z STATS: call_* op count: 138 | FakeTensorMode.__torch_dispatch__:3450 | ProxyTorchDispatchMode.__torch_dispatch__:1188 2024-06-26T08:15:00.4497539Z Dynamo produced 1 graphs covering 138 ops with 0 graph breaks (0 unique) 2024-06-26T08:15:03.7789518Z 2024-06-26T08:15:04.3387447Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:04.3387927Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:04.3455868Z cuda eval phlippe_densenet 2024-06-26T08:15:09.3778730Z pass 2024-06-26T08:15:09.3779503Z TIMING: backend_compile:2.8141 entire_frame_compile:4.67758 2024-06-26T08:15:09.3780582Z STATS: call_* op count: 372 | FakeTensor.__torch_dispatch__:993 | FakeTensorMode.__torch_dispatch__:21942 | ProxyTorchDispatchMode.__torch_dispatch__:4641 2024-06-26T08:15:09.3782470Z Dynamo produced 1 graphs covering 372 ops with 0 graph breaks (0 unique) 2024-06-26T08:15:12.7525797Z 2024-06-26T08:15:13.2435126Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:13.2435656Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:13.2468479Z cuda eval phlippe_resnet 2024-06-26T08:15:15.2614390Z pass 2024-06-26T08:15:15.2614941Z TIMING: backend_compile:1.13079 entire_frame_compile:1.76639 2024-06-26T08:15:15.2616007Z STATS: call_* op count: 142 | FakeTensor.__torch_dispatch__:391 | FakeTensorMode.__torch_dispatch__:8229 | ProxyTorchDispatchMode.__torch_dispatch__:1757 2024-06-26T08:15:15.2617287Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-06-26T08:15:18.5427000Z 2024-06-26T08:15:18.8485131Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:18.8485634Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:18.8486088Z cuda eval pyhpc_equation_of_state 2024-06-26T08:15:20.4156063Z pass 2024-06-26T08:15:20.4156723Z TIMING: backend_compile:0.99569 entire_frame_compile:1.39219 2024-06-26T08:15:20.4157643Z STATS: call_* op count: 732 | FakeTensorMode.__torch_dispatch__:8987 | ProxyTorchDispatchMode.__torch_dispatch__:3732 2024-06-26T08:15:20.4158614Z Dynamo produced 1 graphs covering 732 ops with 0 graph breaks (0 unique) 2024-06-26T08:15:23.6912797Z 2024-06-26T08:15:23.9878739Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:23.9879612Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:23.9884581Z cuda eval pyhpc_isoneutral_mixing 2024-06-26T08:15:33.1527861Z pass 2024-06-26T08:15:33.1529135Z TIMING: backend_compile:7.45291 entire_frame_compile:8.89148 2024-06-26T08:15:33.1533164Z STATS: call_* op count: 1488 | FakeTensorMode.__torch_dispatch__:37150 | FakeTensor.__torch_dispatch__:18 | ProxyTorchDispatchMode.__torch_dispatch__:13834 2024-06-26T08:15:33.1534395Z Dynamo produced 1 graphs covering 1488 ops with 0 graph breaks (0 unique) 2024-06-26T08:15:36.5593824Z 2024-06-26T08:15:37.3092122Z loading model: 0it [00:00, ?it/s]WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-06-26T08:15:37.3903787Z 2024-06-26T08:15:37.3904812Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:37.3906655Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-06-26T08:15:37.3908599Z cuda eval pyhpc_turbulent_kinetic_energy 2024-06-26T08:15:37.4363343Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-06-26T08:15:47.8524315Z pass 2024-06-26T08:15:47.8623826Z TIMING: backend_compile:8.51412 entire_frame_compile:10.1277 2024-06-26T08:15:47.8627464Z STATS: call_* op count: 1992 | FakeTensorMode.__torch_dispatch__:39778 | FakeTensor.__torch_dispatch__:71 | ProxyTorchDispatchMode.__torch_dispatch__:14474 2024-06-26T08:15:47.8629705Z Dynamo produced 1 graphs covering 1992 ops with 0 graph breaks (0 unique) 2024-06-26T08:15:51.2649635Z 2024-06-26T08:15:51.9339310Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:51.9339756Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:51.9352387Z cuda eval pytorch_CycleGAN_and_pix2pix 2024-06-26T08:15:59.3334534Z pass 2024-06-26T08:15:59.3335044Z TIMING: backend_compile:1.19263 entire_frame_compile:1.7582 2024-06-26T08:15:59.3338370Z STATS: call_* op count: 182 | FakeTensorMode.__torch_dispatch__:7533 | FakeTensor.__torch_dispatch__:297 | ProxyTorchDispatchMode.__torch_dispatch__:2092 2024-06-26T08:15:59.3339556Z Dynamo produced 1 graphs covering 182 ops with 0 graph breaks (0 unique) 2024-06-26T08:16:02.6441971Z 2024-06-26T08:16:03.6951159Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:16:03.6951632Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:16:03.6974221Z cuda eval pytorch_stargan 2024-06-26T08:16:21.0571393Z pass 2024-06-26T08:16:21.0614672Z TIMING: backend_compile:2.06977 entire_frame_compile:2.69589 2024-06-26T08:16:21.0615862Z STATS: call_* op count: 112 | FakeTensorMode.__torch_dispatch__:12793 | FakeTensor.__torch_dispatch__:329 | ProxyTorchDispatchMode.__torch_dispatch__:3694 2024-06-26T08:16:21.0617133Z Dynamo produced 1 graphs covering 112 ops with 0 graph breaks (0 unique) 2024-06-26T08:16:24.3854032Z 2024-06-26T08:16:25.0017568Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:16:25.0018031Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:16:25.0043004Z cuda eval pytorch_unet 2024-06-26T08:16:34.9107985Z pass 2024-06-26T08:16:34.9270604Z TIMING: backend_compile:1.60209 entire_frame_compile:2.44844 2024-06-26T08:16:34.9272166Z STATS: call_* op count: 142 | FakeTensor.__torch_dispatch__:462 | FakeTensorMode.__torch_dispatch__:11387 | ProxyTorchDispatchMode.__torch_dispatch__:3036 2024-06-26T08:16:34.9274071Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-06-26T08:16:38.2106456Z 2024-06-26T08:16:38.9805812Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnet152-394f9c45.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnet152-394f9c45.pth 2024-06-26T08:16:38.9997994Z 2024-06-26T08:16:38.9998238Z 2024-06-26T08:16:39.1001848Z 0% 0.00/230M [00:00 2024-06-26T08:23:34.4392395Z torchbench_main() 2024-06-26T08:23:34.4393076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 469, in torchbench_main 2024-06-26T08:23:34.4393877Z main(TorchBenchmarkRunner(), original_dir) 2024-06-26T08:23:34.4394581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3722, in main 2024-06-26T08:23:34.4397093Z process_entry(0, runner, original_dir, args) 2024-06-26T08:23:34.4397913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3654, in process_entry 2024-06-26T08:23:34.4401015Z return run(runner, args, original_dir) 2024-06-26T08:23:34.4401733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4283, in run 2024-06-26T08:23:34.4405627Z assert marked, f"nothing in example_inputs had a dim with {batch_size}" 2024-06-26T08:23:34.4406333Z AssertionError: nothing in example_inputs had a dim with 4 2024-06-26T08:23:37.3988393Z Run failed with return code: 1 2024-06-26T08:23:37.3988917Z Output: None 2024-06-26T08:23:37.3989213Z Error: None 2024-06-26T08:23:39.9636540Z 2024-06-26T08:23:40.1244739Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/shufflenetv2_x1-5666bf0f80.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/shufflenetv2_x1-5666bf0f80.pth 2024-06-26T08:23:40.1486412Z 2024-06-26T08:23:40.1486558Z 2024-06-26T08:23:40.1738330Z 0% 0.00/8.79M [00:00 will be ignored 2024-06-26T08:27:56.2545777Z pass 2024-06-26T08:27:56.2546564Z TIMING: entire_frame_compile:3.21274 backend_compile:2.1415 2024-06-26T08:27:56.2547866Z STATS: call_* op count: 78 | FakeTensor.__torch_dispatch__:398 | FakeTensorMode.__torch_dispatch__:4415 | attempt fast:65 | fast is_contiguous:65 | ProxyTorchDispatchMode.__torch_dispatch__:1170 2024-06-26T08:27:56.2549190Z Dynamo produced 3 graphs covering 78 ops with 7 graph breaks (5 unique) 2024-06-26T08:27:59.6290154Z 2024-06-26T08:28:01.9388690Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:28:01.9389167Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:28:01.9389607Z cuda train mnasnet1_0 2024-06-26T08:28:17.5370532Z W0626 08:28:17.536000 140208790401664 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:28:35.2527170Z pass 2024-06-26T08:28:35.2569083Z TIMING: entire_frame_compile:31.37591 backend_compile:20.73272 2024-06-26T08:28:35.2571510Z STATS: call_* op count: 648 | FakeTensor.__torch_dispatch__:4228 | FakeTensorMode.__torch_dispatch__:50642 | attempt fast:1046 | fast is_contiguous:1046 | ProxyTorchDispatchMode.__torch_dispatch__:12781 2024-06-26T08:28:35.2572865Z Dynamo produced 3 graphs covering 648 ops with 7 graph breaks (5 unique) 2024-06-26T08:28:39.5616160Z 2024-06-26T08:28:41.3751280Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:28:41.3751773Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:28:41.3752204Z cuda train mobilenet_v2 2024-06-26T08:29:01.8079668Z pass 2024-06-26T08:29:01.8125797Z TIMING: entire_frame_compile:18.61818 backend_compile:12.81174 2024-06-26T08:29:01.8129044Z STATS: call_* op count: 157 | FakeTensor.__torch_dispatch__:1151 | FakeTensorMode.__torch_dispatch__:31247 | attempt fast:1046 | fast is_contiguous:1046 | ProxyTorchDispatchMode.__torch_dispatch__:5414 2024-06-26T08:29:01.8130511Z Dynamo produced 2 graphs covering 157 ops with 6 graph breaks (5 unique) 2024-06-26T08:29:06.0158490Z 2024-06-26T08:29:07.8254388Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:29:07.8255053Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:29:07.8255529Z cuda train mobilenet_v2_quantized_qat 2024-06-26T08:29:07.8256035Z Traceback (most recent call last): 2024-06-26T08:29:07.8257077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:29:07.8258082Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:29:07.8259143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in forward_and_backward_pass 2024-06-26T08:29:07.8259967Z pred = mod(*cloned_inputs) 2024-06-26T08:29:07.8263349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 738, in call_wrapped 2024-06-26T08:29:07.8264588Z return self._wrapped_call(self, *args, **kwargs) 2024-06-26T08:29:07.8265645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 316, in __call__ 2024-06-26T08:29:07.8266683Z raise e 2024-06-26T08:29:07.8267670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 303, in __call__ 2024-06-26T08:29:07.8268631Z return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] 2024-06-26T08:29:07.8269985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:29:07.8270862Z return self._call_impl(*args, **kwargs) 2024-06-26T08:29:07.8271799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:29:07.8272624Z return forward_call(*args, **kwargs) 2024-06-26T08:29:07.8273094Z File ".3", line 207, in forward 2024-06-26T08:29:07.8273834Z activation_post_process_101 = self.activation_post_process_101(classifier_1); classifier_1 = None 2024-06-26T08:29:07.8275054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:29:07.8275922Z return self._call_impl(*args, **kwargs) 2024-06-26T08:29:07.8276837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:29:07.8277660Z return forward_call(*args, **kwargs) 2024-06-26T08:29:07.8278612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 343, in forward 2024-06-26T08:29:07.8279516Z return torch.fused_moving_avg_obs_fake_quant( 2024-06-26T08:29:07.8280067Z RuntimeError: expected scalar type Float but found Half 2024-06-26T08:29:07.8280442Z 2024-06-26T08:29:07.8280744Z The above exception was the direct cause of the following exception: 2024-06-26T08:29:07.8281188Z 2024-06-26T08:29:07.8281336Z Traceback (most recent call last): 2024-06-26T08:29:07.8281974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:29:07.8282624Z ) = runner.load_model( 2024-06-26T08:29:07.8283273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:29:07.8284350Z self.validate_model(model, example_inputs) 2024-06-26T08:29:07.8285096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:29:07.8285854Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:29:07.8286316Z RuntimeError: Eager run failed 2024-06-26T08:29:07.8286567Z 2024-06-26T08:29:07.8286687Z eager_fail_to_run 2024-06-26T08:29:11.0771940Z 2024-06-26T08:29:13.2850379Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:29:13.2850989Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:29:13.2851853Z cuda train mobilenet_v3_large 2024-06-26T08:29:36.0238818Z W0626 08:29:36.023000 139889117004416 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:29:56.2094625Z pass 2024-06-26T08:29:56.2144194Z TIMING: entire_frame_compile:40.80964 backend_compile:25.90594 2024-06-26T08:29:56.2146365Z STATS: call_* op count: 731 | FakeTensor.__torch_dispatch__:4772 | FakeTensorMode.__torch_dispatch__:59175 | attempt fast:1363 | fast is_contiguous:1363 | ProxyTorchDispatchMode.__torch_dispatch__:14109 2024-06-26T08:29:56.2147784Z Dynamo produced 3 graphs covering 731 ops with 7 graph breaks (5 unique) 2024-06-26T08:30:00.9763362Z 2024-06-26T08:30:02.9579078Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:30:02.9579789Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:30:02.9583503Z cuda train moco 2024-06-26T08:30:07.5508938Z [rank0]:W0626 08:30:07.550000 140688923243136 torch/_logging/_internal.py:1040] [3/0] Profiler function will be ignored 2024-06-26T08:30:15.1797957Z [rank0]:W0626 08:30:15.178000 140688923243136 torch/_dynamo/backends/distributed.py:88] [4/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance. 2024-06-26T08:30:15.1989512Z [rank0]:W0626 08:30:15.198000 140688923243136 torch/_dynamo/backends/distributed.py:105] [4/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes: 2024-06-26T08:30:15.1994296Z [rank0]:W0626 08:30:15.198000 140688923243136 torch/_dynamo/backends/distributed.py:105] [4/0_1] ┌─────────┬─────────────┬────────────────────────┐ 2024-06-26T08:30:15.1995865Z [rank0]:W0626 08:30:15.198000 140688923243136 torch/_dynamo/backends/distributed.py:105] [4/0_1] │ Index │ Extra Ops │ Extra Param Size (b) │ 2024-06-26T08:30:15.1997402Z [rank0]:W0626 08:30:15.198000 140688923243136 torch/_dynamo/backends/distributed.py:105] [4/0_1] ├─────────┼─────────────┼────────────────────────┤ 2024-06-26T08:30:15.1998888Z [rank0]:W0626 08:30:15.198000 140688923243136 torch/_dynamo/backends/distributed.py:105] [4/0_1] │ 0 │ 161 │ 94032128 │ 2024-06-26T08:30:15.2000412Z [rank0]:W0626 08:30:15.198000 140688923243136 torch/_dynamo/backends/distributed.py:105] [4/0_1] └─────────┴─────────────┴────────────────────────┘ 2024-06-26T08:30:35.5777498Z [rank0]:W0626 08:30:35.576000 140688923243136 torch/_dynamo/variables/tensor.py:714] [7/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T08:30:35.5779391Z [rank0]:W0626 08:30:35.576000 140688923243136 torch/_dynamo/variables/tensor.py:714] [7/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T08:30:35.5781601Z [rank0]:W0626 08:30:35.576000 140688923243136 torch/_dynamo/variables/tensor.py:714] [7/0] or: 2024-06-26T08:30:35.5782923Z [rank0]:W0626 08:30:35.576000 140688923243136 torch/_dynamo/variables/tensor.py:714] [7/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T08:30:35.5784418Z [rank0]:W0626 08:30:35.576000 140688923243136 torch/_dynamo/variables/tensor.py:714] [7/0] to include these operations in the captured graph. 2024-06-26T08:30:35.5785617Z [rank0]:W0626 08:30:35.576000 140688923243136 torch/_dynamo/variables/tensor.py:714] [7/0] 2024-06-26T08:30:45.3059796Z pass 2024-06-26T08:30:45.3355987Z TIMING: entire_frame_compile:37.34828 backend_compile:25.28883 2024-06-26T08:30:45.3357419Z STATS: call_* op count: 907 | FakeTensorMode.__torch_dispatch__:63823 | FakeTensor.__torch_dispatch__:2565 | attempt fast:1979 | fast is_contiguous:1968 | slow no contiguity match:11 | ProxyTorchDispatchMode.__torch_dispatch__:13601 2024-06-26T08:30:45.3359262Z Dynamo produced 10 graphs covering 907 ops with 11 graph breaks (8 unique) 2024-06-26T08:30:46.3547749Z [rank0]:[W626 08:30:46.405772106 ProcessGroupNCCL.cpp:1161] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) 2024-06-26T08:30:50.6239719Z 2024-06-26T08:30:52.4924366Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M 2024-06-26T08:30:52.7157118Z num decayed parameter tensors: 50, with 124,354,560 parameters 2024-06-26T08:30:52.7157998Z num non-decayed parameter tensors: 98, with 121,344 parameters 2024-06-26T08:30:52.7161683Z using fused AdamW: True 2024-06-26T08:30:53.0563249Z 2024-06-26T08:30:53.0564100Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:30:53.0564605Z cuda train nanogpt 2024-06-26T08:31:17.1038900Z W0626 08:31:17.103000 139948479783552 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:31:31.4733248Z pass 2024-06-26T08:31:31.5800081Z TIMING: entire_frame_compile:36.60019 backend_compile:25.29711 2024-06-26T08:31:31.5801557Z STATS: call_* op count: 883 | FakeTensorMode.__torch_dispatch__:55125 | FakeTensor.__torch_dispatch__:3618 | attempt fast:1715 | fast is_contiguous:1715 | ProxyTorchDispatchMode.__torch_dispatch__:15889 2024-06-26T08:31:31.5807203Z Dynamo produced 3 graphs covering 883 ops with 7 graph breaks (5 unique) 2024-06-26T08:31:36.2290916Z 2024-06-26T08:31:38.5528887Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:31:38.5529383Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:31:38.5529861Z cuda train nvidia_deeprecommender 2024-06-26T08:31:40.9912347Z W0626 08:31:40.990000 140154765206144 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:31:42.3044953Z pass 2024-06-26T08:31:42.3342911Z TIMING: entire_frame_compile:2.81164 backend_compile:1.704 2024-06-26T08:31:42.3344250Z STATS: call_* op count: 71 | FakeTensorMode.__torch_dispatch__:4194 | attempt fast:150 | fast is_contiguous:150 | FakeTensor.__torch_dispatch__:283 | ProxyTorchDispatchMode.__torch_dispatch__:1070 2024-06-26T08:31:42.3345574Z Dynamo produced 3 graphs covering 71 ops with 7 graph breaks (5 unique) 2024-06-26T08:31:45.6957245Z 2024-06-26T08:31:46.4986795Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:31:46.4987341Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:31:46.4987849Z cuda train opacus_cifar10 2024-06-26T08:31:46.5089953Z Traceback (most recent call last): 2024-06-26T08:31:46.5090735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:31:46.5091700Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:31:46.5092685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 458, in forward_and_backward_pass 2024-06-26T08:31:46.5093651Z self.grad_scaler.scale(loss).backward() 2024-06-26T08:31:46.5094833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-06-26T08:31:46.5095655Z torch.autograd.backward( 2024-06-26T08:31:46.5096949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 288, in backward 2024-06-26T08:31:46.5097807Z _engine_run_backward( 2024-06-26T08:31:46.5098692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 768, in _engine_run_backward 2024-06-26T08:31:46.5100271Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-06-26T08:31:46.5101766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 97, in __call__ 2024-06-26T08:31:46.5102727Z return self.hook(module, *args, **kwargs) 2024-06-26T08:31:46.5103897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 327, in capture_backprops_hook 2024-06-26T08:31:46.5104912Z activations, backprops = self.rearrange_grad_samples( 2024-06-26T08:31:46.5106058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 383, in rearrange_grad_samples 2024-06-26T08:31:46.5106986Z raise ValueError( 2024-06-26T08:31:46.5107815Z ValueError: No activations detected for , run forward after add_hooks(model) 2024-06-26T08:31:46.5108504Z 2024-06-26T08:31:46.5108806Z The above exception was the direct cause of the following exception: 2024-06-26T08:31:46.5109249Z 2024-06-26T08:31:46.5109401Z Traceback (most recent call last): 2024-06-26T08:31:46.5110053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:31:46.5110709Z ) = runner.load_model( 2024-06-26T08:31:46.5111366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:31:46.5112124Z self.validate_model(model, example_inputs) 2024-06-26T08:31:46.5112882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:31:46.5122217Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:31:46.5122744Z RuntimeError: Eager run failed 2024-06-26T08:31:46.5123004Z 2024-06-26T08:31:46.5123135Z eager_fail_to_run 2024-06-26T08:31:49.7389354Z 2024-06-26T08:31:50.4282094Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:31:50.4282736Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:31:50.4283414Z cuda train phlippe_densenet 2024-06-26T08:32:09.4924946Z pass 2024-06-26T08:32:09.4928748Z TIMING: entire_frame_compile:18.36809 backend_compile:12.72655 2024-06-26T08:32:09.4930727Z STATS: call_* op count: 190 | FakeTensor.__torch_dispatch__:1014 | FakeTensorMode.__torch_dispatch__:31569 | attempt fast:1020 | fast is_contiguous:996 | ProxyTorchDispatchMode.__torch_dispatch__:6410 | slow no contiguity match:24 2024-06-26T08:32:09.4932283Z Dynamo produced 2 graphs covering 190 ops with 6 graph breaks (5 unique) 2024-06-26T08:32:13.6564098Z 2024-06-26T08:32:14.2467419Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:14.2468363Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:14.2469244Z cuda train phlippe_resnet 2024-06-26T08:32:21.1485658Z pass 2024-06-26T08:32:21.1486267Z TIMING: entire_frame_compile:6.51724 backend_compile:4.55262 2024-06-26T08:32:21.1487561Z STATS: call_* op count: 75 | FakeTensor.__torch_dispatch__:412 | FakeTensorMode.__torch_dispatch__:11705 | attempt fast:414 | fast is_contiguous:414 | ProxyTorchDispatchMode.__torch_dispatch__:2419 2024-06-26T08:32:21.1488867Z Dynamo produced 2 graphs covering 75 ops with 6 graph breaks (5 unique) 2024-06-26T08:32:24.6751882Z 2024-06-26T08:32:25.7885007Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:25.7885483Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:32:25.7885966Z cuda train pytorch_CycleGAN_and_pix2pix 2024-06-26T08:32:37.3351046Z pass 2024-06-26T08:32:37.3439132Z TIMING: entire_frame_compile:2.66853 backend_compile:1.97089 2024-06-26T08:32:37.3440932Z STATS: call_* op count: 95 | FakeTensorMode.__torch_dispatch__:9722 | FakeTensor.__torch_dispatch__:321 | ProxyTorchDispatchMode.__torch_dispatch__:3021 2024-06-26T08:32:37.3442083Z Dynamo produced 2 graphs covering 95 ops with 6 graph breaks (5 unique) 2024-06-26T08:32:40.7161076Z 2024-06-26T08:32:42.0504634Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:42.0505508Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:32:42.0506375Z cuda train pytorch_stargan 2024-06-26T08:33:20.0374539Z pass 2024-06-26T08:33:20.0585483Z TIMING: entire_frame_compile:11.7901 backend_compile:8.82403 2024-06-26T08:33:20.0587996Z STATS: call_* op count: 61 | FakeTensorMode.__torch_dispatch__:17856 | FakeTensor.__torch_dispatch__:372 | attempt fast:611 | fast is_contiguous:611 | ProxyTorchDispatchMode.__torch_dispatch__:3815 2024-06-26T08:33:20.0589344Z Dynamo produced 2 graphs covering 61 ops with 6 graph breaks (5 unique) 2024-06-26T08:33:23.9531507Z 2024-06-26T08:33:24.7405650Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:33:24.7406527Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:33:24.7407422Z cuda train pytorch_unet 2024-06-26T08:33:52.5025802Z W0626 08:33:52.501000 139664072123008 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:34:00.8660227Z pass_due_to_skip 2024-06-26T08:34:01.0396221Z TIMING: entire_frame_compile:16.03791 backend_compile:10.58124 2024-06-26T08:34:01.0399069Z STATS: call_* op count: 315 | FakeTensor.__torch_dispatch__:1963 | FakeTensorMode.__torch_dispatch__:25215 | attempt fast:526 | fast is_contiguous:522 | ProxyTorchDispatchMode.__torch_dispatch__:7044 | slow no contiguity match:4 2024-06-26T08:34:01.0400562Z Dynamo produced 3 graphs covering 315 ops with 7 graph breaks (5 unique) 2024-06-26T08:34:04.7874647Z 2024-06-26T08:34:06.9233998Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:34:06.9234492Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:34:06.9234927Z cuda train resnet152 2024-06-26T08:34:56.2822387Z W0626 08:34:56.281000 140070748447360 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:35:49.0714906Z pass 2024-06-26T08:35:49.1361950Z TIMING: entire_frame_compile:95.42912 backend_compile:64.75362 2024-06-26T08:35:49.1366122Z STATS: call_* op count: 1938 | FakeTensor.__torch_dispatch__:11838 | FakeTensorMode.__torch_dispatch__:152687 | attempt fast:3103 | fast is_contiguous:3103 | ProxyTorchDispatchMode.__torch_dispatch__:38630 2024-06-26T08:35:49.1368786Z Dynamo produced 3 graphs covering 1938 ops with 7 graph breaks (5 unique) 2024-06-26T08:35:55.7638078Z 2024-06-26T08:35:56.8574429Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:35:56.8575019Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:35:56.8575455Z cuda train resnet18 2024-06-26T08:36:04.5072364Z pass 2024-06-26T08:36:04.5149000Z TIMING: entire_frame_compile:6.0791 backend_compile:4.59498 2024-06-26T08:36:04.5150272Z STATS: call_* op count: 73 | FakeTensor.__torch_dispatch__:410 | FakeTensorMode.__torch_dispatch__:11098 | attempt fast:412 | fast is_contiguous:412 | ProxyTorchDispatchMode.__torch_dispatch__:2338 2024-06-26T08:36:04.5151573Z Dynamo produced 2 graphs covering 73 ops with 6 graph breaks (5 unique) 2024-06-26T08:36:08.0923143Z 2024-06-26T08:36:09.7218049Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:36:09.7218518Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:36:09.7218970Z cuda train resnet50 2024-06-26T08:36:29.4591480Z pass 2024-06-26T08:36:29.4773275Z TIMING: entire_frame_compile:15.38668 backend_compile:12.01736 2024-06-26T08:36:29.4774590Z STATS: call_* op count: 179 | FakeTensor.__torch_dispatch__:1015 | FakeTensorMode.__torch_dispatch__:28732 | attempt fast:1063 | fast is_contiguous:1063 | ProxyTorchDispatchMode.__torch_dispatch__:6038 2024-06-26T08:36:29.4775893Z Dynamo produced 2 graphs covering 179 ops with 6 graph breaks (5 unique) 2024-06-26T08:36:33.5188270Z 2024-06-26T08:36:35.5214692Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:36:35.5215276Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:36:35.5215720Z cuda train resnet50_quantized_qat 2024-06-26T08:36:35.5216268Z Traceback (most recent call last): 2024-06-26T08:36:35.5217409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:36:35.5218264Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:36:35.5220215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in forward_and_backward_pass 2024-06-26T08:36:35.5221358Z pred = mod(*cloned_inputs) 2024-06-26T08:36:35.5222635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 738, in call_wrapped 2024-06-26T08:36:35.5223857Z return self._wrapped_call(self, *args, **kwargs) 2024-06-26T08:36:35.5225083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 316, in __call__ 2024-06-26T08:36:35.5226169Z raise e 2024-06-26T08:36:35.5227209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 303, in __call__ 2024-06-26T08:36:35.5228264Z return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] 2024-06-26T08:36:35.5229539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:36:35.5230515Z return self._call_impl(*args, **kwargs) 2024-06-26T08:36:35.5231514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:36:35.5232407Z return forward_call(*args, **kwargs) 2024-06-26T08:36:35.5232926Z File ".3", line 167, in forward 2024-06-26T08:36:35.5233623Z activation_post_process_73 = self.activation_post_process_73(fc); fc = None 2024-06-26T08:36:35.5234883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:36:35.5236074Z return self._call_impl(*args, **kwargs) 2024-06-26T08:36:35.5237056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:36:35.5237942Z return forward_call(*args, **kwargs) 2024-06-26T08:36:35.5238969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 343, in forward 2024-06-26T08:36:35.5239939Z return torch.fused_moving_avg_obs_fake_quant( 2024-06-26T08:36:35.5240537Z RuntimeError: expected scalar type Float but found Half 2024-06-26T08:36:35.5240970Z 2024-06-26T08:36:35.5241264Z The above exception was the direct cause of the following exception: 2024-06-26T08:36:35.5241767Z 2024-06-26T08:36:35.5241914Z Traceback (most recent call last): 2024-06-26T08:36:35.5242608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:36:35.5243273Z ) = runner.load_model( 2024-06-26T08:36:35.5244397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:36:35.5245241Z self.validate_model(model, example_inputs) 2024-06-26T08:36:35.5246054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:36:35.5246881Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:36:35.5247403Z RuntimeError: Eager run failed 2024-06-26T08:36:35.5247670Z 2024-06-26T08:36:35.5247791Z eager_fail_to_run 2024-06-26T08:36:38.7869386Z 2024-06-26T08:36:40.5320313Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:36:40.5321188Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:36:40.5322043Z cuda train resnext50_32x4d 2024-06-26T08:37:01.6961250Z W0626 08:37:01.694000 140065049965184 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:37:19.8512595Z pass 2024-06-26T08:37:19.8711466Z TIMING: entire_frame_compile:33.16171 backend_compile:22.28028 2024-06-26T08:37:19.8712752Z STATS: call_* op count: 680 | FakeTensor.__torch_dispatch__:4178 | FakeTensorMode.__torch_dispatch__:52617 | attempt fast:1063 | fast is_contiguous:1063 | ProxyTorchDispatchMode.__torch_dispatch__:13300 2024-06-26T08:37:19.8714059Z Dynamo produced 3 graphs covering 680 ops with 7 graph breaks (5 unique) 2024-06-26T08:37:24.2558574Z 2024-06-26T08:37:30.1676226Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:37:30.1677358Z loading model: 0it [00:05, ?it/s] 2024-06-26T08:37:30.1678213Z cuda train sam 2024-06-26T08:37:30.1684031Z Traceback (most recent call last): 2024-06-26T08:37:30.1684896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:37:30.1685673Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:37:30.1686763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 458, in forward_and_backward_pass 2024-06-26T08:37:30.1687635Z self.grad_scaler.scale(loss).backward() 2024-06-26T08:37:30.1688730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-06-26T08:37:30.1689478Z torch.autograd.backward( 2024-06-26T08:37:30.1690328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 288, in backward 2024-06-26T08:37:30.1691115Z _engine_run_backward( 2024-06-26T08:37:30.1692012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 768, in _engine_run_backward 2024-06-26T08:37:30.1693168Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-06-26T08:37:30.1694114Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn 2024-06-26T08:37:30.1694631Z 2024-06-26T08:37:30.1694932Z The above exception was the direct cause of the following exception: 2024-06-26T08:37:30.1700059Z 2024-06-26T08:37:30.1700294Z Traceback (most recent call last): 2024-06-26T08:37:30.1701194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:37:30.1702091Z ) = runner.load_model( 2024-06-26T08:37:30.1702970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:37:30.1703823Z self.validate_model(model, example_inputs) 2024-06-26T08:37:30.1704584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:37:30.1705352Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:37:30.1705810Z RuntimeError: Eager run failed 2024-06-26T08:37:30.1706069Z 2024-06-26T08:37:30.1706182Z eager_fail_to_run 2024-06-26T08:37:33.4282747Z 2024-06-26T08:37:34.5191752Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:37:34.5192315Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:37:34.5192747Z cuda train shufflenet_v2_x1_0 2024-06-26T08:37:52.7269486Z pass 2024-06-26T08:37:52.7279331Z TIMING: entire_frame_compile:17.20577 backend_compile:13.21358 2024-06-26T08:37:52.7280904Z STATS: call_* op count: 307 | FakeTensor.__torch_dispatch__:1058 | FakeTensorMode.__torch_dispatch__:32616 | attempt fast:1075 | fast is_contiguous:1075 | ProxyTorchDispatchMode.__torch_dispatch__:6897 2024-06-26T08:37:52.7282239Z Dynamo produced 2 graphs covering 307 ops with 6 graph breaks (5 unique) 2024-06-26T08:37:56.9602842Z 2024-06-26T08:37:57.6519724Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:37:57.6520312Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:37:57.6520754Z cuda train soft_actor_critic 2024-06-26T08:37:59.2620361Z W0626 08:37:59.261000 139989716951680 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:37:59.8969634Z pass 2024-06-26T08:37:59.8970306Z TIMING: entire_frame_compile:2.02444 backend_compile:1.28274 2024-06-26T08:37:59.8971746Z STATS: call_* op count: 59 | FakeTensor.__torch_dispatch__:198 | FakeTensorMode.__torch_dispatch__:2601 | attempt fast:46 | fast is_contiguous:46 | ProxyTorchDispatchMode.__torch_dispatch__:686 2024-06-26T08:37:59.8973008Z Dynamo produced 4 graphs covering 59 ops with 6 graph breaks (5 unique) 2024-06-26T08:38:03.2298449Z 2024-06-26T08:38:04.4296141Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:38:04.4296984Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:38:04.4298107Z Traceback (most recent call last): 2024-06-26T08:38:04.4298998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 473, in 2024-06-26T08:38:04.4299715Z torchbench_main() 2024-06-26T08:38:04.4300394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 469, in torchbench_main 2024-06-26T08:38:04.4301279Z main(TorchBenchmarkRunner(), original_dir) 2024-06-26T08:38:04.4302146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3722, in main 2024-06-26T08:38:04.4303416Z process_entry(0, runner, original_dir, args) 2024-06-26T08:38:04.4304340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3654, in process_entry 2024-06-26T08:38:04.4307094Z return run(runner, args, original_dir) 2024-06-26T08:38:04.4308069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4283, in run 2024-06-26T08:38:04.4311850Z assert marked, f"nothing in example_inputs had a dim with {batch_size}" 2024-06-26T08:38:04.4312683Z AssertionError: nothing in example_inputs had a dim with 32 2024-06-26T08:38:05.0909378Z Run failed with return code: 1 2024-06-26T08:38:05.0909924Z Output: None 2024-06-26T08:38:05.0910227Z Error: None 2024-06-26T08:38:07.6320033Z 2024-06-26T08:38:08.6012478Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:38:08.6013010Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:38:08.6013442Z cuda train squeezenet1_1 2024-06-26T08:38:14.7971763Z pass 2024-06-26T08:38:14.7985919Z TIMING: entire_frame_compile:5.08244 backend_compile:4.12473 2024-06-26T08:38:14.7987591Z STATS: call_* op count: 70 | FakeTensor.__torch_dispatch__:359 | FakeTensorMode.__torch_dispatch__:6807 | ProxyTorchDispatchMode.__torch_dispatch__:1964 | attempt fast:8 | fast is_contiguous:8 2024-06-26T08:38:14.7988868Z Dynamo produced 2 graphs covering 70 ops with 6 graph breaks (5 unique) 2024-06-26T08:38:18.3319774Z 2024-06-26T08:38:19.4962306Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:38:19.4962743Z 2024-06-26T08:38:19.8025355Z Loading pipeline components...: 0% 0/6 [00:00 will be ignored 2024-06-26T08:39:49.3749165Z pass 2024-06-26T08:39:49.3798712Z TIMING: entire_frame_compile:53.84069 backend_compile:33.91236 2024-06-26T08:39:49.3801382Z STATS: call_* op count: 974 | FakeTensor.__torch_dispatch__:5293 | FakeTensorMode.__torch_dispatch__:71016 | attempt fast:1536 | fast is_contiguous:1536 | ProxyTorchDispatchMode.__torch_dispatch__:18598 2024-06-26T08:39:49.3802745Z Dynamo produced 3 graphs covering 974 ops with 7 graph breaks (5 unique) 2024-06-26T08:39:54.8092148Z 2024-06-26T08:39:57.3332290Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:39:57.3332782Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:39:57.3333215Z cuda train timm_regnet 2024-06-26T08:40:38.1014542Z pass 2024-06-26T08:40:38.1429585Z TIMING: entire_frame_compile:30.74596 backend_compile:20.8056 2024-06-26T08:40:38.1432598Z STATS: call_* op count: 463 | FakeTensor.__torch_dispatch__:897 | FakeTensorMode.__torch_dispatch__:44732 | attempt fast:1471 | fast is_contiguous:1471 | ProxyTorchDispatchMode.__torch_dispatch__:10183 2024-06-26T08:40:38.1434924Z Dynamo produced 2 graphs covering 463 ops with 6 graph breaks (5 unique) 2024-06-26T08:40:43.1232686Z 2024-06-26T08:40:44.9183314Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:40:44.9184301Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:40:44.9185158Z cuda train timm_resnest 2024-06-26T08:40:59.1869016Z W0626 08:40:59.186000 140089966293632 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:41:08.5969199Z pass 2024-06-26T08:41:08.6042802Z TIMING: entire_frame_compile:19.4551 backend_compile:12.37573 2024-06-26T08:41:08.6046072Z STATS: call_* op count: 431 | FakeTensor.__torch_dispatch__:2248 | FakeTensorMode.__torch_dispatch__:28537 | attempt fast:529 | fast is_contiguous:521 | ProxyTorchDispatchMode.__torch_dispatch__:7360 | slow no contiguity match:8 2024-06-26T08:41:08.6047570Z Dynamo produced 3 graphs covering 431 ops with 7 graph breaks (5 unique) 2024-06-26T08:41:12.6630527Z 2024-06-26T08:41:14.1208173Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:41:14.1208865Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:41:14.1209316Z cuda train timm_vision_transformer 2024-06-26T08:41:34.7454667Z pass 2024-06-26T08:41:34.7606264Z TIMING: entire_frame_compile:19.33743 backend_compile:13.72315 2024-06-26T08:41:34.7608762Z STATS: call_* op count: 381 | FakeTensor.__torch_dispatch__:1075 | FakeTensorMode.__torch_dispatch__:30492 | attempt fast:1591 | fast is_contiguous:1591 | ProxyTorchDispatchMode.__torch_dispatch__:7504 2024-06-26T08:41:34.7610316Z Dynamo produced 2 graphs covering 381 ops with 6 graph breaks (5 unique) 2024-06-26T08:41:39.2177385Z 2024-06-26T08:41:51.6383313Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:41:51.6384428Z loading model: 0it [00:12, ?it/s] 2024-06-26T08:41:51.6389874Z cuda train timm_vision_transformer_large 2024-06-26T08:41:51.6390520Z pass_due_to_skip 2024-06-26T08:41:51.9013312Z TIMING: 2024-06-26T08:41:51.9013735Z STATS: call_* op count: 0 2024-06-26T08:41:51.9014413Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T08:41:55.2800131Z 2024-06-26T08:41:57.0830340Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:41:57.0831036Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:41:57.0831744Z cuda train timm_vovnet 2024-06-26T08:42:18.6429651Z pass 2024-06-26T08:42:18.6600158Z TIMING: entire_frame_compile:15.36605 backend_compile:10.18412 2024-06-26T08:42:18.6604701Z STATS: call_* op count: 174 | FakeTensor.__torch_dispatch__:287 | FakeTensorMode.__torch_dispatch__:21352 | attempt fast:779 | fast is_contiguous:749 | ProxyTorchDispatchMode.__torch_dispatch__:4649 | slow no contiguity match:30 2024-06-26T08:42:18.6606988Z Dynamo produced 2 graphs covering 174 ops with 6 graph breaks (5 unique) 2024-06-26T08:42:22.7211078Z 2024-06-26T08:42:24.4939808Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:42:24.4940402Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:42:24.4940877Z cuda train torch_multimodal_clip 2024-06-26T08:43:00.4726519Z W0626 08:43:00.471000 139961082786432 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:43:30.1309974Z pass 2024-06-26T08:43:30.2610501Z TIMING: entire_frame_compile:60.45573 backend_compile:42.18399 2024-06-26T08:43:30.2613468Z STATS: call_* op count: 963 | FakeTensor.__torch_dispatch__:7957 | FakeTensorMode.__torch_dispatch__:105577 | attempt fast:1485 | fast is_contiguous:1430 | slow no contiguity match:55 | ProxyTorchDispatchMode.__torch_dispatch__:32713 2024-06-26T08:43:30.2615094Z Dynamo produced 3 graphs covering 963 ops with 7 graph breaks (5 unique) 2024-06-26T08:43:35.7836160Z 2024-06-26T08:43:36.3152895Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:43:36.3153367Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:43:36.3156607Z cuda train tts_angular 2024-06-26T08:43:37.1444017Z W0626 08:43:37.143000 140061885264512 torch/_logging/_internal.py:1040] [10/0] Profiler function will be ignored 2024-06-26T08:43:38.7050734Z pass 2024-06-26T08:43:38.7152072Z TIMING: entire_frame_compile:1.9343 backend_compile:0.99841 2024-06-26T08:43:38.7153372Z STATS: call_* op count: 71 | FakeTensorMode.__torch_dispatch__:2622 | FakeTensor.__torch_dispatch__:348 | ProxyTorchDispatchMode.__torch_dispatch__:825 2024-06-26T08:43:38.7154502Z Dynamo produced 5 graphs covering 71 ops with 9 graph breaks (6 unique) 2024-06-26T08:43:42.0590552Z 2024-06-26T08:43:44.2048941Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:43:44.2049430Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:43:44.2049854Z cuda train vgg16 2024-06-26T08:44:04.6254554Z pass 2024-06-26T08:44:04.6503551Z TIMING: entire_frame_compile:3.06382 backend_compile:2.43087 2024-06-26T08:44:04.6507239Z STATS: call_* op count: 44 | FakeTensor.__torch_dispatch__:230 | FakeTensorMode.__torch_dispatch__:4326 | attempt fast:24 | fast is_contiguous:24 | ProxyTorchDispatchMode.__torch_dispatch__:1238 2024-06-26T08:44:04.6508573Z Dynamo produced 2 graphs covering 44 ops with 6 graph breaks (5 unique) 2024-06-26T08:44:08.1446095Z 2024-06-26T08:44:11.2240764Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:44:11.2241361Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:44:11.2241880Z cuda train vision_maskrcnn 2024-06-26T08:44:36.2159785Z W0626 08:44:36.215000 140001053512320 torch/_dynamo/variables/tensor.py:714] [18/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T08:44:36.2163355Z W0626 08:44:36.215000 140001053512320 torch/_dynamo/variables/tensor.py:714] [18/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T08:44:36.2164663Z W0626 08:44:36.215000 140001053512320 torch/_dynamo/variables/tensor.py:714] [18/0] or: 2024-06-26T08:44:36.2165721Z W0626 08:44:36.215000 140001053512320 torch/_dynamo/variables/tensor.py:714] [18/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T08:44:36.2167014Z W0626 08:44:36.215000 140001053512320 torch/_dynamo/variables/tensor.py:714] [18/0] to include these operations in the captured graph. 2024-06-26T08:44:36.2168053Z W0626 08:44:36.215000 140001053512320 torch/_dynamo/variables/tensor.py:714] [18/0] 2024-06-26T08:45:33.1767246Z W0626 08:45:33.176000 140001053512320 torch/_dynamo/convert_frame.py:762] [30/8] torch._dynamo hit config.cache_size_limit (8) 2024-06-26T08:45:33.1769656Z W0626 08:45:33.176000 140001053512320 torch/_dynamo/convert_frame.py:762] [30/8] function: 'roi_align' (/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py:186) 2024-06-26T08:45:33.1772412Z W0626 08:45:33.176000 140001053512320 torch/_dynamo/convert_frame.py:762] [30/8] last reason: tensor 'L['boxes']' size mismatch at index 0. expected 648, actual 619 2024-06-26T08:45:33.1773892Z W0626 08:45:33.176000 140001053512320 torch/_dynamo/convert_frame.py:762] [30/8] To log all recompilation reasons, use TORCH_LOGS="recompiles". 2024-06-26T08:45:33.1775496Z W0626 08:45:33.176000 140001053512320 torch/_dynamo/convert_frame.py:762] [30/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html. 2024-06-26T08:46:04.5619820Z pass 2024-06-26T08:46:04.6283319Z TIMING: entire_frame_compile:104.77006 backend_compile:63.41141 2024-06-26T08:46:04.6286273Z STATS: call_* op count: 2515 | FakeTensorMode.__torch_dispatch__:129988 | FakeTensor.__torch_dispatch__:643 | ProxyTorchDispatchMode.__torch_dispatch__:41391 | attempt fast:1873 | slow no contiguity match:182 | fast is_contiguous:1331 | slow both tensors nontrivially broadcast:360 2024-06-26T08:46:04.6288008Z Dynamo produced 51 graphs covering 2515 ops with 33 graph breaks (8 unique) 2024-06-26T08:46:11.2843264Z 2024-06-26T08:46:13.9358022Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:46:13.9358478Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:46:13.9359950Z cuda train yolov3 2024-06-26T08:47:05.6310345Z W0626 08:47:05.630000 140519147065984 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:48:02.2142390Z E0626 08:48:02.213000 140519147065984 torch/_dynamo/utils.py:1470] RMSE (res-fp64): 0.04169, (ref-fp64): 0.01352 and shape=torch.Size([64]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-26T08:48:02.2144533Z E0626 08:48:02.213000 140519147065984 torch/_dynamo/utils.py:1384] Accuracy failed for key name module_list.1.BatchNorm2d.bias.grad 2024-06-26T08:48:02.2174675Z fail_accuracy 2024-06-26T08:48:02.2777801Z TIMING: entire_frame_compile:89.74688 backend_compile:60.31282 2024-06-26T08:48:02.2779224Z STATS: call_* op count: 1244 | FakeTensor.__torch_dispatch__:8356 | FakeTensorMode.__torch_dispatch__:134935 | attempt fast:4650 | fast is_contiguous:4644 | ProxyTorchDispatchMode.__torch_dispatch__:28065 | slow no contiguity match:6 2024-06-26T08:48:02.2780686Z Dynamo produced 4 graphs covering 1244 ops with 8 graph breaks (5 unique) 2024-06-26T08:48:06.9323480Z accuracy pass_rate=75.68% 2024-06-26T08:48:06.9327875Z calls_captured gmean=0.00x mean=421.757x 2024-06-26T08:48:06.9332575Z unique_graphs gmean=0.00x mean=3.649x 2024-06-26T08:48:06.9336893Z graph_breaks gmean=0.00x mean=6.135x 2024-06-26T08:48:06.9340766Z unique_graph_breaks gmean=0.00x mean=4.243x 2024-06-26T08:48:06.9344745Z autograd_captures gmean=0.00x mean=0.000x 2024-06-26T08:48:06.9348581Z autograd_compiles gmean=0.00x mean=0.000x 2024-06-26T08:48:06.9352599Z cudagraph_skips gmean=0.00x mean=0.000x 2024-06-26T08:48:07.4817924Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamic_aot_eager_torchbench_training.csv 2024-06-26T08:48:07.8111848Z llava XFAIL 2024-06-26T08:48:07.8116055Z maml_omniglot PASS 2024-06-26T08:48:07.8120567Z mnasnet1_0 PASS 2024-06-26T08:48:07.8125276Z mobilenet_v2 PASS 2024-06-26T08:48:07.8129868Z mobilenet_v2_quantized_qat XFAIL 2024-06-26T08:48:07.8134197Z mobilenet_v3_large PASS 2024-06-26T08:48:07.8138845Z moco PASS 2024-06-26T08:48:07.8143294Z nanogpt PASS 2024-06-26T08:48:07.8147706Z nvidia_deeprecommender PASS 2024-06-26T08:48:07.8152500Z opacus_cifar10 XFAIL 2024-06-26T08:48:07.8156487Z phlippe_densenet PASS 2024-06-26T08:48:07.8160892Z phlippe_resnet PASS 2024-06-26T08:48:07.8165228Z pytorch_CycleGAN_and_pix2pix PASS 2024-06-26T08:48:07.8169732Z pytorch_stargan PASS 2024-06-26T08:48:07.8174002Z pytorch_unet XFAIL 2024-06-26T08:48:07.8178665Z resnet152 PASS 2024-06-26T08:48:07.8183196Z resnet18 PASS 2024-06-26T08:48:07.8187489Z resnet50 PASS 2024-06-26T08:48:07.8191878Z resnet50_quantized_qat XFAIL 2024-06-26T08:48:07.8196213Z resnext50_32x4d PASS 2024-06-26T08:48:07.8200728Z sam XFAIL 2024-06-26T08:48:07.8205309Z shufflenet_v2_x1_0 PASS 2024-06-26T08:48:07.8209859Z soft_actor_critic PASS 2024-06-26T08:48:07.8213914Z squeezenet1_1 PASS 2024-06-26T08:48:07.8218561Z stable_diffusion_text_encoder PASS 2024-06-26T08:48:07.8222845Z stable_diffusion_unet XFAIL 2024-06-26T08:48:07.8227302Z timm_efficientnet PASS 2024-06-26T08:48:07.8231545Z timm_regnet PASS 2024-06-26T08:48:07.8235972Z timm_resnest PASS 2024-06-26T08:48:07.8240288Z timm_vision_transformer PASS 2024-06-26T08:48:07.8244799Z timm_vision_transformer_large XFAIL 2024-06-26T08:48:07.8249317Z timm_vovnet PASS 2024-06-26T08:48:07.8253668Z torch_multimodal_clip PASS 2024-06-26T08:48:07.8258135Z tts_angular PASS 2024-06-26T08:48:07.8262692Z vgg16 PASS 2024-06-26T08:48:07.8266975Z vision_maskrcnn PASS 2024-06-26T08:48:07.8271393Z yolov3 XFAIL 2024-06-26T08:48:07.8729005Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamic_aot_eager_torchbench_training.csv 2024-06-26T08:48:08.1845081Z llava PASS 2024-06-26T08:48:08.1848840Z maml_omniglot PASS 2024-06-26T08:48:08.1853856Z mnasnet1_0 PASS 2024-06-26T08:48:08.1859049Z mobilenet_v2 PASS 2024-06-26T08:48:08.1863958Z mobilenet_v2_quantized_qat PASS 2024-06-26T08:48:08.1868891Z mobilenet_v3_large PASS 2024-06-26T08:48:08.1873749Z moco PASS 2024-06-26T08:48:08.1878633Z nanogpt PASS 2024-06-26T08:48:08.1883227Z nvidia_deeprecommender PASS 2024-06-26T08:48:08.1888836Z opacus_cifar10 PASS 2024-06-26T08:48:08.1893389Z phlippe_densenet PASS 2024-06-26T08:48:08.1898553Z phlippe_resnet PASS 2024-06-26T08:48:08.1903422Z pytorch_CycleGAN_and_pix2pix PASS 2024-06-26T08:48:08.1908079Z pytorch_stargan PASS 2024-06-26T08:48:08.1913095Z pytorch_unet PASS 2024-06-26T08:48:08.1917751Z resnet152 PASS 2024-06-26T08:48:08.1922761Z resnet18 PASS 2024-06-26T08:48:08.1927591Z resnet50 PASS 2024-06-26T08:48:08.1932126Z resnet50_quantized_qat PASS 2024-06-26T08:48:08.1936969Z resnext50_32x4d PASS 2024-06-26T08:48:08.1941709Z sam PASS 2024-06-26T08:48:08.1946428Z shufflenet_v2_x1_0 PASS 2024-06-26T08:48:08.1951014Z soft_actor_critic PASS 2024-06-26T08:48:08.1955753Z squeezenet1_1 PASS 2024-06-26T08:48:08.1960489Z stable_diffusion_text_encoder PASS 2024-06-26T08:48:08.1965362Z stable_diffusion_unet PASS 2024-06-26T08:48:08.1970012Z timm_efficientnet PASS 2024-06-26T08:48:08.1974929Z timm_regnet PASS 2024-06-26T08:48:08.1979825Z timm_resnest PASS 2024-06-26T08:48:08.1984492Z timm_vision_transformer PASS 2024-06-26T08:48:08.1989131Z timm_vision_transformer_large PASS 2024-06-26T08:48:08.1993905Z timm_vovnet PASS 2024-06-26T08:48:08.1999399Z torch_multimodal_clip PASS 2024-06-26T08:48:08.2003840Z tts_angular PASS 2024-06-26T08:48:08.2009227Z vgg16 PASS 2024-06-26T08:48:08.2014112Z vision_maskrcnn PASS 2024-06-26T08:48:08.2019670Z yolov3 PASS 2024-06-26T08:48:08.2480830Z + cleanup_workspace 2024-06-26T08:48:08.2481778Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-26T08:48:08.2493071Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-26T08:48:08.2494311Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-26T08:48:08.2494963Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T08:48:08.2495763Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-26T08:48:08.2496754Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-26T08:48:08.2497446Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-26T08:48:10.4874295Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T08:48:10.4874942Z cat test/**/*_toprint.log || true 2024-06-26T08:48:10.4883031Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:48:10.4883536Z env: 2024-06-26T08:48:10.4884049Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:48:10.4884510Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:48:10.4885250Z DOCKER_CONTAINER_ID: ba5a631ec45646ea607916b9626442abdafeeff36170f13db63963abace10ca4 2024-06-26T08:48:10.4885893Z ##[endgroup] 2024-06-26T08:48:10.4947413Z cat: test/**/*_toprint.log: No such file or directory 2024-06-26T08:48:10.4973940Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:48:10.4974407Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:48:10.4981397Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:48:10.4981895Z env: 2024-06-26T08:48:10.4982164Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:48:10.4982614Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:48:10.4983344Z DOCKER_CONTAINER_ID: ba5a631ec45646ea607916b9626442abdafeeff36170f13db63963abace10ca4 2024-06-26T08:48:10.4984012Z MONITOR_SCRIPT_PID: 16077 2024-06-26T08:48:10.4984355Z ##[endgroup] 2024-06-26T08:48:10.5168776Z Prepare all required actions 2024-06-26T08:48:10.5169216Z Getting action download info 2024-06-26T08:48:10.7101669Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T08:48:10.8744580Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T08:48:10.8745036Z with: 2024-06-26T08:48:10.8745600Z file-suffix: test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688917069 2024-06-26T08:48:10.8746298Z s3-bucket: gha-artifacts 2024-06-26T08:48:10.8746854Z env: 2024-06-26T08:48:10.8747118Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:48:10.8747565Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:48:10.8748293Z DOCKER_CONTAINER_ID: ba5a631ec45646ea607916b9626442abdafeeff36170f13db63963abace10ca4 2024-06-26T08:48:10.8748939Z ##[endgroup] 2024-06-26T08:48:10.8773766Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-26T08:48:10.8774405Z # Remove any previous test jsons if they exist 2024-06-26T08:48:10.8775064Z rm -f test-jsons-*.zip 2024-06-26T08:48:10.8775577Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-26T08:48:10.8783656Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:48:10.8784151Z env: 2024-06-26T08:48:10.8784428Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:48:10.8784870Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:48:10.8785841Z DOCKER_CONTAINER_ID: ba5a631ec45646ea607916b9626442abdafeeff36170f13db63963abace10ca4 2024-06-26T08:48:10.8786770Z FILE_SUFFIX: test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688917069 2024-06-26T08:48:10.8787448Z ##[endgroup] 2024-06-26T08:48:10.8965108Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-26T08:48:10.8995124Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-26T08:48:10.8996657Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-26T08:48:10.9002896Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-26T08:48:10.9046317Z ##[group]Run # Remove any previous test reports if they exist 2024-06-26T08:48:10.9046958Z # Remove any previous test reports if they exist 2024-06-26T08:48:10.9047489Z rm -f test-reports-*.zip 2024-06-26T08:48:10.9048077Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-26T08:48:10.9055713Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:48:10.9056201Z env: 2024-06-26T08:48:10.9056483Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:48:10.9057087Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:48:10.9057820Z DOCKER_CONTAINER_ID: ba5a631ec45646ea607916b9626442abdafeeff36170f13db63963abace10ca4 2024-06-26T08:48:10.9058765Z FILE_SUFFIX: test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688917069 2024-06-26T08:48:10.9059563Z ##[endgroup] 2024-06-26T08:48:10.9211825Z adding: test/test-reports/inference_torchbench.csv (deflated 69%) 2024-06-26T08:48:10.9214063Z adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 93%) 2024-06-26T08:48:10.9215552Z adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 74%) 2024-06-26T08:48:10.9216895Z adding: test/test-reports/training_torchbench.csv (deflated 67%) 2024-06-26T08:48:10.9223104Z adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 96%) 2024-06-26T08:48:10.9224480Z adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 80%) 2024-06-26T08:48:10.9249960Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-26T08:48:10.9250845Z # Remove any previous usage logs if they exist 2024-06-26T08:48:10.9251338Z rm -f logs-*.zip 2024-06-26T08:48:10.9252004Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-26T08:48:10.9252883Z # so check to see if the file exists first 2024-06-26T08:48:10.9253377Z if [ -f 'usage_log.txt' ]; then 2024-06-26T08:48:10.9253897Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-26T08:48:10.9254380Z fi 2024-06-26T08:48:10.9254736Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-26T08:48:10.9255298Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-26T08:48:10.9255780Z fi 2024-06-26T08:48:10.9263195Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:48:10.9263692Z env: 2024-06-26T08:48:10.9263968Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:48:10.9264422Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:48:10.9265158Z DOCKER_CONTAINER_ID: ba5a631ec45646ea607916b9626442abdafeeff36170f13db63963abace10ca4 2024-06-26T08:48:10.9266099Z FILE_SUFFIX: test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688917069 2024-06-26T08:48:10.9266774Z ##[endgroup] 2024-06-26T08:48:10.9346656Z adding: usage_log.txt (deflated 92%) 2024-06-26T08:48:10.9454278Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-26T08:48:10.9454984Z # Remove any previous debugging artifacts if they exist 2024-06-26T08:48:10.9455524Z rm -f debug-*.zip 2024-06-26T08:48:10.9455899Z if [ -d 'test/debug' ]; then 2024-06-26T08:48:10.9456393Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-26T08:48:10.9457006Z fi 2024-06-26T08:48:10.9463981Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:48:10.9464474Z env: 2024-06-26T08:48:10.9464749Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:48:10.9465193Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:48:10.9465930Z DOCKER_CONTAINER_ID: ba5a631ec45646ea607916b9626442abdafeeff36170f13db63963abace10ca4 2024-06-26T08:48:10.9466875Z FILE_SUFFIX: test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688917069 2024-06-26T08:48:10.9467567Z ##[endgroup] 2024-06-26T08:48:10.9542950Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:48:10.9543384Z with: 2024-06-26T08:48:10.9543656Z s3-bucket: gha-artifacts 2024-06-26T08:48:10.9544084Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:48:10.9544550Z retention-days: 14 2024-06-26T08:48:10.9544884Z if-no-files-found: warn 2024-06-26T08:48:10.9545232Z path: test-jsons-*.zip 2024-06-26T08:48:10.9545570Z name: artifact 2024-06-26T08:48:10.9545869Z region: us-east-1 2024-06-26T08:48:10.9546164Z env: 2024-06-26T08:48:10.9546432Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:48:10.9546885Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:48:10.9547624Z DOCKER_CONTAINER_ID: ba5a631ec45646ea607916b9626442abdafeeff36170f13db63963abace10ca4 2024-06-26T08:48:10.9548266Z ##[endgroup] 2024-06-26T08:48:11.2734567Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:48:11.2735326Z With the provided path, there will be 1 file uploaded 2024-06-26T08:48:11.2735964Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:48:11.2761826Z Starting upload of test-jsons-test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688917069.zip 2024-06-26T08:48:11.3752061Z Finished upload of test-jsons-test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688917069.zip 2024-06-26T08:48:11.3892114Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:48:11.3892550Z with: 2024-06-26T08:48:11.3892830Z s3-bucket: gha-artifacts 2024-06-26T08:48:11.3893257Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:48:11.3893724Z retention-days: 14 2024-06-26T08:48:11.3894053Z if-no-files-found: error 2024-06-26T08:48:11.3894415Z path: test-reports-*.zip 2024-06-26T08:48:11.3894764Z name: artifact 2024-06-26T08:48:11.3895057Z region: us-east-1 2024-06-26T08:48:11.3895367Z env: 2024-06-26T08:48:11.3895637Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:48:11.3896205Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:48:11.3897080Z DOCKER_CONTAINER_ID: ba5a631ec45646ea607916b9626442abdafeeff36170f13db63963abace10ca4 2024-06-26T08:48:11.3897727Z ##[endgroup] 2024-06-26T08:48:11.6998652Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:48:11.6999695Z With the provided path, there will be 1 file uploaded 2024-06-26T08:48:11.7000558Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:48:11.7026497Z Starting upload of test-reports-test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688917069.zip 2024-06-26T08:48:11.8162630Z Finished upload of test-reports-test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688917069.zip 2024-06-26T08:48:11.8302877Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:48:11.8303317Z with: 2024-06-26T08:48:11.8303612Z s3-bucket: gha-artifacts 2024-06-26T08:48:11.8304035Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:48:11.8304500Z retention-days: 14 2024-06-26T08:48:11.8304844Z if-no-files-found: ignore 2024-06-26T08:48:11.8305200Z path: logs-*.zip 2024-06-26T08:48:11.8305507Z name: artifact 2024-06-26T08:48:11.8305801Z region: us-east-1 2024-06-26T08:48:11.8306097Z env: 2024-06-26T08:48:11.8319983Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:48:11.8320449Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:48:11.8321178Z DOCKER_CONTAINER_ID: ba5a631ec45646ea607916b9626442abdafeeff36170f13db63963abace10ca4 2024-06-26T08:48:11.8321821Z ##[endgroup] 2024-06-26T08:48:12.1393569Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:48:12.1394269Z With the provided path, there will be 1 file uploaded 2024-06-26T08:48:12.1394888Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:48:12.1420662Z Starting upload of logs-test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688917069.zip 2024-06-26T08:48:12.2991371Z Finished upload of logs-test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688917069.zip 2024-06-26T08:48:12.3129110Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:48:12.3129553Z with: 2024-06-26T08:48:12.3129838Z s3-bucket: gha-artifacts 2024-06-26T08:48:12.3130297Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:48:12.3130800Z retention-days: 14 2024-06-26T08:48:12.3131127Z if-no-files-found: ignore 2024-06-26T08:48:12.3131484Z path: debug-*.zip 2024-06-26T08:48:12.3131791Z name: artifact 2024-06-26T08:48:12.3132091Z region: us-east-1 2024-06-26T08:48:12.3132374Z env: 2024-06-26T08:48:12.3132644Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:48:12.3133093Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:48:12.3133823Z DOCKER_CONTAINER_ID: ba5a631ec45646ea607916b9626442abdafeeff36170f13db63963abace10ca4 2024-06-26T08:48:12.3134452Z ##[endgroup] 2024-06-26T08:48:12.6172433Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-26T08:48:12.6309962Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T08:48:12.6310455Z # shellcheck disable=SC2156 2024-06-26T08:48:12.6311252Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T08:48:12.6319134Z shell: /usr/bin/bash -e {0} 2024-06-26T08:48:12.6319489Z env: 2024-06-26T08:48:12.6319765Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:48:12.6320229Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:48:12.6320963Z DOCKER_CONTAINER_ID: ba5a631ec45646ea607916b9626442abdafeeff36170f13db63963abace10ca4 2024-06-26T08:48:12.6321603Z ##[endgroup] 2024-06-26T08:48:12.7978469Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-26T08:48:12.7979081Z with: 2024-06-26T08:48:12.7979342Z env: 2024-06-26T08:48:12.7979628Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:48:12.7980088Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:48:12.7980826Z DOCKER_CONTAINER_ID: ba5a631ec45646ea607916b9626442abdafeeff36170f13db63963abace10ca4 2024-06-26T08:48:12.7981465Z ##[endgroup] 2024-06-26T08:48:12.7998445Z ##[group]Run set -eou pipefail 2024-06-26T08:48:12.7998845Z set -eou pipefail 2024-06-26T08:48:12.7999187Z  2024-06-26T08:48:12.7999684Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-26T08:48:12.8000313Z for _ in $(seq 1440); do 2024-06-26T08:48:12.8000775Z  # Break if no ssh session exists anymore 2024-06-26T08:48:12.8001301Z  if [ "$(who)" = "" ]; then 2024-06-26T08:48:12.8001802Z  break 2024-06-26T08:48:12.8002143Z  fi 2024-06-26T08:48:12.8002433Z  echo "." 2024-06-26T08:48:12.8002758Z  sleep 5 2024-06-26T08:48:12.8003068Z done 2024-06-26T08:48:12.8010630Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:48:12.8011131Z env: 2024-06-26T08:48:12.8011408Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:48:12.8011856Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:48:12.8012581Z DOCKER_CONTAINER_ID: ba5a631ec45646ea607916b9626442abdafeeff36170f13db63963abace10ca4 2024-06-26T08:48:12.8013217Z ##[endgroup] 2024-06-26T08:48:12.8032507Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-26T08:48:12.8069483Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:48:12.8070487Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:48:12.8071076Z # shellcheck disable=SC2046 2024-06-26T08:48:12.8071553Z docker stop $(docker ps -q) || true 2024-06-26T08:48:12.8072023Z # Prune all of the docker images 2024-06-26T08:48:12.8072466Z docker system prune -af 2024-06-26T08:48:12.8079584Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:48:12.8080142Z env: 2024-06-26T08:48:12.8080412Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:48:12.8080863Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:48:12.8081596Z DOCKER_CONTAINER_ID: ba5a631ec45646ea607916b9626442abdafeeff36170f13db63963abace10ca4 2024-06-26T08:48:12.8082237Z ##[endgroup] 2024-06-26T08:48:13.1706745Z ba5a631ec456 2024-06-26T08:48:15.4772197Z Deleted Containers: 2024-06-26T08:48:15.4772823Z ba5a631ec45646ea607916b9626442abdafeeff36170f13db63963abace10ca4 2024-06-26T08:48:15.4773397Z 2024-06-26T08:48:19.3875857Z Deleted Images: 2024-06-26T08:48:19.3878834Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T08:48:19.3881754Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T08:48:19.3883500Z deleted: sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd 2024-06-26T08:48:19.3884635Z deleted: sha256:754ead7012f0b7691885a9660f02e80be6cee8c5665486fb995c1fb06adb6e92 2024-06-26T08:48:19.3885475Z deleted: sha256:fbe4496d8830e11be86d8f222522e7956f32f85bfc06b05ea53961f34a58af74 2024-06-26T08:48:19.3886305Z deleted: sha256:1c83814aa1a1a7e0cc998e3479bdab8e8766b975560cbf00690acf39ed899a80 2024-06-26T08:48:19.3887150Z deleted: sha256:e9ea8e61942c7f8ae4aab97b8d102b01d6bb4b6cf2a605808cd34130a17d8a0f 2024-06-26T08:48:19.3888009Z deleted: sha256:acc5fc32588cd95e082a6ce35e48d0f303ae8f96e3ab65c4e375b85e1f3c4a92 2024-06-26T08:48:19.3888846Z deleted: sha256:a93e53400904d29e4185d3be942ea34f19572ea438962788ae4e9b3d9bf4c498 2024-06-26T08:48:19.3889783Z deleted: sha256:c97092b88b5e4c77c68bbc4b51afab64e38719af1ff58013ec49a696cdd40a6e 2024-06-26T08:48:19.3890604Z deleted: sha256:b132889c90d67ffa53655846603e3fcc2372c23f04dbdbc54690a39d304eb89b 2024-06-26T08:48:19.3891478Z deleted: sha256:915d54081ed607710624abe3f2490114c745858831f4fc13396bd3b4b9a121bc 2024-06-26T08:48:19.3892334Z deleted: sha256:bfa8d446c498c87f535aef102695564de1b22c755e6c63944ae2f5073a2b31ec 2024-06-26T08:48:19.3893167Z deleted: sha256:e71460fb363c29e70bbe357d680aaa7ae694f094ef2f2713609158ad90611744 2024-06-26T08:48:19.3893990Z deleted: sha256:84079bb1b4ee1d0119ae3d3e701de460a1324912297c675425c9a36d30fcdb0f 2024-06-26T08:48:19.3894811Z deleted: sha256:d86ac3ca148d4758cb03b493fdd49109d8f1a2c7c85c4b755493c3c4529553e5 2024-06-26T08:48:19.3895645Z deleted: sha256:532b3ab97c44a673a0d5f62ee46fa6a83248a84de8877754a612355be56ed0a3 2024-06-26T08:48:19.3896465Z deleted: sha256:617e2a35f25ae24500c88c0c9b8475f0687fbcbed3d200013b4c99c6d8941aff 2024-06-26T08:48:19.3897452Z deleted: sha256:b88120bcd5a26a660c09c5df13176e0ef098fb2c8cdb57d74ab0796a3d1194d6 2024-06-26T08:48:19.3898273Z deleted: sha256:c016b2e7764735243205fb76fa9a273bb3dd4bf05532288e3a613ce022131ac3 2024-06-26T08:48:19.3899099Z deleted: sha256:bcaca462c390deddd2995de2d245a7495e7d2e1e94f0edad30a573c83d0588fd 2024-06-26T08:48:19.3899938Z deleted: sha256:9e260eba003d5c25d7b410e5c4c3093845d9e46611896840901c689d7fbf8ee6 2024-06-26T08:48:19.3900744Z deleted: sha256:3e62b86f8b911a9b59c62d55f063e71683ae6a01c22418db5d27930712adaad6 2024-06-26T08:48:19.3901552Z deleted: sha256:905920ccb1d88449b8dde686c34a4b2a8b6f59e78ac44916b189ea3593b9fbbb 2024-06-26T08:48:19.3902358Z deleted: sha256:961201237e03acd5061406238dd7a3c1343ac888d7583212cd0ab5ae1b4f0540 2024-06-26T08:48:19.3903179Z deleted: sha256:70047a2aac62159fa28fb4dbd4567ec2ffbab68fe046967e09acb0eb3e5cb9cb 2024-06-26T08:48:19.3904017Z deleted: sha256:1ec680ad781436ae65fa987f5fa1ed032ecd5c82a6e35eea6707de501f62c0f4 2024-06-26T08:48:19.3904860Z deleted: sha256:ef859623a50e85abf6e6c1dbd6188aa765a66cc7b2bd618a1cbe4825b7b2bd91 2024-06-26T08:48:19.3905700Z deleted: sha256:404ade5cacff59773b93ab438bfa64cd3a171d07f6f168bdc4a5d4bfa0b43526 2024-06-26T08:48:19.3906530Z deleted: sha256:5b27de6ee3d9f2f18217bf18dec725f7b4954c0e48c85d35752634c22d62209b 2024-06-26T08:48:19.3907364Z deleted: sha256:498e39c575b3fa069acaac7e46b91b4e54ae3a05f2ada5c4347091f346f25d4c 2024-06-26T08:48:19.3908173Z deleted: sha256:1405919d2694d65669b1f4c842282021afe89208ce86570e96464903733da2d0 2024-06-26T08:48:19.3908969Z deleted: sha256:d5b41e5d70896c0472fc3e6da0516a17d89cd6635dc75010ecddc31ce9690fad 2024-06-26T08:48:19.3909783Z deleted: sha256:18e9379b62b9d0bb40293d28a652eafaee66a499f66f0776bf14e37a513101b7 2024-06-26T08:48:19.3910614Z deleted: sha256:a781f80ed00ff2926a699fd01a309bb0b4fb7d5655eb652c205bb9c7f811035f 2024-06-26T08:48:19.3911432Z deleted: sha256:c0430973b7cf759075330ee29c79bfd1312c91e6033d9dd883b4b1c29743a171 2024-06-26T08:48:19.3912254Z deleted: sha256:e733b3d66efb9342e5a283376c0e1ddffa47a42fecc26e61ecaf0a3472885d97 2024-06-26T08:48:19.3913075Z deleted: sha256:8929c0b2047f74b84c82c39eff9077a92ae1a828cde00c9ab78fb3e0ed5e0283 2024-06-26T08:48:19.3913894Z deleted: sha256:f5b56155fb58a0a60fe6391d3581cf91610ed69d4f0a321257c19944da296a52 2024-06-26T08:48:19.3914807Z deleted: sha256:b54d2cb355238d1aa91e58df93064447b80f53706d1ca62823acc834f5ac54fe 2024-06-26T08:48:19.3915617Z deleted: sha256:e7311285af9eeddb51aa5629aa98386e89f1456ff32843b6aee6443247edf66c 2024-06-26T08:48:19.3916438Z deleted: sha256:67657ceec347da8dc8085d5731633efeac115ba677fa4b7c5966c4260f7abab0 2024-06-26T08:48:19.3917268Z deleted: sha256:b496e50409dbf414930fc9b4437aa0aa4ce27bc7840e050ae459aaffcf9aca86 2024-06-26T08:48:19.3918090Z deleted: sha256:8f5276170cc4a60d36ad7b964a502306b29546254cedcbcb101ea9123497d2c0 2024-06-26T08:48:19.3918898Z deleted: sha256:53d524d1564745e96a7e515462171b73184d9f1ff5454b32fcfbbca39d361467 2024-06-26T08:48:19.3919723Z deleted: sha256:2acbadb00d69b6aca575f04baba4ea26ae2b5c2d85f09945004ad93a78a8eb75 2024-06-26T08:48:19.3920629Z deleted: sha256:02665efeff878aeb603c1edb2e21cb700c171c025d64f5aae1b2b20b12f3964a 2024-06-26T08:48:19.3921471Z deleted: sha256:19bd23e9a51dfe883a8b8d2ff6d66ff1b3476b8912856406bb79b81762aa6d4b 2024-06-26T08:48:19.3922307Z deleted: sha256:73aa8637bf9299d6298dabd858480c274a62117388f2ca1d5e9eb15c6d1e4155 2024-06-26T08:48:19.3923117Z deleted: sha256:5f8968406305f348b1caebc0e035187d6e9965ace7b9d5f2f21114d37c9fe2e1 2024-06-26T08:48:19.3924136Z deleted: sha256:d827104914db0c503cf5c81f13e2456bf4efce42d4cf1f8d5d70838207278bee 2024-06-26T08:48:19.3924965Z deleted: sha256:867b219724f2f1aaf56f42e3e9f03a5d64bdca8a6a72407efd10328aeea5d510 2024-06-26T08:48:19.3925802Z deleted: sha256:b9ff2a7cceb9237b26bcb17728edf8a558eec5a76f63626e84f80541dfc0f1d4 2024-06-26T08:48:19.3926627Z deleted: sha256:be73f509106819007ce2b0f5a1301ac56ae583cb6c1fb69bee93efa0840368f1 2024-06-26T08:48:19.3927455Z deleted: sha256:2a11c07c915c3fdbb4e73096dea73ea950c970d50fe1a5764060bd62b7b7cf2c 2024-06-26T08:48:19.3928283Z deleted: sha256:5e9989622e2abc1c72f8184343507b45d75e4ff971edb68b96cbcd75ae484ff5 2024-06-26T08:48:19.3929116Z deleted: sha256:05ce642bd9cf5a9205bff107df28b1d6cef7d5f6ab3ea944d28c032ec61270df 2024-06-26T08:48:19.3929929Z deleted: sha256:017702617f6c07b73f2088678f4710b794cfe897a213a5cf122a0e2519eae0ac 2024-06-26T08:48:19.3930738Z deleted: sha256:44b3b37bb1ca39dc5cf356de41a23a5c12f605a1992813e408087fb2b0ad6d1c 2024-06-26T08:48:19.3931564Z deleted: sha256:8c387982e9f27db357a1d149dc149cabc0bd04668ad6956387a5072682dada67 2024-06-26T08:48:19.3932434Z deleted: sha256:10d415dca4e3fbb88a747b40317f3c6333c041ba8f73855f67fb9fe981fae5ee 2024-06-26T08:48:19.3933253Z deleted: sha256:6c65469bf0f53dcd625faefe363b8f3136c08388a6cf64d9158e8576e4ee89b1 2024-06-26T08:48:19.3934093Z deleted: sha256:027db5d8b76deeefbd8c34b485ab7ff14ed2a3e6800425c1ce0d77c2e9fdd4ab 2024-06-26T08:48:19.3934932Z deleted: sha256:24941234ad28cf5c8174d7cf18326c10b4e17f2c2137672ea0322df461597dd7 2024-06-26T08:48:19.3935765Z deleted: sha256:240ea2eebfa8519d6c2d88f4c99e66ce4e038d000f93ce8d03726c01d81180ef 2024-06-26T08:48:19.3936679Z deleted: sha256:09e823cb391a69f6a195e3ca8a41624b5f8b34eee14c6eb36277de814a91966e 2024-06-26T08:48:19.3937475Z deleted: sha256:730479d695169a6593bf0ac16b377491dce65175a6d158e012422367d04577fd 2024-06-26T08:48:19.3938294Z deleted: sha256:b0a30f9f7bd5459ff662453b9346a7ba8ea8db1191528345c1953eab4aa2f54e 2024-06-26T08:48:19.3939130Z deleted: sha256:75cefa87effaa3ad361c74642839923101f0f9982f814f3c9e2ed4ad52f2a008 2024-06-26T08:48:19.3939953Z deleted: sha256:6f97db87ba9512e615e871c6982000602982e1e5a5be334a56e7292251e84e00 2024-06-26T08:48:19.3940773Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-06-26T08:48:19.3941600Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-06-26T08:48:19.3942417Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-06-26T08:48:19.3943241Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-06-26T08:48:19.3944073Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-06-26T08:48:19.3944894Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-06-26T08:48:19.3945843Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-06-26T08:48:19.3946679Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-06-26T08:48:19.3947514Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-06-26T08:48:19.3948348Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-06-26T08:48:19.3949167Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-06-26T08:48:19.3949659Z 2024-06-26T08:48:19.3952720Z Total reclaimed space: 41.17GB 2024-06-26T08:48:19.4026373Z Post job cleanup. 2024-06-26T08:48:19.4074051Z Post job cleanup. 2024-06-26T08:48:19.4893783Z [command]/usr/bin/git version 2024-06-26T08:48:19.4937254Z git version 2.40.1 2024-06-26T08:48:19.4980375Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/f155ff37-a7ba-4201-ae7b-ce4e4f6c825a' before making global git config changes 2024-06-26T08:48:19.4982290Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T08:48:19.4998802Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T08:48:19.5022606Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T08:48:19.5052280Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T08:48:19.5271586Z Entering 'android/libs/fbjni' 2024-06-26T08:48:19.5309304Z Entering 'third_party/FP16' 2024-06-26T08:48:19.5345315Z Entering 'third_party/FXdiv' 2024-06-26T08:48:19.5381468Z Entering 'third_party/NNPACK' 2024-06-26T08:48:19.5418655Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:48:19.5454511Z Entering 'third_party/XNNPACK' 2024-06-26T08:48:19.5501840Z Entering 'third_party/benchmark' 2024-06-26T08:48:19.5537453Z Entering 'third_party/cpp-httplib' 2024-06-26T08:48:19.5573639Z Entering 'third_party/cpuinfo' 2024-06-26T08:48:19.5609116Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:48:19.5647627Z Entering 'third_party/cutlass' 2024-06-26T08:48:19.5688493Z Entering 'third_party/eigen' 2024-06-26T08:48:19.5726611Z Entering 'third_party/fbgemm' 2024-06-26T08:48:19.5766062Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:48:19.5801065Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:48:19.5838031Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:48:19.5877956Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:48:19.5912696Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:48:19.5949217Z Entering 'third_party/flatbuffers' 2024-06-26T08:48:19.5989486Z Entering 'third_party/fmt' 2024-06-26T08:48:19.6027637Z Entering 'third_party/foxi' 2024-06-26T08:48:19.6064002Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:48:19.6102510Z Entering 'third_party/gloo' 2024-06-26T08:48:19.6140751Z Entering 'third_party/googletest' 2024-06-26T08:48:19.6178687Z Entering 'third_party/ideep' 2024-06-26T08:48:19.6212698Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:48:19.6253548Z Entering 'third_party/ittapi' 2024-06-26T08:48:19.6290306Z Entering 'third_party/kineto' 2024-06-26T08:48:19.6325785Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:48:19.6360145Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:48:19.6396090Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:48:19.6430867Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:48:19.6465394Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:48:19.6499245Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:48:19.6533905Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:48:19.6569113Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:48:19.6604751Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:48:19.6640440Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:48:19.6676726Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:48:19.6710933Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:48:19.6746950Z Entering 'third_party/mimalloc' 2024-06-26T08:48:19.6784410Z Entering 'third_party/nccl/nccl' 2024-06-26T08:48:19.6818383Z Entering 'third_party/nlohmann' 2024-06-26T08:48:19.6855396Z Entering 'third_party/onnx' 2024-06-26T08:48:19.6903845Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:48:19.6939212Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:48:19.6975069Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:48:19.7012417Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:48:19.7046882Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:48:19.7079544Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:48:19.7116171Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:48:19.7152099Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:48:19.7188772Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:48:19.7222624Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:48:19.7257247Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:48:19.7292893Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:48:19.7328270Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:48:19.7379302Z Entering 'third_party/pocketfft' 2024-06-26T08:48:19.7414259Z Entering 'third_party/protobuf' 2024-06-26T08:48:19.7450939Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:48:19.7484604Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:48:19.7520377Z Entering 'third_party/psimd' 2024-06-26T08:48:19.7553400Z Entering 'third_party/pthreadpool' 2024-06-26T08:48:19.7591418Z Entering 'third_party/pybind11' 2024-06-26T08:48:19.7625638Z Entering 'third_party/python-peachpy' 2024-06-26T08:48:19.7660719Z Entering 'third_party/sleef' 2024-06-26T08:48:19.7695912Z Entering 'third_party/tensorpipe' 2024-06-26T08:48:19.7729758Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:48:19.7765261Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:48:19.7797945Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:48:19.7834415Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:48:19.7869315Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:48:19.7919978Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T08:48:19.7942886Z http.https://github.com/.extraheader 2024-06-26T08:48:19.7952157Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T08:48:19.7980309Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T08:48:19.8188269Z Entering 'android/libs/fbjni' 2024-06-26T08:48:19.8210208Z http.https://github.com/.extraheader 2024-06-26T08:48:19.8235192Z Entering 'third_party/FP16' 2024-06-26T08:48:19.8258503Z http.https://github.com/.extraheader 2024-06-26T08:48:19.8282378Z Entering 'third_party/FXdiv' 2024-06-26T08:48:19.8304122Z http.https://github.com/.extraheader 2024-06-26T08:48:19.8326253Z Entering 'third_party/NNPACK' 2024-06-26T08:48:19.8348027Z http.https://github.com/.extraheader 2024-06-26T08:48:19.8370697Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:48:19.8392694Z http.https://github.com/.extraheader 2024-06-26T08:48:19.8414999Z Entering 'third_party/XNNPACK' 2024-06-26T08:48:19.8437457Z http.https://github.com/.extraheader 2024-06-26T08:48:19.8471418Z Entering 'third_party/benchmark' 2024-06-26T08:48:19.8494409Z http.https://github.com/.extraheader 2024-06-26T08:48:19.8517822Z Entering 'third_party/cpp-httplib' 2024-06-26T08:48:19.8539784Z http.https://github.com/.extraheader 2024-06-26T08:48:19.8563294Z Entering 'third_party/cpuinfo' 2024-06-26T08:48:19.8585787Z http.https://github.com/.extraheader 2024-06-26T08:48:19.8609084Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:48:19.8630445Z http.https://github.com/.extraheader 2024-06-26T08:48:19.8651746Z Entering 'third_party/cutlass' 2024-06-26T08:48:19.8675408Z http.https://github.com/.extraheader 2024-06-26T08:48:19.8702992Z Entering 'third_party/eigen' 2024-06-26T08:48:19.8725429Z http.https://github.com/.extraheader 2024-06-26T08:48:19.8749780Z Entering 'third_party/fbgemm' 2024-06-26T08:48:19.8771958Z http.https://github.com/.extraheader 2024-06-26T08:48:19.8795231Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:48:19.8817566Z http.https://github.com/.extraheader 2024-06-26T08:48:19.8840565Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:48:19.8862564Z http.https://github.com/.extraheader 2024-06-26T08:48:19.8886573Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:48:19.8907966Z http.https://github.com/.extraheader 2024-06-26T08:48:19.8937403Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:48:19.8959016Z http.https://github.com/.extraheader 2024-06-26T08:48:19.8981429Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:48:19.9002681Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9027900Z Entering 'third_party/flatbuffers' 2024-06-26T08:48:19.9050428Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9076344Z Entering 'third_party/fmt' 2024-06-26T08:48:19.9099024Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9123397Z Entering 'third_party/foxi' 2024-06-26T08:48:19.9146531Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9170345Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:48:19.9192471Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9216576Z Entering 'third_party/gloo' 2024-06-26T08:48:19.9237951Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9261667Z Entering 'third_party/googletest' 2024-06-26T08:48:19.9282411Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9304847Z Entering 'third_party/ideep' 2024-06-26T08:48:19.9327007Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9349852Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:48:19.9372093Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9400463Z Entering 'third_party/ittapi' 2024-06-26T08:48:19.9422949Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9446975Z Entering 'third_party/kineto' 2024-06-26T08:48:19.9468902Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9490728Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:48:19.9512550Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9535205Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:48:19.9556163Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9582011Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:48:19.9602505Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9627723Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:48:19.9649685Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9674609Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:48:19.9695916Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9718790Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:48:19.9741079Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9765886Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:48:19.9786997Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9810443Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:48:19.9832663Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9857206Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:48:19.9879420Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9903500Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:48:19.9925524Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9950913Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:48:19.9972943Z http.https://github.com/.extraheader 2024-06-26T08:48:19.9995804Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:48:20.0017049Z http.https://github.com/.extraheader 2024-06-26T08:48:20.0042554Z Entering 'third_party/mimalloc' 2024-06-26T08:48:20.0064656Z http.https://github.com/.extraheader 2024-06-26T08:48:20.0089854Z Entering 'third_party/nccl/nccl' 2024-06-26T08:48:20.0112796Z http.https://github.com/.extraheader 2024-06-26T08:48:20.0136209Z Entering 'third_party/nlohmann' 2024-06-26T08:48:20.0158028Z http.https://github.com/.extraheader 2024-06-26T08:48:20.0183348Z Entering 'third_party/onnx' 2024-06-26T08:48:20.0204801Z http.https://github.com/.extraheader 2024-06-26T08:48:20.0239877Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:48:20.0262692Z http.https://github.com/.extraheader 2024-06-26T08:48:20.0285742Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:48:20.0307368Z http.https://github.com/.extraheader 2024-06-26T08:48:20.0333069Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:48:20.0355650Z http.https://github.com/.extraheader 2024-06-26T08:48:20.0381924Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:48:20.0403435Z http.https://github.com/.extraheader 2024-06-26T08:48:20.0427186Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:48:20.0447372Z http.https://github.com/.extraheader 2024-06-26T08:48:20.0471289Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:48:20.0493836Z http.https://github.com/.extraheader 2024-06-26T08:48:20.0517337Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:48:20.0541496Z http.https://github.com/.extraheader 2024-06-26T08:48:20.0566820Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:48:20.0588850Z http.https://github.com/.extraheader 2024-06-26T08:48:20.0610464Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:48:20.0631413Z http.https://github.com/.extraheader 2024-06-26T08:48:20.0653806Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:48:20.0674568Z http.https://github.com/.extraheader 2024-06-26T08:48:20.0698254Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:48:20.0719144Z http.https://github.com/.extraheader 2024-06-26T08:48:20.0745243Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:48:20.0766954Z http.https://github.com/.extraheader 2024-06-26T08:48:20.0791857Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:48:20.0813207Z http.https://github.com/.extraheader 2024-06-26T08:48:20.0853423Z Entering 'third_party/pocketfft' 2024-06-26T08:48:20.0877127Z http.https://github.com/.extraheader 2024-06-26T08:48:20.0901016Z Entering 'third_party/protobuf' 2024-06-26T08:48:20.0922999Z http.https://github.com/.extraheader 2024-06-26T08:48:20.0947835Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:48:20.0968993Z http.https://github.com/.extraheader 2024-06-26T08:48:20.0993279Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:48:20.1015447Z http.https://github.com/.extraheader 2024-06-26T08:48:20.1040888Z Entering 'third_party/psimd' 2024-06-26T08:48:20.1064168Z http.https://github.com/.extraheader 2024-06-26T08:48:20.1087996Z Entering 'third_party/pthreadpool' 2024-06-26T08:48:20.1110787Z http.https://github.com/.extraheader 2024-06-26T08:48:20.1133904Z Entering 'third_party/pybind11' 2024-06-26T08:48:20.1153810Z http.https://github.com/.extraheader 2024-06-26T08:48:20.1179330Z Entering 'third_party/python-peachpy' 2024-06-26T08:48:20.1202790Z http.https://github.com/.extraheader 2024-06-26T08:48:20.1226220Z Entering 'third_party/sleef' 2024-06-26T08:48:20.1248707Z http.https://github.com/.extraheader 2024-06-26T08:48:20.1272569Z Entering 'third_party/tensorpipe' 2024-06-26T08:48:20.1295384Z http.https://github.com/.extraheader 2024-06-26T08:48:20.1319108Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:48:20.1342019Z http.https://github.com/.extraheader 2024-06-26T08:48:20.1365235Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:48:20.1386282Z http.https://github.com/.extraheader 2024-06-26T08:48:20.1407684Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:48:20.1428932Z http.https://github.com/.extraheader 2024-06-26T08:48:20.1453388Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:48:20.1473327Z http.https://github.com/.extraheader 2024-06-26T08:48:20.1497816Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:48:20.1518581Z http.https://github.com/.extraheader 2024-06-26T08:48:20.1619167Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-26T08:48:20.1636267Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T08:48:20.1642893Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:48:20.1643409Z ##[endgroup] 2024-06-26T08:48:22.1181418Z Cleaning up orphan processes