2024-06-26T07:04:31.8744752Z Current runner version: '2.317.0' 2024-06-26T07:04:31.8750971Z Runner name: 'i-06b804ca2116ce394' 2024-06-26T07:04:31.8751818Z Runner group name: 'Default' 2024-06-26T07:04:31.8752792Z Machine name: 'ip-10-0-25-189' 2024-06-26T07:04:31.8757691Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T07:04:31.8759702Z Actions: read 2024-06-26T07:04:31.8760313Z Attestations: read 2024-06-26T07:04:31.8760892Z Checks: read 2024-06-26T07:04:31.8761443Z Contents: read 2024-06-26T07:04:31.8761996Z Deployments: read 2024-06-26T07:04:31.8762558Z Discussions: read 2024-06-26T07:04:31.8763135Z Issues: read 2024-06-26T07:04:31.8763682Z Metadata: read 2024-06-26T07:04:31.8764221Z Packages: read 2024-06-26T07:04:31.8764783Z Pages: read 2024-06-26T07:04:31.8765294Z PullRequests: read 2024-06-26T07:04:31.8765890Z RepositoryProjects: read 2024-06-26T07:04:31.8766598Z SecurityEvents: read 2024-06-26T07:04:31.8767187Z Statuses: read 2024-06-26T07:04:31.8767703Z ##[endgroup] 2024-06-26T07:04:31.8770983Z Secret source: Actions 2024-06-26T07:04:31.8771821Z Prepare workflow directory 2024-06-26T07:04:32.2352584Z Prepare all required actions 2024-06-26T07:04:32.2535355Z Getting action download info 2024-06-26T07:04:32.4357324Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T07:04:32.8150314Z Download action repository 'pytorch/pytorch@main' (SHA:c422a9549d2b320b47a9391f6cbcf592b2876c88) 2024-06-26T07:04:35.5134982Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-26T07:04:35.6541481Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T07:04:35.9541284Z Getting action download info 2024-06-26T07:04:36.0670614Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T07:04:36.2784061Z Getting action download info 2024-06-26T07:04:36.3752520Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T07:04:36.5179716Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/129470 (b8c4c54d347aa776934c60784e35936878ef18dc) 2024-06-26T07:04:36.5182002Z ##[group] Inputs 2024-06-26T07:04:36.5182518Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:04:36.5190958Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:04:36.5200161Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:36.5201502Z sync-tag: 2024-06-26T07:04:36.5202348Z timeout-minutes: 240 2024-06-26T07:04:36.5202709Z use-gha: 2024-06-26T07:04:36.5203028Z dashboard-tag: 2024-06-26T07:04:36.5203376Z s3-bucket: gha-artifacts 2024-06-26T07:04:36.5203760Z aws-role-to-assume: 2024-06-26T07:04:36.5204127Z ##[endgroup] 2024-06-26T07:04:36.5205283Z Complete job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:04:36.5767831Z A job started hook has been configured by the self-hosted runner administrator 2024-06-26T07:04:36.5905362Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T07:04:36.5915604Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:36.5916176Z ##[endgroup] 2024-06-26T07:04:37.2365485Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-26T07:04:37.2366099Z with: 2024-06-26T07:04:37.2366764Z github-secret: *** 2024-06-26T07:04:37.2367834Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-26T07:04:37.2369023Z activate-with-label: false 2024-06-26T07:04:37.2369402Z label: with-ssh 2024-06-26T07:04:37.2369735Z remove-existing-keys: true 2024-06-26T07:04:37.2370108Z fail-silently: true 2024-06-26T07:04:37.2370432Z env: 2024-06-26T07:04:37.2370710Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:37.2371071Z ##[endgroup] 2024-06-26T07:04:37.3232004Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-26T07:04:37.3233155Z ciflow reference detected, attempting to extract PR number 2024-06-26T07:04:37.5856916Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-06-26T07:04:37.6515692Z No SSH keys found for user pytorch-bot[bot] 2024-06-26T07:04:37.6516522Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T07:04:37.7175865Z No SSH keys found for user leslie-fang-intel 2024-06-26T07:04:37.7286672Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T07:04:37.7287279Z with: 2024-06-26T07:04:37.7287564Z submodules: recursive 2024-06-26T07:04:37.7287908Z fetch-depth: 0 2024-06-26T07:04:37.7288208Z env: 2024-06-26T07:04:37.7288483Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:37.7288836Z ##[endgroup] 2024-06-26T07:04:37.7482583Z ##[group]Run retry () { 2024-06-26T07:04:37.7482993Z retry () { 2024-06-26T07:04:37.7483554Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T07:04:37.7484196Z } 2024-06-26T07:04:37.7484600Z echo "${GITHUB_WORKSPACE}" 2024-06-26T07:04:37.7485122Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T07:04:37.7485701Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:04:37.7486257Z else 2024-06-26T07:04:37.7486643Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:04:37.7487161Z fi 2024-06-26T07:04:37.7487641Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T07:04:37.7496118Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:37.7496722Z env: 2024-06-26T07:04:37.7497086Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:37.7497525Z NO_SUDO: 2024-06-26T07:04:37.7497835Z ##[endgroup] 2024-06-26T07:04:37.7520941Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:04:39.9514397Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T07:04:39.9514904Z with: 2024-06-26T07:04:39.9515241Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:04:39.9515709Z fetch-depth: 0 2024-06-26T07:04:39.9516037Z submodules: recursive 2024-06-26T07:04:39.9516402Z quiet-checkout: true 2024-06-26T07:04:39.9516900Z repository: pytorch/pytorch 2024-06-26T07:04:39.9517411Z token: *** 2024-06-26T07:04:39.9517713Z ssh-strict: true 2024-06-26T07:04:39.9518054Z persist-credentials: true 2024-06-26T07:04:39.9518434Z clean: true 2024-06-26T07:04:39.9518773Z sparse-checkout-cone-mode: true 2024-06-26T07:04:39.9519188Z lfs: false 2024-06-26T07:04:39.9519504Z set-safe-directory: true 2024-06-26T07:04:39.9519862Z env: 2024-06-26T07:04:39.9520138Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:39.9520510Z ##[endgroup] 2024-06-26T07:04:40.0449778Z Syncing repository: pytorch/pytorch 2024-06-26T07:04:40.0451385Z ##[group]Getting Git version info 2024-06-26T07:04:40.0452455Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:04:40.0453396Z [command]/usr/bin/git version 2024-06-26T07:04:40.0453778Z git version 2.40.1 2024-06-26T07:04:40.0455186Z ##[endgroup] 2024-06-26T07:04:40.0466743Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/8491900e-ee16-4ace-be1d-f86c0d6e4d80' before making global git config changes 2024-06-26T07:04:40.0468169Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T07:04:40.0469430Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:04:40.0502792Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:04:40.0507072Z ##[group]Initializing the repository 2024-06-26T07:04:40.0510220Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:04:40.0534705Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T07:04:40.0535827Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T07:04:40.0536815Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T07:04:40.0537438Z hint: 2024-06-26T07:04:40.0537906Z hint: git config --global init.defaultBranch 2024-06-26T07:04:40.0538431Z hint: 2024-06-26T07:04:40.0538979Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T07:04:40.0539900Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T07:04:40.0540790Z hint: 2024-06-26T07:04:40.0541135Z hint: git branch -m 2024-06-26T07:04:40.0541931Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T07:04:40.0543994Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T07:04:40.0565092Z ##[endgroup] 2024-06-26T07:04:40.0565718Z ##[group]Disabling automatic garbage collection 2024-06-26T07:04:40.0568116Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T07:04:40.0588653Z ##[endgroup] 2024-06-26T07:04:40.0589466Z ##[group]Setting up auth 2024-06-26T07:04:40.0595765Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T07:04:40.0618535Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T07:04:40.0809571Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T07:04:40.0832615Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T07:04:40.1034807Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:04:40.1076290Z ##[endgroup] 2024-06-26T07:04:40.1076867Z ##[group]Fetching the repository 2024-06-26T07:04:40.1082924Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T07:04:42.3903515Z remote: Enumerating objects: 987619 2024-06-26T07:04:42.3904548Z remote: Enumerating objects: 990209, done. 2024-06-26T07:04:42.3905358Z remote: Counting objects: 0% (1/2590) 2024-06-26T07:04:42.3910530Z remote: Counting objects: 1% (26/2590) 2024-06-26T07:04:42.3911571Z remote: Counting objects: 2% (52/2590) 2024-06-26T07:04:42.3914146Z remote: Counting objects: 3% (78/2590) 2024-06-26T07:04:42.3916665Z remote: Counting objects: 4% (104/2590) 2024-06-26T07:04:42.3917236Z remote: Counting objects: 5% (130/2590) 2024-06-26T07:04:42.3917896Z remote: Counting objects: 6% (156/2590) 2024-06-26T07:04:42.3918954Z remote: Counting objects: 7% (182/2590) 2024-06-26T07:04:42.3922041Z remote: Counting objects: 8% (208/2590) 2024-06-26T07:04:42.3922606Z remote: Counting objects: 9% (234/2590) 2024-06-26T07:04:42.3923307Z remote: Counting objects: 10% (259/2590) 2024-06-26T07:04:42.3923933Z remote: Counting objects: 11% (285/2590) 2024-06-26T07:04:42.3924498Z remote: Counting objects: 12% (311/2590) 2024-06-26T07:04:42.3925050Z remote: Counting objects: 13% (337/2590) 2024-06-26T07:04:42.3925633Z remote: Counting objects: 14% (363/2590) 2024-06-26T07:04:42.3926374Z remote: Counting objects: 15% (389/2590) 2024-06-26T07:04:42.3926930Z remote: Counting objects: 16% (415/2590) 2024-06-26T07:04:42.3927476Z remote: Counting objects: 17% (441/2590) 2024-06-26T07:04:42.3928172Z remote: Counting objects: 18% (467/2590) 2024-06-26T07:04:42.3928855Z remote: Counting objects: 19% (493/2590) 2024-06-26T07:04:42.3929410Z remote: Counting objects: 20% (518/2590) 2024-06-26T07:04:42.3930052Z remote: Counting objects: 21% (544/2590) 2024-06-26T07:04:42.3930786Z remote: Counting objects: 22% (570/2590) 2024-06-26T07:04:42.3931475Z remote: Counting objects: 23% (596/2590) 2024-06-26T07:04:42.3932470Z remote: Counting objects: 24% (622/2590) 2024-06-26T07:04:42.3933169Z remote: Counting objects: 25% (648/2590) 2024-06-26T07:04:42.3933804Z remote: Counting objects: 26% (674/2590) 2024-06-26T07:04:42.3934356Z remote: Counting objects: 27% (700/2590) 2024-06-26T07:04:42.3935228Z remote: Counting objects: 28% (726/2590) 2024-06-26T07:04:42.3935966Z remote: Counting objects: 29% (752/2590) 2024-06-26T07:04:42.3936592Z remote: Counting objects: 30% (777/2590) 2024-06-26T07:04:42.3937226Z remote: Counting objects: 31% (803/2590) 2024-06-26T07:04:42.3937915Z remote: Counting objects: 32% (829/2590) 2024-06-26T07:04:42.3938540Z remote: Counting objects: 33% (855/2590) 2024-06-26T07:04:42.3939095Z remote: Counting objects: 34% (881/2590) 2024-06-26T07:04:42.3939629Z remote: Counting objects: 35% (907/2590) 2024-06-26T07:04:42.3940318Z remote: Counting objects: 36% (933/2590) 2024-06-26T07:04:42.3940999Z remote: Counting objects: 37% (959/2590) 2024-06-26T07:04:42.3941602Z remote: Counting objects: 38% (985/2590) 2024-06-26T07:04:42.3942161Z remote: Counting objects: 39% (1011/2590) 2024-06-26T07:04:42.3942906Z remote: Counting objects: 40% (1036/2590) 2024-06-26T07:04:42.3943743Z remote: Counting objects: 41% (1062/2590) 2024-06-26T07:04:42.3944309Z remote: Counting objects: 42% (1088/2590) 2024-06-26T07:04:42.3944971Z remote: Counting objects: 43% (1114/2590) 2024-06-26T07:04:42.3945635Z remote: Counting objects: 44% (1140/2590) 2024-06-26T07:04:42.3946283Z remote: Counting objects: 45% (1166/2590) 2024-06-26T07:04:42.3946843Z remote: Counting objects: 46% (1192/2590) 2024-06-26T07:04:42.3947524Z remote: Counting objects: 47% (1218/2590) 2024-06-26T07:04:42.3948196Z remote: Counting objects: 48% (1244/2590) 2024-06-26T07:04:42.3948847Z remote: Counting objects: 49% (1270/2590) 2024-06-26T07:04:42.3949415Z remote: Counting objects: 50% (1295/2590) 2024-06-26T07:04:42.3950182Z remote: Counting objects: 51% (1321/2590) 2024-06-26T07:04:42.3950903Z remote: Counting objects: 52% (1347/2590) 2024-06-26T07:04:42.3951620Z remote: Counting objects: 53% (1373/2590) 2024-06-26T07:04:42.3952180Z remote: Counting objects: 54% (1399/2590) 2024-06-26T07:04:42.3952736Z remote: Counting objects: 55% (1425/2590) 2024-06-26T07:04:42.3953278Z remote: Counting objects: 56% (1451/2590) 2024-06-26T07:04:42.3953991Z remote: Counting objects: 57% (1477/2590) 2024-06-26T07:04:42.3954707Z remote: Counting objects: 58% (1503/2590) 2024-06-26T07:04:42.3955302Z remote: Counting objects: 59% (1529/2590) 2024-06-26T07:04:42.3956017Z remote: Counting objects: 60% (1554/2590) 2024-06-26T07:04:42.3956722Z remote: Counting objects: 61% (1580/2590) 2024-06-26T07:04:42.3957289Z remote: Counting objects: 62% (1606/2590) 2024-06-26T07:04:42.3957868Z remote: Counting objects: 63% (1632/2590) 2024-06-26T07:04:42.3958418Z remote: Counting objects: 64% (1658/2590) 2024-06-26T07:04:42.3958988Z remote: Counting objects: 65% (1684/2590) 2024-06-26T07:04:42.3959653Z remote: Counting objects: 66% (1710/2590) 2024-06-26T07:04:42.3960205Z remote: Counting objects: 67% (1736/2590) 2024-06-26T07:04:42.3960759Z remote: Counting objects: 68% (1762/2590) 2024-06-26T07:04:42.3961311Z remote: Counting objects: 69% (1788/2590) 2024-06-26T07:04:42.3961863Z remote: Counting objects: 70% (1813/2590) 2024-06-26T07:04:42.3962422Z remote: Counting objects: 71% (1839/2590) 2024-06-26T07:04:42.3962977Z remote: Counting objects: 72% (1865/2590) 2024-06-26T07:04:42.3963533Z remote: Counting objects: 73% (1891/2590) 2024-06-26T07:04:42.3964108Z remote: Counting objects: 74% (1917/2590) 2024-06-26T07:04:42.3964670Z remote: Counting objects: 75% (1943/2590) 2024-06-26T07:04:42.3965228Z remote: Counting objects: 76% (1969/2590) 2024-06-26T07:04:42.3965783Z remote: Counting objects: 77% (1995/2590) 2024-06-26T07:04:42.3966342Z remote: Counting objects: 78% (2021/2590) 2024-06-26T07:04:42.3966989Z remote: Counting objects: 79% (2047/2590) 2024-06-26T07:04:42.3967552Z remote: Counting objects: 80% (2072/2590) 2024-06-26T07:04:42.3968101Z remote: Counting objects: 81% (2098/2590) 2024-06-26T07:04:42.3968654Z remote: Counting objects: 82% (2124/2590) 2024-06-26T07:04:42.3969209Z remote: Counting objects: 83% (2150/2590) 2024-06-26T07:04:42.3969766Z remote: Counting objects: 84% (2176/2590) 2024-06-26T07:04:42.3970312Z remote: Counting objects: 85% (2202/2590) 2024-06-26T07:04:42.3970876Z remote: Counting objects: 86% (2228/2590) 2024-06-26T07:04:42.3971429Z remote: Counting objects: 87% (2254/2590) 2024-06-26T07:04:42.3972135Z remote: Counting objects: 88% (2280/2590) 2024-06-26T07:04:42.3972689Z remote: Counting objects: 89% (2306/2590) 2024-06-26T07:04:42.3973245Z remote: Counting objects: 90% (2331/2590) 2024-06-26T07:04:42.3973803Z remote: Counting objects: 91% (2357/2590) 2024-06-26T07:04:42.3974370Z remote: Counting objects: 92% (2383/2590) 2024-06-26T07:04:42.3974917Z remote: Counting objects: 93% (2409/2590) 2024-06-26T07:04:42.3975476Z remote: Counting objects: 94% (2435/2590) 2024-06-26T07:04:42.3976038Z remote: Counting objects: 95% (2461/2590) 2024-06-26T07:04:42.3976598Z remote: Counting objects: 96% (2487/2590) 2024-06-26T07:04:42.3977146Z remote: Counting objects: 97% (2513/2590) 2024-06-26T07:04:42.3977698Z remote: Counting objects: 98% (2539/2590) 2024-06-26T07:04:42.3978261Z remote: Counting objects: 99% (2565/2590) 2024-06-26T07:04:42.3978820Z remote: Counting objects: 100% (2590/2590) 2024-06-26T07:04:42.3979407Z remote: Counting objects: 100% (2590/2590), done. 2024-06-26T07:04:42.4113223Z remote: Compressing objects: 0% (1/1068) 2024-06-26T07:04:42.4551689Z remote: Compressing objects: 1% (11/1068) 2024-06-26T07:04:42.5008673Z remote: Compressing objects: 2% (22/1068) 2024-06-26T07:04:42.5414767Z remote: Compressing objects: 3% (33/1068) 2024-06-26T07:04:42.5639527Z remote: Compressing objects: 4% (43/1068) 2024-06-26T07:04:42.5937600Z remote: Compressing objects: 5% (54/1068) 2024-06-26T07:04:42.6136993Z remote: Compressing objects: 6% (65/1068) 2024-06-26T07:04:42.6555154Z remote: Compressing objects: 7% (75/1068) 2024-06-26T07:04:42.7508859Z remote: Compressing objects: 8% (86/1068) 2024-06-26T07:04:42.8323021Z remote: Compressing objects: 9% (97/1068) 2024-06-26T07:04:42.8575526Z remote: Compressing objects: 10% (107/1068) 2024-06-26T07:04:42.8761620Z remote: Compressing objects: 11% (118/1068) 2024-06-26T07:04:42.8863113Z remote: Compressing objects: 12% (129/1068) 2024-06-26T07:04:42.8863759Z remote: Compressing objects: 13% (139/1068) 2024-06-26T07:04:42.8997050Z remote: Compressing objects: 14% (150/1068) 2024-06-26T07:04:42.8997646Z remote: Compressing objects: 15% (161/1068) 2024-06-26T07:04:42.9065575Z remote: Compressing objects: 16% (171/1068) 2024-06-26T07:04:42.9066171Z remote: Compressing objects: 17% (182/1068) 2024-06-26T07:04:42.9066753Z remote: Compressing objects: 18% (193/1068) 2024-06-26T07:04:42.9077813Z remote: Compressing objects: 19% (203/1068) 2024-06-26T07:04:42.9096190Z remote: Compressing objects: 20% (214/1068) 2024-06-26T07:04:42.9110882Z remote: Compressing objects: 21% (225/1068) 2024-06-26T07:04:42.9128562Z remote: Compressing objects: 22% (235/1068) 2024-06-26T07:04:42.9151807Z remote: Compressing objects: 23% (246/1068) 2024-06-26T07:04:42.9166041Z remote: Compressing objects: 24% (257/1068) 2024-06-26T07:04:42.9176660Z remote: Compressing objects: 25% (267/1068) 2024-06-26T07:04:42.9192039Z remote: Compressing objects: 26% (278/1068) 2024-06-26T07:04:42.9199129Z remote: Compressing objects: 27% (289/1068) 2024-06-26T07:04:42.9201541Z remote: Compressing objects: 28% (300/1068) 2024-06-26T07:04:42.9207411Z remote: Compressing objects: 29% (310/1068) 2024-06-26T07:04:42.9218003Z remote: Compressing objects: 30% (321/1068) 2024-06-26T07:04:42.9227732Z remote: Compressing objects: 31% (332/1068) 2024-06-26T07:04:42.9236986Z remote: Compressing objects: 32% (342/1068) 2024-06-26T07:04:42.9246472Z remote: Compressing objects: 33% (353/1068) 2024-06-26T07:04:42.9266499Z remote: Compressing objects: 34% (364/1068) 2024-06-26T07:04:42.9281026Z remote: Compressing objects: 35% (374/1068) 2024-06-26T07:04:42.9301936Z remote: Compressing objects: 36% (385/1068) 2024-06-26T07:04:42.9317486Z remote: Compressing objects: 37% (396/1068) 2024-06-26T07:04:42.9333605Z remote: Compressing objects: 38% (406/1068) 2024-06-26T07:04:42.9348832Z remote: Compressing objects: 39% (417/1068) 2024-06-26T07:04:42.9366587Z remote: Compressing objects: 40% (428/1068) 2024-06-26T07:04:42.9387498Z remote: Compressing objects: 41% (438/1068) 2024-06-26T07:04:42.9404008Z remote: Compressing objects: 42% (449/1068) 2024-06-26T07:04:42.9416760Z remote: Compressing objects: 43% (460/1068) 2024-06-26T07:04:42.9421557Z remote: Compressing objects: 44% (470/1068) 2024-06-26T07:04:42.9430068Z remote: Compressing objects: 45% (481/1068) 2024-06-26T07:04:42.9434898Z remote: Compressing objects: 46% (492/1068) 2024-06-26T07:04:42.9450842Z remote: Compressing objects: 47% (502/1068) 2024-06-26T07:04:42.9463482Z remote: Compressing objects: 48% (513/1068) 2024-06-26T07:04:42.9479744Z remote: Compressing objects: 49% (524/1068) 2024-06-26T07:04:42.9480447Z remote: Compressing objects: 50% (534/1068) 2024-06-26T07:04:42.9486007Z remote: Compressing objects: 51% (545/1068) 2024-06-26T07:04:42.9500597Z remote: Compressing objects: 52% (556/1068) 2024-06-26T07:04:42.9516080Z remote: Compressing objects: 53% (567/1068) 2024-06-26T07:04:42.9536099Z remote: Compressing objects: 54% (577/1068) 2024-06-26T07:04:42.9544000Z remote: Compressing objects: 55% (588/1068) 2024-06-26T07:04:42.9554028Z remote: Compressing objects: 56% (599/1068) 2024-06-26T07:04:42.9568271Z remote: Compressing objects: 57% (609/1068) 2024-06-26T07:04:42.9580534Z remote: Compressing objects: 58% (620/1068) 2024-06-26T07:04:42.9588496Z remote: Compressing objects: 59% (631/1068) 2024-06-26T07:04:42.9599488Z remote: Compressing objects: 60% (641/1068) 2024-06-26T07:04:42.9613534Z remote: Compressing objects: 61% (652/1068) 2024-06-26T07:04:42.9622444Z remote: Compressing objects: 62% (663/1068) 2024-06-26T07:04:42.9630864Z remote: Compressing objects: 63% (673/1068) 2024-06-26T07:04:42.9640305Z remote: Compressing objects: 64% (684/1068) 2024-06-26T07:04:42.9650923Z remote: Compressing objects: 65% (695/1068) 2024-06-26T07:04:42.9653720Z remote: Compressing objects: 66% (705/1068) 2024-06-26T07:04:42.9666180Z remote: Compressing objects: 67% (716/1068) 2024-06-26T07:04:42.9671994Z remote: Compressing objects: 68% (727/1068) 2024-06-26T07:04:42.9674590Z remote: Compressing objects: 69% (737/1068) 2024-06-26T07:04:42.9677193Z remote: Compressing objects: 70% (748/1068) 2024-06-26T07:04:42.9679895Z remote: Compressing objects: 71% (759/1068) 2024-06-26T07:04:42.9682351Z remote: Compressing objects: 72% (769/1068) 2024-06-26T07:04:42.9684974Z remote: Compressing objects: 73% (780/1068) 2024-06-26T07:04:42.9685560Z remote: Compressing objects: 74% (791/1068) 2024-06-26T07:04:42.9687597Z remote: Compressing objects: 75% (801/1068) 2024-06-26T07:04:42.9688176Z remote: Compressing objects: 76% (812/1068) 2024-06-26T07:04:42.9688889Z remote: Compressing objects: 77% (823/1068) 2024-06-26T07:04:42.9689471Z remote: Compressing objects: 78% (834/1068) 2024-06-26T07:04:42.9693634Z remote: Compressing objects: 79% (844/1068) 2024-06-26T07:04:42.9694293Z remote: Compressing objects: 80% (855/1068) 2024-06-26T07:04:42.9696557Z remote: Compressing objects: 81% (866/1068) 2024-06-26T07:04:42.9698917Z remote: Compressing objects: 82% (876/1068) 2024-06-26T07:04:42.9701537Z remote: Compressing objects: 83% (887/1068) 2024-06-26T07:04:42.9702136Z remote: Compressing objects: 84% (898/1068) 2024-06-26T07:04:42.9704154Z remote: Compressing objects: 85% (908/1068) 2024-06-26T07:04:42.9706790Z remote: Compressing objects: 86% (919/1068) 2024-06-26T07:04:42.9709316Z remote: Compressing objects: 87% (930/1068) 2024-06-26T07:04:42.9711734Z remote: Compressing objects: 88% (940/1068) 2024-06-26T07:04:42.9714462Z remote: Compressing objects: 89% (951/1068) 2024-06-26T07:04:42.9715074Z remote: Compressing objects: 90% (962/1068) 2024-06-26T07:04:42.9715654Z remote: Compressing objects: 91% (972/1068) 2024-06-26T07:04:42.9717084Z remote: Compressing objects: 92% (983/1068) 2024-06-26T07:04:42.9719596Z remote: Compressing objects: 93% (994/1068) 2024-06-26T07:04:42.9720205Z remote: Compressing objects: 94% (1004/1068) 2024-06-26T07:04:42.9722298Z remote: Compressing objects: 95% (1015/1068) 2024-06-26T07:04:42.9722906Z remote: Compressing objects: 96% (1026/1068) 2024-06-26T07:04:42.9724749Z remote: Compressing objects: 97% (1036/1068) 2024-06-26T07:04:42.9725360Z remote: Compressing objects: 98% (1047/1068) 2024-06-26T07:04:42.9727394Z remote: Compressing objects: 99% (1058/1068) 2024-06-26T07:04:42.9728354Z remote: Compressing objects: 100% (1068/1068) 2024-06-26T07:04:42.9728984Z remote: Compressing objects: 100% (1068/1068), done. 2024-06-26T07:05:03.1286894Z remote: Total 990209 (delta 2055), reused 1861 (delta 1521), pack-reused 987619 2024-06-26T07:05:22.8865817Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T07:05:22.8883519Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:05:22.8886860Z ##[endgroup] 2024-06-26T07:05:22.8891239Z ##[group]Determining the checkout info 2024-06-26T07:05:22.8893216Z ##[endgroup] 2024-06-26T07:05:22.8894017Z ##[group]Checking out the ref 2024-06-26T07:05:22.8895167Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:05:24.1009332Z ##[endgroup] 2024-06-26T07:05:24.1010053Z ##[group]Setting up auth for fetching submodules 2024-06-26T07:05:24.1014035Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:05:24.1054488Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T07:05:24.1075344Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T07:05:24.1096052Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T07:05:24.1112979Z ##[endgroup] 2024-06-26T07:05:24.1113788Z ##[group]Fetching submodules 2024-06-26T07:05:24.1117330Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T07:05:24.1327616Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T07:05:24.1537503Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T07:05:24.1540135Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T07:05:24.1541720Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T07:05:24.1543398Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T07:05:24.1545411Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T07:05:24.1547241Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T07:05:24.1548876Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T07:05:24.1551731Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T07:05:24.1554203Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T07:05:24.1557091Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T07:05:24.1559734Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T07:05:24.1562535Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T07:05:24.1565371Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T07:05:24.1568499Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T07:05:24.1571299Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T07:05:24.1575726Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T07:05:24.1578985Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:05:24.1582001Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T07:05:24.1585463Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T07:05:24.1588669Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T07:05:24.1592169Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T07:05:24.1595673Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T07:05:24.1599454Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T07:05:24.1603006Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T07:05:24.1606718Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T07:05:24.1610384Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T07:05:24.1614980Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T07:05:24.1618600Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T07:05:24.1622658Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T07:05:24.1626804Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T07:05:24.1631021Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T07:05:24.1635132Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T07:05:24.1639340Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T07:05:24.1643466Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T07:05:24.1647954Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T07:05:24.1669272Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T07:05:24.4336865Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T07:05:24.6502363Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T07:05:24.8068143Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T07:05:25.0602187Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T07:05:26.9980357Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T07:05:37.0246616Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T07:05:37.3915898Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T07:05:37.9512755Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T07:05:38.5309501Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T07:05:39.6639728Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T07:05:41.3749022Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T07:05:47.2761452Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T07:05:48.5409134Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T07:05:50.2387556Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T07:05:51.4350416Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T07:05:51.6212428Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T07:05:52.0192570Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T07:05:52.3376659Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T07:05:53.3313761Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T07:05:53.7422423Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T07:05:53.9933465Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T07:05:55.4707370Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T07:05:56.2138373Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T07:05:56.8882916Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T07:06:02.5924936Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T07:06:05.1540244Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T07:06:10.6848289Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T07:06:10.9281917Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T07:06:18.9477307Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T07:06:19.1101575Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T07:06:19.2953876Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T07:06:20.2480642Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T07:06:20.5157940Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T07:06:21.1281182Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T07:06:21.5343914Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T07:06:21.5427540Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T07:06:21.5493395Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T07:06:21.5670993Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T07:06:21.5972756Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T07:06:22.3747521Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T07:06:22.3929330Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T07:06:22.4274225Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T07:06:22.5116149Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T07:06:22.5383365Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T07:06:22.9672549Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T07:06:23.1798249Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T07:06:23.2421277Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T07:06:23.2432024Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:06:23.2433656Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:06:23.2435556Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:06:23.2438025Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:06:23.2439985Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:06:23.2459556Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T07:06:24.2386775Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T07:06:24.7948273Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T07:06:26.5196736Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T07:06:27.5842976Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T07:06:27.9447216Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T07:06:28.0290177Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T07:06:28.3708006Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T07:06:28.4245403Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T07:06:28.4330294Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T07:06:28.5234512Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T07:06:28.5521725Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T07:06:28.5588546Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T07:06:28.5892932Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T07:06:28.6092888Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T07:06:28.6458032Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:06:28.6554060Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T07:06:28.6564893Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T07:06:28.6580050Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T07:06:40.6631841Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T07:06:40.6771033Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T07:06:40.7550407Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T07:06:40.7561779Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:06:40.7563694Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:06:40.7565594Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:06:40.7584013Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T07:06:41.2993323Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T07:06:42.4740215Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T07:06:43.4597236Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T07:06:43.4606798Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:06:43.4608968Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:06:43.4610977Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:06:43.4613918Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:06:43.4616047Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:06:43.4618493Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:06:43.4620657Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:06:43.4623297Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:06:43.4643378Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T07:06:44.4716293Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T07:06:44.8999284Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T07:06:46.0972567Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T07:06:46.4169690Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T07:06:46.8688208Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T07:06:47.7885669Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T07:06:53.2859912Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T07:06:53.6283921Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T07:06:53.6421436Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T07:06:53.6718702Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T07:06:53.6814910Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T07:06:53.6824575Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:06:53.6841030Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T07:06:54.0189552Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T07:06:54.0326472Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T07:06:54.0665287Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T07:06:54.1475779Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T07:06:54.1597376Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T07:06:54.1890810Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T07:06:54.2385516Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T07:06:54.2683458Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T07:06:54.2882027Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T07:06:54.3743290Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T07:06:54.6461277Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T07:06:54.6485487Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T07:06:54.6487256Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T07:06:54.6503145Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T07:06:55.0141929Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T07:06:56.0947690Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T07:06:56.1221913Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T07:06:56.1749921Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T07:06:56.1761221Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:06:56.1769641Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:06:56.1771537Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:06:56.1773968Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:06:56.1776165Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:06:56.1778525Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:06:56.1780710Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:06:56.1782533Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:06:56.1791289Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T07:06:56.6033943Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T07:06:57.7249899Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T07:06:58.0467831Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T07:07:03.5993507Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T07:07:03.8641706Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T07:07:04.0423826Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T07:07:04.3476614Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T07:07:11.6963789Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T07:07:11.7299088Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T07:07:11.7426719Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T07:07:11.8259732Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T07:07:11.8353297Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T07:07:11.8462447Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T07:07:11.8575572Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T07:07:11.8584847Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:07:11.8586991Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:07:11.8603598Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T07:07:13.6533961Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T07:07:14.8223644Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T07:07:14.8586794Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:07:15.2298813Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T07:07:15.2377947Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T07:07:15.4550311Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T07:07:15.4563280Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:07:15.4565244Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T07:07:15.4581439Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T07:07:15.8371462Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T07:07:16.7647393Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T07:07:16.8274062Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T07:07:16.8336938Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T07:07:16.8426578Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T07:07:16.8705037Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T07:07:16.8929020Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T07:07:16.9264460Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T07:07:16.9468675Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T07:07:16.9479979Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:07:16.9481743Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:07:16.9483373Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:07:16.9485217Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:07:16.9503620Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T07:07:17.9038948Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T07:07:18.1667278Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T07:07:19.2300425Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T07:07:20.2927731Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T07:07:20.3043432Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T07:07:20.3604927Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T07:07:20.3836491Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T07:07:20.3846565Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:07:20.3864560Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T07:07:20.5717765Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T07:07:20.5744293Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T07:07:20.5952048Z Entering 'android/libs/fbjni' 2024-06-26T07:07:20.5981628Z Entering 'third_party/FP16' 2024-06-26T07:07:20.6007109Z Entering 'third_party/FXdiv' 2024-06-26T07:07:20.6038262Z Entering 'third_party/NNPACK' 2024-06-26T07:07:20.6065947Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:07:20.6093706Z Entering 'third_party/XNNPACK' 2024-06-26T07:07:20.6131198Z Entering 'third_party/benchmark' 2024-06-26T07:07:20.6161204Z Entering 'third_party/cpp-httplib' 2024-06-26T07:07:20.6189518Z Entering 'third_party/cpuinfo' 2024-06-26T07:07:20.6218671Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:07:20.6248678Z Entering 'third_party/cutlass' 2024-06-26T07:07:20.6285578Z Entering 'third_party/eigen' 2024-06-26T07:07:20.6315036Z Entering 'third_party/fbgemm' 2024-06-26T07:07:20.6342539Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:07:20.6369312Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:07:20.6397351Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:07:20.6429208Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:07:20.6457881Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:07:20.6487435Z Entering 'third_party/flatbuffers' 2024-06-26T07:07:20.6518287Z Entering 'third_party/fmt' 2024-06-26T07:07:20.6548933Z Entering 'third_party/foxi' 2024-06-26T07:07:20.6577289Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:07:20.6605883Z Entering 'third_party/gloo' 2024-06-26T07:07:20.6632604Z Entering 'third_party/googletest' 2024-06-26T07:07:20.6660881Z Entering 'third_party/ideep' 2024-06-26T07:07:20.6687219Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:07:20.6720692Z Entering 'third_party/ittapi' 2024-06-26T07:07:20.6750468Z Entering 'third_party/kineto' 2024-06-26T07:07:20.6780594Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:07:20.6810017Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:07:20.6840841Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:07:20.6868253Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:07:20.6896905Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:07:20.6925468Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:07:20.6954520Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:07:20.6981968Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:07:20.7011546Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:07:20.7039874Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:07:20.7070237Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:07:20.7098897Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:07:20.7127495Z Entering 'third_party/mimalloc' 2024-06-26T07:07:20.7155613Z Entering 'third_party/nccl/nccl' 2024-06-26T07:07:20.7183607Z Entering 'third_party/nlohmann' 2024-06-26T07:07:20.7212860Z Entering 'third_party/onnx' 2024-06-26T07:07:20.7254486Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:07:20.7282194Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:07:20.7313141Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:07:20.7342469Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:07:20.7369896Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:07:20.7396909Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:07:20.7424780Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:07:20.7453776Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:07:20.7481145Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:07:20.7508024Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:07:20.7535429Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:07:20.7565612Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:07:20.7595850Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:07:20.7639389Z Entering 'third_party/pocketfft' 2024-06-26T07:07:20.7667650Z Entering 'third_party/protobuf' 2024-06-26T07:07:20.7699122Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:07:20.7726861Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:07:20.7757828Z Entering 'third_party/psimd' 2024-06-26T07:07:20.7785244Z Entering 'third_party/pthreadpool' 2024-06-26T07:07:20.7814813Z Entering 'third_party/pybind11' 2024-06-26T07:07:20.7845103Z Entering 'third_party/python-peachpy' 2024-06-26T07:07:20.7873113Z Entering 'third_party/sleef' 2024-06-26T07:07:20.7898766Z Entering 'third_party/tensorpipe' 2024-06-26T07:07:20.7928967Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:07:20.7956360Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:07:20.7984591Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:07:20.8013827Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:07:20.8039752Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:07:20.8081508Z ##[endgroup] 2024-06-26T07:07:20.8083490Z ##[group]Persisting credentials for submodules 2024-06-26T07:07:20.8087209Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T07:07:20.8300126Z Entering 'android/libs/fbjni' 2024-06-26T07:07:20.8336059Z Entering 'third_party/FP16' 2024-06-26T07:07:20.8369766Z Entering 'third_party/FXdiv' 2024-06-26T07:07:20.8402651Z Entering 'third_party/NNPACK' 2024-06-26T07:07:20.8438806Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:07:20.8472171Z Entering 'third_party/XNNPACK' 2024-06-26T07:07:20.8520432Z Entering 'third_party/benchmark' 2024-06-26T07:07:20.8557732Z Entering 'third_party/cpp-httplib' 2024-06-26T07:07:20.8593813Z Entering 'third_party/cpuinfo' 2024-06-26T07:07:20.8630749Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:07:20.8667472Z Entering 'third_party/cutlass' 2024-06-26T07:07:20.8709419Z Entering 'third_party/eigen' 2024-06-26T07:07:20.8748623Z Entering 'third_party/fbgemm' 2024-06-26T07:07:20.8783198Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:07:20.8817489Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:07:20.8853231Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:07:20.8894397Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:07:20.8929583Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:07:20.8965905Z Entering 'third_party/flatbuffers' 2024-06-26T07:07:20.9005579Z Entering 'third_party/fmt' 2024-06-26T07:07:20.9043381Z Entering 'third_party/foxi' 2024-06-26T07:07:20.9079273Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:07:20.9114802Z Entering 'third_party/gloo' 2024-06-26T07:07:20.9149908Z Entering 'third_party/googletest' 2024-06-26T07:07:20.9185275Z Entering 'third_party/ideep' 2024-06-26T07:07:20.9216704Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:07:20.9257198Z Entering 'third_party/ittapi' 2024-06-26T07:07:20.9290368Z Entering 'third_party/kineto' 2024-06-26T07:07:20.9324133Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:07:20.9360347Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:07:20.9397463Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:07:20.9433573Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:07:20.9467279Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:07:20.9501709Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:07:20.9538173Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:07:20.9573486Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:07:20.9608571Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:07:20.9646068Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:07:20.9680179Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:07:20.9716506Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:07:20.9749793Z Entering 'third_party/mimalloc' 2024-06-26T07:07:20.9781553Z Entering 'third_party/nccl/nccl' 2024-06-26T07:07:20.9818494Z Entering 'third_party/nlohmann' 2024-06-26T07:07:20.9855973Z Entering 'third_party/onnx' 2024-06-26T07:07:20.9903287Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:07:20.9938228Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:07:20.9975265Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:07:21.0012566Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:07:21.0047465Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:07:21.0080365Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:07:21.0116876Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:07:21.0150293Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:07:21.0185007Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:07:21.0217001Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:07:21.0248340Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:07:21.0284090Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:07:21.0320106Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:07:21.0371198Z Entering 'third_party/pocketfft' 2024-06-26T07:07:21.0406506Z Entering 'third_party/protobuf' 2024-06-26T07:07:21.0444496Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:07:21.0480569Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:07:21.0514802Z Entering 'third_party/psimd' 2024-06-26T07:07:21.0547871Z Entering 'third_party/pthreadpool' 2024-06-26T07:07:21.0582791Z Entering 'third_party/pybind11' 2024-06-26T07:07:21.0619429Z Entering 'third_party/python-peachpy' 2024-06-26T07:07:21.0653243Z Entering 'third_party/sleef' 2024-06-26T07:07:21.0689240Z Entering 'third_party/tensorpipe' 2024-06-26T07:07:21.0720673Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:07:21.0755711Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:07:21.0789700Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:07:21.0824173Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:07:21.0857110Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:07:21.0906364Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T07:07:21.1119295Z Entering 'android/libs/fbjni' 2024-06-26T07:07:21.1152119Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T07:07:21.1163809Z Entering 'third_party/FP16' 2024-06-26T07:07:21.1196235Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T07:07:21.1209894Z Entering 'third_party/FXdiv' 2024-06-26T07:07:21.1243031Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T07:07:21.1255209Z Entering 'third_party/NNPACK' 2024-06-26T07:07:21.1286316Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T07:07:21.1298449Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:07:21.1329550Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T07:07:21.1343761Z Entering 'third_party/XNNPACK' 2024-06-26T07:07:21.1380983Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T07:07:21.1405075Z Entering 'third_party/benchmark' 2024-06-26T07:07:21.1439146Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:07:21.1451845Z Entering 'third_party/cpp-httplib' 2024-06-26T07:07:21.1484768Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T07:07:21.1497674Z Entering 'third_party/cpuinfo' 2024-06-26T07:07:21.1529348Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:07:21.1542550Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:07:21.1571559Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T07:07:21.1584519Z Entering 'third_party/cutlass' 2024-06-26T07:07:21.1616709Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:07:21.1636015Z Entering 'third_party/eigen' 2024-06-26T07:07:21.1667973Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T07:07:21.1683003Z Entering 'third_party/fbgemm' 2024-06-26T07:07:21.1715181Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T07:07:21.1727063Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:07:21.1761372Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T07:07:21.1773654Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:07:21.1805190Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:07:21.1817160Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:07:21.1848373Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:07:21.1866826Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:07:21.1898629Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:07:21.1911108Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:07:21.1942156Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T07:07:21.1956952Z Entering 'third_party/flatbuffers' 2024-06-26T07:07:21.1990812Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T07:07:21.2004833Z Entering 'third_party/fmt' 2024-06-26T07:07:21.2037952Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:07:21.2052569Z Entering 'third_party/foxi' 2024-06-26T07:07:21.2084789Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T07:07:21.2096640Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:07:21.2127999Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T07:07:21.2140675Z Entering 'third_party/gloo' 2024-06-26T07:07:21.2173906Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T07:07:21.2187600Z Entering 'third_party/googletest' 2024-06-26T07:07:21.2218692Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:07:21.2231758Z Entering 'third_party/ideep' 2024-06-26T07:07:21.2263032Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T07:07:21.2276574Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:07:21.2309580Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T07:07:21.2327591Z Entering 'third_party/ittapi' 2024-06-26T07:07:21.2361518Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T07:07:21.2375397Z Entering 'third_party/kineto' 2024-06-26T07:07:21.2407446Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T07:07:21.2419998Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:07:21.2450196Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T07:07:21.2461915Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:07:21.2494988Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T07:07:21.2509147Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:07:21.2542639Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T07:07:21.2555849Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:07:21.2588611Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:07:21.2600785Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:07:21.2634375Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T07:07:21.2646262Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:07:21.2679462Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T07:07:21.2693043Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:07:21.2723696Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T07:07:21.2736975Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:07:21.2768992Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:07:21.2781642Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:07:21.2814348Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T07:07:21.2828348Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:07:21.2859595Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T07:07:21.2873192Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:07:21.2904172Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T07:07:21.2916865Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:07:21.2948401Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T07:07:21.2962808Z Entering 'third_party/mimalloc' 2024-06-26T07:07:21.2995819Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T07:07:21.3008106Z Entering 'third_party/nccl/nccl' 2024-06-26T07:07:21.3040778Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T07:07:21.3052335Z Entering 'third_party/nlohmann' 2024-06-26T07:07:21.3085175Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T07:07:21.3098706Z Entering 'third_party/onnx' 2024-06-26T07:07:21.3130016Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T07:07:21.3153738Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:07:21.3186479Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:07:21.3199300Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:07:21.3229738Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:07:21.3242632Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:07:21.3275913Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T07:07:21.3289039Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:07:21.3320924Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:07:21.3333896Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:07:21.3364928Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:07:21.3377934Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:07:21.3409653Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T07:07:21.3422372Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:07:21.3455275Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T07:07:21.3469743Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:07:21.3501132Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T07:07:21.3513558Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:07:21.3548850Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T07:07:21.3561904Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:07:21.3593004Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T07:07:21.3604848Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:07:21.3636572Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T07:07:21.3650547Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:07:21.3681372Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T07:07:21.3695210Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:07:21.3726611Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T07:07:21.3755800Z Entering 'third_party/pocketfft' 2024-06-26T07:07:21.3787997Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T07:07:21.3800426Z Entering 'third_party/protobuf' 2024-06-26T07:07:21.3833719Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T07:07:21.3848666Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:07:21.3883060Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:07:21.3896424Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:07:21.3929282Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:07:21.3942703Z Entering 'third_party/psimd' 2024-06-26T07:07:21.3975763Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T07:07:21.3989562Z Entering 'third_party/pthreadpool' 2024-06-26T07:07:21.4020202Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T07:07:21.4032762Z Entering 'third_party/pybind11' 2024-06-26T07:07:21.4067037Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:07:21.4080078Z Entering 'third_party/python-peachpy' 2024-06-26T07:07:21.4113941Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T07:07:21.4127116Z Entering 'third_party/sleef' 2024-06-26T07:07:21.4161769Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T07:07:21.4175591Z Entering 'third_party/tensorpipe' 2024-06-26T07:07:21.4207351Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T07:07:21.4220081Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:07:21.4249372Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:07:21.4260603Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:07:21.4291767Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T07:07:21.4303951Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:07:21.4336248Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T07:07:21.4349147Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:07:21.4380175Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:07:21.4392629Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:07:21.4424848Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T07:07:21.5361545Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T07:07:21.5574738Z Entering 'android/libs/fbjni' 2024-06-26T07:07:21.5600777Z Entering 'third_party/FP16' 2024-06-26T07:07:21.5629917Z Entering 'third_party/FXdiv' 2024-06-26T07:07:21.5656217Z Entering 'third_party/NNPACK' 2024-06-26T07:07:21.5681985Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:07:21.5712371Z Entering 'third_party/XNNPACK' 2024-06-26T07:07:21.5749367Z Entering 'third_party/benchmark' 2024-06-26T07:07:21.5778161Z Entering 'third_party/cpp-httplib' 2024-06-26T07:07:21.5805376Z Entering 'third_party/cpuinfo' 2024-06-26T07:07:21.5834136Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:07:21.5862497Z Entering 'third_party/cutlass' 2024-06-26T07:07:21.5897176Z Entering 'third_party/eigen' 2024-06-26T07:07:21.5925860Z Entering 'third_party/fbgemm' 2024-06-26T07:07:21.5953447Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:07:21.5981682Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:07:21.6007924Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:07:21.6041572Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:07:21.6069970Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:07:21.6099630Z Entering 'third_party/flatbuffers' 2024-06-26T07:07:21.6132380Z Entering 'third_party/fmt' 2024-06-26T07:07:21.6162227Z Entering 'third_party/foxi' 2024-06-26T07:07:21.6190475Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:07:21.6219241Z Entering 'third_party/gloo' 2024-06-26T07:07:21.6248868Z Entering 'third_party/googletest' 2024-06-26T07:07:21.6279674Z Entering 'third_party/ideep' 2024-06-26T07:07:21.6307026Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:07:21.6342144Z Entering 'third_party/ittapi' 2024-06-26T07:07:21.6369779Z Entering 'third_party/kineto' 2024-06-26T07:07:21.6397974Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:07:21.6427451Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:07:21.6456566Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:07:21.6485338Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:07:21.6512393Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:07:21.6537886Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:07:21.6566099Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:07:21.6594052Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:07:21.6621976Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:07:21.6649915Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:07:21.6678098Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:07:21.6706856Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:07:21.6738106Z Entering 'third_party/mimalloc' 2024-06-26T07:07:21.6766835Z Entering 'third_party/nccl/nccl' 2024-06-26T07:07:21.6798335Z Entering 'third_party/nlohmann' 2024-06-26T07:07:21.6828084Z Entering 'third_party/onnx' 2024-06-26T07:07:21.6868713Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:07:21.6898715Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:07:21.6930236Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:07:21.6960661Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:07:21.6989346Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:07:21.7017897Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:07:21.7044313Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:07:21.7073920Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:07:21.7103182Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:07:21.7130891Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:07:21.7159480Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:07:21.7187291Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:07:21.7214813Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:07:21.7256945Z Entering 'third_party/pocketfft' 2024-06-26T07:07:21.7283812Z Entering 'third_party/protobuf' 2024-06-26T07:07:21.7315748Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:07:21.7342091Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:07:21.7369812Z Entering 'third_party/psimd' 2024-06-26T07:07:21.7399554Z Entering 'third_party/pthreadpool' 2024-06-26T07:07:21.7429509Z Entering 'third_party/pybind11' 2024-06-26T07:07:21.7455481Z Entering 'third_party/python-peachpy' 2024-06-26T07:07:21.7483699Z Entering 'third_party/sleef' 2024-06-26T07:07:21.7511258Z Entering 'third_party/tensorpipe' 2024-06-26T07:07:21.7540810Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:07:21.7568610Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:07:21.7596245Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:07:21.7624534Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:07:21.7649015Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:07:21.7691015Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T07:07:21.7907856Z Entering 'android/libs/fbjni' 2024-06-26T07:07:21.7938100Z Entering 'third_party/FP16' 2024-06-26T07:07:21.7965165Z Entering 'third_party/FXdiv' 2024-06-26T07:07:21.7992040Z Entering 'third_party/NNPACK' 2024-06-26T07:07:21.8023124Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:07:21.8051030Z Entering 'third_party/XNNPACK' 2024-06-26T07:07:21.8091384Z Entering 'third_party/benchmark' 2024-06-26T07:07:21.8121964Z Entering 'third_party/cpp-httplib' 2024-06-26T07:07:21.8150088Z Entering 'third_party/cpuinfo' 2024-06-26T07:07:21.8177197Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:07:21.8203877Z Entering 'third_party/cutlass' 2024-06-26T07:07:21.8237123Z Entering 'third_party/eigen' 2024-06-26T07:07:21.8268806Z Entering 'third_party/fbgemm' 2024-06-26T07:07:21.8298864Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:07:21.8326204Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:07:21.8354635Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:07:21.8387650Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:07:21.8415976Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:07:21.8444376Z Entering 'third_party/flatbuffers' 2024-06-26T07:07:21.8473828Z Entering 'third_party/fmt' 2024-06-26T07:07:21.8501590Z Entering 'third_party/foxi' 2024-06-26T07:07:21.8528442Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:07:21.8558411Z Entering 'third_party/gloo' 2024-06-26T07:07:21.8586098Z Entering 'third_party/googletest' 2024-06-26T07:07:21.8615481Z Entering 'third_party/ideep' 2024-06-26T07:07:21.8644562Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:07:21.8677106Z Entering 'third_party/ittapi' 2024-06-26T07:07:21.8706064Z Entering 'third_party/kineto' 2024-06-26T07:07:21.8731418Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:07:21.8760602Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:07:21.8788665Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:07:21.8815276Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:07:21.8841117Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:07:21.8868893Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:07:21.8898136Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:07:21.8925742Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:07:21.8953661Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:07:21.8980312Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:07:21.9008052Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:07:21.9036012Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:07:21.9065433Z Entering 'third_party/mimalloc' 2024-06-26T07:07:21.9094869Z Entering 'third_party/nccl/nccl' 2024-06-26T07:07:21.9122091Z Entering 'third_party/nlohmann' 2024-06-26T07:07:21.9152763Z Entering 'third_party/onnx' 2024-06-26T07:07:21.9193218Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:07:21.9221245Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:07:21.9253823Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:07:21.9282834Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:07:21.9311338Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:07:21.9339935Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:07:21.9368797Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:07:21.9399855Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:07:21.9425369Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:07:21.9451388Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:07:21.9479654Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:07:21.9509441Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:07:21.9538478Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:07:21.9582261Z Entering 'third_party/pocketfft' 2024-06-26T07:07:21.9611519Z Entering 'third_party/protobuf' 2024-06-26T07:07:21.9644104Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:07:21.9671186Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:07:21.9699050Z Entering 'third_party/psimd' 2024-06-26T07:07:21.9728376Z Entering 'third_party/pthreadpool' 2024-06-26T07:07:21.9758535Z Entering 'third_party/pybind11' 2024-06-26T07:07:21.9788106Z Entering 'third_party/python-peachpy' 2024-06-26T07:07:21.9816278Z Entering 'third_party/sleef' 2024-06-26T07:07:21.9842565Z Entering 'third_party/tensorpipe' 2024-06-26T07:07:21.9871738Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:07:21.9899451Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:07:21.9927141Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:07:21.9956692Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:07:21.9982814Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:07:22.0021502Z ##[endgroup] 2024-06-26T07:07:22.0049784Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T07:07:22.0067275Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T07:07:22.0246725Z Prepare all required actions 2024-06-26T07:07:22.0247199Z Getting action download info 2024-06-26T07:07:22.1580903Z ##[group]Run ./.github/actions/setup-linux 2024-06-26T07:07:22.1581502Z env: 2024-06-26T07:07:22.1581846Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:22.1582201Z ##[endgroup] 2024-06-26T07:07:22.1639062Z ##[group]Run set -euo pipefail 2024-06-26T07:07:22.1639531Z set -euo pipefail 2024-06-26T07:07:22.1639954Z function get_ec2_metadata() { 2024-06-26T07:07:22.1640541Z  # Pulled from instance metadata endpoint for EC2 2024-06-26T07:07:22.1641518Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-26T07:07:22.1642366Z  category=$1 2024-06-26T07:07:22.1643073Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-26T07:07:22.1643823Z  runner_name_str=i-06b804ca2116ce394 2024-06-26T07:07:22.1644358Z  if [[ -f /.inarc ]]; then 2024-06-26T07:07:22.1644879Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-26T07:07:22.1645544Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-26T07:07:22.1646540Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-26T07:07:22.1647180Z  else 2024-06-26T07:07:22.1647687Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-26T07:07:22.1648291Z  fi 2024-06-26T07:07:22.1648584Z } 2024-06-26T07:07:22.1648951Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-26T07:07:22.1649570Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-26T07:07:22.1650265Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-26T07:07:22.1650867Z echo "system info $(uname -a)" 2024-06-26T07:07:22.1659233Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:22.1659754Z env: 2024-06-26T07:07:22.1660041Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:22.1660399Z ##[endgroup] 2024-06-26T07:07:22.1728407Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-26T07:07:22.1768456Z instance-id: i-06b804ca2116ce394 2024-06-26T07:07:22.1807421Z instance-type: g5.4xlarge 2024-06-26T07:07:22.1814046Z system info Linux ip-10-0-25-189.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-26T07:07:22.1834107Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:07:22.1835112Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:07:22.1843086Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:22.1843607Z env: 2024-06-26T07:07:22.1843896Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:22.1844267Z ##[endgroup] 2024-06-26T07:07:22.1906509Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-26T07:07:22.1907258Z if systemctl is-active --quiet docker; then 2024-06-26T07:07:22.1907895Z  echo "Docker daemon is running..."; 2024-06-26T07:07:22.1908503Z else 2024-06-26T07:07:22.1909026Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-26T07:07:22.1909660Z fi 2024-06-26T07:07:22.1916416Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:22.1916945Z env: 2024-06-26T07:07:22.1917265Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:22.1917647Z ##[endgroup] 2024-06-26T07:07:22.1952841Z Docker daemon is running... 2024-06-26T07:07:22.1999437Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:07:22.2000044Z with: 2024-06-26T07:07:22.2000323Z shell: bash 2024-06-26T07:07:22.2000628Z timeout_minutes: 5 2024-06-26T07:07:22.2000965Z max_attempts: 3 2024-06-26T07:07:22.2001290Z retry_wait_seconds: 30 2024-06-26T07:07:22.2002998Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-26T07:07:22.2004654Z polling_interval_seconds: 1 2024-06-26T07:07:22.2005051Z warning_on_retry: true 2024-06-26T07:07:22.2005418Z continue_on_error: false 2024-06-26T07:07:22.2005768Z env: 2024-06-26T07:07:22.2006039Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:22.2006410Z AWS_RETRY_MODE: standard 2024-06-26T07:07:22.2006776Z AWS_MAX_ATTEMPTS: 5 2024-06-26T07:07:22.2007120Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:07:22.2007516Z ##[endgroup] 2024-06-26T07:07:23.0344610Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:07:23.0345647Z Configure a credential helper to remove this warning. See 2024-06-26T07:07:23.0346657Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:07:23.0347255Z 2024-06-26T07:07:23.0347419Z Login Succeeded 2024-06-26T07:07:23.2508001Z Command completed after 1 attempt(s). 2024-06-26T07:07:23.2549674Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:07:23.2550461Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:07:23.2551175Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:07:23.2559102Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:23.2559635Z env: 2024-06-26T07:07:23.2559928Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:23.2560300Z ##[endgroup] 2024-06-26T07:07:23.2622559Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:07:23.2623408Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:07:23.2624133Z # shellcheck disable=SC2046 2024-06-26T07:07:23.2624626Z docker stop $(docker ps -q) || true 2024-06-26T07:07:23.2625156Z # Prune all of the docker images 2024-06-26T07:07:23.2625634Z docker system prune -af 2024-06-26T07:07:23.2633386Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:23.2633938Z env: 2024-06-26T07:07:23.2634224Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:23.2634592Z ##[endgroup] 2024-06-26T07:07:23.2876545Z "docker stop" requires at least 1 argument. 2024-06-26T07:07:23.2877170Z See 'docker stop --help'. 2024-06-26T07:07:23.2877422Z 2024-06-26T07:07:23.2877656Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T07:07:23.2878508Z 2024-06-26T07:07:23.2878769Z Stop one or more running containers 2024-06-26T07:07:23.3026610Z Total reclaimed space: 0B 2024-06-26T07:07:23.3056513Z ##[group]Run set +e 2024-06-26T07:07:23.3056869Z set +e 2024-06-26T07:07:23.3057171Z set -x 2024-06-26T07:07:23.3057470Z  2024-06-26T07:07:23.3057798Z PT_DOMAIN=download.pytorch.org 2024-06-26T07:07:23.3058665Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-26T07:07:23.3059866Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-26T07:07:23.3060820Z # one is returned at random 2024-06-26T07:07:23.3061399Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-26T07:07:23.3061948Z  2024-06-26T07:07:23.3062270Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:07:23.3062941Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-26T07:07:23.3063849Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-26T07:07:23.3064445Z  2024-06-26T07:07:23.3064774Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:07:23.3065357Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-26T07:07:23.3065898Z  exit 1 2024-06-26T07:07:23.3066210Z  fi 2024-06-26T07:07:23.3066496Z fi 2024-06-26T07:07:23.3066885Z  2024-06-26T07:07:23.3067265Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-26T07:07:23.3067867Z  # Clean up any old records first 2024-06-26T07:07:23.3068403Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-26T07:07:23.3068884Z fi 2024-06-26T07:07:23.3069155Z  2024-06-26T07:07:23.3069600Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-26T07:07:23.3070199Z cat /etc/hosts 2024-06-26T07:07:23.3077888Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:23.3078419Z env: 2024-06-26T07:07:23.3078706Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:23.3079069Z ##[endgroup] 2024-06-26T07:07:23.3101668Z + PT_DOMAIN=download.pytorch.org 2024-06-26T07:07:23.3105986Z ++ dig -4 +short download.pytorch.org 2024-06-26T07:07:23.3106450Z ++ tail -n1 2024-06-26T07:07:23.3167545Z + RESOLVED_IP=18.67.76.42 2024-06-26T07:07:23.3167992Z + '[' -z 18.67.76.42 ']' 2024-06-26T07:07:23.3168453Z + grep -r download.pytorch.org /etc/hosts 2024-06-26T07:07:23.3173909Z 18.160.18.56 download.pytorch.org 2024-06-26T07:07:23.3174683Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-06-26T07:07:23.3270854Z + echo '18.67.76.42 download.pytorch.org' 2024-06-26T07:07:23.3271393Z + sudo tee -a /etc/hosts 2024-06-26T07:07:23.3525929Z 18.67.76.42 download.pytorch.org 2024-06-26T07:07:23.3539286Z + cat /etc/hosts 2024-06-26T07:07:23.3543948Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-26T07:07:23.3556131Z ::1 localhost6 localhost6.localdomain6 2024-06-26T07:07:23.3556674Z 18.67.76.42 download.pytorch.org 2024-06-26T07:07:23.3716799Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T07:07:23.3717441Z with: 2024-06-26T07:07:23.3718671Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:23.3720059Z docker-build-dir: .ci/docker 2024-06-26T07:07:23.3720466Z working-directory: . 2024-06-26T07:07:23.3720994Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:23.3721564Z force-push: false 2024-06-26T07:07:23.3721922Z env: 2024-06-26T07:07:23.3722235Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:23.3722697Z ##[endgroup] 2024-06-26T07:07:23.3798949Z ##[group]Run set -ex 2024-06-26T07:07:23.3799318Z set -ex 2024-06-26T07:07:23.3799615Z  2024-06-26T07:07:23.3800204Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T07:07:23.3801282Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T07:07:23.3802170Z # job could then download the pre-built image as usual 2024-06-26T07:07:23.3802975Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T07:07:23.3803701Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:23.3804383Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:23.3805100Z  2024-06-26T07:07:23.3805636Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T07:07:23.3806321Z  exit 0 2024-06-26T07:07:23.3806623Z else 2024-06-26T07:07:23.3806995Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:23.3807473Z fi 2024-06-26T07:07:23.3807746Z  2024-06-26T07:07:23.3808237Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T07:07:23.3809172Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T07:07:23.3810018Z  # use it as it is, but first let's extract the tag 2024-06-26T07:07:23.3810773Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T07:07:23.3811557Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:23.3812609Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:23.3813227Z else 2024-06-26T07:07:23.3813683Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T07:07:23.3814397Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:23.3815381Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:23.3816211Z fi 2024-06-26T07:07:23.3824017Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:23.3824550Z env: 2024-06-26T07:07:23.3824831Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:23.3825198Z REPO_NAME: pytorch 2024-06-26T07:07:23.3826460Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:23.3827804Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:07:23.3828328Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:23.3828889Z ##[endgroup] 2024-06-26T07:07:23.3849630Z + [[ ! -d .ci/docker ]] 2024-06-26T07:07:23.3850041Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T07:07:23.3850431Z + echo skip=false 2024-06-26T07:07:23.3852831Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T07:07:23.3855626Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:23.3856919Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T07:07:23.3861696Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:23.3862378Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:23.3864311Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:23.3889528Z ##[group]Run set +e 2024-06-26T07:07:23.3889916Z set +e 2024-06-26T07:07:23.3890238Z set -x 2024-06-26T07:07:23.3890549Z  2024-06-26T07:07:23.3890847Z login() { 2024-06-26T07:07:23.3891577Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:07:23.3892645Z } 2024-06-26T07:07:23.3892926Z  2024-06-26T07:07:23.3893212Z retry () { 2024-06-26T07:07:23.3893617Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:07:23.3894099Z } 2024-06-26T07:07:23.3894380Z  2024-06-26T07:07:23.3894709Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:07:23.3895149Z  2024-06-26T07:07:23.3895642Z # Check if image already exists, if it does then skip building it 2024-06-26T07:07:23.3896516Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T07:07:23.3897065Z  exit 0 2024-06-26T07:07:23.3897389Z fi 2024-06-26T07:07:23.3897671Z  2024-06-26T07:07:23.3898218Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T07:07:23.3899132Z # be empty. The default action would be to continue rebuild the image 2024-06-26T07:07:23.3899930Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T07:07:23.3900653Z  # if we're on the base branch then use the parent commit 2024-06-26T07:07:23.3901291Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T07:07:23.3901766Z else 2024-06-26T07:07:23.3902269Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T07:07:23.3902995Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T07:07:23.3903548Z fi 2024-06-26T07:07:23.3903919Z  2024-06-26T07:07:23.3904251Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T07:07:23.3904780Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:23.3905267Z  2024-06-26T07:07:23.3905970Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T07:07:23.3906795Z  exit 0 2024-06-26T07:07:23.3907103Z fi 2024-06-26T07:07:23.3907417Z  2024-06-26T07:07:23.3907906Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T07:07:23.3908978Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T07:07:23.3909853Z  exit 1 2024-06-26T07:07:23.3910170Z fi 2024-06-26T07:07:23.3910464Z  2024-06-26T07:07:23.3910990Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T07:07:23.3912010Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T07:07:23.3912943Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T07:07:23.3913993Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T07:07:23.3915182Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T07:07:23.3915978Z fi 2024-06-26T07:07:23.3916267Z  2024-06-26T07:07:23.3916648Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:23.3923547Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:23.3924099Z env: 2024-06-26T07:07:23.3924388Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:23.3924788Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:07:23.3925307Z BASE_REVISION: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:07:23.3926766Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:23.3928168Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:23.3928845Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:23.3929424Z ##[endgroup] 2024-06-26T07:07:23.3948827Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:23.3949675Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:23.3950546Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:07:23.3951622Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:23.8040560Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:07:23.8041471Z Configure a credential helper to remove this warning. See 2024-06-26T07:07:23.8042653Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:07:23.8043227Z 2024-06-26T07:07:23.8043367Z Login Succeeded 2024-06-26T07:07:23.8053440Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:24.0018501Z { 2024-06-26T07:07:24.0018919Z "schemaVersion": 2, 2024-06-26T07:07:24.0019572Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T07:07:24.0020207Z "config": { 2024-06-26T07:07:24.0020683Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T07:07:24.0021271Z "size": 48681, 2024-06-26T07:07:24.0021840Z "digest": "sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd" 2024-06-26T07:07:24.0022705Z }, 2024-06-26T07:07:24.0023021Z "layers": [ 2024-06-26T07:07:24.0023400Z { 2024-06-26T07:07:24.0023861Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0024482Z "size": 28580681, 2024-06-26T07:07:24.0025074Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-06-26T07:07:24.0025758Z }, 2024-06-26T07:07:24.0026002Z { 2024-06-26T07:07:24.0026456Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0027092Z "size": 7943451, 2024-06-26T07:07:24.0027681Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-06-26T07:07:24.0028356Z }, 2024-06-26T07:07:24.0028617Z { 2024-06-26T07:07:24.0029082Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0029703Z "size": 55728572, 2024-06-26T07:07:24.0030282Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-06-26T07:07:24.0031034Z }, 2024-06-26T07:07:24.0031509Z { 2024-06-26T07:07:24.0032190Z + exit 0 2024-06-26T07:07:24.0032677Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0033284Z "size": 186, 2024-06-26T07:07:24.0033869Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-06-26T07:07:24.0034541Z }, 2024-06-26T07:07:24.0034795Z { 2024-06-26T07:07:24.0035249Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0035848Z "size": 6886, 2024-06-26T07:07:24.0036417Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-06-26T07:07:24.0037301Z }, 2024-06-26T07:07:24.0037575Z { 2024-06-26T07:07:24.0038062Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0038653Z "size": 1329001756, 2024-06-26T07:07:24.0039338Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-06-26T07:07:24.0040033Z }, 2024-06-26T07:07:24.0040288Z { 2024-06-26T07:07:24.0040734Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0041341Z "size": 62501, 2024-06-26T07:07:24.0041922Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-06-26T07:07:24.0042580Z }, 2024-06-26T07:07:24.0042829Z { 2024-06-26T07:07:24.0043280Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0043881Z "size": 1684, 2024-06-26T07:07:24.0044445Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-06-26T07:07:24.0045106Z }, 2024-06-26T07:07:24.0045365Z { 2024-06-26T07:07:24.0045815Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0046405Z "size": 1523, 2024-06-26T07:07:24.0046996Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-06-26T07:07:24.0047684Z }, 2024-06-26T07:07:24.0047926Z { 2024-06-26T07:07:24.0048381Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0048979Z "size": 2528295403, 2024-06-26T07:07:24.0049701Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-06-26T07:07:24.0050363Z }, 2024-06-26T07:07:24.0050621Z { 2024-06-26T07:07:24.0051074Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0051668Z "size": 86016, 2024-06-26T07:07:24.0052532Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-06-26T07:07:24.0053303Z }, 2024-06-26T07:07:24.0053559Z { 2024-06-26T07:07:24.0054016Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0054602Z "size": 1823, 2024-06-26T07:07:24.0055165Z "digest": "sha256:309b872da805fcdf74137a887464876fa0615dc8b24199307119eb65fc2bfb49" 2024-06-26T07:07:24.0055827Z }, 2024-06-26T07:07:24.0056070Z { 2024-06-26T07:07:24.0056519Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0057121Z "size": 246737542, 2024-06-26T07:07:24.0057712Z "digest": "sha256:e82932cb002c06a782077b8e5965f8733eb8563406a23cab260d9d8c4a994f7d" 2024-06-26T07:07:24.0058375Z }, 2024-06-26T07:07:24.0058636Z { 2024-06-26T07:07:24.0059086Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0059694Z "size": 545, 2024-06-26T07:07:24.0060277Z "digest": "sha256:ca9dd73e3e9378225ebdcfd12199ef44649e34fe55003cfbc8afe2e8a4527c5b" 2024-06-26T07:07:24.0060963Z }, 2024-06-26T07:07:24.0061215Z { 2024-06-26T07:07:24.0061672Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0062271Z "size": 1283, 2024-06-26T07:07:24.0062836Z "digest": "sha256:16d76a79d5d6616e228537785ec83a869f68a402520bbc6e826201621ca6cb98" 2024-06-26T07:07:24.0063569Z }, 2024-06-26T07:07:24.0063815Z { 2024-06-26T07:07:24.0064268Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0064864Z "size": 485, 2024-06-26T07:07:24.0065448Z "digest": "sha256:4bdf41c9ecbe1651e0986ccbea4663a0556af27b7145a679cfa45c35dc96b891" 2024-06-26T07:07:24.0066137Z }, 2024-06-26T07:07:24.0066394Z { 2024-06-26T07:07:24.0066850Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0067451Z "size": 91710937, 2024-06-26T07:07:24.0068056Z "digest": "sha256:f4863801e30cdbcd1d4e6cb1cbbcde7dbcaf6ae317ca80429295741ce60ac31d" 2024-06-26T07:07:24.0068747Z }, 2024-06-26T07:07:24.0069012Z { 2024-06-26T07:07:24.0069467Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0070168Z "size": 3233, 2024-06-26T07:07:24.0070766Z "digest": "sha256:d37dd9ed8948a0d02e1198bbedadbca36477a0980dd553196385aab0c025581d" 2024-06-26T07:07:24.0071450Z }, 2024-06-26T07:07:24.0071696Z { 2024-06-26T07:07:24.0072154Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0072757Z "size": 1905, 2024-06-26T07:07:24.0073355Z "digest": "sha256:cf2f34f225dbed17c1a9a8b9ea4ad030aba797774a716dc0a82fe212324aa751" 2024-06-26T07:07:24.0074040Z }, 2024-06-26T07:07:24.0074305Z { 2024-06-26T07:07:24.0074761Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0075348Z "size": 702, 2024-06-26T07:07:24.0075934Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:07:24.0076614Z }, 2024-06-26T07:07:24.0076875Z { 2024-06-26T07:07:24.0077326Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0077985Z "size": 2784789002, 2024-06-26T07:07:24.0078603Z "digest": "sha256:3d914950f8addfc4d657a2365c76e1143eb80b710cd3f6e9ae05640217fc6347" 2024-06-26T07:07:24.0079274Z }, 2024-06-26T07:07:24.0079520Z { 2024-06-26T07:07:24.0079989Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0080600Z "size": 381, 2024-06-26T07:07:24.0081179Z "digest": "sha256:96d0e2032184c57e62a652aaffcf0241f0d0c7a23e894ca0ad192c4a64a36583" 2024-06-26T07:07:24.0081849Z }, 2024-06-26T07:07:24.0082179Z { 2024-06-26T07:07:24.0082635Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0083230Z "size": 11879, 2024-06-26T07:07:24.0083814Z "digest": "sha256:06e12ce59790cb8b091e5d619c2a62a28329008ee184a8b9a562e3db12471f30" 2024-06-26T07:07:24.0084488Z }, 2024-06-26T07:07:24.0084750Z { 2024-06-26T07:07:24.0085200Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0085802Z "size": 803, 2024-06-26T07:07:24.0086390Z "digest": "sha256:ed712f63cde7070bdec2a6179c4c35e752c6991ddbc69fe9f2724a9903250ceb" 2024-06-26T07:07:24.0087074Z }, 2024-06-26T07:07:24.0087318Z { 2024-06-26T07:07:24.0087781Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0088435Z "size": 106, 2024-06-26T07:07:24.0089016Z "digest": "sha256:a97a71ef8a4e66234dfc2098834f8395ac5630240e60164d1a251fdb0115ba85" 2024-06-26T07:07:24.0089677Z }, 2024-06-26T07:07:24.0089934Z { 2024-06-26T07:07:24.0090402Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0090997Z "size": 503, 2024-06-26T07:07:24.0091566Z "digest": "sha256:25cd0c482ad612f936b35b4f220c580e16be982226aac8801a6a849a6bfdc221" 2024-06-26T07:07:24.0092360Z }, 2024-06-26T07:07:24.0092613Z { 2024-06-26T07:07:24.0093057Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0093663Z "size": 121477177, 2024-06-26T07:07:24.0094276Z "digest": "sha256:9bf94bc456584365fda21d0ed83b7cc690b08a71727b71c4c1e667e15028a386" 2024-06-26T07:07:24.0094951Z }, 2024-06-26T07:07:24.0095201Z { 2024-06-26T07:07:24.0095662Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0096264Z "size": 109, 2024-06-26T07:07:24.0096839Z "digest": "sha256:de45a14c9b227e4b0c9d45d42223b50a71c03522ddd9e2ca0d9517835206c823" 2024-06-26T07:07:24.0097529Z }, 2024-06-26T07:07:24.0097813Z { 2024-06-26T07:07:24.0098271Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0098868Z "size": 491, 2024-06-26T07:07:24.0099445Z "digest": "sha256:e99241e77209b0b35a0fb59a61c0821fc161190178af7e4af45e6eeaa043375e" 2024-06-26T07:07:24.0100113Z }, 2024-06-26T07:07:24.0100377Z { 2024-06-26T07:07:24.0100827Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0101435Z "size": 291, 2024-06-26T07:07:24.0102028Z "digest": "sha256:35e8f4e2a42f9df0c969dedd2199cf71719cf5ecfa676dfa014bbdeb4fdb295f" 2024-06-26T07:07:24.0102821Z }, 2024-06-26T07:07:24.0103066Z { 2024-06-26T07:07:24.0103572Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0104172Z "size": 101, 2024-06-26T07:07:24.0104732Z "digest": "sha256:a823ffa4c541394b52953391dc0a6867e26c37fe4a71fa022a2cc3f07f0ba70d" 2024-06-26T07:07:24.0105400Z }, 2024-06-26T07:07:24.0105660Z { 2024-06-26T07:07:24.0106119Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0106718Z "size": 1473, 2024-06-26T07:07:24.0107292Z "digest": "sha256:44ba272eae2064603605a6238b3f35cc6578859d660068e702a047a9cd8ab2fd" 2024-06-26T07:07:24.0108009Z }, 2024-06-26T07:07:24.0108267Z { 2024-06-26T07:07:24.0108713Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0109322Z "size": 446710781, 2024-06-26T07:07:24.0109925Z "digest": "sha256:da02d278e5240c285bc855ce4e42cd6121d5b903787f5c14f03e575ad1268650" 2024-06-26T07:07:24.0110583Z }, 2024-06-26T07:07:24.0110859Z { 2024-06-26T07:07:24.0111315Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0111922Z "size": 161, 2024-06-26T07:07:24.0112483Z "digest": "sha256:73a0ce681c295135456e190578733273276b38b4dd5babbb39298f384d43b1c6" 2024-06-26T07:07:24.0113148Z }, 2024-06-26T07:07:24.0113411Z { 2024-06-26T07:07:24.0113872Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0114459Z "size": 564, 2024-06-26T07:07:24.0115121Z "digest": "sha256:49737bd5ae709dbf7956e21fc8b05ba009e9e6c9b130b0126a73bd3d3f5efc78" 2024-06-26T07:07:24.0115805Z }, 2024-06-26T07:07:24.0116055Z { 2024-06-26T07:07:24.0116508Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0117120Z "size": 35874600, 2024-06-26T07:07:24.0117733Z "digest": "sha256:9cd4c745f8e4f9d5c9bb8050923575afad21f5b7852975a4a1981bc9eff282c5" 2024-06-26T07:07:24.0118405Z }, 2024-06-26T07:07:24.0118669Z { 2024-06-26T07:07:24.0119145Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0119750Z "size": 104, 2024-06-26T07:07:24.0120312Z "digest": "sha256:b2950a0365345d4c17a48627c905fd8930b3b3154c727f0526eb9ca547102bb7" 2024-06-26T07:07:24.0120981Z }, 2024-06-26T07:07:24.0121251Z { 2024-06-26T07:07:24.0121713Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0122316Z "size": 425, 2024-06-26T07:07:24.0122892Z "digest": "sha256:25e19d81d9fc081962ff4cbc09e098f4035419822712c19a410f5282300dfb9e" 2024-06-26T07:07:24.0123579Z }, 2024-06-26T07:07:24.0123825Z { 2024-06-26T07:07:24.0124287Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0124899Z "size": 20262084, 2024-06-26T07:07:24.0125518Z "digest": "sha256:60156b40fcbec2c25f0170b0a82fa8cd9303e0daae8bb1811fd0fc6562e88e8e" 2024-06-26T07:07:24.0126198Z }, 2024-06-26T07:07:24.0126470Z { 2024-06-26T07:07:24.0126948Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0127566Z "size": 439, 2024-06-26T07:07:24.0128136Z "digest": "sha256:7a868239227fd19b30a19927bbc8f058d247403fe5b343c55f4b4b8987918911" 2024-06-26T07:07:24.0128809Z }, 2024-06-26T07:07:24.0129075Z { 2024-06-26T07:07:24.0129534Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0130150Z "size": 702, 2024-06-26T07:07:24.0130754Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:07:24.0131458Z }, 2024-06-26T07:07:24.0131710Z { 2024-06-26T07:07:24.0132363Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0132978Z "size": 143, 2024-06-26T07:07:24.0133568Z "digest": "sha256:83911d7c7ecf11b5932bb7449975c52e7f737a5b867691935e1ccb85ceb25b52" 2024-06-26T07:07:24.0134229Z }, 2024-06-26T07:07:24.0134490Z { 2024-06-26T07:07:24.0134968Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0135650Z "size": 135, 2024-06-26T07:07:24.0136244Z "digest": "sha256:b3c5d66c679f58c71febdd502889bd8cd3d36e5001754cdf1e30eabe21135cce" 2024-06-26T07:07:24.0136939Z }, 2024-06-26T07:07:24.0137203Z { 2024-06-26T07:07:24.0137655Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0138267Z "size": 4911495247, 2024-06-26T07:07:24.0138870Z "digest": "sha256:84b2c7a899820f6acdb22148c5e8346d8b26c33269fd01842c9573515251e1e1" 2024-06-26T07:07:24.0139545Z }, 2024-06-26T07:07:24.0139804Z { 2024-06-26T07:07:24.0140280Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0140894Z "size": 190, 2024-06-26T07:07:24.0141500Z "digest": "sha256:c1068d03a811d8dead5da9e82fdc8869205fc4ee9cb92d4f7a4acfbb5ba83950" 2024-06-26T07:07:24.0142183Z }, 2024-06-26T07:07:24.0142451Z { 2024-06-26T07:07:24.0142924Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0143566Z "size": 565, 2024-06-26T07:07:24.0144138Z "digest": "sha256:671b4a301d81074fca17d36851f81801552c4ecb2f90ff3412aa9ab6a1a91371" 2024-06-26T07:07:24.0144806Z }, 2024-06-26T07:07:24.0145066Z { 2024-06-26T07:07:24.0145515Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0146129Z "size": 43163095, 2024-06-26T07:07:24.0146754Z "digest": "sha256:67760cf5ec46ab58d3ca98637fd9b26cb47a67446bfd0db176fe5e023606aa0d" 2024-06-26T07:07:24.0147432Z }, 2024-06-26T07:07:24.0147755Z { 2024-06-26T07:07:24.0148223Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0148840Z "size": 106, 2024-06-26T07:07:24.0149437Z "digest": "sha256:ddc9740bcea2b301e99b50525059aea8d9c7121a1c8eab566561a8968438a8c2" 2024-06-26T07:07:24.0150106Z }, 2024-06-26T07:07:24.0150375Z { 2024-06-26T07:07:24.0150851Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0151478Z "size": 1213, 2024-06-26T07:07:24.0152083Z "digest": "sha256:fb2e977685609fcc01de170ba7f50a573e73e6520e47253717e92a28952c5ee8" 2024-06-26T07:07:24.0152763Z }, 2024-06-26T07:07:24.0153020Z { 2024-06-26T07:07:24.0153476Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0154080Z "size": 702, 2024-06-26T07:07:24.0154685Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:07:24.0155374Z }, 2024-06-26T07:07:24.0155625Z { 2024-06-26T07:07:24.0156093Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0156704Z "size": 138, 2024-06-26T07:07:24.0157299Z "digest": "sha256:b599c5a8c7cb461cbd7b3c98b99960b8eb04506e05e79db1cc0cfadb14a47e91" 2024-06-26T07:07:24.0157983Z }, 2024-06-26T07:07:24.0158249Z { 2024-06-26T07:07:24.0158722Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0159337Z "size": 120, 2024-06-26T07:07:24.0159943Z "digest": "sha256:e08e88c1366827455ee7ddbf92b6ae615ceb3cbd3fb11cd03ddd31bd3fa94a9c" 2024-06-26T07:07:24.0160641Z }, 2024-06-26T07:07:24.0160906Z { 2024-06-26T07:07:24.0161356Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0161982Z "size": 1904157495, 2024-06-26T07:07:24.0162608Z "digest": "sha256:12ba7431ff38de8cf6af3052a2a52e79f4e712b8c99f5bafde3c20e63b1f570f" 2024-06-26T07:07:24.0163295Z }, 2024-06-26T07:07:24.0163550Z { 2024-06-26T07:07:24.0164013Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0164632Z "size": 172, 2024-06-26T07:07:24.0165231Z "digest": "sha256:cf7c2cb896cba6a594028cc0fd246e127a2f4481d48905a3aa12c5f5872332d4" 2024-06-26T07:07:24.0165908Z }, 2024-06-26T07:07:24.0166169Z { 2024-06-26T07:07:24.0166624Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0167226Z "size": 908, 2024-06-26T07:07:24.0167825Z "digest": "sha256:3a778ddfac2af5b3a11f49ddf2f3f1ad1731f74ba691f434ab69966d1e2a189f" 2024-06-26T07:07:24.0168598Z }, 2024-06-26T07:07:24.0168890Z { 2024-06-26T07:07:24.0169336Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0169954Z "size": 702, 2024-06-26T07:07:24.0170537Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:07:24.0171218Z }, 2024-06-26T07:07:24.0171466Z { 2024-06-26T07:07:24.0172033Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0172657Z "size": 134, 2024-06-26T07:07:24.0173229Z "digest": "sha256:236c8f0245f4b40616a4322eaf1e1b3e178d5990db1fcaba39622ea4e79c06a1" 2024-06-26T07:07:24.0173899Z }, 2024-06-26T07:07:24.0174160Z { 2024-06-26T07:07:24.0174621Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0175317Z "size": 32, 2024-06-26T07:07:24.0175991Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:07:24.0176720Z }, 2024-06-26T07:07:24.0176985Z { 2024-06-26T07:07:24.0177490Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0178137Z "size": 156, 2024-06-26T07:07:24.0178770Z "digest": "sha256:dda1ce302b9a9df0c8f1145d2e3eaa03c96909bb45a792451f1fcb555b06738d" 2024-06-26T07:07:24.0179473Z }, 2024-06-26T07:07:24.0179721Z { 2024-06-26T07:07:24.0180186Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0180794Z "size": 1841, 2024-06-26T07:07:24.0181488Z "digest": "sha256:e797fe86caef19319e03f8dca92cf96838ab3bb3644f4447c6dfc04230118afa" 2024-06-26T07:07:24.0182157Z }, 2024-06-26T07:07:24.0182415Z { 2024-06-26T07:07:24.0182867Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0183551Z "size": 7529784, 2024-06-26T07:07:24.0184140Z "digest": "sha256:967190076fc0ab2bb2fc0d74e58edd3d662a537d702ff3f8b6e876230154a16f" 2024-06-26T07:07:24.0184812Z }, 2024-06-26T07:07:24.0185070Z { 2024-06-26T07:07:24.0185568Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0186204Z "size": 165, 2024-06-26T07:07:24.0186783Z "digest": "sha256:4fb0a83f06855534ef82b80dcf7c685303edb2bcbd446196066dc2ad2dd71281" 2024-06-26T07:07:24.0187464Z }, 2024-06-26T07:07:24.0187719Z { 2024-06-26T07:07:24.0188175Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0188780Z "size": 7944, 2024-06-26T07:07:24.0189363Z "digest": "sha256:68af2ec4e30a33f26947ece9f3a31b7d3b04c2bca5865d4b48618457832a1da2" 2024-06-26T07:07:24.0190042Z }, 2024-06-26T07:07:24.0190307Z { 2024-06-26T07:07:24.0190761Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0191365Z "size": 8070, 2024-06-26T07:07:24.0191938Z "digest": "sha256:cc91933e985a044c6df53caa15488d2285896f5e991f38ec068c253ceec53f14" 2024-06-26T07:07:24.0192610Z }, 2024-06-26T07:07:24.0192865Z { 2024-06-26T07:07:24.0193324Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0193927Z "size": 303, 2024-06-26T07:07:24.0194505Z "digest": "sha256:9f3187916e7dcad33cc572f8f0e949d2f507b887f99cfd6acefd14826b44960d" 2024-06-26T07:07:24.0195180Z }, 2024-06-26T07:07:24.0195431Z { 2024-06-26T07:07:24.0195892Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0196499Z "size": 7629071, 2024-06-26T07:07:24.0197069Z "digest": "sha256:52608399485738d10d779254946c9e1d536010dd45593c69bf1eb9f62f85dc24" 2024-06-26T07:07:24.0197715Z }, 2024-06-26T07:07:24.0197966Z { 2024-06-26T07:07:24.0198426Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0199032Z "size": 108, 2024-06-26T07:07:24.0199607Z "digest": "sha256:05a6f6dcc1c00bac6660923a22adc7d7c34416718f8c89daf9b3351d9365f694" 2024-06-26T07:07:24.0200288Z }, 2024-06-26T07:07:24.0200544Z { 2024-06-26T07:07:24.0200995Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0201684Z "size": 54145775, 2024-06-26T07:07:24.0202288Z "digest": "sha256:6bc9dc8315b09b27c976bfba9700767ba412d72fd0528b18bbe8f8e62005ec22" 2024-06-26T07:07:24.0202964Z }, 2024-06-26T07:07:24.0203206Z { 2024-06-26T07:07:24.0203664Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0204260Z "size": 474, 2024-06-26T07:07:24.0204838Z "digest": "sha256:1b0e27c35f07b3fbf0aeb548d1949bf1d60a420136da21789d94d196c891d981" 2024-06-26T07:07:24.0205518Z }, 2024-06-26T07:07:24.0205782Z { 2024-06-26T07:07:24.0206233Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0206836Z "size": 1374858747, 2024-06-26T07:07:24.0207443Z "digest": "sha256:95237e7311a1c5ce6937db7196c866632f44e8788514a5c0eb4a3f04da4017e6" 2024-06-26T07:07:24.0208110Z }, 2024-06-26T07:07:24.0208377Z { 2024-06-26T07:07:24.0208833Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0209442Z "size": 106, 2024-06-26T07:07:24.0210045Z "digest": "sha256:789eff6c51aada70bf75eb1acafed63c0d839b768638001500616ff6e53b854a" 2024-06-26T07:07:24.0210745Z }, 2024-06-26T07:07:24.0210997Z { 2024-06-26T07:07:24.0211476Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0212202Z "size": 558, 2024-06-26T07:07:24.0212793Z "digest": "sha256:de0f8a8c45c6077b3c5ba3b84d7f07de98353fd618dad9a1e2f368ffe00493c7" 2024-06-26T07:07:24.0213478Z }, 2024-06-26T07:07:24.0213826Z { 2024-06-26T07:07:24.0214297Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0214900Z "size": 46248513, 2024-06-26T07:07:24.0215513Z "digest": "sha256:2060028f74888d218bebdd10d723f66c33ee1c708935c4ffd4654d34912ac4dc" 2024-06-26T07:07:24.0216182Z }, 2024-06-26T07:07:24.0216445Z { 2024-06-26T07:07:24.0216905Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0217523Z "size": 111, 2024-06-26T07:07:24.0218124Z "digest": "sha256:e412c7f3b2ded440992955bd92512345cbece8a8894cd3592f714ebc758ad1a0" 2024-06-26T07:07:24.0218796Z }, 2024-06-26T07:07:24.0219045Z { 2024-06-26T07:07:24.0219507Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0220103Z "size": 32, 2024-06-26T07:07:24.0220677Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:07:24.0221354Z }, 2024-06-26T07:07:24.0221614Z { 2024-06-26T07:07:24.0222068Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0222670Z "size": 32, 2024-06-26T07:07:24.0223255Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:07:24.0224012Z }, 2024-06-26T07:07:24.0224262Z { 2024-06-26T07:07:24.0224704Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0225304Z "size": 32, 2024-06-26T07:07:24.0225885Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:07:24.0226559Z }, 2024-06-26T07:07:24.0226805Z { 2024-06-26T07:07:24.0227268Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:24.0227868Z "size": 32, 2024-06-26T07:07:24.0228436Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:07:24.0229125Z } 2024-06-26T07:07:24.0229380Z ] 2024-06-26T07:07:24.0229638Z } 2024-06-26T07:07:24.0329892Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:07:24.0330423Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:07:24.0331004Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-26T07:07:24.0339187Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:24.0339724Z env: 2024-06-26T07:07:24.0340015Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:24.0341304Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:24.0342624Z ##[endgroup] 2024-06-26T07:07:24.0378987Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:24.0430290Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T07:07:24.0430924Z with: 2024-06-26T07:07:24.0432126Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:24.0433656Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:24.0434216Z env: 2024-06-26T07:07:24.0434499Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:24.0434868Z ##[endgroup] 2024-06-26T07:07:24.0450683Z ##[group]Run set -x 2024-06-26T07:07:24.0451030Z set -x 2024-06-26T07:07:24.0451342Z set +e 2024-06-26T07:07:24.0451643Z  2024-06-26T07:07:24.0452081Z login() { 2024-06-26T07:07:24.0452806Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:07:24.0453600Z } 2024-06-26T07:07:24.0453883Z  2024-06-26T07:07:24.0454190Z retry () { 2024-06-26T07:07:24.0454594Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:07:24.0455071Z } 2024-06-26T07:07:24.0455361Z  2024-06-26T07:07:24.0455687Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:07:24.0456224Z  2024-06-26T07:07:24.0456503Z set -e 2024-06-26T07:07:24.0457016Z # ignore output since only exit code is used for conditional 2024-06-26T07:07:24.0457843Z # only pull docker image if it's not available locally 2024-06-26T07:07:24.0458686Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T07:07:24.0459446Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T07:07:24.0459916Z fi 2024-06-26T07:07:24.0466879Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:24.0467420Z env: 2024-06-26T07:07:24.0467706Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:24.0468994Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:24.0470463Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:24.0471030Z ##[endgroup] 2024-06-26T07:07:24.0491107Z + set +e 2024-06-26T07:07:24.0491704Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:24.0492501Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:24.0494357Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:07:24.0495179Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:24.4662893Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:07:24.4663933Z Configure a credential helper to remove this warning. See 2024-06-26T07:07:24.4665014Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:07:24.4665763Z 2024-06-26T07:07:24.4665932Z Login Succeeded 2024-06-26T07:07:24.4675389Z + set -e 2024-06-26T07:07:24.4676827Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:24.4801189Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:24.4803585Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:24.6769543Z 91382da70d5719cd7007b6b80b71d2f48398f6b7: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-06-26T07:07:24.6771081Z 7a2c55901189: Pulling fs layer 2024-06-26T07:07:24.6771488Z 224fe954d725: Pulling fs layer 2024-06-26T07:07:24.6772342Z 75722010b82e: Pulling fs layer 2024-06-26T07:07:24.6772828Z d527cbbb87e3: Pulling fs layer 2024-06-26T07:07:24.6773243Z b57676e46aee: Pulling fs layer 2024-06-26T07:07:24.6773636Z a8c1e85b5e14: Pulling fs layer 2024-06-26T07:07:24.6774041Z a41a8d1c11c8: Pulling fs layer 2024-06-26T07:07:24.6774426Z 0c1227890755: Pulling fs layer 2024-06-26T07:07:24.6774803Z d8d1234baab3: Pulling fs layer 2024-06-26T07:07:24.6775190Z 7ed32bc8e469: Pulling fs layer 2024-06-26T07:07:24.6775574Z ec1e7978c1fe: Pulling fs layer 2024-06-26T07:07:24.6775954Z 309b872da805: Pulling fs layer 2024-06-26T07:07:24.6776335Z e82932cb002c: Pulling fs layer 2024-06-26T07:07:24.6776721Z ca9dd73e3e93: Pulling fs layer 2024-06-26T07:07:24.6777104Z 16d76a79d5d6: Pulling fs layer 2024-06-26T07:07:24.6777488Z 4bdf41c9ecbe: Pulling fs layer 2024-06-26T07:07:24.6777877Z f4863801e30c: Pulling fs layer 2024-06-26T07:07:24.6778261Z d37dd9ed8948: Pulling fs layer 2024-06-26T07:07:24.6778652Z cf2f34f225db: Pulling fs layer 2024-06-26T07:07:24.6779051Z b57676e46aee: Waiting 2024-06-26T07:07:24.6779509Z df546d7b3e34: Pulling fs layer 2024-06-26T07:07:24.6780022Z a8c1e85b5e14: Waiting 2024-06-26T07:07:24.6780568Z a41a8d1c11c8: Waiting 2024-06-26T07:07:24.6781083Z 3d914950f8ad: Pulling fs layer 2024-06-26T07:07:24.6781839Z 96d0e2032184: Pulling fs layer 2024-06-26T07:07:24.6782278Z 06e12ce59790: Pulling fs layer 2024-06-26T07:07:24.6782666Z ed712f63cde7: Pulling fs layer 2024-06-26T07:07:24.6783041Z 0c1227890755: Waiting 2024-06-26T07:07:24.6783462Z a97a71ef8a4e: Pulling fs layer 2024-06-26T07:07:24.6784252Z 25cd0c482ad6: Pulling fs layer 2024-06-26T07:07:24.6784815Z 9bf94bc45658: Pulling fs layer 2024-06-26T07:07:24.6785391Z de45a14c9b22: Pulling fs layer 2024-06-26T07:07:24.6785926Z e99241e77209: Pulling fs layer 2024-06-26T07:07:24.6786317Z 35e8f4e2a42f: Pulling fs layer 2024-06-26T07:07:24.6786779Z a823ffa4c541: Pulling fs layer 2024-06-26T07:07:24.6787216Z 44ba272eae20: Pulling fs layer 2024-06-26T07:07:24.6787766Z da02d278e524: Pulling fs layer 2024-06-26T07:07:24.6788403Z 73a0ce681c29: Pulling fs layer 2024-06-26T07:07:24.6788966Z d8d1234baab3: Waiting 2024-06-26T07:07:24.6789532Z 49737bd5ae70: Pulling fs layer 2024-06-26T07:07:24.6789948Z 9cd4c745f8e4: Pulling fs layer 2024-06-26T07:07:24.6790328Z 7ed32bc8e469: Waiting 2024-06-26T07:07:24.6790685Z b2950a036534: Pulling fs layer 2024-06-26T07:07:24.6791079Z 25e19d81d9fc: Pulling fs layer 2024-06-26T07:07:24.6791450Z ca9dd73e3e93: Waiting 2024-06-26T07:07:24.6791950Z d37dd9ed8948: Waiting 2024-06-26T07:07:24.6792653Z 60156b40fcbe: Pulling fs layer 2024-06-26T07:07:24.6793340Z ec1e7978c1fe: Waiting 2024-06-26T07:07:24.6793759Z 7a868239227f: Pulling fs layer 2024-06-26T07:07:24.6794129Z 16d76a79d5d6: Waiting 2024-06-26T07:07:24.6794468Z cf2f34f225db: Waiting 2024-06-26T07:07:24.6794875Z 83911d7c7ecf: Pulling fs layer 2024-06-26T07:07:24.6795415Z 4bdf41c9ecbe: Waiting 2024-06-26T07:07:24.6795909Z b3c5d66c679f: Pulling fs layer 2024-06-26T07:07:24.6796439Z 309b872da805: Waiting 2024-06-26T07:07:24.6796892Z f4863801e30c: Waiting 2024-06-26T07:07:24.6797282Z df546d7b3e34: Waiting 2024-06-26T07:07:24.6797619Z e82932cb002c: Waiting 2024-06-26T07:07:24.6797963Z 84b2c7a89982: Pulling fs layer 2024-06-26T07:07:24.6798360Z c1068d03a811: Pulling fs layer 2024-06-26T07:07:24.6798735Z 3d914950f8ad: Waiting 2024-06-26T07:07:24.6799201Z d527cbbb87e3: Waiting 2024-06-26T07:07:24.6799551Z 671b4a301d81: Pulling fs layer 2024-06-26T07:07:24.6799970Z 60156b40fcbe: Waiting 2024-06-26T07:07:24.6800362Z 25cd0c482ad6: Waiting 2024-06-26T07:07:24.6800693Z ed712f63cde7: Waiting 2024-06-26T07:07:24.6801111Z 67760cf5ec46: Pulling fs layer 2024-06-26T07:07:24.6801510Z ddc9740bcea2: Pulling fs layer 2024-06-26T07:07:24.6801899Z 49737bd5ae70: Waiting 2024-06-26T07:07:24.6802245Z fb2e97768560: Pulling fs layer 2024-06-26T07:07:24.6802690Z a823ffa4c541: Waiting 2024-06-26T07:07:24.6803031Z b599c5a8c7cb: Pulling fs layer 2024-06-26T07:07:24.6803528Z 96d0e2032184: Waiting 2024-06-26T07:07:24.6803907Z e08e88c13668: Pulling fs layer 2024-06-26T07:07:24.6804441Z 12ba7431ff38: Pulling fs layer 2024-06-26T07:07:24.6804865Z 06e12ce59790: Waiting 2024-06-26T07:07:24.6805213Z cf7c2cb896cb: Pulling fs layer 2024-06-26T07:07:24.6805619Z 3a778ddfac2a: Pulling fs layer 2024-06-26T07:07:24.6806005Z 44ba272eae20: Waiting 2024-06-26T07:07:24.6806334Z e99241e77209: Waiting 2024-06-26T07:07:24.6806662Z 35e8f4e2a42f: Waiting 2024-06-26T07:07:24.6807006Z 236c8f0245f4: Pulling fs layer 2024-06-26T07:07:24.6807396Z 4f4fb700ef54: Pulling fs layer 2024-06-26T07:07:24.6807801Z dda1ce302b9a: Pulling fs layer 2024-06-26T07:07:24.6808205Z e797fe86caef: Pulling fs layer 2024-06-26T07:07:24.6808577Z da02d278e524: Waiting 2024-06-26T07:07:24.6808902Z 83911d7c7ecf: Waiting 2024-06-26T07:07:24.6809228Z 7a868239227f: Waiting 2024-06-26T07:07:24.6809675Z 73a0ce681c29: Waiting 2024-06-26T07:07:24.6809998Z c1068d03a811: Waiting 2024-06-26T07:07:24.6810374Z 84b2c7a89982: Waiting 2024-06-26T07:07:24.6810709Z 9cd4c745f8e4: Waiting 2024-06-26T07:07:24.6811030Z b2950a036534: Waiting 2024-06-26T07:07:24.6811362Z a97a71ef8a4e: Waiting 2024-06-26T07:07:24.6811696Z 25e19d81d9fc: Waiting 2024-06-26T07:07:24.6812204Z b599c5a8c7cb: Waiting 2024-06-26T07:07:24.6812535Z de45a14c9b22: Waiting 2024-06-26T07:07:24.6812859Z 67760cf5ec46: Waiting 2024-06-26T07:07:24.6813321Z 967190076fc0: Pulling fs layer 2024-06-26T07:07:24.6813704Z e08e88c13668: Waiting 2024-06-26T07:07:24.6814029Z ddc9740bcea2: Waiting 2024-06-26T07:07:24.6814367Z 9bf94bc45658: Waiting 2024-06-26T07:07:24.6814707Z 4fb0a83f0685: Pulling fs layer 2024-06-26T07:07:24.6815104Z 68af2ec4e30a: Pulling fs layer 2024-06-26T07:07:24.6815503Z cc91933e985a: Pulling fs layer 2024-06-26T07:07:24.6815890Z cf7c2cb896cb: Waiting 2024-06-26T07:07:24.6816238Z 9f3187916e7d: Pulling fs layer 2024-06-26T07:07:24.6816638Z 526083994857: Pulling fs layer 2024-06-26T07:07:24.6817024Z fb2e97768560: Waiting 2024-06-26T07:07:24.6817368Z 05a6f6dcc1c0: Pulling fs layer 2024-06-26T07:07:24.6817765Z 6bc9dc8315b0: Pulling fs layer 2024-06-26T07:07:24.6818153Z 68af2ec4e30a: Waiting 2024-06-26T07:07:24.6818496Z 1b0e27c35f07: Pulling fs layer 2024-06-26T07:07:24.6818896Z 4fb0a83f0685: Waiting 2024-06-26T07:07:24.6819333Z e797fe86caef: Waiting 2024-06-26T07:07:24.6819930Z 95237e7311a1: Pulling fs layer 2024-06-26T07:07:24.6820486Z 967190076fc0: Waiting 2024-06-26T07:07:24.6820950Z cc91933e985a: Waiting 2024-06-26T07:07:24.6821429Z 789eff6c51aa: Pulling fs layer 2024-06-26T07:07:24.6821962Z 9f3187916e7d: Waiting 2024-06-26T07:07:24.6822426Z 236c8f0245f4: Waiting 2024-06-26T07:07:24.6822892Z de0f8a8c45c6: Pulling fs layer 2024-06-26T07:07:24.6823546Z 2060028f7488: Pulling fs layer 2024-06-26T07:07:24.6824067Z 95237e7311a1: Waiting 2024-06-26T07:07:24.6824521Z 05a6f6dcc1c0: Waiting 2024-06-26T07:07:24.6825011Z e412c7f3b2de: Pulling fs layer 2024-06-26T07:07:24.6825560Z 3a778ddfac2a: Waiting 2024-06-26T07:07:24.6826029Z 1b0e27c35f07: Waiting 2024-06-26T07:07:24.6826486Z de0f8a8c45c6: Waiting 2024-06-26T07:07:24.6826938Z 2060028f7488: Waiting 2024-06-26T07:07:24.6827405Z 4f4fb700ef54: Waiting 2024-06-26T07:07:24.8066290Z 224fe954d725: Verifying Checksum 2024-06-26T07:07:24.8066922Z 224fe954d725: Download complete 2024-06-26T07:07:24.8794709Z d527cbbb87e3: Verifying Checksum 2024-06-26T07:07:24.8795328Z d527cbbb87e3: Download complete 2024-06-26T07:07:24.9756203Z b57676e46aee: Download complete 2024-06-26T07:07:25.0223511Z 7a2c55901189: Verifying Checksum 2024-06-26T07:07:25.0224300Z 7a2c55901189: Download complete 2024-06-26T07:07:25.1026354Z a41a8d1c11c8: Verifying Checksum 2024-06-26T07:07:25.1027045Z a41a8d1c11c8: Download complete 2024-06-26T07:07:25.1979918Z 0c1227890755: Download complete 2024-06-26T07:07:25.2782341Z 75722010b82e: Download complete 2024-06-26T07:07:25.2895819Z d8d1234baab3: Verifying Checksum 2024-06-26T07:07:25.2896338Z d8d1234baab3: Download complete 2024-06-26T07:07:25.3497438Z ec1e7978c1fe: Verifying Checksum 2024-06-26T07:07:25.3498026Z ec1e7978c1fe: Download complete 2024-06-26T07:07:25.4093756Z 309b872da805: Download complete 2024-06-26T07:07:26.0432191Z 7a2c55901189: Pull complete 2024-06-26T07:07:26.3616910Z 224fe954d725: Pull complete 2024-06-26T07:07:27.3156695Z 75722010b82e: Pull complete 2024-06-26T07:07:27.3738255Z d527cbbb87e3: Pull complete 2024-06-26T07:07:27.4148501Z b57676e46aee: Pull complete 2024-06-26T07:07:27.9218736Z e82932cb002c: Verifying Checksum 2024-06-26T07:07:27.9219263Z e82932cb002c: Download complete 2024-06-26T07:07:27.9923704Z ca9dd73e3e93: Verifying Checksum 2024-06-26T07:07:27.9924236Z ca9dd73e3e93: Download complete 2024-06-26T07:07:28.0656458Z 16d76a79d5d6: Verifying Checksum 2024-06-26T07:07:28.0657004Z 16d76a79d5d6: Download complete 2024-06-26T07:07:28.1390556Z 4bdf41c9ecbe: Verifying Checksum 2024-06-26T07:07:28.1391076Z 4bdf41c9ecbe: Download complete 2024-06-26T07:07:29.1006700Z f4863801e30c: Verifying Checksum 2024-06-26T07:07:29.1007274Z f4863801e30c: Download complete 2024-06-26T07:07:29.1764853Z d37dd9ed8948: Download complete 2024-06-26T07:07:29.2540341Z cf2f34f225db: Download complete 2024-06-26T07:07:29.3201297Z df546d7b3e34: Download complete 2024-06-26T07:07:38.3125377Z a8c1e85b5e14: Verifying Checksum 2024-06-26T07:07:38.3126020Z a8c1e85b5e14: Download complete 2024-06-26T07:07:38.3950130Z 96d0e2032184: Verifying Checksum 2024-06-26T07:07:38.3950955Z 96d0e2032184: Download complete 2024-06-26T07:07:38.4750159Z 06e12ce59790: Download complete 2024-06-26T07:07:38.5495584Z ed712f63cde7: Verifying Checksum 2024-06-26T07:07:38.5496224Z ed712f63cde7: Download complete 2024-06-26T07:07:38.6381852Z a97a71ef8a4e: Verifying Checksum 2024-06-26T07:07:38.6382492Z a97a71ef8a4e: Download complete 2024-06-26T07:07:38.7177089Z 25cd0c482ad6: Download complete 2024-06-26T07:07:39.9890935Z 9bf94bc45658: Verifying Checksum 2024-06-26T07:07:39.9891429Z 9bf94bc45658: Download complete 2024-06-26T07:07:40.0673726Z de45a14c9b22: Verifying Checksum 2024-06-26T07:07:40.0674400Z de45a14c9b22: Download complete 2024-06-26T07:07:40.1314757Z e99241e77209: Download complete 2024-06-26T07:07:40.2221906Z 35e8f4e2a42f: Verifying Checksum 2024-06-26T07:07:40.2222438Z 35e8f4e2a42f: Download complete 2024-06-26T07:07:40.2896168Z a823ffa4c541: Download complete 2024-06-26T07:07:40.3624547Z 44ba272eae20: Verifying Checksum 2024-06-26T07:07:44.8935081Z da02d278e524: Verifying Checksum 2024-06-26T07:07:44.8935790Z da02d278e524: Download complete 2024-06-26T07:07:45.0020978Z 73a0ce681c29: Verifying Checksum 2024-06-26T07:07:45.0021613Z 73a0ce681c29: Download complete 2024-06-26T07:07:45.0917875Z 49737bd5ae70: Verifying Checksum 2024-06-26T07:07:45.0918356Z 49737bd5ae70: Download complete 2024-06-26T07:07:45.5086854Z 9cd4c745f8e4: Verifying Checksum 2024-06-26T07:07:45.5087526Z 9cd4c745f8e4: Download complete 2024-06-26T07:07:45.5977250Z b2950a036534: Verifying Checksum 2024-06-26T07:07:45.5977709Z b2950a036534: Download complete 2024-06-26T07:07:45.6891401Z 25e19d81d9fc: Verifying Checksum 2024-06-26T07:07:45.6892141Z 25e19d81d9fc: Download complete 2024-06-26T07:07:45.9908191Z 60156b40fcbe: Verifying Checksum 2024-06-26T07:07:45.9908765Z 60156b40fcbe: Download complete 2024-06-26T07:07:46.0639890Z 7a868239227f: Download complete 2024-06-26T07:07:46.1345456Z 83911d7c7ecf: Verifying Checksum 2024-06-26T07:07:46.1345934Z 83911d7c7ecf: Download complete 2024-06-26T07:07:46.2053084Z b3c5d66c679f: Verifying Checksum 2024-06-26T07:07:46.2053661Z b3c5d66c679f: Download complete 2024-06-26T07:07:50.6213695Z 7ed32bc8e469: Verifying Checksum 2024-06-26T07:07:50.6214201Z 7ed32bc8e469: Download complete 2024-06-26T07:07:50.7058322Z c1068d03a811: Verifying Checksum 2024-06-26T07:07:50.7058989Z c1068d03a811: Download complete 2024-06-26T07:07:50.7904616Z 671b4a301d81: Verifying Checksum 2024-06-26T07:07:50.7905165Z 671b4a301d81: Download complete 2024-06-26T07:07:51.2980725Z 67760cf5ec46: Verifying Checksum 2024-06-26T07:07:51.2981230Z 67760cf5ec46: Download complete 2024-06-26T07:07:51.3813376Z ddc9740bcea2: Verifying Checksum 2024-06-26T07:07:51.3813865Z ddc9740bcea2: Download complete 2024-06-26T07:07:51.4055228Z a8c1e85b5e14: Pull complete 2024-06-26T07:07:51.4610309Z fb2e97768560: Download complete 2024-06-26T07:07:51.5350326Z b599c5a8c7cb: Verifying Checksum 2024-06-26T07:07:51.5350900Z b599c5a8c7cb: Download complete 2024-06-26T07:07:51.6140829Z e08e88c13668: Verifying Checksum 2024-06-26T07:07:51.6141314Z e08e88c13668: Download complete 2024-06-26T07:07:51.6361186Z a41a8d1c11c8: Pull complete 2024-06-26T07:07:51.8751822Z 0c1227890755: Pull complete 2024-06-26T07:07:52.0652080Z d8d1234baab3: Pull complete 2024-06-26T07:07:57.2186233Z 3d914950f8ad: Verifying Checksum 2024-06-26T07:07:57.2186837Z 3d914950f8ad: Download complete 2024-06-26T07:07:57.2934592Z cf7c2cb896cb: Download complete 2024-06-26T07:07:57.3867815Z 3a778ddfac2a: Verifying Checksum 2024-06-26T07:07:57.3868714Z 3a778ddfac2a: Download complete 2024-06-26T07:07:57.4629177Z 236c8f0245f4: Download complete 2024-06-26T07:07:57.4715162Z 4f4fb700ef54: Verifying Checksum 2024-06-26T07:07:57.4715808Z 4f4fb700ef54: Download complete 2024-06-26T07:07:57.5549636Z dda1ce302b9a: Verifying Checksum 2024-06-26T07:07:57.5550306Z dda1ce302b9a: Download complete 2024-06-26T07:07:57.6335993Z e797fe86caef: Download complete 2024-06-26T07:07:57.8001392Z 967190076fc0: Verifying Checksum 2024-06-26T07:07:57.8001878Z 967190076fc0: Download complete 2024-06-26T07:07:57.8661561Z 4fb0a83f0685: Verifying Checksum 2024-06-26T07:07:57.8662194Z 4fb0a83f0685: Download complete 2024-06-26T07:07:57.9420131Z 68af2ec4e30a: Download complete 2024-06-26T07:07:58.0159434Z cc91933e985a: Download complete 2024-06-26T07:07:58.0972633Z 9f3187916e7d: Verifying Checksum 2024-06-26T07:07:58.0973378Z 9f3187916e7d: Download complete 2024-06-26T07:07:58.2252471Z 526083994857: Verifying Checksum 2024-06-26T07:07:58.2253042Z 526083994857: Download complete 2024-06-26T07:07:58.2979855Z 05a6f6dcc1c0: Verifying Checksum 2024-06-26T07:07:58.2980497Z 05a6f6dcc1c0: Download complete 2024-06-26T07:07:58.8916030Z 6bc9dc8315b0: Verifying Checksum 2024-06-26T07:07:58.8916723Z 6bc9dc8315b0: Download complete 2024-06-26T07:07:58.9625908Z 1b0e27c35f07: Download complete 2024-06-26T07:08:21.3301818Z 12ba7431ff38: Verifying Checksum 2024-06-26T07:08:21.3303924Z 12ba7431ff38: Download complete 2024-06-26T07:08:21.4056875Z 789eff6c51aa: Verifying Checksum 2024-06-26T07:08:21.4057354Z 789eff6c51aa: Download complete 2024-06-26T07:08:21.4745225Z de0f8a8c45c6: Verifying Checksum 2024-06-26T07:08:21.4746013Z de0f8a8c45c6: Download complete 2024-06-26T07:08:22.9696170Z 2060028f7488: Verifying Checksum 2024-06-26T07:08:22.9696666Z 2060028f7488: Download complete 2024-06-26T07:08:23.0513445Z e412c7f3b2de: Verifying Checksum 2024-06-26T07:08:23.0513916Z e412c7f3b2de: Download complete 2024-06-26T07:08:26.7304394Z 95237e7311a1: Verifying Checksum 2024-06-26T07:08:26.7304926Z 95237e7311a1: Download complete 2024-06-26T07:08:49.7506257Z 84b2c7a89982: Verifying Checksum 2024-06-26T07:08:49.7506748Z 84b2c7a89982: Download complete 2024-06-26T07:09:07.6673835Z 7ed32bc8e469: Pull complete 2024-06-26T07:09:07.8997384Z ec1e7978c1fe: Pull complete 2024-06-26T07:09:08.1281344Z 309b872da805: Pull complete 2024-06-26T07:09:15.5514719Z e82932cb002c: Pull complete 2024-06-26T07:09:15.7740982Z ca9dd73e3e93: Pull complete 2024-06-26T07:09:16.0108161Z 16d76a79d5d6: Pull complete 2024-06-26T07:09:16.2381471Z 4bdf41c9ecbe: Pull complete 2024-06-26T07:09:18.6468996Z f4863801e30c: Pull complete 2024-06-26T07:09:18.8825007Z d37dd9ed8948: Pull complete 2024-06-26T07:09:19.0931764Z cf2f34f225db: Pull complete 2024-06-26T07:09:19.3123796Z df546d7b3e34: Pull complete 2024-06-26T07:10:08.2998135Z 3d914950f8ad: Pull complete 2024-06-26T07:10:08.4439274Z 96d0e2032184: Pull complete 2024-06-26T07:10:08.5878383Z 06e12ce59790: Pull complete 2024-06-26T07:10:08.6762230Z ed712f63cde7: Pull complete 2024-06-26T07:10:08.7895552Z a97a71ef8a4e: Pull complete 2024-06-26T07:10:08.9108098Z 25cd0c482ad6: Pull complete 2024-06-26T07:10:12.4392148Z 9bf94bc45658: Pull complete 2024-06-26T07:10:12.6630465Z de45a14c9b22: Pull complete 2024-06-26T07:10:12.8943020Z e99241e77209: Pull complete 2024-06-26T07:10:13.1188246Z 35e8f4e2a42f: Pull complete 2024-06-26T07:10:13.3487023Z a823ffa4c541: Pull complete 2024-06-26T07:10:13.5594064Z 44ba272eae20: Pull complete 2024-06-26T07:10:21.9187895Z da02d278e524: Pull complete 2024-06-26T07:10:22.0517168Z 73a0ce681c29: Pull complete 2024-06-26T07:10:22.2860084Z 49737bd5ae70: Pull complete 2024-06-26T07:10:23.2396934Z 9cd4c745f8e4: Pull complete 2024-06-26T07:10:23.4733518Z b2950a036534: Pull complete 2024-06-26T07:10:23.6363662Z 25e19d81d9fc: Pull complete 2024-06-26T07:10:24.1092754Z 60156b40fcbe: Pull complete 2024-06-26T07:10:24.2841715Z 7a868239227f: Pull complete 2024-06-26T07:10:24.6616956Z 83911d7c7ecf: Pull complete 2024-06-26T07:10:24.8847854Z b3c5d66c679f: Pull complete 2024-06-26T07:11:04.1378205Z 84b2c7a89982: Pull complete 2024-06-26T07:11:04.3581094Z c1068d03a811: Pull complete 2024-06-26T07:11:04.5295737Z 671b4a301d81: Pull complete 2024-06-26T07:11:06.4826214Z 67760cf5ec46: Pull complete 2024-06-26T07:11:06.5218837Z ddc9740bcea2: Pull complete 2024-06-26T07:11:06.5660289Z fb2e97768560: Pull complete 2024-06-26T07:11:06.6897378Z b599c5a8c7cb: Pull complete 2024-06-26T07:11:06.7315256Z e08e88c13668: Pull complete 2024-06-26T07:11:40.8474691Z 12ba7431ff38: Pull complete 2024-06-26T07:11:41.0715605Z cf7c2cb896cb: Pull complete 2024-06-26T07:11:41.2916518Z 3a778ddfac2a: Pull complete 2024-06-26T07:11:41.7389409Z 236c8f0245f4: Pull complete 2024-06-26T07:11:41.9329128Z 4f4fb700ef54: Pull complete 2024-06-26T07:11:42.0816382Z dda1ce302b9a: Pull complete 2024-06-26T07:11:42.3012860Z e797fe86caef: Pull complete 2024-06-26T07:11:42.6748678Z 967190076fc0: Pull complete 2024-06-26T07:11:42.8873625Z 4fb0a83f0685: Pull complete 2024-06-26T07:11:43.1099677Z 68af2ec4e30a: Pull complete 2024-06-26T07:11:43.3251245Z cc91933e985a: Pull complete 2024-06-26T07:11:43.5283370Z 9f3187916e7d: Pull complete 2024-06-26T07:11:44.6136440Z 526083994857: Pull complete 2024-06-26T07:11:44.8422307Z 05a6f6dcc1c0: Pull complete 2024-06-26T07:11:46.8382871Z 6bc9dc8315b0: Pull complete 2024-06-26T07:11:46.9319648Z 1b0e27c35f07: Pull complete 2024-06-26T07:12:00.6822314Z 95237e7311a1: Pull complete 2024-06-26T07:12:00.7662615Z 789eff6c51aa: Pull complete 2024-06-26T07:12:00.8315221Z de0f8a8c45c6: Pull complete 2024-06-26T07:12:01.6647506Z 2060028f7488: Pull complete 2024-06-26T07:12:01.8939282Z e412c7f3b2de: Pull complete 2024-06-26T07:12:02.9728260Z Digest: sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:12:03.0192652Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:12:03.0413934Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:12:03.0466404Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:12:03.0467412Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:12:03.0475696Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:03.0476242Z env: 2024-06-26T07:12:03.0476528Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:03.0476901Z ##[endgroup] 2024-06-26T07:12:03.0645247Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-26T07:12:03.0645847Z with: 2024-06-26T07:12:03.0646149Z driver-version: 550.54.15 2024-06-26T07:12:03.0646518Z env: 2024-06-26T07:12:03.0646806Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:03.0647169Z ##[endgroup] 2024-06-26T07:12:03.0783639Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:12:03.0784250Z with: 2024-06-26T07:12:03.0784540Z timeout_minutes: 10 2024-06-26T07:12:03.0784898Z max_attempts: 3 2024-06-26T07:12:03.0821742Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-26T07:12:03.0859426Z retry_wait_seconds: 10 2024-06-26T07:12:03.0859823Z polling_interval_seconds: 1 2024-06-26T07:12:03.0860222Z warning_on_retry: true 2024-06-26T07:12:03.0860597Z continue_on_error: false 2024-06-26T07:12:03.0860962Z env: 2024-06-26T07:12:03.0861247Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:03.0861721Z DRIVER_VERSION: 550.54.15 2024-06-26T07:12:03.0862088Z ##[endgroup] 2024-06-26T07:12:03.1398320Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-26T07:12:03.1399154Z + pre_install_nvidia_driver_amzn2 2024-06-26T07:12:03.1399871Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-26T07:12:03.4425904Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:12:03.4816276Z No Match for argument: nvidia-driver-latest-dkms 2024-06-26T07:12:03.5053454Z No Packages marked for removal 2024-06-26T07:12:03.5174278Z + install_nvidia_driver_common 2024-06-26T07:12:03.5176788Z + echo 'Before installing NVIDIA driver' 2024-06-26T07:12:03.5177374Z + lspci 2024-06-26T07:12:03.5177731Z Before installing NVIDIA driver 2024-06-26T07:12:03.5256832Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:12:03.5257942Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:12:03.5259345Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:12:03.5260527Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:12:03.5261688Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:12:03.5262795Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:12:03.5263941Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:12:03.5265068Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:12:03.5265933Z + lsmod 2024-06-26T07:12:03.5273144Z Module Size Used by 2024-06-26T07:12:03.5273797Z nvidia_modeset 1339392 0 2024-06-26T07:12:03.5274370Z veth 16384 0 2024-06-26T07:12:03.5274929Z nvidia_uvm 4599808 0 2024-06-26T07:12:03.5275455Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:12:03.5276152Z drm 421888 1 nvidia 2024-06-26T07:12:03.5276968Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:12:03.5277664Z backlight 16384 1 nvidia_modeset 2024-06-26T07:12:03.5278145Z xt_conntrack 16384 1 2024-06-26T07:12:03.5278551Z ipt_MASQUERADE 16384 1 2024-06-26T07:12:03.5279010Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:12:03.5279516Z nf_conntrack_netlink 49152 0 2024-06-26T07:12:03.5279990Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:12:03.5280489Z xfrm_user 45056 1 2024-06-26T07:12:03.5281062Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:12:03.5281679Z iptable_nat 16384 1 2024-06-26T07:12:03.5282215Z nf_conntrack_ipv4 16384 3 2024-06-26T07:12:03.5282742Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:12:03.5283322Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:12:03.5283894Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T07:12:03.5284980Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T07:12:03.5285928Z xt_addrtype 16384 2 2024-06-26T07:12:03.5286335Z iptable_filter 16384 1 2024-06-26T07:12:03.5286743Z br_netfilter 24576 0 2024-06-26T07:12:03.5287178Z bridge 172032 1 br_netfilter 2024-06-26T07:12:03.5287640Z stp 16384 1 bridge 2024-06-26T07:12:03.5288106Z llc 16384 2 bridge,stp 2024-06-26T07:12:03.5288555Z overlay 86016 0 2024-06-26T07:12:03.5288953Z sunrpc 393216 1 2024-06-26T07:12:03.5289342Z dm_mirror 28672 0 2024-06-26T07:12:03.5289771Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:12:03.5290294Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:12:03.5290850Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:12:03.5291321Z dax 69632 1 dm_mod 2024-06-26T07:12:03.5291878Z crc32_pclmul 16384 0 2024-06-26T07:12:03.5292532Z ghash_clmulni_intel 16384 0 2024-06-26T07:12:03.5292938Z pcbc 16384 0 2024-06-26T07:12:03.5293324Z aesni_intel 188416 0 2024-06-26T07:12:03.5293749Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:12:03.5294237Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:12:03.5294720Z glue_helper 16384 1 aesni_intel 2024-06-26T07:12:03.5295329Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:12:03.5295920Z mousedev 24576 0 2024-06-26T07:12:03.5296320Z psmouse 32768 0 2024-06-26T07:12:03.5296709Z evdev 20480 3 2024-06-26T07:12:03.5297104Z button 16384 0 2024-06-26T07:12:03.5297497Z ena 139264 0 2024-06-26T07:12:03.5297891Z ptp 20480 1 ena 2024-06-26T07:12:03.5298318Z pps_core 20480 1 ptp 2024-06-26T07:12:03.5298756Z crc32c_intel 24576 0 2024-06-26T07:12:03.5299148Z autofs4 49152 2 2024-06-26T07:12:03.5299540Z + modinfo nvidia 2024-06-26T07:12:03.5306896Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:12:03.5307929Z alias: char-major-195-* 2024-06-26T07:12:03.5308479Z version: 550.54.15 2024-06-26T07:12:03.5308965Z supported: external 2024-06-26T07:12:03.5309370Z license: NVIDIA 2024-06-26T07:12:03.5309763Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:12:03.5310280Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:12:03.5310766Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:12:03.5311310Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:12:03.5312045Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:12:03.5312780Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:12:03.5313401Z depends: i2c-core,drm 2024-06-26T07:12:03.5314100Z retpoline: Y 2024-06-26T07:12:03.5314579Z name: nvidia 2024-06-26T07:12:03.5315454Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:12:03.5316537Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:12:03.5317527Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:12:03.5318411Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:12:03.5318885Z parm: NVreg_RmLogonRC:int 2024-06-26T07:12:03.5319335Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:12:03.5319816Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:12:03.5320272Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:12:03.5320738Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:12:03.5321292Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:12:03.5321893Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:12:03.5322404Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:12:03.5322889Z parm: NVreg_EnableMSI:int 2024-06-26T07:12:03.5323321Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:12:03.5323809Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:12:03.5324369Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:12:03.5324991Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:12:03.5325618Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:12:03.5326367Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:12:03.5327009Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:12:03.5327664Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:12:03.5328287Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:12:03.5328807Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:12:03.5329372Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:12:03.5329963Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:12:03.5330604Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:12:03.5331102Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:12:03.5331614Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:12:03.5332316Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:12:03.5332786Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:12:03.5333317Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:12:03.5333874Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:12:03.5334376Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:12:03.5334884Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:12:03.5335411Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:12:03.5335922Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:12:03.5336441Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:12:03.5336949Z parm: NVreg_RmMsg:charp 2024-06-26T07:12:03.5337387Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:12:03.5337886Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:12:03.5338373Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:12:03.5338856Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:12:03.5339361Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:12:03.5339874Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:12:03.5340344Z parm: rm_firmware_active:charp 2024-06-26T07:12:03.5340780Z + HAS_NVIDIA_DRIVER=0 2024-06-26T07:12:03.5341175Z ++ command -v nvidia-smi 2024-06-26T07:12:03.5341575Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-26T07:12:03.5341951Z + set +e 2024-06-26T07:12:03.5342476Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-26T07:12:05.5150525Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-26T07:12:05.5151166Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:12:05.5151778Z + '[' 0 -ne 0 ']' 2024-06-26T07:12:05.5152174Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-26T07:12:05.5152886Z + HAS_NVIDIA_DRIVER=1 2024-06-26T07:12:05.5153676Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-26T07:12:05.5154456Z + set -e 2024-06-26T07:12:05.5154763Z + '[' 1 -eq 0 ']' 2024-06-26T07:12:05.5155384Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-26T07:12:05.5156145Z + post_install_nvidia_driver_common 2024-06-26T07:12:05.5156585Z + sudo modprobe nvidia 2024-06-26T07:12:05.5247357Z + echo 'After installing NVIDIA driver' 2024-06-26T07:12:05.5248044Z + lspci 2024-06-26T07:12:05.5248429Z After installing NVIDIA driver 2024-06-26T07:12:05.5326212Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:12:05.5327079Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:12:05.5328138Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:12:05.5329033Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:12:05.5329870Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:12:05.5330703Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:12:05.5331497Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:12:05.5332710Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:12:05.5333361Z + lsmod 2024-06-26T07:12:05.5341205Z Module Size Used by 2024-06-26T07:12:05.5341849Z nvidia_modeset 1339392 0 2024-06-26T07:12:05.5342346Z veth 16384 0 2024-06-26T07:12:05.5342885Z nvidia_uvm 4599808 0 2024-06-26T07:12:05.5343757Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:12:05.5344450Z drm 421888 1 nvidia 2024-06-26T07:12:05.5345036Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:12:05.5345554Z backlight 16384 1 nvidia_modeset 2024-06-26T07:12:05.5346246Z xt_conntrack 16384 1 2024-06-26T07:12:05.5346649Z ipt_MASQUERADE 16384 1 2024-06-26T07:12:05.5347113Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:12:05.5347614Z nf_conntrack_netlink 49152 0 2024-06-26T07:12:05.5348168Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:12:05.5348829Z xfrm_user 45056 1 2024-06-26T07:12:05.5349377Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:12:05.5349963Z iptable_nat 16384 1 2024-06-26T07:12:05.5350500Z nf_conntrack_ipv4 16384 3 2024-06-26T07:12:05.5351054Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:12:05.5351576Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:12:05.5352129Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T07:12:05.5353191Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T07:12:05.5354199Z xt_addrtype 16384 2 2024-06-26T07:12:05.5354600Z iptable_filter 16384 1 2024-06-26T07:12:05.5355002Z br_netfilter 24576 0 2024-06-26T07:12:05.5355420Z bridge 172032 1 br_netfilter 2024-06-26T07:12:05.5355990Z stp 16384 1 bridge 2024-06-26T07:12:05.5356433Z llc 16384 2 bridge,stp 2024-06-26T07:12:05.5356925Z overlay 86016 0 2024-06-26T07:12:05.5357312Z sunrpc 393216 1 2024-06-26T07:12:05.5357704Z dm_mirror 28672 0 2024-06-26T07:12:05.5358134Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:12:05.5358660Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:12:05.5359203Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:12:05.5359691Z dax 69632 1 dm_mod 2024-06-26T07:12:05.5360124Z crc32_pclmul 16384 0 2024-06-26T07:12:05.5360622Z ghash_clmulni_intel 16384 0 2024-06-26T07:12:05.5361168Z pcbc 16384 0 2024-06-26T07:12:05.5361568Z aesni_intel 188416 0 2024-06-26T07:12:05.5361988Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:12:05.5362461Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:12:05.5362950Z glue_helper 16384 1 aesni_intel 2024-06-26T07:12:05.5363552Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:12:05.5364143Z mousedev 24576 0 2024-06-26T07:12:05.5364532Z psmouse 32768 0 2024-06-26T07:12:05.5365007Z evdev 20480 3 2024-06-26T07:12:05.5365396Z button 16384 0 2024-06-26T07:12:05.5365776Z ena 139264 0 2024-06-26T07:12:05.5366162Z ptp 20480 1 ena 2024-06-26T07:12:05.5366584Z pps_core 20480 1 ptp 2024-06-26T07:12:05.5367008Z crc32c_intel 24576 0 2024-06-26T07:12:05.5367403Z autofs4 49152 2 2024-06-26T07:12:05.5367774Z + modinfo nvidia 2024-06-26T07:12:05.5368576Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:12:05.5369337Z alias: char-major-195-* 2024-06-26T07:12:05.5369732Z version: 550.54.15 2024-06-26T07:12:05.5370097Z supported: external 2024-06-26T07:12:05.5370452Z license: NVIDIA 2024-06-26T07:12:05.5370850Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:12:05.5371372Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:12:05.5371868Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:12:05.5372603Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:12:05.5373223Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:12:05.5373747Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:12:05.5374299Z depends: i2c-core,drm 2024-06-26T07:12:05.5374680Z retpoline: Y 2024-06-26T07:12:05.5375004Z name: nvidia 2024-06-26T07:12:05.5375596Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:12:05.5376481Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:12:05.5377186Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:12:05.5377841Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:12:05.5378316Z parm: NVreg_RmLogonRC:int 2024-06-26T07:12:05.5378783Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:12:05.5379277Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:12:05.5379746Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:12:05.5380223Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:12:05.5380802Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:12:05.5381418Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:12:05.5381933Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:12:05.5382403Z parm: NVreg_EnableMSI:int 2024-06-26T07:12:05.5382851Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:12:05.5383369Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:12:05.5384026Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:12:05.5384717Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:12:05.5385322Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:12:05.5385984Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:12:05.5386620Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:12:05.5387285Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:12:05.5387939Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:12:05.5388470Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:12:05.5389043Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:12:05.5389651Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:12:05.5390199Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:12:05.5390781Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:12:05.5391305Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:12:05.5391810Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:12:05.5392297Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:12:05.5392907Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:12:05.5393474Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:12:05.5394042Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:12:05.5394579Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:12:05.5395114Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:12:05.5395644Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:12:05.5396185Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:12:05.5396710Z parm: NVreg_RmMsg:charp 2024-06-26T07:12:05.5397147Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:12:05.5397661Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:12:05.5398252Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:12:05.5398753Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:12:05.5399259Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:12:05.5399787Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:12:05.5400288Z parm: rm_firmware_active:charp 2024-06-26T07:12:05.5400728Z + set +e 2024-06-26T07:12:05.5401042Z + nvidia-smi 2024-06-26T07:12:07.1070265Z Wed Jun 26 07:12:07 2024 2024-06-26T07:12:07.1071057Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:12:07.1072041Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-26T07:12:07.1072896Z |-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:12:07.1073757Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-26T07:12:07.1074763Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-26T07:12:07.1075769Z | | | MIG M. | 2024-06-26T07:12:07.1076345Z |=========================================+========================+======================| 2024-06-26T07:12:07.1196333Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-06-26T07:12:07.1197283Z | 0% 30C P0 59W / 300W | 0MiB / 23028MiB | 0% Default | 2024-06-26T07:12:07.1198124Z | | | N/A | 2024-06-26T07:12:07.1198946Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:12:07.1199582Z 2024-06-26T07:12:07.1200272Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:12:07.1201079Z | Processes: | 2024-06-26T07:12:07.1201917Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-26T07:12:07.1202677Z | ID ID Usage | 2024-06-26T07:12:07.1203274Z |=========================================================================================| 2024-06-26T07:12:07.1203978Z | No running processes found | 2024-06-26T07:12:07.1204798Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:12:07.7427714Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-26T07:12:09.3008174Z NVIDIA A10G 2024-06-26T07:12:09.7493127Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:12:09.7493851Z + '[' 0 -eq 0 ']' 2024-06-26T07:12:09.7494412Z + echo 'INFO: Ignoring allowed status 0' 2024-06-26T07:12:09.7495230Z + set -e 2024-06-26T07:12:09.7495544Z INFO: Ignoring allowed status 0 2024-06-26T07:12:09.7497272Z == Installing nvidia container toolkit for amzn2 == 2024-06-26T07:12:09.7499198Z + sudo yum install -y yum-utils 2024-06-26T07:12:10.0545945Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:12:11.5669157Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-26T07:12:11.5670606Z Nothing to do 2024-06-26T07:12:11.7168737Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-06-26T07:12:11.7169874Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:12:11.7171460Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:12:12.0552238Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:12:12.0801308Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:12:12.0802707Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:12:12.0803766Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:12:12.0917601Z + sudo yum install -y nvidia-docker2 2024-06-26T07:12:12.4020303Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:12:13.9011362Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-26T07:12:13.9012264Z Nothing to do 2024-06-26T07:12:14.0534695Z + sudo systemctl restart docker 2024-06-26T07:12:49.1821931Z Command completed after 1 attempt(s). 2024-06-26T07:12:49.1881547Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:12:49.1882402Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:12:49.1883165Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T07:12:49.1883922Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:12:49.1892164Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:49.1892710Z env: 2024-06-26T07:12:49.1893007Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:49.1893497Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:49.1894017Z ##[endgroup] 2024-06-26T07:12:49.4107057Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:12:49.4269375Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-06-26T07:12:49.4370879Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-06-26T07:12:49.5478639Z Prepare all required actions 2024-06-26T07:12:49.5479170Z Getting action download info 2024-06-26T07:12:49.6666190Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T07:12:49.8646309Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T07:12:50.0005969Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T07:12:50.0006474Z with: 2024-06-26T07:12:50.0006808Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:12:50.0007285Z s3-bucket: gha-artifacts 2024-06-26T07:12:50.0007629Z env: 2024-06-26T07:12:50.0007911Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:50.0008363Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:50.0008870Z ##[endgroup] 2024-06-26T07:12:50.0041201Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:12:50.0041677Z with: 2024-06-26T07:12:50.0042047Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:12:50.0042528Z s3-bucket: gha-artifacts 2024-06-26T07:12:50.0042895Z region: us-east-1 2024-06-26T07:12:50.0043207Z env: 2024-06-26T07:12:50.0043478Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:50.0043972Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:50.0044516Z ##[endgroup] 2024-06-26T07:12:50.4191844Z (node:23546) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:12:50.4192649Z 2024-06-26T07:12:50.4192925Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:12:50.4193722Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:12:50.4194772Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:12:50.5054322Z Found 1 objects with prefix pytorch/pytorch/9673646806/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-06-26T07:12:50.5055551Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:12:57.9904511Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:12:57.9909839Z Artifact download has finished successfully 2024-06-26T07:12:58.0066079Z ##[group]Run unzip -o artifacts.zip 2024-06-26T07:12:58.0066540Z unzip -o artifacts.zip 2024-06-26T07:12:58.0074458Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:58.0074992Z env: 2024-06-26T07:12:58.0075279Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:58.0075766Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:58.0076288Z ##[endgroup] 2024-06-26T07:12:58.0106422Z Archive: artifacts.zip 2024-06-26T07:12:58.0107907Z creating: dist/ 2024-06-26T07:12:59.9935698Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:12:59.9936710Z creating: build/custom_test_artifacts/ 2024-06-26T07:12:59.9937339Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T07:12:59.9938123Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T07:12:59.9939037Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:12:59.9942508Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:12:59.9944123Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T07:12:59.9945160Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:12:59.9946522Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:12:59.9947821Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:12:59.9949073Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:12:59.9950440Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:12:59.9951550Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:12:59.9952652Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:12:59.9953941Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:12:59.9955240Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:12:59.9956504Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:12:59.9957764Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:12:59.9959062Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:12:59.9960365Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:12:59.9961516Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:12:59.9962635Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:12:59.9995586Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:13:00.0033380Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:13:00.0035008Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:13:00.0077699Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:13:00.0079298Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:13:00.0080896Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:13:00.0082547Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:13:00.0084135Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:13:00.0085694Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:13:00.0087252Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:13:00.0088799Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:13:00.0090317Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:13:00.0091768Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:13:00.0093409Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:13:00.0094894Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:13:00.0096269Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:13:00.0097603Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:13:00.0098964Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:13:00.0144347Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:13:00.0202263Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:13:00.0203745Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:13:00.0204864Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:13:00.0205815Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:13:00.0206807Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T07:13:00.0207834Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T07:13:00.0208978Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T07:13:00.0210267Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T07:13:00.0211513Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T07:13:00.0212910Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T07:13:00.0214105Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T07:13:00.0215315Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T07:13:00.0216527Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T07:13:00.0217731Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T07:13:00.0218921Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T07:13:00.0228032Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T07:13:00.0355458Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T07:13:00.0356557Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T07:13:00.0357776Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T07:13:00.0359137Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T07:13:00.0360424Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T07:13:00.0361651Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T07:13:00.0362909Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T07:13:00.0364188Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T07:13:00.0365461Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T07:13:00.0366719Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T07:13:00.0367969Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T07:13:00.0379410Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T07:13:00.0458892Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T07:13:00.0460217Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:13:00.0461411Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:13:00.0462587Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T07:13:00.0463658Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T07:13:00.0464649Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T07:13:00.0465641Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-26T07:13:00.0466586Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T07:13:00.0467439Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T07:13:00.0468330Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T07:13:00.0572846Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T07:13:00.0634087Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T07:13:00.0635877Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T07:13:00.0636787Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T07:13:00.0637691Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:13:00.0640859Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:13:00.0642022Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T07:13:00.0643089Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:13:00.0644438Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:13:00.0645718Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:13:00.0646929Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:13:00.0648427Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:13:00.0649734Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:13:00.0651106Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:13:00.0652835Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:13:00.0654327Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:13:00.0655584Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:13:00.0656844Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:13:00.0658133Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:13:00.0659422Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:13:00.0660560Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:13:00.0661651Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:13:00.0692986Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:13:00.0732521Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:13:00.0735759Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:13:00.0776325Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:13:00.0778408Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:13:00.0780599Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:13:00.0782597Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:13:00.0784279Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:13:00.0785817Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:13:00.0787381Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:13:00.0788932Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:13:00.0790472Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:13:00.0791906Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:13:00.0793278Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:13:00.0794647Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:13:00.0796022Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:13:00.0797349Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:13:00.0798693Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:13:00.0840706Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:13:00.0899028Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:13:00.0900774Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:13:00.0902212Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:13:00.0903479Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:13:00.0904664Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T07:13:00.0905720Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T07:13:00.0906900Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T07:13:00.0908245Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T07:13:00.0909531Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T07:13:00.0910731Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T07:13:00.0911987Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T07:13:00.0913350Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T07:13:00.0914600Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T07:13:00.0915833Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T07:13:00.0917066Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T07:13:00.0924447Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T07:13:00.0986035Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T07:13:00.0987741Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:13:00.0989096Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:13:00.0990298Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T07:13:00.0991607Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T07:13:00.0992601Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T07:13:00.0993583Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-26T07:13:00.0994513Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T07:13:00.0995361Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T07:13:00.0996249Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T07:13:00.1039837Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T07:13:00.1040870Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T07:13:00.1041715Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T07:13:00.1042695Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:13:00.1047401Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:13:00.1048571Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T07:13:00.1049785Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:13:00.1051204Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:13:00.1052728Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:13:00.1054175Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:13:00.1055784Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:13:00.1057252Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:13:00.1058552Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:13:00.1060330Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:13:00.1061711Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:13:00.1063037Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:13:00.1064470Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:13:00.1065822Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:13:00.1067182Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:13:00.1068515Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:13:00.1069709Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:13:00.1099288Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:13:00.1137150Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:13:00.1138971Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:13:00.1181439Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:13:00.1183430Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:13:00.1185127Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:13:00.1186903Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:13:00.1188577Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:13:00.1204182Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:13:00.1206233Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:13:00.1208376Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:13:00.1210215Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:13:00.1228628Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:13:00.1230266Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:13:00.1231697Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:13:00.1233171Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:13:00.1234583Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:13:00.1236009Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:13:00.1270194Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:13:00.1328089Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:13:00.1329499Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:13:00.1330679Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:13:00.1331703Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:13:00.1332980Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T07:13:00.1334106Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T07:13:00.1335384Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T07:13:00.1337068Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T07:13:00.1338436Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T07:13:00.1339719Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T07:13:00.1341033Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T07:13:00.1342475Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T07:13:00.1343894Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T07:13:00.1345225Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T07:13:00.1346548Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T07:13:00.1347953Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T07:13:00.1461340Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T07:13:00.1462662Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T07:13:00.1464042Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T07:13:00.1465526Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T07:13:00.1466953Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T07:13:00.1468316Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T07:13:00.1469704Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T07:13:00.1471097Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T07:13:00.1472489Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T07:13:00.1473885Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T07:13:00.1475267Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T07:13:00.1487200Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T07:13:00.1540393Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T07:13:00.1541862Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:13:00.1543192Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:13:00.1544354Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T07:13:00.1545442Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T07:13:00.1546548Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T07:13:00.1547646Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-26T07:13:00.1548664Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T07:13:00.1549600Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T07:13:00.1550684Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T07:13:00.1648938Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T07:13:00.1689757Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T07:13:00.1690463Z creating: build/lib/ 2024-06-26T07:13:00.1775506Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T07:13:00.2215489Z inflating: build/lib/libprotobuf.a 2024-06-26T07:13:00.2225315Z inflating: build/lib/libpthreadpool.a 2024-06-26T07:13:00.2232921Z inflating: build/lib/libcpuinfo.a 2024-06-26T07:13:00.2240687Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T07:13:00.2242040Z inflating: build/lib/libclog.a 2024-06-26T07:13:00.2259799Z inflating: build/lib/libnnpack.a 2024-06-26T07:13:00.2262085Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T07:13:00.2325017Z inflating: build/lib/libgtest.a 2024-06-26T07:13:00.2399507Z inflating: build/lib/libbenchmark.a 2024-06-26T07:13:00.2461582Z inflating: build/lib/libasmjit.a 2024-06-26T07:13:00.2469184Z inflating: build/lib/libittnotify.a 2024-06-26T07:13:00.2496308Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T07:13:00.2619331Z inflating: build/lib/libgloo.a 2024-06-26T07:13:00.2640784Z inflating: build/lib/libfmt.a 2024-06-26T07:13:00.2730102Z inflating: build/lib/libc10.so 2024-06-26T07:13:00.2731858Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T07:13:00.2732874Z inflating: build/lib/libfoxi_loader.a 2024-06-26T07:13:00.2734272Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T07:13:00.2753570Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T07:13:00.2771176Z inflating: build/lib/libgmock.a 2024-06-26T07:13:00.2771862Z inflating: build/lib/libgtest_main.a 2024-06-26T07:13:00.2772805Z inflating: build/lib/libbenchmark_main.a 2024-06-26T07:13:00.3288547Z inflating: build/lib/libprotoc.a 2024-06-26T07:13:00.3659334Z inflating: build/lib/libgloo_cuda.a 2024-06-26T07:13:01.3405804Z inflating: build/lib/libdnnl.a 2024-06-26T07:13:01.3956933Z inflating: build/lib/libtensorpipe.a 2024-06-26T07:13:01.4010072Z inflating: build/lib/libc10_cuda.so 2024-06-26T07:13:01.4010771Z inflating: build/lib/libgmock_main.a 2024-06-26T07:13:01.5213506Z inflating: build/lib/libfbgemm.a 2024-06-26T07:13:01.5459025Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-26T07:13:01.5956714Z inflating: build/lib/libkineto.a 2024-06-26T07:13:01.6134964Z inflating: build/lib/libXNNPACK.a 2024-06-26T07:13:01.6176176Z inflating: build/lib/libonnx_proto.a 2024-06-26T07:13:01.6841446Z inflating: build/lib/libonnx.a 2024-06-26T07:13:04.0112563Z inflating: build/lib/libtorch_cpu.so 2024-06-26T07:13:04.0117004Z inflating: build/lib/libunbox_lib.a 2024-06-26T07:13:04.0121134Z inflating: build/lib/libshm.so 2024-06-26T07:13:05.9965980Z inflating: build/lib/libtorch_cuda.so 2024-06-26T07:13:05.9966714Z inflating: build/lib/libtorch.so 2024-06-26T07:13:06.7926270Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-26T07:13:06.7928270Z inflating: build/lib/libc10d_cuda_test.so 2024-06-26T07:13:06.9823561Z inflating: build/lib/libtorch_python.so 2024-06-26T07:13:06.9890520Z inflating: build/lib/libtorchbind_test.so 2024-06-26T07:13:06.9911487Z inflating: build/lib/libjitbackend_test.so 2024-06-26T07:13:06.9936511Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T07:13:06.9970374Z inflating: build/lib/libnnapi_backend.so 2024-06-26T07:13:06.9971056Z creating: build/bin/ 2024-06-26T07:13:07.0019881Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T07:13:07.0070279Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T07:13:07.0119607Z inflating: build/bin/c10_Device_test 2024-06-26T07:13:07.0177962Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T07:13:07.0230318Z inflating: build/bin/c10_Scalar_test 2024-06-26T07:13:07.0278238Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T07:13:07.0331512Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T07:13:07.0381477Z inflating: build/bin/c10_SymInt_test 2024-06-26T07:13:07.0435985Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T07:13:07.0490963Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T07:13:07.0542941Z inflating: build/bin/c10_Bitset_test 2024-06-26T07:13:07.0612568Z inflating: build/bin/c10_cow_test 2024-06-26T07:13:07.0660421Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T07:13:07.0709446Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T07:13:07.0758973Z inflating: build/bin/c10_Half_test 2024-06-26T07:13:07.0814166Z inflating: build/bin/c10_LeftRight_test 2024-06-26T07:13:07.0868128Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T07:13:07.0917001Z inflating: build/bin/c10_Synchronized_test 2024-06-26T07:13:07.0971309Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T07:13:07.1021796Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T07:13:07.1071827Z inflating: build/bin/c10_TypeList_test 2024-06-26T07:13:07.1119811Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T07:13:07.1170360Z inflating: build/bin/c10_accumulate_test 2024-06-26T07:13:07.1224787Z inflating: build/bin/c10_bfloat16_test 2024-06-26T07:13:07.1274047Z inflating: build/bin/c10_bit_cast_test 2024-06-26T07:13:07.1329116Z inflating: build/bin/c10_complex_math_test 2024-06-26T07:13:07.1383762Z inflating: build/bin/c10_complex_test 2024-06-26T07:13:07.1435248Z inflating: build/bin/c10_exception_test 2024-06-26T07:13:07.1485497Z inflating: build/bin/c10_flags_test 2024-06-26T07:13:07.1536547Z inflating: build/bin/c10_generic_math_test 2024-06-26T07:13:07.1696628Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T07:13:07.1747133Z inflating: build/bin/c10_irange_test 2024-06-26T07:13:07.1801305Z inflating: build/bin/c10_lazy_test 2024-06-26T07:13:07.1858728Z inflating: build/bin/c10_logging_test 2024-06-26T07:13:07.1933648Z inflating: build/bin/c10_optional_test 2024-06-26T07:13:07.1996113Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T07:13:07.2048778Z inflating: build/bin/c10_registry_test 2024-06-26T07:13:07.2195349Z inflating: build/bin/c10_small_vector_test 2024-06-26T07:13:07.2245747Z inflating: build/bin/c10_ssize_test 2024-06-26T07:13:07.2296452Z inflating: build/bin/c10_string_util_test 2024-06-26T07:13:07.2353870Z inflating: build/bin/c10_string_view_test 2024-06-26T07:13:07.2403943Z inflating: build/bin/c10_tempfile_test 2024-06-26T07:13:07.2451815Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T07:13:07.2506726Z inflating: build/bin/c10_typeid_test 2024-06-26T07:13:07.2940348Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T07:13:07.3374003Z inflating: build/bin/protoc 2024-06-26T07:13:07.3425938Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-26T07:13:07.3477696Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-26T07:13:07.3529708Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-26T07:13:07.3580492Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-26T07:13:07.3632321Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T07:13:07.3684271Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T07:13:07.3732234Z inflating: build/bin/c10_cuda_CUDATest 2024-06-26T07:13:07.3783936Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-26T07:13:07.4103700Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T07:13:07.4436586Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T07:13:07.4783014Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T07:13:07.4834870Z inflating: build/bin/FileStoreTest 2024-06-26T07:13:07.4889325Z inflating: build/bin/TCPStoreTest 2024-06-26T07:13:07.4941599Z inflating: build/bin/HashStoreTest 2024-06-26T07:13:07.4955052Z inflating: build/bin/ProcessGroupMPITest 2024-06-26T07:13:07.5007813Z inflating: build/bin/test_edge_op_registration 2024-06-26T07:13:07.5012267Z inflating: build/bin/torch_shm_manager 2024-06-26T07:13:07.5015397Z inflating: build/bin/example_allreduce 2024-06-26T07:13:07.5069316Z inflating: build/bin/test_dist_autograd 2024-06-26T07:13:07.5137138Z inflating: build/bin/test_cpp_rpc 2024-06-26T07:13:07.5139657Z inflating: build/bin/parallel_benchmark 2024-06-26T07:13:07.5206783Z inflating: build/bin/test_mobile_nnc 2024-06-26T07:13:07.5215893Z inflating: build/bin/aot_model_compiler_test 2024-06-26T07:13:07.5549647Z inflating: build/bin/test_lazy 2024-06-26T07:13:07.6695746Z inflating: build/bin/test_api 2024-06-26T07:13:07.6767352Z inflating: build/bin/Dict_test 2024-06-26T07:13:07.6818446Z inflating: build/bin/Dimname_test 2024-06-26T07:13:07.6881593Z inflating: build/bin/MaybeOwned_test 2024-06-26T07:13:07.6937778Z inflating: build/bin/NamedTensor_test 2024-06-26T07:13:07.6995701Z inflating: build/bin/apply_utils_test 2024-06-26T07:13:07.7053511Z inflating: build/bin/atest 2024-06-26T07:13:07.7115228Z inflating: build/bin/basic 2024-06-26T07:13:07.7168837Z inflating: build/bin/broadcast_test 2024-06-26T07:13:07.7218629Z inflating: build/bin/cpu_allocator_test 2024-06-26T07:13:07.7275542Z inflating: build/bin/cpu_generator_test 2024-06-26T07:13:07.7327734Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T07:13:07.7419590Z inflating: build/bin/cpu_rng_test 2024-06-26T07:13:07.7468571Z inflating: build/bin/dispatch_key_set_test 2024-06-26T07:13:07.7517949Z inflating: build/bin/dlconvertor_test 2024-06-26T07:13:07.7574994Z inflating: build/bin/extension_backend_test 2024-06-26T07:13:07.7628982Z inflating: build/bin/half_test 2024-06-26T07:13:07.7722471Z inflating: build/bin/ivalue_test 2024-06-26T07:13:07.7770967Z inflating: build/bin/lazy_tensor_test 2024-06-26T07:13:07.7824292Z inflating: build/bin/math_kernel_test 2024-06-26T07:13:07.7877348Z inflating: build/bin/memory_format_test 2024-06-26T07:13:07.7929277Z inflating: build/bin/memory_overlapping_test 2024-06-26T07:13:07.7981407Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T07:13:07.8036135Z inflating: build/bin/native_test 2024-06-26T07:13:07.8085740Z inflating: build/bin/operator_name_test 2024-06-26T07:13:07.8136113Z inflating: build/bin/operators_test 2024-06-26T07:13:07.8187019Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T07:13:07.8252971Z inflating: build/bin/pow_test 2024-06-26T07:13:07.8308577Z inflating: build/bin/quantized_test 2024-06-26T07:13:07.8357654Z inflating: build/bin/reduce_ops_test 2024-06-26T07:13:07.8407691Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T07:13:07.8463314Z inflating: build/bin/scalar_tensor_test 2024-06-26T07:13:07.8520646Z inflating: build/bin/scalar_test 2024-06-26T07:13:07.8571494Z inflating: build/bin/StorageUtils_test 2024-06-26T07:13:07.8622657Z inflating: build/bin/stride_properties_test 2024-06-26T07:13:07.8699015Z inflating: build/bin/tensor_iterator_test 2024-06-26T07:13:07.8751649Z inflating: build/bin/test_parallel 2024-06-26T07:13:07.8754591Z inflating: build/bin/thread_init_test 2024-06-26T07:13:07.8809381Z inflating: build/bin/type_ptr_test 2024-06-26T07:13:07.8867235Z inflating: build/bin/type_test 2024-06-26T07:13:07.8918116Z inflating: build/bin/undefined_tensor_test 2024-06-26T07:13:07.8919636Z inflating: build/bin/verify_api_visibility 2024-06-26T07:13:07.8987315Z inflating: build/bin/legacy_vmap_test 2024-06-26T07:13:07.9037749Z inflating: build/bin/weakref_test 2024-06-26T07:13:07.9087949Z inflating: build/bin/wrapdim_test 2024-06-26T07:13:07.9138191Z inflating: build/bin/xla_tensor_test 2024-06-26T07:13:07.9196568Z inflating: build/bin/IListRef_test 2024-06-26T07:13:07.9299719Z inflating: build/bin/List_test 2024-06-26T07:13:07.9364629Z inflating: build/bin/KernelFunction_test 2024-06-26T07:13:07.9483536Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T07:13:07.9577379Z inflating: build/bin/kernel_function_test 2024-06-26T07:13:07.9701956Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T07:13:07.9802752Z inflating: build/bin/kernel_lambda_test 2024-06-26T07:13:07.9861900Z inflating: build/bin/kernel_stackbased_test 2024-06-26T07:13:07.9955401Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T07:13:08.0005077Z inflating: build/bin/CppSignature_test 2024-06-26T07:13:08.0059502Z inflating: build/bin/backend_fallback_test 2024-06-26T07:13:08.0107110Z inflating: build/bin/op_allowlist_test 2024-06-26T07:13:08.0400017Z inflating: build/bin/op_registration_test 2024-06-26T07:13:08.0461502Z inflating: build/bin/inline_container_test 2024-06-26T07:13:08.0513073Z inflating: build/bin/cuda_apply_test 2024-06-26T07:13:08.0563466Z inflating: build/bin/cuda_allocator_test 2024-06-26T07:13:08.0620563Z inflating: build/bin/cuda_atomic_ops_test 2024-06-26T07:13:08.0673542Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-26T07:13:08.0741470Z inflating: build/bin/cuda_complex_math_test 2024-06-26T07:13:08.0799443Z inflating: build/bin/cuda_complex_test 2024-06-26T07:13:08.0847798Z inflating: build/bin/cuda_device_test 2024-06-26T07:13:08.0903792Z inflating: build/bin/cuda_cub_test 2024-06-26T07:13:08.0953675Z inflating: build/bin/cuda_dlconvertor_test 2024-06-26T07:13:08.1017008Z inflating: build/bin/cuda_distributions_test 2024-06-26T07:13:08.1072550Z inflating: build/bin/cuda_generator_test 2024-06-26T07:13:08.1121206Z inflating: build/bin/cuda_half_test 2024-06-26T07:13:08.1171208Z inflating: build/bin/cuda_integer_divider_test 2024-06-26T07:13:08.1219383Z inflating: build/bin/cuda_optional_test 2024-06-26T07:13:08.1270036Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-26T07:13:08.1321546Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-26T07:13:08.1370444Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-26T07:13:08.1429884Z inflating: build/bin/cuda_stream_test 2024-06-26T07:13:08.1477996Z inflating: build/bin/cuda_cudnn_test 2024-06-26T07:13:08.1529009Z inflating: build/bin/cuda_vectorized_test 2024-06-26T07:13:08.1543974Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T07:13:08.1607979Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T07:13:08.1664305Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-26T07:13:08.1726262Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-26T07:13:08.1787109Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-26T07:13:08.2594025Z inflating: build/bin/test_tensorexpr 2024-06-26T07:13:08.3154183Z inflating: build/bin/test_jit 2024-06-26T07:13:08.3154809Z creating: .additional_ci_files/ 2024-06-26T07:13:08.3208532Z inflating: .additional_ci_files/test-times.json 2024-06-26T07:13:08.3420021Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T07:13:08.3448078Z ##[group]Run rm artifacts.zip 2024-06-26T07:13:08.3448554Z rm artifacts.zip 2024-06-26T07:13:08.3457123Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:13:08.3457657Z env: 2024-06-26T07:13:08.3457965Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:08.3458441Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:08.3458961Z ##[endgroup] 2024-06-26T07:13:08.4162789Z ##[group]Run df -H 2024-06-26T07:13:08.4163125Z df -H 2024-06-26T07:13:08.4170798Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:13:08.4171460Z env: 2024-06-26T07:13:08.4171749Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:08.4172447Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:08.4172972Z ##[endgroup] 2024-06-26T07:13:08.4199371Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T07:13:08.4199958Z devtmpfs 34G 0 34G 0% /dev 2024-06-26T07:13:08.4200872Z tmpfs 34G 0 34G 0% /dev/shm 2024-06-26T07:13:08.4201505Z tmpfs 34G 455k 34G 1% /run 2024-06-26T07:13:08.4202019Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-06-26T07:13:08.4202536Z /dev/nvme0n1p1 162G 43G 119G 27% / 2024-06-26T07:13:08.4243777Z Prepare all required actions 2024-06-26T07:13:08.4244237Z Getting action download info 2024-06-26T07:13:08.5500819Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T07:13:08.5501311Z with: 2024-06-26T07:13:08.5501571Z env: 2024-06-26T07:13:08.5501856Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:08.5502351Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:08.5502859Z ##[endgroup] 2024-06-26T07:13:08.5535305Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:13:08.5535797Z with: 2024-06-26T07:13:08.5536071Z name: td_results 2024-06-26T07:13:08.5536426Z s3-bucket: gha-artifacts 2024-06-26T07:13:08.5536804Z region: us-east-1 2024-06-26T07:13:08.5537120Z env: 2024-06-26T07:13:08.5537404Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:08.5537900Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:08.5538427Z ##[endgroup] 2024-06-26T07:13:08.9623100Z (node:23567) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:13:08.9623970Z 2024-06-26T07:13:08.9624252Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:13:08.9625057Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:13:08.9626051Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:13:09.0353704Z Found 0 objects with prefix pytorch/pytorch/9673646806/td_results/ 2024-06-26T07:13:09.0360218Z Artifact download has finished successfully 2024-06-26T07:13:09.0489239Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T07:13:09.0489756Z mkdir -p .additional_ci_files 2024-06-26T07:13:09.0490355Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T07:13:09.0498704Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:13:09.0499246Z env: 2024-06-26T07:13:09.0499571Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:09.0500095Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:09.0500637Z ##[endgroup] 2024-06-26T07:13:09.0536830Z mv: cannot stat 'td_results.json': No such file or directory 2024-06-26T07:13:09.0547166Z ##[error]Process completed with exit code 1. 2024-06-26T07:13:09.0593671Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T07:13:09.0594182Z .github/scripts/parse_ref.py 2024-06-26T07:13:09.0601712Z shell: /usr/bin/bash -e {0} 2024-06-26T07:13:09.0602075Z env: 2024-06-26T07:13:09.0602367Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:09.0602857Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:09.0603385Z ##[endgroup] 2024-06-26T07:13:09.0834695Z Prepare all required actions 2024-06-26T07:13:09.0874532Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T07:13:09.0875028Z with: 2024-06-26T07:13:09.0875598Z github-token: *** 2024-06-26T07:13:09.0875914Z env: 2024-06-26T07:13:09.0876194Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:09.0876672Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:09.0877196Z ##[endgroup] 2024-06-26T07:13:09.0892661Z ##[group]Run set -eux 2024-06-26T07:13:09.0893019Z set -eux 2024-06-26T07:13:09.0893660Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T07:13:09.0901349Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:13:09.0902017Z env: 2024-06-26T07:13:09.0902308Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:09.0902792Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:09.0903557Z GITHUB_TOKEN: *** 2024-06-26T07:13:09.0903878Z ##[endgroup] 2024-06-26T07:13:09.0923738Z + python3 .github/scripts/get_workflow_job_id.py 9673646806 i-06b804ca2116ce394 2024-06-26T07:13:09.7452461Z setting job-id=26688915472 2024-06-26T07:13:09.7453849Z setting job-name=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:13:09.7668376Z Prepare all required actions 2024-06-26T07:13:09.7668845Z Getting action download info 2024-06-26T07:13:09.8804868Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T07:13:09.8805358Z with: 2024-06-26T07:13:09.8805846Z github-token: *** 2024-06-26T07:13:09.8813834Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:13:09.8822684Z job-name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:13:09.8823744Z env: 2024-06-26T07:13:09.8824027Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:09.8824515Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:09.8825037Z ##[endgroup] 2024-06-26T07:13:09.8867771Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:13:09.8868387Z with: 2024-06-26T07:13:09.8868672Z shell: bash 2024-06-26T07:13:09.8868985Z timeout_minutes: 10 2024-06-26T07:13:09.8869342Z max_attempts: 5 2024-06-26T07:13:09.8869725Z retry_wait_seconds: 30 2024-06-26T07:13:09.8870986Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:13:09.8872318Z polling_interval_seconds: 1 2024-06-26T07:13:09.8872714Z warning_on_retry: true 2024-06-26T07:13:09.8873073Z continue_on_error: false 2024-06-26T07:13:09.8873440Z env: 2024-06-26T07:13:09.8873727Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:09.8874211Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:09.8874900Z GITHUB_TOKEN: *** 2024-06-26T07:13:09.8875238Z ##[endgroup] 2024-06-26T07:13:09.9367902Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:13:10.1533725Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:13:10.1697110Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-06-26T07:13:10.1838321Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-06-26T07:13:10.1848498Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-06-26T07:13:10.2042637Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2) 2024-06-26T07:13:10.2052629Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-06-26T07:13:10.2072451Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-06-26T07:13:10.9370209Z Command completed after 1 attempt(s). 2024-06-26T07:13:10.9410036Z ##[group]Run set -x 2024-06-26T07:13:10.9410383Z set -x 2024-06-26T07:13:10.9410689Z  2024-06-26T07:13:10.9411286Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:13:10.9412359Z # in runner workspace 2024-06-26T07:13:10.9413071Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T07:13:10.9421134Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:13:10.9421674Z env: 2024-06-26T07:13:10.9421966Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:10.9422449Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:10.9422975Z ##[endgroup] 2024-06-26T07:13:10.9444573Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T07:13:10.9647104Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:13:10.9647704Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:13:10.9648199Z echo "Job name: ${JOB_NAME}" 2024-06-26T07:13:10.9648637Z  2024-06-26T07:13:10.9649236Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:13:10.9649979Z # in runner workspace 2024-06-26T07:13:10.9650610Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T07:13:10.9651315Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T07:13:10.9651820Z  --job-name "${JOB_NAME}" \ 2024-06-26T07:13:10.9659746Z  --test-matrix "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-06-26T07:13:10.9667921Z  --selected-test-configs "" \ 2024-06-26T07:13:10.9668399Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T07:13:10.9668845Z  --tag "${TAG}" \ 2024-06-26T07:13:10.9669250Z  --event-name "${EVENT_NAME}" \ 2024-06-26T07:13:10.9669721Z  --schedule "${SCHEDULE}" \ 2024-06-26T07:13:10.9670263Z  --branch "${HEAD_BRANCH}" 2024-06-26T07:13:10.9677967Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:13:10.9678493Z env: 2024-06-26T07:13:10.9678771Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:10.9679231Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:10.9679936Z GITHUB_TOKEN: *** 2024-06-26T07:13:10.9680843Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:13:10.9681837Z PR_NUMBER: 2024-06-26T07:13:10.9682149Z TAG: ciflow/inductor/129470 2024-06-26T07:13:10.9682526Z EVENT_NAME: push 2024-06-26T07:13:10.9682843Z SCHEDULE: 2024-06-26T07:13:10.9683143Z HEAD_BRANCH: 2024-06-26T07:13:10.9683442Z ##[endgroup] 2024-06-26T07:13:10.9719293Z Workflow: inductor-periodic 2024-06-26T07:13:10.9720437Z Job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:13:11.1889974Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T07:13:11.3178422Z ##[group]Run echo "Filtered matrix:" 2024-06-26T07:13:11.3178919Z echo "Filtered matrix:" 2024-06-26T07:13:11.3186885Z echo "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-06-26T07:13:11.3195140Z  2024-06-26T07:13:11.3195435Z echo 2024-06-26T07:13:11.3195843Z echo "Is the current job unstable? False" 2024-06-26T07:13:11.3196348Z  2024-06-26T07:13:11.3196625Z echo 2024-06-26T07:13:11.3197009Z echo "Is keep-going label set? False" 2024-06-26T07:13:11.3197490Z  2024-06-26T07:13:11.3197765Z echo 2024-06-26T07:13:11.3198104Z echo "Renabled issues? " 2024-06-26T07:13:11.3205937Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:13:11.3206590Z env: 2024-06-26T07:13:11.3206873Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:11.3207365Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:11.3207897Z ##[endgroup] 2024-06-26T07:13:11.3227974Z Filtered matrix: 2024-06-26T07:13:11.3237320Z {include: [{config: dynamo_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-06-26T07:13:11.3245283Z 2024-06-26T07:13:11.3245438Z Is the current job unstable? False 2024-06-26T07:13:11.3245757Z 2024-06-26T07:13:11.3245968Z Is keep-going label set? False 2024-06-26T07:13:11.3246244Z 2024-06-26T07:13:11.3246379Z Renabled issues? 2024-06-26T07:13:11.3289449Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:13:11.3290209Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:13:11.3298385Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:13:11.3298922Z env: 2024-06-26T07:13:11.3299213Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:11.3299704Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:11.3300234Z JOB_TIMEOUT: 240 2024-06-26T07:13:11.3300554Z ##[endgroup] 2024-06-26T07:13:11.3370916Z ##[group]Run set -x 2024-06-26T07:13:11.3371312Z set -x 2024-06-26T07:13:11.3371623Z  2024-06-26T07:13:11.3372153Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T07:13:11.3372741Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T07:13:11.3373351Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T07:13:11.3373905Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-26T07:13:11.3374366Z else 2024-06-26T07:13:11.3374717Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:13:11.3375178Z fi 2024-06-26T07:13:11.3375464Z  2024-06-26T07:13:11.3375969Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T07:13:11.3376798Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T07:13:11.3377543Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T07:13:11.3378172Z # shellcheck disable=SC2086,SC2090 2024-06-26T07:13:11.3378655Z container_name=$(docker run \ 2024-06-26T07:13:11.3379115Z  ${GPU_FLAG:-} \ 2024-06-26T07:13:11.3379509Z  -e BUILD_ENVIRONMENT \ 2024-06-26T07:13:11.3379935Z  -e PR_NUMBER \ 2024-06-26T07:13:11.3380308Z  -e GITHUB_ACTIONS \ 2024-06-26T07:13:11.3380728Z  -e GITHUB_REPOSITORY \ 2024-06-26T07:13:11.3381165Z  -e GITHUB_WORKFLOW \ 2024-06-26T07:13:11.3381579Z  -e GITHUB_JOB \ 2024-06-26T07:13:11.3382072Z  -e GITHUB_RUN_ID \ 2024-06-26T07:13:11.3382492Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T07:13:11.3382932Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T07:13:11.3383425Z  -e JOB_ID \ 2024-06-26T07:13:11.3383782Z  -e JOB_NAME \ 2024-06-26T07:13:11.3384149Z  -e BASE_SHA \ 2024-06-26T07:13:11.3384505Z  -e BRANCH \ 2024-06-26T07:13:11.3384839Z  -e SHA1 \ 2024-06-26T07:13:11.3385197Z  -e AWS_DEFAULT_REGION \ 2024-06-26T07:13:11.3385622Z  -e IN_WHEEL_TEST \ 2024-06-26T07:13:11.3386018Z  -e SHARD_NUMBER \ 2024-06-26T07:13:11.3386397Z  -e TEST_CONFIG \ 2024-06-26T07:13:11.3386799Z  -e NUM_TEST_SHARDS \ 2024-06-26T07:13:11.3387232Z  -e REENABLED_ISSUES \ 2024-06-26T07:13:11.3387677Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T07:13:11.3388129Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T07:13:11.3388560Z  -e NO_TEST_TIMEOUT \ 2024-06-26T07:13:11.3388960Z  -e NO_TD \ 2024-06-26T07:13:11.3389313Z  -e TD_DISTRIBUTED \ 2024-06-26T07:13:11.3389720Z  -e PR_LABELS \ 2024-06-26T07:13:11.3390158Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T07:13:11.3390658Z  -e SCCACHE_BUCKET \ 2024-06-26T07:13:11.3391073Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-26T07:13:11.3391521Z  -e XLA_CUDA \ 2024-06-26T07:13:11.3391959Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T07:13:11.3392512Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T07:13:11.3393079Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T07:13:11.3393635Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-26T07:13:11.3394139Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-26T07:13:11.3394590Z  -e DASHBOARD_TAG \ 2024-06-26T07:13:11.3395092Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T07:13:11.3395703Z  --security-opt seccomp=unconfined \ 2024-06-26T07:13:11.3396214Z  --cap-add=SYS_PTRACE \ 2024-06-26T07:13:11.3396632Z  --ipc=host \ 2024-06-26T07:13:11.3397007Z  --shm-size="${SHM_SIZE}" \ 2024-06-26T07:13:11.3397427Z  --tty \ 2024-06-26T07:13:11.3397760Z  --detach \ 2024-06-26T07:13:11.3398134Z  --name="${container_name}" \ 2024-06-26T07:13:11.3398580Z  --user jenkins \ 2024-06-26T07:13:11.3399187Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T07:13:11.3399814Z  -w /var/lib/jenkins/workspace \ 2024-06-26T07:13:11.3400299Z  "${DOCKER_IMAGE}" 2024-06-26T07:13:11.3400669Z ) 2024-06-26T07:13:11.3401113Z # Propagate download.pytorch.org IP to container 2024-06-26T07:13:11.3402163Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-26T07:13:11.3403274Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-26T07:13:11.3404301Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-26T07:13:11.3411871Z shell: /usr/bin/bash -e {0} 2024-06-26T07:13:11.3412745Z env: 2024-06-26T07:13:11.3413048Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:11.3413535Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:11.3414228Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:13:11.3414793Z PR_NUMBER: 2024-06-26T07:13:11.3415150Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T07:13:11.3415622Z GITHUB_WORKFLOW: inductor-periodic 2024-06-26T07:13:11.3416065Z GITHUB_JOB: test 2024-06-26T07:13:11.3416420Z GITHUB_RUN_ID: 9673646806 2024-06-26T07:13:11.3416819Z GITHUB_RUN_NUMBER: 45567 2024-06-26T07:13:11.3417196Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T07:13:11.3417559Z JOB_ID: 26688915472 2024-06-26T07:13:11.3418524Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:13:11.3419637Z BRANCH: 2024-06-26T07:13:11.3419992Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:11.3420565Z BASE_SHA: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:11.3421111Z TEST_CONFIG: dynamo_eager_torchbench 2024-06-26T07:13:11.3421561Z SHARD_NUMBER: 1 2024-06-26T07:13:11.3421886Z NUM_TEST_SHARDS: 2 2024-06-26T07:13:11.3422243Z REENABLED_ISSUES: 2024-06-26T07:13:11.3422621Z CONTINUE_THROUGH_ERROR: False 2024-06-26T07:13:11.3423031Z VERBOSE_TEST_LOGS: False 2024-06-26T07:13:11.3423466Z NO_TEST_TIMEOUT: False 2024-06-26T07:13:11.3423825Z NO_TD: False 2024-06-26T07:13:11.3424159Z TD_DISTRIBUTED: False 2024-06-26T07:13:11.3424611Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T07:13:11.3425204Z SCCACHE_S3_KEY_PREFIX: inductor-periodic 2024-06-26T07:13:11.3425683Z SHM_SIZE: 2g 2024-06-26T07:13:11.3426919Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:13:11.3428242Z XLA_CUDA: 2024-06-26T07:13:11.3428779Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:13:11.3429472Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T07:13:11.3429960Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T07:13:11.3430407Z DASHBOARD_TAG: 2024-06-26T07:13:11.3430982Z HUGGING_FACE_HUB_TOKEN: *** 2024-06-26T07:13:11.3431379Z ##[endgroup] 2024-06-26T07:13:11.3453770Z + [[ dynamo_eager_torchbench == \m\u\l\t\i\g\p\u ]] 2024-06-26T07:13:11.3454510Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-06-26T07:13:11.3455064Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:13:11.3461254Z +++ nproc --ignore=2 2024-06-26T07:13:11.3475024Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9673646806 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:13:22.5913524Z + container_name=272d959298f66930256ee50b307d2e87a6d4a8766a7c2b0ccd856d88c6b87252 2024-06-26T07:13:22.5914302Z + grep download.pytorch.org /etc/hosts 2024-06-26T07:13:22.5916669Z + docker exec -i 272d959298f66930256ee50b307d2e87a6d4a8766a7c2b0ccd856d88c6b87252 sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-26T07:13:22.6529730Z + echo DOCKER_CONTAINER_ID=272d959298f66930256ee50b307d2e87a6d4a8766a7c2b0ccd856d88c6b87252 2024-06-26T07:13:22.6532242Z ++ echo dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:13:22.6534131Z + docker exec -t 272d959298f66930256ee50b307d2e87a6d4a8766a7c2b0ccd856d88c6b87252 sh -c 'pip install dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-26T07:13:23.0304134Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb8c4c54) 2024-06-26T07:13:23.3455830Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T07:13:23.3460195Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T07:13:23.3462581Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T07:13:23.3465041Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T07:13:23.3467660Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T07:13:23.3470866Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T07:13:23.3496710Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.3.0) 2024-06-26T07:13:23.3555079Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.21.2) 2024-06-26T07:13:23.3987485Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T07:13:23.4145764Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T07:13:24.7272337Z Installing collected packages: torch 2024-06-26T07:13:33.4350769Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-06-26T07:13:33.4353550Z timm 0.9.7 requires torchvision, which is not installed. 2024-06-26T07:13:33.4354794Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T07:13:33.5315132Z ++ dirname .ci/pytorch/test.sh 2024-06-26T07:13:33.5319577Z + source .ci/pytorch/common.sh 2024-06-26T07:13:33.5320612Z +++ dirname .ci/pytorch/common.sh 2024-06-26T07:13:33.5326493Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T07:13:33.5327525Z +++ declare -f -t trap_add 2024-06-26T07:13:33.5334299Z ++ set -ex 2024-06-26T07:13:33.5335122Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:13:33.5335699Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T07:13:33.5336219Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-06-26T07:13:33.5337329Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-26T07:13:33.5346581Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-26T07:13:33.5347052Z + trap_add cleanup_workspace EXIT 2024-06-26T07:13:33.5347494Z + trap_add_cmd=cleanup_workspace 2024-06-26T07:13:33.5347878Z + shift 2024-06-26T07:13:33.5348172Z + for trap_add_name in "$@" 2024-06-26T07:13:33.5351465Z +++ trap -p EXIT 2024-06-26T07:13:33.5353296Z ++ eval 'extract_trap_cmd ' 2024-06-26T07:13:33.5353801Z +++ extract_trap_cmd 2024-06-26T07:13:33.5354203Z +++ printf '%s\n' '' 2024-06-26T07:13:33.5354597Z ++ printf '%s\n' cleanup_workspace 2024-06-26T07:13:33.5356154Z + trap -- ' 2024-06-26T07:13:33.5356548Z cleanup_workspace' EXIT 2024-06-26T07:13:33.5357027Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-26T07:13:33.9988480Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-26T07:13:34.0001720Z + echo 'Environment variables:' 2024-06-26T07:13:34.0002226Z Environment variables: 2024-06-26T07:13:34.0002566Z + env 2024-06-26T07:13:34.0007139Z INSTALLED_DB=yes 2024-06-26T07:13:34.0007643Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:13:34.0008497Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:13:34.0009073Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:13:34.0009826Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:13:34.0010716Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:13:34.0011492Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:13:34.0012536Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:13:34.0013261Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:13:34.0014013Z HOSTNAME=272d959298f6 2024-06-26T07:13:34.0015028Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_afe4a476-9747-4f1c-82cf-ed04a73ebc78 2024-06-26T07:13:34.0015932Z GITHUB_ACTION=__self 2024-06-26T07:13:34.0016306Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:13:34.0020869Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:13:34.0025546Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:13:34.0026087Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:13:34.0026461Z GITHUB_RUN_NUMBER=45567 2024-06-26T07:13:34.0026838Z TEST_CONFIG=dynamo_eager_torchbench 2024-06-26T07:13:34.0027272Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:13:34.0027774Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:13:34.0028285Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:13:34.0028863Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:13:34.0029418Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:13:34.0029941Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:13:34.0030525Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:13:34.0031031Z GITHUB_REF_TYPE=tag 2024-06-26T07:13:34.0031367Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:13:34.0031764Z NCCL_VERSION=2.17.1-1 2024-06-26T07:13:34.0032382Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:34.0032869Z XLA_CUDA= 2024-06-26T07:13:34.0033393Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:13:34.0034998Z *** 2024-06-26T07:13:34.0035314Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:13:34.0035745Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:13:34.0036129Z GITHUB_ACTIONS=true 2024-06-26T07:13:34.0036466Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:34.0037014Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:13:34.0037586Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:13:34.0038093Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:34.0038622Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:13:34.0039129Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:34.0040148Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:13:34.0041007Z UCC_HOME=/usr 2024-06-26T07:13:34.0041340Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:13:34.0041770Z VERBOSE_TEST_LOGS=False 2024-06-26T07:13:34.0042124Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:13:34.0042586Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:13:34.0043121Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:13:34.0043610Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:13:34.0044001Z SHARD_NUMBER=1 2024-06-26T07:13:34.0044312Z GITHUB_REF_PROTECTED=false 2024-06-26T07:13:34.0044682Z HOME=/var/lib/jenkins 2024-06-26T07:13:34.0045052Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:13:34.0045677Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:13:34.0046174Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:13:34.0046752Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-06-26T07:13:34.0047194Z CUDA_VERSION=12.1.1 2024-06-26T07:13:34.0047616Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:13:34.0048084Z NUM_TEST_SHARDS=2 2024-06-26T07:13:34.0048395Z UCX_HOME=/usr 2024-06-26T07:13:34.0048934Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:13:34.0050117Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_afe4a476-9747-4f1c-82cf-ed04a73ebc78 2024-06-26T07:13:34.0051713Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:13:34.0053614Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_afe4a476-9747-4f1c-82cf-ed04a73ebc78 2024-06-26T07:13:34.0054907Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:13:34.0055635Z GITHUB_EVENT_NAME=push 2024-06-26T07:13:34.0055966Z DASHBOARD_TAG= 2024-06-26T07:13:34.0056273Z GITHUB_RUN_ID=9673646806 2024-06-26T07:13:34.0056739Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:13:34.0057295Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:13:34.0058387Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_afe4a476-9747-4f1c-82cf-ed04a73ebc78 2024-06-26T07:13:34.0059402Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:13:34.0059825Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:13:34.0060214Z PR_NUMBER= 2024-06-26T07:13:34.0060509Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:13:34.0060856Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:13:34.0061321Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:13:34.0061802Z TERM=xterm 2024-06-26T07:13:34.0062145Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:13:34.0062587Z INSTALLED_VISION=yes 2024-06-26T07:13:34.0062898Z BRANCH= 2024-06-26T07:13:34.0063187Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:13:34.0063711Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:13:34.0064160Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:13:34.0064956Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:13:34.0065789Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:13:34.0066411Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:13:34.0066920Z REENABLED_ISSUES= 2024-06-26T07:13:34.0067216Z SHLVL=1 2024-06-26T07:13:34.0067483Z MAX_JOBS=14 2024-06-26T07:13:34.0067801Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:13:34.0068161Z NVARCH=x86_64 2024-06-26T07:13:34.0068467Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:13:34.0068942Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:34.0069512Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:13:34.0070012Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:13:34.0070729Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:13:34.0071356Z GITHUB_JOB=test 2024-06-26T07:13:34.0071758Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:13:34.0072345Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:13:34.0072898Z NO_TEST_TIMEOUT=False 2024-06-26T07:13:34.0073234Z TD_DISTRIBUTED=False 2024-06-26T07:13:34.0073618Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:13:34.0074077Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:13:34.0074513Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:13:34.0074894Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:13:34.0075253Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:13:34.0075611Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:13:34.0076757Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:13:34.0078040Z GITHUB_BASE_REF= 2024-06-26T07:13:34.0078365Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:13:34.0078752Z CI=true 2024-06-26T07:13:34.0079065Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:13:34.0079509Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:13:34.0079896Z JOB_ID=26688915472 2024-06-26T07:13:34.0080216Z INSTALLED_PROTOBUF=yes 2024-06-26T07:13:34.0080552Z GITHUB_HEAD_REF= 2024-06-26T07:13:34.0080855Z GITHUB_ACTION_REF= 2024-06-26T07:13:34.0081293Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:13:34.0081850Z GITHUB_WORKFLOW=inductor-periodic 2024-06-26T07:13:34.0082272Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:13:34.0083237Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_afe4a476-9747-4f1c-82cf-ed04a73ebc78 2024-06-26T07:13:34.0084148Z NO_TD=False 2024-06-26T07:13:34.0084477Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:13:34.0084893Z _=/usr/bin/env 2024-06-26T07:13:34.0085370Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T07:13:34.0151630Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-26T07:13:34.0152819Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T07:13:34.0153974Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-26T07:13:34.0154925Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-26T07:13:34.0155597Z + BUILD_DIR=build 2024-06-26T07:13:34.0155947Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T07:13:34.0156464Z + BUILD_BIN_DIR=build/bin 2024-06-26T07:13:34.0156930Z + SHARD_NUMBER=1 2024-06-26T07:13:34.0157263Z + NUM_TEST_SHARDS=2 2024-06-26T07:13:34.0157591Z + export VALGRIND=ON 2024-06-26T07:13:34.0157918Z + VALGRIND=ON 2024-06-26T07:13:34.0158387Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-06-26T07:13:34.0158917Z + [[ 0 == \1 ]] 2024-06-26T07:13:34.0159219Z + [[ False == \1 ]] 2024-06-26T07:13:34.0159705Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-06-26T07:13:34.0160276Z ++ realpath build/custom_test_artifacts 2024-06-26T07:13:34.0162976Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-26T07:13:34.0163832Z + [[ -n '' ]] 2024-06-26T07:13:34.0164185Z + echo 'Environment variables' 2024-06-26T07:13:34.0164578Z Environment variables 2024-06-26T07:13:34.0164913Z + env 2024-06-26T07:13:34.0167753Z INSTALLED_DB=yes 2024-06-26T07:13:34.0168273Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:13:34.0168986Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:13:34.0169544Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:13:34.0170431Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:13:34.0171263Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:13:34.0171833Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:13:34.0172667Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:13:34.0173256Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:13:34.0173899Z HOSTNAME=272d959298f6 2024-06-26T07:13:34.0174959Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_afe4a476-9747-4f1c-82cf-ed04a73ebc78 2024-06-26T07:13:34.0175850Z GITHUB_ACTION=__self 2024-06-26T07:13:34.0176200Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:13:34.0180515Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:13:34.0185153Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:13:34.0185685Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:13:34.0186036Z GITHUB_RUN_NUMBER=45567 2024-06-26T07:13:34.0186400Z TEST_CONFIG=dynamo_eager_torchbench 2024-06-26T07:13:34.0186842Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:13:34.0187321Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:13:34.0187800Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:13:34.0188258Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:13:34.0188694Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:13:34.0189147Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:13:34.0189663Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:13:34.0190145Z GITHUB_REF_TYPE=tag 2024-06-26T07:13:34.0190479Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:13:34.0190856Z NCCL_VERSION=2.17.1-1 2024-06-26T07:13:34.0191255Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:34.0191719Z XLA_CUDA= 2024-06-26T07:13:34.0192206Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:13:34.0192611Z *** 2024-06-26T07:13:34.0192894Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:13:34.0193301Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:13:34.0193674Z GITHUB_ACTIONS=true 2024-06-26T07:13:34.0194001Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:34.0194507Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:13:34.0195076Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:13:34.0195558Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:34.0196073Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:13:34.0196577Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:34.0197567Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:13:34.0198399Z UCC_HOME=/usr 2024-06-26T07:13:34.0198728Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:13:34.0199141Z VERBOSE_TEST_LOGS=False 2024-06-26T07:13:34.0199490Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:13:34.0199952Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:13:34.0200472Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:13:34.0200950Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:13:34.0201341Z SHARD_NUMBER=1 2024-06-26T07:13:34.0201646Z GITHUB_REF_PROTECTED=false 2024-06-26T07:13:34.0202010Z HOME=/var/lib/jenkins 2024-06-26T07:13:34.0202388Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:13:34.0202942Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:13:34.0203434Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:13:34.0204011Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-06-26T07:13:34.0204504Z CUDA_VERSION=12.1.1 2024-06-26T07:13:34.0204931Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:13:34.0205382Z NUM_TEST_SHARDS=2 2024-06-26T07:13:34.0205690Z UCX_HOME=/usr 2024-06-26T07:13:34.0206229Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:13:34.0207392Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_afe4a476-9747-4f1c-82cf-ed04a73ebc78 2024-06-26T07:13:34.0208960Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:13:34.0210479Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_afe4a476-9747-4f1c-82cf-ed04a73ebc78 2024-06-26T07:13:34.0211696Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:13:34.0212840Z GITHUB_EVENT_NAME=push 2024-06-26T07:13:34.0213273Z DASHBOARD_TAG= 2024-06-26T07:13:34.0213705Z GITHUB_RUN_ID=9673646806 2024-06-26T07:13:34.0214353Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:13:34.0215114Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:13:34.0216613Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_afe4a476-9747-4f1c-82cf-ed04a73ebc78 2024-06-26T07:13:34.0218170Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:13:34.0218763Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:13:34.0219285Z PR_NUMBER= 2024-06-26T07:13:34.0219695Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:13:34.0220159Z VALGRIND=ON 2024-06-26T07:13:34.0220583Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:13:34.0221219Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:13:34.0221937Z TERM=xterm 2024-06-26T07:13:34.0222458Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:13:34.0223098Z INSTALLED_VISION=yes 2024-06-26T07:13:34.0223678Z BRANCH= 2024-06-26T07:13:34.0224099Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:13:34.0224722Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:13:34.0225439Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:13:34.0226639Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:13:34.0227904Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:13:34.0228675Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:13:34.0229399Z REENABLED_ISSUES= 2024-06-26T07:13:34.0229845Z SHLVL=1 2024-06-26T07:13:34.0230220Z MAX_JOBS=14 2024-06-26T07:13:34.0230685Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:13:34.0231201Z NVARCH=x86_64 2024-06-26T07:13:34.0231657Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:13:34.0232346Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:34.0233156Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:13:34.0233900Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:13:34.0234976Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:13:34.0235895Z GITHUB_JOB=test 2024-06-26T07:13:34.0236523Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:13:34.0237406Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:13:34.0238186Z NO_TEST_TIMEOUT=False 2024-06-26T07:13:34.0238673Z TD_DISTRIBUTED=False 2024-06-26T07:13:34.0239274Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:13:34.0239999Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:13:34.0240650Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:13:34.0241201Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:13:34.0241708Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:13:34.0242247Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:13:34.0243919Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:13:34.0245810Z GITHUB_BASE_REF= 2024-06-26T07:13:34.0246295Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:13:34.0246866Z CI=true 2024-06-26T07:13:34.0247368Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:13:34.0264330Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:13:34.0264900Z JOB_ID=26688915472 2024-06-26T07:13:34.0265368Z INSTALLED_PROTOBUF=yes 2024-06-26T07:13:34.0265867Z GITHUB_HEAD_REF= 2024-06-26T07:13:34.0266341Z GITHUB_ACTION_REF= 2024-06-26T07:13:34.0267030Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:13:34.0267860Z GITHUB_WORKFLOW=inductor-periodic 2024-06-26T07:13:34.0268479Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:13:34.0269954Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_afe4a476-9747-4f1c-82cf-ed04a73ebc78 2024-06-26T07:13:34.0271282Z NO_TD=False 2024-06-26T07:13:34.0271731Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:13:34.0272298Z _=/usr/bin/env 2024-06-26T07:13:34.0272856Z + echo 'Testing pytorch' 2024-06-26T07:13:34.0273398Z Testing pytorch 2024-06-26T07:13:34.0273925Z + export LANG=C.UTF-8 2024-06-26T07:13:34.0274463Z + LANG=C.UTF-8 2024-06-26T07:13:34.0274904Z + PR_NUMBER= 2024-06-26T07:13:34.0275420Z + [[ dynamo_eager_torchbench == \d\e\f\a\u\l\t ]] 2024-06-26T07:13:34.0276272Z + [[ dynamo_eager_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T07:13:34.0277076Z + [[ dynamo_eager_torchbench == \s\l\o\w ]] 2024-06-26T07:13:34.0278120Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-06-26T07:13:34.0279410Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-06-26T07:13:34.0280279Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:13:34.0281055Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:13:34.0281784Z + [[ dynamo_eager_torchbench == *crossref* ]] 2024-06-26T07:13:34.0282697Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:13:34.0283655Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-06-26T07:13:34.0284629Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:13:34.0285534Z + pip_install --user ninja==1.10.2 2024-06-26T07:13:34.0286382Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T07:13:34.3911440Z Collecting ninja==1.10.2 2024-06-26T07:13:34.4073363Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T07:13:34.4227817Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T07:13:35.7121115Z Installing collected packages: ninja 2024-06-26T07:13:35.7189113Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T07:13:35.7190525Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T07:13:35.7231467Z Successfully installed ninja-1.10.2 2024-06-26T07:13:35.8269866Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:13:35.8272304Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:13:35.8273984Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:13:35.8274527Z + install_tlparse 2024-06-26T07:13:35.8274936Z + pip_install --user tlparse==0.3.7 2024-06-26T07:13:35.8275510Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T07:13:36.1821811Z Collecting tlparse==0.3.7 2024-06-26T07:13:36.1981439Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T07:13:36.2093847Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T07:13:37.5381278Z Installing collected packages: tlparse 2024-06-26T07:13:37.5748414Z Successfully installed tlparse-0.3.7 2024-06-26T07:13:37.6710488Z ++ python -m site --user-base 2024-06-26T07:13:37.6864809Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:13:37.6867024Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-06-26T07:13:37.6867719Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-06-26T07:13:37.6868601Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:13:37.6869948Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-06-26T07:13:37.6871638Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-06-26T07:13:37.6872692Z + cd test 2024-06-26T07:13:37.6873471Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T07:13:39.1622770Z + [[ dynamo_eager_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T07:13:39.1623661Z + [[ dynamo_eager_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T07:13:39.1624495Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:13:39.1625026Z + pushd test 2024-06-26T07:13:39.1625356Z ~/workspace/test ~/workspace 2024-06-26T07:13:39.1625901Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T07:13:40.6401192Z + CUDA_VERSION=12.1 2024-06-26T07:13:40.6402259Z + '[' 12.1 == 12.4 ']' 2024-06-26T07:13:40.6402724Z + ISCUDA124= 2024-06-26T07:13:40.6403130Z + popd 2024-06-26T07:13:40.6403505Z ~/workspace 2024-06-26T07:13:40.6407430Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T07:13:40.6408071Z + [[ dynamo_eager_torchbench == *dynamo_eager* ]] 2024-06-26T07:13:40.6408808Z + DYNAMO_BENCHMARK_FLAGS+=(--backend eager) 2024-06-26T07:13:40.6409319Z + [[ dynamo_eager_torchbench == *dynamic* ]] 2024-06-26T07:13:40.6409836Z + [[ dynamo_eager_torchbench == *cpu_inductor* ]] 2024-06-26T07:13:40.6410407Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T07:13:40.6435386Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:13:40.6436331Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-06-26T07:13:40.6437435Z + cd test 2024-06-26T07:13:40.6437906Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T07:13:41.9789275Z PyTorch built with: 2024-06-26T07:13:41.9789952Z - GCC 9.4 2024-06-26T07:13:41.9790402Z - C++ Version: 201703 2024-06-26T07:13:41.9791359Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:13:41.9792543Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:13:41.9793255Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:13:41.9793785Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T07:13:41.9794293Z - NNPACK is enabled 2024-06-26T07:13:41.9794685Z - CPU capability usage: AVX2 2024-06-26T07:13:41.9795120Z - CUDA Runtime 12.1 2024-06-26T07:13:41.9795653Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-06-26T07:13:41.9796295Z - CuDNN 90.1 (built against CUDA 12.4) 2024-06-26T07:13:41.9796768Z - Magma 2.6.1 2024-06-26T07:13:41.9805123Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T07:13:41.9813207Z 2024-06-26T07:13:42.2533320Z + cd test 2024-06-26T07:13:42.2534190Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T07:13:43.4668606Z ATen/Parallel: 2024-06-26T07:13:43.4669156Z at::get_num_threads() : 8 2024-06-26T07:13:43.4669688Z at::get_num_interop_threads() : 16 2024-06-26T07:13:43.4670270Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:13:43.4670812Z omp_get_max_threads() : 8 2024-06-26T07:13:43.4672000Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:13:43.4672893Z mkl_get_max_threads() : 8 2024-06-26T07:13:43.4673534Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:13:43.4674216Z std::thread::hardware_concurrency() : 16 2024-06-26T07:13:43.4674673Z Environment variables: 2024-06-26T07:13:43.4675042Z OMP_NUM_THREADS : [not set] 2024-06-26T07:13:43.4675761Z MKL_NUM_THREADS : [not set] 2024-06-26T07:13:43.4676153Z ATen parallel backend: OpenMP 2024-06-26T07:13:43.4676422Z 2024-06-26T07:13:43.7069997Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:13:43.7070675Z + [[ dynamo_eager_torchbench == *backward* ]] 2024-06-26T07:13:43.7071233Z + [[ dynamo_eager_torchbench == *xla* ]] 2024-06-26T07:13:43.7071787Z + [[ dynamo_eager_torchbench == *executorch* ]] 2024-06-26T07:13:43.7072522Z + [[ dynamo_eager_torchbench == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T07:13:43.7073305Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:13:43.7074067Z + [[ dynamo_eager_torchbench == distributed ]] 2024-06-26T07:13:43.7074808Z + [[ dynamo_eager_torchbench == *inductor_distributed* ]] 2024-06-26T07:13:43.7075726Z + [[ dynamo_eager_torchbench == *inductor-halide* ]] 2024-06-26T07:13:43.7076644Z + [[ dynamo_eager_torchbench == *inductor-micro-benchmark* ]] 2024-06-26T07:13:43.7077470Z + [[ dynamo_eager_torchbench == *huggingface* ]] 2024-06-26T07:13:43.7078180Z + [[ dynamo_eager_torchbench == *timm* ]] 2024-06-26T07:13:43.7078841Z + [[ dynamo_eager_torchbench == *torchbench* ]] 2024-06-26T07:13:43.7079599Z + [[ dynamo_eager_torchbench == *cpu_inductor* ]] 2024-06-26T07:13:43.7080284Z + install_torchaudio cuda 2024-06-26T07:13:43.7080793Z + local commit 2024-06-26T07:13:43.7081242Z ++ get_pinned_commit audio 2024-06-26T07:13:43.7081642Z ++ cat .github/ci_commit_pins/audio.txt 2024-06-26T07:13:43.7085614Z + commit=b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:43.7086132Z + [[ cuda == \c\u\d\a ]] 2024-06-26T07:13:43.7086569Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-06-26T07:13:43.7087548Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:43.7089127Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:44.0208555Z Collecting git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:44.0211499Z Cloning https://github.com/pytorch/audio.git (to revision b829e936f7cc61b48149f5f957a451a38bf2a178) to /tmp/pip-req-build-fj4kdiqy 2024-06-26T07:13:44.0229210Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-fj4kdiqy 2024-06-26T07:13:44.7459318Z Running command git rev-parse -q --verify 'sha^b829e936f7cc61b48149f5f957a451a38bf2a178' 2024-06-26T07:13:44.7474481Z Running command git fetch -q https://github.com/pytorch/audio.git b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:45.1240601Z Running command git checkout -q b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:45.2387743Z Resolved https://github.com/pytorch/audio.git to commit b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:45.2388878Z Running command git submodule update --init --recursive -q 2024-06-26T07:13:47.3958085Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:13:47.3992975Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.4.0a0+b829e93) (2.5.0a0+gitb8c4c54) 2024-06-26T07:13:47.4057699Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (3.13.1) 2024-06-26T07:13:47.4063088Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (4.12.2) 2024-06-26T07:13:47.4065820Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (1.12.1) 2024-06-26T07:13:47.4068848Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (2.8.8) 2024-06-26T07:13:47.4071840Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (3.1.4) 2024-06-26T07:13:47.4074967Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (2024.2.0) 2024-06-26T07:13:47.4495490Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.4.0a0+b829e93) (2.1.5) 2024-06-26T07:13:47.4645909Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchaudio==2.4.0a0+b829e93) (1.3.0) 2024-06-26T07:13:47.4715918Z Building wheels for collected packages: torchaudio 2024-06-26T07:16:13.8802338Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-06-26T07:16:13.8830955Z [?25h Created wheel for torchaudio: filename=torchaudio-2.4.0a0+b829e93-cp310-cp310-linux_x86_64.whl size=2308817 sha256=b5587ea75589391354105f56c6bb0d7c439644fc7f5a61f4853f1513e8103ee7 2024-06-26T07:16:13.8832782Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/30/b3/9c/882f183f00af2e582825654a92a04400924519922db53ff522 2024-06-26T07:16:13.8858615Z Successfully built torchaudio 2024-06-26T07:16:15.0856577Z Installing collected packages: torchaudio 2024-06-26T07:16:15.2858177Z Successfully installed torchaudio-2.4.0a0+b829e93 2024-06-26T07:16:15.4237525Z + install_torchtext 2024-06-26T07:16:15.4238140Z + local data_commit 2024-06-26T07:16:15.4238731Z + local text_commit 2024-06-26T07:16:15.4242784Z ++ get_pinned_commit data 2024-06-26T07:16:15.4243324Z ++ cat .github/ci_commit_pins/data.txt 2024-06-26T07:16:15.4252802Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:16:15.4254869Z ++ get_pinned_commit text 2024-06-26T07:16:15.4255376Z ++ cat .github/ci_commit_pins/text.txt 2024-06-26T07:16:15.4265657Z + text_commit=b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:16:15.4266957Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:16:15.4268781Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:16:15.7373513Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:16:15.7377080Z Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-q3p3u1k0 2024-06-26T07:16:15.7391357Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-q3p3u1k0 2024-06-26T07:16:16.1847452Z Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f' 2024-06-26T07:16:16.1865835Z Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:16:16.4061833Z Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:16:16.5606697Z Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:16:16.5607856Z Running command git submodule update --init --recursive -q 2024-06-26T07:17:28.0087367Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:17:28.0137062Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.19) 2024-06-26T07:17:28.0140609Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.32.3) 2024-06-26T07:17:28.0145333Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.5.0a0+gitb8c4c54) 2024-06-26T07:17:28.0205703Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1) 2024-06-26T07:17:28.0211628Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.12.2) 2024-06-26T07:17:28.0215432Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.12.1) 2024-06-26T07:17:28.0217313Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8) 2024-06-26T07:17:28.0219677Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4) 2024-06-26T07:17:28.0222907Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.2.0) 2024-06-26T07:17:28.0404027Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2) 2024-06-26T07:17:28.0410027Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7) 2024-06-26T07:17:28.0415378Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.6.2) 2024-06-26T07:17:28.0861189Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5) 2024-06-26T07:17:28.1018978Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0) 2024-06-26T07:17:28.1098344Z Building wheels for collected packages: torchdata 2024-06-26T07:17:29.8367312Z Building wheel for torchdata (setup.py) ... [?25l- \ | done 2024-06-26T07:17:29.8372813Z [?25h Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182956 sha256=cf29047f27850dad9c5aded3fcc2a2262662f8eb9b8328ecb6b03ee8877a9920 2024-06-26T07:17:29.8374815Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce 2024-06-26T07:17:29.8399591Z Successfully built torchdata 2024-06-26T07:17:31.0327304Z Installing collected packages: torchdata 2024-06-26T07:17:31.1210089Z Successfully installed torchdata-0.7.0a0+11bb5b8 2024-06-26T07:17:33.6094297Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:17:33.6095956Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:17:33.9217288Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:17:33.9221069Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-z0pema1l 2024-06-26T07:17:33.9238663Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-z0pema1l 2024-06-26T07:17:34.7119891Z Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db' 2024-06-26T07:17:34.7139513Z Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:17:35.1635421Z Running command git checkout -q b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:17:35.3187112Z Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:17:35.3188506Z Running command git submodule update --init --recursive -q 2024-06-26T07:17:42.3754088Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:17:42.3803961Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.4) 2024-06-26T07:17:42.3806958Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.32.3) 2024-06-26T07:17:42.3809228Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.5.0a0+gitb8c4c54) 2024-06-26T07:17:42.3812575Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.21.2) 2024-06-26T07:17:42.3815738Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8) 2024-06-26T07:17:42.3874505Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2) 2024-06-26T07:17:42.3879744Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7) 2024-06-26T07:17:42.3885799Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.19) 2024-06-26T07:17:42.3890451Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.6.2) 2024-06-26T07:17:42.3945924Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1) 2024-06-26T07:17:42.3952163Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.12.2) 2024-06-26T07:17:42.3954624Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.12.1) 2024-06-26T07:17:42.3957524Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8) 2024-06-26T07:17:42.3960244Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4) 2024-06-26T07:17:42.3962698Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.2.0) 2024-06-26T07:17:42.4640543Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5) 2024-06-26T07:17:42.4797866Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0) 2024-06-26T07:17:42.4876737Z Building wheels for collected packages: torchtext 2024-06-26T07:18:18.7601197Z Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-06-26T07:18:18.7628595Z [?25h Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2045297 sha256=899d16dcc110582705cbb733fc7aebd73c8ed85640e6ecccafe1c3965f769244 2024-06-26T07:18:18.7631595Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf 2024-06-26T07:18:18.7655079Z Successfully built torchtext 2024-06-26T07:18:19.9441634Z Installing collected packages: torchtext 2024-06-26T07:18:20.0425651Z Successfully installed torchtext-0.16.0a0+b0ebddc 2024-06-26T07:18:20.1653677Z + install_torchvision 2024-06-26T07:18:20.1654170Z + local orig_preload 2024-06-26T07:18:20.1654622Z + local commit 2024-06-26T07:18:20.1656327Z ++ get_pinned_commit vision 2024-06-26T07:18:20.1656908Z ++ cat .github/ci_commit_pins/vision.txt 2024-06-26T07:18:20.1665218Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:18:20.1665870Z + orig_preload= 2024-06-26T07:18:20.1666320Z + '[' -n '' ']' 2024-06-26T07:18:20.1667232Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:18:20.1668843Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:18:20.4789516Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:18:20.4793320Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-m57isgrh 2024-06-26T07:18:20.4809881Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-m57isgrh 2024-06-26T07:18:21.9150153Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-06-26T07:18:21.9168733Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:18:23.1363349Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:18:23.4039760Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:18:25.7337251Z Preparing metadata (setup.py) ... [?25l- \ done 2024-06-26T07:18:25.7387553Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-06-26T07:18:25.7389852Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+gitb8c4c54) 2024-06-26T07:18:25.7395437Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-06-26T07:18:25.7596754Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-06-26T07:18:25.7603218Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-06-26T07:18:25.7605836Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1) 2024-06-26T07:18:25.7608182Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-06-26T07:18:25.7610597Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-06-26T07:18:25.7613854Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-06-26T07:18:25.8039288Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-06-26T07:18:25.8192960Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-06-26T07:18:25.8266932Z Building wheels for collected packages: torchvision 2024-06-26T07:19:38.7069427Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done 2024-06-26T07:19:38.7097123Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2112990 sha256=ef9d487a4a8ff3cd221cc53872f70abe5786b74fc70190d381b7f673529ca5e0 2024-06-26T07:19:38.7099431Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-06-26T07:19:38.7134382Z Successfully built torchvision 2024-06-26T07:19:39.8990318Z Installing collected packages: torchvision 2024-06-26T07:19:40.2782885Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-06-26T07:19:40.3876764Z + '[' -n '' ']' 2024-06-26T07:19:40.3877122Z + id=0 2024-06-26T07:19:40.3877487Z + pip_install opencv-python==4.8.0.74 2024-06-26T07:19:40.3881285Z + pip install --progress-bar off opencv-python==4.8.0.74 2024-06-26T07:19:40.8150883Z Collecting opencv-python==4.8.0.74 2024-06-26T07:19:40.8304862Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2024-06-26T07:19:40.8474830Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.21.2) 2024-06-26T07:19:40.8514697Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2024-06-26T07:19:42.5457627Z Installing collected packages: opencv-python 2024-06-26T07:19:43.3616966Z Successfully installed opencv-python-4.8.0.74 2024-06-26T07:19:43.4596160Z + [[ dynamo_eager_torchbench == *inductor_torchbench_smoketest_perf* ]] 2024-06-26T07:19:43.4597269Z + [[ dynamo_eager_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]] 2024-06-26T07:19:43.4598272Z + [[ dynamo_eager_torchbench == *torchbench_gcp_smoketest* ]] 2024-06-26T07:19:43.4598930Z + checkout_install_torchbench 2024-06-26T07:19:43.4599317Z + local commit 2024-06-26T07:19:43.4599666Z ++ get_pinned_commit torchbench 2024-06-26T07:19:43.4600122Z ++ cat .github/ci_commit_pins/torchbench.txt 2024-06-26T07:19:43.4606875Z + commit=23512dbebd44a11eb84afbf53c3c071dd105297e 2024-06-26T07:19:43.4607572Z + git clone https://github.com/pytorch/benchmark torchbench 2024-06-26T07:19:43.4616542Z Cloning into 'torchbench'... 2024-06-26T07:19:43.5986217Z remote: Enumerating objects: 30187, done. 2024-06-26T07:19:43.5987009Z remote: Counting objects: 0% (1/546) 2024-06-26T07:19:43.5988068Z remote: Counting objects: 1% (6/546) 2024-06-26T07:19:43.5988808Z remote: Counting objects: 2% (11/546) 2024-06-26T07:19:43.5989517Z remote: Counting objects: 3% (17/546) 2024-06-26T07:19:43.5990130Z remote: Counting objects: 4% (22/546) 2024-06-26T07:19:43.5990709Z remote: Counting objects: 5% (28/546) 2024-06-26T07:19:43.5991445Z remote: Counting objects: 6% (33/546) 2024-06-26T07:19:43.5992022Z remote: Counting objects: 7% (39/546) 2024-06-26T07:19:43.5992564Z remote: Counting objects: 8% (44/546) 2024-06-26T07:19:43.5993159Z remote: Counting objects: 9% (50/546) 2024-06-26T07:19:43.5993691Z remote: Counting objects: 10% (55/546) 2024-06-26T07:19:43.5994302Z remote: Counting objects: 11% (61/546) 2024-06-26T07:19:43.5995075Z remote: Counting objects: 12% (66/546) 2024-06-26T07:19:43.5995640Z remote: Counting objects: 13% (71/546) 2024-06-26T07:19:43.5996364Z remote: Counting objects: 14% (77/546) 2024-06-26T07:19:43.5997024Z remote: Counting objects: 15% (82/546) 2024-06-26T07:19:43.5997694Z remote: Counting objects: 16% (88/546) 2024-06-26T07:19:43.5998400Z remote: Counting objects: 17% (93/546) 2024-06-26T07:19:43.5999100Z remote: Counting objects: 18% (99/546) 2024-06-26T07:19:43.5999660Z remote: Counting objects: 19% (104/546) 2024-06-26T07:19:43.6000204Z remote: Counting objects: 20% (110/546) 2024-06-26T07:19:43.6000781Z remote: Counting objects: 21% (115/546) 2024-06-26T07:19:43.6001523Z remote: Counting objects: 22% (121/546) 2024-06-26T07:19:43.6002450Z remote: Counting objects: 23% (126/546) 2024-06-26T07:19:43.6003116Z remote: Counting objects: 24% (132/546) 2024-06-26T07:19:43.6003657Z remote: Counting objects: 25% (137/546) 2024-06-26T07:19:43.6004186Z remote: Counting objects: 26% (142/546) 2024-06-26T07:19:43.6004725Z remote: Counting objects: 27% (148/546) 2024-06-26T07:19:43.6005386Z remote: Counting objects: 28% (153/546) 2024-06-26T07:19:43.6006093Z remote: Counting objects: 29% (159/546) 2024-06-26T07:19:43.6006813Z remote: Counting objects: 30% (164/546) 2024-06-26T07:19:43.6007539Z remote: Counting objects: 31% (170/546) 2024-06-26T07:19:43.6008180Z remote: Counting objects: 32% (175/546) 2024-06-26T07:19:43.6008714Z remote: Counting objects: 33% (181/546) 2024-06-26T07:19:43.6009249Z remote: Counting objects: 34% (186/546) 2024-06-26T07:19:43.6009786Z remote: Counting objects: 35% (192/546) 2024-06-26T07:19:43.6010323Z remote: Counting objects: 36% (197/546) 2024-06-26T07:19:43.6010859Z remote: Counting objects: 37% (203/546) 2024-06-26T07:19:43.6011393Z remote: Counting objects: 38% (208/546) 2024-06-26T07:19:43.6012291Z remote: Counting objects: 39% (213/546) 2024-06-26T07:19:43.6012845Z remote: Counting objects: 40% (219/546) 2024-06-26T07:19:43.6013461Z remote: Counting objects: 41% (224/546) 2024-06-26T07:19:43.6014163Z remote: Counting objects: 42% (230/546) 2024-06-26T07:19:43.6014719Z remote: Counting objects: 43% (235/546) 2024-06-26T07:19:43.6015259Z remote: Counting objects: 44% (241/546) 2024-06-26T07:19:43.6015794Z remote: Counting objects: 45% (246/546) 2024-06-26T07:19:43.6016338Z remote: Counting objects: 46% (252/546) 2024-06-26T07:19:43.6016883Z remote: Counting objects: 47% (257/546) 2024-06-26T07:19:43.6017416Z remote: Counting objects: 48% (263/546) 2024-06-26T07:19:43.6017952Z remote: Counting objects: 49% (268/546) 2024-06-26T07:19:43.6018494Z remote: Counting objects: 50% (273/546) 2024-06-26T07:19:43.6019037Z remote: Counting objects: 51% (279/546) 2024-06-26T07:19:43.6019576Z remote: Counting objects: 52% (284/546) 2024-06-26T07:19:43.6020122Z remote: Counting objects: 53% (290/546) 2024-06-26T07:19:43.6020654Z remote: Counting objects: 54% (295/546) 2024-06-26T07:19:43.6021196Z remote: Counting objects: 55% (301/546) 2024-06-26T07:19:43.6021747Z remote: Counting objects: 56% (306/546) 2024-06-26T07:19:43.6022463Z remote: Counting objects: 57% (312/546) 2024-06-26T07:19:43.6023011Z remote: Counting objects: 58% (317/546) 2024-06-26T07:19:43.6023659Z remote: Counting objects: 59% (323/546) 2024-06-26T07:19:43.6024198Z remote: Counting objects: 60% (328/546) 2024-06-26T07:19:43.6024742Z remote: Counting objects: 61% (334/546) 2024-06-26T07:19:43.6025282Z remote: Counting objects: 62% (339/546) 2024-06-26T07:19:43.6025826Z remote: Counting objects: 63% (344/546) 2024-06-26T07:19:43.6026363Z remote: Counting objects: 64% (350/546) 2024-06-26T07:19:43.6026916Z remote: Counting objects: 65% (355/546) 2024-06-26T07:19:43.6027454Z remote: Counting objects: 66% (361/546) 2024-06-26T07:19:43.6027983Z remote: Counting objects: 67% (366/546) 2024-06-26T07:19:43.6028525Z remote: Counting objects: 68% (372/546) 2024-06-26T07:19:43.6029069Z remote: Counting objects: 69% (377/546) 2024-06-26T07:19:43.6029610Z remote: Counting objects: 70% (383/546) 2024-06-26T07:19:43.6030151Z remote: Counting objects: 71% (388/546) 2024-06-26T07:19:43.6030688Z remote: Counting objects: 72% (394/546) 2024-06-26T07:19:43.6031234Z remote: Counting objects: 73% (399/546) 2024-06-26T07:19:43.6031773Z remote: Counting objects: 74% (405/546) 2024-06-26T07:19:43.6032328Z remote: Counting objects: 75% (410/546) 2024-06-26T07:19:43.6032864Z remote: Counting objects: 76% (415/546) 2024-06-26T07:19:43.6033403Z remote: Counting objects: 77% (421/546) 2024-06-26T07:19:43.6033948Z remote: Counting objects: 78% (426/546) 2024-06-26T07:19:43.6034577Z remote: Counting objects: 79% (432/546) 2024-06-26T07:19:43.6035121Z remote: Counting objects: 80% (437/546) 2024-06-26T07:19:43.6035661Z remote: Counting objects: 81% (443/546) 2024-06-26T07:19:43.6036199Z remote: Counting objects: 82% (448/546) 2024-06-26T07:19:43.6036728Z remote: Counting objects: 83% (454/546) 2024-06-26T07:19:43.6037271Z remote: Counting objects: 84% (459/546) 2024-06-26T07:19:43.6037824Z remote: Counting objects: 85% (465/546) 2024-06-26T07:19:43.6038381Z remote: Counting objects: 86% (470/546) 2024-06-26T07:19:43.6038908Z remote: Counting objects: 87% (476/546) 2024-06-26T07:19:43.6039470Z remote: Counting objects: 88% (481/546) 2024-06-26T07:19:43.6040027Z remote: Counting objects: 89% (486/546) 2024-06-26T07:19:43.6040578Z remote: Counting objects: 90% (492/546) 2024-06-26T07:19:43.6041111Z remote: Counting objects: 91% (497/546) 2024-06-26T07:19:43.6041658Z remote: Counting objects: 92% (503/546) 2024-06-26T07:19:43.6042209Z remote: Counting objects: 93% (508/546) 2024-06-26T07:19:43.6042754Z remote: Counting objects: 94% (514/546) 2024-06-26T07:19:43.6043288Z remote: Counting objects: 95% (519/546) 2024-06-26T07:19:43.6043827Z remote: Counting objects: 96% (525/546) 2024-06-26T07:19:43.6044529Z remote: Counting objects: 97% (530/546) 2024-06-26T07:19:43.6045154Z remote: Counting objects: 98% (536/546) 2024-06-26T07:19:43.6045750Z remote: Counting objects: 99% (541/546) 2024-06-26T07:19:43.6046467Z remote: Counting objects: 100% (546/546) 2024-06-26T07:19:43.6047069Z remote: Counting objects: 100% (546/546), done. 2024-06-26T07:19:43.6047674Z remote: Compressing objects: 0% (1/319) 2024-06-26T07:19:43.6048269Z remote: Compressing objects: 1% (4/319) 2024-06-26T07:19:43.6048862Z remote: Compressing objects: 2% (7/319) 2024-06-26T07:19:43.6049451Z remote: Compressing objects: 3% (10/319) 2024-06-26T07:19:43.6050027Z remote: Compressing objects: 4% (13/319) 2024-06-26T07:19:43.6050614Z remote: Compressing objects: 5% (16/319) 2024-06-26T07:19:43.6051189Z remote: Compressing objects: 6% (20/319) 2024-06-26T07:19:43.6051773Z remote: Compressing objects: 7% (23/319) 2024-06-26T07:19:43.6052533Z remote: Compressing objects: 8% (26/319) 2024-06-26T07:19:43.6053081Z remote: Compressing objects: 9% (29/319) 2024-06-26T07:19:43.6054124Z remote: Compressing objects: 10% (32/319) 2024-06-26T07:19:43.6066365Z remote: Compressing objects: 11% (36/319) 2024-06-26T07:19:43.6067124Z remote: Compressing objects: 12% (39/319) 2024-06-26T07:19:43.6067707Z remote: Compressing objects: 13% (42/319) 2024-06-26T07:19:43.6074327Z remote: Compressing objects: 14% (45/319) 2024-06-26T07:19:43.6075108Z remote: Compressing objects: 15% (48/319) 2024-06-26T07:19:43.6075836Z remote: Compressing objects: 16% (52/319) 2024-06-26T07:19:43.6076572Z remote: Compressing objects: 17% (55/319) 2024-06-26T07:19:43.6077279Z remote: Compressing objects: 18% (58/319) 2024-06-26T07:19:43.6077969Z remote: Compressing objects: 19% (61/319) 2024-06-26T07:19:43.6078537Z remote: Compressing objects: 20% (64/319) 2024-06-26T07:19:43.6079091Z remote: Compressing objects: 21% (67/319) 2024-06-26T07:19:43.6079651Z remote: Compressing objects: 22% (71/319) 2024-06-26T07:19:43.6080268Z remote: Compressing objects: 23% (74/319) 2024-06-26T07:19:43.6080895Z remote: Compressing objects: 24% (77/319) 2024-06-26T07:19:43.6081594Z remote: Compressing objects: 25% (80/319) 2024-06-26T07:19:43.6082382Z remote: Compressing objects: 26% (83/319) 2024-06-26T07:19:43.6082997Z remote: Compressing objects: 27% (87/319) 2024-06-26T07:19:43.6083551Z remote: Compressing objects: 28% (90/319) 2024-06-26T07:19:43.6084107Z remote: Compressing objects: 29% (93/319) 2024-06-26T07:19:43.6084780Z remote: Compressing objects: 30% (96/319) 2024-06-26T07:19:43.6085455Z remote: Compressing objects: 31% (99/319) 2024-06-26T07:19:43.6086153Z remote: Compressing objects: 32% (103/319) 2024-06-26T07:19:43.6086731Z remote: Compressing objects: 33% (106/319) 2024-06-26T07:19:43.6087451Z remote: Compressing objects: 34% (109/319) 2024-06-26T07:19:43.6088088Z remote: Compressing objects: 35% (112/319) 2024-06-26T07:19:43.6088777Z remote: Compressing objects: 36% (115/319) 2024-06-26T07:19:43.6089429Z remote: Compressing objects: 37% (119/319) 2024-06-26T07:19:43.6090005Z remote: Compressing objects: 38% (122/319) 2024-06-26T07:19:43.6090578Z remote: Compressing objects: 39% (125/319) 2024-06-26T07:19:43.6091138Z remote: Compressing objects: 40% (128/319) 2024-06-26T07:19:43.6092118Z remote: Compressing objects: 41% (131/319) 2024-06-26T07:19:43.6092853Z remote: Compressing objects: 42% (134/319) 2024-06-26T07:19:43.6093571Z remote: Compressing objects: 43% (138/319) 2024-06-26T07:19:43.6094133Z remote: Compressing objects: 44% (141/319) 2024-06-26T07:19:43.6094743Z remote: Compressing objects: 45% (144/319) 2024-06-26T07:19:43.6095315Z remote: Compressing objects: 46% (147/319) 2024-06-26T07:19:43.6095889Z remote: Compressing objects: 47% (150/319) 2024-06-26T07:19:43.6096446Z remote: Compressing objects: 48% (154/319) 2024-06-26T07:19:43.6097024Z remote: Compressing objects: 49% (157/319) 2024-06-26T07:19:43.6097598Z remote: Compressing objects: 50% (160/319) 2024-06-26T07:19:43.6098179Z remote: Compressing objects: 51% (163/319) 2024-06-26T07:19:43.6098752Z remote: Compressing objects: 52% (166/319) 2024-06-26T07:19:43.6099331Z remote: Compressing objects: 53% (170/319) 2024-06-26T07:19:43.6099893Z remote: Compressing objects: 54% (173/319) 2024-06-26T07:19:43.6100455Z remote: Compressing objects: 55% (176/319) 2024-06-26T07:19:43.6101012Z remote: Compressing objects: 56% (179/319) 2024-06-26T07:19:43.6101586Z remote: Compressing objects: 57% (182/319) 2024-06-26T07:19:43.6102158Z remote: Compressing objects: 58% (186/319) 2024-06-26T07:19:43.6102744Z remote: Compressing objects: 59% (189/319) 2024-06-26T07:19:43.6103389Z remote: Compressing objects: 60% (192/319) 2024-06-26T07:19:43.6103964Z remote: Compressing objects: 61% (195/319) 2024-06-26T07:19:43.6104533Z remote: Compressing objects: 62% (198/319) 2024-06-26T07:19:43.6105090Z remote: Compressing objects: 63% (201/319) 2024-06-26T07:19:43.6105667Z remote: Compressing objects: 64% (205/319) 2024-06-26T07:19:43.6106370Z remote: Compressing objects: 65% (208/319) 2024-06-26T07:19:43.6106955Z remote: Compressing objects: 66% (211/319) 2024-06-26T07:19:43.6107509Z remote: Compressing objects: 67% (214/319) 2024-06-26T07:19:43.6108079Z remote: Compressing objects: 68% (217/319) 2024-06-26T07:19:43.6108643Z remote: Compressing objects: 69% (221/319) 2024-06-26T07:19:43.6109219Z remote: Compressing objects: 70% (224/319) 2024-06-26T07:19:43.6109790Z remote: Compressing objects: 71% (227/319) 2024-06-26T07:19:43.6110365Z remote: Compressing objects: 72% (230/319) 2024-06-26T07:19:43.6110928Z remote: Compressing objects: 73% (233/319) 2024-06-26T07:19:43.6111501Z remote: Compressing objects: 74% (237/319) 2024-06-26T07:19:43.6112112Z remote: Compressing objects: 75% (240/319) 2024-06-26T07:19:43.6112674Z remote: Compressing objects: 76% (243/319) 2024-06-26T07:19:43.6113239Z remote: Compressing objects: 77% (246/319) 2024-06-26T07:19:43.6113803Z remote: Compressing objects: 78% (249/319) 2024-06-26T07:19:43.6114371Z remote: Compressing objects: 79% (253/319) 2024-06-26T07:19:43.6114954Z remote: Compressing objects: 80% (256/319) 2024-06-26T07:19:43.6115524Z remote: Compressing objects: 81% (259/319) 2024-06-26T07:19:43.6116097Z remote: Compressing objects: 82% (262/319) 2024-06-26T07:19:43.6116656Z remote: Compressing objects: 83% (265/319) 2024-06-26T07:19:43.6117229Z remote: Compressing objects: 84% (268/319) 2024-06-26T07:19:43.6117800Z remote: Compressing objects: 85% (272/319) 2024-06-26T07:19:43.6118457Z remote: Compressing objects: 86% (275/319) 2024-06-26T07:19:43.6119023Z remote: Compressing objects: 87% (278/319) 2024-06-26T07:19:43.6119590Z remote: Compressing objects: 88% (281/319) 2024-06-26T07:19:43.6120161Z remote: Compressing objects: 89% (284/319) 2024-06-26T07:19:43.6120725Z remote: Compressing objects: 90% (288/319) 2024-06-26T07:19:43.6121296Z remote: Compressing objects: 91% (291/319) 2024-06-26T07:19:43.6121872Z remote: Compressing objects: 92% (294/319) 2024-06-26T07:19:43.6122443Z remote: Compressing objects: 93% (297/319) 2024-06-26T07:19:43.6123008Z remote: Compressing objects: 94% (300/319) 2024-06-26T07:19:43.6123579Z remote: Compressing objects: 95% (304/319) 2024-06-26T07:19:43.6124153Z remote: Compressing objects: 96% (307/319) 2024-06-26T07:19:43.6124724Z remote: Compressing objects: 97% (310/319) 2024-06-26T07:19:43.6125285Z remote: Compressing objects: 98% (313/319) 2024-06-26T07:19:43.6125862Z remote: Compressing objects: 99% (316/319) 2024-06-26T07:19:43.6126433Z remote: Compressing objects: 100% (319/319) 2024-06-26T07:19:43.6127045Z remote: Compressing objects: 100% (319/319), done. 2024-06-26T07:19:43.6178091Z Receiving objects: 0% (1/30187) 2024-06-26T07:19:43.6232896Z Receiving objects: 1% (302/30187) 2024-06-26T07:19:43.6279491Z Receiving objects: 2% (604/30187) 2024-06-26T07:19:43.6319315Z Receiving objects: 3% (906/30187) 2024-06-26T07:19:43.6351368Z Receiving objects: 4% (1208/30187) 2024-06-26T07:19:43.6382406Z Receiving objects: 5% (1510/30187) 2024-06-26T07:19:43.6411902Z Receiving objects: 6% (1812/30187) 2024-06-26T07:19:43.6449112Z Receiving objects: 7% (2114/30187) 2024-06-26T07:19:43.6479778Z Receiving objects: 8% (2415/30187) 2024-06-26T07:19:43.6510210Z Receiving objects: 9% (2717/30187) 2024-06-26T07:19:43.6546486Z Receiving objects: 10% (3019/30187) 2024-06-26T07:19:43.6607651Z Receiving objects: 11% (3321/30187) 2024-06-26T07:19:43.6671086Z Receiving objects: 12% (3623/30187) 2024-06-26T07:19:43.6727921Z Receiving objects: 13% (3925/30187) 2024-06-26T07:19:43.6897020Z Receiving objects: 14% (4227/30187) 2024-06-26T07:19:43.6998783Z Receiving objects: 15% (4529/30187) 2024-06-26T07:19:43.7090104Z Receiving objects: 16% (4830/30187) 2024-06-26T07:19:43.7140731Z Receiving objects: 17% (5132/30187) 2024-06-26T07:19:43.7181642Z Receiving objects: 18% (5434/30187) 2024-06-26T07:19:43.7206980Z Receiving objects: 19% (5736/30187) 2024-06-26T07:19:43.7269770Z Receiving objects: 20% (6038/30187) 2024-06-26T07:19:43.7289140Z Receiving objects: 21% (6340/30187) 2024-06-26T07:19:43.7309721Z Receiving objects: 22% (6642/30187) 2024-06-26T07:19:43.8361754Z Receiving objects: 23% (6944/30187) 2024-06-26T07:19:44.6135096Z Receiving objects: 24% (7245/30187) 2024-06-26T07:19:44.7071140Z Receiving objects: 24% (7435/30187), 81.09 MiB | 81.25 MiB/s 2024-06-26T07:19:45.1908785Z Receiving objects: 25% (7547/30187), 81.09 MiB | 81.25 MiB/s 2024-06-26T07:19:45.2572940Z Receiving objects: 26% (7849/30187), 119.54 MiB | 79.79 MiB/s 2024-06-26T07:19:45.3237536Z Receiving objects: 27% (8151/30187), 119.54 MiB | 79.79 MiB/s 2024-06-26T07:19:45.3894430Z Receiving objects: 28% (8453/30187), 119.54 MiB | 79.79 MiB/s 2024-06-26T07:19:45.4552249Z Receiving objects: 29% (8755/30187), 119.54 MiB | 79.79 MiB/s 2024-06-26T07:19:45.5209204Z Receiving objects: 30% (9057/30187), 119.54 MiB | 79.79 MiB/s 2024-06-26T07:19:45.5866905Z Receiving objects: 31% (9358/30187), 119.54 MiB | 79.79 MiB/s 2024-06-26T07:19:45.6134861Z Receiving objects: 32% (9660/30187), 119.54 MiB | 79.79 MiB/s 2024-06-26T07:19:45.6529404Z Receiving objects: 32% (9781/30187), 161.91 MiB | 81.03 MiB/s 2024-06-26T07:19:45.8740473Z Receiving objects: 33% (9962/30187), 161.91 MiB | 81.03 MiB/s 2024-06-26T07:19:45.9462311Z Receiving objects: 34% (10264/30187), 161.91 MiB | 81.03 MiB/s 2024-06-26T07:19:45.9583797Z Receiving objects: 35% (10566/30187), 161.91 MiB | 81.03 MiB/s 2024-06-26T07:19:46.1952217Z Receiving objects: 36% (10868/30187), 161.91 MiB | 81.03 MiB/s 2024-06-26T07:19:46.4862714Z Receiving objects: 37% (11170/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5087917Z Receiving objects: 38% (11472/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5102865Z Receiving objects: 39% (11773/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5130138Z Receiving objects: 40% (12075/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5144681Z Receiving objects: 41% (12377/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5158190Z Receiving objects: 42% (12679/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5169081Z Receiving objects: 43% (12981/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5178810Z Receiving objects: 44% (13283/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5198136Z Receiving objects: 45% (13585/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5209596Z Receiving objects: 46% (13887/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5214284Z Receiving objects: 47% (14188/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5220513Z Receiving objects: 48% (14490/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5241652Z Receiving objects: 49% (14792/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5258210Z Receiving objects: 50% (15094/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5268299Z Receiving objects: 51% (15396/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5462122Z Receiving objects: 52% (15698/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5467226Z Receiving objects: 53% (16000/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5474200Z Receiving objects: 54% (16301/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5480523Z Receiving objects: 55% (16603/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5485640Z Receiving objects: 56% (16905/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5502486Z Receiving objects: 57% (17207/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5512628Z Receiving objects: 58% (17509/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5521506Z Receiving objects: 59% (17811/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5534731Z Receiving objects: 60% (18113/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5543530Z Receiving objects: 61% (18415/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5553745Z Receiving objects: 62% (18716/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5680705Z Receiving objects: 63% (19018/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5686735Z Receiving objects: 64% (19320/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5693395Z Receiving objects: 65% (19622/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5699593Z Receiving objects: 66% (19924/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5708770Z Receiving objects: 67% (20226/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.5717030Z Receiving objects: 68% (20528/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.6005287Z Receiving objects: 69% (20830/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.6135166Z Receiving objects: 70% (21131/30187), 192.56 MiB | 77.11 MiB/s 2024-06-26T07:19:46.6437024Z Receiving objects: 70% (21376/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.6437761Z Receiving objects: 71% (21433/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.6445468Z Receiving objects: 72% (21735/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.6457870Z Receiving objects: 73% (22037/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.6471782Z Receiving objects: 74% (22339/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.6483049Z Receiving objects: 75% (22641/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.6492827Z Receiving objects: 76% (22943/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.6512403Z Receiving objects: 77% (23244/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.6519484Z Receiving objects: 78% (23546/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.6528260Z Receiving objects: 79% (23848/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.6551992Z Receiving objects: 80% (24150/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.8618444Z Receiving objects: 81% (24452/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.8828242Z Receiving objects: 82% (24754/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.8860853Z Receiving objects: 83% (25056/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.8870581Z Receiving objects: 84% (25358/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.8880947Z Receiving objects: 85% (25659/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.8911038Z Receiving objects: 86% (25961/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.8939469Z Receiving objects: 87% (26263/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.9069928Z Receiving objects: 88% (26565/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.9075950Z Receiving objects: 89% (26867/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.9082784Z Receiving objects: 90% (27169/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.9087993Z Receiving objects: 91% (27471/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.9094554Z Receiving objects: 92% (27773/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.9100327Z Receiving objects: 93% (28074/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.9210343Z Receiving objects: 94% (28376/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.9225555Z Receiving objects: 95% (28678/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.9235283Z Receiving objects: 96% (28980/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.9253154Z Receiving objects: 97% (29282/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.9267585Z Receiving objects: 98% (29584/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.9307475Z Receiving objects: 99% (29886/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.9308446Z remote: Total 30187 (delta 255), reused 486 (delta 215), pack-reused 29641 2024-06-26T07:19:46.9324667Z Receiving objects: 100% (30187/30187), 227.45 MiB | 75.89 MiB/s 2024-06-26T07:19:46.9325409Z Receiving objects: 100% (30187/30187), 245.25 MiB | 73.91 MiB/s, done. 2024-06-26T07:19:46.9353929Z Resolving deltas: 0% (0/16204) 2024-06-26T07:19:46.9362252Z Resolving deltas: 1% (164/16204) 2024-06-26T07:19:46.9377628Z Resolving deltas: 2% (327/16204) 2024-06-26T07:19:46.9391987Z Resolving deltas: 3% (515/16204) 2024-06-26T07:19:46.9412321Z Resolving deltas: 4% (649/16204) 2024-06-26T07:19:46.9424028Z Resolving deltas: 5% (821/16204) 2024-06-26T07:19:46.9431030Z Resolving deltas: 6% (975/16204) 2024-06-26T07:19:46.9441839Z Resolving deltas: 7% (1138/16204) 2024-06-26T07:19:46.9451276Z Resolving deltas: 8% (1303/16204) 2024-06-26T07:19:46.9460327Z Resolving deltas: 9% (1476/16204) 2024-06-26T07:19:46.9471671Z Resolving deltas: 10% (1631/16204) 2024-06-26T07:19:46.9483895Z Resolving deltas: 11% (1783/16204) 2024-06-26T07:19:46.9492182Z Resolving deltas: 12% (1961/16204) 2024-06-26T07:19:46.9505867Z Resolving deltas: 13% (2108/16204) 2024-06-26T07:19:46.9515325Z Resolving deltas: 14% (2269/16204) 2024-06-26T07:19:46.9533553Z Resolving deltas: 15% (2433/16204) 2024-06-26T07:19:46.9549871Z Resolving deltas: 16% (2620/16204) 2024-06-26T07:19:46.9565221Z Resolving deltas: 17% (2759/16204) 2024-06-26T07:19:46.9593855Z Resolving deltas: 18% (2922/16204) 2024-06-26T07:19:46.9621902Z Resolving deltas: 19% (3088/16204) 2024-06-26T07:19:46.9640789Z Resolving deltas: 20% (3255/16204) 2024-06-26T07:19:46.9643408Z Resolving deltas: 21% (3524/16204) 2024-06-26T07:19:46.9652885Z Resolving deltas: 22% (3575/16204) 2024-06-26T07:19:46.9661996Z Resolving deltas: 23% (3735/16204) 2024-06-26T07:19:46.9672960Z Resolving deltas: 24% (3903/16204) 2024-06-26T07:19:46.9683684Z Resolving deltas: 25% (4095/16204) 2024-06-26T07:19:46.9721054Z Resolving deltas: 26% (4223/16204) 2024-06-26T07:19:46.9741311Z Resolving deltas: 27% (4389/16204) 2024-06-26T07:19:46.9753971Z Resolving deltas: 28% (4566/16204) 2024-06-26T07:19:46.9768308Z Resolving deltas: 29% (4704/16204) 2024-06-26T07:19:46.9786546Z Resolving deltas: 30% (4867/16204) 2024-06-26T07:19:46.9798133Z Resolving deltas: 31% (5069/16204) 2024-06-26T07:19:46.9807394Z Resolving deltas: 32% (5202/16204) 2024-06-26T07:19:46.9816263Z Resolving deltas: 33% (5377/16204) 2024-06-26T07:19:46.9828423Z Resolving deltas: 34% (5510/16204) 2024-06-26T07:19:46.9848113Z Resolving deltas: 35% (5678/16204) 2024-06-26T07:19:46.9862934Z Resolving deltas: 36% (5845/16204) 2024-06-26T07:19:46.9881753Z Resolving deltas: 37% (6000/16204) 2024-06-26T07:19:46.9887329Z Resolving deltas: 38% (6204/16204) 2024-06-26T07:19:46.9908128Z Resolving deltas: 39% (6331/16204) 2024-06-26T07:19:46.9917777Z Resolving deltas: 40% (6494/16204) 2024-06-26T07:19:46.9924570Z Resolving deltas: 41% (6675/16204) 2024-06-26T07:19:46.9947346Z Resolving deltas: 42% (6824/16204) 2024-06-26T07:19:46.9951902Z Resolving deltas: 43% (7032/16204) 2024-06-26T07:19:46.9959783Z Resolving deltas: 44% (7160/16204) 2024-06-26T07:19:46.9969688Z Resolving deltas: 45% (7314/16204) 2024-06-26T07:19:46.9992377Z Resolving deltas: 46% (7456/16204) 2024-06-26T07:19:47.0008624Z Resolving deltas: 47% (7627/16204) 2024-06-26T07:19:47.0019854Z Resolving deltas: 48% (7779/16204) 2024-06-26T07:19:47.0040771Z Resolving deltas: 49% (7940/16204) 2024-06-26T07:19:47.0049733Z Resolving deltas: 50% (8119/16204) 2024-06-26T07:19:47.0055623Z Resolving deltas: 51% (8267/16204) 2024-06-26T07:19:47.0065159Z Resolving deltas: 52% (8432/16204) 2024-06-26T07:19:47.0084818Z Resolving deltas: 53% (8595/16204) 2024-06-26T07:19:47.0087430Z Resolving deltas: 54% (8896/16204) 2024-06-26T07:19:47.0092303Z Resolving deltas: 55% (8954/16204) 2024-06-26T07:19:47.0100203Z Resolving deltas: 56% (9081/16204) 2024-06-26T07:19:47.0115097Z Resolving deltas: 57% (9241/16204) 2024-06-26T07:19:47.0122282Z Resolving deltas: 58% (9405/16204) 2024-06-26T07:19:47.0131549Z Resolving deltas: 59% (9614/16204) 2024-06-26T07:19:47.0139828Z Resolving deltas: 61% (9885/16204) 2024-06-26T07:19:47.0151144Z Resolving deltas: 62% (10105/16204) 2024-06-26T07:19:47.0159049Z Resolving deltas: 64% (10381/16204) 2024-06-26T07:19:47.0166366Z Resolving deltas: 65% (10560/16204) 2024-06-26T07:19:47.0172684Z Resolving deltas: 66% (10734/16204) 2024-06-26T07:19:47.0192063Z Resolving deltas: 67% (10903/16204) 2024-06-26T07:19:47.0203515Z Resolving deltas: 69% (11207/16204) 2024-06-26T07:19:47.0220353Z Resolving deltas: 70% (11353/16204) 2024-06-26T07:19:47.0226796Z Resolving deltas: 71% (11506/16204) 2024-06-26T07:19:47.0238165Z Resolving deltas: 72% (11671/16204) 2024-06-26T07:19:47.0252182Z Resolving deltas: 73% (11830/16204) 2024-06-26T07:19:47.0265428Z Resolving deltas: 74% (11992/16204) 2024-06-26T07:19:47.0299968Z Resolving deltas: 75% (12175/16204) 2024-06-26T07:19:47.0303819Z Resolving deltas: 78% (12715/16204) 2024-06-26T07:19:47.0313983Z Resolving deltas: 79% (12803/16204) 2024-06-26T07:19:47.0321843Z Resolving deltas: 80% (12966/16204) 2024-06-26T07:19:47.0334851Z Resolving deltas: 81% (13136/16204) 2024-06-26T07:19:47.0344106Z Resolving deltas: 82% (13313/16204) 2024-06-26T07:19:47.0351285Z Resolving deltas: 83% (13451/16204) 2024-06-26T07:19:47.0362704Z Resolving deltas: 84% (13615/16204) 2024-06-26T07:19:47.0382978Z Resolving deltas: 85% (13775/16204) 2024-06-26T07:19:47.0392674Z Resolving deltas: 86% (13936/16204) 2024-06-26T07:19:47.0401492Z Resolving deltas: 87% (14100/16204) 2024-06-26T07:19:47.0407523Z Resolving deltas: 88% (14310/16204) 2024-06-26T07:19:47.0415290Z Resolving deltas: 89% (14443/16204) 2024-06-26T07:19:47.0424401Z Resolving deltas: 90% (14604/16204) 2024-06-26T07:19:47.0431862Z Resolving deltas: 91% (14765/16204) 2024-06-26T07:19:47.0442882Z Resolving deltas: 92% (14915/16204) 2024-06-26T07:19:47.0447306Z Resolving deltas: 93% (15111/16204) 2024-06-26T07:19:47.0456645Z Resolving deltas: 94% (15240/16204) 2024-06-26T07:19:47.0464434Z Resolving deltas: 95% (15422/16204) 2024-06-26T07:19:47.0471899Z Resolving deltas: 96% (15561/16204) 2024-06-26T07:19:47.0477896Z Resolving deltas: 97% (15720/16204) 2024-06-26T07:19:47.0485311Z Resolving deltas: 98% (15880/16204) 2024-06-26T07:19:47.0518309Z Resolving deltas: 99% (16043/16204) 2024-06-26T07:19:47.0518749Z Resolving deltas: 100% (16204/16204) 2024-06-26T07:19:47.0519203Z Resolving deltas: 100% (16204/16204), done. 2024-06-26T07:19:48.1068359Z + pushd torchbench 2024-06-26T07:19:48.1068810Z ~/workspace/torchbench ~/workspace 2024-06-26T07:19:48.1069381Z + git checkout 23512dbebd44a11eb84afbf53c3c071dd105297e 2024-06-26T07:19:48.1634343Z Note: switching to '23512dbebd44a11eb84afbf53c3c071dd105297e'. 2024-06-26T07:19:48.1634961Z 2024-06-26T07:19:48.1635466Z You are in 'detached HEAD' state. You can look around, make experimental 2024-06-26T07:19:48.1636700Z changes and commit them, and you can discard any commits you make in this 2024-06-26T07:19:48.1637512Z state without impacting any branches by switching back to a branch. 2024-06-26T07:19:48.1638129Z 2024-06-26T07:19:48.1638576Z If you want to create a new branch to retain commits you create, you may 2024-06-26T07:19:48.1639552Z do so (now or later) by using -c with the switch command. Example: 2024-06-26T07:19:48.1640138Z 2024-06-26T07:19:48.1640343Z git switch -c 2024-06-26T07:19:48.1640643Z 2024-06-26T07:19:48.1640782Z Or undo this operation with: 2024-06-26T07:19:48.1641036Z 2024-06-26T07:19:48.1641162Z git switch - 2024-06-26T07:19:48.1641345Z 2024-06-26T07:19:48.1641704Z Turn off this advice by setting config variable advice.detachedHead to false 2024-06-26T07:19:48.1642234Z 2024-06-26T07:19:48.1642395Z HEAD is now at 23512dbe Pin numpy version 2024-06-26T07:19:48.1642851Z + '[' '' ']' 2024-06-26T07:19:48.1643214Z + python install.py --continue_on_fail 2024-06-26T07:19:50.1246228Z checking packages torch, torchvision, torchaudio are installed...OK 2024-06-26T07:20:05.6027691Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2024-06-26T07:20:07.9605384Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2024-06-26T07:20:10.5434715Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2024-06-26T07:20:12.8907088Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2024-06-26T07:20:12.9057323Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2024-06-26T07:20:21.5405116Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2024-06-26T07:20:24.9217801Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2024-06-26T07:20:28.2715636Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2024-06-26T07:20:31.6416035Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2024-06-26T07:20:31.6418885Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2024-06-26T07:20:33.4548781Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2024-06-26T07:20:36.9846217Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2024-06-26T07:20:37.0001134Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2024-06-26T07:21:31.8162301Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2024-06-26T07:21:44.3272221Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2024-06-26T07:21:52.0856489Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2024-06-26T07:21:59.5044072Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2024-06-26T07:22:09.2650380Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2024-06-26T07:22:17.0190642Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2024-06-26T07:22:23.9905817Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2024-06-26T07:22:31.6464495Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2024-06-26T07:22:39.8642681Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2024-06-26T07:22:47.6215754Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2024-06-26T07:22:55.0240724Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2024-06-26T07:23:02.4633770Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2024-06-26T07:23:06.4318426Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2024-06-26T07:23:45.8235217Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2024-06-26T07:23:51.8131649Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2024-06-26T07:23:58.5166684Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2024-06-26T07:24:09.4176973Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2024-06-26T07:24:11.5925191Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2024-06-26T07:24:14.0410555Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2024-06-26T07:24:21.9013668Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2024-06-26T07:24:30.0743891Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2024-06-26T07:24:36.8601571Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2024-06-26T07:24:45.6686881Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2024-06-26T07:24:52.7244541Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2024-06-26T07:24:58.9968027Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2024-06-26T07:25:06.3181405Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2024-06-26T07:25:22.6183913Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2024-06-26T07:25:29.9671801Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2024-06-26T07:25:35.2738090Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2024-06-26T07:25:41.3017628Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2024-06-26T07:25:49.1187845Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2024-06-26T07:25:49.1190354Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2024-06-26T07:26:02.5408668Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2024-06-26T07:26:07.8249373Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2024-06-26T07:26:07.8251728Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2024-06-26T07:26:16.4035915Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2024-06-26T07:26:18.6536921Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2024-06-26T07:26:20.9242733Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2024-06-26T07:27:00.7720845Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2024-06-26T07:28:20.7760193Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2024-06-26T07:28:20.7918053Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2024-06-26T07:28:23.3301380Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2024-06-26T07:28:23.3457614Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2024-06-26T07:28:23.3616641Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2024-06-26T07:28:23.3763634Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2024-06-26T07:28:23.3911232Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2024-06-26T07:28:23.4065881Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2024-06-26T07:28:43.8800980Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2024-06-26T07:28:43.8804029Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2024-06-26T07:28:46.1711192Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2024-06-26T07:28:49.5548312Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2024-06-26T07:28:49.5705501Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2024-06-26T07:28:49.5861819Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2024-06-26T07:28:49.6011291Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2024-06-26T07:28:49.6163159Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2024-06-26T07:28:49.6314408Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2024-06-26T07:28:56.7949919Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2024-06-26T07:28:59.2684846Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2024-06-26T07:29:03.7589746Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2024-06-26T07:29:03.7747685Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2024-06-26T07:29:03.7905176Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2024-06-26T07:29:03.8057477Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2024-06-26T07:29:03.8210329Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2024-06-26T07:29:03.8360066Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2024-06-26T07:29:15.8262095Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2024-06-26T07:29:28.6465331Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2024-06-26T07:29:28.6621814Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2024-06-26T07:29:28.6625591Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2024-06-26T07:29:28.6629661Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2024-06-26T07:29:32.1774091Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2024-06-26T07:29:35.0679842Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2024-06-26T07:29:35.0839796Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2024-06-26T07:29:59.7179345Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2024-06-26T07:30:06.4129251Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2024-06-26T07:30:11.4349326Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2024-06-26T07:30:18.4346739Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2024-06-26T07:30:18.4503584Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2024-06-26T07:30:18.4662180Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2024-06-26T07:30:18.4818849Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2024-06-26T07:30:18.4977545Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2024-06-26T07:30:18.5132839Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2024-06-26T07:30:18.5280207Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2024-06-26T07:30:18.5432199Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2024-06-26T07:30:29.9077258Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2024-06-26T07:30:42.7226754Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2024-06-26T07:30:42.7383137Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2024-06-26T07:30:45.6899902Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2024-06-26T07:30:48.8612624Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2024-06-26T07:30:49.2835173Z + popd 2024-06-26T07:30:49.2835493Z ~/workspace 2024-06-26T07:30:49.2835875Z + [[ dynamo_eager_torchbench != *cpu_inductor* ]] 2024-06-26T07:30:49.2838035Z + install_torchrec_and_fbgemm 2024-06-26T07:30:49.2838492Z + local torchrec_commit 2024-06-26T07:30:49.2838890Z ++ get_pinned_commit torchrec 2024-06-26T07:30:49.2839337Z ++ cat .github/ci_commit_pins/torchrec.txt 2024-06-26T07:30:49.2870680Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T07:30:49.2871238Z + local fbgemm_commit 2024-06-26T07:30:49.2872751Z ++ get_pinned_commit fbgemm 2024-06-26T07:30:49.2873164Z ++ cat .github/ci_commit_pins/fbgemm.txt 2024-06-26T07:30:49.2882202Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:30:49.2882919Z + pip_uninstall torchrec-nightly 2024-06-26T07:30:49.2883409Z + pip uninstall -y torchrec-nightly 2024-06-26T07:30:49.6488166Z WARNING: Skipping torchrec-nightly as it is not installed. 2024-06-26T07:30:49.6936669Z + pip_uninstall fbgemm-gpu-nightly 2024-06-26T07:30:49.6937735Z + pip uninstall -y fbgemm-gpu-nightly 2024-06-26T07:30:50.0372485Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed. 2024-06-26T07:30:50.0788302Z + pip_install setuptools-git-versioning scikit-build pyre-extensions 2024-06-26T07:30:50.0789562Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions 2024-06-26T07:30:50.5180209Z Collecting setuptools-git-versioning 2024-06-26T07:30:50.5318830Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB) 2024-06-26T07:30:50.6298870Z Collecting scikit-build 2024-06-26T07:30:50.6323892Z Downloading scikit_build-0.18.0-py3-none-any.whl.metadata (17 kB) 2024-06-26T07:30:50.7328264Z Collecting pyre-extensions 2024-06-26T07:30:50.7355591Z Downloading pyre_extensions-0.0.30-py3-none-any.whl.metadata (4.0 kB) 2024-06-26T07:30:50.7804926Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.1) 2024-06-26T07:30:50.7807075Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (69.5.1) 2024-06-26T07:30:50.8382544Z Collecting toml>=0.10.2 (from setuptools-git-versioning) 2024-06-26T07:30:50.8406993Z Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) 2024-06-26T07:30:50.9015028Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0) 2024-06-26T07:30:50.9024525Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1) 2024-06-26T07:30:50.9029883Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.43.0) 2024-06-26T07:30:50.9541708Z Collecting typing-inspect (from pyre-extensions) 2024-06-26T07:30:50.9567333Z Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB) 2024-06-26T07:30:51.0010981Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.2) 2024-06-26T07:30:51.0237353Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0) 2024-06-26T07:30:51.0280499Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB) 2024-06-26T07:30:51.0828167Z Downloading scikit_build-0.18.0-py3-none-any.whl (85 kB) 2024-06-26T07:30:51.1395712Z Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB) 2024-06-26T07:30:51.1979233Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2024-06-26T07:30:51.2477169Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2024-06-26T07:30:53.4231927Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions 2024-06-26T07:30:53.7578920Z Successfully installed pyre-extensions-0.0.30 scikit-build-0.18.0 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0 2024-06-26T07:30:53.8798198Z + CUDA_PATH=/usr/local/cuda-12.1 2024-06-26T07:30:53.8799558Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-06-26T07:30:53.8801681Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-06-26T07:30:54.2183395Z Collecting fbgemm-gpu 2024-06-26T07:30:54.2185008Z Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-kni7r2i6/fbgemm-gpu_07de2e0ac7284ad183d93f8c46e9038e 2024-06-26T07:30:54.2201196Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-kni7r2i6/fbgemm-gpu_07de2e0ac7284ad183d93f8c46e9038e 2024-06-26T07:30:55.1462891Z Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd' 2024-06-26T07:30:55.1478110Z Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:30:55.5448727Z Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:30:56.0084830Z Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:30:56.0085979Z Running command git submodule update --init --recursive -q 2024-06-26T07:31:03.4860222Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:31:03.4880138Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.21.2) 2024-06-26T07:31:03.4886749Z Building wheels for collected packages: fbgemm-gpu 2024-06-26T08:12:59.2949958Z Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-06-26T08:12:59.4751301Z [?25h Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=252630018 sha256=8cf7eeacfc9cbe816aa954024d9325eccf15a5dc2c8c3383764ee9804b595c23 2024-06-26T08:12:59.4753173Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39 2024-06-26T08:12:59.4771479Z Successfully built fbgemm-gpu 2024-06-26T08:13:01.6256488Z Installing collected packages: fbgemm-gpu 2024-06-26T08:13:04.7438440Z Successfully installed fbgemm-gpu-0.4.1rc0.post421 2024-06-26T08:13:05.3671576Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:13:05.3673246Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:13:05.7053668Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:13:05.7057318Z Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-dy7in_3v 2024-06-26T08:13:05.7073930Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-dy7in_3v 2024-06-26T08:13:06.3251491Z Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2' 2024-06-26T08:13:06.3269978Z Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:13:06.5990663Z Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:13:06.9092679Z Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:13:07.1370588Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T08:13:07.1424954Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9) 2024-06-26T08:13:07.1426916Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3) 2024-06-26T08:13:07.1428777Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0) 2024-06-26T08:13:07.1953875Z Collecting torchmetrics (from torchrec==0.3.2) 2024-06-26T08:13:07.2140315Z Downloading torchmetrics-1.4.0.post0-py3-none-any.whl.metadata (19 kB) 2024-06-26T08:13:07.2199898Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.4) 2024-06-26T08:13:07.2203121Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421) 2024-06-26T08:13:07.2216987Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.21.2) 2024-06-26T08:13:07.2237954Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.10.0) 2024-06-26T08:13:07.2912748Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0) 2024-06-26T08:13:07.2917691Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-06-26T08:13:07.2923019Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-06-26T08:13:07.3701484Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.1) 2024-06-26T08:13:07.3707028Z Requirement already satisfied: torch>=1.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.5.0a0+gitb8c4c54) 2024-06-26T08:13:07.3901140Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2) 2024-06-26T08:13:07.3942063Z Downloading lightning_utilities-0.11.2-py3-none-any.whl.metadata (4.7 kB) 2024-06-26T08:13:07.4110618Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (69.5.1) 2024-06-26T08:13:07.4114181Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.2) 2024-06-26T08:13:07.4146670Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0) 2024-06-26T08:13:07.4216825Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.13.1) 2024-06-26T08:13:07.4219751Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.12.1) 2024-06-26T08:13:07.4222520Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.8.8) 2024-06-26T08:13:07.4225944Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.1.4) 2024-06-26T08:13:07.4228359Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2024.2.0) 2024-06-26T08:13:07.4893587Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.1.5) 2024-06-26T08:13:07.5202404Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.3.0) 2024-06-26T08:13:07.5356337Z Downloading torchmetrics-1.4.0.post0-py3-none-any.whl (868 kB) 2024-06-26T08:13:07.5567090Z Downloading lightning_utilities-0.11.2-py3-none-any.whl (26 kB) 2024-06-26T08:13:07.5662410Z Building wheels for collected packages: torchrec 2024-06-26T08:13:08.0021491Z Building wheel for torchrec (setup.py) ... [?25l- \ | done 2024-06-26T08:13:08.0032326Z [?25h Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374488 sha256=41de40fcc977e5cfcb4660f52b09983fedffb2ffd914dc33691910148bd8cfd0 2024-06-26T08:13:08.0034155Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8 2024-06-26T08:13:08.0055805Z Successfully built torchrec 2024-06-26T08:13:10.1063727Z Installing collected packages: lightning-utilities, torchmetrics, torchrec 2024-06-26T08:13:10.6633853Z Successfully installed lightning-utilities-0.11.2 torchmetrics-1.4.0.post0 torchrec-0.3.2 2024-06-26T08:13:10.7965178Z ++ pwd 2024-06-26T08:13:10.7967165Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2024-06-26T08:13:10.7967775Z + test_dynamo_benchmark torchbench 0 2024-06-26T08:13:10.7970265Z ++ pwd 2024-06-26T08:13:10.7973958Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:13:10.7974720Z + local suite=torchbench 2024-06-26T08:13:10.7976774Z + shift 2024-06-26T08:13:10.7977212Z + local shard_id=0 2024-06-26T08:13:10.7977667Z + shift 2024-06-26T08:13:10.7978052Z + [[ dynamo_eager_torchbench == *perf_compare* ]] 2024-06-26T08:13:10.7978581Z + [[ dynamo_eager_torchbench == *perf* ]] 2024-06-26T08:13:10.7979240Z + [[ dynamo_eager_torchbench == *cpu_inductor* ]] 2024-06-26T08:13:10.7980343Z + [[ dynamo_eager_torchbench == *aot_inductor* ]] 2024-06-26T08:13:10.7981312Z + test_single_dynamo_benchmark inference torchbench 0 --inference --bfloat16 2024-06-26T08:13:10.7981966Z ++ pwd 2024-06-26T08:13:10.7982437Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:13:10.7983266Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:13:10.8007887Z + local name=inference 2024-06-26T08:13:10.8008560Z + shift 2024-06-26T08:13:10.8008963Z + local suite=torchbench 2024-06-26T08:13:10.8009413Z + shift 2024-06-26T08:13:10.8009784Z + local shard_id=0 2024-06-26T08:13:10.8010198Z + shift 2024-06-26T08:13:10.8010559Z + partition_flags=() 2024-06-26T08:13:10.8011252Z + local partition_flags 2024-06-26T08:13:10.8011830Z + [[ -n 2 ]] 2024-06-26T08:13:10.8012581Z + [[ -n 0 ]] 2024-06-26T08:13:10.8013442Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-06-26T08:13:10.8014476Z + [[ dynamo_eager_torchbench == *perf_compare* ]] 2024-06-26T08:13:10.8015203Z + [[ dynamo_eager_torchbench == *perf* ]] 2024-06-26T08:13:10.8015902Z + [[ dynamo_eager_torchbench == *aot_inductor* ]] 2024-06-26T08:13:10.8018375Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --backend eager --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2024-06-26T08:13:16.4266730Z 2024-06-26T08:13:16.9838042Z loading model: 0it [00:00, ?it/s]/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/optim/apply_optimizer_in_backward.py:47: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead. 2024-06-26T08:13:16.9841627Z from torch.distributed.optim import _apply_optimizer_in_backward 2024-06-26T08:13:17.1745827Z 2024-06-26T08:13:17.1746457Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:17.1746988Z cuda eval torchrec_dlrm 2024-06-26T08:13:17.1757569Z Traceback (most recent call last): 2024-06-26T08:13:17.1758413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:13:17.1759547Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:13:17.1760395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 447, in forward_pass 2024-06-26T08:13:17.1761364Z return mod(*inputs) 2024-06-26T08:13:17.1762703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:13:17.1763712Z return self._call_impl(*args, **kwargs) 2024-06-26T08:13:17.1764740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:13:17.1765655Z return forward_call(*args, **kwargs) 2024-06-26T08:13:17.1766596Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward 2024-06-26T08:13:17.1767644Z logits = self.model(batch.dense_features, batch.sparse_features) 2024-06-26T08:13:17.1768849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:13:17.1769840Z return self._call_impl(*args, **kwargs) 2024-06-26T08:13:17.1770833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:13:17.1771758Z return forward_call(*args, **kwargs) 2024-06-26T08:13:17.1772978Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 572, in forward 2024-06-26T08:13:17.1773868Z concatenated_dense = self.inter_arch( 2024-06-26T08:13:17.1774925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:13:17.1775917Z return self._call_impl(*args, **kwargs) 2024-06-26T08:13:17.1776933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:13:17.1777854Z return forward_call(*args, **kwargs) 2024-06-26T08:13:17.1778788Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 284, in forward 2024-06-26T08:13:17.1779794Z return self.crossnet(combined_values.reshape([B, -1])) 2024-06-26T08:13:17.1780938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:13:17.1781918Z return self._call_impl(*args, **kwargs) 2024-06-26T08:13:17.1783061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:13:17.1784068Z return forward_call(*args, **kwargs) 2024-06-26T08:13:17.1785047Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/crossnet.py", line 175, in forward 2024-06-26T08:13:17.1786083Z x_l_v = torch.nn.functional.linear(x_l, self.V_kernels[layer]) 2024-06-26T08:13:17.1787008Z RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::BFloat16 2024-06-26T08:13:17.1787642Z 2024-06-26T08:13:17.1787957Z The above exception was the direct cause of the following exception: 2024-06-26T08:13:17.1788454Z 2024-06-26T08:13:17.1788600Z Traceback (most recent call last): 2024-06-26T08:13:17.1789302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:13:17.1790021Z ) = runner.load_model( 2024-06-26T08:13:17.1790727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:13:17.1791564Z self.validate_model(model, example_inputs) 2024-06-26T08:13:17.1792387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:13:17.1793232Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:13:17.1793730Z RuntimeError: Eager run failed 2024-06-26T08:13:17.1794006Z 2024-06-26T08:13:17.1794120Z eager_fail_to_run 2024-06-26T08:13:20.5118308Z 2024-06-26T08:13:22.4658372Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:22.4659249Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:13:22.4723850Z cuda eval BERT_pytorch 2024-06-26T08:13:25.9148651Z pass 2024-06-26T08:13:25.9151879Z TIMING: backend_compile:0.00024 entire_frame_compile:2.63645 2024-06-26T08:13:25.9152817Z STATS: call_* op count: 1076 | FakeTensor.__torch_dispatch__:897 | FakeTensorMode.__torch_dispatch__:4194 2024-06-26T08:13:25.9153821Z Dynamo produced 1 graphs covering 1076 ops with 0 graph breaks (0 unique) 2024-06-26T08:13:29.2426062Z 2024-06-26T08:13:31.7982055Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:31.7982552Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:13:31.8034187Z cuda eval Background_Matting 2024-06-26T08:13:31.8037140Z pass_due_to_skip 2024-06-26T08:13:31.8554740Z TIMING: 2024-06-26T08:13:31.8555157Z STATS: call_* op count: 0 2024-06-26T08:13:31.8555749Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T08:13:35.0957674Z 2024-06-26T08:13:37.7471006Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:37.7471506Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:13:37.7497088Z cuda eval LearningToPaint 2024-06-26T08:13:38.8915691Z pass 2024-06-26T08:13:38.8916775Z TIMING: backend_compile:7e-05 entire_frame_compile:0.65254 2024-06-26T08:13:38.8917765Z STATS: call_* op count: 142 | FakeTensor.__torch_dispatch__:408 | FakeTensorMode.__torch_dispatch__:1950 2024-06-26T08:13:38.8921020Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-06-26T08:13:42.4129415Z 2024-06-26T08:13:43.6613814Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/vgg16-397923af.pth 2024-06-26T08:13:43.6815430Z 2024-06-26T08:13:43.6815437Z 2024-06-26T08:13:43.7819276Z 0% 0.00/528M [00:00 will be ignored 2024-06-26T08:26:01.8142913Z pass 2024-06-26T08:26:01.8601955Z TIMING: entire_frame_compile:6.76138 backend_compile:0.00042 2024-06-26T08:26:01.8603013Z STATS: call_* op count: 672 | FakeTensorMode.__torch_dispatch__:6352 | FakeTensor.__torch_dispatch__:2579 2024-06-26T08:26:01.8604284Z Dynamo produced 3 graphs covering 672 ops with 7 graph breaks (5 unique) 2024-06-26T08:26:05.2734322Z 2024-06-26T08:26:06.4682002Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:26:06.4682494Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:26:06.4682954Z cuda train alexnet 2024-06-26T08:26:08.0502236Z pass 2024-06-26T08:26:08.0687194Z TIMING: entire_frame_compile:0.26644 backend_compile:9e-05 2024-06-26T08:26:08.0688381Z STATS: call_* op count: 26 | FakeTensor.__torch_dispatch__:130 | FakeTensorMode.__torch_dispatch__:542 2024-06-26T08:26:08.0689405Z Dynamo produced 2 graphs covering 26 ops with 6 graph breaks (5 unique) 2024-06-26T08:26:11.3387262Z 2024-06-26T08:26:13.0243588Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:26:13.0244085Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:26:13.0244622Z cuda train basic_gnn_edgecnn 2024-06-26T08:26:15.1183135Z W0626 08:26:15.117000 140559877603968 torch/_logging/_internal.py:1040] [24/0] Profiler function will be ignored 2024-06-26T08:26:15.6542545Z pass 2024-06-26T08:26:15.6724288Z TIMING: entire_frame_compile:1.69996 backend_compile:0.00038 2024-06-26T08:26:15.6725344Z STATS: call_* op count: 103 | FakeTensorMode.__torch_dispatch__:920 | FakeTensor.__torch_dispatch__:388 2024-06-26T08:26:15.6726446Z Dynamo produced 8 graphs covering 103 ops with 20 graph breaks (6 unique) 2024-06-26T08:26:19.1408952Z 2024-06-26T08:26:20.8678089Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:26:20.8678659Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:26:20.8679145Z cuda train basic_gnn_gcn 2024-06-26T08:26:22.0167385Z W0626 08:26:22.015000 139849981239936 torch/_logging/_internal.py:1040] [18/0] Profiler function will be ignored 2024-06-26T08:26:22.3527407Z pass 2024-06-26T08:26:22.3637886Z TIMING: entire_frame_compile:0.90886 backend_compile:0.00043 2024-06-26T08:26:22.3638925Z STATS: call_* op count: 99 | FakeTensorMode.__torch_dispatch__:393 | FakeTensor.__torch_dispatch__:182 2024-06-26T08:26:22.3639948Z Dynamo produced 10 graphs covering 99 ops with 13 graph breaks (6 unique) 2024-06-26T08:26:25.8018880Z 2024-06-26T08:26:27.4807070Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:26:27.4807631Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:26:27.4808095Z cuda train basic_gnn_gin 2024-06-26T08:26:28.5978392Z W0626 08:26:28.597000 139625726759552 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:26:29.1333864Z pass 2024-06-26T08:26:29.1427108Z TIMING: entire_frame_compile:1.10578 backend_compile:0.00017 2024-06-26T08:26:29.1428180Z STATS: call_* op count: 113 | FakeTensorMode.__torch_dispatch__:500 | FakeTensor.__torch_dispatch__:248 2024-06-26T08:26:29.1429537Z Dynamo produced 3 graphs covering 113 ops with 7 graph breaks (5 unique) 2024-06-26T08:26:32.5688527Z 2024-06-26T08:26:34.2947481Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:26:34.2948111Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:26:34.2948603Z cuda train basic_gnn_sage 2024-06-26T08:26:35.3236108Z W0626 08:26:35.322000 140618381894272 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:26:35.7702506Z pass 2024-06-26T08:26:35.7788143Z TIMING: entire_frame_compile:0.93143 backend_compile:0.00016 2024-06-26T08:26:35.7789153Z STATS: call_* op count: 107 | FakeTensorMode.__torch_dispatch__:460 | FakeTensor.__torch_dispatch__:191 2024-06-26T08:26:35.7790399Z Dynamo produced 3 graphs covering 107 ops with 7 graph breaks (5 unique) 2024-06-26T08:26:39.2126062Z 2024-06-26T08:26:39.7018568Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:26:39.7019063Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:26:39.7019528Z cuda train dcgan 2024-06-26T08:26:40.2336935Z pass 2024-06-26T08:26:40.2354301Z TIMING: entire_frame_compile:0.27138 backend_compile:8e-05 2024-06-26T08:26:40.2355218Z STATS: call_* op count: 17 | FakeTensor.__torch_dispatch__:102 | FakeTensorMode.__torch_dispatch__:464 2024-06-26T08:26:40.2356222Z Dynamo produced 2 graphs covering 17 ops with 6 graph breaks (5 unique) 2024-06-26T08:26:43.4679106Z 2024-06-26T08:26:46.8693203Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:26:46.8693694Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:26:46.8694159Z cuda train demucs 2024-06-26T08:27:42.2999243Z pass 2024-06-26T08:27:42.3219287Z TIMING: entire_frame_compile:0.77686 backend_compile:0.00027 2024-06-26T08:27:42.3220456Z STATS: call_* op count: 88 | FakeTensorMode.__torch_dispatch__:1698 | FakeTensor.__torch_dispatch__:434 2024-06-26T08:27:42.3222982Z Dynamo produced 6 graphs covering 88 ops with 9 graph breaks (6 unique) 2024-06-26T08:27:45.6684721Z 2024-06-26T08:27:47.7494074Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:27:47.7494573Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:27:47.7495043Z cuda train densenet121 2024-06-26T08:27:57.9181705Z pass 2024-06-26T08:27:57.9299046Z TIMING: entire_frame_compile:4.50122 backend_compile:0.00017 2024-06-26T08:27:57.9300312Z STATS: call_* op count: 435 | FakeTensor.__torch_dispatch__:2271 | FakeTensorMode.__torch_dispatch__:14776 2024-06-26T08:27:57.9301340Z Dynamo produced 2 graphs covering 435 ops with 6 graph breaks (5 unique) 2024-06-26T08:28:01.3368813Z 2024-06-26T08:28:05.9023658Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:28:05.9024150Z loading model: 0it [00:04, ?it/s] 2024-06-26T08:28:05.9024633Z cuda train detectron2_maskrcnn_r_50_c4 2024-06-26T08:28:05.9036784Z Traceback (most recent call last): 2024-06-26T08:28:05.9037762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:28:05.9038612Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:28:05.9039586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in forward_and_backward_pass 2024-06-26T08:28:05.9040530Z pred = mod(*cloned_inputs) 2024-06-26T08:28:05.9041674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:28:05.9042668Z return self._call_impl(*args, **kwargs) 2024-06-26T08:28:05.9043680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:28:05.9044600Z return forward_call(*args, **kwargs) 2024-06-26T08:28:05.9045643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-06-26T08:28:05.9050106Z return self.inference(batched_inputs) 2024-06-26T08:28:05.9051645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference 2024-06-26T08:28:05.9053319Z results, _ = self.roi_heads(images, features, proposals, None) 2024-06-26T08:28:05.9054531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:28:05.9055826Z return self._call_impl(*args, **kwargs) 2024-06-26T08:28:05.9057099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:28:05.9058033Z return forward_call(*args, **kwargs) 2024-06-26T08:28:05.9059104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward 2024-06-26T08:28:05.9060380Z box_features = self._shared_roi_transform( 2024-06-26T08:28:05.9061589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform 2024-06-26T08:28:05.9062667Z x = self.pooler(features, boxes) 2024-06-26T08:28:05.9063803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:28:05.9064783Z return self._call_impl(*args, **kwargs) 2024-06-26T08:28:05.9065804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:28:05.9066731Z return forward_call(*args, **kwargs) 2024-06-26T08:28:05.9067718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward 2024-06-26T08:28:05.9068704Z return self.level_poolers[0](x[0], pooler_fmt_boxes) 2024-06-26T08:28:05.9069832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:28:05.9070819Z return self._call_impl(*args, **kwargs) 2024-06-26T08:28:05.9071822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:28:05.9072736Z return forward_call(*args, **kwargs) 2024-06-26T08:28:05.9073705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward 2024-06-26T08:28:05.9074704Z return roi_align( 2024-06-26T08:28:05.9075588Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-06-26T08:28:05.9076844Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-06-26T08:28:05.9078163Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-06-26T08:28:05.9079384Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-06-26T08:28:05.9080759Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-06-26T08:28:05.9081742Z v1 = masked_index(y_low, x_low) 2024-06-26T08:28:05.9082724Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-06-26T08:28:05.9083623Z return input[ 2024-06-26T08:28:05.9087172Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 21890.87 GiB. GPU 0 has a total capacity of 21.98 GiB of which 16.54 GiB is free. Process 29776 has 5.42 GiB memory in use. Of the allocated memory 4.58 GiB is allocated by PyTorch, and 558.89 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T08:28:05.9090410Z 2024-06-26T08:28:05.9090729Z The above exception was the direct cause of the following exception: 2024-06-26T08:28:05.9091235Z 2024-06-26T08:28:05.9091381Z Traceback (most recent call last): 2024-06-26T08:28:05.9092327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:28:05.9093028Z ) = runner.load_model( 2024-06-26T08:28:05.9093725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:28:05.9094544Z self.validate_model(model, example_inputs) 2024-06-26T08:28:05.9095340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:28:05.9096153Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:28:05.9096632Z RuntimeError: Eager run failed 2024-06-26T08:28:05.9096904Z 2024-06-26T08:28:05.9097019Z eager_fail_to_run 2024-06-26T08:28:09.2562878Z 2024-06-26T08:28:17.4028083Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:28:17.4028580Z loading model: 0it [00:08, ?it/s] 2024-06-26T08:28:17.4029029Z cuda train dlrm 2024-06-26T08:28:18.4385326Z pass 2024-06-26T08:28:18.4784561Z TIMING: entire_frame_compile:0.39813 backend_compile:0.0001 2024-06-26T08:28:18.4785580Z STATS: call_* op count: 40 | FakeTensor.__torch_dispatch__:150 | FakeTensorMode.__torch_dispatch__:771 2024-06-26T08:28:18.4786565Z Dynamo produced 2 graphs covering 40 ops with 6 graph breaks (5 unique) 2024-06-26T08:28:21.7367274Z 2024-06-26T08:28:23.1119348Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:28:23.1119839Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:28:23.1120300Z cuda train drq 2024-06-26T08:28:23.9354633Z W0626 08:28:23.934000 139856093102720 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:28:25.0930918Z pass 2024-06-26T08:28:25.0953807Z TIMING: entire_frame_compile:1.7195 backend_compile:0.0003 2024-06-26T08:28:25.0954804Z STATS: call_* op count: 137 | FakeTensorMode.__torch_dispatch__:876 | FakeTensor.__torch_dispatch__:361 2024-06-26T08:28:25.0955903Z Dynamo produced 5 graphs covering 137 ops with 7 graph breaks (5 unique) 2024-06-26T08:28:28.5017858Z 2024-06-26T08:28:30.0955435Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:28:35.8758661Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:28:35.8759368Z cuda train fastNLP_Bert 2024-06-26T08:28:35.8760914Z W0626 08:28:35.875000 140183195947648 torch/_dynamo/variables/tensor.py:714] [3/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T08:28:35.8762560Z W0626 08:28:35.875000 140183195947648 torch/_dynamo/variables/tensor.py:714] [3/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T08:28:35.8763871Z W0626 08:28:35.875000 140183195947648 torch/_dynamo/variables/tensor.py:714] [3/0] or: 2024-06-26T08:28:35.8765206Z W0626 08:28:35.875000 140183195947648 torch/_dynamo/variables/tensor.py:714] [3/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T08:28:35.8766786Z W0626 08:28:35.875000 140183195947648 torch/_dynamo/variables/tensor.py:714] [3/0] to include these operations in the captured graph. 2024-06-26T08:28:35.8768068Z W0626 08:28:35.875000 140183195947648 torch/_dynamo/variables/tensor.py:714] [3/0] 2024-06-26T08:28:39.0325020Z pass 2024-06-26T08:28:39.1240946Z TIMING: entire_frame_compile:2.95595 backend_compile:0.00036 2024-06-26T08:28:39.1242132Z STATS: call_* op count: 448 | FakeTensorMode.__torch_dispatch__:4457 | FakeTensor.__torch_dispatch__:632 2024-06-26T08:28:39.1243163Z Dynamo produced 7 graphs covering 448 ops with 10 graph breaks (6 unique) 2024-06-26T08:28:42.4837882Z 2024-06-26T08:28:43.2518794Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:28:43.2519377Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:28:43.2519852Z cuda train functorch_dp_cifar10 2024-06-26T08:28:44.3559411Z W0626 08:28:44.355000 139988268114560 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:28:47.2758382Z pass 2024-06-26T08:28:47.2821002Z TIMING: entire_frame_compile:3.51626 backend_compile:0.00029 2024-06-26T08:28:47.2822227Z STATS: call_* op count: 277 | FakeTensor.__torch_dispatch__:1587 | FakeTensorMode.__torch_dispatch__:1888 2024-06-26T08:28:47.2823364Z Dynamo produced 3 graphs covering 277 ops with 7 graph breaks (5 unique) 2024-06-26T08:28:50.6478274Z 2024-06-26T08:28:51.1595882Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:28:51.1596391Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:28:51.1596867Z cuda train functorch_maml_omniglot 2024-06-26T08:28:51.6100633Z W0626 08:28:51.609000 140583243223680 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:28:52.2140756Z pass 2024-06-26T08:28:52.2142040Z TIMING: entire_frame_compile:0.81575 backend_compile:0.00017 2024-06-26T08:28:52.2143113Z STATS: call_* op count: 78 | FakeTensor.__torch_dispatch__:375 | FakeTensorMode.__torch_dispatch__:498 2024-06-26T08:28:52.2144216Z Dynamo produced 3 graphs covering 78 ops with 7 graph breaks (5 unique) 2024-06-26T08:28:55.5144401Z 2024-06-26T08:28:56.7315814Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:28:56.7316380Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:28:56.7316846Z cuda train hf_Albert 2024-06-26T08:29:05.7757861Z W0626 08:29:05.775000 140329777857152 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:29:06.8912324Z pass 2024-06-26T08:29:06.9638924Z TIMING: entire_frame_compile:3.43214 backend_compile:0.00027 2024-06-26T08:29:06.9640068Z STATS: call_* op count: 542 | FakeTensorMode.__torch_dispatch__:8918 | FakeTensor.__torch_dispatch__:1793 2024-06-26T08:29:06.9641317Z Dynamo produced 3 graphs covering 542 ops with 6 graph breaks (5 unique) 2024-06-26T08:29:10.3442243Z 2024-06-26T08:29:14.0413337Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:29:14.0413825Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:29:14.0414293Z cuda train hf_Bart 2024-06-26T08:29:26.3104928Z W0626 08:29:26.309000 140055356560000 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:29:36.0897906Z pass 2024-06-26T08:29:36.1373877Z TIMING: entire_frame_compile:12.27161 backend_compile:0.00088 2024-06-26T08:29:36.1376732Z STATS: call_* op count: 1250 | FakeTensorMode.__torch_dispatch__:9789 | FakeTensor.__torch_dispatch__:6609 2024-06-26T08:29:36.1377790Z Dynamo produced 3 graphs covering 1250 ops with 6 graph breaks (5 unique) 2024-06-26T08:29:39.7729327Z 2024-06-26T08:29:42.1541811Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:29:42.1542305Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:29:42.1544506Z cuda train hf_Bert 2024-06-26T08:29:54.3374312Z W0626 08:29:54.336000 140335434248832 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:30:02.0683301Z pass 2024-06-26T08:30:02.1073947Z TIMING: entire_frame_compile:12.15043 backend_compile:0.00069 2024-06-26T08:30:02.1074977Z STATS: call_* op count: 995 | FakeTensorMode.__torch_dispatch__:8340 | FakeTensor.__torch_dispatch__:5084 2024-06-26T08:30:02.1076009Z Dynamo produced 3 graphs covering 995 ops with 6 graph breaks (5 unique) 2024-06-26T08:30:05.6432814Z 2024-06-26T08:30:10.2684205Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:30:10.2684744Z loading model: 0it [00:04, ?it/s] 2024-06-26T08:30:10.2685207Z cuda train hf_Bert_large 2024-06-26T08:30:38.7717753Z W0626 08:30:38.770000 139908162663040 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:30:53.6036303Z pass 2024-06-26T08:30:53.7527104Z TIMING: entire_frame_compile:23.60595 backend_compile:0.00128 2024-06-26T08:30:53.7529242Z STATS: call_* op count: 1919 | FakeTensorMode.__torch_dispatch__:16212 | FakeTensor.__torch_dispatch__:9884 2024-06-26T08:30:53.7531532Z Dynamo produced 3 graphs covering 1919 ops with 6 graph breaks (5 unique) 2024-06-26T08:30:57.6354069Z 2024-06-26T08:31:00.9495014Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:31:00.9495965Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:31:00.9496453Z cuda train hf_BigBird 2024-06-26T08:31:12.2093974Z WARNING:common:fp64 golden ref were not generated for hf_BigBird. Setting accuracy check to cosine 2024-06-26T08:31:27.1601358Z W0626 08:31:27.159000 140289283240576 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:31:35.5237988Z pass 2024-06-26T08:31:35.5947075Z TIMING: entire_frame_compile:19.07879 backend_compile:0.00158 2024-06-26T08:31:35.5948529Z STATS: call_* op count: 3975 | FakeTensorMode.__torch_dispatch__:37294 | FakeTensor.__torch_dispatch__:3857 2024-06-26T08:31:35.5949558Z Dynamo produced 3 graphs covering 3975 ops with 6 graph breaks (5 unique) 2024-06-26T08:31:39.2687198Z 2024-06-26T08:31:41.3287142Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:31:41.3288153Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:31:41.3289084Z cuda train hf_DistilBert 2024-06-26T08:31:47.1566276Z W0626 08:31:47.155000 140202042663552 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:31:51.2853545Z pass 2024-06-26T08:31:51.3065428Z TIMING: entire_frame_compile:5.30059 backend_compile:0.00042 2024-06-26T08:31:51.3066374Z STATS: call_* op count: 540 | FakeTensor.__torch_dispatch__:2658 | FakeTensorMode.__torch_dispatch__:4363 2024-06-26T08:31:51.3067377Z Dynamo produced 3 graphs covering 540 ops with 6 graph breaks (5 unique) 2024-06-26T08:31:54.7283601Z 2024-06-26T08:31:57.6862701Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:31:57.6863225Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:31:57.6863751Z cuda train hf_GPT2 2024-06-26T08:32:05.9704848Z W0626 08:32:05.969000 140529570288256 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:32:11.6664815Z pass 2024-06-26T08:32:11.7426909Z TIMING: entire_frame_compile:9.15312 backend_compile:0.00063 2024-06-26T08:32:11.7428274Z STATS: call_* op count: 1088 | FakeTensorMode.__torch_dispatch__:6557 | FakeTensor.__torch_dispatch__:3149 2024-06-26T08:32:11.7429297Z Dynamo produced 3 graphs covering 1088 ops with 6 graph breaks (5 unique) 2024-06-26T08:32:15.3256167Z 2024-06-26T08:32:27.7637301Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:27.7637896Z loading model: 0it [00:12, ?it/s] 2024-06-26T08:32:27.7638423Z cuda train hf_GPT2_large 2024-06-26T08:32:27.7640442Z pass_due_to_skip 2024-06-26T08:32:28.2603893Z TIMING: 2024-06-26T08:32:28.2604480Z STATS: call_* op count: 0 2024-06-26T08:32:28.2605543Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T08:32:31.5020458Z 2024-06-26T08:32:32.9580542Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:32.9581141Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:32:32.9584448Z cuda train hf_Reformer 2024-06-26T08:32:43.4355504Z W0626 08:32:43.434000 139988667146880 torch/_logging/_internal.py:1040] [32/0] Profiler function will be ignored 2024-06-26T08:32:46.8714794Z pass 2024-06-26T08:32:46.9357259Z TIMING: entire_frame_compile:5.94973 backend_compile:0.00105 2024-06-26T08:32:46.9358286Z STATS: call_* op count: 688 | FakeTensorMode.__torch_dispatch__:3263 | FakeTensor.__torch_dispatch__:1549 2024-06-26T08:32:46.9359308Z Dynamo produced 17 graphs covering 688 ops with 25 graph breaks (9 unique) 2024-06-26T08:32:50.4282912Z 2024-06-26T08:32:54.4674003Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:54.4674513Z loading model: 0it [00:04, ?it/s] 2024-06-26T08:32:54.4674975Z cuda train hf_T5_base 2024-06-26T08:32:57.6001355Z WARNING:common:fp64 golden ref were not generated for hf_T5_base. Setting accuracy check to cosine 2024-06-26T08:32:59.7060863Z ERROR:common: 2024-06-26T08:32:59.7061394Z Traceback (most recent call last): 2024-06-26T08:32:59.7062331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2564, in check_accuracy 2024-06-26T08:32:59.7063285Z correct_rerun_result = self.run_n_iterations( 2024-06-26T08:32:59.7064216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2359, in run_n_iterations 2024-06-26T08:32:59.7065120Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-06-26T08:32:59.7066368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 458, in forward_and_backward_pass 2024-06-26T08:32:59.7067869Z self.grad_scaler.scale(loss).backward() 2024-06-26T08:32:59.7068941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-06-26T08:32:59.7069779Z torch.autograd.backward( 2024-06-26T08:32:59.7070688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 288, in backward 2024-06-26T08:32:59.7071558Z _engine_run_backward( 2024-06-26T08:32:59.7072518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 768, in _engine_run_backward 2024-06-26T08:32:59.7073883Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-06-26T08:32:59.7077924Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 192.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 88.44 MiB is free. Process 30648 has 21.88 GiB memory in use. Of the allocated memory 20.95 GiB is allocated by PyTorch, and 595.70 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T08:32:59.7918705Z eager_2nd_run_OOM 2024-06-26T08:32:59.7951516Z TIMING: 2024-06-26T08:32:59.7951824Z STATS: call_* op count: 0 2024-06-26T08:32:59.7952404Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T08:33:03.0633495Z 2024-06-26T08:33:12.8220262Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:33:12.8220756Z loading model: 0it [00:09, ?it/s] 2024-06-26T08:33:12.8221214Z cuda train hf_T5_large 2024-06-26T08:33:12.8224618Z pass_due_to_skip 2024-06-26T08:33:13.2882978Z TIMING: 2024-06-26T08:33:13.2883390Z STATS: call_* op count: 0 2024-06-26T08:33:13.2884002Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T08:33:16.5243994Z 2024-06-26T08:33:17.5564934Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:33:17.5565435Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:33:17.5565899Z cuda train hf_Whisper 2024-06-26T08:33:21.0958891Z W0626 08:33:21.095000 139782563897984 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:33:23.8823389Z pass 2024-06-26T08:33:23.8942505Z TIMING: entire_frame_compile:3.54621 backend_compile:0.00032 2024-06-26T08:33:23.8943720Z STATS: call_* op count: 347 | FakeTensor.__torch_dispatch__:1819 | FakeTensorMode.__torch_dispatch__:2668 2024-06-26T08:33:23.8944737Z Dynamo produced 3 graphs covering 347 ops with 6 graph breaks (5 unique) 2024-06-26T08:33:27.2738978Z 2024-06-26T08:33:27.5423401Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:33:27.5424091Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:33:27.5424764Z cuda train hf_distil_whisper 2024-06-26T08:33:27.5429228Z Traceback (most recent call last): 2024-06-26T08:33:27.5430257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:33:27.5431073Z ) = runner.load_model( 2024-06-26T08:33:27.5431966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 320, in load_model 2024-06-26T08:33:27.5433004Z benchmark = benchmark_cls( 2024-06-26T08:33:27.5433926Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-26T08:33:27.5434813Z obj = type.__call__(cls, *args, **kwargs) 2024-06-26T08:33:27.5435823Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper/__init__.py", line 12, in __init__ 2024-06-26T08:33:27.5436946Z raise NotImplementedError("Training is not implemented.") 2024-06-26T08:33:27.5437650Z NotImplementedError: Training is not implemented. 2024-06-26T08:33:27.5438086Z 2024-06-26T08:33:27.5438221Z model_fail_to_load 2024-06-26T08:33:30.6978512Z 2024-06-26T08:33:31.0780774Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:33:31.0781593Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:33:31.0782314Z cuda train lennard_jones 2024-06-26T08:33:31.4482928Z W0626 08:33:31.447000 140473864716928 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:33:31.8760359Z pass 2024-06-26T08:33:31.8761065Z TIMING: entire_frame_compile:0.58821 backend_compile:0.00015 2024-06-26T08:33:31.8762357Z STATS: call_* op count: 61 | FakeTensor.__torch_dispatch__:275 | FakeTensorMode.__torch_dispatch__:368 2024-06-26T08:33:31.8763726Z Dynamo produced 3 graphs covering 61 ops with 7 graph breaks (5 unique) 2024-06-26T08:33:32.5597944Z accuracy pass_rate=81.82% 2024-06-26T08:33:32.5601326Z calls_captured gmean=0.00x mean=448.030x 2024-06-26T08:33:32.5604898Z unique_graphs gmean=0.00x mean=3.303x 2024-06-26T08:33:32.5609023Z graph_breaks gmean=0.00x mean=6.545x 2024-06-26T08:33:32.5613133Z unique_graph_breaks gmean=0.00x mean=4.333x 2024-06-26T08:33:32.5617115Z autograd_captures gmean=0.00x mean=0.000x 2024-06-26T08:33:32.5620789Z autograd_compiles gmean=0.00x mean=0.000x 2024-06-26T08:33:32.5624725Z cudagraph_skips gmean=0.00x mean=0.000x 2024-06-26T08:33:33.1064292Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamo_eager_torchbench_training.csv 2024-06-26T08:33:33.4357156Z torchrec_dlrm PASS 2024-06-26T08:33:33.4360853Z BERT_pytorch PASS 2024-06-26T08:33:33.4366338Z Background_Matting XFAIL 2024-06-26T08:33:33.4371698Z LearningToPaint PASS 2024-06-26T08:33:33.4377073Z Super_SloMo PASS 2024-06-26T08:33:33.4381956Z alexnet PASS 2024-06-26T08:33:33.4387939Z basic_gnn_edgecnn PASS 2024-06-26T08:33:33.4393576Z basic_gnn_gcn PASS 2024-06-26T08:33:33.4399129Z basic_gnn_gin PASS 2024-06-26T08:33:33.4404275Z basic_gnn_sage PASS 2024-06-26T08:33:33.4409350Z dcgan PASS 2024-06-26T08:33:33.4413999Z demucs PASS 2024-06-26T08:33:33.4419089Z densenet121 PASS 2024-06-26T08:33:33.4423854Z detectron2_maskrcnn_r_50_c4 XFAIL 2024-06-26T08:33:33.4428892Z dlrm PASS 2024-06-26T08:33:33.4433605Z drq PASS 2024-06-26T08:33:33.4438542Z fastNLP_Bert PASS 2024-06-26T08:33:33.4443573Z functorch_dp_cifar10 PASS 2024-06-26T08:33:33.4448545Z functorch_maml_omniglot PASS 2024-06-26T08:33:33.4453729Z hf_Albert PASS 2024-06-26T08:33:33.4458706Z hf_Bart PASS 2024-06-26T08:33:33.4463539Z hf_Bert PASS 2024-06-26T08:33:33.4468526Z hf_Bert_large PASS 2024-06-26T08:33:33.4473575Z hf_BigBird PASS 2024-06-26T08:33:33.4478492Z hf_DistilBert PASS 2024-06-26T08:33:33.4483375Z hf_GPT2 PASS 2024-06-26T08:33:33.4488318Z hf_GPT2_large XFAIL 2024-06-26T08:33:33.4493655Z hf_Reformer PASS 2024-06-26T08:33:33.4498332Z hf_T5_base XFAIL 2024-06-26T08:33:33.4503362Z hf_T5_large XFAIL 2024-06-26T08:33:33.4508476Z hf_Whisper PASS 2024-06-26T08:33:33.4513484Z hf_distil_whisper XFAIL 2024-06-26T08:33:33.4518387Z lennard_jones PASS 2024-06-26T08:33:33.4982145Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamo_eager_torchbench_training.csv 2024-06-26T08:33:33.8100732Z torchrec_dlrm PASS 2024-06-26T08:33:33.8104697Z BERT_pytorch PASS 2024-06-26T08:33:33.8109622Z Background_Matting PASS 2024-06-26T08:33:33.8113827Z LearningToPaint PASS 2024-06-26T08:33:33.8118064Z Super_SloMo PASS 2024-06-26T08:33:33.8122610Z alexnet PASS 2024-06-26T08:33:33.8127716Z basic_gnn_edgecnn PASS 2024-06-26T08:33:33.8131455Z basic_gnn_gcn PASS 2024-06-26T08:33:33.8136200Z basic_gnn_gin PASS 2024-06-26T08:33:33.8140364Z basic_gnn_sage PASS 2024-06-26T08:33:33.8145019Z dcgan PASS 2024-06-26T08:33:33.8149253Z demucs PASS 2024-06-26T08:33:33.8153866Z densenet121 PASS 2024-06-26T08:33:33.8158035Z detectron2_maskrcnn_r_50_c4 PASS 2024-06-26T08:33:33.8162649Z dlrm PASS 2024-06-26T08:33:33.8166895Z drq PASS 2024-06-26T08:33:33.8171367Z fastNLP_Bert PASS 2024-06-26T08:33:33.8175893Z functorch_dp_cifar10 PASS 2024-06-26T08:33:33.8180303Z functorch_maml_omniglot PASS 2024-06-26T08:33:33.8184640Z hf_Albert PASS 2024-06-26T08:33:33.8189180Z hf_Bart PASS 2024-06-26T08:33:33.8193419Z hf_Bert PASS 2024-06-26T08:33:33.8197947Z hf_Bert_large PASS 2024-06-26T08:33:33.8202157Z hf_BigBird PASS 2024-06-26T08:33:33.8206614Z hf_DistilBert PASS 2024-06-26T08:33:33.8210949Z hf_GPT2 PASS 2024-06-26T08:33:33.8215613Z hf_GPT2_large PASS 2024-06-26T08:33:33.8219847Z hf_Reformer PASS 2024-06-26T08:33:33.8224510Z hf_T5_base PASS 2024-06-26T08:33:33.8228858Z hf_T5_large PASS 2024-06-26T08:33:33.8233213Z hf_Whisper PASS 2024-06-26T08:33:33.8237450Z hf_distil_whisper PASS 2024-06-26T08:33:33.8242029Z lennard_jones PASS 2024-06-26T08:33:33.8694917Z + cleanup_workspace 2024-06-26T08:33:33.8696000Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-26T08:33:33.8697241Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-26T08:33:33.8698266Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-26T08:33:33.8698969Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T08:33:33.8699827Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-26T08:33:33.8700787Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-26T08:33:33.8701552Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-26T08:33:34.4504661Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T08:33:34.4505214Z cat test/**/*_toprint.log || true 2024-06-26T08:33:34.4513258Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:33:34.4513799Z env: 2024-06-26T08:33:34.4514075Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:34.4514559Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:34.4515364Z DOCKER_CONTAINER_ID: 272d959298f66930256ee50b307d2e87a6d4a8766a7c2b0ccd856d88c6b87252 2024-06-26T08:33:34.4516075Z ##[endgroup] 2024-06-26T08:33:34.4580071Z cat: test/**/*_toprint.log: No such file or directory 2024-06-26T08:33:34.4644564Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:33:34.4645063Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:33:34.4652481Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:33:34.4653066Z env: 2024-06-26T08:33:34.4653351Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:34.4653829Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:34.4654628Z DOCKER_CONTAINER_ID: 272d959298f66930256ee50b307d2e87a6d4a8766a7c2b0ccd856d88c6b87252 2024-06-26T08:33:34.4655335Z MONITOR_SCRIPT_PID: 23515 2024-06-26T08:33:34.4655705Z ##[endgroup] 2024-06-26T08:33:34.4875083Z Prepare all required actions 2024-06-26T08:33:34.4875552Z Getting action download info 2024-06-26T08:33:34.6483375Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T08:33:34.8033040Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T08:33:34.8033523Z with: 2024-06-26T08:33:34.8034096Z file-suffix: test-dynamo_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688915472 2024-06-26T08:33:34.8034819Z s3-bucket: gha-artifacts 2024-06-26T08:33:34.8035171Z env: 2024-06-26T08:33:34.8035437Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:34.8035903Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:34.8036675Z DOCKER_CONTAINER_ID: 272d959298f66930256ee50b307d2e87a6d4a8766a7c2b0ccd856d88c6b87252 2024-06-26T08:33:34.8037357Z ##[endgroup] 2024-06-26T08:33:34.8062346Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-26T08:33:34.8063050Z # Remove any previous test jsons if they exist 2024-06-26T08:33:34.8063660Z rm -f test-jsons-*.zip 2024-06-26T08:33:34.8064207Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-26T08:33:34.8072195Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:33:34.8072737Z env: 2024-06-26T08:33:34.8073035Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:34.8073513Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:34.8074322Z DOCKER_CONTAINER_ID: 272d959298f66930256ee50b307d2e87a6d4a8766a7c2b0ccd856d88c6b87252 2024-06-26T08:33:34.8075326Z FILE_SUFFIX: test-dynamo_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688915472 2024-06-26T08:33:34.8076053Z ##[endgroup] 2024-06-26T08:33:34.8247950Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-26T08:33:34.8277334Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-26T08:33:34.8278104Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-26T08:33:34.8283884Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-26T08:33:34.8310571Z ##[group]Run # Remove any previous test reports if they exist 2024-06-26T08:33:34.8311266Z # Remove any previous test reports if they exist 2024-06-26T08:33:34.8311864Z rm -f test-reports-*.zip 2024-06-26T08:33:34.8312521Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-26T08:33:34.8320158Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:33:34.8320690Z env: 2024-06-26T08:33:34.8320990Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:34.8321474Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:34.8322273Z DOCKER_CONTAINER_ID: 272d959298f66930256ee50b307d2e87a6d4a8766a7c2b0ccd856d88c6b87252 2024-06-26T08:33:34.8323279Z FILE_SUFFIX: test-dynamo_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688915472 2024-06-26T08:33:34.8324015Z ##[endgroup] 2024-06-26T08:33:34.8487542Z adding: test/test-reports/inference_torchbench.csv (deflated 73%) 2024-06-26T08:33:34.8492811Z adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 92%) 2024-06-26T08:33:34.8494936Z adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 79%) 2024-06-26T08:33:34.8496241Z adding: test/test-reports/training_torchbench.csv (deflated 64%) 2024-06-26T08:33:34.8502541Z adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 97%) 2024-06-26T08:33:34.8504050Z adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 77%) 2024-06-26T08:33:34.8546671Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-26T08:33:34.8547352Z # Remove any previous usage logs if they exist 2024-06-26T08:33:34.8547894Z rm -f logs-*.zip 2024-06-26T08:33:34.8548622Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-26T08:33:34.8549612Z # so check to see if the file exists first 2024-06-26T08:33:34.8550149Z if [ -f 'usage_log.txt' ]; then 2024-06-26T08:33:34.8550707Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-26T08:33:34.8551228Z fi 2024-06-26T08:33:34.8551618Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-26T08:33:34.8552234Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-26T08:33:34.8552767Z fi 2024-06-26T08:33:34.8560209Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:33:34.8560756Z env: 2024-06-26T08:33:34.8561043Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:34.8561530Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:34.8562338Z DOCKER_CONTAINER_ID: 272d959298f66930256ee50b307d2e87a6d4a8766a7c2b0ccd856d88c6b87252 2024-06-26T08:33:34.8563346Z FILE_SUFFIX: test-dynamo_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688915472 2024-06-26T08:33:34.8564068Z ##[endgroup] 2024-06-26T08:33:34.8633624Z adding: usage_log.txt (deflated 92%) 2024-06-26T08:33:34.8700775Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-26T08:33:34.8701567Z # Remove any previous debugging artifacts if they exist 2024-06-26T08:33:34.8702301Z rm -f debug-*.zip 2024-06-26T08:33:34.8702705Z if [ -d 'test/debug' ]; then 2024-06-26T08:33:34.8703316Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-26T08:33:34.8703823Z fi 2024-06-26T08:33:34.8710589Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:33:34.8711132Z env: 2024-06-26T08:33:34.8711427Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:34.8711908Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:34.8712774Z DOCKER_CONTAINER_ID: 272d959298f66930256ee50b307d2e87a6d4a8766a7c2b0ccd856d88c6b87252 2024-06-26T08:33:34.8713797Z FILE_SUFFIX: test-dynamo_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688915472 2024-06-26T08:33:34.8714529Z ##[endgroup] 2024-06-26T08:33:34.8789924Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:33:34.8790397Z with: 2024-06-26T08:33:34.8790694Z s3-bucket: gha-artifacts 2024-06-26T08:33:34.8791156Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:33:34.8791677Z retention-days: 14 2024-06-26T08:33:34.8792027Z if-no-files-found: warn 2024-06-26T08:33:34.8792437Z path: test-jsons-*.zip 2024-06-26T08:33:34.8792822Z name: artifact 2024-06-26T08:33:34.8793143Z region: us-east-1 2024-06-26T08:33:34.8793452Z env: 2024-06-26T08:33:34.8793738Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:34.8794223Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:34.8795025Z DOCKER_CONTAINER_ID: 272d959298f66930256ee50b307d2e87a6d4a8766a7c2b0ccd856d88c6b87252 2024-06-26T08:33:34.8795729Z ##[endgroup] 2024-06-26T08:33:35.1922803Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:33:35.1923454Z With the provided path, there will be 1 file uploaded 2024-06-26T08:33:35.1924161Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:33:35.1949779Z Starting upload of test-jsons-test-dynamo_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688915472.zip 2024-06-26T08:33:35.3641363Z Finished upload of test-jsons-test-dynamo_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688915472.zip 2024-06-26T08:33:35.3783523Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:33:35.3784000Z with: 2024-06-26T08:33:35.3784287Z s3-bucket: gha-artifacts 2024-06-26T08:33:35.3784740Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:33:35.3785252Z retention-days: 14 2024-06-26T08:33:35.3785600Z if-no-files-found: error 2024-06-26T08:33:35.3785989Z path: test-reports-*.zip 2024-06-26T08:33:35.3786361Z name: artifact 2024-06-26T08:33:35.3786670Z region: us-east-1 2024-06-26T08:33:35.3786984Z env: 2024-06-26T08:33:35.3787272Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:35.3787750Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:35.3788686Z DOCKER_CONTAINER_ID: 272d959298f66930256ee50b307d2e87a6d4a8766a7c2b0ccd856d88c6b87252 2024-06-26T08:33:35.3789385Z ##[endgroup] 2024-06-26T08:33:35.6896950Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:33:35.6898243Z With the provided path, there will be 1 file uploaded 2024-06-26T08:33:35.6899566Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:33:35.6923377Z Starting upload of test-reports-test-dynamo_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688915472.zip 2024-06-26T08:33:35.7947518Z Finished upload of test-reports-test-dynamo_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688915472.zip 2024-06-26T08:33:35.8088641Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:33:35.8089104Z with: 2024-06-26T08:33:35.8089385Z s3-bucket: gha-artifacts 2024-06-26T08:33:35.8089819Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:33:35.8090312Z retention-days: 14 2024-06-26T08:33:35.8090664Z if-no-files-found: ignore 2024-06-26T08:33:35.8091035Z path: logs-*.zip 2024-06-26T08:33:35.8091348Z name: artifact 2024-06-26T08:33:35.8091647Z region: us-east-1 2024-06-26T08:33:35.8092187Z env: 2024-06-26T08:33:35.8092475Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:35.8093101Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:35.8093872Z DOCKER_CONTAINER_ID: 272d959298f66930256ee50b307d2e87a6d4a8766a7c2b0ccd856d88c6b87252 2024-06-26T08:33:35.8094556Z ##[endgroup] 2024-06-26T08:33:36.1157759Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:33:36.1158631Z With the provided path, there will be 1 file uploaded 2024-06-26T08:33:36.1159346Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:33:36.1184902Z Starting upload of logs-test-dynamo_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688915472.zip 2024-06-26T08:33:36.2430268Z Finished upload of logs-test-dynamo_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688915472.zip 2024-06-26T08:33:36.2571532Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:33:36.2572232Z with: 2024-06-26T08:33:36.2572559Z s3-bucket: gha-artifacts 2024-06-26T08:33:36.2573005Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:33:36.2573528Z retention-days: 14 2024-06-26T08:33:36.2573873Z if-no-files-found: ignore 2024-06-26T08:33:36.2574254Z path: debug-*.zip 2024-06-26T08:33:36.2574578Z name: artifact 2024-06-26T08:33:36.2574893Z region: us-east-1 2024-06-26T08:33:36.2575211Z env: 2024-06-26T08:33:36.2575495Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:36.2575980Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:36.2576770Z DOCKER_CONTAINER_ID: 272d959298f66930256ee50b307d2e87a6d4a8766a7c2b0ccd856d88c6b87252 2024-06-26T08:33:36.2577473Z ##[endgroup] 2024-06-26T08:33:36.5627881Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-26T08:33:36.5767241Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T08:33:36.5767759Z # shellcheck disable=SC2156 2024-06-26T08:33:36.5768639Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T08:33:36.5777039Z shell: /usr/bin/bash -e {0} 2024-06-26T08:33:36.5777422Z env: 2024-06-26T08:33:36.5777720Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:36.5778205Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:36.5779008Z DOCKER_CONTAINER_ID: 272d959298f66930256ee50b307d2e87a6d4a8766a7c2b0ccd856d88c6b87252 2024-06-26T08:33:36.5779703Z ##[endgroup] 2024-06-26T08:33:36.7453057Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-26T08:33:36.7453651Z with: 2024-06-26T08:33:36.7453920Z env: 2024-06-26T08:33:36.7454206Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:36.7454690Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:36.7455487Z DOCKER_CONTAINER_ID: 272d959298f66930256ee50b307d2e87a6d4a8766a7c2b0ccd856d88c6b87252 2024-06-26T08:33:36.7456184Z ##[endgroup] 2024-06-26T08:33:36.7474411Z ##[group]Run set -eou pipefail 2024-06-26T08:33:36.7474837Z set -eou pipefail 2024-06-26T08:33:36.7475211Z  2024-06-26T08:33:36.7475774Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-26T08:33:36.7476483Z for _ in $(seq 1440); do 2024-06-26T08:33:36.7477153Z  # Break if no ssh session exists anymore 2024-06-26T08:33:36.7477687Z  if [ "$(who)" = "" ]; then 2024-06-26T08:33:36.7478132Z  break 2024-06-26T08:33:36.7478502Z  fi 2024-06-26T08:33:36.7478817Z  echo "." 2024-06-26T08:33:36.7479166Z  sleep 5 2024-06-26T08:33:36.7479505Z done 2024-06-26T08:33:36.7486917Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:33:36.7487452Z env: 2024-06-26T08:33:36.7487740Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:36.7488225Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:36.7489026Z DOCKER_CONTAINER_ID: 272d959298f66930256ee50b307d2e87a6d4a8766a7c2b0ccd856d88c6b87252 2024-06-26T08:33:36.7489721Z ##[endgroup] 2024-06-26T08:33:36.7510140Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-26T08:33:36.7553739Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:33:36.7554579Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:33:36.7555230Z # shellcheck disable=SC2046 2024-06-26T08:33:36.7555733Z docker stop $(docker ps -q) || true 2024-06-26T08:33:36.7556240Z # Prune all of the docker images 2024-06-26T08:33:36.7556733Z docker system prune -af 2024-06-26T08:33:36.7563765Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:33:36.7564291Z env: 2024-06-26T08:33:36.7564581Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:36.7565057Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:36.7565958Z DOCKER_CONTAINER_ID: 272d959298f66930256ee50b307d2e87a6d4a8766a7c2b0ccd856d88c6b87252 2024-06-26T08:33:36.7566649Z ##[endgroup] 2024-06-26T08:33:37.1234737Z 272d959298f6 2024-06-26T08:33:39.3946080Z Deleted Containers: 2024-06-26T08:33:39.3946768Z 272d959298f66930256ee50b307d2e87a6d4a8766a7c2b0ccd856d88c6b87252 2024-06-26T08:33:39.3947290Z 2024-06-26T08:33:43.2813186Z Deleted Images: 2024-06-26T08:33:43.2814772Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T08:33:43.2817346Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T08:33:43.2819078Z deleted: sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd 2024-06-26T08:33:43.2819996Z deleted: sha256:754ead7012f0b7691885a9660f02e80be6cee8c5665486fb995c1fb06adb6e92 2024-06-26T08:33:43.2820918Z deleted: sha256:fbe4496d8830e11be86d8f222522e7956f32f85bfc06b05ea53961f34a58af74 2024-06-26T08:33:43.2821850Z deleted: sha256:1c83814aa1a1a7e0cc998e3479bdab8e8766b975560cbf00690acf39ed899a80 2024-06-26T08:33:43.2822795Z deleted: sha256:e9ea8e61942c7f8ae4aab97b8d102b01d6bb4b6cf2a605808cd34130a17d8a0f 2024-06-26T08:33:43.2823849Z deleted: sha256:acc5fc32588cd95e082a6ce35e48d0f303ae8f96e3ab65c4e375b85e1f3c4a92 2024-06-26T08:33:43.2824762Z deleted: sha256:a93e53400904d29e4185d3be942ea34f19572ea438962788ae4e9b3d9bf4c498 2024-06-26T08:33:43.2825860Z deleted: sha256:c97092b88b5e4c77c68bbc4b51afab64e38719af1ff58013ec49a696cdd40a6e 2024-06-26T08:33:43.2826784Z deleted: sha256:b132889c90d67ffa53655846603e3fcc2372c23f04dbdbc54690a39d304eb89b 2024-06-26T08:33:43.2827671Z deleted: sha256:915d54081ed607710624abe3f2490114c745858831f4fc13396bd3b4b9a121bc 2024-06-26T08:33:43.2828566Z deleted: sha256:bfa8d446c498c87f535aef102695564de1b22c755e6c63944ae2f5073a2b31ec 2024-06-26T08:33:43.2829482Z deleted: sha256:e71460fb363c29e70bbe357d680aaa7ae694f094ef2f2713609158ad90611744 2024-06-26T08:33:43.2830389Z deleted: sha256:84079bb1b4ee1d0119ae3d3e701de460a1324912297c675425c9a36d30fcdb0f 2024-06-26T08:33:43.2831304Z deleted: sha256:d86ac3ca148d4758cb03b493fdd49109d8f1a2c7c85c4b755493c3c4529553e5 2024-06-26T08:33:43.2832227Z deleted: sha256:532b3ab97c44a673a0d5f62ee46fa6a83248a84de8877754a612355be56ed0a3 2024-06-26T08:33:43.2833150Z deleted: sha256:617e2a35f25ae24500c88c0c9b8475f0687fbcbed3d200013b4c99c6d8941aff 2024-06-26T08:33:43.2834096Z deleted: sha256:b88120bcd5a26a660c09c5df13176e0ef098fb2c8cdb57d74ab0796a3d1194d6 2024-06-26T08:33:43.2835008Z deleted: sha256:c016b2e7764735243205fb76fa9a273bb3dd4bf05532288e3a613ce022131ac3 2024-06-26T08:33:43.2835923Z deleted: sha256:bcaca462c390deddd2995de2d245a7495e7d2e1e94f0edad30a573c83d0588fd 2024-06-26T08:33:43.2836854Z deleted: sha256:9e260eba003d5c25d7b410e5c4c3093845d9e46611896840901c689d7fbf8ee6 2024-06-26T08:33:43.2837749Z deleted: sha256:3e62b86f8b911a9b59c62d55f063e71683ae6a01c22418db5d27930712adaad6 2024-06-26T08:33:43.2838652Z deleted: sha256:905920ccb1d88449b8dde686c34a4b2a8b6f59e78ac44916b189ea3593b9fbbb 2024-06-26T08:33:43.2839547Z deleted: sha256:961201237e03acd5061406238dd7a3c1343ac888d7583212cd0ab5ae1b4f0540 2024-06-26T08:33:43.2840578Z deleted: sha256:70047a2aac62159fa28fb4dbd4567ec2ffbab68fe046967e09acb0eb3e5cb9cb 2024-06-26T08:33:43.2841518Z deleted: sha256:1ec680ad781436ae65fa987f5fa1ed032ecd5c82a6e35eea6707de501f62c0f4 2024-06-26T08:33:43.2842456Z deleted: sha256:ef859623a50e85abf6e6c1dbd6188aa765a66cc7b2bd618a1cbe4825b7b2bd91 2024-06-26T08:33:43.2843380Z deleted: sha256:404ade5cacff59773b93ab438bfa64cd3a171d07f6f168bdc4a5d4bfa0b43526 2024-06-26T08:33:43.2844370Z deleted: sha256:5b27de6ee3d9f2f18217bf18dec725f7b4954c0e48c85d35752634c22d62209b 2024-06-26T08:33:43.2845304Z deleted: sha256:498e39c575b3fa069acaac7e46b91b4e54ae3a05f2ada5c4347091f346f25d4c 2024-06-26T08:33:43.2846208Z deleted: sha256:1405919d2694d65669b1f4c842282021afe89208ce86570e96464903733da2d0 2024-06-26T08:33:43.2847092Z deleted: sha256:d5b41e5d70896c0472fc3e6da0516a17d89cd6635dc75010ecddc31ce9690fad 2024-06-26T08:33:43.2848005Z deleted: sha256:18e9379b62b9d0bb40293d28a652eafaee66a499f66f0776bf14e37a513101b7 2024-06-26T08:33:43.2848973Z deleted: sha256:a781f80ed00ff2926a699fd01a309bb0b4fb7d5655eb652c205bb9c7f811035f 2024-06-26T08:33:43.2850164Z deleted: sha256:c0430973b7cf759075330ee29c79bfd1312c91e6033d9dd883b4b1c29743a171 2024-06-26T08:33:43.2851275Z deleted: sha256:e733b3d66efb9342e5a283376c0e1ddffa47a42fecc26e61ecaf0a3472885d97 2024-06-26T08:33:43.2852413Z deleted: sha256:8929c0b2047f74b84c82c39eff9077a92ae1a828cde00c9ab78fb3e0ed5e0283 2024-06-26T08:33:43.2853307Z deleted: sha256:f5b56155fb58a0a60fe6391d3581cf91610ed69d4f0a321257c19944da296a52 2024-06-26T08:33:43.2854195Z deleted: sha256:b54d2cb355238d1aa91e58df93064447b80f53706d1ca62823acc834f5ac54fe 2024-06-26T08:33:43.2855081Z deleted: sha256:e7311285af9eeddb51aa5629aa98386e89f1456ff32843b6aee6443247edf66c 2024-06-26T08:33:43.2855969Z deleted: sha256:67657ceec347da8dc8085d5731633efeac115ba677fa4b7c5966c4260f7abab0 2024-06-26T08:33:43.2856889Z deleted: sha256:b496e50409dbf414930fc9b4437aa0aa4ce27bc7840e050ae459aaffcf9aca86 2024-06-26T08:33:43.2857783Z deleted: sha256:8f5276170cc4a60d36ad7b964a502306b29546254cedcbcb101ea9123497d2c0 2024-06-26T08:33:43.2858659Z deleted: sha256:53d524d1564745e96a7e515462171b73184d9f1ff5454b32fcfbbca39d361467 2024-06-26T08:33:43.2859557Z deleted: sha256:2acbadb00d69b6aca575f04baba4ea26ae2b5c2d85f09945004ad93a78a8eb75 2024-06-26T08:33:43.2860833Z deleted: sha256:02665efeff878aeb603c1edb2e21cb700c171c025d64f5aae1b2b20b12f3964a 2024-06-26T08:33:43.2861757Z deleted: sha256:19bd23e9a51dfe883a8b8d2ff6d66ff1b3476b8912856406bb79b81762aa6d4b 2024-06-26T08:33:43.2862662Z deleted: sha256:73aa8637bf9299d6298dabd858480c274a62117388f2ca1d5e9eb15c6d1e4155 2024-06-26T08:33:43.2863620Z deleted: sha256:5f8968406305f348b1caebc0e035187d6e9965ace7b9d5f2f21114d37c9fe2e1 2024-06-26T08:33:43.2864505Z deleted: sha256:d827104914db0c503cf5c81f13e2456bf4efce42d4cf1f8d5d70838207278bee 2024-06-26T08:33:43.2865396Z deleted: sha256:867b219724f2f1aaf56f42e3e9f03a5d64bdca8a6a72407efd10328aeea5d510 2024-06-26T08:33:43.2866309Z deleted: sha256:b9ff2a7cceb9237b26bcb17728edf8a558eec5a76f63626e84f80541dfc0f1d4 2024-06-26T08:33:43.2867211Z deleted: sha256:be73f509106819007ce2b0f5a1301ac56ae583cb6c1fb69bee93efa0840368f1 2024-06-26T08:33:43.2868104Z deleted: sha256:2a11c07c915c3fdbb4e73096dea73ea950c970d50fe1a5764060bd62b7b7cf2c 2024-06-26T08:33:43.2869016Z deleted: sha256:5e9989622e2abc1c72f8184343507b45d75e4ff971edb68b96cbcd75ae484ff5 2024-06-26T08:33:43.2869923Z deleted: sha256:05ce642bd9cf5a9205bff107df28b1d6cef7d5f6ab3ea944d28c032ec61270df 2024-06-26T08:33:43.2870821Z deleted: sha256:017702617f6c07b73f2088678f4710b794cfe897a213a5cf122a0e2519eae0ac 2024-06-26T08:33:43.2871703Z deleted: sha256:44b3b37bb1ca39dc5cf356de41a23a5c12f605a1992813e408087fb2b0ad6d1c 2024-06-26T08:33:43.2872597Z deleted: sha256:8c387982e9f27db357a1d149dc149cabc0bd04668ad6956387a5072682dada67 2024-06-26T08:33:43.2873486Z deleted: sha256:10d415dca4e3fbb88a747b40317f3c6333c041ba8f73855f67fb9fe981fae5ee 2024-06-26T08:33:43.2874434Z deleted: sha256:6c65469bf0f53dcd625faefe363b8f3136c08388a6cf64d9158e8576e4ee89b1 2024-06-26T08:33:43.2875452Z deleted: sha256:027db5d8b76deeefbd8c34b485ab7ff14ed2a3e6800425c1ce0d77c2e9fdd4ab 2024-06-26T08:33:43.2876342Z deleted: sha256:24941234ad28cf5c8174d7cf18326c10b4e17f2c2137672ea0322df461597dd7 2024-06-26T08:33:43.2877238Z deleted: sha256:240ea2eebfa8519d6c2d88f4c99e66ce4e038d000f93ce8d03726c01d81180ef 2024-06-26T08:33:43.2878150Z deleted: sha256:09e823cb391a69f6a195e3ca8a41624b5f8b34eee14c6eb36277de814a91966e 2024-06-26T08:33:43.2879026Z deleted: sha256:730479d695169a6593bf0ac16b377491dce65175a6d158e012422367d04577fd 2024-06-26T08:33:43.2879901Z deleted: sha256:b0a30f9f7bd5459ff662453b9346a7ba8ea8db1191528345c1953eab4aa2f54e 2024-06-26T08:33:43.2880798Z deleted: sha256:75cefa87effaa3ad361c74642839923101f0f9982f814f3c9e2ed4ad52f2a008 2024-06-26T08:33:43.2881689Z deleted: sha256:6f97db87ba9512e615e871c6982000602982e1e5a5be334a56e7292251e84e00 2024-06-26T08:33:43.2882574Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-06-26T08:33:43.2883472Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-06-26T08:33:43.2884354Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-06-26T08:33:43.2885244Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-06-26T08:33:43.2886143Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-06-26T08:33:43.2887035Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-06-26T08:33:43.2887936Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-06-26T08:33:43.2888845Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-06-26T08:33:43.2889754Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-06-26T08:33:43.2890652Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-06-26T08:33:43.2891549Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-06-26T08:33:43.2892272Z 2024-06-26T08:33:43.2892419Z Total reclaimed space: 40.92GB 2024-06-26T08:33:43.2958599Z Post job cleanup. 2024-06-26T08:33:43.3008272Z Post job cleanup. 2024-06-26T08:33:43.3833159Z [command]/usr/bin/git version 2024-06-26T08:33:43.3875554Z git version 2.40.1 2024-06-26T08:33:43.3915224Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/e56e5f19-dadd-4167-abc5-4e5f4e25262d' before making global git config changes 2024-06-26T08:33:43.3916655Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T08:33:43.3920856Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T08:33:43.3951016Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T08:33:43.3980274Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T08:33:43.4206974Z Entering 'android/libs/fbjni' 2024-06-26T08:33:43.4249120Z Entering 'third_party/FP16' 2024-06-26T08:33:43.4286962Z Entering 'third_party/FXdiv' 2024-06-26T08:33:43.4324281Z Entering 'third_party/NNPACK' 2024-06-26T08:33:43.4360979Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:33:43.4397400Z Entering 'third_party/XNNPACK' 2024-06-26T08:33:43.4444373Z Entering 'third_party/benchmark' 2024-06-26T08:33:43.4486007Z Entering 'third_party/cpp-httplib' 2024-06-26T08:33:43.4523635Z Entering 'third_party/cpuinfo' 2024-06-26T08:33:43.4559829Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:33:43.4595930Z Entering 'third_party/cutlass' 2024-06-26T08:33:43.4636914Z Entering 'third_party/eigen' 2024-06-26T08:33:43.4676605Z Entering 'third_party/fbgemm' 2024-06-26T08:33:43.4711107Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:33:43.4746031Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:33:43.4780699Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:33:43.4820103Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:33:43.4854335Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:33:43.4890080Z Entering 'third_party/flatbuffers' 2024-06-26T08:33:43.4928844Z Entering 'third_party/fmt' 2024-06-26T08:33:43.4962336Z Entering 'third_party/foxi' 2024-06-26T08:33:43.4997635Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:33:43.5031027Z Entering 'third_party/gloo' 2024-06-26T08:33:43.5066434Z Entering 'third_party/googletest' 2024-06-26T08:33:43.5101133Z Entering 'third_party/ideep' 2024-06-26T08:33:43.5136876Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:33:43.5177432Z Entering 'third_party/ittapi' 2024-06-26T08:33:43.5212627Z Entering 'third_party/kineto' 2024-06-26T08:33:43.5247403Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:33:43.5283456Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:33:43.5319865Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:33:43.5355034Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:33:43.5389904Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:33:43.5424191Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:33:43.5460981Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:33:43.5495928Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:33:43.5531655Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:33:43.5567973Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:33:43.5604321Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:33:43.5640369Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:33:43.5677893Z Entering 'third_party/mimalloc' 2024-06-26T08:33:43.5712506Z Entering 'third_party/nccl/nccl' 2024-06-26T08:33:43.5749069Z Entering 'third_party/nlohmann' 2024-06-26T08:33:43.5785250Z Entering 'third_party/onnx' 2024-06-26T08:33:43.5832633Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:33:43.5868267Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:33:43.5901998Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:33:43.5939597Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:33:43.5975910Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:33:43.6007978Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:33:43.6041911Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:33:43.6078807Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:33:43.6111847Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:33:43.6148003Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:33:43.6179477Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:33:43.6216070Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:33:43.6251286Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:33:43.6303096Z Entering 'third_party/pocketfft' 2024-06-26T08:33:43.6338066Z Entering 'third_party/protobuf' 2024-06-26T08:33:43.6375274Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:33:43.6409552Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:33:43.6442277Z Entering 'third_party/psimd' 2024-06-26T08:33:43.6478972Z Entering 'third_party/pthreadpool' 2024-06-26T08:33:43.6512175Z Entering 'third_party/pybind11' 2024-06-26T08:33:43.6549456Z Entering 'third_party/python-peachpy' 2024-06-26T08:33:43.6584044Z Entering 'third_party/sleef' 2024-06-26T08:33:43.6619635Z Entering 'third_party/tensorpipe' 2024-06-26T08:33:43.6656110Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:33:43.6689851Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:33:43.6723819Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:33:43.6759113Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:33:43.6791594Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:33:43.6842462Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T08:33:43.6863026Z http.https://github.com/.extraheader 2024-06-26T08:33:43.6871413Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T08:33:43.6899723Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T08:33:43.7117321Z Entering 'android/libs/fbjni' 2024-06-26T08:33:43.7140141Z http.https://github.com/.extraheader 2024-06-26T08:33:43.7165009Z Entering 'third_party/FP16' 2024-06-26T08:33:43.7186111Z http.https://github.com/.extraheader 2024-06-26T08:33:43.7208658Z Entering 'third_party/FXdiv' 2024-06-26T08:33:43.7231670Z http.https://github.com/.extraheader 2024-06-26T08:33:43.7254833Z Entering 'third_party/NNPACK' 2024-06-26T08:33:43.7277111Z http.https://github.com/.extraheader 2024-06-26T08:33:43.7302582Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:33:43.7324445Z http.https://github.com/.extraheader 2024-06-26T08:33:43.7347728Z Entering 'third_party/XNNPACK' 2024-06-26T08:33:43.7370086Z http.https://github.com/.extraheader 2024-06-26T08:33:43.7405799Z Entering 'third_party/benchmark' 2024-06-26T08:33:43.7430351Z http.https://github.com/.extraheader 2024-06-26T08:33:43.7456810Z Entering 'third_party/cpp-httplib' 2024-06-26T08:33:43.7477487Z http.https://github.com/.extraheader 2024-06-26T08:33:43.7500320Z Entering 'third_party/cpuinfo' 2024-06-26T08:33:43.7523401Z http.https://github.com/.extraheader 2024-06-26T08:33:43.7549239Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:33:43.7570137Z http.https://github.com/.extraheader 2024-06-26T08:33:43.7592949Z Entering 'third_party/cutlass' 2024-06-26T08:33:43.7615526Z http.https://github.com/.extraheader 2024-06-26T08:33:43.7644385Z Entering 'third_party/eigen' 2024-06-26T08:33:43.7667898Z http.https://github.com/.extraheader 2024-06-26T08:33:43.7694286Z Entering 'third_party/fbgemm' 2024-06-26T08:33:43.7716570Z http.https://github.com/.extraheader 2024-06-26T08:33:43.7741208Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:33:43.7763368Z http.https://github.com/.extraheader 2024-06-26T08:33:43.7786199Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:33:43.7806069Z http.https://github.com/.extraheader 2024-06-26T08:33:43.7828922Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:33:43.7849539Z http.https://github.com/.extraheader 2024-06-26T08:33:43.7877773Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:33:43.7899336Z http.https://github.com/.extraheader 2024-06-26T08:33:43.7922740Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:33:43.7944384Z http.https://github.com/.extraheader 2024-06-26T08:33:43.7969677Z Entering 'third_party/flatbuffers' 2024-06-26T08:33:43.7992478Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8018110Z Entering 'third_party/fmt' 2024-06-26T08:33:43.8039851Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8064449Z Entering 'third_party/foxi' 2024-06-26T08:33:43.8084762Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8107354Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:33:43.8129230Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8154720Z Entering 'third_party/gloo' 2024-06-26T08:33:43.8176612Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8199394Z Entering 'third_party/googletest' 2024-06-26T08:33:43.8221872Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8245277Z Entering 'third_party/ideep' 2024-06-26T08:33:43.8268827Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8292288Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:33:43.8314447Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8342871Z Entering 'third_party/ittapi' 2024-06-26T08:33:43.8365798Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8390893Z Entering 'third_party/kineto' 2024-06-26T08:33:43.8414383Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8438321Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:33:43.8460658Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8485264Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:33:43.8507615Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8529970Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:33:43.8552190Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8577763Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:33:43.8599780Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8625045Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:33:43.8646336Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8669282Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:33:43.8691264Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8716691Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:33:43.8738585Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8761938Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:33:43.8783148Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8806615Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:33:43.8828205Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8851432Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:33:43.8873495Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8896672Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:33:43.8918139Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8941360Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:33:43.8963548Z http.https://github.com/.extraheader 2024-06-26T08:33:43.8991606Z Entering 'third_party/mimalloc' 2024-06-26T08:33:43.9014515Z http.https://github.com/.extraheader 2024-06-26T08:33:43.9037746Z Entering 'third_party/nccl/nccl' 2024-06-26T08:33:43.9059319Z http.https://github.com/.extraheader 2024-06-26T08:33:43.9083798Z Entering 'third_party/nlohmann' 2024-06-26T08:33:43.9106505Z http.https://github.com/.extraheader 2024-06-26T08:33:43.9131496Z Entering 'third_party/onnx' 2024-06-26T08:33:43.9155479Z http.https://github.com/.extraheader 2024-06-26T08:33:43.9193050Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:33:43.9214910Z http.https://github.com/.extraheader 2024-06-26T08:33:43.9239271Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:33:43.9261410Z http.https://github.com/.extraheader 2024-06-26T08:33:43.9289080Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:33:43.9313080Z http.https://github.com/.extraheader 2024-06-26T08:33:43.9340305Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:33:43.9362419Z http.https://github.com/.extraheader 2024-06-26T08:33:43.9386675Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:33:43.9408087Z http.https://github.com/.extraheader 2024-06-26T08:33:43.9430995Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:33:43.9452941Z http.https://github.com/.extraheader 2024-06-26T08:33:43.9477107Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:33:43.9497988Z http.https://github.com/.extraheader 2024-06-26T08:33:43.9522611Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:33:43.9544708Z http.https://github.com/.extraheader 2024-06-26T08:33:43.9568540Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:33:43.9590455Z http.https://github.com/.extraheader 2024-06-26T08:33:43.9614061Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:33:43.9635443Z http.https://github.com/.extraheader 2024-06-26T08:33:43.9657970Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:33:43.9679757Z http.https://github.com/.extraheader 2024-06-26T08:33:43.9705913Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:33:43.9726951Z http.https://github.com/.extraheader 2024-06-26T08:33:43.9751756Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:33:43.9773069Z http.https://github.com/.extraheader 2024-06-26T08:33:43.9810906Z Entering 'third_party/pocketfft' 2024-06-26T08:33:43.9835378Z http.https://github.com/.extraheader 2024-06-26T08:33:43.9861317Z Entering 'third_party/protobuf' 2024-06-26T08:33:43.9884288Z http.https://github.com/.extraheader 2024-06-26T08:33:43.9910259Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:33:43.9930258Z http.https://github.com/.extraheader 2024-06-26T08:33:43.9951886Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:33:43.9974436Z http.https://github.com/.extraheader 2024-06-26T08:33:44.0000506Z Entering 'third_party/psimd' 2024-06-26T08:33:44.0023639Z http.https://github.com/.extraheader 2024-06-26T08:33:44.0046648Z Entering 'third_party/pthreadpool' 2024-06-26T08:33:44.0068512Z http.https://github.com/.extraheader 2024-06-26T08:33:44.0094625Z Entering 'third_party/pybind11' 2024-06-26T08:33:44.0117240Z http.https://github.com/.extraheader 2024-06-26T08:33:44.0141060Z Entering 'third_party/python-peachpy' 2024-06-26T08:33:44.0163216Z http.https://github.com/.extraheader 2024-06-26T08:33:44.0185868Z Entering 'third_party/sleef' 2024-06-26T08:33:44.0205950Z http.https://github.com/.extraheader 2024-06-26T08:33:44.0229116Z Entering 'third_party/tensorpipe' 2024-06-26T08:33:44.0249804Z http.https://github.com/.extraheader 2024-06-26T08:33:44.0272743Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:33:44.0293931Z http.https://github.com/.extraheader 2024-06-26T08:33:44.0316692Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:33:44.0338376Z http.https://github.com/.extraheader 2024-06-26T08:33:44.0361841Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:33:44.0384403Z http.https://github.com/.extraheader 2024-06-26T08:33:44.0406582Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:33:44.0428879Z http.https://github.com/.extraheader 2024-06-26T08:33:44.0452613Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:33:44.0474540Z http.https://github.com/.extraheader 2024-06-26T08:33:44.0591855Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-26T08:33:44.0609523Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T08:33:44.0616599Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:33:44.0617159Z ##[endgroup] 2024-06-26T08:33:44.7322920Z Cleaning up orphan processes