2024-08-20T22:09:55.2688835Z Current runner version: '2.319.1' 2024-08-20T22:09:55.2695185Z Runner name: 'i-0573eecbda81770a3' 2024-08-20T22:09:55.2696015Z Runner group name: 'Default' 2024-08-20T22:09:55.2696891Z Machine name: 'ip-10-0-36-93' 2024-08-20T22:09:55.2714279Z Testing runner upgrade compatibility 2024-08-20T22:09:55.4142824Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T22:09:55.4145091Z Actions: read 2024-08-20T22:09:55.4145596Z Attestations: read 2024-08-20T22:09:55.4146159Z Checks: read 2024-08-20T22:09:55.4146715Z Contents: read 2024-08-20T22:09:55.4147237Z Deployments: read 2024-08-20T22:09:55.4147747Z Discussions: read 2024-08-20T22:09:55.4148318Z Issues: read 2024-08-20T22:09:55.4148803Z Metadata: read 2024-08-20T22:09:55.4149288Z Packages: read 2024-08-20T22:09:55.4149845Z Pages: read 2024-08-20T22:09:55.4150351Z PullRequests: read 2024-08-20T22:09:55.4150902Z RepositoryProjects: read 2024-08-20T22:09:55.4151525Z SecurityEvents: read 2024-08-20T22:09:55.4152085Z Statuses: read 2024-08-20T22:09:55.4152787Z ##[endgroup] 2024-08-20T22:09:55.4155970Z Secret source: Actions 2024-08-20T22:09:55.4156758Z Prepare workflow directory 2024-08-20T22:09:55.5078828Z Prepare all required actions 2024-08-20T22:09:55.5243766Z Getting action download info 2024-08-20T22:09:55.6841665Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T22:09:56.1484647Z Download action repository 'pytorch/pytorch@main' (SHA:1ae5d5bb62141d7c9b1b0b66c66a462b4e10b1f2) 2024-08-20T22:10:00.0893961Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T22:10:00.2765220Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T22:10:00.6153021Z Getting action download info 2024-08-20T22:10:00.7119822Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T22:10:00.8758707Z Getting action download info 2024-08-20T22:10:00.9743489Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T22:10:01.1144495Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3) 2024-08-20T22:10:01.1146609Z ##[group] Inputs 2024-08-20T22:10:01.1147098Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:10:01.1156303Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:10:01.1166459Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:01.1167640Z sync-tag: 2024-08-20T22:10:01.1168486Z timeout-minutes: 240 2024-08-20T22:10:01.1168894Z use-gha: 2024-08-20T22:10:01.1169195Z dashboard-tag: 2024-08-20T22:10:01.1169520Z s3-bucket: gha-artifacts 2024-08-20T22:10:01.1169877Z aws-role-to-assume: 2024-08-20T22:10:01.1170204Z ##[endgroup] 2024-08-20T22:10:01.1171103Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:10:01.1891690Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T22:10:01.2030493Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T22:10:01.2042191Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:01.2042720Z ##[endgroup] 2024-08-20T22:10:02.5723879Z Runner Type: amz2023.linux.g5.4xlarge.nvidia.gpu 2024-08-20T22:10:02.5724470Z Instance Type: g5.4xlarge 2024-08-20T22:10:02.5725162Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T22:10:02.5725704Z AMI ID: ami-06c68f701d8090592 2024-08-20T22:10:08.3217623Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T22:10:08.3218221Z with: 2024-08-20T22:10:08.3218936Z github-secret: *** 2024-08-20T22:10:08.3219848Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:08.3220826Z activate-with-label: false 2024-08-20T22:10:08.3221186Z label: with-ssh 2024-08-20T22:10:08.3221505Z remove-existing-keys: true 2024-08-20T22:10:08.3221858Z fail-silently: true 2024-08-20T22:10:08.3222168Z env: 2024-08-20T22:10:08.3222440Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:08.3222780Z ##[endgroup] 2024-08-20T22:10:08.4118574Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T22:10:08.4121534Z ciflow reference detected, attempting to extract PR number 2024-08-20T22:10:08.7680508Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-08-20T22:10:08.8459853Z No SSH keys found for user pytorch-bot[bot] 2024-08-20T22:10:08.8460476Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T22:10:08.9278126Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T22:10:08.9295493Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T22:10:08.9322670Z Login using: ssh ec2-user@ec2-54-210-132-48.compute-1.amazonaws.com 2024-08-20T22:10:08.9323535Z All testing is done inside the container, to start an interactive session run: 2024-08-20T22:10:08.9324571Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:08.9454720Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T22:10:08.9455288Z with: 2024-08-20T22:10:08.9455591Z submodules: recursive 2024-08-20T22:10:08.9455948Z fetch-depth: 0 2024-08-20T22:10:08.9456264Z env: 2024-08-20T22:10:08.9456562Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:08.9456919Z ##[endgroup] 2024-08-20T22:10:08.9645870Z ##[group]Run retry () { 2024-08-20T22:10:08.9646263Z retry () { 2024-08-20T22:10:08.9646986Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T22:10:08.9647651Z } 2024-08-20T22:10:08.9648056Z echo "${GITHUB_WORKSPACE}" 2024-08-20T22:10:08.9648503Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T22:10:08.9648981Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:08.9649440Z else 2024-08-20T22:10:08.9649783Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:08.9650209Z fi 2024-08-20T22:10:08.9650551Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T22:10:08.9664607Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:08.9665131Z env: 2024-08-20T22:10:08.9665431Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:08.9665789Z NO_SUDO: 2024-08-20T22:10:08.9666085Z ##[endgroup] 2024-08-20T22:10:08.9698147Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:09.0489574Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T22:10:09.0490025Z with: 2024-08-20T22:10:09.0490362Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:10:09.0490809Z fetch-depth: 0 2024-08-20T22:10:09.0491129Z submodules: recursive 2024-08-20T22:10:09.0491469Z quiet-checkout: true 2024-08-20T22:10:09.0491828Z repository: pytorch/pytorch 2024-08-20T22:10:09.0492310Z token: *** 2024-08-20T22:10:09.0492608Z ssh-strict: true 2024-08-20T22:10:09.0492932Z persist-credentials: true 2024-08-20T22:10:09.0493292Z clean: true 2024-08-20T22:10:09.0493633Z sparse-checkout-cone-mode: true 2024-08-20T22:10:09.0494022Z lfs: false 2024-08-20T22:10:09.0494324Z set-safe-directory: true 2024-08-20T22:10:09.0494664Z env: 2024-08-20T22:10:09.0494940Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:09.0495275Z ##[endgroup] 2024-08-20T22:10:09.1466795Z Syncing repository: pytorch/pytorch 2024-08-20T22:10:09.1468271Z ##[group]Getting Git version info 2024-08-20T22:10:09.1468996Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:09.1469873Z [command]/usr/bin/git version 2024-08-20T22:10:09.1470254Z git version 2.40.1 2024-08-20T22:10:09.1479726Z ##[endgroup] 2024-08-20T22:10:09.1501213Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/36493bc1-10d8-40ad-8bf7-828668eeaa7d' before making global git config changes 2024-08-20T22:10:09.1503085Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:10:09.1504957Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:09.1558550Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:09.1562913Z ##[group]Initializing the repository 2024-08-20T22:10:09.1566593Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:09.1611003Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T22:10:09.1611942Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T22:10:09.1612843Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T22:10:09.1613477Z hint: 2024-08-20T22:10:09.1614168Z hint: git config --global init.defaultBranch 2024-08-20T22:10:09.1614669Z hint: 2024-08-20T22:10:09.1615205Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T22:10:09.1616082Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T22:10:09.1617304Z hint: 2024-08-20T22:10:09.1617869Z hint: git branch -m 2024-08-20T22:10:09.1618773Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T22:10:09.1627122Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T22:10:09.1670547Z ##[endgroup] 2024-08-20T22:10:09.1671408Z ##[group]Disabling automatic garbage collection 2024-08-20T22:10:09.1674249Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T22:10:09.1715440Z ##[endgroup] 2024-08-20T22:10:09.1716251Z ##[group]Setting up auth 2024-08-20T22:10:09.1722769Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:10:09.1763863Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:10:09.2157122Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:10:09.2199546Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:10:09.2555210Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:10:09.2614737Z ##[endgroup] 2024-08-20T22:10:09.2615595Z ##[group]Fetching the repository 2024-08-20T22:10:09.2622101Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T22:10:12.5261557Z remote: Enumerating objects: 1008509 2024-08-20T22:10:12.5262589Z remote: Enumerating objects: 1010229, done. 2024-08-20T22:10:12.5263823Z remote: Counting objects: 0% (1/1720) 2024-08-20T22:10:12.5264966Z remote: Counting objects: 1% (18/1720) 2024-08-20T22:10:12.5266277Z remote: Counting objects: 2% (35/1720) 2024-08-20T22:10:12.5267112Z remote: Counting objects: 3% (52/1720) 2024-08-20T22:10:12.5267771Z remote: Counting objects: 4% (69/1720) 2024-08-20T22:10:12.5268441Z remote: Counting objects: 5% (86/1720) 2024-08-20T22:10:12.5269160Z remote: Counting objects: 6% (104/1720) 2024-08-20T22:10:12.5269780Z remote: Counting objects: 7% (121/1720) 2024-08-20T22:10:12.5270410Z remote: Counting objects: 8% (138/1720) 2024-08-20T22:10:12.5270987Z remote: Counting objects: 9% (155/1720) 2024-08-20T22:10:12.5271488Z remote: Counting objects: 10% (172/1720) 2024-08-20T22:10:12.5271999Z remote: Counting objects: 11% (190/1720) 2024-08-20T22:10:12.5272502Z remote: Counting objects: 12% (207/1720) 2024-08-20T22:10:12.5273006Z remote: Counting objects: 13% (224/1720) 2024-08-20T22:10:12.5273503Z remote: Counting objects: 14% (241/1720) 2024-08-20T22:10:12.5274021Z remote: Counting objects: 15% (258/1720) 2024-08-20T22:10:12.5274528Z remote: Counting objects: 16% (276/1720) 2024-08-20T22:10:12.5275072Z remote: Counting objects: 17% (293/1720) 2024-08-20T22:10:12.5275737Z remote: Counting objects: 18% (310/1720) 2024-08-20T22:10:12.5276394Z remote: Counting objects: 19% (327/1720) 2024-08-20T22:10:12.5277014Z remote: Counting objects: 20% (344/1720) 2024-08-20T22:10:12.5277591Z remote: Counting objects: 21% (362/1720) 2024-08-20T22:10:12.5278203Z remote: Counting objects: 22% (379/1720) 2024-08-20T22:10:12.5278718Z remote: Counting objects: 23% (396/1720) 2024-08-20T22:10:12.5279233Z remote: Counting objects: 24% (413/1720) 2024-08-20T22:10:12.5279736Z remote: Counting objects: 25% (430/1720) 2024-08-20T22:10:12.5280248Z remote: Counting objects: 26% (448/1720) 2024-08-20T22:10:12.5280758Z remote: Counting objects: 27% (465/1720) 2024-08-20T22:10:12.5281629Z remote: Counting objects: 28% (482/1720) 2024-08-20T22:10:12.5282139Z remote: Counting objects: 29% (499/1720) 2024-08-20T22:10:12.5282651Z remote: Counting objects: 30% (516/1720) 2024-08-20T22:10:12.5283161Z remote: Counting objects: 31% (534/1720) 2024-08-20T22:10:12.5283670Z remote: Counting objects: 32% (551/1720) 2024-08-20T22:10:12.5284170Z remote: Counting objects: 33% (568/1720) 2024-08-20T22:10:12.5284810Z remote: Counting objects: 34% (585/1720) 2024-08-20T22:10:12.5285327Z remote: Counting objects: 35% (602/1720) 2024-08-20T22:10:12.5285837Z remote: Counting objects: 36% (620/1720) 2024-08-20T22:10:12.5286340Z remote: Counting objects: 37% (637/1720) 2024-08-20T22:10:12.5286853Z remote: Counting objects: 38% (654/1720) 2024-08-20T22:10:12.5287361Z remote: Counting objects: 39% (671/1720) 2024-08-20T22:10:12.5287872Z remote: Counting objects: 40% (688/1720) 2024-08-20T22:10:12.5288386Z remote: Counting objects: 41% (706/1720) 2024-08-20T22:10:12.5288899Z remote: Counting objects: 42% (723/1720) 2024-08-20T22:10:12.5289437Z remote: Counting objects: 43% (740/1720) 2024-08-20T22:10:12.5289947Z remote: Counting objects: 44% (757/1720) 2024-08-20T22:10:12.5290458Z remote: Counting objects: 45% (774/1720) 2024-08-20T22:10:12.5290963Z remote: Counting objects: 46% (792/1720) 2024-08-20T22:10:12.5291482Z remote: Counting objects: 47% (809/1720) 2024-08-20T22:10:12.5291994Z remote: Counting objects: 48% (826/1720) 2024-08-20T22:10:12.5292503Z remote: Counting objects: 49% (843/1720) 2024-08-20T22:10:12.5293015Z remote: Counting objects: 50% (860/1720) 2024-08-20T22:10:12.5293526Z remote: Counting objects: 51% (878/1720) 2024-08-20T22:10:12.5294038Z remote: Counting objects: 52% (895/1720) 2024-08-20T22:10:12.5294542Z remote: Counting objects: 53% (912/1720) 2024-08-20T22:10:12.5295058Z remote: Counting objects: 54% (929/1720) 2024-08-20T22:10:12.5295569Z remote: Counting objects: 55% (946/1720) 2024-08-20T22:10:12.5296079Z remote: Counting objects: 56% (964/1720) 2024-08-20T22:10:12.5296583Z remote: Counting objects: 57% (981/1720) 2024-08-20T22:10:12.5297094Z remote: Counting objects: 58% (998/1720) 2024-08-20T22:10:12.5297614Z remote: Counting objects: 59% (1015/1720) 2024-08-20T22:10:12.5298157Z remote: Counting objects: 60% (1032/1720) 2024-08-20T22:10:12.5298685Z remote: Counting objects: 61% (1050/1720) 2024-08-20T22:10:12.5299214Z remote: Counting objects: 62% (1067/1720) 2024-08-20T22:10:12.5299755Z remote: Counting objects: 63% (1084/1720) 2024-08-20T22:10:12.5300284Z remote: Counting objects: 64% (1101/1720) 2024-08-20T22:10:12.5300812Z remote: Counting objects: 65% (1118/1720) 2024-08-20T22:10:12.5301341Z remote: Counting objects: 66% (1136/1720) 2024-08-20T22:10:12.5301870Z remote: Counting objects: 67% (1153/1720) 2024-08-20T22:10:12.5302393Z remote: Counting objects: 68% (1170/1720) 2024-08-20T22:10:12.5302921Z remote: Counting objects: 69% (1187/1720) 2024-08-20T22:10:12.5303440Z remote: Counting objects: 70% (1204/1720) 2024-08-20T22:10:12.5303968Z remote: Counting objects: 71% (1222/1720) 2024-08-20T22:10:12.5304494Z remote: Counting objects: 72% (1239/1720) 2024-08-20T22:10:12.5305026Z remote: Counting objects: 73% (1256/1720) 2024-08-20T22:10:12.5305545Z remote: Counting objects: 74% (1273/1720) 2024-08-20T22:10:12.5306070Z remote: Counting objects: 75% (1290/1720) 2024-08-20T22:10:12.5306599Z remote: Counting objects: 76% (1308/1720) 2024-08-20T22:10:12.5307123Z remote: Counting objects: 77% (1325/1720) 2024-08-20T22:10:12.5307645Z remote: Counting objects: 78% (1342/1720) 2024-08-20T22:10:12.5308266Z remote: Counting objects: 79% (1359/1720) 2024-08-20T22:10:12.5308796Z remote: Counting objects: 80% (1376/1720) 2024-08-20T22:10:12.5309322Z remote: Counting objects: 81% (1394/1720) 2024-08-20T22:10:12.5310130Z remote: Counting objects: 82% (1411/1720) 2024-08-20T22:10:12.5310889Z remote: Counting objects: 83% (1428/1720) 2024-08-20T22:10:12.5311408Z remote: Counting objects: 84% (1445/1720) 2024-08-20T22:10:12.5311921Z remote: Counting objects: 85% (1462/1720) 2024-08-20T22:10:12.5312582Z remote: Counting objects: 86% (1480/1720) 2024-08-20T22:10:12.5313095Z remote: Counting objects: 87% (1497/1720) 2024-08-20T22:10:12.5313604Z remote: Counting objects: 88% (1514/1720) 2024-08-20T22:10:12.5314114Z remote: Counting objects: 89% (1531/1720) 2024-08-20T22:10:12.5314620Z remote: Counting objects: 90% (1548/1720) 2024-08-20T22:10:12.5315126Z remote: Counting objects: 91% (1566/1720) 2024-08-20T22:10:12.5315641Z remote: Counting objects: 92% (1583/1720) 2024-08-20T22:10:12.5316151Z remote: Counting objects: 93% (1600/1720) 2024-08-20T22:10:12.5316659Z remote: Counting objects: 94% (1617/1720) 2024-08-20T22:10:12.5317167Z remote: Counting objects: 95% (1634/1720) 2024-08-20T22:10:12.5317684Z remote: Counting objects: 96% (1652/1720) 2024-08-20T22:10:12.5318335Z remote: Counting objects: 97% (1669/1720) 2024-08-20T22:10:12.5318856Z remote: Counting objects: 98% (1686/1720) 2024-08-20T22:10:12.5319377Z remote: Counting objects: 99% (1703/1720) 2024-08-20T22:10:12.5319896Z remote: Counting objects: 100% (1720/1720) 2024-08-20T22:10:12.5320453Z remote: Counting objects: 100% (1720/1720), done. 2024-08-20T22:10:12.5514611Z remote: Compressing objects: 0% (1/836) 2024-08-20T22:10:12.5831950Z remote: Compressing objects: 1% (9/836) 2024-08-20T22:10:12.6140383Z remote: Compressing objects: 2% (17/836) 2024-08-20T22:10:12.6587220Z remote: Compressing objects: 3% (26/836) 2024-08-20T22:10:12.7408635Z remote: Compressing objects: 4% (34/836) 2024-08-20T22:10:12.7875017Z remote: Compressing objects: 5% (42/836) 2024-08-20T22:10:12.8536900Z remote: Compressing objects: 6% (51/836) 2024-08-20T22:10:12.8869885Z remote: Compressing objects: 7% (59/836) 2024-08-20T22:10:12.9266804Z remote: Compressing objects: 8% (67/836) 2024-08-20T22:10:12.9479649Z remote: Compressing objects: 9% (76/836) 2024-08-20T22:10:12.9740149Z remote: Compressing objects: 10% (84/836) 2024-08-20T22:10:12.9976037Z remote: Compressing objects: 11% (92/836) 2024-08-20T22:10:13.0148603Z remote: Compressing objects: 12% (101/836) 2024-08-20T22:10:13.0339039Z remote: Compressing objects: 13% (109/836) 2024-08-20T22:10:13.0459520Z remote: Compressing objects: 14% (118/836) 2024-08-20T22:10:13.0546785Z remote: Compressing objects: 15% (126/836) 2024-08-20T22:10:13.0637391Z remote: Compressing objects: 16% (134/836) 2024-08-20T22:10:13.0702054Z remote: Compressing objects: 17% (143/836) 2024-08-20T22:10:13.0744988Z remote: Compressing objects: 18% (151/836) 2024-08-20T22:10:13.0795678Z remote: Compressing objects: 19% (159/836) 2024-08-20T22:10:13.0833031Z remote: Compressing objects: 20% (168/836) 2024-08-20T22:10:13.0856092Z remote: Compressing objects: 21% (176/836) 2024-08-20T22:10:13.0862222Z remote: Compressing objects: 22% (184/836) 2024-08-20T22:10:13.0871339Z remote: Compressing objects: 23% (193/836) 2024-08-20T22:10:13.0885530Z remote: Compressing objects: 24% (201/836) 2024-08-20T22:10:13.0893380Z remote: Compressing objects: 25% (209/836) 2024-08-20T22:10:13.0906120Z remote: Compressing objects: 26% (218/836) 2024-08-20T22:10:13.0911961Z remote: Compressing objects: 27% (226/836) 2024-08-20T22:10:13.0935012Z remote: Compressing objects: 28% (235/836) 2024-08-20T22:10:13.0949788Z remote: Compressing objects: 29% (243/836) 2024-08-20T22:10:13.0963689Z remote: Compressing objects: 30% (251/836) 2024-08-20T22:10:13.0979670Z remote: Compressing objects: 31% (260/836) 2024-08-20T22:10:13.0987232Z remote: Compressing objects: 32% (268/836) 2024-08-20T22:10:13.1000794Z remote: Compressing objects: 33% (276/836) 2024-08-20T22:10:13.1010245Z remote: Compressing objects: 34% (285/836) 2024-08-20T22:10:13.1019026Z remote: Compressing objects: 35% (293/836) 2024-08-20T22:10:13.1032022Z remote: Compressing objects: 36% (301/836) 2024-08-20T22:10:13.1040511Z remote: Compressing objects: 37% (310/836) 2024-08-20T22:10:13.1054065Z remote: Compressing objects: 38% (318/836) 2024-08-20T22:10:13.1062478Z remote: Compressing objects: 39% (327/836) 2024-08-20T22:10:13.1070049Z remote: Compressing objects: 40% (335/836) 2024-08-20T22:10:13.1077440Z remote: Compressing objects: 41% (343/836) 2024-08-20T22:10:13.1086847Z remote: Compressing objects: 42% (352/836) 2024-08-20T22:10:13.1092957Z remote: Compressing objects: 43% (360/836) 2024-08-20T22:10:13.1101941Z remote: Compressing objects: 44% (368/836) 2024-08-20T22:10:13.1107465Z remote: Compressing objects: 45% (377/836) 2024-08-20T22:10:13.1112982Z remote: Compressing objects: 46% (385/836) 2024-08-20T22:10:13.1117937Z remote: Compressing objects: 47% (393/836) 2024-08-20T22:10:13.1122215Z remote: Compressing objects: 48% (402/836) 2024-08-20T22:10:13.1126695Z remote: Compressing objects: 49% (410/836) 2024-08-20T22:10:13.1129820Z remote: Compressing objects: 50% (418/836) 2024-08-20T22:10:13.1133791Z remote: Compressing objects: 51% (427/836) 2024-08-20T22:10:13.1136876Z remote: Compressing objects: 52% (435/836) 2024-08-20T22:10:13.1139142Z remote: Compressing objects: 53% (444/836) 2024-08-20T22:10:13.1142512Z remote: Compressing objects: 54% (452/836) 2024-08-20T22:10:13.1144352Z remote: Compressing objects: 55% (460/836) 2024-08-20T22:10:13.1146112Z remote: Compressing objects: 56% (469/836) 2024-08-20T22:10:13.1147719Z remote: Compressing objects: 57% (477/836) 2024-08-20T22:10:13.1148826Z remote: Compressing objects: 58% (485/836) 2024-08-20T22:10:13.1150444Z remote: Compressing objects: 59% (494/836) 2024-08-20T22:10:13.1153485Z remote: Compressing objects: 60% (502/836) 2024-08-20T22:10:13.1154032Z remote: Compressing objects: 61% (510/836) 2024-08-20T22:10:13.1154566Z remote: Compressing objects: 62% (519/836) 2024-08-20T22:10:13.1155090Z remote: Compressing objects: 63% (527/836) 2024-08-20T22:10:13.1157380Z remote: Compressing objects: 64% (536/836) 2024-08-20T22:10:13.1164633Z remote: Compressing objects: 65% (544/836) 2024-08-20T22:10:13.1170864Z remote: Compressing objects: 66% (552/836) 2024-08-20T22:10:13.1175724Z remote: Compressing objects: 67% (561/836) 2024-08-20T22:10:13.1180049Z remote: Compressing objects: 68% (569/836) 2024-08-20T22:10:13.1185142Z remote: Compressing objects: 69% (577/836) 2024-08-20T22:10:13.1189771Z remote: Compressing objects: 70% (586/836) 2024-08-20T22:10:13.1195845Z remote: Compressing objects: 71% (594/836) 2024-08-20T22:10:13.1198188Z remote: Compressing objects: 72% (602/836) 2024-08-20T22:10:13.1202954Z remote: Compressing objects: 73% (611/836) 2024-08-20T22:10:13.1206831Z remote: Compressing objects: 74% (619/836) 2024-08-20T22:10:13.1210660Z remote: Compressing objects: 75% (627/836) 2024-08-20T22:10:13.1214696Z remote: Compressing objects: 76% (636/836) 2024-08-20T22:10:13.1218348Z remote: Compressing objects: 77% (644/836) 2024-08-20T22:10:13.1220941Z remote: Compressing objects: 78% (653/836) 2024-08-20T22:10:13.1223161Z remote: Compressing objects: 79% (661/836) 2024-08-20T22:10:13.1226682Z remote: Compressing objects: 80% (669/836) 2024-08-20T22:10:13.1229651Z remote: Compressing objects: 81% (678/836) 2024-08-20T22:10:13.1232936Z remote: Compressing objects: 82% (686/836) 2024-08-20T22:10:13.1235917Z remote: Compressing objects: 83% (694/836) 2024-08-20T22:10:13.1238812Z remote: Compressing objects: 84% (703/836) 2024-08-20T22:10:13.1240242Z remote: Compressing objects: 85% (711/836) 2024-08-20T22:10:13.1243069Z remote: Compressing objects: 86% (719/836) 2024-08-20T22:10:13.1245212Z remote: Compressing objects: 87% (728/836) 2024-08-20T22:10:13.1249015Z remote: Compressing objects: 88% (736/836) 2024-08-20T22:10:13.1252161Z remote: Compressing objects: 89% (745/836) 2024-08-20T22:10:13.1254955Z remote: Compressing objects: 90% (753/836) 2024-08-20T22:10:13.1257167Z remote: Compressing objects: 91% (761/836) 2024-08-20T22:10:13.1257725Z remote: Compressing objects: 92% (770/836) 2024-08-20T22:10:13.1258271Z remote: Compressing objects: 93% (778/836) 2024-08-20T22:10:13.1261732Z remote: Compressing objects: 94% (786/836) 2024-08-20T22:10:13.1263323Z remote: Compressing objects: 95% (795/836) 2024-08-20T22:10:13.1265503Z remote: Compressing objects: 96% (803/836) 2024-08-20T22:10:13.1267238Z remote: Compressing objects: 97% (811/836) 2024-08-20T22:10:13.1269377Z remote: Compressing objects: 98% (820/836) 2024-08-20T22:10:13.1271557Z remote: Compressing objects: 99% (828/836) 2024-08-20T22:10:13.1272485Z remote: Compressing objects: 100% (836/836) 2024-08-20T22:10:13.1273109Z remote: Compressing objects: 100% (836/836), done. 2024-08-20T22:10:36.4807150Z remote: Total 1010229 (delta 1121), reused 1351 (delta 881), pack-reused 1008509 (from 1) 2024-08-20T22:10:59.8053231Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T22:10:59.8090084Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:10:59.8097044Z ##[endgroup] 2024-08-20T22:10:59.8099418Z ##[group]Determining the checkout info 2024-08-20T22:10:59.8100061Z ##[endgroup] 2024-08-20T22:10:59.8100569Z ##[group]Checking out the ref 2024-08-20T22:10:59.8101345Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:11:01.5228333Z ##[endgroup] 2024-08-20T22:11:01.5228962Z ##[group]Setting up auth for fetching submodules 2024-08-20T22:11:01.5230849Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:11:01.5295737Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T22:11:01.5335936Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T22:11:01.5375881Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T22:11:01.5414164Z ##[endgroup] 2024-08-20T22:11:01.5414714Z ##[group]Fetching submodules 2024-08-20T22:11:01.5417984Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T22:11:01.5792375Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T22:11:01.6157855Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T22:11:01.6159472Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T22:11:01.6163079Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T22:11:01.6166355Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T22:11:01.6170442Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T22:11:01.6173771Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T22:11:01.6177410Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T22:11:01.6181120Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T22:11:01.6184855Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T22:11:01.6188489Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T22:11:01.6192311Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T22:11:01.6196194Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T22:11:01.6200292Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T22:11:01.6204395Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T22:11:01.6208313Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T22:11:01.6214654Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:11:01.6222260Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T22:11:01.6226583Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T22:11:01.6230806Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T22:11:01.6235459Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T22:11:01.6240216Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T22:11:01.6244793Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T22:11:01.6249386Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T22:11:01.6254203Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T22:11:01.6258674Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T22:11:01.6263806Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T22:11:01.6268510Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T22:11:01.6273569Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T22:11:01.6278693Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T22:11:01.6283845Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T22:11:01.6288948Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T22:11:01.6294385Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T22:11:01.6299424Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T22:11:01.6308295Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T22:11:01.6347034Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T22:11:01.9565128Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T22:11:02.1554153Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T22:11:02.3648761Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T22:11:02.6097976Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T22:11:04.6180541Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T22:11:16.3349553Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T22:11:16.7507811Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T22:11:17.1932188Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T22:11:17.7924482Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T22:11:19.0477000Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T22:11:21.1366755Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T22:11:27.1758724Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T22:11:28.6389567Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T22:11:30.4241996Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T22:11:31.7148375Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T22:11:32.1234345Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T22:11:32.4450298Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T22:11:33.5872455Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T22:11:33.9990082Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T22:11:34.2558003Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T22:11:35.9601508Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T22:11:36.7007875Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T22:11:37.4001348Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T22:11:43.7665772Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T22:11:45.7314854Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T22:11:49.9844328Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T22:11:50.2100094Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T22:11:59.4163555Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T22:11:59.6046405Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T22:11:59.7951461Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T22:12:00.7715406Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T22:12:01.1240621Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T22:12:01.7903947Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T22:12:02.2403132Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T22:12:02.2550521Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T22:12:02.2664886Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T22:12:02.2968467Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T22:12:02.3394490Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T22:12:03.5047982Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T22:12:03.5326659Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T22:12:03.5850352Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T22:12:03.6969196Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T22:12:03.7373013Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T22:12:04.3580183Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T22:12:04.6437181Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T22:12:04.7377943Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T22:12:04.7400958Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:12:04.7404423Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:12:04.7407823Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:12:04.7411463Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:12:04.7416765Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:12:04.7448175Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T22:12:05.8165527Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T22:12:06.4384855Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T22:12:08.5305846Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T22:12:09.5976718Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T22:12:09.9521983Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T22:12:10.0625113Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T22:12:10.5727578Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T22:12:10.6426998Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T22:12:10.6577665Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T22:12:10.8042619Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T22:12:10.8496725Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T22:12:10.8947360Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T22:12:10.9262545Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T22:12:10.9780725Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:12:10.9941541Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T22:12:10.9960384Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T22:12:10.9990614Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T22:12:24.3083398Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T22:12:24.3306982Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T22:12:24.4302507Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T22:12:24.4327081Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:12:24.4330034Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:12:24.4333794Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:12:24.4365164Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T22:12:25.0071797Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T22:12:26.2263364Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T22:12:27.3900514Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T22:12:27.3922509Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:12:27.3925854Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:12:27.3929718Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:12:27.3933317Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:12:27.3937127Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:12:27.3941072Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:12:27.3944839Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:12:27.3948756Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:12:27.3982139Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T22:12:28.2840392Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T22:12:28.6445831Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T22:12:29.8281431Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T22:12:30.1021321Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T22:12:30.6151980Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T22:12:31.6652710Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T22:12:38.5778792Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T22:12:38.9812057Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T22:12:39.0038387Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T22:12:39.0469071Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T22:12:39.0632562Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T22:12:39.0652247Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:12:39.0684030Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T22:12:39.3737872Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T22:12:39.3962646Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T22:12:39.4444421Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T22:12:39.5660352Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T22:12:39.5863296Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T22:12:39.6319276Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T22:12:39.6972820Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T22:12:39.7417426Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T22:12:39.7715686Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T22:12:39.8966534Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T22:12:40.4093109Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T22:12:40.4131082Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T22:12:40.4133899Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T22:12:40.4167026Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T22:12:40.9236890Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T22:12:41.9070798Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T22:12:41.9484019Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T22:12:42.0394481Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T22:12:42.0416960Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:12:42.0420275Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:12:42.0423449Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:12:42.0427160Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:12:42.0430945Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:12:42.0434313Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:12:42.0438019Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:12:42.0441534Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:12:42.0473932Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T22:12:42.5476218Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T22:12:43.6080397Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T22:12:43.9156838Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T22:12:50.1587183Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T22:12:50.4152303Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T22:12:50.6351554Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T22:12:50.9392493Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T22:12:57.7967408Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T22:12:57.8442600Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T22:12:57.8630725Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T22:12:57.9877146Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T22:12:58.0044673Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T22:12:58.0246567Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T22:12:58.0453263Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T22:12:58.0471959Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:12:58.0475295Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:12:58.0505954Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T22:12:59.9863933Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T22:13:01.3408806Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T22:13:01.3948828Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:13:02.0645119Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T22:13:02.0783311Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T22:13:02.3884826Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T22:13:02.3912587Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:02.3915936Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:02.3947500Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T22:13:02.8663462Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T22:13:03.9572461Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T22:13:04.0388702Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T22:13:04.0508270Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T22:13:04.0662749Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T22:13:04.1106504Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T22:13:04.1446684Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T22:13:04.1936804Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T22:13:04.2280808Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T22:13:04.2302594Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:04.2305991Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:04.2309646Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:04.2314159Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:04.2345019Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T22:13:05.3159806Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T22:13:05.5452301Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T22:13:06.7356563Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T22:13:07.7251927Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T22:13:07.7446832Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T22:13:07.8233347Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T22:13:07.8576802Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T22:13:07.8596086Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:07.8626945Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T22:13:08.0627821Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T22:13:08.0674960Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T22:13:08.1039976Z Entering 'android/libs/fbjni' 2024-08-20T22:13:08.1091425Z Entering 'third_party/FP16' 2024-08-20T22:13:08.1143827Z Entering 'third_party/FXdiv' 2024-08-20T22:13:08.1195603Z Entering 'third_party/NNPACK' 2024-08-20T22:13:08.1248997Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:08.1300465Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:08.1370928Z Entering 'third_party/benchmark' 2024-08-20T22:13:08.1422885Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:08.1473712Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:08.1525818Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:08.1576985Z Entering 'third_party/cutlass' 2024-08-20T22:13:08.1635601Z Entering 'third_party/eigen' 2024-08-20T22:13:08.1688875Z Entering 'third_party/fbgemm' 2024-08-20T22:13:08.1739795Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:08.1790319Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:08.1841416Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:08.1898528Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:08.1949856Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:08.2004723Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:08.2062283Z Entering 'third_party/fmt' 2024-08-20T22:13:08.2112455Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:08.2163865Z Entering 'third_party/gloo' 2024-08-20T22:13:08.2215593Z Entering 'third_party/googletest' 2024-08-20T22:13:08.2267996Z Entering 'third_party/ideep' 2024-08-20T22:13:08.2316571Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:08.2374686Z Entering 'third_party/ittapi' 2024-08-20T22:13:08.2426098Z Entering 'third_party/kineto' 2024-08-20T22:13:08.2476407Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:08.2526357Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:08.2579165Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:08.2636653Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:08.2687796Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:08.2738537Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:08.2796095Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:08.2846758Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:08.2896236Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:08.2949474Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:08.3001850Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:08.3052317Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:08.3105722Z Entering 'third_party/mimalloc' 2024-08-20T22:13:08.3159376Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:08.3210978Z Entering 'third_party/nlohmann' 2024-08-20T22:13:08.3263824Z Entering 'third_party/onnx' 2024-08-20T22:13:08.3327707Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:08.3378242Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:08.3434541Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:08.3485511Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:08.3535428Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:08.3585008Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:08.3636573Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:08.3687924Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:08.3738760Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:08.3789004Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:08.3838391Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:08.3891201Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:08.3944887Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:08.4022206Z Entering 'third_party/pocketfft' 2024-08-20T22:13:08.4072926Z Entering 'third_party/protobuf' 2024-08-20T22:13:08.4127070Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:08.4176717Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:08.4230535Z Entering 'third_party/psimd' 2024-08-20T22:13:08.4281500Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:08.4335688Z Entering 'third_party/pybind11' 2024-08-20T22:13:08.4386559Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:08.4437584Z Entering 'third_party/sleef' 2024-08-20T22:13:08.4488676Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:08.4539842Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:08.4592105Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:08.4642072Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:08.4691783Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:08.4742229Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:08.4816309Z ##[endgroup] 2024-08-20T22:13:08.4819511Z ##[group]Persisting credentials for submodules 2024-08-20T22:13:08.4821932Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T22:13:08.5185633Z Entering 'android/libs/fbjni' 2024-08-20T22:13:08.5252963Z Entering 'third_party/FP16' 2024-08-20T22:13:08.5320510Z Entering 'third_party/FXdiv' 2024-08-20T22:13:08.5386922Z Entering 'third_party/NNPACK' 2024-08-20T22:13:08.5459837Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:08.5526920Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:08.5608547Z Entering 'third_party/benchmark' 2024-08-20T22:13:08.5675904Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:08.5742418Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:08.5810533Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:08.5879056Z Entering 'third_party/cutlass' 2024-08-20T22:13:08.5953740Z Entering 'third_party/eigen' 2024-08-20T22:13:08.6022676Z Entering 'third_party/fbgemm' 2024-08-20T22:13:08.6087823Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:08.6158439Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:08.6225426Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:08.6298452Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:08.6365736Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:08.6434628Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:08.6505077Z Entering 'third_party/fmt' 2024-08-20T22:13:08.6572787Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:08.6640625Z Entering 'third_party/gloo' 2024-08-20T22:13:08.6707323Z Entering 'third_party/googletest' 2024-08-20T22:13:08.6774229Z Entering 'third_party/ideep' 2024-08-20T22:13:08.6838842Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:08.6912068Z Entering 'third_party/ittapi' 2024-08-20T22:13:08.6978531Z Entering 'third_party/kineto' 2024-08-20T22:13:08.7043211Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:08.7108426Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:08.7178127Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:08.7244802Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:08.7311547Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:08.7375994Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:08.7446742Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:08.7513746Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:08.7582937Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:08.7650344Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:08.7720294Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:08.7786653Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:08.7856056Z Entering 'third_party/mimalloc' 2024-08-20T22:13:08.7923383Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:08.7990369Z Entering 'third_party/nlohmann' 2024-08-20T22:13:08.8062723Z Entering 'third_party/onnx' 2024-08-20T22:13:08.8141807Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:08.8209272Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:08.8283904Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:08.8351211Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:08.8418333Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:08.8484548Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:08.8551326Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:08.8618210Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:08.8683396Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:08.8751170Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:08.8815494Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:08.8883534Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:08.8952704Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:08.9041029Z Entering 'third_party/pocketfft' 2024-08-20T22:13:08.9107027Z Entering 'third_party/protobuf' 2024-08-20T22:13:08.9176172Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:08.9244130Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:08.9315282Z Entering 'third_party/psimd' 2024-08-20T22:13:08.9381975Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:08.9449295Z Entering 'third_party/pybind11' 2024-08-20T22:13:08.9516860Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:08.9582768Z Entering 'third_party/sleef' 2024-08-20T22:13:08.9650377Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:08.9716955Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:08.9783666Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:08.9850405Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:08.9917753Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:08.9986455Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:09.0078638Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T22:13:09.0442787Z Entering 'android/libs/fbjni' 2024-08-20T22:13:09.0503593Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T22:13:09.0525413Z Entering 'third_party/FP16' 2024-08-20T22:13:09.0586317Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T22:13:09.0608715Z Entering 'third_party/FXdiv' 2024-08-20T22:13:09.0670825Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T22:13:09.0691556Z Entering 'third_party/NNPACK' 2024-08-20T22:13:09.0753339Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T22:13:09.0774215Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:09.0835216Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T22:13:09.0856202Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:09.0917086Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T22:13:09.0954356Z Entering 'third_party/benchmark' 2024-08-20T22:13:09.1015152Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:09.1035643Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:09.1097715Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T22:13:09.1118981Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:09.1180003Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:13:09.1201824Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:09.1263379Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T22:13:09.1284062Z Entering 'third_party/cutlass' 2024-08-20T22:13:09.1345603Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:13:09.1374395Z Entering 'third_party/eigen' 2024-08-20T22:13:09.1435168Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T22:13:09.1458342Z Entering 'third_party/fbgemm' 2024-08-20T22:13:09.1520926Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T22:13:09.1540312Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:09.1601196Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T22:13:09.1622322Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:09.1684807Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:13:09.1704623Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:09.1767089Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:13:09.1793337Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:09.1855207Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:09.1875766Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:09.1938637Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T22:13:09.1960835Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:09.2026899Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T22:13:09.2052314Z Entering 'third_party/fmt' 2024-08-20T22:13:09.2114084Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:13:09.2135097Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:09.2196952Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T22:13:09.2218852Z Entering 'third_party/gloo' 2024-08-20T22:13:09.2279590Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T22:13:09.2299849Z Entering 'third_party/googletest' 2024-08-20T22:13:09.2360810Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:09.2382063Z Entering 'third_party/ideep' 2024-08-20T22:13:09.2444834Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T22:13:09.2463596Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:09.2525307Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T22:13:09.2555156Z Entering 'third_party/ittapi' 2024-08-20T22:13:09.2615805Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T22:13:09.2636980Z Entering 'third_party/kineto' 2024-08-20T22:13:09.2707460Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T22:13:09.2728207Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:09.2790160Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T22:13:09.2809004Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:09.2871282Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T22:13:09.2892857Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:09.2955658Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T22:13:09.2978381Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:09.3040475Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:13:09.3060695Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:09.3123377Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T22:13:09.3141552Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:09.3206176Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T22:13:09.3228449Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:09.3289559Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T22:13:09.3312980Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:09.3374562Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:09.3395209Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:09.3461667Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T22:13:09.3482811Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:09.3545044Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T22:13:09.3567771Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:09.3629223Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T22:13:09.3649111Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:09.3711923Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T22:13:09.3733268Z Entering 'third_party/mimalloc' 2024-08-20T22:13:09.3794168Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T22:13:09.3816981Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:09.3878184Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T22:13:09.3899039Z Entering 'third_party/nlohmann' 2024-08-20T22:13:09.3962059Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T22:13:09.3983991Z Entering 'third_party/onnx' 2024-08-20T22:13:09.4045825Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T22:13:09.4079949Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:09.4142826Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:09.4164032Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:09.4228890Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:09.4255717Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:09.4319167Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T22:13:09.4340823Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:09.4402611Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:09.4426056Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:09.4487728Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:09.4507794Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:09.4568999Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T22:13:09.4588841Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:09.4652506Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T22:13:09.4674057Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:09.4736557Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T22:13:09.4757505Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:09.4819624Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T22:13:09.4840452Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:09.4902740Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T22:13:09.4923044Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:09.4984369Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T22:13:09.5011891Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:09.5072940Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T22:13:09.5095848Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:09.5157302Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T22:13:09.5199844Z Entering 'third_party/pocketfft' 2024-08-20T22:13:09.5260633Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T22:13:09.5281432Z Entering 'third_party/protobuf' 2024-08-20T22:13:09.5344639Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T22:13:09.5366571Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:09.5430273Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:09.5449296Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:09.5510551Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:09.5535015Z Entering 'third_party/psimd' 2024-08-20T22:13:09.5598505Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T22:13:09.5620171Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:09.5681247Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T22:13:09.5701882Z Entering 'third_party/pybind11' 2024-08-20T22:13:09.5762806Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:09.5784341Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:09.5845571Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T22:13:09.5866541Z Entering 'third_party/sleef' 2024-08-20T22:13:09.5932481Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T22:13:09.5953481Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:09.6017203Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T22:13:09.6037272Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:09.6098859Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:09.6119861Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:09.6180647Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T22:13:09.6200997Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:09.6263207Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T22:13:09.6284064Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:09.6345965Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:09.6364588Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:09.6426587Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T22:13:09.7494186Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T22:13:09.7856542Z Entering 'android/libs/fbjni' 2024-08-20T22:13:09.7909243Z Entering 'third_party/FP16' 2024-08-20T22:13:09.7962190Z Entering 'third_party/FXdiv' 2024-08-20T22:13:09.8013855Z Entering 'third_party/NNPACK' 2024-08-20T22:13:09.8065756Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:09.8118764Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:09.8186484Z Entering 'third_party/benchmark' 2024-08-20T22:13:09.8239010Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:09.8290787Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:09.8343682Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:09.8395243Z Entering 'third_party/cutlass' 2024-08-20T22:13:09.8454949Z Entering 'third_party/eigen' 2024-08-20T22:13:09.8508644Z Entering 'third_party/fbgemm' 2024-08-20T22:13:09.8561884Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:09.8613486Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:09.8663947Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:09.8721676Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:09.8771440Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:09.8825051Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:09.8880207Z Entering 'third_party/fmt' 2024-08-20T22:13:09.8935236Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:09.8988224Z Entering 'third_party/gloo' 2024-08-20T22:13:09.9041295Z Entering 'third_party/googletest' 2024-08-20T22:13:09.9093391Z Entering 'third_party/ideep' 2024-08-20T22:13:09.9144252Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:09.9201395Z Entering 'third_party/ittapi' 2024-08-20T22:13:09.9252880Z Entering 'third_party/kineto' 2024-08-20T22:13:09.9303074Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:09.9352450Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:09.9403867Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:09.9454692Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:09.9504309Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:09.9554037Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:09.9607923Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:09.9657803Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:09.9707484Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:09.9760108Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:09.9811867Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:09.9862799Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:09.9915620Z Entering 'third_party/mimalloc' 2024-08-20T22:13:09.9968278Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:10.0021457Z Entering 'third_party/nlohmann' 2024-08-20T22:13:10.0074642Z Entering 'third_party/onnx' 2024-08-20T22:13:10.0142534Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:10.0193620Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:10.0252911Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:10.0305121Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:10.0358377Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:10.0413081Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:10.0462990Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:10.0514788Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:10.0564609Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:10.0614469Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:10.0663433Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:10.0716484Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:10.0770114Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:10.0841929Z Entering 'third_party/pocketfft' 2024-08-20T22:13:10.0893704Z Entering 'third_party/protobuf' 2024-08-20T22:13:10.0949255Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:10.0997738Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:10.1056932Z Entering 'third_party/psimd' 2024-08-20T22:13:10.1108929Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:10.1161292Z Entering 'third_party/pybind11' 2024-08-20T22:13:10.1213742Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:10.1265632Z Entering 'third_party/sleef' 2024-08-20T22:13:10.1317804Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:10.1369072Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:10.1419525Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:10.1470111Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:10.1519653Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:10.1570609Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:10.1647592Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T22:13:10.2007426Z Entering 'android/libs/fbjni' 2024-08-20T22:13:10.2058684Z Entering 'third_party/FP16' 2024-08-20T22:13:10.2109523Z Entering 'third_party/FXdiv' 2024-08-20T22:13:10.2160340Z Entering 'third_party/NNPACK' 2024-08-20T22:13:10.2212185Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:10.2265039Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:10.2336815Z Entering 'third_party/benchmark' 2024-08-20T22:13:10.2389134Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:10.2442384Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:10.2493992Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:10.2547844Z Entering 'third_party/cutlass' 2024-08-20T22:13:10.2606438Z Entering 'third_party/eigen' 2024-08-20T22:13:10.2662471Z Entering 'third_party/fbgemm' 2024-08-20T22:13:10.2715457Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:10.2766356Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:10.2822384Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:10.2878710Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:10.2928855Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:10.2981423Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:10.3036719Z Entering 'third_party/fmt' 2024-08-20T22:13:10.3088689Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:10.3141416Z Entering 'third_party/gloo' 2024-08-20T22:13:10.3194400Z Entering 'third_party/googletest' 2024-08-20T22:13:10.3247835Z Entering 'third_party/ideep' 2024-08-20T22:13:10.3298495Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:10.3362826Z Entering 'third_party/ittapi' 2024-08-20T22:13:10.3415507Z Entering 'third_party/kineto' 2024-08-20T22:13:10.3466727Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:10.3521806Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:10.3573753Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:10.3625082Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:10.3675790Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:10.3724929Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:10.3780178Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:10.3830467Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:10.3881029Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:10.3932536Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:10.3985939Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:10.4037343Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:10.4095180Z Entering 'third_party/mimalloc' 2024-08-20T22:13:10.4148822Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:10.4200729Z Entering 'third_party/nlohmann' 2024-08-20T22:13:10.4254930Z Entering 'third_party/onnx' 2024-08-20T22:13:10.4319726Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:10.4370866Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:10.4427307Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:10.4480447Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:10.4529349Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:10.4578978Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:10.4629956Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:10.4682191Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:10.4731051Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:10.4780595Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:10.4829168Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:10.4882576Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:10.4935530Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:10.5006277Z Entering 'third_party/pocketfft' 2024-08-20T22:13:10.5059644Z Entering 'third_party/protobuf' 2024-08-20T22:13:10.5114618Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:10.5166185Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:10.5221147Z Entering 'third_party/psimd' 2024-08-20T22:13:10.5273769Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:10.5327154Z Entering 'third_party/pybind11' 2024-08-20T22:13:10.5379042Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:10.5431955Z Entering 'third_party/sleef' 2024-08-20T22:13:10.5484644Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:10.5536791Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:10.5585738Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:10.5634642Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:10.5684631Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:10.5732764Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:10.5806192Z ##[endgroup] 2024-08-20T22:13:10.5857562Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T22:13:10.5894277Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T22:13:10.6118826Z Prepare all required actions 2024-08-20T22:13:10.6119309Z Getting action download info 2024-08-20T22:13:10.7370510Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T22:13:10.7370946Z env: 2024-08-20T22:13:10.7371235Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:10.7371591Z ##[endgroup] 2024-08-20T22:13:10.7431475Z ##[group]Run set -euo pipefail 2024-08-20T22:13:10.7431932Z set -euo pipefail 2024-08-20T22:13:10.7432354Z function get_ec2_metadata() { 2024-08-20T22:13:10.7432913Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T22:13:10.7433816Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T22:13:10.7434599Z  category=$1 2024-08-20T22:13:10.7435170Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T22:13:10.7435835Z  runner_name_str=i-0573eecbda81770a3 2024-08-20T22:13:10.7436360Z  if [[ -f /.inarc ]]; then 2024-08-20T22:13:10.7436878Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T22:13:10.7437469Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T22:13:10.7438305Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T22:13:10.7438933Z  else 2024-08-20T22:13:10.7439447Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T22:13:10.7440050Z  fi 2024-08-20T22:13:10.7440371Z } 2024-08-20T22:13:10.7440756Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T22:13:10.7441362Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T22:13:10.7442037Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T22:13:10.7442635Z echo "system info $(uname -a)" 2024-08-20T22:13:10.7453769Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:10.7454278Z env: 2024-08-20T22:13:10.7454580Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:10.7454945Z ##[endgroup] 2024-08-20T22:13:10.7564788Z ami-id: ami-06c68f701d8090592 2024-08-20T22:13:10.7627688Z instance-id: i-0573eecbda81770a3 2024-08-20T22:13:10.7684512Z instance-type: g5.4xlarge 2024-08-20T22:13:10.7698287Z system info Linux ip-10-0-36-93.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T22:13:10.7723817Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:13:10.7724743Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:13:10.7734536Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:10.7735217Z env: 2024-08-20T22:13:10.7735523Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:10.7735895Z ##[endgroup] 2024-08-20T22:13:10.7843172Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T22:13:10.7843790Z if systemctl is-active --quiet docker; then 2024-08-20T22:13:10.7844338Z  echo "Docker daemon is running..."; 2024-08-20T22:13:10.7844804Z else 2024-08-20T22:13:10.7845305Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T22:13:10.7845896Z fi 2024-08-20T22:13:10.7854621Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:10.7855122Z env: 2024-08-20T22:13:10.7855420Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:10.7855792Z ##[endgroup] 2024-08-20T22:13:10.7951860Z Docker daemon is running... 2024-08-20T22:13:10.7998726Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:13:10.7999291Z with: 2024-08-20T22:13:10.7999594Z shell: bash 2024-08-20T22:13:10.7999903Z timeout_minutes: 5 2024-08-20T22:13:10.8000242Z max_attempts: 3 2024-08-20T22:13:10.8000571Z retry_wait_seconds: 30 2024-08-20T22:13:10.8003645Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T22:13:10.8006562Z polling_interval_seconds: 1 2024-08-20T22:13:10.8006940Z warning_on_retry: true 2024-08-20T22:13:10.8007292Z continue_on_error: false 2024-08-20T22:13:10.8007695Z env: 2024-08-20T22:13:10.8007987Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:10.8008356Z AWS_RETRY_MODE: standard 2024-08-20T22:13:10.8008715Z AWS_MAX_ATTEMPTS: 5 2024-08-20T22:13:10.8009069Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T22:13:10.8009460Z ##[endgroup] 2024-08-20T22:13:11.9480588Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:11.9481641Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:11.9482753Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:11.9483428Z 2024-08-20T22:13:11.9483590Z Login Succeeded 2024-08-20T22:13:12.8566015Z Command completed after 1 attempt(s). 2024-08-20T22:13:12.8637011Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:12.8637725Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:12.8638498Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:12.8648950Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:12.8649452Z env: 2024-08-20T22:13:12.8649743Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:12.8650107Z ##[endgroup] 2024-08-20T22:13:12.8747535Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:12.8748308Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:12.8748901Z # shellcheck disable=SC2046 2024-08-20T22:13:12.8749389Z docker stop $(docker ps -q) || true 2024-08-20T22:13:12.8749887Z # Prune all of the docker images 2024-08-20T22:13:12.8750359Z docker system prune -af 2024-08-20T22:13:12.8759393Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:12.8759902Z env: 2024-08-20T22:13:12.8760195Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:12.8760558Z ##[endgroup] 2024-08-20T22:13:12.9084000Z "docker stop" requires at least 1 argument. 2024-08-20T22:13:12.9084912Z See 'docker stop --help'. 2024-08-20T22:13:12.9085618Z 2024-08-20T22:13:12.9085917Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T22:13:12.9086401Z 2024-08-20T22:13:12.9086615Z Stop one or more running containers 2024-08-20T22:13:12.9292865Z Total reclaimed space: 0B 2024-08-20T22:13:12.9340426Z ##[group]Run set +e 2024-08-20T22:13:12.9340785Z set +e 2024-08-20T22:13:12.9341101Z set -x 2024-08-20T22:13:12.9341410Z  2024-08-20T22:13:12.9341742Z PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:12.9342523Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T22:13:12.9343583Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T22:13:12.9344333Z # one is returned at random 2024-08-20T22:13:12.9344886Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T22:13:12.9345402Z  2024-08-20T22:13:12.9345750Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:12.9346376Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T22:13:12.9347327Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T22:13:12.9347885Z  2024-08-20T22:13:12.9348222Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:12.9348783Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T22:13:12.9349302Z  exit 1 2024-08-20T22:13:12.9349625Z  fi 2024-08-20T22:13:12.9349928Z fi 2024-08-20T22:13:12.9350222Z  2024-08-20T22:13:12.9350596Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T22:13:12.9351117Z  # Clean up any old records first 2024-08-20T22:13:12.9351638Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T22:13:12.9352111Z fi 2024-08-20T22:13:12.9352404Z  2024-08-20T22:13:12.9352854Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T22:13:12.9353416Z cat /etc/hosts 2024-08-20T22:13:12.9362520Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:12.9363023Z env: 2024-08-20T22:13:12.9363328Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:12.9363690Z ##[endgroup] 2024-08-20T22:13:12.9395597Z + PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:12.9401975Z ++ dig -4 +short download.pytorch.org 2024-08-20T22:13:12.9402652Z ++ tail -n1 2024-08-20T22:13:12.9896931Z + RESOLVED_IP=18.160.10.28 2024-08-20T22:13:12.9897531Z + '[' -z 18.160.10.28 ']' 2024-08-20T22:13:12.9898097Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T22:13:12.9915264Z + echo '18.160.10.28 download.pytorch.org' 2024-08-20T22:13:12.9915784Z + sudo tee -a /etc/hosts 2024-08-20T22:13:13.2283089Z 18.160.10.28 download.pytorch.org 2024-08-20T22:13:13.2306499Z + cat /etc/hosts 2024-08-20T22:13:13.2317768Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T22:13:13.2329910Z ::1 localhost6 localhost6.localdomain6 2024-08-20T22:13:13.2330661Z 18.160.10.28 download.pytorch.org 2024-08-20T22:13:13.2483642Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T22:13:13.2484247Z with: 2024-08-20T22:13:13.2485373Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:13.2486607Z docker-build-dir: .ci/docker 2024-08-20T22:13:13.2487007Z working-directory: . 2024-08-20T22:13:13.2487492Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:13.2488030Z force-push: false 2024-08-20T22:13:13.2488357Z env: 2024-08-20T22:13:13.2488654Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:13.2489023Z ##[endgroup] 2024-08-20T22:13:13.2508726Z ##[group]Run set -ex 2024-08-20T22:13:13.2509097Z set -ex 2024-08-20T22:13:13.2509403Z  2024-08-20T22:13:13.2510495Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T22:13:13.2511650Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T22:13:13.2512456Z # job could then download the pre-built image as usual 2024-08-20T22:13:13.2513209Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T22:13:13.2513900Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:13.2514549Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:13.2515121Z  2024-08-20T22:13:13.2515640Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T22:13:13.2516283Z  exit 0 2024-08-20T22:13:13.2516611Z else 2024-08-20T22:13:13.2516996Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:13.2517470Z fi 2024-08-20T22:13:13.2517779Z  2024-08-20T22:13:13.2518340Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T22:13:13.2519194Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T22:13:13.2519980Z  # use it as it is, but first let's extract the tag 2024-08-20T22:13:13.2520688Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T22:13:13.2521423Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:13.2522137Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:13.2522707Z else 2024-08-20T22:13:13.2523162Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T22:13:13.2523834Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:13.2524732Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:13.2525485Z fi 2024-08-20T22:13:13.2537242Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:13.2537760Z env: 2024-08-20T22:13:13.2538055Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:13.2538427Z REPO_NAME: pytorch 2024-08-20T22:13:13.2539560Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:13.2540811Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:13.2541321Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:13.2541850Z ##[endgroup] 2024-08-20T22:13:13.2572086Z + [[ ! -d .ci/docker ]] 2024-08-20T22:13:13.2572639Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T22:13:13.2573131Z + echo skip=false 2024-08-20T22:13:13.2575178Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T22:13:13.2579826Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:13.2581278Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T22:13:13.2607227Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:13.2608160Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:13.2610285Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:13.2643469Z ##[group]Run set +e 2024-08-20T22:13:13.2644020Z set +e 2024-08-20T22:13:13.2644411Z set -x 2024-08-20T22:13:13.2644811Z  2024-08-20T22:13:13.2645414Z login() { 2024-08-20T22:13:13.2646237Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:13.2647090Z } 2024-08-20T22:13:13.2647561Z  2024-08-20T22:13:13.2647952Z retry () { 2024-08-20T22:13:13.2648468Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:13.2649042Z } 2024-08-20T22:13:13.2649525Z  2024-08-20T22:13:13.2650001Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:13.2650538Z  2024-08-20T22:13:13.2651111Z # Check if image already exists, if it does then skip building it 2024-08-20T22:13:13.2651922Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T22:13:13.2652557Z  exit 0 2024-08-20T22:13:13.2652924Z fi 2024-08-20T22:13:13.2653325Z  2024-08-20T22:13:13.2653965Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T22:13:13.2654918Z # be empty. The default action would be to continue rebuild the image 2024-08-20T22:13:13.2655716Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T22:13:13.2656595Z  # if we're on the base branch then use the parent commit 2024-08-20T22:13:13.2657298Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T22:13:13.2657819Z else 2024-08-20T22:13:13.2658394Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T22:13:13.2659192Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T22:13:13.2659791Z fi 2024-08-20T22:13:13.2660217Z  2024-08-20T22:13:13.2660625Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T22:13:13.2661219Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:13.2661829Z  2024-08-20T22:13:13.2662566Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T22:13:13.2663388Z  exit 0 2024-08-20T22:13:13.2663926Z fi 2024-08-20T22:13:13.2664295Z  2024-08-20T22:13:13.2664879Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T22:13:13.2665915Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T22:13:13.2666791Z  exit 1 2024-08-20T22:13:13.2667236Z fi 2024-08-20T22:13:13.2667637Z  2024-08-20T22:13:13.2668183Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T22:13:13.2669350Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T22:13:13.2670293Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T22:13:13.2671332Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T22:13:13.2672651Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T22:13:13.2673504Z fi 2024-08-20T22:13:13.2673918Z  2024-08-20T22:13:13.2674430Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:13.2683253Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:13.2683889Z env: 2024-08-20T22:13:13.2684274Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:13.2684781Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:13.2685426Z BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:13:13.2686924Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:13.2688530Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:13.2689276Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:13.2690063Z ##[endgroup] 2024-08-20T22:13:13.2724986Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:13.2725943Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:13.2727184Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:13.2729467Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:13.7992532Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:13.7993411Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:13.7994361Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:13.7994989Z 2024-08-20T22:13:13.7996407Z Login Succeeded 2024-08-20T22:13:13.8020996Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:14.0150872Z { 2024-08-20T22:13:14.0152018Z "schemaVersion": 2, 2024-08-20T22:13:14.0152748Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T22:13:14.0153446Z "config": { 2024-08-20T22:13:14.0154203Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T22:13:14.0155716Z "size": 48711, 2024-08-20T22:13:14.0156402Z "digest": "sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb" 2024-08-20T22:13:14.0157502Z }, 2024-08-20T22:13:14.0158049Z "layers": [ 2024-08-20T22:13:14.0158693Z { 2024-08-20T22:13:14.0159864Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0160569Z "size": 28580681, 2024-08-20T22:13:14.0161290Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-20T22:13:14.0162285Z }, 2024-08-20T22:13:14.0162716Z { 2024-08-20T22:13:14.0163776Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0165303Z "size": 7943451, 2024-08-20T22:13:14.0166371Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-08-20T22:13:14.0167307Z }, 2024-08-20T22:13:14.0167832Z { 2024-08-20T22:13:14.0168456Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0169115Z "size": 55728572, 2024-08-20T22:13:14.0169863Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-08-20T22:13:14.0170576Z }, 2024-08-20T22:13:14.0170910Z { 2024-08-20T22:13:14.0171496Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0172152Z "size": 186, 2024-08-20T22:13:14.0172864Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-08-20T22:13:14.0173626Z }, 2024-08-20T22:13:14.0173983Z { 2024-08-20T22:13:14.0174534Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0175194Z "size": 6886, 2024-08-20T22:13:14.0175845Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-08-20T22:13:14.0176575Z }, 2024-08-20T22:13:14.0177177Z { 2024-08-20T22:13:14.0177785Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0178439Z "size": 1329001756, 2024-08-20T22:13:14.0179163Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-08-20T22:13:14.0180175Z }, 2024-08-20T22:13:14.0180482Z { 2024-08-20T22:13:14.0181084Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0181728Z "size": 62501, 2024-08-20T22:13:14.0182373Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-08-20T22:13:14.0183444Z }, 2024-08-20T22:13:14.0183916Z { 2024-08-20T22:13:14.0184715Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0185524Z "size": 1684, 2024-08-20T22:13:14.0186162Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-08-20T22:13:14.0187123Z }, 2024-08-20T22:13:14.0187543Z { 2024-08-20T22:13:14.0188043Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0188765Z "size": 1523, 2024-08-20T22:13:14.0189827Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-08-20T22:13:14.0190636Z }, 2024-08-20T22:13:14.0190973Z { 2024-08-20T22:13:14.0191498Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0192223Z "size": 2528295403, 2024-08-20T22:13:14.0192885Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-08-20T22:13:14.0193561Z }, 2024-08-20T22:13:14.0194053Z { 2024-08-20T22:13:14.0194592Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0195281Z "size": 86016, 2024-08-20T22:13:14.0196418Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-08-20T22:13:14.0197446Z }, 2024-08-20T22:13:14.0197784Z { 2024-08-20T22:13:14.0198491Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0199138Z "size": 1822, 2024-08-20T22:13:14.0199797Z "digest": "sha256:ac23c5fcc1ef310e9cd0d138b180281188e20a3e28c51cd14751622e3c7d92ea" 2024-08-20T22:13:14.0200582Z }, 2024-08-20T22:13:14.0200908Z { 2024-08-20T22:13:14.0201537Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0202246Z "size": 246761563, 2024-08-20T22:13:14.0202887Z "digest": "sha256:ca426ffc9d2bd051ad1ff5c4f02fd5d9be161e70953cc37882c815080c708f4d" 2024-08-20T22:13:14.0203701Z }, 2024-08-20T22:13:14.0204103Z { 2024-08-20T22:13:14.0204634Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0205264Z "size": 544, 2024-08-20T22:13:14.0205941Z "digest": "sha256:4373aa743caf6660c671295b837048c8350a2165f2336dc3b7b6c96462579e50" 2024-08-20T22:13:14.0206737Z }, 2024-08-20T22:13:14.0207092Z { 2024-08-20T22:13:14.0207642Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0208317Z "size": 1258, 2024-08-20T22:13:14.0208990Z "digest": "sha256:75b2abb16c685854c017d0cf75c626526c956734c656d9e78fdd4520317e9b3c" 2024-08-20T22:13:14.0209730Z }, 2024-08-20T22:13:14.0210450Z { 2024-08-20T22:13:14.0210971Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0211666Z "size": 484, 2024-08-20T22:13:14.0212259Z "digest": "sha256:ef1e3ef1e85f0124270b8506721219150e230184181f8207d64463bf5c67e656" 2024-08-20T22:13:14.0212942Z }, 2024-08-20T22:13:14.0213347Z { 2024-08-20T22:13:14.0213869Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0214478Z "size": 91711766, 2024-08-20T22:13:14.0215194Z "digest": "sha256:0bf61114d37b2d09065aa6852b1e78a32580a1ff011dd5d23f03b0f5c4e710af" 2024-08-20T22:13:14.0215886Z }, 2024-08-20T22:13:14.0216284Z { 2024-08-20T22:13:14.0216862Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0217581Z "size": 3390, 2024-08-20T22:13:14.0218467Z "digest": "sha256:2df849f72ba4258df10c9c45ace21a66d93b73196e78e96f12c13098822f60e2" 2024-08-20T22:13:14.0219188Z }, 2024-08-20T22:13:14.0219523Z { 2024-08-20T22:13:14.0220172Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0220874Z "size": 1909, 2024-08-20T22:13:14.0221467Z "digest": "sha256:34c47d63c5de27418739aa048b4dd781c3006d65691e9b903c95304d0509d89c" 2024-08-20T22:13:14.0222235Z }, 2024-08-20T22:13:14.0222574Z { 2024-08-20T22:13:14.0223112Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0223757Z "size": 701, 2024-08-20T22:13:14.0224461Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:14.0225182Z }, 2024-08-20T22:13:14.0225577Z { 2024-08-20T22:13:14.0226068Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0226863Z "size": 2864254879, 2024-08-20T22:13:14.0227582Z "digest": "sha256:6efb96fc87ea1bca54876d85331e5585e28f325a1d60ee1aef7811eabb74f933" 2024-08-20T22:13:14.0228253Z }, 2024-08-20T22:13:14.0228618Z { 2024-08-20T22:13:14.0229180Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0229864Z "size": 380, 2024-08-20T22:13:14.0230473Z "digest": "sha256:b119c4fc6c58d9e72744201ff4850b34184a94eb3566031b7d36e87ecce950a6" 2024-08-20T22:13:14.0231281Z }, 2024-08-20T22:13:14.0231643Z { 2024-08-20T22:13:14.0232166Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0232996Z + exit 0 2024-08-20T22:13:14.0233362Z "size": 12900, 2024-08-20T22:13:14.0234053Z "digest": "sha256:33b639226eeefa41d7bb26a8ecc20d271ceef45571dabfd80aaee3634926381b" 2024-08-20T22:13:14.0234802Z }, 2024-08-20T22:13:14.0235141Z { 2024-08-20T22:13:14.0235727Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0236339Z "size": 803, 2024-08-20T22:13:14.0236953Z "digest": "sha256:d3547bd43146930aa830d54562c98aa678e355cf5f84527e9717eb72fffe8ca3" 2024-08-20T22:13:14.0237771Z }, 2024-08-20T22:13:14.0238167Z { 2024-08-20T22:13:14.0238677Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0239393Z "size": 106, 2024-08-20T22:13:14.0240006Z "digest": "sha256:5645520256fa6d7f2270608eaed53849a131d03fbf4c5d4472c9cfde1a439e47" 2024-08-20T22:13:14.0240655Z }, 2024-08-20T22:13:14.0241063Z { 2024-08-20T22:13:14.0241583Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0242242Z "size": 503, 2024-08-20T22:13:14.0242881Z "digest": "sha256:829ed7d91e3818d0e11834c71551117c7aeec0a51d798e93d641faa0c02582cb" 2024-08-20T22:13:14.0243565Z }, 2024-08-20T22:13:14.0244000Z { 2024-08-20T22:13:14.0244528Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0245168Z "size": 121477267, 2024-08-20T22:13:14.0245830Z "digest": "sha256:385d0421d67c533d628985c453a186028a61f6ad4209ae519ec3d4ecea661d3e" 2024-08-20T22:13:14.0246567Z }, 2024-08-20T22:13:14.0246878Z { 2024-08-20T22:13:14.0247428Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0248113Z "size": 109, 2024-08-20T22:13:14.0248757Z "digest": "sha256:7cbfce999d76851e07dd71056a6d852629d789ccfd46be44cc405b451fb8bb58" 2024-08-20T22:13:14.0249431Z }, 2024-08-20T22:13:14.0249837Z { 2024-08-20T22:13:14.0250369Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0251118Z "size": 490, 2024-08-20T22:13:14.0251693Z "digest": "sha256:6658e558ebe496f0d628947534b0ea1e44808f84f2dfd8a9a704e8aa5329d791" 2024-08-20T22:13:14.0252398Z }, 2024-08-20T22:13:14.0252775Z { 2024-08-20T22:13:14.0253251Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0253909Z "size": 297, 2024-08-20T22:13:14.0254560Z "digest": "sha256:33444a5a7e01a3010fd1cc16b32a66892a08600d7f700747d23faf3c36624f3a" 2024-08-20T22:13:14.0255365Z }, 2024-08-20T22:13:14.0255667Z { 2024-08-20T22:13:14.0256231Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0256885Z "size": 103, 2024-08-20T22:13:14.0257574Z "digest": "sha256:7c47122f4e09b47e891b4ae9e499a35a4eb46faacae87d0e946615adda6e0706" 2024-08-20T22:13:14.0258287Z }, 2024-08-20T22:13:14.0258647Z { 2024-08-20T22:13:14.0259157Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0259873Z "size": 1473, 2024-08-20T22:13:14.0260521Z "digest": "sha256:1811ff4d3556ac255604f346d879528aff2aadd7652c2d39baf4fe6eaa60aec3" 2024-08-20T22:13:14.0261211Z }, 2024-08-20T22:13:14.0261623Z { 2024-08-20T22:13:14.0262105Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0262734Z "size": 594729516, 2024-08-20T22:13:14.0263540Z "digest": "sha256:7754ee068b33f760d9c320c26a1a3935247ace0c8c463a976105c1586a43fb1c" 2024-08-20T22:13:14.0264289Z }, 2024-08-20T22:13:14.0264623Z { 2024-08-20T22:13:14.0265190Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0265823Z "size": 164, 2024-08-20T22:13:14.0266462Z "digest": "sha256:0479fbac3361713c075bf788e778117917dcacdd1ff71ed79a789dd2016a1714" 2024-08-20T22:13:14.0267166Z }, 2024-08-20T22:13:14.0267505Z { 2024-08-20T22:13:14.0268040Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0268682Z "size": 564, 2024-08-20T22:13:14.0269294Z "digest": "sha256:f1d6874dee9c34d154278d2b6935d04447f07fa89d25d4a09398234040d8a7e8" 2024-08-20T22:13:14.0269998Z }, 2024-08-20T22:13:14.0270386Z { 2024-08-20T22:13:14.0270963Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0271673Z "size": 35874203, 2024-08-20T22:13:14.0272372Z "digest": "sha256:afc429e87b593a64f3e0a1be51e09e966ea1aa71e6e8a0bec4edc9f2f7bc1c05" 2024-08-20T22:13:14.0273101Z }, 2024-08-20T22:13:14.0273399Z { 2024-08-20T22:13:14.0273987Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0274617Z "size": 104, 2024-08-20T22:13:14.0275193Z "digest": "sha256:019620471c2592cd22437fe8ed1a94c784114cf7445c5c2ab36dd0bae727fc66" 2024-08-20T22:13:14.0275959Z }, 2024-08-20T22:13:14.0276300Z { 2024-08-20T22:13:14.0276859Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0277548Z "size": 425, 2024-08-20T22:13:14.0278277Z "digest": "sha256:27aeb1b2cd5e28d8f0e54fc1090baf88abc29ebf5fce088b493fcae99c9550d1" 2024-08-20T22:13:14.0279060Z }, 2024-08-20T22:13:14.0279433Z { 2024-08-20T22:13:14.0279943Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0280646Z "size": 20262183, 2024-08-20T22:13:14.0281280Z "digest": "sha256:829c0e19f4b7797e9120a86172fabfd710f9b3971ffd714a616931a3fadc7192" 2024-08-20T22:13:14.0281980Z }, 2024-08-20T22:13:14.0282355Z { 2024-08-20T22:13:14.0282876Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0283509Z "size": 438, 2024-08-20T22:13:14.0284167Z "digest": "sha256:560cd6122a3868e7e82e75eb103d182edf6512114d61ccdfae099af00cb16b4c" 2024-08-20T22:13:14.0284936Z }, 2024-08-20T22:13:14.0285276Z { 2024-08-20T22:13:14.0285859Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0286457Z "size": 701, 2024-08-20T22:13:14.0287112Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:14.0287851Z }, 2024-08-20T22:13:14.0288186Z { 2024-08-20T22:13:14.0288687Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0289376Z "size": 141, 2024-08-20T22:13:14.0290049Z "digest": "sha256:5a4eb19d72198a01f3babd4711c40fcf7c6e5cb9d5cfedc507b0061408d47097" 2024-08-20T22:13:14.0290772Z }, 2024-08-20T22:13:14.0291198Z { 2024-08-20T22:13:14.0291708Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0292457Z "size": 135, 2024-08-20T22:13:14.0293092Z "digest": "sha256:da6a838a66c72b5de9a5590c233779880ea901c1264940a1c618825d2501ffd1" 2024-08-20T22:13:14.0293806Z }, 2024-08-20T22:13:14.0294141Z { 2024-08-20T22:13:14.0294703Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0295325Z "size": 4838068375, 2024-08-20T22:13:14.0295967Z "digest": "sha256:cbb79a7053537bed2d80b77198ea552b985563b11d3f8616593c6ab575a3a090" 2024-08-20T22:13:14.0296729Z }, 2024-08-20T22:13:14.0297067Z { 2024-08-20T22:13:14.0297618Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0298337Z "size": 194, 2024-08-20T22:13:14.0298955Z "digest": "sha256:4d4fa6802ab3e6b36d8d4adb650046e9370c4a16a19876970559ef8a312ca855" 2024-08-20T22:13:14.0299611Z }, 2024-08-20T22:13:14.0300023Z { 2024-08-20T22:13:14.0300630Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0301333Z "size": 564, 2024-08-20T22:13:14.0301924Z "digest": "sha256:0f085d4fa279366fcd52d303b7ad8abddc5962a8872174b8109f3f5d5ca5f2cf" 2024-08-20T22:13:14.0302617Z }, 2024-08-20T22:13:14.0303033Z { 2024-08-20T22:13:14.0303561Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0304171Z "size": 43162921, 2024-08-20T22:13:14.0304944Z "digest": "sha256:4a0412373305517489d891a4749e6d2b80b4aeb7df3b8adca0ed0abb2f3e5300" 2024-08-20T22:13:14.0305646Z }, 2024-08-20T22:13:14.0305949Z { 2024-08-20T22:13:14.0306541Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0307173Z "size": 106, 2024-08-20T22:13:14.0307807Z "digest": "sha256:302c8c1779444e54919f83e91c6d5e1fdea653fc899be802b3a2a34d34d99d5d" 2024-08-20T22:13:14.0308511Z }, 2024-08-20T22:13:14.0308845Z { 2024-08-20T22:13:14.0309404Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0310405Z "size": 1380, 2024-08-20T22:13:14.0310996Z "digest": "sha256:02970347dff622ee45629bc28788716d00528464f71f88ec7b74ae5f44a486e2" 2024-08-20T22:13:14.0311773Z }, 2024-08-20T22:13:14.0312165Z { 2024-08-20T22:13:14.0312702Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0313298Z "size": 701, 2024-08-20T22:13:14.0313968Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:14.0314680Z }, 2024-08-20T22:13:14.0314977Z { 2024-08-20T22:13:14.0315547Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0316202Z "size": 139, 2024-08-20T22:13:14.0316812Z "digest": "sha256:916c5e8580164926a3aec87e88a5d1c2d02756e49fdeed4198e1e6304efa2fd3" 2024-08-20T22:13:14.0317518Z }, 2024-08-20T22:13:14.0317940Z { 2024-08-20T22:13:14.0318541Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0319263Z "size": 118, 2024-08-20T22:13:14.0319900Z "digest": "sha256:c42b3602a1fb4df5111224650f15adffb025979ae9c614807379ea84e189533b" 2024-08-20T22:13:14.0320587Z }, 2024-08-20T22:13:14.0320992Z { 2024-08-20T22:13:14.0321468Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0322104Z "size": 4324672728, 2024-08-20T22:13:14.0322830Z "digest": "sha256:11fad972ef6cb9d2dc236504b0ee5b6e5ea996e4576ece0ed3262fbb9ee2de60" 2024-08-20T22:13:14.0323527Z }, 2024-08-20T22:13:14.0323827Z { 2024-08-20T22:13:14.0324407Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0325099Z "size": 174, 2024-08-20T22:13:14.0325774Z "digest": "sha256:f957c16244819132b478272aaf24231699c4fcbd495c229a574f94daab8d6b86" 2024-08-20T22:13:14.0326421Z }, 2024-08-20T22:13:14.0326753Z { 2024-08-20T22:13:14.0327336Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0327977Z "size": 907, 2024-08-20T22:13:14.0328577Z "digest": "sha256:768c31dca016b80bbdc973c8ee818034bdbd17b7ea9bdadbf0ec45ea51005720" 2024-08-20T22:13:14.0329497Z }, 2024-08-20T22:13:14.0329846Z { 2024-08-20T22:13:14.0330330Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0331048Z "size": 701, 2024-08-20T22:13:14.0331745Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:14.0332461Z }, 2024-08-20T22:13:14.0332816Z { 2024-08-20T22:13:14.0333330Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0333983Z "size": 134, 2024-08-20T22:13:14.0334663Z "digest": "sha256:8c32d0ecebbe2054c96a3a5c1f3d6c8b87c2e826686b369e5242cd55d0f01c2d" 2024-08-20T22:13:14.0335323Z }, 2024-08-20T22:13:14.0335679Z { 2024-08-20T22:13:14.0336236Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0336830Z "size": 32, 2024-08-20T22:13:14.0337607Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:14.0338428Z }, 2024-08-20T22:13:14.0338796Z { 2024-08-20T22:13:14.0339274Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0339997Z "size": 158, 2024-08-20T22:13:14.0340642Z "digest": "sha256:78ee77eddef23f20afc78e2924f4467041cadb754634ee3df431c68211cf676d" 2024-08-20T22:13:14.0341340Z }, 2024-08-20T22:13:14.0341714Z { 2024-08-20T22:13:14.0342231Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0342858Z "size": 1841, 2024-08-20T22:13:14.0343545Z "digest": "sha256:37513e9b43d7123e7ddee09100f62a72b57af7b3833eaecd515e247e2f63e13f" 2024-08-20T22:13:14.0344201Z }, 2024-08-20T22:13:14.0344538Z { 2024-08-20T22:13:14.0345193Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0345785Z "size": 7529776, 2024-08-20T22:13:14.0346422Z "digest": "sha256:a7a5392e2bf6b73d9be292cc22a99ad4637636c15733beb7afbe6403b5ba21a8" 2024-08-20T22:13:14.0347188Z }, 2024-08-20T22:13:14.0347523Z { 2024-08-20T22:13:14.0348026Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0348713Z "size": 163, 2024-08-20T22:13:14.0349336Z "digest": "sha256:b10de02255c021b2366b85bf12ddf593aac89f88964201c455b71a145bc5ca2e" 2024-08-20T22:13:14.0350144Z }, 2024-08-20T22:13:14.0350440Z { 2024-08-20T22:13:14.0350950Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0351721Z "size": 7943, 2024-08-20T22:13:14.0352306Z "digest": "sha256:e56eeff2db93ffe5002a2bb764795b7333321c6802cc28bef0801a619fe6a07e" 2024-08-20T22:13:14.0353002Z }, 2024-08-20T22:13:14.0353402Z { 2024-08-20T22:13:14.0353913Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0354525Z "size": 8068, 2024-08-20T22:13:14.0355209Z "digest": "sha256:70ac3e38d0fd1f0f46ad710c3fc485fd01406606054e8cf84a63af1570bc9f3a" 2024-08-20T22:13:14.0355909Z }, 2024-08-20T22:13:14.0356263Z { 2024-08-20T22:13:14.0356800Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0357435Z "size": 302, 2024-08-20T22:13:14.0358139Z "digest": "sha256:eb09cbf3e4168749cc8605e3188ed35070f6037d13930e443c38d8941734488f" 2024-08-20T22:13:14.0358950Z }, 2024-08-20T22:13:14.0359255Z { 2024-08-20T22:13:14.0359782Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0360458Z "size": 7629638, 2024-08-20T22:13:14.0361066Z "digest": "sha256:ee1273153675e6543b58f5be50321abee5e91f7351701d5d501a6837eaf676ef" 2024-08-20T22:13:14.0361751Z }, 2024-08-20T22:13:14.0362135Z { 2024-08-20T22:13:14.0362666Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0363259Z "size": 108, 2024-08-20T22:13:14.0363931Z "digest": "sha256:e22979f9e0eedfa16f98eb4fa26f30a50c237f5ffd2f6e5b898f2b908058496d" 2024-08-20T22:13:14.0364646Z }, 2024-08-20T22:13:14.0365053Z { 2024-08-20T22:13:14.0365571Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0366317Z "size": 54145661, 2024-08-20T22:13:14.0366967Z "digest": "sha256:6c1f0aabce7c84f9b0b9ab3b96325778dba1c1465b7463c5b4c6be1cf52b74db" 2024-08-20T22:13:14.0367737Z }, 2024-08-20T22:13:14.0368038Z { 2024-08-20T22:13:14.0368550Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0369244Z "size": 473, 2024-08-20T22:13:14.0369888Z "digest": "sha256:7ebe49c3a23e71b289ce012b48dad35791a2f2aa7fcee2326a64d763b0d14df8" 2024-08-20T22:13:14.0370607Z }, 2024-08-20T22:13:14.0371002Z { 2024-08-20T22:13:14.0371514Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0372196Z "size": 1374858766, 2024-08-20T22:13:14.0372888Z "digest": "sha256:5d115b29b5d53e5418762bc41ae975dd245bcffc3d058b8902c58146d505cb37" 2024-08-20T22:13:14.0373568Z }, 2024-08-20T22:13:14.0373969Z { 2024-08-20T22:13:14.0374529Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0375173Z "size": 106, 2024-08-20T22:13:14.0375856Z "digest": "sha256:7f2d7a04fbe65bbe1e895138e0cfeeef53ad926c890bab8e7e132ad1e655e48f" 2024-08-20T22:13:14.0376525Z }, 2024-08-20T22:13:14.0376885Z { 2024-08-20T22:13:14.0377627Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0378268Z "size": 559, 2024-08-20T22:13:14.0378933Z "digest": "sha256:d79fd7d526989e554455eda7d459f2c88444030f61710101e52ec8c7a49b7b79" 2024-08-20T22:13:14.0379708Z }, 2024-08-20T22:13:14.0380084Z { 2024-08-20T22:13:14.0380593Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0381232Z "size": 46248488, 2024-08-20T22:13:14.0381884Z "digest": "sha256:9752d6fbbcab4e7403f26777c5b65911320de6389c81b833a47ba45c4f3354b4" 2024-08-20T22:13:14.0382572Z }, 2024-08-20T22:13:14.0382952Z { 2024-08-20T22:13:14.0383456Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0384087Z "size": 111, 2024-08-20T22:13:14.0384750Z "digest": "sha256:83808b2eff4d71ed46db7c29f66fe918f328405c25b314225dfad01b4868d1f4" 2024-08-20T22:13:14.0385422Z }, 2024-08-20T22:13:14.0385873Z { 2024-08-20T22:13:14.0386636Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0387276Z "size": 32, 2024-08-20T22:13:14.0387985Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:14.0388759Z }, 2024-08-20T22:13:14.0389248Z { 2024-08-20T22:13:14.0389844Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0390438Z "size": 32, 2024-08-20T22:13:14.0391059Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:14.0391820Z }, 2024-08-20T22:13:14.0392123Z { 2024-08-20T22:13:14.0392719Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0393439Z "size": 32, 2024-08-20T22:13:14.0394055Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:14.0394710Z }, 2024-08-20T22:13:14.0395220Z { 2024-08-20T22:13:14.0395833Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:14.0396497Z "size": 32, 2024-08-20T22:13:14.0397129Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:14.0397819Z } 2024-08-20T22:13:14.0398239Z ] 2024-08-20T22:13:14.0398619Z } 2024-08-20T22:13:14.0702870Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:14.0703366Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:14.0703912Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T22:13:14.0714304Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:14.0714806Z env: 2024-08-20T22:13:14.0715090Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:14.0716255Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:14.0717421Z ##[endgroup] 2024-08-20T22:13:14.0750114Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:14.0800326Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T22:13:14.0800904Z with: 2024-08-20T22:13:14.0801963Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:14.0803262Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:14.0803779Z env: 2024-08-20T22:13:14.0804061Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:14.0804419Z ##[endgroup] 2024-08-20T22:13:14.0825136Z ##[group]Run set -x 2024-08-20T22:13:14.0825646Z set -x 2024-08-20T22:13:14.0825960Z set +e 2024-08-20T22:13:14.0826270Z  2024-08-20T22:13:14.0826553Z login() { 2024-08-20T22:13:14.0827213Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:14.0828080Z } 2024-08-20T22:13:14.0828372Z  2024-08-20T22:13:14.0828689Z retry () { 2024-08-20T22:13:14.0829087Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:14.0829539Z } 2024-08-20T22:13:14.0829825Z  2024-08-20T22:13:14.0830161Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:14.0830688Z  2024-08-20T22:13:14.0830984Z set -e 2024-08-20T22:13:14.0831466Z # ignore output since only exit code is used for conditional 2024-08-20T22:13:14.0832172Z # only pull docker image if it's not available locally 2024-08-20T22:13:14.0832936Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T22:13:14.0833644Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T22:13:14.0834099Z fi 2024-08-20T22:13:14.0842354Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:14.0842975Z env: 2024-08-20T22:13:14.0843272Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:14.0844416Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:14.0845757Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:14.0846287Z ##[endgroup] 2024-08-20T22:13:14.0874697Z + set +e 2024-08-20T22:13:14.0875507Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:14.0876237Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:14.0878407Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:14.0880149Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:14.6095470Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:14.6096422Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:14.6097337Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:14.6097876Z 2024-08-20T22:13:14.6098001Z Login Succeeded 2024-08-20T22:13:14.6119051Z + set -e 2024-08-20T22:13:14.6120360Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:14.6276515Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:14.6278788Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:14.9144984Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-08-20T22:13:14.9147053Z 7a2c55901189: Pulling fs layer 2024-08-20T22:13:14.9147557Z 224fe954d725: Pulling fs layer 2024-08-20T22:13:14.9148316Z 75722010b82e: Pulling fs layer 2024-08-20T22:13:14.9148837Z d527cbbb87e3: Pulling fs layer 2024-08-20T22:13:14.9149322Z b57676e46aee: Pulling fs layer 2024-08-20T22:13:14.9149804Z a8c1e85b5e14: Pulling fs layer 2024-08-20T22:13:14.9150497Z a41a8d1c11c8: Pulling fs layer 2024-08-20T22:13:14.9151021Z 0c1227890755: Pulling fs layer 2024-08-20T22:13:14.9151482Z d8d1234baab3: Pulling fs layer 2024-08-20T22:13:14.9151976Z 7ed32bc8e469: Pulling fs layer 2024-08-20T22:13:14.9152631Z ec1e7978c1fe: Pulling fs layer 2024-08-20T22:13:14.9153120Z ac23c5fcc1ef: Pulling fs layer 2024-08-20T22:13:14.9153520Z ca426ffc9d2b: Pulling fs layer 2024-08-20T22:13:14.9154127Z 4373aa743caf: Pulling fs layer 2024-08-20T22:13:14.9154500Z d527cbbb87e3: Waiting 2024-08-20T22:13:14.9154852Z 75b2abb16c68: Pulling fs layer 2024-08-20T22:13:14.9155225Z ef1e3ef1e85f: Pulling fs layer 2024-08-20T22:13:14.9155601Z 0bf61114d37b: Pulling fs layer 2024-08-20T22:13:14.9155975Z 2df849f72ba4: Pulling fs layer 2024-08-20T22:13:14.9156344Z b57676e46aee: Waiting 2024-08-20T22:13:14.9156673Z 34c47d63c5de: Pulling fs layer 2024-08-20T22:13:14.9157052Z 1fef4bd19792: Pulling fs layer 2024-08-20T22:13:14.9157499Z 6efb96fc87ea: Pulling fs layer 2024-08-20T22:13:14.9158082Z b119c4fc6c58: Pulling fs layer 2024-08-20T22:13:14.9158545Z a8c1e85b5e14: Waiting 2024-08-20T22:13:14.9158990Z 33b639226eee: Pulling fs layer 2024-08-20T22:13:14.9159461Z a41a8d1c11c8: Waiting 2024-08-20T22:13:14.9159883Z d3547bd43146: Pulling fs layer 2024-08-20T22:13:14.9160358Z 5645520256fa: Pulling fs layer 2024-08-20T22:13:14.9160817Z ca426ffc9d2b: Waiting 2024-08-20T22:13:14.9161166Z 829ed7d91e38: Pulling fs layer 2024-08-20T22:13:14.9161551Z 385d0421d67c: Pulling fs layer 2024-08-20T22:13:14.9161929Z 7cbfce999d76: Pulling fs layer 2024-08-20T22:13:14.9162314Z 6658e558ebe4: Pulling fs layer 2024-08-20T22:13:14.9162665Z 0c1227890755: Waiting 2024-08-20T22:13:14.9163001Z 33444a5a7e01: Pulling fs layer 2024-08-20T22:13:14.9163374Z 7c47122f4e09: Pulling fs layer 2024-08-20T22:13:14.9163753Z 1811ff4d3556: Pulling fs layer 2024-08-20T22:13:14.9164119Z 7754ee068b33: Pulling fs layer 2024-08-20T22:13:14.9164482Z b119c4fc6c58: Waiting 2024-08-20T22:13:14.9164821Z 0479fbac3361: Pulling fs layer 2024-08-20T22:13:14.9165181Z 33b639226eee: Waiting 2024-08-20T22:13:14.9165515Z f1d6874dee9c: Pulling fs layer 2024-08-20T22:13:14.9165894Z afc429e87b59: Pulling fs layer 2024-08-20T22:13:14.9166261Z 2df849f72ba4: Waiting 2024-08-20T22:13:14.9166578Z 829ed7d91e38: Waiting 2024-08-20T22:13:14.9166908Z 019620471c25: Pulling fs layer 2024-08-20T22:13:14.9167278Z 34c47d63c5de: Waiting 2024-08-20T22:13:14.9167618Z 27aeb1b2cd5e: Pulling fs layer 2024-08-20T22:13:14.9168126Z 6658e558ebe4: Waiting 2024-08-20T22:13:14.9168466Z 829c0e19f4b7: Pulling fs layer 2024-08-20T22:13:14.9168843Z 560cd6122a38: Pulling fs layer 2024-08-20T22:13:14.9169199Z 4373aa743caf: Waiting 2024-08-20T22:13:14.9169520Z 5645520256fa: Waiting 2024-08-20T22:13:14.9170000Z ef1e3ef1e85f: Waiting 2024-08-20T22:13:14.9170397Z 5a4eb19d7219: Pulling fs layer 2024-08-20T22:13:14.9170775Z da6a838a66c7: Pulling fs layer 2024-08-20T22:13:14.9171144Z cbb79a705353: Pulling fs layer 2024-08-20T22:13:14.9171506Z 7cbfce999d76: Waiting 2024-08-20T22:13:14.9171840Z 4d4fa6802ab3: Pulling fs layer 2024-08-20T22:13:14.9172214Z 0f085d4fa279: Pulling fs layer 2024-08-20T22:13:14.9172569Z ac23c5fcc1ef: Waiting 2024-08-20T22:13:14.9172905Z 4a0412373305: Pulling fs layer 2024-08-20T22:13:14.9173261Z 385d0421d67c: Waiting 2024-08-20T22:13:14.9173588Z 302c8c177944: Pulling fs layer 2024-08-20T22:13:14.9173957Z 02970347dff6: Pulling fs layer 2024-08-20T22:13:14.9174316Z ec1e7978c1fe: Waiting 2024-08-20T22:13:14.9174641Z 0479fbac3361: Waiting 2024-08-20T22:13:14.9174963Z 916c5e858016: Pulling fs layer 2024-08-20T22:13:14.9175318Z 1811ff4d3556: Waiting 2024-08-20T22:13:14.9175645Z c42b3602a1fb: Pulling fs layer 2024-08-20T22:13:14.9176006Z f1d6874dee9c: Waiting 2024-08-20T22:13:14.9176319Z 5a4eb19d7219: Waiting 2024-08-20T22:13:14.9176638Z 7ed32bc8e469: Waiting 2024-08-20T22:13:14.9177126Z 11fad972ef6c: Pulling fs layer 2024-08-20T22:13:14.9177485Z cbb79a705353: Waiting 2024-08-20T22:13:14.9177805Z 33444a5a7e01: Waiting 2024-08-20T22:13:14.9178122Z 7c47122f4e09: Waiting 2024-08-20T22:13:14.9178449Z f957c1624481: Pulling fs layer 2024-08-20T22:13:14.9178823Z 768c31dca016: Pulling fs layer 2024-08-20T22:13:14.9179183Z 916c5e858016: Waiting 2024-08-20T22:13:14.9179502Z 27aeb1b2cd5e: Waiting 2024-08-20T22:13:14.9179834Z 8c32d0ecebbe: Pulling fs layer 2024-08-20T22:13:14.9180221Z 4f4fb700ef54: Pulling fs layer 2024-08-20T22:13:14.9180608Z da6a838a66c7: Waiting 2024-08-20T22:13:14.9180966Z 78ee77eddef2: Pulling fs layer 2024-08-20T22:13:14.9181426Z 37513e9b43d7: Pulling fs layer 2024-08-20T22:13:14.9181784Z 019620471c25: Waiting 2024-08-20T22:13:14.9182098Z 02970347dff6: Waiting 2024-08-20T22:13:14.9182417Z 6efb96fc87ea: Waiting 2024-08-20T22:13:14.9182746Z a7a5392e2bf6: Pulling fs layer 2024-08-20T22:13:14.9183121Z b10de02255c0: Pulling fs layer 2024-08-20T22:13:14.9183491Z 4d4fa6802ab3: Waiting 2024-08-20T22:13:14.9183822Z e56eeff2db93: Pulling fs layer 2024-08-20T22:13:14.9184184Z 4a0412373305: Waiting 2024-08-20T22:13:14.9184500Z 0f085d4fa279: Waiting 2024-08-20T22:13:14.9184840Z 70ac3e38d0fd: Pulling fs layer 2024-08-20T22:13:14.9185212Z eb09cbf3e416: Pulling fs layer 2024-08-20T22:13:14.9185591Z ee1273153675: Pulling fs layer 2024-08-20T22:13:14.9185962Z e22979f9e0ee: Pulling fs layer 2024-08-20T22:13:14.9186337Z 6c1f0aabce7c: Pulling fs layer 2024-08-20T22:13:14.9186735Z c42b3602a1fb: Waiting 2024-08-20T22:13:14.9187210Z 7ebe49c3a23e: Pulling fs layer 2024-08-20T22:13:14.9187617Z e56eeff2db93: Waiting 2024-08-20T22:13:14.9187997Z 5d115b29b5d5: Pulling fs layer 2024-08-20T22:13:14.9188372Z 7f2d7a04fbe6: Pulling fs layer 2024-08-20T22:13:14.9188898Z 829c0e19f4b7: Waiting 2024-08-20T22:13:14.9189242Z d79fd7d52698: Pulling fs layer 2024-08-20T22:13:14.9189664Z ee1273153675: Waiting 2024-08-20T22:13:14.9190105Z 9752d6fbbcab: Pulling fs layer 2024-08-20T22:13:14.9190558Z 83808b2eff4d: Pulling fs layer 2024-08-20T22:13:14.9190955Z 7ebe49c3a23e: Waiting 2024-08-20T22:13:14.9191335Z 5d115b29b5d5: Waiting 2024-08-20T22:13:14.9191655Z e22979f9e0ee: Waiting 2024-08-20T22:13:14.9192013Z 6c1f0aabce7c: Waiting 2024-08-20T22:13:14.9192342Z 7f2d7a04fbe6: Waiting 2024-08-20T22:13:14.9192654Z a7a5392e2bf6: Waiting 2024-08-20T22:13:14.9193036Z d79fd7d52698: Waiting 2024-08-20T22:13:14.9193359Z b10de02255c0: Waiting 2024-08-20T22:13:14.9193711Z 8c32d0ecebbe: Waiting 2024-08-20T22:13:14.9194051Z eb09cbf3e416: Waiting 2024-08-20T22:13:14.9194363Z 4f4fb700ef54: Waiting 2024-08-20T22:13:14.9194736Z 83808b2eff4d: Waiting 2024-08-20T22:13:14.9195062Z 11fad972ef6c: Waiting 2024-08-20T22:13:14.9195415Z 78ee77eddef2: Waiting 2024-08-20T22:13:14.9195758Z 37513e9b43d7: Waiting 2024-08-20T22:13:14.9196120Z 302c8c177944: Waiting 2024-08-20T22:13:15.0494382Z 224fe954d725: Verifying Checksum 2024-08-20T22:13:15.0494859Z 224fe954d725: Download complete 2024-08-20T22:13:15.1761796Z d527cbbb87e3: Verifying Checksum 2024-08-20T22:13:15.1762422Z d527cbbb87e3: Download complete 2024-08-20T22:13:15.2466613Z b57676e46aee: Download complete 2024-08-20T22:13:15.2618676Z 7a2c55901189: Verifying Checksum 2024-08-20T22:13:15.2619135Z 7a2c55901189: Download complete 2024-08-20T22:13:15.3246539Z a41a8d1c11c8: Verifying Checksum 2024-08-20T22:13:15.3247202Z a41a8d1c11c8: Download complete 2024-08-20T22:13:15.4000594Z 0c1227890755: Verifying Checksum 2024-08-20T22:13:15.4001141Z 0c1227890755: Download complete 2024-08-20T22:13:15.4593336Z d8d1234baab3: Verifying Checksum 2024-08-20T22:13:15.4593865Z d8d1234baab3: Download complete 2024-08-20T22:13:15.5274714Z 75722010b82e: Verifying Checksum 2024-08-20T22:13:15.5275202Z 75722010b82e: Download complete 2024-08-20T22:13:15.5973857Z ec1e7978c1fe: Verifying Checksum 2024-08-20T22:13:15.6638677Z ac23c5fcc1ef: Download complete 2024-08-20T22:13:16.4556573Z 7a2c55901189: Pull complete 2024-08-20T22:13:16.7672447Z 224fe954d725: Pull complete 2024-08-20T22:13:17.7120642Z 75722010b82e: Pull complete 2024-08-20T22:13:17.7334009Z d527cbbb87e3: Pull complete 2024-08-20T22:13:17.7585208Z b57676e46aee: Pull complete 2024-08-20T22:13:18.1809444Z ca426ffc9d2b: Verifying Checksum 2024-08-20T22:13:18.1810272Z ca426ffc9d2b: Download complete 2024-08-20T22:13:18.2592831Z 4373aa743caf: Verifying Checksum 2024-08-20T22:13:18.2593381Z 4373aa743caf: Download complete 2024-08-20T22:13:18.3363132Z 75b2abb16c68: Verifying Checksum 2024-08-20T22:13:18.3363689Z 75b2abb16c68: Download complete 2024-08-20T22:13:18.3994605Z ef1e3ef1e85f: Verifying Checksum 2024-08-20T22:13:18.3995159Z ef1e3ef1e85f: Download complete 2024-08-20T22:13:19.3590028Z 0bf61114d37b: Verifying Checksum 2024-08-20T22:13:19.3590520Z 0bf61114d37b: Download complete 2024-08-20T22:13:19.4230818Z 2df849f72ba4: Download complete 2024-08-20T22:13:19.4973441Z 34c47d63c5de: Download complete 2024-08-20T22:13:19.5587665Z 1fef4bd19792: Download complete 2024-08-20T22:13:28.5834159Z a8c1e85b5e14: Download complete 2024-08-20T22:13:28.6844808Z b119c4fc6c58: Download complete 2024-08-20T22:13:28.7560968Z 33b639226eee: Download complete 2024-08-20T22:13:28.8377906Z d3547bd43146: Download complete 2024-08-20T22:13:28.9058799Z 5645520256fa: Verifying Checksum 2024-08-20T22:13:29.0001412Z 829ed7d91e38: Download complete 2024-08-20T22:13:30.2991210Z 385d0421d67c: Verifying Checksum 2024-08-20T22:13:30.2991859Z 385d0421d67c: Download complete 2024-08-20T22:13:30.3850483Z 7cbfce999d76: Verifying Checksum 2024-08-20T22:13:30.3851112Z 7cbfce999d76: Download complete 2024-08-20T22:13:30.4473875Z 6658e558ebe4: Download complete 2024-08-20T22:13:30.5259246Z 33444a5a7e01: Download complete 2024-08-20T22:13:30.6024006Z 7c47122f4e09: Verifying Checksum 2024-08-20T22:13:30.6024624Z 7c47122f4e09: Download complete 2024-08-20T22:13:30.6642346Z 1811ff4d3556: Verifying Checksum 2024-08-20T22:13:30.6642964Z 1811ff4d3556: Download complete 2024-08-20T22:13:36.6549659Z 7754ee068b33: Verifying Checksum 2024-08-20T22:13:36.6550341Z 7754ee068b33: Download complete 2024-08-20T22:13:36.7721646Z 0479fbac3361: Verifying Checksum 2024-08-20T22:13:36.7722127Z 0479fbac3361: Download complete 2024-08-20T22:13:36.8618041Z f1d6874dee9c: Download complete 2024-08-20T22:13:37.2777569Z afc429e87b59: Verifying Checksum 2024-08-20T22:13:37.2778023Z afc429e87b59: Download complete 2024-08-20T22:13:37.3480826Z 019620471c25: Verifying Checksum 2024-08-20T22:13:37.3481446Z 019620471c25: Download complete 2024-08-20T22:13:37.4149957Z 27aeb1b2cd5e: Download complete 2024-08-20T22:13:37.6638400Z 829c0e19f4b7: Verifying Checksum 2024-08-20T22:13:37.6639047Z 829c0e19f4b7: Download complete 2024-08-20T22:13:37.7318122Z 560cd6122a38: Verifying Checksum 2024-08-20T22:13:37.7318579Z 560cd6122a38: Download complete 2024-08-20T22:13:37.7999631Z 5a4eb19d7219: Download complete 2024-08-20T22:13:37.8673827Z da6a838a66c7: Download complete 2024-08-20T22:13:40.7882248Z 7ed32bc8e469: Verifying Checksum 2024-08-20T22:13:40.7882704Z 7ed32bc8e469: Download complete 2024-08-20T22:13:40.8661859Z 4d4fa6802ab3: Download complete 2024-08-20T22:13:40.9301333Z 0f085d4fa279: Download complete 2024-08-20T22:13:41.4185868Z 4a0412373305: Verifying Checksum 2024-08-20T22:13:41.4186312Z 4a0412373305: Download complete 2024-08-20T22:13:41.4918635Z 302c8c177944: Verifying Checksum 2024-08-20T22:13:41.4919107Z 302c8c177944: Download complete 2024-08-20T22:13:41.5636707Z 02970347dff6: Download complete 2024-08-20T22:13:41.6290756Z 916c5e858016: Verifying Checksum 2024-08-20T22:13:41.6291217Z 916c5e858016: Download complete 2024-08-20T22:13:41.6937368Z c42b3602a1fb: Verifying Checksum 2024-08-20T22:13:41.8819426Z a8c1e85b5e14: Pull complete 2024-08-20T22:13:42.1082560Z a41a8d1c11c8: Pull complete 2024-08-20T22:13:42.3186514Z 0c1227890755: Pull complete 2024-08-20T22:13:42.5323127Z d8d1234baab3: Pull complete 2024-08-20T22:13:48.2513452Z 6efb96fc87ea: Verifying Checksum 2024-08-20T22:13:48.2514115Z 6efb96fc87ea: Download complete 2024-08-20T22:13:48.3298147Z f957c1624481: Verifying Checksum 2024-08-20T22:13:48.3298821Z f957c1624481: Download complete 2024-08-20T22:13:48.4116561Z 768c31dca016: Verifying Checksum 2024-08-20T22:13:48.4117044Z 768c31dca016: Download complete 2024-08-20T22:13:48.4805689Z 8c32d0ecebbe: Verifying Checksum 2024-08-20T22:13:48.4806218Z 8c32d0ecebbe: Download complete 2024-08-20T22:13:48.4893331Z 4f4fb700ef54: Verifying Checksum 2024-08-20T22:13:48.4893918Z 4f4fb700ef54: Download complete 2024-08-20T22:13:48.5522159Z 78ee77eddef2: Verifying Checksum 2024-08-20T22:13:48.5522780Z 78ee77eddef2: Download complete 2024-08-20T22:13:48.6229159Z 37513e9b43d7: Verifying Checksum 2024-08-20T22:13:48.6229773Z 37513e9b43d7: Download complete 2024-08-20T22:13:48.7502284Z a7a5392e2bf6: Verifying Checksum 2024-08-20T22:13:48.7502851Z a7a5392e2bf6: Download complete 2024-08-20T22:13:48.8207716Z b10de02255c0: Download complete 2024-08-20T22:13:48.8896165Z e56eeff2db93: Download complete 2024-08-20T22:13:49.0263102Z eb09cbf3e416: Download complete 2024-08-20T22:13:49.1486765Z ee1273153675: Verifying Checksum 2024-08-20T22:13:49.2186040Z e22979f9e0ee: Download complete 2024-08-20T22:13:49.8076288Z 6c1f0aabce7c: Verifying Checksum 2024-08-20T22:13:49.8076879Z 6c1f0aabce7c: Download complete 2024-08-20T22:13:49.8746432Z 7ebe49c3a23e: Verifying Checksum 2024-08-20T22:13:49.8747076Z 7ebe49c3a23e: Download complete 2024-08-20T22:14:16.2521790Z 5d115b29b5d5: Verifying Checksum 2024-08-20T22:14:16.2522405Z 5d115b29b5d5: Download complete 2024-08-20T22:14:16.3181444Z 7f2d7a04fbe6: Verifying Checksum 2024-08-20T22:14:16.3181898Z 7f2d7a04fbe6: Download complete 2024-08-20T22:14:16.3945639Z d79fd7d52698: Verifying Checksum 2024-08-20T22:14:16.3946226Z d79fd7d52698: Download complete 2024-08-20T22:14:17.8658885Z 9752d6fbbcab: Verifying Checksum 2024-08-20T22:14:17.8659360Z 9752d6fbbcab: Download complete 2024-08-20T22:14:17.9381386Z 83808b2eff4d: Verifying Checksum 2024-08-20T22:14:53.8820624Z 11fad972ef6c: Download complete 2024-08-20T22:14:56.9978713Z cbb79a705353: Verifying Checksum 2024-08-20T22:14:56.9979194Z cbb79a705353: Download complete 2024-08-20T22:15:11.7004031Z 7ed32bc8e469: Pull complete 2024-08-20T22:15:11.9177255Z ec1e7978c1fe: Pull complete 2024-08-20T22:15:12.1271880Z ac23c5fcc1ef: Pull complete 2024-08-20T22:15:20.7223659Z ca426ffc9d2b: Pull complete 2024-08-20T22:15:20.9322748Z 4373aa743caf: Pull complete 2024-08-20T22:15:21.1496954Z 75b2abb16c68: Pull complete 2024-08-20T22:15:21.3528210Z ef1e3ef1e85f: Pull complete 2024-08-20T22:15:23.9582473Z 0bf61114d37b: Pull complete 2024-08-20T22:15:24.1686509Z 2df849f72ba4: Pull complete 2024-08-20T22:15:24.3849567Z 34c47d63c5de: Pull complete 2024-08-20T22:15:24.5616885Z 1fef4bd19792: Pull complete 2024-08-20T22:16:23.0901410Z 6efb96fc87ea: Pull complete 2024-08-20T22:16:23.2528284Z b119c4fc6c58: Pull complete 2024-08-20T22:16:23.4599008Z 33b639226eee: Pull complete 2024-08-20T22:16:23.6752365Z d3547bd43146: Pull complete 2024-08-20T22:16:23.8723793Z 5645520256fa: Pull complete 2024-08-20T22:16:24.0807226Z 829ed7d91e38: Pull complete 2024-08-20T22:16:27.5367566Z 385d0421d67c: Pull complete 2024-08-20T22:16:27.7076778Z 7cbfce999d76: Pull complete 2024-08-20T22:16:27.9249927Z 6658e558ebe4: Pull complete 2024-08-20T22:16:28.1408531Z 33444a5a7e01: Pull complete 2024-08-20T22:16:28.2934426Z 7c47122f4e09: Pull complete 2024-08-20T22:16:28.4429671Z 1811ff4d3556: Pull complete 2024-08-20T22:16:39.1163301Z 7754ee068b33: Pull complete 2024-08-20T22:16:39.3139165Z 0479fbac3361: Pull complete 2024-08-20T22:16:39.5212506Z f1d6874dee9c: Pull complete 2024-08-20T22:16:40.5020659Z afc429e87b59: Pull complete 2024-08-20T22:16:40.6890501Z 019620471c25: Pull complete 2024-08-20T22:16:40.8811935Z 27aeb1b2cd5e: Pull complete 2024-08-20T22:16:41.3582175Z 829c0e19f4b7: Pull complete 2024-08-20T22:16:41.5485513Z 560cd6122a38: Pull complete 2024-08-20T22:16:41.8562050Z 5a4eb19d7219: Pull complete 2024-08-20T22:16:42.0420541Z da6a838a66c7: Pull complete 2024-08-20T22:17:20.9872301Z cbb79a705353: Pull complete 2024-08-20T22:17:21.2051935Z 4d4fa6802ab3: Pull complete 2024-08-20T22:17:21.3342180Z 0f085d4fa279: Pull complete 2024-08-20T22:17:23.6893071Z 4a0412373305: Pull complete 2024-08-20T22:17:23.7085931Z 302c8c177944: Pull complete 2024-08-20T22:17:23.7299461Z 02970347dff6: Pull complete 2024-08-20T22:17:23.7705070Z 916c5e858016: Pull complete 2024-08-20T22:17:23.7898799Z c42b3602a1fb: Pull complete 2024-08-20T22:18:45.7427139Z 11fad972ef6c: Pull complete 2024-08-20T22:18:45.9516392Z f957c1624481: Pull complete 2024-08-20T22:18:46.1690843Z 768c31dca016: Pull complete 2024-08-20T22:18:46.5745631Z 8c32d0ecebbe: Pull complete 2024-08-20T22:18:46.7824165Z 4f4fb700ef54: Pull complete 2024-08-20T22:18:46.9980529Z 78ee77eddef2: Pull complete 2024-08-20T22:18:47.2098304Z 37513e9b43d7: Pull complete 2024-08-20T22:18:47.5785128Z a7a5392e2bf6: Pull complete 2024-08-20T22:18:47.8032206Z b10de02255c0: Pull complete 2024-08-20T22:18:48.0131197Z e56eeff2db93: Pull complete 2024-08-20T22:18:48.2151297Z 70ac3e38d0fd: Pull complete 2024-08-20T22:18:48.4234541Z eb09cbf3e416: Pull complete 2024-08-20T22:18:49.8145007Z ee1273153675: Pull complete 2024-08-20T22:18:50.0335011Z e22979f9e0ee: Pull complete 2024-08-20T22:18:52.1762875Z 6c1f0aabce7c: Pull complete 2024-08-20T22:18:52.3911108Z 7ebe49c3a23e: Pull complete 2024-08-20T22:19:05.7575444Z 5d115b29b5d5: Pull complete 2024-08-20T22:19:05.9250063Z 7f2d7a04fbe6: Pull complete 2024-08-20T22:19:06.1050454Z d79fd7d52698: Pull complete 2024-08-20T22:19:06.9309241Z 9752d6fbbcab: Pull complete 2024-08-20T22:19:07.1308247Z 83808b2eff4d: Pull complete 2024-08-20T22:19:07.9623601Z Digest: sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T22:19:08.0024325Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:19:08.0287483Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:19:08.0364992Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:19:08.0365904Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:19:08.0375913Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:19:08.0376415Z env: 2024-08-20T22:19:08.0376701Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:08.0377063Z ##[endgroup] 2024-08-20T22:19:08.0579169Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-20T22:19:08.0579714Z with: 2024-08-20T22:19:08.0580005Z driver-version: 550.54.15 2024-08-20T22:19:08.0580344Z env: 2024-08-20T22:19:08.0580635Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:08.0580981Z ##[endgroup] 2024-08-20T22:19:08.0656490Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:19:08.0657053Z with: 2024-08-20T22:19:08.0657341Z timeout_minutes: 10 2024-08-20T22:19:08.0657675Z max_attempts: 3 2024-08-20T22:19:08.0689900Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-20T22:19:08.0722682Z retry_wait_seconds: 10 2024-08-20T22:19:08.0723053Z polling_interval_seconds: 1 2024-08-20T22:19:08.0723427Z warning_on_retry: true 2024-08-20T22:19:08.0723774Z continue_on_error: false 2024-08-20T22:19:08.0724256Z env: 2024-08-20T22:19:08.0724532Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:08.0724882Z DRIVER_VERSION: 550.54.15 2024-08-20T22:19:08.0725246Z ##[endgroup] 2024-08-20T22:19:08.1546464Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-20T22:19:08.1547784Z + pre_install_nvidia_driver_amzn2 2024-08-20T22:19:08.1549699Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-20T22:19:08.6198194Z No match for argument: nvidia-driver-latest-dkms 2024-08-20T22:19:08.6198777Z No packages marked for removal. 2024-08-20T22:19:08.6260872Z Dependencies resolved. 2024-08-20T22:19:08.6270745Z Nothing to do. 2024-08-20T22:19:08.6271387Z Complete! 2024-08-20T22:19:08.7035638Z + install_nvidia_driver_common 2024-08-20T22:19:08.7043816Z + echo 'Before installing NVIDIA driver' 2024-08-20T22:19:08.7044386Z + lspci 2024-08-20T22:19:08.7044741Z Before installing NVIDIA driver 2024-08-20T22:19:08.7907510Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:19:08.7908296Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:19:08.7909366Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:19:08.7910684Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:19:08.7911784Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:19:08.7912593Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:19:08.7913339Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:19:08.7914152Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:19:08.7914753Z + lsmod 2024-08-20T22:19:08.7952764Z Module Size Used by 2024-08-20T22:19:08.7953354Z xt_conntrack 16384 1 2024-08-20T22:19:08.7953837Z nft_chain_nat 16384 3 2024-08-20T22:19:08.7954220Z xt_MASQUERADE 20480 1 2024-08-20T22:19:08.7954687Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:19:08.7955196Z nf_conntrack_netlink 57344 0 2024-08-20T22:19:08.7955781Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:19:08.7956523Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:19:08.7957044Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:19:08.7957630Z xfrm_user 57344 1 2024-08-20T22:19:08.7958185Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:19:08.7958714Z xt_addrtype 16384 2 2024-08-20T22:19:08.7959229Z nft_compat 20480 4 2024-08-20T22:19:08.7959824Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:19:08.7960510Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:19:08.7961239Z br_netfilter 36864 0 2024-08-20T22:19:08.7961779Z bridge 307200 1 br_netfilter 2024-08-20T22:19:08.7962223Z stp 16384 1 bridge 2024-08-20T22:19:08.7962655Z llc 16384 2 bridge,stp 2024-08-20T22:19:08.7963080Z overlay 167936 0 2024-08-20T22:19:08.7963473Z tls 114688 0 2024-08-20T22:19:08.7963845Z nls_ascii 16384 1 2024-08-20T22:19:08.7964227Z nls_cp437 20480 1 2024-08-20T22:19:08.7964603Z vfat 24576 1 2024-08-20T22:19:08.7964982Z fat 86016 1 vfat 2024-08-20T22:19:08.7965376Z sunrpc 692224 1 2024-08-20T22:19:08.7965752Z ena 167936 0 2024-08-20T22:19:08.7966133Z ptp 36864 1 ena 2024-08-20T22:19:08.7966537Z pps_core 24576 1 ptp 2024-08-20T22:19:08.7966940Z ghash_clmulni_intel 16384 0 2024-08-20T22:19:08.7967340Z aesni_intel 393216 0 2024-08-20T22:19:08.7967716Z i8042 45056 0 2024-08-20T22:19:08.7968090Z serio 28672 3 i8042 2024-08-20T22:19:08.7968522Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:19:08.7969243Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:19:08.7969746Z button 24576 0 2024-08-20T22:19:08.7970123Z sch_fq_codel 20480 17 2024-08-20T22:19:08.7970543Z dm_mod 188416 0 2024-08-20T22:19:08.7971038Z fuse 163840 1 2024-08-20T22:19:08.7971521Z configfs 57344 1 2024-08-20T22:19:08.7972007Z dax 45056 1 dm_mod 2024-08-20T22:19:08.7972519Z loop 36864 0 2024-08-20T22:19:08.7972995Z dmi_sysfs 20480 0 2024-08-20T22:19:08.7973465Z crc32_pclmul 16384 0 2024-08-20T22:19:08.7973837Z crc32c_intel 24576 0 2024-08-20T22:19:08.7974215Z efivarfs 24576 1 2024-08-20T22:19:08.7974591Z + modinfo nvidia 2024-08-20T22:19:08.7975301Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:19:08.7976040Z alias: char-major-195-* 2024-08-20T22:19:08.7976470Z version: 550.54.15 2024-08-20T22:19:08.7976839Z supported: external 2024-08-20T22:19:08.7977198Z license: NVIDIA 2024-08-20T22:19:08.7977587Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:19:08.7978084Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:19:08.7978558Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:19:08.7979046Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:19:08.7979702Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:19:08.7980280Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:19:08.7980791Z depends: i2c-core,drm 2024-08-20T22:19:08.7981152Z retpoline: Y 2024-08-20T22:19:08.7981469Z name: nvidia 2024-08-20T22:19:08.7982265Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:19:08.7983192Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:19:08.7983839Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:19:08.7984432Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:19:08.7984878Z parm: NVreg_RmLogonRC:int 2024-08-20T22:19:08.7985313Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:19:08.7985764Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:19:08.7986200Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:19:08.7986642Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:19:08.7987171Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:19:08.7987725Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:19:08.7988201Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:19:08.7988634Z parm: NVreg_EnableMSI:int 2024-08-20T22:19:08.7989049Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:19:08.7989509Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:19:08.7990040Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:19:08.7990616Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:19:08.7991175Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:19:08.7991792Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:08.7992389Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:19:08.7993003Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:08.7993590Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:19:08.7994085Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:19:08.7994619Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:19:08.7995168Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:19:08.7995661Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:19:08.7996136Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:19:08.7996622Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:19:08.7997102Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:19:08.7997548Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:19:08.7998282Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:19:08.7998821Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:19:08.7999312Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:19:08.7999807Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:19:08.8000326Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:19:08.8000877Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:19:08.8001392Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:19:08.8001879Z parm: NVreg_RmMsg:charp 2024-08-20T22:19:08.8002304Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:19:08.8002785Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:19:08.8003267Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:19:08.8003738Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:19:08.8004228Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:19:08.8004724Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:19:08.8005185Z parm: rm_firmware_active:charp 2024-08-20T22:19:08.8005618Z + HAS_NVIDIA_DRIVER=0 2024-08-20T22:19:08.8006050Z ++ command -v nvidia-smi 2024-08-20T22:19:08.8006471Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-20T22:19:08.8006850Z + set +e 2024-08-20T22:19:08.8007379Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-20T22:19:11.0404123Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-20T22:19:11.0404742Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:19:11.0405442Z + '[' 0 -ne 0 ']' 2024-08-20T22:19:11.0405873Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-20T22:19:11.0406260Z + HAS_NVIDIA_DRIVER=1 2024-08-20T22:19:11.0407359Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-20T22:19:11.0408187Z + set -e 2024-08-20T22:19:11.0408606Z + '[' 1 -eq 0 ']' 2024-08-20T22:19:11.0409327Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-20T22:19:11.0410438Z + post_install_nvidia_driver_common 2024-08-20T22:19:11.0410993Z + sudo modprobe nvidia 2024-08-20T22:19:11.2204827Z + echo 'After installing NVIDIA driver' 2024-08-20T22:19:11.2205416Z + lspci 2024-08-20T22:19:11.2206246Z After installing NVIDIA driver 2024-08-20T22:19:11.2309358Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:19:11.2310364Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:19:11.2311568Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:19:11.2312379Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:19:11.2313222Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:19:11.2313995Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:19:11.2314709Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:19:11.2315529Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:19:11.2316264Z + lsmod 2024-08-20T22:19:11.2343767Z Module Size Used by 2024-08-20T22:19:11.2344237Z nvidia_uvm 4706304 0 2024-08-20T22:19:11.2344745Z nvidia 54071296 1 nvidia_uvm 2024-08-20T22:19:11.2345258Z drm 602112 1 nvidia 2024-08-20T22:19:11.2345773Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:19:11.2346341Z backlight 24576 1 drm 2024-08-20T22:19:11.2346795Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:19:11.2347221Z xt_conntrack 16384 1 2024-08-20T22:19:11.2347597Z nft_chain_nat 16384 3 2024-08-20T22:19:11.2347980Z xt_MASQUERADE 20480 1 2024-08-20T22:19:11.2348445Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:19:11.2348940Z nf_conntrack_netlink 57344 0 2024-08-20T22:19:11.2349512Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:19:11.2350386Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:19:11.2350853Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:19:11.2351281Z xfrm_user 57344 1 2024-08-20T22:19:11.2351684Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:19:11.2352106Z xt_addrtype 16384 2 2024-08-20T22:19:11.2352489Z nft_compat 20480 4 2024-08-20T22:19:11.2352946Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:19:11.2353561Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:19:11.2354105Z br_netfilter 36864 0 2024-08-20T22:19:11.2354520Z bridge 307200 1 br_netfilter 2024-08-20T22:19:11.2354964Z stp 16384 1 bridge 2024-08-20T22:19:11.2355394Z llc 16384 2 bridge,stp 2024-08-20T22:19:11.2355817Z overlay 167936 0 2024-08-20T22:19:11.2356189Z tls 114688 0 2024-08-20T22:19:11.2356564Z nls_ascii 16384 1 2024-08-20T22:19:11.2356947Z nls_cp437 20480 1 2024-08-20T22:19:11.2357320Z vfat 24576 1 2024-08-20T22:19:11.2357783Z fat 86016 1 vfat 2024-08-20T22:19:11.2358181Z sunrpc 692224 1 2024-08-20T22:19:11.2358562Z ena 167936 0 2024-08-20T22:19:11.2358940Z ptp 36864 1 ena 2024-08-20T22:19:11.2359337Z pps_core 24576 1 ptp 2024-08-20T22:19:11.2359741Z ghash_clmulni_intel 16384 0 2024-08-20T22:19:11.2360128Z aesni_intel 393216 0 2024-08-20T22:19:11.2360499Z i8042 45056 0 2024-08-20T22:19:11.2360869Z serio 28672 3 i8042 2024-08-20T22:19:11.2361461Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:19:11.2361992Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:19:11.2362576Z button 24576 0 2024-08-20T22:19:11.2363064Z sch_fq_codel 20480 17 2024-08-20T22:19:11.2363593Z dm_mod 188416 0 2024-08-20T22:19:11.2364023Z fuse 163840 1 2024-08-20T22:19:11.2364389Z configfs 57344 1 2024-08-20T22:19:11.2364761Z dax 45056 1 dm_mod 2024-08-20T22:19:11.2365164Z loop 36864 0 2024-08-20T22:19:11.2365534Z dmi_sysfs 20480 0 2024-08-20T22:19:11.2365909Z crc32_pclmul 16384 0 2024-08-20T22:19:11.2366272Z crc32c_intel 24576 0 2024-08-20T22:19:11.2366644Z efivarfs 24576 1 2024-08-20T22:19:11.2367005Z + modinfo nvidia 2024-08-20T22:19:11.2367684Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:19:11.2368385Z alias: char-major-195-* 2024-08-20T22:19:11.2368768Z version: 550.54.15 2024-08-20T22:19:11.2369117Z supported: external 2024-08-20T22:19:11.2369456Z license: NVIDIA 2024-08-20T22:19:11.2369911Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:19:11.2370403Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:19:11.2370863Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:19:11.2371320Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:19:11.2371820Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:19:11.2372364Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:19:11.2372864Z depends: i2c-core,drm 2024-08-20T22:19:11.2373228Z retpoline: Y 2024-08-20T22:19:11.2373546Z name: nvidia 2024-08-20T22:19:11.2374175Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:19:11.2374870Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:19:11.2375521Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:19:11.2376112Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:19:11.2376560Z parm: NVreg_RmLogonRC:int 2024-08-20T22:19:11.2376988Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:19:11.2377585Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:19:11.2378024Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:19:11.2378466Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:19:11.2378990Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:19:11.2379551Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:19:11.2380032Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:19:11.2380469Z parm: NVreg_EnableMSI:int 2024-08-20T22:19:11.2380882Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:19:11.2381340Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:19:11.2381871Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:19:11.2382505Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:19:11.2383055Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:19:11.2383664Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:11.2384268Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:19:11.2384880Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:11.2385465Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:19:11.2385956Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:19:11.2386492Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:19:11.2387040Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:19:11.2387528Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:19:11.2387994Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:19:11.2388477Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:19:11.2388948Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:19:11.2389492Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:19:11.2390005Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:19:11.2390532Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:19:11.2391028Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:19:11.2391517Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:19:11.2392021Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:19:11.2392509Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:19:11.2393006Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:19:11.2393490Z parm: NVreg_RmMsg:charp 2024-08-20T22:19:11.2393905Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:19:11.2394375Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:19:11.2394842Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:19:11.2395300Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:19:11.2395789Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:19:11.2396269Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:19:11.2396720Z parm: rm_firmware_active:charp 2024-08-20T22:19:11.2397123Z + set +e 2024-08-20T22:19:11.2397440Z + nvidia-smi 2024-08-20T22:19:12.8010645Z Tue Aug 20 22:19:12 2024 2024-08-20T22:19:12.8011375Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:12.8012204Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:19:12.8013005Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:12.8013810Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:19:12.8014703Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:19:12.8015399Z | | | MIG M. | 2024-08-20T22:19:12.8015972Z |=========================================+========================+======================| 2024-08-20T22:19:12.8110217Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:19:12.8111225Z | 0% 29C P0 58W / 300W | 0MiB / 23028MiB | 5% Default | 2024-08-20T22:19:12.8111916Z | | | N/A | 2024-08-20T22:19:12.8112603Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:12.8113205Z 2024-08-20T22:19:12.8113866Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:12.8114508Z | Processes: | 2024-08-20T22:19:12.8115215Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:19:12.8115918Z | ID ID Usage | 2024-08-20T22:19:12.8116496Z |=========================================================================================| 2024-08-20T22:19:12.8117161Z | No running processes found | 2024-08-20T22:19:12.8118043Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:13.4111651Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-20T22:19:14.9781731Z NVIDIA A10G 2024-08-20T22:19:15.4346480Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:19:15.4347118Z + '[' 0 -eq 0 ']' 2024-08-20T22:19:15.4347533Z + echo 'INFO: Ignoring allowed status 0' 2024-08-20T22:19:15.4347969Z + set -e 2024-08-20T22:19:15.4348275Z INFO: Ignoring allowed status 0 2024-08-20T22:19:15.4355435Z == Installing nvidia container toolkit for amzn2023 == 2024-08-20T22:19:15.4360065Z + sudo yum install -y yum-utils 2024-08-20T22:19:15.8371702Z Last metadata expiration check: 0:09:36 ago on Tue Aug 20 22:09:39 2024. 2024-08-20T22:19:15.8595162Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-08-20T22:19:15.8897725Z Dependencies resolved. 2024-08-20T22:19:15.9023656Z Nothing to do. 2024-08-20T22:19:15.9024080Z Complete! 2024-08-20T22:19:15.9867877Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-08-20T22:19:15.9869150Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:15.9870634Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:16.2789257Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:16.3567080Z + sudo yum install -y nvidia-docker2 2024-08-20T22:19:16.8740968Z nvidia-container-toolkit 10 kB/s | 833 B 00:00 2024-08-20T22:19:16.8966066Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-08-20T22:19:16.9263781Z Dependencies resolved. 2024-08-20T22:19:16.9388305Z Nothing to do. 2024-08-20T22:19:16.9388938Z Complete! 2024-08-20T22:19:17.0353633Z + sudo systemctl restart docker 2024-08-20T22:20:12.1509515Z Tue Aug 20 22:20:12 2024 2024-08-20T22:20:12.1510795Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:12.1511877Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:20:12.1512734Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:20:12.1513545Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:20:12.1515285Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:20:12.1516230Z | | | MIG M. | 2024-08-20T22:20:12.1516891Z |=========================================+========================+======================| 2024-08-20T22:20:12.1630233Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:20:12.1631521Z | 0% 29C P0 58W / 300W | 0MiB / 23028MiB | 5% Default | 2024-08-20T22:20:12.1632348Z | | | N/A | 2024-08-20T22:20:12.1633280Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:20:12.1633928Z 2024-08-20T22:20:12.1634824Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:12.1635568Z | Processes: | 2024-08-20T22:20:12.1636315Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:20:12.1637159Z | ID ID Usage | 2024-08-20T22:20:12.1637855Z |=========================================================================================| 2024-08-20T22:20:12.1638595Z | No running processes found | 2024-08-20T22:20:12.1639376Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:13.1956759Z Command completed after 1 attempt(s). 2024-08-20T22:20:13.2036914Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:13.2037692Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:13.2038497Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T22:20:13.2039143Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:13.2052891Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:13.2053389Z env: 2024-08-20T22:20:13.2053682Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:13.2054202Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:13.2054753Z ##[endgroup] 2024-08-20T22:20:13.5083939Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:13.9226855Z Collecting psutil==5.9.1 2024-08-20T22:20:13.9584907Z Downloading psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-08-20T22:20:14.0394876Z Collecting nvidia-ml-py==11.525.84 2024-08-20T22:20:14.0438523Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-08-20T22:20:14.1267365Z Installing collected packages: psutil, nvidia-ml-py 2024-08-20T22:20:14.2840799Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-08-20T22:20:14.4801119Z Prepare all required actions 2024-08-20T22:20:14.4801636Z Getting action download info 2024-08-20T22:20:14.6155590Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T22:20:14.7827388Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T22:20:14.9047962Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T22:20:14.9048452Z with: 2024-08-20T22:20:14.9048795Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:14.9049247Z s3-bucket: gha-artifacts 2024-08-20T22:20:14.9049597Z env: 2024-08-20T22:20:14.9049884Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:14.9050331Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:14.9050820Z ##[endgroup] 2024-08-20T22:20:14.9084752Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:14.9085187Z with: 2024-08-20T22:20:14.9085564Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:14.9086022Z s3-bucket: gha-artifacts 2024-08-20T22:20:14.9086366Z region: us-east-1 2024-08-20T22:20:14.9086670Z env: 2024-08-20T22:20:14.9086952Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:14.9087588Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:14.9088081Z ##[endgroup] 2024-08-20T22:20:15.3830903Z (node:52502) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:15.3832258Z 2024-08-20T22:20:15.3832766Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:15.3834192Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:15.3835671Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:15.4455111Z Found 1 objects with prefix pytorch/pytorch/10479310725/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-08-20T22:20:15.4456234Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:25.0782288Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:25.0788598Z Artifact download has finished successfully 2024-08-20T22:20:25.1138486Z ##[group]Run unzip -o artifacts.zip 2024-08-20T22:20:25.1138937Z unzip -o artifacts.zip 2024-08-20T22:20:25.1147766Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:25.1148279Z env: 2024-08-20T22:20:25.1148586Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:25.1149045Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:25.1149542Z ##[endgroup] 2024-08-20T22:20:25.1242806Z Archive: artifacts.zip 2024-08-20T22:20:25.1244248Z creating: dist/ 2024-08-20T22:20:27.2566686Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:20:27.2567376Z creating: build/custom_test_artifacts/ 2024-08-20T22:20:27.2567985Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T22:20:27.2568754Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T22:20:27.2569602Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:27.2576000Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:27.2576965Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:27.2577916Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:27.2578920Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:27.2579897Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:27.2581008Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:27.2582731Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:27.2583775Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:27.2584794Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:27.2586367Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:27.2588065Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:27.2590333Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:27.2591480Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:27.2592772Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:27.2594257Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:27.2595307Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:27.2596327Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:27.2636354Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:27.2676295Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:27.2677738Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:27.2723474Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:27.2724902Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:27.2726339Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:27.2727805Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:27.2729233Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:27.2730621Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:27.2732017Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:27.2733406Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:27.2734761Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:27.2736046Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:27.2737297Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:27.2738524Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:27.2739761Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:27.2740956Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:27.2742176Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:27.2799857Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:27.2859054Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:27.2860260Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:27.2861268Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:27.2862155Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:27.2863072Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:27.2864014Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T22:20:27.2865052Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:27.2866210Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T22:20:27.2867325Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T22:20:27.2868379Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T22:20:27.2869628Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:27.2870718Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T22:20:27.2871807Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:27.2872887Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T22:20:27.2874011Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T22:20:27.2891986Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T22:20:27.3035803Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T22:20:27.3036813Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T22:20:27.3037927Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:27.3039237Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T22:20:27.3040405Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T22:20:27.3041498Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T22:20:27.3042638Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:27.3043789Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T22:20:27.3044937Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:27.3046080Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T22:20:27.3047203Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T22:20:27.3065733Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T22:20:27.3150034Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T22:20:27.3152419Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:27.3154225Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:27.3155380Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T22:20:27.3156319Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T22:20:27.3157231Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:27.3158227Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-20T22:20:27.3159095Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T22:20:27.3159885Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T22:20:27.3160713Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T22:20:27.3280329Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T22:20:27.3342928Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T22:20:27.3343683Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T22:20:27.3344399Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T22:20:27.3345229Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:27.3351902Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:27.3353033Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:27.3353970Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:27.3354964Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:27.3355939Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:27.3357041Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:27.3358396Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:27.3359402Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:27.3360400Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:27.3361795Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:27.3363459Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:27.3365669Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:27.3366805Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:27.3368088Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:27.3369453Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:27.3370499Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:27.3371506Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:27.3411373Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:27.3451111Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:27.3452540Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:27.3497927Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:27.3499334Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:27.3500962Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:27.3502428Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:27.3503885Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:27.3505269Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:27.3506658Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:27.3508039Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:27.3509382Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:27.3510834Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:27.3512068Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:27.3513608Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:27.3515015Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:27.3516370Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:27.3517765Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:27.3573473Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:27.3634264Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:27.3635450Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:27.3636440Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:27.3637313Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:27.3638285Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:27.3639242Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T22:20:27.3640317Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T22:20:27.3641506Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T22:20:27.3642657Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T22:20:27.3643739Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T22:20:27.3644862Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T22:20:27.3645985Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T22:20:27.3647100Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T22:20:27.3648211Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T22:20:27.3649316Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T22:20:27.3666447Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T22:20:27.3731087Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T22:20:27.3732279Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:27.3733361Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:27.3734338Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T22:20:27.3735249Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T22:20:27.3736283Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:27.3737191Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-20T22:20:27.3739951Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T22:20:27.3740932Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T22:20:27.3741758Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T22:20:27.3792086Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T22:20:27.3793611Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T22:20:27.3794640Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T22:20:27.3795534Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:27.3800303Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:27.3801301Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:27.3802312Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:27.3803378Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:27.3804427Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:27.3805597Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:27.3806780Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:27.3807854Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:27.3808915Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:27.3810504Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:27.3812295Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:27.3814175Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:27.3815386Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:27.3817066Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:27.3818494Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:27.3819603Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:27.3820675Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:27.3860233Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:27.3899701Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:27.3901360Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:27.3946823Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:27.3948305Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:27.3949790Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:27.3951301Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:27.3952778Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:27.3954281Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:27.3955752Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:27.3957356Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:27.3958854Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:27.3960196Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:27.3961493Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:27.3962783Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:27.3964089Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:27.3965339Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:27.3966629Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:27.4022638Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:27.4082938Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:27.4084179Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:27.4085234Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:27.4086157Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:27.4087123Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:27.4088134Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T22:20:27.4089268Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:27.4090513Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T22:20:27.4091724Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T22:20:27.4092866Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T22:20:27.4094115Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:27.4095452Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T22:20:27.4096646Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:27.4097836Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T22:20:27.4099028Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T22:20:27.4100280Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T22:20:27.4224712Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T22:20:27.4225886Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T22:20:27.4227080Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:27.4228402Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T22:20:27.4229668Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T22:20:27.4231064Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T22:20:27.4232307Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:27.4233555Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T22:20:27.4234805Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:27.4236047Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T22:20:27.4237281Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T22:20:27.4254293Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T22:20:27.4310214Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T22:20:27.4311510Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:27.4312657Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:27.4313698Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T22:20:27.4314675Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T22:20:27.4315645Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:27.4316613Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-20T22:20:27.4318897Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T22:20:27.4319947Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T22:20:27.4320938Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T22:20:27.4424076Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T22:20:27.4468004Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T22:20:27.4469284Z creating: build/lib/ 2024-08-20T22:20:27.4556553Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T22:20:27.5013620Z inflating: build/lib/libprotobuf.a 2024-08-20T22:20:27.5023324Z inflating: build/lib/libpthreadpool.a 2024-08-20T22:20:27.5031657Z inflating: build/lib/libcpuinfo.a 2024-08-20T22:20:27.5039946Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T22:20:27.5040622Z inflating: build/lib/libclog.a 2024-08-20T22:20:27.5059327Z inflating: build/lib/libnnpack.a 2024-08-20T22:20:27.5062073Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T22:20:27.5127457Z inflating: build/lib/libgtest.a 2024-08-20T22:20:27.5202394Z inflating: build/lib/libbenchmark.a 2024-08-20T22:20:27.5266890Z inflating: build/lib/libasmjit.a 2024-08-20T22:20:27.5274267Z inflating: build/lib/libittnotify.a 2024-08-20T22:20:27.5302033Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T22:20:27.5429302Z inflating: build/lib/libgloo.a 2024-08-20T22:20:27.5450445Z inflating: build/lib/libfmt.a 2024-08-20T22:20:27.5545597Z inflating: build/lib/libc10.so 2024-08-20T22:20:27.5547535Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-20T22:20:27.5549108Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T22:20:27.5568701Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T22:20:27.6072742Z inflating: build/lib/libprotoc.a 2024-08-20T22:20:27.6090233Z inflating: build/lib/libgmock.a 2024-08-20T22:20:27.6090849Z inflating: build/lib/libgtest_main.a 2024-08-20T22:20:27.6091893Z inflating: build/lib/libbenchmark_main.a 2024-08-20T22:20:27.6480902Z inflating: build/lib/libgloo_cuda.a 2024-08-20T22:20:28.6682333Z inflating: build/lib/libdnnl.a 2024-08-20T22:20:28.7254276Z inflating: build/lib/libtensorpipe.a 2024-08-20T22:20:28.7311869Z inflating: build/lib/libc10_cuda.so 2024-08-20T22:20:28.7312554Z inflating: build/lib/libgmock_main.a 2024-08-20T22:20:28.8606216Z inflating: build/lib/libfbgemm.a 2024-08-20T22:20:28.8866481Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-20T22:20:28.9372612Z inflating: build/lib/libkineto.a 2024-08-20T22:20:28.9556817Z inflating: build/lib/libXNNPACK.a 2024-08-20T22:20:28.9599371Z inflating: build/lib/libonnx_proto.a 2024-08-20T22:20:29.0290725Z inflating: build/lib/libonnx.a 2024-08-20T22:20:31.4765656Z inflating: build/lib/libtorch_cpu.so 2024-08-20T22:20:31.4770583Z inflating: build/lib/libunbox_lib.a 2024-08-20T22:20:31.4775127Z inflating: build/lib/libshm.so 2024-08-20T22:20:33.5434385Z inflating: build/lib/libtorch_cuda.so 2024-08-20T22:20:33.5436650Z inflating: build/lib/libtorch.so 2024-08-20T22:20:33.5439358Z inflating: build/lib/libc10d_cuda_test.so 2024-08-20T22:20:34.3631106Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-20T22:20:34.5624328Z inflating: build/lib/libtorch_python.so 2024-08-20T22:20:34.5696554Z inflating: build/lib/libtorchbind_test.so 2024-08-20T22:20:34.5717648Z inflating: build/lib/libjitbackend_test.so 2024-08-20T22:20:34.5743864Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T22:20:34.5769440Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T22:20:34.5804172Z inflating: build/lib/libnnapi_backend.so 2024-08-20T22:20:34.5804657Z creating: build/bin/ 2024-08-20T22:20:34.5855475Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T22:20:34.5907020Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T22:20:34.5958621Z inflating: build/bin/c10_Device_test 2024-08-20T22:20:34.6017805Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T22:20:34.6071363Z inflating: build/bin/c10_Scalar_test 2024-08-20T22:20:34.6120897Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T22:20:34.6172244Z inflating: build/bin/c10_SymInt_test 2024-08-20T22:20:34.6226929Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T22:20:34.6282968Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T22:20:34.6339783Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T22:20:34.6411023Z inflating: build/bin/c10_cow_test 2024-08-20T22:20:34.6464651Z inflating: build/bin/c10_Bitset_test 2024-08-20T22:20:34.6513442Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T22:20:34.6564549Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T22:20:34.6614774Z inflating: build/bin/c10_Half_test 2024-08-20T22:20:34.6671828Z inflating: build/bin/c10_LeftRight_test 2024-08-20T22:20:34.6726726Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T22:20:34.6776696Z inflating: build/bin/c10_Synchronized_test 2024-08-20T22:20:34.6832647Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T22:20:34.6884264Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T22:20:34.6935781Z inflating: build/bin/c10_TypeList_test 2024-08-20T22:20:34.6984847Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T22:20:34.7037286Z inflating: build/bin/c10_accumulate_test 2024-08-20T22:20:34.7092799Z inflating: build/bin/c10_bfloat16_test 2024-08-20T22:20:34.7143896Z inflating: build/bin/c10_bit_cast_test 2024-08-20T22:20:34.7200496Z inflating: build/bin/c10_complex_math_test 2024-08-20T22:20:34.7256318Z inflating: build/bin/c10_complex_test 2024-08-20T22:20:34.7309251Z inflating: build/bin/c10_exception_test 2024-08-20T22:20:34.7360556Z inflating: build/bin/c10_flags_test 2024-08-20T22:20:34.7410617Z inflating: build/bin/c10_generic_math_test 2024-08-20T22:20:34.7576281Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T22:20:34.7627523Z inflating: build/bin/c10_irange_test 2024-08-20T22:20:34.7681904Z inflating: build/bin/c10_lazy_test 2024-08-20T22:20:34.7739247Z inflating: build/bin/c10_logging_test 2024-08-20T22:20:34.7801823Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T22:20:34.7877597Z inflating: build/bin/c10_optional_test 2024-08-20T22:20:34.8028851Z inflating: build/bin/c10_small_vector_test 2024-08-20T22:20:34.8083389Z inflating: build/bin/c10_registry_test 2024-08-20T22:20:34.8135291Z inflating: build/bin/c10_ssize_test 2024-08-20T22:20:34.8187636Z inflating: build/bin/c10_string_util_test 2024-08-20T22:20:34.8239072Z inflating: build/bin/c10_tempfile_test 2024-08-20T22:20:34.8298351Z inflating: build/bin/c10_string_view_test 2024-08-20T22:20:34.8347326Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T22:20:34.8403828Z inflating: build/bin/c10_typeid_test 2024-08-20T22:20:34.8852478Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T22:20:34.9300422Z inflating: build/bin/protoc 2024-08-20T22:20:34.9354477Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-20T22:20:34.9406886Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-20T22:20:34.9460600Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-20T22:20:34.9512966Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-20T22:20:34.9566327Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-20T22:20:34.9619477Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-20T22:20:34.9669428Z inflating: build/bin/c10_cuda_CUDATest 2024-08-20T22:20:34.9722177Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-20T22:20:35.0053209Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T22:20:35.0399233Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T22:20:35.0759573Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T22:20:35.0812340Z inflating: build/bin/BackoffTest 2024-08-20T22:20:35.0865632Z inflating: build/bin/FileStoreTest 2024-08-20T22:20:35.0921796Z inflating: build/bin/TCPStoreTest 2024-08-20T22:20:35.0975509Z inflating: build/bin/HashStoreTest 2024-08-20T22:20:35.0989296Z inflating: build/bin/ProcessGroupMPITest 2024-08-20T22:20:35.1043827Z inflating: build/bin/test_edge_op_registration 2024-08-20T22:20:35.1048431Z inflating: build/bin/torch_shm_manager 2024-08-20T22:20:35.1051574Z inflating: build/bin/example_allreduce 2024-08-20T22:20:35.1106851Z inflating: build/bin/test_dist_autograd 2024-08-20T22:20:35.1176274Z inflating: build/bin/test_cpp_rpc 2024-08-20T22:20:35.1179221Z inflating: build/bin/parallel_benchmark 2024-08-20T22:20:35.1247768Z inflating: build/bin/test_mobile_nnc 2024-08-20T22:20:35.1256870Z inflating: build/bin/aot_model_compiler_test 2024-08-20T22:20:35.1604738Z inflating: build/bin/test_lazy 2024-08-20T22:20:35.2788688Z inflating: build/bin/test_api 2024-08-20T22:20:35.2863000Z inflating: build/bin/Dict_test 2024-08-20T22:20:35.2915485Z inflating: build/bin/Dimname_test 2024-08-20T22:20:35.2980300Z inflating: build/bin/MaybeOwned_test 2024-08-20T22:20:35.3038385Z inflating: build/bin/NamedTensor_test 2024-08-20T22:20:35.3097573Z inflating: build/bin/apply_utils_test 2024-08-20T22:20:35.3157475Z inflating: build/bin/atest 2024-08-20T22:20:35.3220735Z inflating: build/bin/basic 2024-08-20T22:20:35.3276130Z inflating: build/bin/broadcast_test 2024-08-20T22:20:35.3327314Z inflating: build/bin/cpu_allocator_test 2024-08-20T22:20:35.3385539Z inflating: build/bin/cpu_generator_test 2024-08-20T22:20:35.3439554Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T22:20:35.3533434Z inflating: build/bin/cpu_rng_test 2024-08-20T22:20:35.3584269Z inflating: build/bin/dispatch_key_set_test 2024-08-20T22:20:35.3635305Z inflating: build/bin/dlconvertor_test 2024-08-20T22:20:35.3693645Z inflating: build/bin/extension_backend_test 2024-08-20T22:20:35.3748874Z inflating: build/bin/half_test 2024-08-20T22:20:35.3845796Z inflating: build/bin/ivalue_test 2024-08-20T22:20:35.3895885Z inflating: build/bin/lazy_tensor_test 2024-08-20T22:20:35.3951292Z inflating: build/bin/math_kernel_test 2024-08-20T22:20:35.4005252Z inflating: build/bin/memory_format_test 2024-08-20T22:20:35.4058550Z inflating: build/bin/memory_overlapping_test 2024-08-20T22:20:35.4112280Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T22:20:35.4168850Z inflating: build/bin/native_test 2024-08-20T22:20:35.4220256Z inflating: build/bin/operator_name_test 2024-08-20T22:20:35.4272099Z inflating: build/bin/operators_test 2024-08-20T22:20:35.4324392Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T22:20:35.4391331Z inflating: build/bin/pow_test 2024-08-20T22:20:35.4449434Z inflating: build/bin/quantized_test 2024-08-20T22:20:35.4499376Z inflating: build/bin/reduce_ops_test 2024-08-20T22:20:35.4550914Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T22:20:35.4608626Z inflating: build/bin/scalar_tensor_test 2024-08-20T22:20:35.4667072Z inflating: build/bin/scalar_test 2024-08-20T22:20:35.4719417Z inflating: build/bin/StorageUtils_test 2024-08-20T22:20:35.4772754Z inflating: build/bin/stride_properties_test 2024-08-20T22:20:35.4852235Z inflating: build/bin/tensor_iterator_test 2024-08-20T22:20:35.4906935Z inflating: build/bin/test_parallel 2024-08-20T22:20:35.4909561Z inflating: build/bin/thread_init_test 2024-08-20T22:20:35.4965850Z inflating: build/bin/type_ptr_test 2024-08-20T22:20:35.5026444Z inflating: build/bin/type_test 2024-08-20T22:20:35.5079411Z inflating: build/bin/undefined_tensor_test 2024-08-20T22:20:35.5081118Z inflating: build/bin/verify_api_visibility 2024-08-20T22:20:35.5151201Z inflating: build/bin/legacy_vmap_test 2024-08-20T22:20:35.5203553Z inflating: build/bin/weakref_test 2024-08-20T22:20:35.5255605Z inflating: build/bin/wrapdim_test 2024-08-20T22:20:35.5308033Z inflating: build/bin/xla_tensor_test 2024-08-20T22:20:35.5368236Z inflating: build/bin/IListRef_test 2024-08-20T22:20:35.5475556Z inflating: build/bin/List_test 2024-08-20T22:20:35.5542633Z inflating: build/bin/KernelFunction_test 2024-08-20T22:20:35.5665666Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T22:20:35.5762441Z inflating: build/bin/kernel_function_test 2024-08-20T22:20:35.5891898Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T22:20:35.5996792Z inflating: build/bin/kernel_lambda_test 2024-08-20T22:20:35.6057980Z inflating: build/bin/kernel_stackbased_test 2024-08-20T22:20:35.6155170Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T22:20:35.6207017Z inflating: build/bin/CppSignature_test 2024-08-20T22:20:35.6263821Z inflating: build/bin/backend_fallback_test 2024-08-20T22:20:35.6312777Z inflating: build/bin/op_allowlist_test 2024-08-20T22:20:35.6616448Z inflating: build/bin/op_registration_test 2024-08-20T22:20:35.6680601Z inflating: build/bin/inline_container_test 2024-08-20T22:20:35.6733882Z inflating: build/bin/cuda_apply_test 2024-08-20T22:20:35.6786921Z inflating: build/bin/cuda_allocator_test 2024-08-20T22:20:35.6846119Z inflating: build/bin/cuda_atomic_ops_test 2024-08-20T22:20:35.6901066Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-20T22:20:35.6971922Z inflating: build/bin/cuda_complex_math_test 2024-08-20T22:20:35.7031222Z inflating: build/bin/cuda_complex_test 2024-08-20T22:20:35.7081402Z inflating: build/bin/cuda_device_test 2024-08-20T22:20:35.7139279Z inflating: build/bin/cuda_cub_test 2024-08-20T22:20:35.7191323Z inflating: build/bin/cuda_dlconvertor_test 2024-08-20T22:20:35.7256102Z inflating: build/bin/cuda_distributions_test 2024-08-20T22:20:35.7313595Z inflating: build/bin/cuda_generator_test 2024-08-20T22:20:35.7363069Z inflating: build/bin/cuda_half_test 2024-08-20T22:20:35.7414562Z inflating: build/bin/cuda_integer_divider_test 2024-08-20T22:20:35.7464230Z inflating: build/bin/cuda_optional_test 2024-08-20T22:20:35.7516285Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-20T22:20:35.7569628Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-20T22:20:35.7619780Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-20T22:20:35.7680778Z inflating: build/bin/cuda_stream_test 2024-08-20T22:20:35.7730885Z inflating: build/bin/cuda_cudnn_test 2024-08-20T22:20:35.7783210Z inflating: build/bin/cuda_vectorized_test 2024-08-20T22:20:35.7797763Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T22:20:35.7864200Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T22:20:35.7922118Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-20T22:20:35.7986567Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-20T22:20:35.8048531Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-20T22:20:35.8880156Z inflating: build/bin/test_tensorexpr 2024-08-20T22:20:35.9464147Z inflating: build/bin/test_jit 2024-08-20T22:20:35.9465054Z creating: .additional_ci_files/ 2024-08-20T22:20:35.9525136Z inflating: .additional_ci_files/test-times.json 2024-08-20T22:20:35.9765585Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T22:20:35.9817665Z ##[group]Run rm artifacts.zip 2024-08-20T22:20:35.9818068Z rm artifacts.zip 2024-08-20T22:20:35.9828561Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:35.9829077Z env: 2024-08-20T22:20:35.9829368Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:35.9829879Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:35.9830372Z ##[endgroup] 2024-08-20T22:20:36.1259651Z ##[group]Run df -H 2024-08-20T22:20:36.1259975Z df -H 2024-08-20T22:20:36.1269063Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:36.1269565Z env: 2024-08-20T22:20:36.1269845Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.1270302Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.1270791Z ##[endgroup] 2024-08-20T22:20:36.1319266Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T22:20:36.1319800Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T22:20:36.1320277Z tmpfs 34G 0 34G 0% /dev/shm 2024-08-20T22:20:36.1320746Z tmpfs 14G 553k 14G 1% /run 2024-08-20T22:20:36.1321190Z /dev/nvme0n1p1 161G 53G 109G 33% / 2024-08-20T22:20:36.1321871Z tmpfs 34G 4.1k 34G 1% /tmp 2024-08-20T22:20:36.1322374Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T22:20:36.1322902Z tmpfs 6.7G 0 6.7G 0% /run/user/0 2024-08-20T22:20:36.1362466Z Prepare all required actions 2024-08-20T22:20:36.1362902Z Getting action download info 2024-08-20T22:20:36.2408595Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T22:20:36.2409060Z with: 2024-08-20T22:20:36.2409316Z env: 2024-08-20T22:20:36.2409597Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.2410437Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.2410947Z ##[endgroup] 2024-08-20T22:20:36.2445204Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:36.2445656Z with: 2024-08-20T22:20:36.2445943Z name: td_results 2024-08-20T22:20:36.2446278Z s3-bucket: gha-artifacts 2024-08-20T22:20:36.2446652Z region: us-east-1 2024-08-20T22:20:36.2446974Z env: 2024-08-20T22:20:36.2447274Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.2447737Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.2448240Z ##[endgroup] 2024-08-20T22:20:36.7203132Z (node:52526) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:36.7204099Z 2024-08-20T22:20:36.7204372Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:36.7205093Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:36.7205964Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:36.7854306Z Found 0 objects with prefix pytorch/pytorch/10479310725/td_results/ 2024-08-20T22:20:36.7861716Z Artifact download has finished successfully 2024-08-20T22:20:36.8184406Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T22:20:36.8184895Z mkdir -p .additional_ci_files 2024-08-20T22:20:36.8185465Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T22:20:36.8194536Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:36.8195044Z env: 2024-08-20T22:20:36.8195340Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.8195822Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.8196325Z ##[endgroup] 2024-08-20T22:20:36.8253126Z mv: cannot stat 'td_results.json': No such file or directory 2024-08-20T22:20:36.8266338Z ##[error]Process completed with exit code 1. 2024-08-20T22:20:36.8354974Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T22:20:36.8355459Z .github/scripts/parse_ref.py 2024-08-20T22:20:36.8364344Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:36.8364702Z env: 2024-08-20T22:20:36.8364982Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.8365439Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.8365924Z ##[endgroup] 2024-08-20T22:20:36.8650624Z Prepare all required actions 2024-08-20T22:20:36.8712353Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T22:20:36.8712805Z with: 2024-08-20T22:20:36.8713358Z github-token: *** 2024-08-20T22:20:36.8713670Z env: 2024-08-20T22:20:36.8713950Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.8714408Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.8714903Z ##[endgroup] 2024-08-20T22:20:36.8832312Z ##[group]Run set -eux 2024-08-20T22:20:36.8832662Z set -eux 2024-08-20T22:20:36.8833252Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T22:20:36.8842202Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:36.8842695Z env: 2024-08-20T22:20:36.8842982Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.8843434Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.8844072Z GITHUB_TOKEN: *** 2024-08-20T22:20:36.8844390Z ##[endgroup] 2024-08-20T22:20:36.8873254Z + python3 .github/scripts/get_workflow_job_id.py 10479310725 i-0573eecbda81770a3 2024-08-20T22:20:38.5210970Z setting job-id=29026529883 2024-08-20T22:20:38.5212083Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:38.5452548Z Prepare all required actions 2024-08-20T22:20:38.5452999Z Getting action download info 2024-08-20T22:20:38.6545338Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T22:20:38.6545993Z with: 2024-08-20T22:20:38.6546494Z github-token: *** 2024-08-20T22:20:38.6555306Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:20:38.6565065Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:38.6565831Z env: 2024-08-20T22:20:38.6566120Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:38.6566576Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:38.6567056Z ##[endgroup] 2024-08-20T22:20:38.6707257Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:20:38.6707814Z with: 2024-08-20T22:20:38.6708091Z shell: bash 2024-08-20T22:20:38.6708401Z timeout_minutes: 10 2024-08-20T22:20:38.6708735Z max_attempts: 5 2024-08-20T22:20:38.6709063Z retry_wait_seconds: 30 2024-08-20T22:20:38.6710498Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:38.6711699Z polling_interval_seconds: 1 2024-08-20T22:20:38.6712079Z warning_on_retry: true 2024-08-20T22:20:38.6712427Z continue_on_error: false 2024-08-20T22:20:38.6712755Z env: 2024-08-20T22:20:38.6713027Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:38.6713469Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:38.6714100Z GITHUB_TOKEN: *** 2024-08-20T22:20:38.6714408Z ##[endgroup] 2024-08-20T22:20:38.7475775Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:38.9868831Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:39.1645428Z Collecting requests==2.27.1 2024-08-20T22:20:39.1970139Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-08-20T22:20:39.4323658Z Collecting pyyaml==6.0.1 2024-08-20T22:20:39.4359478Z Downloading PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (738 kB) 2024-08-20T22:20:39.4912014Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-08-20T22:20:39.5555762Z Collecting certifi>=2017.4.17 2024-08-20T22:20:39.5591164Z Downloading certifi-2024.7.4-py3-none-any.whl (162 kB) 2024-08-20T22:20:39.9085556Z Collecting charset-normalizer~=2.0.0 2024-08-20T22:20:39.9126481Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-08-20T22:20:39.9339557Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-08-20T22:20:40.0096706Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2024-08-20T22:20:40.3082897Z Successfully installed certifi-2024.7.4 charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1 2024-08-20T22:20:40.7255898Z Command completed after 1 attempt(s). 2024-08-20T22:20:40.7317853Z ##[group]Run set -x 2024-08-20T22:20:40.7318322Z set -x 2024-08-20T22:20:40.7318632Z  2024-08-20T22:20:40.7319185Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:40.7319864Z # in runner workspace 2024-08-20T22:20:40.7320404Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T22:20:40.7329569Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:40.7330075Z env: 2024-08-20T22:20:40.7330365Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:40.7330825Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:40.7331307Z ##[endgroup] 2024-08-20T22:20:40.7362215Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T22:20:40.7616224Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:40.7616786Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:40.7617264Z echo "Job name: ${JOB_NAME}" 2024-08-20T22:20:40.7617677Z  2024-08-20T22:20:40.7618234Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:40.7618921Z # in runner workspace 2024-08-20T22:20:40.7619509Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T22:20:40.7620148Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T22:20:40.7620626Z  --job-name "${JOB_NAME}" \ 2024-08-20T22:20:40.7629862Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-08-20T22:20:40.7639231Z  --selected-test-configs "" \ 2024-08-20T22:20:40.7639718Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T22:20:40.7640155Z  --tag "${TAG}" \ 2024-08-20T22:20:40.7640722Z  --event-name "${EVENT_NAME}" \ 2024-08-20T22:20:40.7641176Z  --schedule "${SCHEDULE}" \ 2024-08-20T22:20:40.7641614Z  --branch "${HEAD_BRANCH}" 2024-08-20T22:20:40.7650480Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:40.7651003Z env: 2024-08-20T22:20:40.7651310Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:40.7651791Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:40.7652480Z GITHUB_TOKEN: *** 2024-08-20T22:20:40.7653202Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:40.7653981Z PR_NUMBER: 2024-08-20T22:20:40.7654303Z TAG: ciflow/inductor/133712 2024-08-20T22:20:40.7654813Z EVENT_NAME: push 2024-08-20T22:20:40.7655141Z SCHEDULE: 2024-08-20T22:20:40.7655438Z HEAD_BRANCH: 2024-08-20T22:20:40.7655759Z ##[endgroup] 2024-08-20T22:20:40.7685340Z Workflow: inductor 2024-08-20T22:20:40.7686354Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:41.0072875Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T22:20:41.1561714Z ##[group]Run echo "Filtered matrix:" 2024-08-20T22:20:41.1562183Z echo "Filtered matrix:" 2024-08-20T22:20:41.1571276Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" 2024-08-20T22:20:41.1580444Z  2024-08-20T22:20:41.1580733Z echo 2024-08-20T22:20:41.1581128Z echo "Is the current job unstable? False" 2024-08-20T22:20:41.1581596Z  2024-08-20T22:20:41.1581891Z echo 2024-08-20T22:20:41.1582267Z echo "Is keep-going label set? False" 2024-08-20T22:20:41.1582722Z  2024-08-20T22:20:41.1583039Z echo 2024-08-20T22:20:41.1583396Z echo "Renabled issues? " 2024-08-20T22:20:41.1592387Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:41.1592891Z env: 2024-08-20T22:20:41.1593228Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:41.1593714Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:41.1594210Z ##[endgroup] 2024-08-20T22:20:41.1624369Z Filtered matrix: 2024-08-20T22:20:41.1634774Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: amz2023.linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}]} 2024-08-20T22:20:41.1643732Z 2024-08-20T22:20:41.1643908Z Is the current job unstable? False 2024-08-20T22:20:41.1644203Z 2024-08-20T22:20:41.1644585Z Is keep-going label set? False 2024-08-20T22:20:41.1644843Z 2024-08-20T22:20:41.1644985Z Renabled issues? 2024-08-20T22:20:41.1702169Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:41.1702886Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:41.1711577Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:41.1712098Z env: 2024-08-20T22:20:41.1712402Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:41.1712884Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:41.1713393Z JOB_TIMEOUT: 240 2024-08-20T22:20:41.1713728Z ##[endgroup] 2024-08-20T22:20:41.1803817Z ##[group]Run set -x 2024-08-20T22:20:41.1804233Z set -x 2024-08-20T22:20:41.1804544Z  2024-08-20T22:20:41.1804904Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T22:20:41.1805442Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T22:20:41.1806004Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T22:20:41.1806515Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T22:20:41.1806931Z else 2024-08-20T22:20:41.1807275Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:41.1807706Z fi 2024-08-20T22:20:41.1808006Z  2024-08-20T22:20:41.1808482Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T22:20:41.1809233Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T22:20:41.1810234Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T22:20:41.1811036Z # shellcheck disable=SC2086,SC2090 2024-08-20T22:20:41.1811498Z container_name=$(docker run \ 2024-08-20T22:20:41.1811928Z  ${GPU_FLAG:-} \ 2024-08-20T22:20:41.1812307Z  -e BUILD_ENVIRONMENT \ 2024-08-20T22:20:41.1812719Z  -e PR_NUMBER \ 2024-08-20T22:20:41.1813088Z  -e GITHUB_ACTIONS \ 2024-08-20T22:20:41.1813488Z  -e GITHUB_REPOSITORY \ 2024-08-20T22:20:41.1813903Z  -e GITHUB_WORKFLOW \ 2024-08-20T22:20:41.1814297Z  -e GITHUB_JOB \ 2024-08-20T22:20:41.1814663Z  -e GITHUB_RUN_ID \ 2024-08-20T22:20:41.1815058Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T22:20:41.1815469Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T22:20:41.1815874Z  -e JOB_ID \ 2024-08-20T22:20:41.1816219Z  -e JOB_NAME \ 2024-08-20T22:20:41.1816577Z  -e BASE_SHA \ 2024-08-20T22:20:41.1816935Z  -e BRANCH \ 2024-08-20T22:20:41.1817272Z  -e SHA1 \ 2024-08-20T22:20:41.1817635Z  -e AWS_DEFAULT_REGION \ 2024-08-20T22:20:41.1818046Z  -e IN_WHEEL_TEST \ 2024-08-20T22:20:41.1818436Z  -e SHARD_NUMBER \ 2024-08-20T22:20:41.1818810Z  -e TEST_CONFIG \ 2024-08-20T22:20:41.1819193Z  -e NUM_TEST_SHARDS \ 2024-08-20T22:20:41.1819593Z  -e REENABLED_ISSUES \ 2024-08-20T22:20:41.1820015Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T22:20:41.1820439Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T22:20:41.1820846Z  -e TEST_SHOWLOCALS \ 2024-08-20T22:20:41.1821248Z  -e NO_TEST_TIMEOUT \ 2024-08-20T22:20:41.1821631Z  -e NO_TD \ 2024-08-20T22:20:41.1821976Z  -e TD_DISTRIBUTED \ 2024-08-20T22:20:41.1822363Z  -e PR_LABELS \ 2024-08-20T22:20:41.1822777Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T22:20:41.1823233Z  -e SCCACHE_BUCKET \ 2024-08-20T22:20:41.1823637Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T22:20:41.1824062Z  -e XLA_CUDA \ 2024-08-20T22:20:41.1824474Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T22:20:41.1824981Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T22:20:41.1825509Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T22:20:41.1826029Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T22:20:41.1826505Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T22:20:41.1826960Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T22:20:41.1827410Z  -e DASHBOARD_TAG \ 2024-08-20T22:20:41.1827895Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T22:20:41.1828455Z  --security-opt seccomp=unconfined \ 2024-08-20T22:20:41.1828925Z  --cap-add=SYS_PTRACE \ 2024-08-20T22:20:41.1829322Z  --ipc=host \ 2024-08-20T22:20:41.1829692Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T22:20:41.1830097Z  --tty \ 2024-08-20T22:20:41.1830420Z  --detach \ 2024-08-20T22:20:41.1830951Z  --name="${container_name}" \ 2024-08-20T22:20:41.1831382Z  --user jenkins \ 2024-08-20T22:20:41.1831873Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T22:20:41.1832432Z  -w /var/lib/jenkins/workspace \ 2024-08-20T22:20:41.1832877Z  "${DOCKER_IMAGE}" 2024-08-20T22:20:41.1833238Z ) 2024-08-20T22:20:41.1833655Z # Propagate download.pytorch.org IP to container 2024-08-20T22:20:41.1834572Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T22:20:41.1835528Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T22:20:41.1836432Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T22:20:41.1845706Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:41.1846070Z env: 2024-08-20T22:20:41.1846364Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:41.1846924Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:41.1847533Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:41.1848039Z PR_NUMBER: 2024-08-20T22:20:41.1848383Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T22:20:41.1848810Z GITHUB_WORKFLOW: inductor 2024-08-20T22:20:41.1849176Z GITHUB_JOB: test 2024-08-20T22:20:41.1849503Z GITHUB_RUN_ID: 10479310725 2024-08-20T22:20:41.1849880Z GITHUB_RUN_NUMBER: 88082 2024-08-20T22:20:41.1850249Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T22:20:41.1850597Z JOB_ID: 29026529883 2024-08-20T22:20:41.1851307Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:41.1852083Z BRANCH: 2024-08-20T22:20:41.1852437Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:41.1852971Z BASE_SHA: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:41.1853514Z TEST_CONFIG: aot_inductor_huggingface 2024-08-20T22:20:41.1853952Z SHARD_NUMBER: 1 2024-08-20T22:20:41.1854275Z NUM_TEST_SHARDS: 1 2024-08-20T22:20:41.1854614Z REENABLED_ISSUES: 2024-08-20T22:20:41.1854960Z CONTINUE_THROUGH_ERROR: False 2024-08-20T22:20:41.1855358Z VERBOSE_TEST_LOGS: False 2024-08-20T22:20:41.1855727Z TEST_SHOWLOCALS: False 2024-08-20T22:20:41.1856082Z NO_TEST_TIMEOUT: False 2024-08-20T22:20:41.1856432Z NO_TD: False 2024-08-20T22:20:41.1856748Z TD_DISTRIBUTED: False 2024-08-20T22:20:41.1857175Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T22:20:41.1857672Z SCCACHE_S3_KEY_PREFIX: inductor 2024-08-20T22:20:41.1858068Z SHM_SIZE: 2g 2024-08-20T22:20:41.1859158Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:20:41.1860326Z XLA_CUDA: 2024-08-20T22:20:41.1860807Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:20:41.1861435Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T22:20:41.1861880Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T22:20:41.1862300Z DASHBOARD_TAG: 2024-08-20T22:20:41.1862814Z HUGGING_FACE_HUB_TOKEN: *** 2024-08-20T22:20:41.1863239Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T22:20:41.1863639Z ##[endgroup] 2024-08-20T22:20:41.1891917Z + [[ aot_inductor_huggingface == \m\u\l\t\i\g\p\u ]] 2024-08-20T22:20:41.1893011Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-08-20T22:20:41.1893870Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:41.1900141Z +++ nproc --ignore=2 2024-08-20T22:20:41.2103295Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479310725 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:20:46.9413308Z + container_name=72f7e9e6370f06a2257acc464c18d7b985c3bee0aa44f4ab35ce823e71dc1762 2024-08-20T22:20:46.9416197Z + grep download.pytorch.org /etc/hosts 2024-08-20T22:20:46.9418301Z + docker exec -i 72f7e9e6370f06a2257acc464c18d7b985c3bee0aa44f4ab35ce823e71dc1762 sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T22:20:47.0484153Z + echo DOCKER_CONTAINER_ID=72f7e9e6370f06a2257acc464c18d7b985c3bee0aa44f4ab35ce823e71dc1762 2024-08-20T22:20:47.0489000Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:20:47.0492122Z + docker exec -t 72f7e9e6370f06a2257acc464c18d7b985c3bee0aa44f4ab35ce823e71dc1762 sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T22:20:47.4470500Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T22:20:47.7713878Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T22:20:47.7717542Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T22:20:47.7719615Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T22:20:47.7721395Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T22:20:47.7724434Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T22:20:47.7728609Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1) 2024-08-20T22:20:47.7744659Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T22:20:47.7758079Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T22:20:47.7775081Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4) 2024-08-20T22:20:47.8160819Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T22:20:48.3986122Z Installing collected packages: torch 2024-08-20T22:20:58.1552884Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T22:20:58.1554613Z timm 1.0.9.dev0 requires torchvision, which is not installed. 2024-08-20T22:20:58.1555469Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T22:20:58.2372459Z + export TERM=vt100 2024-08-20T22:20:58.2372944Z + TERM=vt100 2024-08-20T22:20:58.2375008Z ++ dirname .ci/pytorch/test.sh 2024-08-20T22:20:58.2384128Z + source .ci/pytorch/common.sh 2024-08-20T22:20:58.2388118Z +++ dirname .ci/pytorch/common.sh 2024-08-20T22:20:58.2399017Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T22:20:58.2400584Z +++ declare -f -t trap_add 2024-08-20T22:20:58.2407498Z ++ set -ex 2024-08-20T22:20:58.2408073Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:20:58.2408589Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T22:20:58.2409116Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-08-20T22:20:58.2412223Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T22:20:58.2432982Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T22:20:58.2433523Z + trap_add cleanup_workspace EXIT 2024-08-20T22:20:58.2434348Z + trap_add_cmd=cleanup_workspace 2024-08-20T22:20:58.2434749Z + shift 2024-08-20T22:20:58.2435053Z + for trap_add_name in "$@" 2024-08-20T22:20:58.2441665Z +++ trap -p EXIT 2024-08-20T22:20:58.2444945Z ++ eval 'extract_trap_cmd ' 2024-08-20T22:20:58.2445338Z +++ extract_trap_cmd 2024-08-20T22:20:58.2445704Z +++ printf '%s\n' '' 2024-08-20T22:20:58.2446106Z ++ printf '%s\n' cleanup_workspace 2024-08-20T22:20:58.2448517Z + trap -- ' 2024-08-20T22:20:58.2448876Z cleanup_workspace' EXIT 2024-08-20T22:20:58.2449350Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T22:20:58.9450494Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T22:20:58.9473039Z + echo 'Environment variables:' 2024-08-20T22:20:58.9473458Z Environment variables: 2024-08-20T22:20:58.9473799Z + env 2024-08-20T22:20:58.9485129Z INSTALLED_DB=yes 2024-08-20T22:20:58.9485713Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:20:58.9486221Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:20:58.9486775Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:20:58.9487445Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:20:58.9488044Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:20:58.9488551Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:20:58.9491948Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:20:58.9492842Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:58.9493547Z HOSTNAME=72f7e9e6370f 2024-08-20T22:20:58.9494645Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_78dc5258-4b60-4db3-852d-86b7d1c98d09 2024-08-20T22:20:58.9495496Z GITHUB_ACTION=__self 2024-08-20T22:20:58.9495946Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:20:58.9500156Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:20:58.9504390Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:20:58.9504970Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:20:58.9505322Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:20:58.9505732Z TEST_CONFIG=aot_inductor_huggingface 2024-08-20T22:20:58.9506167Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:20:58.9506719Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:20:58.9507195Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:20:58.9507705Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:20:58.9508412Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:20:58.9508838Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:20:58.9509356Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:20:58.9510156Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:20:58.9510656Z GITHUB_REF_TYPE=tag 2024-08-20T22:20:58.9511033Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:20:58.9511443Z NCCL_VERSION=2.17.1-1 2024-08-20T22:20:58.9511868Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:58.9512325Z XLA_CUDA= 2024-08-20T22:20:58.9512880Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:20:58.9515211Z *** 2024-08-20T22:20:58.9515557Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:20:58.9516001Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:20:58.9516372Z GITHUB_ACTIONS=true 2024-08-20T22:20:58.9516708Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:58.9517297Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:20:58.9517959Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:20:58.9518663Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:58.9519187Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:20:58.9519751Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:58.9520526Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:20:58.9521225Z UCC_HOME=/usr 2024-08-20T22:20:58.9521566Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:20:58.9521962Z VERBOSE_TEST_LOGS=False 2024-08-20T22:20:58.9522298Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:20:58.9522760Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:20:58.9523297Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:20:58.9523795Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:20:58.9524175Z SHARD_NUMBER=1 2024-08-20T22:20:58.9524488Z GITHUB_REF_PROTECTED=false 2024-08-20T22:20:58.9524836Z HOME=/var/lib/jenkins 2024-08-20T22:20:58.9525223Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:20:58.9525691Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:20:58.9526171Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:20:58.9526655Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:20:58.9527028Z CUDA_VERSION=12.1.1 2024-08-20T22:20:58.9527462Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:20:58.9527896Z NUM_TEST_SHARDS=1 2024-08-20T22:20:58.9528202Z UCX_HOME=/usr 2024-08-20T22:20:58.9528723Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:20:58.9529792Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_78dc5258-4b60-4db3-852d-86b7d1c98d09 2024-08-20T22:20:58.9531146Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:58.9532409Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_78dc5258-4b60-4db3-852d-86b7d1c98d09 2024-08-20T22:20:58.9533517Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:20:58.9534179Z GITHUB_EVENT_NAME=push 2024-08-20T22:20:58.9534512Z DASHBOARD_TAG= 2024-08-20T22:20:58.9534821Z GITHUB_RUN_ID=10479310725 2024-08-20T22:20:58.9535290Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:20:58.9535840Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:20:58.9536831Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_78dc5258-4b60-4db3-852d-86b7d1c98d09 2024-08-20T22:20:58.9537735Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:20:58.9538152Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:20:58.9538537Z PR_NUMBER= 2024-08-20T22:20:58.9538822Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:20:58.9539165Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:20:58.9539609Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:20:58.9540073Z TERM=vt100 2024-08-20T22:20:58.9540411Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:20:58.9540830Z INSTALLED_VISION=yes 2024-08-20T22:20:58.9541151Z BRANCH= 2024-08-20T22:20:58.9541610Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:20:58.9542022Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:20:58.9542451Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:20:58.9543226Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:20:58.9544042Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:20:58.9544523Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:20:58.9545035Z REENABLED_ISSUES= 2024-08-20T22:20:58.9545337Z SHLVL=1 2024-08-20T22:20:58.9545616Z MAX_JOBS=14 2024-08-20T22:20:58.9545945Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:20:58.9546297Z NVARCH=x86_64 2024-08-20T22:20:58.9546596Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:20:58.9547041Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:58.9547576Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:20:58.9548068Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:20:58.9548747Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:20:58.9549423Z GITHUB_JOB=test 2024-08-20T22:20:58.9549840Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:20:58.9550397Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:20:58.9550905Z NO_TEST_TIMEOUT=False 2024-08-20T22:20:58.9551246Z TD_DISTRIBUTED=False 2024-08-20T22:20:58.9551643Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:20:58.9552086Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:20:58.9552509Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:20:58.9552881Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:20:58.9553238Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:20:58.9553594Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:20:58.9554612Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:20:58.9555722Z GITHUB_BASE_REF= 2024-08-20T22:20:58.9556058Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:20:58.9556438Z CI=true 2024-08-20T22:20:58.9556763Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:20:58.9557187Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:20:58.9557569Z JOB_ID=29026529883 2024-08-20T22:20:58.9557985Z INSTALLED_PROTOBUF=yes 2024-08-20T22:20:58.9558325Z GITHUB_HEAD_REF= 2024-08-20T22:20:58.9558638Z GITHUB_ACTION_REF= 2024-08-20T22:20:58.9559100Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:20:58.9559566Z TEST_SHOWLOCALS=False 2024-08-20T22:20:58.9559908Z GITHUB_WORKFLOW=inductor 2024-08-20T22:20:58.9560272Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:20:58.9561180Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_78dc5258-4b60-4db3-852d-86b7d1c98d09 2024-08-20T22:20:58.9561983Z NO_TD=False 2024-08-20T22:20:58.9562296Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:20:58.9562674Z _=/usr/bin/env 2024-08-20T22:20:58.9563158Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T22:20:58.9720958Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-20T22:20:58.9721980Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:20:58.9722977Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-20T22:20:58.9723929Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-20T22:20:58.9724558Z + BUILD_DIR=build 2024-08-20T22:20:58.9724943Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T22:20:58.9725358Z + BUILD_BIN_DIR=build/bin 2024-08-20T22:20:58.9725754Z + SHARD_NUMBER=1 2024-08-20T22:20:58.9726049Z + NUM_TEST_SHARDS=1 2024-08-20T22:20:58.9726432Z + export VALGRIND=ON 2024-08-20T22:20:58.9726753Z + VALGRIND=ON 2024-08-20T22:20:58.9727311Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-08-20T22:20:58.9727822Z + [[ 0 == \1 ]] 2024-08-20T22:20:58.9728178Z + [[ False == \1 ]] 2024-08-20T22:20:58.9728678Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-08-20T22:20:58.9729465Z ++ realpath build/custom_test_artifacts 2024-08-20T22:20:58.9741106Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T22:20:58.9742022Z + [[ -n '' ]] 2024-08-20T22:20:58.9742508Z + echo 'Environment variables' 2024-08-20T22:20:58.9742901Z Environment variables 2024-08-20T22:20:58.9743235Z + env 2024-08-20T22:20:58.9751797Z INSTALLED_DB=yes 2024-08-20T22:20:58.9752747Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:20:58.9753307Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:20:58.9753845Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:20:58.9754654Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:20:58.9755487Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:20:58.9756171Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:20:58.9756860Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:20:58.9757614Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:58.9758371Z HOSTNAME=72f7e9e6370f 2024-08-20T22:20:58.9759835Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_78dc5258-4b60-4db3-852d-86b7d1c98d09 2024-08-20T22:20:58.9760691Z GITHUB_ACTION=__self 2024-08-20T22:20:58.9761061Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:20:58.9765396Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:20:58.9769679Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:20:58.9770216Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:20:58.9770589Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:20:58.9770957Z TEST_CONFIG=aot_inductor_huggingface 2024-08-20T22:20:58.9771394Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:20:58.9771892Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:20:58.9772380Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:20:58.9772838Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:20:58.9773265Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:20:58.9773678Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:20:58.9774141Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:20:58.9774643Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:20:58.9775117Z GITHUB_REF_TYPE=tag 2024-08-20T22:20:58.9775505Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:20:58.9775911Z NCCL_VERSION=2.17.1-1 2024-08-20T22:20:58.9776335Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:58.9776806Z XLA_CUDA= 2024-08-20T22:20:58.9777328Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:20:58.9777736Z *** 2024-08-20T22:20:58.9778028Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:20:58.9778430Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:20:58.9778805Z GITHUB_ACTIONS=true 2024-08-20T22:20:58.9779142Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:58.9779657Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:20:58.9780213Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:20:58.9780706Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:58.9781278Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:20:58.9781771Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:58.9782565Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:20:58.9783293Z UCC_HOME=/usr 2024-08-20T22:20:58.9783645Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:20:58.9784225Z VERBOSE_TEST_LOGS=False 2024-08-20T22:20:58.9784588Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:20:58.9785067Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:20:58.9785570Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:20:58.9786060Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:20:58.9786455Z SHARD_NUMBER=1 2024-08-20T22:20:58.9786774Z GITHUB_REF_PROTECTED=false 2024-08-20T22:20:58.9787144Z HOME=/var/lib/jenkins 2024-08-20T22:20:58.9787523Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:20:58.9787977Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:20:58.9788464Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:20:58.9788956Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:20:58.9789340Z CUDA_VERSION=12.1.1 2024-08-20T22:20:58.9789781Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:20:58.9790266Z NUM_TEST_SHARDS=1 2024-08-20T22:20:58.9790571Z UCX_HOME=/usr 2024-08-20T22:20:58.9791098Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:20:58.9792274Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_78dc5258-4b60-4db3-852d-86b7d1c98d09 2024-08-20T22:20:58.9793571Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:58.9794838Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_78dc5258-4b60-4db3-852d-86b7d1c98d09 2024-08-20T22:20:58.9796009Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:20:58.9796679Z GITHUB_EVENT_NAME=push 2024-08-20T22:20:58.9797019Z DASHBOARD_TAG= 2024-08-20T22:20:58.9797340Z GITHUB_RUN_ID=10479310725 2024-08-20T22:20:58.9797916Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:20:58.9798475Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:20:58.9799474Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_78dc5258-4b60-4db3-852d-86b7d1c98d09 2024-08-20T22:20:58.9800404Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:20:58.9800828Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:20:58.9801218Z PR_NUMBER= 2024-08-20T22:20:58.9801510Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:20:58.9801841Z VALGRIND=ON 2024-08-20T22:20:58.9802155Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:20:58.9802605Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:20:58.9803078Z TERM=vt100 2024-08-20T22:20:58.9803437Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:20:58.9803863Z INSTALLED_VISION=yes 2024-08-20T22:20:58.9804186Z BRANCH= 2024-08-20T22:20:58.9804486Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:20:58.9804897Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:20:58.9805375Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:20:58.9806145Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:20:58.9806905Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:20:58.9807409Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:20:58.9807878Z REENABLED_ISSUES= 2024-08-20T22:20:58.9808195Z SHLVL=1 2024-08-20T22:20:58.9808472Z MAX_JOBS=14 2024-08-20T22:20:58.9808805Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:20:58.9809165Z NVARCH=x86_64 2024-08-20T22:20:58.9809477Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:20:58.9810222Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:58.9810757Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:20:58.9811258Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:20:58.9811918Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:20:58.9812493Z GITHUB_JOB=test 2024-08-20T22:20:58.9812903Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:20:58.9813588Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:20:58.9814157Z NO_TEST_TIMEOUT=False 2024-08-20T22:20:58.9814523Z TD_DISTRIBUTED=False 2024-08-20T22:20:58.9814923Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:20:58.9815600Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:20:58.9816043Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:20:58.9816419Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:20:58.9816773Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:20:58.9817135Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:20:58.9818156Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:20:58.9819209Z GITHUB_BASE_REF= 2024-08-20T22:20:58.9819546Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:20:58.9819930Z CI=true 2024-08-20T22:20:58.9820263Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:20:58.9820683Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:20:58.9821061Z JOB_ID=29026529883 2024-08-20T22:20:58.9821384Z INSTALLED_PROTOBUF=yes 2024-08-20T22:20:58.9821726Z GITHUB_HEAD_REF= 2024-08-20T22:20:58.9822033Z GITHUB_ACTION_REF= 2024-08-20T22:20:58.9822481Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:20:58.9823185Z TEST_SHOWLOCALS=False 2024-08-20T22:20:58.9823612Z GITHUB_WORKFLOW=inductor 2024-08-20T22:20:58.9823964Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:20:58.9824886Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_78dc5258-4b60-4db3-852d-86b7d1c98d09 2024-08-20T22:20:58.9825772Z NO_TD=False 2024-08-20T22:20:58.9826085Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:20:58.9826444Z _=/usr/bin/env 2024-08-20T22:20:58.9826801Z + echo 'Testing pytorch' 2024-08-20T22:20:58.9827144Z Testing pytorch 2024-08-20T22:20:58.9827467Z + export LANG=C.UTF-8 2024-08-20T22:20:58.9827812Z + LANG=C.UTF-8 2024-08-20T22:20:58.9828102Z + PR_NUMBER= 2024-08-20T22:20:58.9828456Z + [[ aot_inductor_huggingface == \d\e\f\a\u\l\t ]] 2024-08-20T22:20:58.9828995Z + [[ aot_inductor_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T22:20:58.9829513Z + [[ aot_inductor_huggingface == \s\l\o\w ]] 2024-08-20T22:20:58.9830162Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-08-20T22:20:58.9830876Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-08-20T22:20:58.9831416Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:20:58.9831893Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:20:58.9832357Z + [[ aot_inductor_huggingface == *crossref* ]] 2024-08-20T22:20:58.9832960Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:20:58.9833595Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-08-20T22:20:58.9834255Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:20:58.9834828Z + pip_install --user ninja==1.10.2 2024-08-20T22:20:58.9835373Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T22:20:59.8331271Z Collecting ninja==1.10.2 2024-08-20T22:20:59.8493866Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T22:20:59.9006484Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T22:21:00.5152616Z Installing collected packages: ninja 2024-08-20T22:21:00.5231765Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T22:21:00.5233041Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T22:21:00.5684396Z Successfully installed ninja-1.10.2 2024-08-20T22:21:00.6496552Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:00.6498642Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:00.6500054Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:00.6500581Z + install_tlparse 2024-08-20T22:21:00.6501235Z + pip_install --user tlparse==0.3.25 2024-08-20T22:21:00.6501827Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T22:21:01.0797560Z Collecting tlparse==0.3.25 2024-08-20T22:21:01.0951124Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T22:21:01.1514871Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T22:21:01.7957956Z Installing collected packages: tlparse 2024-08-20T22:21:01.8718464Z Successfully installed tlparse-0.3.25 2024-08-20T22:21:01.9500225Z ++ python -m site --user-base 2024-08-20T22:21:01.9743689Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:01.9745257Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-08-20T22:21:01.9746299Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-08-20T22:21:01.9746977Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:21:01.9747930Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-08-20T22:21:01.9749084Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-08-20T22:21:01.9749793Z + cd test 2024-08-20T22:21:01.9750346Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T22:21:03.5966206Z + [[ aot_inductor_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T22:21:03.5966882Z + [[ aot_inductor_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T22:21:03.5968867Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T22:21:03.5969663Z + [[ aot_inductor_huggingface == *pr_time_benchmarks* ]] 2024-08-20T22:21:03.5970400Z + [[ aot_inductor_huggingface == *dynamo_eager* ]] 2024-08-20T22:21:03.5970997Z + [[ aot_inductor_huggingface == *aot_eager* ]] 2024-08-20T22:21:03.5971532Z + [[ aot_inductor_huggingface == *aot_inductor* ]] 2024-08-20T22:21:03.5972302Z + DYNAMO_BENCHMARK_FLAGS+=(--export-aot-inductor) 2024-08-20T22:21:03.5972810Z + [[ aot_inductor_huggingface == *dynamic* ]] 2024-08-20T22:21:03.5973275Z + [[ aot_inductor_huggingface == *cpu* ]] 2024-08-20T22:21:03.5973777Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-20T22:21:03.6002455Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:03.6003180Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-08-20T22:21:03.6005422Z + cd test 2024-08-20T22:21:03.6005964Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T22:21:05.0594171Z PyTorch built with: 2024-08-20T22:21:05.0595319Z - GCC 9.4 2024-08-20T22:21:05.0596233Z - C++ Version: 201703 2024-08-20T22:21:05.0598589Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:05.0600023Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:05.0600708Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:05.0601240Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T22:21:05.0601735Z - NNPACK is enabled 2024-08-20T22:21:05.0602137Z - CPU capability usage: AVX2 2024-08-20T22:21:05.0602566Z - CUDA Runtime 12.1 2024-08-20T22:21:05.0603102Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-08-20T22:21:05.0603705Z - CuDNN 90.1 (built against CUDA 12.4) 2024-08-20T22:21:05.0604156Z - Magma 2.6.1 2024-08-20T22:21:05.0611418Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T22:21:05.0617604Z 2024-08-20T22:21:05.3788971Z + cd test 2024-08-20T22:21:05.3789816Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T22:21:06.7096300Z ATen/Parallel: 2024-08-20T22:21:06.7096815Z at::get_num_threads() : 8 2024-08-20T22:21:06.7097333Z at::get_num_interop_threads() : 16 2024-08-20T22:21:06.7097882Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:06.7098381Z omp_get_max_threads() : 8 2024-08-20T22:21:06.7099530Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:06.7100313Z mkl_get_max_threads() : 8 2024-08-20T22:21:06.7100943Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:06.7101579Z std::thread::hardware_concurrency() : 16 2024-08-20T22:21:06.7102004Z Environment variables: 2024-08-20T22:21:06.7102348Z OMP_NUM_THREADS : [not set] 2024-08-20T22:21:06.7102720Z MKL_NUM_THREADS : [not set] 2024-08-20T22:21:06.7103096Z ATen parallel backend: OpenMP 2024-08-20T22:21:06.7103347Z 2024-08-20T22:21:06.9828965Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:06.9829655Z + [[ aot_inductor_huggingface == *backward* ]] 2024-08-20T22:21:06.9830223Z + [[ aot_inductor_huggingface == *xla* ]] 2024-08-20T22:21:06.9830755Z + [[ aot_inductor_huggingface == *executorch* ]] 2024-08-20T22:21:06.9831284Z + [[ aot_inductor_huggingface == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T22:21:06.9831957Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:06.9832524Z + [[ aot_inductor_huggingface == distributed ]] 2024-08-20T22:21:06.9833069Z + [[ aot_inductor_huggingface == *inductor_distributed* ]] 2024-08-20T22:21:06.9833687Z + [[ aot_inductor_huggingface == *inductor-halide* ]] 2024-08-20T22:21:06.9834408Z + [[ aot_inductor_huggingface == *inductor-micro-benchmark* ]] 2024-08-20T22:21:06.9835087Z + [[ aot_inductor_huggingface == *huggingface* ]] 2024-08-20T22:21:06.9835546Z + install_torchvision 2024-08-20T22:21:06.9835877Z + local orig_preload 2024-08-20T22:21:06.9836205Z + local commit 2024-08-20T22:21:06.9836533Z ++ get_pinned_commit vision 2024-08-20T22:21:06.9836922Z ++ cat .github/ci_commit_pins/vision.txt 2024-08-20T22:21:06.9852052Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:06.9852612Z + orig_preload= 2024-08-20T22:21:06.9853023Z + '[' -n '' ']' 2024-08-20T22:21:06.9853916Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:06.9855354Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:07.3178974Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:07.3184035Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-1n0w8afj 2024-08-20T22:21:07.3214728Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-1n0w8afj 2024-08-20T22:21:08.7963784Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-08-20T22:21:08.7991109Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:10.1140285Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:10.4105013Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:12.8766898Z Preparing metadata (setup.py) ... [?25l- \ done 2024-08-20T22:21:12.8800599Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4) 2024-08-20T22:21:12.8803981Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+git40ec5f6) 2024-08-20T22:21:12.8807082Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-08-20T22:21:12.8877459Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-08-20T22:21:12.8882129Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-08-20T22:21:12.8884618Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-08-20T22:21:12.8888304Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-08-20T22:21:12.8891544Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.6.1) 2024-08-20T22:21:12.8896777Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1) 2024-08-20T22:21:12.8911470Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-08-20T22:21:12.9385993Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-08-20T22:21:12.9448483Z Building wheels for collected packages: torchvision 2024-08-20T22:22:29.9455860Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-08-20T22:22:29.9488354Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2116189 sha256=5e29e609e42938e300fc4434e1a942498f411a98c5a47ff71ff4dbfd91b625cc 2024-08-20T22:22:29.9490267Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-08-20T22:22:29.9530604Z Successfully built torchvision 2024-08-20T22:22:30.4385914Z Installing collected packages: torchvision 2024-08-20T22:22:30.8598063Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-08-20T22:22:31.0001767Z + '[' -n '' ']' 2024-08-20T22:22:31.0002223Z + id=0 2024-08-20T22:22:31.0002613Z + test_dynamo_benchmark huggingface 0 2024-08-20T22:22:31.0005108Z ++ pwd 2024-08-20T22:22:31.0008431Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:22:31.0009150Z + local suite=huggingface 2024-08-20T22:22:31.0009500Z + shift 2024-08-20T22:22:31.0009766Z + local shard_id=0 2024-08-20T22:22:31.0010386Z + shift 2024-08-20T22:22:31.0010835Z + [[ aot_inductor_huggingface == *perf_compare* ]] 2024-08-20T22:22:31.0011488Z + [[ aot_inductor_huggingface == *perf* ]] 2024-08-20T22:22:31.0015845Z + [[ aot_inductor_huggingface == *cpu* ]] 2024-08-20T22:22:31.0017805Z + [[ aot_inductor_huggingface == *aot_inductor* ]] 2024-08-20T22:22:31.0018785Z + test_single_dynamo_benchmark inference huggingface 0 --inference --bfloat16 2024-08-20T22:22:31.0019360Z ++ pwd 2024-08-20T22:22:31.0019830Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:22:31.0020499Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:22:31.0046729Z + local name=inference 2024-08-20T22:22:31.0047208Z + shift 2024-08-20T22:22:31.0047616Z + local suite=huggingface 2024-08-20T22:22:31.0048098Z + shift 2024-08-20T22:22:31.0048469Z + local shard_id=0 2024-08-20T22:22:31.0048867Z + shift 2024-08-20T22:22:31.0049229Z + partition_flags=() 2024-08-20T22:22:31.0049566Z + local partition_flags 2024-08-20T22:22:31.0050569Z + [[ -n 1 ]] 2024-08-20T22:22:31.0051071Z + [[ -n 0 ]] 2024-08-20T22:22:31.0051891Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T22:22:31.0052690Z + [[ aot_inductor_huggingface == *perf_compare* ]] 2024-08-20T22:22:31.0053846Z + [[ aot_inductor_huggingface == *perf* ]] 2024-08-20T22:22:31.0054352Z + [[ aot_inductor_huggingface == *aot_inductor* ]] 2024-08-20T22:22:31.0054893Z + [[ aot_inductor_huggingface != *cpu_aot_inductor* ]] 2024-08-20T22:22:31.0055408Z + export TORCHINDUCTOR_ABI_COMPATIBLE=1 2024-08-20T22:22:31.0055865Z + TORCHINDUCTOR_ABI_COMPATIBLE=1 2024-08-20T22:22:31.0056303Z + [[ aot_inductor_huggingface == *_avx2* ]] 2024-08-20T22:22:31.0056781Z + [[ aot_inductor_huggingface == *_avx512* ]] 2024-08-20T22:22:31.0058484Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --export-aot-inductor --device cuda --inference --bfloat16 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv 2024-08-20T22:22:34.0411725Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:22:34.0413363Z warnings.warn( 2024-08-20T22:22:34.0926679Z 2024-08-20T22:22:34.0928083Z config.json: 0% 0.00/694 [00:00at(103); 2024-08-20T22:36:45.3798400Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-08-20T22:36:45.3800549Z /tmp/tmp4g15lx_b/c535zg7kobdf7445qtnzrn5qgpdjh7vhkqvcadfv6hiqo7sejpkc/ckiqqctyodmcr6cx5vnuemxstrcnxv5b57c2mbj3by6pu37bjmyy.cpp:1711:10: warning: variable ‘L__self___fnet_pooler_dense_bias’ set but not used [-Wunused-but-set-variable] 2024-08-20T22:36:45.3802984Z 1711 | auto L__self___fnet_pooler_dense_bias = constants_->at(104); 2024-08-20T22:36:45.3803698Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-08-20T22:36:50.6063321Z pass 2024-08-20T22:36:50.6203815Z TIMING: backend_compile:0.00013 entire_frame_compile:1.32832 total_wall_time:1.32832 2024-08-20T22:36:50.6205038Z STATS: call_* op count: 232 | FakeTensorMode.__torch_dispatch__:8887 | FakeTensor.__torch_dispatch__:1557 | ProxyTorchDispatchMode.__torch_dispatch__:3016 2024-08-20T22:36:50.6206174Z Dynamo produced 1 graphs covering 232 ops with 0 graph breaks (0 unique) 2024-08-20T22:36:54.7957479Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:36:54.7959269Z warnings.warn( 2024-08-20T22:36:55.0084297Z 2024-08-20T22:36:57.1249424Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:36:57.1249923Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:36:57.1303024Z cuda eval LayoutLMForMaskedLM 2024-08-20T22:37:25.3439605Z /tmp/tmp4g15lx_b/cudqh3kfhha3bata3vkftnddzxpmvxcta4gmfsbt45dwwkvn3fle/cgokcdc745je2bsah5amguv4ta7smon4wagdagobuaqt6tq4ne5m.cpp: In member function ‘void torch::aot_inductor::AOTInductorModel::run_impl(AtenTensorOpaque**, AtenTensorOpaque**, torch::aot_inductor::DeviceStreamType, AOTIProxyExecutorHandle)’: 2024-08-20T22:37:25.3447717Z /tmp/tmp4g15lx_b/cudqh3kfhha3bata3vkftnddzxpmvxcta4gmfsbt45dwwkvn3fle/cgokcdc745je2bsah5amguv4ta7smon4wagdagobuaqt6tq4ne5m.cpp:2679:10: warning: variable ‘L__self___layoutlm_pooler_dense_weight’ set but not used [-Wunused-but-set-variable] 2024-08-20T22:37:25.3449439Z 2679 | auto L__self___layoutlm_pooler_dense_weight = constants_->at(201); 2024-08-20T22:37:25.3450041Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-08-20T22:37:25.3451731Z /tmp/tmp4g15lx_b/cudqh3kfhha3bata3vkftnddzxpmvxcta4gmfsbt45dwwkvn3fle/cgokcdc745je2bsah5amguv4ta7smon4wagdagobuaqt6tq4ne5m.cpp:2680:10: warning: variable ‘L__self___layoutlm_pooler_dense_bias’ set but not used [-Wunused-but-set-variable] 2024-08-20T22:37:25.3453384Z 2680 | auto L__self___layoutlm_pooler_dense_bias = constants_->at(202); 2024-08-20T22:37:25.3454052Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-08-20T22:37:28.9782679Z pass 2024-08-20T22:37:28.9968697Z TIMING: backend_compile:0.00017 entire_frame_compile:3.11674 total_wall_time:3.11674 2024-08-20T22:37:28.9970114Z STATS: call_* op count: 396 | FakeTensorMode.__torch_dispatch__:19319 | FakeTensor.__torch_dispatch__:3150 | ProxyTorchDispatchMode.__torch_dispatch__:7012 2024-08-20T22:37:28.9971343Z Dynamo produced 1 graphs covering 396 ops with 0 graph breaks (0 unique) 2024-08-20T22:37:33.4197683Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:37:33.4199719Z warnings.warn( 2024-08-20T22:37:33.6324250Z 2024-08-20T22:37:35.4713773Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:37:35.4714852Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:37:35.4767764Z cuda eval LayoutLMForSequenceClassification 2024-08-20T22:38:06.8167821Z pass 2024-08-20T22:38:06.8312764Z TIMING: backend_compile:0.00016 entire_frame_compile:3.07107 total_wall_time:3.07107 2024-08-20T22:38:06.8315276Z STATS: call_* op count: 394 | FakeTensorMode.__torch_dispatch__:19116 | FakeTensor.__torch_dispatch__:3084 | ProxyTorchDispatchMode.__torch_dispatch__:6918 2024-08-20T22:38:06.8316416Z Dynamo produced 1 graphs covering 394 ops with 0 graph breaks (0 unique) 2024-08-20T22:38:11.2358713Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:38:11.2360669Z warnings.warn( 2024-08-20T22:38:11.4215519Z 2024-08-20T22:38:22.0904159Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:38:22.0904632Z loading model: 0it [00:10, ?it/s] 2024-08-20T22:38:22.1022668Z cuda eval M2M100ForConditionalGeneration 2024-08-20T22:38:22.1759776Z WARNING:common:fp64 golden ref were not generated for M2M100ForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T22:39:35.7351102Z pass 2024-08-20T22:39:35.7670484Z TIMING: backend_compile:0.00033 entire_frame_compile:3.72736 total_wall_time:3.72736 2024-08-20T22:39:35.7672642Z STATS: call_* op count: 1290 | FakeTensorMode.__torch_dispatch__:49484 | FakeTensor.__torch_dispatch__:7808 | ProxyTorchDispatchMode.__torch_dispatch__:17602 2024-08-20T22:39:35.7673778Z Dynamo produced 1 graphs covering 1290 ops with 0 graph breaks (0 unique) 2024-08-20T22:39:40.9511974Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:39:40.9515269Z warnings.warn( 2024-08-20T22:39:41.3049956Z 2024-08-20T22:39:45.2977304Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:39:45.2977765Z loading model: 0it [00:03, ?it/s] 2024-08-20T22:39:45.3052973Z cuda eval MBartForCausalLM 2024-08-20T22:39:45.3336475Z WARNING:common:fp64 golden ref were not generated for MBartForCausalLM. Setting accuracy check to cosine 2024-08-20T22:40:19.4449652Z pass 2024-08-20T22:40:19.4726107Z TIMING: backend_compile:0.00018 entire_frame_compile:1.65873 total_wall_time:1.65873 2024-08-20T22:40:19.4728774Z STATS: call_* op count: 482 | FakeTensorMode.__torch_dispatch__:19046 | FakeTensor.__torch_dispatch__:3204 | ProxyTorchDispatchMode.__torch_dispatch__:6673 2024-08-20T22:40:19.4729936Z Dynamo produced 1 graphs covering 482 ops with 0 graph breaks (0 unique) 2024-08-20T22:40:23.9038186Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:40:23.9039836Z warnings.warn( 2024-08-20T22:40:24.0863303Z 2024-08-20T22:40:31.1064507Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:40:31.1067594Z loading model: 0it [00:07, ?it/s] 2024-08-20T22:40:31.1188062Z cuda eval MBartForConditionalGeneration 2024-08-20T22:40:31.2110269Z WARNING:common:fp64 golden ref were not generated for MBartForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T22:41:42.7311834Z pass 2024-08-20T22:41:42.7658586Z TIMING: backend_compile:0.00032 entire_frame_compile:3.91918 total_wall_time:3.91918 2024-08-20T22:41:42.7663749Z STATS: call_* op count: 1263 | FakeTensorMode.__torch_dispatch__:49446 | FakeTensor.__torch_dispatch__:8269 | ProxyTorchDispatchMode.__torch_dispatch__:17813 2024-08-20T22:41:42.7666048Z Dynamo produced 1 graphs covering 1263 ops with 0 graph breaks (0 unique) 2024-08-20T22:41:48.0363860Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:41:48.0365490Z warnings.warn( 2024-08-20T22:41:48.2332236Z 2024-08-20T22:41:52.2929355Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:41:52.2930097Z loading model: 0it [00:04, ?it/s] 2024-08-20T22:41:52.2985081Z cuda eval MT5ForConditionalGeneration 2024-08-20T22:42:40.7652450Z pass 2024-08-20T22:42:40.7819934Z TIMING: backend_compile:0.00031 entire_frame_compile:4.71919 total_wall_time:4.71919 2024-08-20T22:42:40.7822743Z STATS: call_* op count: 1216 | FakeTensorMode.__torch_dispatch__:34045 | FakeTensor.__torch_dispatch__:4751 | ProxyTorchDispatchMode.__torch_dispatch__:11577 2024-08-20T22:42:40.7823900Z Dynamo produced 1 graphs covering 1216 ops with 0 graph breaks (0 unique) 2024-08-20T22:42:45.4801873Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:42:45.4803590Z warnings.warn( 2024-08-20T22:42:45.7092179Z 2024-08-20T22:42:45.7131106Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2024-08-20T22:42:50.5129282Z 2024-08-20T22:42:50.5130098Z loading model: 0it [00:04, ?it/s] 2024-08-20T22:42:50.5229376Z cuda eval MegatronBertForCausalLM 2024-08-20T22:43:47.5458602Z pass 2024-08-20T22:43:47.5765822Z TIMING: backend_compile:0.00023 entire_frame_compile:5.72085 total_wall_time:5.72085 2024-08-20T22:43:47.5768634Z STATS: call_* op count: 721 | FakeTensorMode.__torch_dispatch__:35795 | FakeTensor.__torch_dispatch__:5968 | ProxyTorchDispatchMode.__torch_dispatch__:13234 2024-08-20T22:43:47.5770172Z Dynamo produced 1 graphs covering 721 ops with 0 graph breaks (0 unique) 2024-08-20T22:43:52.5094166Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:43:52.5095752Z warnings.warn( 2024-08-20T22:43:52.7095418Z 2024-08-20T22:43:57.1363520Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:43:57.1363993Z loading model: 0it [00:04, ?it/s] 2024-08-20T22:43:57.1466883Z cuda eval MegatronBertForQuestionAnswering 2024-08-20T22:44:53.8792527Z pass 2024-08-20T22:44:53.9044340Z TIMING: backend_compile:0.00024 entire_frame_compile:5.68825 total_wall_time:5.68825 2024-08-20T22:44:53.9046735Z STATS: call_* op count: 724 | FakeTensorMode.__torch_dispatch__:35770 | FakeTensor.__torch_dispatch__:5951 | ProxyTorchDispatchMode.__torch_dispatch__:13228 2024-08-20T22:44:53.9047889Z Dynamo produced 1 graphs covering 724 ops with 0 graph breaks (0 unique) 2024-08-20T22:44:58.8696823Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:44:58.8698448Z warnings.warn( 2024-08-20T22:44:59.0801825Z 2024-08-20T22:45:00.3577425Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:45:00.3578339Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:45:00.3852906Z cuda eval MobileBertForMaskedLM 2024-08-20T22:46:44.0766384Z pass 2024-08-20T22:46:44.0894399Z TIMING: backend_compile:0.00043 entire_frame_compile:14.69506 total_wall_time:14.69506 2024-08-20T22:46:44.0896973Z STATS: call_* op count: 1447 | FakeTensorMode.__torch_dispatch__:73117 | FakeTensor.__torch_dispatch__:10103 | ProxyTorchDispatchMode.__torch_dispatch__:25859 2024-08-20T22:46:44.0898129Z Dynamo produced 1 graphs covering 1447 ops with 0 graph breaks (0 unique) 2024-08-20T22:46:49.6602934Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:46:49.6604629Z warnings.warn( 2024-08-20T22:46:49.8738719Z 2024-08-20T22:46:50.9461993Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:46:50.9462443Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:46:50.9736019Z cuda eval MobileBertForQuestionAnswering 2024-08-20T22:48:31.8569095Z pass 2024-08-20T22:48:31.8657394Z TIMING: backend_compile:0.00044 entire_frame_compile:14.69716 total_wall_time:14.69716 2024-08-20T22:48:31.8660423Z STATS: call_* op count: 1451 | FakeTensorMode.__torch_dispatch__:73089 | FakeTensor.__torch_dispatch__:10092 | ProxyTorchDispatchMode.__torch_dispatch__:25871 2024-08-20T22:48:31.8661564Z Dynamo produced 1 graphs covering 1451 ops with 0 graph breaks (0 unique) 2024-08-20T22:48:37.3627166Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:48:37.3628823Z warnings.warn( 2024-08-20T22:48:37.7255907Z 2024-08-20T22:48:40.1410169Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:48:40.1410768Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:48:40.1458290Z cuda eval OPTForCausalLM 2024-08-20T22:48:40.1643661Z WARNING:common:fp64 golden ref were not generated for OPTForCausalLM. Setting accuracy check to cosine 2024-08-20T22:49:11.9119463Z pass 2024-08-20T22:49:11.9304388Z TIMING: backend_compile:0.00018 entire_frame_compile:1.62035 total_wall_time:1.62035 2024-08-20T22:49:11.9307795Z STATS: call_* op count: 534 | FakeTensorMode.__torch_dispatch__:18969 | FakeTensor.__torch_dispatch__:2974 | ProxyTorchDispatchMode.__torch_dispatch__:6574 2024-08-20T22:49:11.9308926Z Dynamo produced 1 graphs covering 534 ops with 0 graph breaks (0 unique) 2024-08-20T22:49:16.3136877Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:49:16.3138569Z warnings.warn( 2024-08-20T22:49:16.4898224Z 2024-08-20T22:49:18.3905067Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:49:18.3905584Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:49:18.3945985Z cuda eval PLBartForCausalLM 2024-08-20T22:49:18.5047382Z WARNING:common:fp64 golden ref were not generated for PLBartForCausalLM. Setting accuracy check to cosine 2024-08-20T22:49:40.5668859Z pass 2024-08-20T22:49:40.5806864Z TIMING: backend_compile:0.00013 entire_frame_compile:0.89771 total_wall_time:0.89771 2024-08-20T22:49:40.5809635Z STATS: call_* op count: 253 | FakeTensorMode.__torch_dispatch__:10005 | FakeTensor.__torch_dispatch__:1650 | ProxyTorchDispatchMode.__torch_dispatch__:3456 2024-08-20T22:49:40.5811054Z Dynamo produced 1 graphs covering 253 ops with 0 graph breaks (0 unique) 2024-08-20T22:49:44.7975012Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:49:44.7976727Z warnings.warn( 2024-08-20T22:49:44.9926535Z 2024-08-20T22:49:48.4484808Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:49:48.4485402Z loading model: 0it [00:03, ?it/s] 2024-08-20T22:49:48.4554556Z cuda eval PLBartForConditionalGeneration 2024-08-20T22:49:48.5052379Z WARNING:common:fp64 golden ref were not generated for PLBartForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T22:50:25.9761347Z pass 2024-08-20T22:50:25.9942429Z TIMING: backend_compile:0.00021 entire_frame_compile:2.11053 total_wall_time:2.11053 2024-08-20T22:50:25.9946469Z STATS: call_* op count: 656 | FakeTensorMode.__torch_dispatch__:25398 | FakeTensor.__torch_dispatch__:4209 | ProxyTorchDispatchMode.__torch_dispatch__:9116 2024-08-20T22:50:25.9947610Z Dynamo produced 1 graphs covering 656 ops with 0 graph breaks (0 unique) 2024-08-20T22:50:30.5308019Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:50:30.5310453Z warnings.warn( 2024-08-20T22:50:30.7783157Z 2024-08-20T22:50:36.9224740Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:50:36.9225225Z loading model: 0it [00:06, ?it/s] 2024-08-20T22:50:36.9299728Z cuda eval PegasusForCausalLM 2024-08-20T22:50:36.9600839Z WARNING:common:fp64 golden ref were not generated for PegasusForCausalLM. Setting accuracy check to cosine 2024-08-20T22:51:11.3086565Z pass 2024-08-20T22:51:11.3312300Z TIMING: backend_compile:0.00017 entire_frame_compile:1.642 total_wall_time:1.642 2024-08-20T22:51:11.3315474Z STATS: call_* op count: 478 | FakeTensorMode.__torch_dispatch__:18892 | FakeTensor.__torch_dispatch__:3166 | ProxyTorchDispatchMode.__torch_dispatch__:6620 2024-08-20T22:51:11.3316607Z Dynamo produced 1 graphs covering 478 ops with 0 graph breaks (0 unique) 2024-08-20T22:51:15.7967816Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:51:15.7969727Z warnings.warn( 2024-08-20T22:51:16.0292297Z 2024-08-20T22:51:26.3204024Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:51:26.3204751Z loading model: 0it [00:10, ?it/s] 2024-08-20T22:51:26.3331974Z cuda eval PegasusForConditionalGeneration 2024-08-20T22:51:26.4030575Z WARNING:common:fp64 golden ref were not generated for PegasusForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T22:52:36.5797863Z pass 2024-08-20T22:52:36.6043171Z TIMING: backend_compile:0.00035 entire_frame_compile:3.90633 total_wall_time:3.90633 2024-08-20T22:52:36.6046763Z STATS: call_* op count: 1242 | FakeTensorMode.__torch_dispatch__:48821 | FakeTensor.__torch_dispatch__:8163 | ProxyTorchDispatchMode.__torch_dispatch__:17600 2024-08-20T22:52:36.6048509Z Dynamo produced 1 graphs covering 1242 ops with 0 graph breaks (0 unique) 2024-08-20T22:52:41.9161151Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:52:41.9162796Z warnings.warn( 2024-08-20T22:52:42.1375299Z 2024-08-20T22:52:42.1404014Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2024-08-20T22:52:44.3663068Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-08-20T22:52:44.3664961Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-08-20T22:52:44.5850387Z 2024-08-20T22:52:44.5850896Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:52:44.5903025Z cuda eval RobertaForCausalLM 2024-08-20T22:53:16.2717915Z pass 2024-08-20T22:53:16.2870052Z TIMING: backend_compile:0.00016 entire_frame_compile:3.00699 total_wall_time:3.00699 2024-08-20T22:53:16.2874466Z STATS: call_* op count: 382 | FakeTensorMode.__torch_dispatch__:18733 | FakeTensor.__torch_dispatch__:3112 | ProxyTorchDispatchMode.__torch_dispatch__:6849 2024-08-20T22:53:16.2875604Z Dynamo produced 1 graphs covering 382 ops with 0 graph breaks (0 unique) 2024-08-20T22:53:20.6757827Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:53:20.6759584Z warnings.warn( 2024-08-20T22:53:20.8799804Z 2024-08-20T22:53:22.6461200Z loading model: 0it [00:00, ?it/s]We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-08-20T22:53:22.6463977Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-08-20T22:53:22.8838302Z 2024-08-20T22:53:22.8838869Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:53:22.8891006Z cuda eval RobertaForQuestionAnswering 2024-08-20T22:53:54.1430088Z pass 2024-08-20T22:53:54.1512513Z TIMING: backend_compile:0.00016 entire_frame_compile:3.03014 total_wall_time:3.03014 2024-08-20T22:53:54.1515235Z STATS: call_* op count: 384 | FakeTensorMode.__torch_dispatch__:18698 | FakeTensor.__torch_dispatch__:3095 | ProxyTorchDispatchMode.__torch_dispatch__:6837 2024-08-20T22:53:54.1516366Z Dynamo produced 1 graphs covering 384 ops with 0 graph breaks (0 unique) 2024-08-20T22:53:58.5952143Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:53:58.5953762Z warnings.warn( 2024-08-20T22:53:58.7908813Z 2024-08-20T22:53:59.4035385Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:53:59.4036116Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:53:59.4073981Z cuda eval Speech2Text2ForCausalLM 2024-08-20T22:53:59.5155316Z WARNING:common:fp64 golden ref were not generated for Speech2Text2ForCausalLM. Setting accuracy check to cosine 2024-08-20T22:54:19.0747732Z pass 2024-08-20T22:54:19.0768663Z TIMING: backend_compile:0.00014 entire_frame_compile:0.84773 total_wall_time:0.84773 2024-08-20T22:54:19.0769872Z STATS: call_* op count: 259 | FakeTensorMode.__torch_dispatch__:10000 | FakeTensor.__torch_dispatch__:1541 | ProxyTorchDispatchMode.__torch_dispatch__:3397 2024-08-20T22:54:19.0771008Z Dynamo produced 1 graphs covering 259 ops with 0 graph breaks (0 unique) 2024-08-20T22:54:23.2674198Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:54:23.2675930Z warnings.warn( 2024-08-20T22:54:23.5592037Z 2024-08-20T22:54:25.0620243Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:54:25.0620732Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:54:25.0659325Z cuda eval T5ForConditionalGeneration 2024-08-20T22:54:58.8157770Z pass 2024-08-20T22:54:58.8545152Z TIMING: backend_compile:0.00024 entire_frame_compile:3.25516 total_wall_time:3.25516 2024-08-20T22:54:58.8548697Z STATS: call_* op count: 831 | FakeTensorMode.__torch_dispatch__:23777 | FakeTensor.__torch_dispatch__:3249 | ProxyTorchDispatchMode.__torch_dispatch__:8034 2024-08-20T22:54:58.8549908Z Dynamo produced 1 graphs covering 831 ops with 0 graph breaks (0 unique) 2024-08-20T22:55:03.2970886Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:55:03.2972602Z warnings.warn( 2024-08-20T22:55:03.5048719Z 2024-08-20T22:55:04.9285251Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:55:04.9285731Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:55:04.9320728Z cuda eval T5Small 2024-08-20T22:55:26.5731890Z pass 2024-08-20T22:55:26.6119346Z TIMING: backend_compile:0.00026 entire_frame_compile:3.2512 total_wall_time:3.2512 2024-08-20T22:55:26.6122195Z STATS: call_* op count: 831 | FakeTensorMode.__torch_dispatch__:23777 | FakeTensor.__torch_dispatch__:3249 | ProxyTorchDispatchMode.__torch_dispatch__:8034 2024-08-20T22:55:26.6123637Z Dynamo produced 1 graphs covering 831 ops with 0 graph breaks (0 unique) 2024-08-20T22:55:31.0165876Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:55:31.0167601Z warnings.warn( 2024-08-20T22:55:31.2528545Z 2024-08-20T22:55:35.2488073Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:55:35.2488565Z loading model: 0it [00:03, ?it/s] 2024-08-20T22:55:35.2567833Z cuda eval TrOCRForCausalLM 2024-08-20T22:55:35.2879927Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine 2024-08-20T22:56:09.0094998Z pass 2024-08-20T22:56:09.0350529Z TIMING: backend_compile:0.00018 entire_frame_compile:1.63485 total_wall_time:1.63485 2024-08-20T22:56:09.0352353Z STATS: call_* op count: 479 | FakeTensorMode.__torch_dispatch__:18894 | FakeTensor.__torch_dispatch__:3170 | ProxyTorchDispatchMode.__torch_dispatch__:6624 2024-08-20T22:56:09.0353944Z Dynamo produced 1 graphs covering 479 ops with 0 graph breaks (0 unique) 2024-08-20T22:56:13.4083648Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:56:13.4085296Z warnings.warn( 2024-08-20T22:56:13.6029369Z 2024-08-20T22:56:23.5284785Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:56:23.5285325Z loading model: 0it [00:09, ?it/s] 2024-08-20T22:56:23.5380233Z cuda eval XGLMForCausalLM 2024-08-20T22:56:23.5879405Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine 2024-08-20T22:57:27.6682265Z pass 2024-08-20T22:57:27.7032336Z TIMING: backend_compile:0.00027 entire_frame_compile:2.96688 total_wall_time:2.96688 2024-08-20T22:57:27.7033528Z STATS: call_* op count: 991 | FakeTensorMode.__torch_dispatch__:37706 | FakeTensor.__torch_dispatch__:6280 | ProxyTorchDispatchMode.__torch_dispatch__:13308 2024-08-20T22:57:27.7034643Z Dynamo produced 1 graphs covering 991 ops with 0 graph breaks (0 unique) 2024-08-20T22:57:32.7103608Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:57:32.7105254Z warnings.warn( 2024-08-20T22:57:32.9276959Z 2024-08-20T22:57:37.4224673Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:57:37.4225265Z loading model: 0it [00:04, ?it/s] 2024-08-20T22:57:37.4318014Z cuda eval XLNetLMHeadModel 2024-08-20T22:58:02.4368980Z W0820 22:58:02.436000 10735 torch/_inductor/utils.py:1391] DeviceCopy in input program 2024-08-20T22:58:51.5075828Z pass 2024-08-20T22:58:51.5471726Z TIMING: backend_compile:0.00027 entire_frame_compile:6.20212 total_wall_time:6.20212 2024-08-20T22:58:51.5472908Z STATS: call_* op count: 816 | FakeTensorMode.__torch_dispatch__:63291 | FakeTensor.__torch_dispatch__:8021 | ProxyTorchDispatchMode.__torch_dispatch__:20588 2024-08-20T22:58:51.5474684Z Dynamo produced 1 graphs covering 816 ops with 0 graph breaks (0 unique) 2024-08-20T22:58:56.9297462Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:58:56.9299193Z warnings.warn( 2024-08-20T22:58:57.1342885Z 2024-08-20T22:58:59.3188451Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:58:59.3189341Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:58:59.3265280Z cuda eval YituTechConvBert 2024-08-20T22:59:41.4592019Z pass 2024-08-20T22:59:41.4794209Z TIMING: backend_compile:0.00023 entire_frame_compile:4.26778 total_wall_time:4.26778 2024-08-20T22:59:41.4797445Z STATS: call_* op count: 634 | FakeTensorMode.__torch_dispatch__:29257 | FakeTensor.__torch_dispatch__:4577 | ProxyTorchDispatchMode.__torch_dispatch__:9947 2024-08-20T22:59:41.4798675Z Dynamo produced 1 graphs covering 634 ops with 0 graph breaks (0 unique) 2024-08-20T22:59:43.2332532Z accuracy pass_rate=95.56% 2024-08-20T22:59:43.2334809Z calls_captured gmean=0.00x mean=612.711x 2024-08-20T22:59:43.2338988Z unique_graphs gmean=0.00x mean=0.956x 2024-08-20T22:59:43.2343346Z graph_breaks gmean=0.00x mean=0.000x 2024-08-20T22:59:43.2347448Z unique_graph_breaks gmean=0.00x mean=0.000x 2024-08-20T22:59:43.2351426Z autograd_captures gmean=0.00x mean=0.000x 2024-08-20T22:59:43.2355784Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-20T22:59:43.2360297Z cudagraph_skips gmean=0.00x mean=0.000x 2024-08-20T22:59:48.0518085Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/aot_inductor_huggingface_inference.csv 2024-08-20T22:59:48.3146157Z AlbertForMaskedLM PASS 2024-08-20T22:59:48.3150867Z AlbertForQuestionAnswering PASS 2024-08-20T22:59:48.3155221Z BartForCausalLM PASS 2024-08-20T22:59:48.3159525Z BartForConditionalGeneration PASS 2024-08-20T22:59:48.3164633Z BertForMaskedLM PASS 2024-08-20T22:59:48.3169366Z BertForQuestionAnswering PASS 2024-08-20T22:59:48.3174321Z BlenderbotForCausalLM XFAIL 2024-08-20T22:59:48.3179350Z BlenderbotSmallForCausalLM PASS 2024-08-20T22:59:48.3184478Z BlenderbotSmallForConditionalGeneration PASS 2024-08-20T22:59:48.3188894Z CamemBert PASS 2024-08-20T22:59:48.3194010Z DebertaForMaskedLM PASS 2024-08-20T22:59:48.3198688Z DebertaForQuestionAnswering PASS 2024-08-20T22:59:48.3203942Z DebertaV2ForMaskedLM XFAIL 2024-08-20T22:59:48.3208856Z DebertaV2ForQuestionAnswering PASS 2024-08-20T22:59:48.3214257Z DistilBertForMaskedLM PASS 2024-08-20T22:59:48.3219073Z DistilBertForQuestionAnswering PASS 2024-08-20T22:59:48.3223993Z DistillGPT2 PASS 2024-08-20T22:59:48.3228850Z ElectraForCausalLM PASS 2024-08-20T22:59:48.3234170Z ElectraForQuestionAnswering PASS 2024-08-20T22:59:48.3238910Z GPT2ForSequenceClassification PASS 2024-08-20T22:59:48.3243951Z GoogleFnet PASS 2024-08-20T22:59:48.3248702Z LayoutLMForMaskedLM PASS 2024-08-20T22:59:48.3253641Z LayoutLMForSequenceClassification PASS 2024-08-20T22:59:48.3258359Z M2M100ForConditionalGeneration PASS 2024-08-20T22:59:48.3263734Z MBartForCausalLM PASS 2024-08-20T22:59:48.3268150Z MBartForConditionalGeneration PASS 2024-08-20T22:59:48.3273208Z MT5ForConditionalGeneration PASS 2024-08-20T22:59:48.3277783Z MegatronBertForCausalLM PASS 2024-08-20T22:59:48.3282951Z MegatronBertForQuestionAnswering PASS 2024-08-20T22:59:48.3288110Z MobileBertForMaskedLM PASS 2024-08-20T22:59:48.3292977Z MobileBertForQuestionAnswering PASS 2024-08-20T22:59:48.3297997Z OPTForCausalLM PASS 2024-08-20T22:59:48.3302727Z PLBartForCausalLM PASS 2024-08-20T22:59:48.3307513Z PLBartForConditionalGeneration PASS 2024-08-20T22:59:48.3313389Z PegasusForCausalLM PASS 2024-08-20T22:59:48.3318112Z PegasusForConditionalGeneration PASS 2024-08-20T22:59:48.3323466Z RobertaForCausalLM PASS 2024-08-20T22:59:48.3328245Z RobertaForQuestionAnswering PASS 2024-08-20T22:59:48.3333200Z Speech2Text2ForCausalLM PASS 2024-08-20T22:59:48.3338222Z T5ForConditionalGeneration PASS 2024-08-20T22:59:48.3342878Z T5Small PASS 2024-08-20T22:59:48.3347626Z TrOCRForCausalLM PASS 2024-08-20T22:59:48.3352670Z XGLMForCausalLM PASS 2024-08-20T22:59:48.3357780Z XLNetLMHeadModel PASS 2024-08-20T22:59:48.3362732Z YituTechConvBert PASS 2024-08-20T22:59:48.3798771Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/aot_inductor_huggingface_inference.csv 2024-08-20T22:59:48.6424206Z AlbertForMaskedLM PASS 2024-08-20T22:59:48.6430444Z AlbertForQuestionAnswering PASS 2024-08-20T22:59:48.6435924Z BartForCausalLM PASS 2024-08-20T22:59:48.6441612Z BartForConditionalGeneration PASS 2024-08-20T22:59:48.6446938Z BertForMaskedLM PASS 2024-08-20T22:59:48.6452012Z BertForQuestionAnswering PASS 2024-08-20T22:59:48.6457251Z BlenderbotForCausalLM PASS 2024-08-20T22:59:48.6461816Z BlenderbotSmallForCausalLM PASS 2024-08-20T22:59:48.6466934Z BlenderbotSmallForConditionalGeneration PASS 2024-08-20T22:59:48.6471760Z CamemBert PASS 2024-08-20T22:59:48.6477015Z DebertaForMaskedLM PASS 2024-08-20T22:59:48.6482175Z DebertaForQuestionAnswering PASS 2024-08-20T22:59:48.6487532Z DebertaV2ForMaskedLM PASS 2024-08-20T22:59:48.6492763Z DebertaV2ForQuestionAnswering PASS 2024-08-20T22:59:48.6498124Z DistilBertForMaskedLM PASS 2024-08-20T22:59:48.6503235Z DistilBertForQuestionAnswering PASS 2024-08-20T22:59:48.6508404Z DistillGPT2 PASS 2024-08-20T22:59:48.6514425Z ElectraForCausalLM PASS 2024-08-20T22:59:48.6519878Z ElectraForQuestionAnswering PASS 2024-08-20T22:59:48.6525045Z GPT2ForSequenceClassification PASS 2024-08-20T22:59:48.6530268Z GoogleFnet PASS 2024-08-20T22:59:48.6535453Z LayoutLMForMaskedLM PASS 2024-08-20T22:59:48.6540740Z LayoutLMForSequenceClassification PASS 2024-08-20T22:59:48.6545833Z M2M100ForConditionalGeneration PASS 2024-08-20T22:59:48.6550928Z MBartForCausalLM PASS 2024-08-20T22:59:48.6556250Z MBartForConditionalGeneration PASS 2024-08-20T22:59:48.6561513Z MT5ForConditionalGeneration PASS 2024-08-20T22:59:48.6566689Z MegatronBertForCausalLM PASS 2024-08-20T22:59:48.6571841Z MegatronBertForQuestionAnswering PASS 2024-08-20T22:59:48.6577028Z MobileBertForMaskedLM PASS 2024-08-20T22:59:48.6582148Z MobileBertForQuestionAnswering PASS 2024-08-20T22:59:48.6587320Z OPTForCausalLM PASS 2024-08-20T22:59:48.6592660Z PLBartForCausalLM PASS 2024-08-20T22:59:48.6597597Z PLBartForConditionalGeneration PASS 2024-08-20T22:59:48.6603162Z PegasusForCausalLM PASS 2024-08-20T22:59:48.6608082Z PegasusForConditionalGeneration PASS 2024-08-20T22:59:48.6613772Z RobertaForCausalLM PASS 2024-08-20T22:59:48.6618970Z RobertaForQuestionAnswering PASS 2024-08-20T22:59:48.6624212Z Speech2Text2ForCausalLM PASS 2024-08-20T22:59:48.6629306Z T5ForConditionalGeneration PASS 2024-08-20T22:59:48.6634409Z T5Small PASS 2024-08-20T22:59:48.6639682Z TrOCRForCausalLM PASS 2024-08-20T22:59:48.6644924Z XGLMForCausalLM PASS 2024-08-20T22:59:48.6650155Z XLNetLMHeadModel PASS 2024-08-20T22:59:48.6655317Z YituTechConvBert PASS 2024-08-20T22:59:48.7088666Z + cleanup_workspace 2024-08-20T22:59:48.7091561Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-20T22:59:48.7092847Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-20T22:59:48.7094927Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-20T22:59:48.7095572Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-20T22:59:48.7096385Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-20T22:59:48.7097269Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-20T22:59:48.7097983Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-20T22:59:49.4589719Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-20T22:59:49.4590244Z cat test/**/*_toprint.log || true 2024-08-20T22:59:49.4604135Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:59:49.4604640Z env: 2024-08-20T22:59:49.4604919Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:59:49.4605375Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:59:49.4606104Z DOCKER_CONTAINER_ID: 72f7e9e6370f06a2257acc464c18d7b985c3bee0aa44f4ab35ce823e71dc1762 2024-08-20T22:59:49.4606758Z ##[endgroup] 2024-08-20T22:59:49.4697961Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-20T22:59:49.4731570Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-20T22:59:49.4732044Z kill "$MONITOR_SCRIPT_PID" 2024-08-20T22:59:49.4740595Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:59:49.4741097Z env: 2024-08-20T22:59:49.4741375Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:59:49.4741833Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:59:49.4742563Z DOCKER_CONTAINER_ID: 72f7e9e6370f06a2257acc464c18d7b985c3bee0aa44f4ab35ce823e71dc1762 2024-08-20T22:59:49.4743218Z MONITOR_SCRIPT_PID: 52492 2024-08-20T22:59:49.4743572Z ##[endgroup] 2024-08-20T22:59:49.4944615Z Prepare all required actions 2024-08-20T22:59:49.4945058Z Getting action download info 2024-08-20T22:59:49.7355160Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-20T22:59:49.9134840Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-20T22:59:49.9135312Z with: 2024-08-20T22:59:49.9135906Z file-suffix: test-aot_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529883 2024-08-20T22:59:49.9136824Z s3-bucket: gha-artifacts 2024-08-20T22:59:49.9137166Z env: 2024-08-20T22:59:49.9137433Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:59:49.9137886Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:59:49.9138615Z DOCKER_CONTAINER_ID: 72f7e9e6370f06a2257acc464c18d7b985c3bee0aa44f4ab35ce823e71dc1762 2024-08-20T22:59:49.9139254Z ##[endgroup] 2024-08-20T22:59:49.9165704Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-20T22:59:49.9166344Z # Remove any previous test jsons if they exist 2024-08-20T22:59:49.9166859Z rm -f test-jsons-*.zip 2024-08-20T22:59:49.9167377Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-20T22:59:49.9177282Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:59:49.9177792Z env: 2024-08-20T22:59:49.9178074Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:59:49.9178515Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:59:49.9179245Z DOCKER_CONTAINER_ID: 72f7e9e6370f06a2257acc464c18d7b985c3bee0aa44f4ab35ce823e71dc1762 2024-08-20T22:59:49.9180201Z FILE_SUFFIX: test-aot_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529883 2024-08-20T22:59:49.9180908Z ##[endgroup] 2024-08-20T22:59:49.9572578Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-20T22:59:49.9618037Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-20T22:59:49.9623710Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-20T22:59:49.9636702Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-20T22:59:49.9649288Z adding: test/slow_tests.json (deflated 82%) 2024-08-20T22:59:49.9707054Z ##[group]Run # Remove any previous test reports if they exist 2024-08-20T22:59:49.9707890Z # Remove any previous test reports if they exist 2024-08-20T22:59:49.9708422Z rm -f test-reports-*.zip 2024-08-20T22:59:49.9709020Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-20T22:59:49.9718581Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:59:49.9719087Z env: 2024-08-20T22:59:49.9719384Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:59:49.9719852Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:59:49.9720595Z DOCKER_CONTAINER_ID: 72f7e9e6370f06a2257acc464c18d7b985c3bee0aa44f4ab35ce823e71dc1762 2024-08-20T22:59:49.9721561Z FILE_SUFFIX: test-aot_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529883 2024-08-20T22:59:49.9722286Z ##[endgroup] 2024-08-20T22:59:49.9983223Z adding: test/test-reports/inference_huggingface.csv (deflated 77%) 2024-08-20T22:59:50.0016079Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-20T22:59:50.0016735Z # Remove any previous usage logs if they exist 2024-08-20T22:59:50.0017256Z rm -f logs-*.zip 2024-08-20T22:59:50.0017942Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-20T22:59:50.0018749Z # so check to see if the file exists first 2024-08-20T22:59:50.0019252Z if [ -f 'usage_log.txt' ]; then 2024-08-20T22:59:50.0019791Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-20T22:59:50.0020287Z fi 2024-08-20T22:59:50.0020660Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-20T22:59:50.0021237Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-20T22:59:50.0021741Z fi 2024-08-20T22:59:50.0030591Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:59:50.0031122Z env: 2024-08-20T22:59:50.0031418Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:59:50.0031894Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:59:50.0032649Z DOCKER_CONTAINER_ID: 72f7e9e6370f06a2257acc464c18d7b985c3bee0aa44f4ab35ce823e71dc1762 2024-08-20T22:59:50.0033623Z FILE_SUFFIX: test-aot_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529883 2024-08-20T22:59:50.0034345Z ##[endgroup] 2024-08-20T22:59:50.0105281Z adding: usage_log.txt (deflated 92%) 2024-08-20T22:59:50.0199582Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-20T22:59:50.0200301Z # Remove any previous debugging artifacts if they exist 2024-08-20T22:59:50.0200858Z rm -f debug-*.zip 2024-08-20T22:59:50.0201248Z if [ -d 'test/debug' ]; then 2024-08-20T22:59:50.0201751Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-20T22:59:50.0202243Z fi 2024-08-20T22:59:50.0210748Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:59:50.0211263Z env: 2024-08-20T22:59:50.0211550Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:59:50.0212013Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:59:50.0212753Z DOCKER_CONTAINER_ID: 72f7e9e6370f06a2257acc464c18d7b985c3bee0aa44f4ab35ce823e71dc1762 2024-08-20T22:59:50.0213709Z FILE_SUFFIX: test-aot_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529883 2024-08-20T22:59:50.0214413Z ##[endgroup] 2024-08-20T22:59:50.0308487Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:59:50.0308919Z with: 2024-08-20T22:59:50.0309212Z s3-bucket: gha-artifacts 2024-08-20T22:59:50.0309650Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:59:50.0310702Z retention-days: 14 2024-08-20T22:59:50.0311102Z if-no-files-found: warn 2024-08-20T22:59:50.0311458Z path: test-jsons-*.zip 2024-08-20T22:59:50.0311816Z name: artifact 2024-08-20T22:59:50.0312123Z region: us-east-1 2024-08-20T22:59:50.0312431Z env: 2024-08-20T22:59:50.0312710Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:59:50.0313157Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:59:50.0314111Z DOCKER_CONTAINER_ID: 72f7e9e6370f06a2257acc464c18d7b985c3bee0aa44f4ab35ce823e71dc1762 2024-08-20T22:59:50.0314744Z ##[endgroup] 2024-08-20T22:59:50.4108174Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:59:50.4108781Z With the provided path, there will be 1 file uploaded 2024-08-20T22:59:50.4109411Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:59:50.5589801Z Starting upload of test-jsons-test-aot_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529883.zip 2024-08-20T22:59:50.7036274Z Finished upload of test-jsons-test-aot_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529883.zip 2024-08-20T22:59:50.7322083Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:59:50.7322535Z with: 2024-08-20T22:59:50.7322814Z s3-bucket: gha-artifacts 2024-08-20T22:59:50.7323243Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:59:50.7323712Z retention-days: 14 2024-08-20T22:59:50.7324060Z if-no-files-found: error 2024-08-20T22:59:50.7324416Z path: test-reports-*.zip 2024-08-20T22:59:50.7324766Z name: artifact 2024-08-20T22:59:50.7325068Z region: us-east-1 2024-08-20T22:59:50.7325369Z env: 2024-08-20T22:59:50.7325636Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:59:50.7326084Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:59:50.7326860Z DOCKER_CONTAINER_ID: 72f7e9e6370f06a2257acc464c18d7b985c3bee0aa44f4ab35ce823e71dc1762 2024-08-20T22:59:50.7327488Z ##[endgroup] 2024-08-20T22:59:51.0928492Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:59:51.0929217Z With the provided path, there will be 1 file uploaded 2024-08-20T22:59:51.0929849Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:59:51.0986233Z Starting upload of test-reports-test-aot_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529883.zip 2024-08-20T22:59:51.1780077Z Finished upload of test-reports-test-aot_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529883.zip 2024-08-20T22:59:51.2087129Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:59:51.2087591Z with: 2024-08-20T22:59:51.2087877Z s3-bucket: gha-artifacts 2024-08-20T22:59:51.2088310Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:59:51.2088788Z retention-days: 14 2024-08-20T22:59:51.2089127Z if-no-files-found: ignore 2024-08-20T22:59:51.2089497Z path: logs-*.zip 2024-08-20T22:59:51.2089815Z name: artifact 2024-08-20T22:59:51.2090118Z region: us-east-1 2024-08-20T22:59:51.2090428Z env: 2024-08-20T22:59:51.2090719Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:59:51.2091181Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:59:51.2091929Z DOCKER_CONTAINER_ID: 72f7e9e6370f06a2257acc464c18d7b985c3bee0aa44f4ab35ce823e71dc1762 2024-08-20T22:59:51.2092580Z ##[endgroup] 2024-08-20T22:59:51.5660215Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:59:51.5661410Z With the provided path, there will be 1 file uploaded 2024-08-20T22:59:51.5662683Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:59:51.5717605Z Starting upload of logs-test-aot_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529883.zip 2024-08-20T22:59:51.7304174Z Finished upload of logs-test-aot_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529883.zip 2024-08-20T22:59:51.7594211Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:59:51.7594650Z with: 2024-08-20T22:59:51.7594935Z s3-bucket: gha-artifacts 2024-08-20T22:59:51.7595380Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:59:51.7595863Z retention-days: 14 2024-08-20T22:59:51.7596205Z if-no-files-found: ignore 2024-08-20T22:59:51.7596591Z path: debug-*.zip 2024-08-20T22:59:51.7596917Z name: artifact 2024-08-20T22:59:51.7597225Z region: us-east-1 2024-08-20T22:59:51.7597542Z env: 2024-08-20T22:59:51.7597835Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:59:51.7598419Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:59:51.7599358Z DOCKER_CONTAINER_ID: 72f7e9e6370f06a2257acc464c18d7b985c3bee0aa44f4ab35ce823e71dc1762 2024-08-20T22:59:51.7599998Z ##[endgroup] 2024-08-20T22:59:52.1085662Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-20T22:59:52.1377055Z ##[group]Run # shellcheck disable=SC2156 2024-08-20T22:59:52.1377540Z # shellcheck disable=SC2156 2024-08-20T22:59:52.1378325Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-20T22:59:52.1388235Z shell: /usr/bin/bash -e {0} 2024-08-20T22:59:52.1388595Z env: 2024-08-20T22:59:52.1388877Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:59:52.1389333Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:59:52.1390063Z DOCKER_CONTAINER_ID: 72f7e9e6370f06a2257acc464c18d7b985c3bee0aa44f4ab35ce823e71dc1762 2024-08-20T22:59:52.1390712Z ##[endgroup] 2024-08-20T22:59:52.3998995Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-20T22:59:52.3999595Z with: 2024-08-20T22:59:52.3999888Z env: 2024-08-20T22:59:52.4000191Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:59:52.4000662Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:59:52.4001418Z DOCKER_CONTAINER_ID: 72f7e9e6370f06a2257acc464c18d7b985c3bee0aa44f4ab35ce823e71dc1762 2024-08-20T22:59:52.4002075Z ##[endgroup] 2024-08-20T22:59:52.4022344Z ##[group]Run set -eou pipefail 2024-08-20T22:59:52.4022770Z set -eou pipefail 2024-08-20T22:59:52.4023139Z  2024-08-20T22:59:52.4023659Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-20T22:59:52.4024299Z for _ in $(seq 1440); do 2024-08-20T22:59:52.4024783Z  # Break if no ssh session exists anymore 2024-08-20T22:59:52.4025291Z  if [ "$(who)" = "" ]; then 2024-08-20T22:59:52.4025710Z  break 2024-08-20T22:59:52.4026065Z  fi 2024-08-20T22:59:52.4026372Z  echo "." 2024-08-20T22:59:52.4026707Z  sleep 5 2024-08-20T22:59:52.4027028Z done 2024-08-20T22:59:52.4035806Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:59:52.4036311Z env: 2024-08-20T22:59:52.4036602Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:59:52.4037068Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:59:52.4037862Z DOCKER_CONTAINER_ID: 72f7e9e6370f06a2257acc464c18d7b985c3bee0aa44f4ab35ce823e71dc1762 2024-08-20T22:59:52.4038597Z ##[endgroup] 2024-08-20T22:59:52.4068884Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-20T22:59:52.4177282Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:59:52.4178055Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:59:52.4178651Z # shellcheck disable=SC2046 2024-08-20T22:59:52.4179137Z docker stop $(docker ps -q) || true 2024-08-20T22:59:52.4179636Z # Prune all of the docker images 2024-08-20T22:59:52.4180100Z docker system prune -af 2024-08-20T22:59:52.4189877Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:59:52.4190387Z env: 2024-08-20T22:59:52.4190677Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:59:52.4191141Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:59:52.4191879Z DOCKER_CONTAINER_ID: 72f7e9e6370f06a2257acc464c18d7b985c3bee0aa44f4ab35ce823e71dc1762 2024-08-20T22:59:52.4192523Z ##[endgroup] 2024-08-20T22:59:53.0533599Z 72f7e9e6370f 2024-08-20T22:59:53.7429261Z Deleted Containers: 2024-08-20T22:59:53.7429829Z 72f7e9e6370f06a2257acc464c18d7b985c3bee0aa44f4ab35ce823e71dc1762 2024-08-20T22:59:53.7430261Z 2024-08-20T23:00:02.6467809Z Deleted Images: 2024-08-20T23:00:02.6469672Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T23:00:02.6473000Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T23:00:02.6474700Z deleted: sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb 2024-08-20T23:00:02.6475547Z deleted: sha256:c2ce10014dd98f6abbfbf2265f9780a8204c2a7a27463e61ee565573a23d141d 2024-08-20T23:00:02.6476556Z deleted: sha256:ae249c0e6e6013bd6ffea2ca0b0160c015c9ebf7c03070bc2f983a9ff2415bf5 2024-08-20T23:00:02.6477650Z deleted: sha256:8a73883c1067c0e29ff128f553518682b47ed9707d2f35e7d5138e2caf029d61 2024-08-20T23:00:02.6478926Z deleted: sha256:9c040db729fa084584aaf5573082408a2196bc64e5952270c79dc610ed4680b6 2024-08-20T23:00:02.6479801Z deleted: sha256:17311eeeb8b71b16699880c0f646e185107496c4d41da5d23370603768ec030c 2024-08-20T23:00:02.6480634Z deleted: sha256:07cc10bece8501b1c8035d4385ad9e9867dd6e92fa32218a4dcb3e42af7e0411 2024-08-20T23:00:02.6481717Z deleted: sha256:f3e32d05099d1a3115a4722bc2e94a48752c3e965446773673808252671bb967 2024-08-20T23:00:02.6482700Z deleted: sha256:be7c3f41475e3e72ffe3ac2f92ab2f4cf49d5496b2aae738942ac3bb687d742c 2024-08-20T23:00:02.6483618Z deleted: sha256:4fcac490be825129b468cf458089f1fcf8fd56afc44ef4ab27c2eff41ab1ac90 2024-08-20T23:00:02.6484495Z deleted: sha256:31c66d80a2a4792897f267d363c0cf84fb821c5d4a275ab1636f778fd0284c08 2024-08-20T23:00:02.6485307Z deleted: sha256:8a8bcb6085e7e438e5533e8a5613e2ae5888543853568d023af3e2623a1593a9 2024-08-20T23:00:02.6486125Z deleted: sha256:2f6cb60029c79d11567b738f5bdbfb9cb1838cc0f194c4a5e6ae685178d2ed34 2024-08-20T23:00:02.6486948Z deleted: sha256:e546391804d3313f558e1ac16e93cb29a417a92e4eccaf0d4bd12af3bbcf39e3 2024-08-20T23:00:02.6487758Z deleted: sha256:1623b2aa16f2bdc6b572f6799498d9a92fa8114d07c440599b17aea72e22f8b1 2024-08-20T23:00:02.6488587Z deleted: sha256:c0386ca58b7cb7afee015b7e58d2f70525eaf73d6da2cc63868b20e14b1f9497 2024-08-20T23:00:02.6489430Z deleted: sha256:dcbf534ff289186e246531925fed90f72516eb3d63663bad234d611cdc025d62 2024-08-20T23:00:02.6490260Z deleted: sha256:0bab537af579f56c9cd51c7073fd53635fffb424b363236e0de38a7dd0e60ae8 2024-08-20T23:00:02.6491082Z deleted: sha256:d85a852ae1ee027e2fc17240ed68b87957b0a6d210c9119752d8979a0dcd55cf 2024-08-20T23:00:02.6491902Z deleted: sha256:f3821d761b555362e44cf9a1447d8fd2d95b37d54aea67c7519cd9e5db0c4d7e 2024-08-20T23:00:02.6492726Z deleted: sha256:c05e84a545db5dfcb0847c73d2b0f4749fa67aabce05705632ff7f76c84d4688 2024-08-20T23:00:02.6493557Z deleted: sha256:27054eae7ca67b2bd5ca0fd3f4488bf5c5f78e95487d8c45616f3d753d48a8b5 2024-08-20T23:00:02.6494373Z deleted: sha256:9804139f6ae97796aaa993fe4b89ea0cf111eb4d30a79f4c87c339c2e1e04fa0 2024-08-20T23:00:02.6495186Z deleted: sha256:ad6ae2fc594aaa359ff4e287323836570454eb24dd162930655e414d1d77c6dd 2024-08-20T23:00:02.6496009Z deleted: sha256:9cfbb36e328af52a248853a46aab88e266c09e2b44a03e0a42194a031b9b9453 2024-08-20T23:00:02.6496851Z deleted: sha256:a8a842bd7f37b8d50b6d12d6f95f46bbba57953db466fcbde9ba522625f024ef 2024-08-20T23:00:02.6497693Z deleted: sha256:da9738654b3c86f89ebeed2560f1bae093f1219f1eff90b0394b7f211ecabd04 2024-08-20T23:00:02.6498521Z deleted: sha256:cc683bf0cb245394e0f568984b2dd4fce1b6a3a7fe7c44734bae1aa1caaa3251 2024-08-20T23:00:02.6499351Z deleted: sha256:965f220e71c7d1ec19aaf41b1529ab287beb9d86eb8482c4c14094485ab34475 2024-08-20T23:00:02.6500182Z deleted: sha256:da32ecec41a73ce7dd29464f90c7951510a18cd378753659a78cd7fabd10dc36 2024-08-20T23:00:02.6501011Z deleted: sha256:d828d187cee9ad48dbc3702fb61d359619dd8a1ed582878680b51686bf117e5a 2024-08-20T23:00:02.6501823Z deleted: sha256:712111f4b12ff8e2787597d5fdcb043bd0824e55ddd97f262ef550506ca195f0 2024-08-20T23:00:02.6502643Z deleted: sha256:a064eea405958ad14dc6c794e5df9fd1d4515424272801ec559d45a59317309a 2024-08-20T23:00:02.6503463Z deleted: sha256:e124382d136bd5e9a8b409ca79c1f1ef76458deaf444b1b2060d3fa99dbeb506 2024-08-20T23:00:02.6504328Z deleted: sha256:6e10fe757cc190a1b31b47010f4f7610e02cb1b4b438f190dcd301aa4fe2278e 2024-08-20T23:00:02.6505267Z deleted: sha256:4cea352e10f46ca4729f96ac6c4600fc53d4c36e7658c2e0f19f4f309cd4a2a7 2024-08-20T23:00:02.6506118Z deleted: sha256:518bba00fe1d0933053066d21cac0235c9fb485abbc41557ec184b293095c924 2024-08-20T23:00:02.6507196Z deleted: sha256:a64f9ca42e9f27ff2690c15a5c84312e6efaccad45668d9b496cb542165cfb5a 2024-08-20T23:00:02.6508078Z deleted: sha256:096ea0fa14a6c420658f5276b8a8f5d264c0de7e18161ceda2cfc2f4a92cd9bf 2024-08-20T23:00:02.6508903Z deleted: sha256:6f91c94ce6e3f8034c91c66ee25c66a3d85dba9d813b749bd6d2000942a99edc 2024-08-20T23:00:02.6509726Z deleted: sha256:8f8d255eeae546cead8f9a94d237fa57a99940c0f5d51bc29a9d41c7df9cffd9 2024-08-20T23:00:02.6510813Z deleted: sha256:fe05276f4a49f91eef56bfbc3da87cfff5eb116a827fa2d30efe42ce1c134b40 2024-08-20T23:00:02.6511641Z deleted: sha256:5f83cd75d0669c813d4f58978c92e14305cd8f6a07eb9a60665a583460fa7b59 2024-08-20T23:00:02.6512447Z deleted: sha256:8f7ab6366169d39d8953129f650f1927d2fb1328b3f11e2fd9076adb5f46b93a 2024-08-20T23:00:02.6513436Z deleted: sha256:4146cfd5dba0f5308954749f98596b14a1d8f6d23bd93f7ad86c6c3d72239328 2024-08-20T23:00:02.6514286Z deleted: sha256:c6d83cb9ccc606ff63bb3c4d5a5e1ff1450c619b5aae494303a94ae25971cca0 2024-08-20T23:00:02.6515163Z deleted: sha256:71e037e0a3e0963c4565534a7387c082afcb04475d3ac629acf7a0ab2f49fadc 2024-08-20T23:00:02.6515961Z deleted: sha256:4b31135c564686bdaf23262b901237880aea635655721cc7e8c95dc42b05e06d 2024-08-20T23:00:02.6516767Z deleted: sha256:d09bd5c2f35cedcb846543941ee64ce1f684b52e4285696c93b3fb2032da8579 2024-08-20T23:00:02.6517604Z deleted: sha256:4de7fb7d4fc22212aae5fc1ceb637526916d0cced240d7c93d5edd6b0ea28013 2024-08-20T23:00:02.6518518Z deleted: sha256:69890e3e82746e10aff2f1084f11afb4b95a4a6c26b30f64b7001bc0a5fe32bf 2024-08-20T23:00:02.6519331Z deleted: sha256:0156df53b3224e645336ea52b4bb816f1e8d8c9b591748cd645e5e5649fa507b 2024-08-20T23:00:02.6520144Z deleted: sha256:0664624cb9cc4a386f31bf49919a8d8a11d9fea428d05585d9e6228f754bc7cd 2024-08-20T23:00:02.6520975Z deleted: sha256:8c2ddd5fa535c768c058a23f5cc4b6c64cdd71bb7458491282aabc7f7bb94d69 2024-08-20T23:00:02.6521809Z deleted: sha256:8dd03dbe0e1c5f47dd333074dff4bd00415f2f63727a65bf17866774e2ebbd0d 2024-08-20T23:00:02.6522629Z deleted: sha256:3c56b63e094bd18fad9da7a541d537210bca3b730c590d3641213259f6908d6f 2024-08-20T23:00:02.6523443Z deleted: sha256:51484fef07bd1266d4d6ff3d767751172660fc4cc6a1ffe883523a63c985e31e 2024-08-20T23:00:02.6524246Z deleted: sha256:03e83c94023982ab7dd8dc65a6df2024e342f1b760c91ec7a680ee45824869c7 2024-08-20T23:00:02.6525088Z deleted: sha256:58440ab3e8175b5de94a76485c8689a6dc186015475ebf4b386ebba309c5da34 2024-08-20T23:00:02.6525928Z deleted: sha256:9a980f3383e8aa511f0ef68982cb6dfc53b38d1ac8c3d673cb3ba3d45158c91e 2024-08-20T23:00:02.6526761Z deleted: sha256:cb0269c9cf1c47b6c4188ca7be4db261b5d3fc4b4a5153629e222dba7b1f9f98 2024-08-20T23:00:02.6527594Z deleted: sha256:41f1ba870cdffc05e0bf8d87b640fb9d4c153d7f4f3b775f37f94e9f3d584aad 2024-08-20T23:00:02.6528425Z deleted: sha256:7b502ab352f48e8b6eb5e6824a29f308952a5ee7c7d987562e1df7e47639d7f1 2024-08-20T23:00:02.6529243Z deleted: sha256:6ba5217e18738660e6ca8b7a7ec88845e70b0fd781301520bf46c71d99df3144 2024-08-20T23:00:02.6530063Z deleted: sha256:dd5a1c233594780163726bdbbf5de00f7b6b8dca1dace8a001340714c54baedb 2024-08-20T23:00:02.6530884Z deleted: sha256:c27f0db27da73e01f67f6cb78d10f355e350d829479ba4351d82055bb29f6597 2024-08-20T23:00:02.6531705Z deleted: sha256:4e3ce3c854f35c08be3dd95e0d6e718512a885e4614291d7d8b7cd23f48b7dd8 2024-08-20T23:00:02.6532546Z deleted: sha256:c2d6fb088bbec16e60cf0ad74b016f7287c92e659ec2db01271bc309b4dc6d83 2024-08-20T23:00:02.6533387Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-08-20T23:00:02.6534208Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-08-20T23:00:02.6535022Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-08-20T23:00:02.6535844Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-08-20T23:00:02.6536678Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-08-20T23:00:02.6538582Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-08-20T23:00:02.6539413Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-08-20T23:00:02.6540254Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-08-20T23:00:02.6541106Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-08-20T23:00:02.6541947Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-08-20T23:00:02.6542764Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-20T23:00:02.6543258Z 2024-08-20T23:00:02.6543415Z Total reclaimed space: 39.92GB 2024-08-20T23:00:02.6610202Z Post job cleanup. 2024-08-20T23:00:02.6660648Z Post job cleanup. 2024-08-20T23:00:02.7625914Z [command]/usr/bin/git version 2024-08-20T23:00:02.7676008Z git version 2.40.1 2024-08-20T23:00:02.7715884Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/dd0283c6-d411-48ee-a074-9ad344cb0e35' before making global git config changes 2024-08-20T23:00:02.7717258Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T23:00:02.7721538Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T23:00:02.7770794Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T23:00:02.7816212Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T23:00:02.8195111Z Entering 'android/libs/fbjni' 2024-08-20T23:00:02.8259633Z Entering 'third_party/FP16' 2024-08-20T23:00:02.8326340Z Entering 'third_party/FXdiv' 2024-08-20T23:00:02.8392858Z Entering 'third_party/NNPACK' 2024-08-20T23:00:02.8463892Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T23:00:02.8534657Z Entering 'third_party/XNNPACK' 2024-08-20T23:00:02.8619891Z Entering 'third_party/benchmark' 2024-08-20T23:00:02.8688536Z Entering 'third_party/cpp-httplib' 2024-08-20T23:00:02.8758888Z Entering 'third_party/cpuinfo' 2024-08-20T23:00:02.8827932Z Entering 'third_party/cudnn_frontend' 2024-08-20T23:00:02.8897106Z Entering 'third_party/cutlass' 2024-08-20T23:00:02.8974232Z Entering 'third_party/eigen' 2024-08-20T23:00:02.9046734Z Entering 'third_party/fbgemm' 2024-08-20T23:00:02.9115695Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T23:00:02.9182443Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T23:00:02.9248420Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T23:00:02.9321581Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T23:00:02.9386881Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T23:00:02.9456375Z Entering 'third_party/flatbuffers' 2024-08-20T23:00:02.9528523Z Entering 'third_party/fmt' 2024-08-20T23:00:02.9595829Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T23:00:02.9664571Z Entering 'third_party/gloo' 2024-08-20T23:00:02.9732988Z Entering 'third_party/googletest' 2024-08-20T23:00:02.9803223Z Entering 'third_party/ideep' 2024-08-20T23:00:02.9869298Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T23:00:02.9943835Z Entering 'third_party/ittapi' 2024-08-20T23:00:03.0012395Z Entering 'third_party/kineto' 2024-08-20T23:00:03.0079415Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T23:00:03.0145173Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T23:00:03.0215813Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T23:00:03.0282750Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T23:00:03.0350272Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T23:00:03.0415995Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T23:00:03.0486656Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T23:00:03.0554888Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T23:00:03.0621692Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T23:00:03.0689344Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T23:00:03.0760184Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T23:00:03.0826356Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T23:00:03.0894749Z Entering 'third_party/mimalloc' 2024-08-20T23:00:03.0963863Z Entering 'third_party/nccl/nccl' 2024-08-20T23:00:03.1032700Z Entering 'third_party/nlohmann' 2024-08-20T23:00:03.1101970Z Entering 'third_party/onnx' 2024-08-20T23:00:03.1185218Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T23:00:03.1256512Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T23:00:03.1328778Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T23:00:03.1398230Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T23:00:03.1464848Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T23:00:03.1531567Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T23:00:03.1596769Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T23:00:03.1663926Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T23:00:03.1733831Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T23:00:03.1798801Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T23:00:03.1863616Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T23:00:03.1933301Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T23:00:03.2003737Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T23:00:03.2091809Z Entering 'third_party/pocketfft' 2024-08-20T23:00:03.2161656Z Entering 'third_party/protobuf' 2024-08-20T23:00:03.2232986Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T23:00:03.2299085Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T23:00:03.2369697Z Entering 'third_party/psimd' 2024-08-20T23:00:03.2440561Z Entering 'third_party/pthreadpool' 2024-08-20T23:00:03.2509170Z Entering 'third_party/pybind11' 2024-08-20T23:00:03.2579713Z Entering 'third_party/python-peachpy' 2024-08-20T23:00:03.2648271Z Entering 'third_party/sleef' 2024-08-20T23:00:03.2717107Z Entering 'third_party/tensorpipe' 2024-08-20T23:00:03.2784540Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T23:00:03.2852470Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T23:00:03.2919775Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T23:00:03.2985817Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T23:00:03.3049957Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T23:00:03.3144951Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T23:00:03.3181156Z http.https://github.com/.extraheader 2024-08-20T23:00:03.3190825Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-20T23:00:03.3237009Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T23:00:03.3602295Z Entering 'android/libs/fbjni' 2024-08-20T23:00:03.3647767Z http.https://github.com/.extraheader 2024-08-20T23:00:03.3692283Z Entering 'third_party/FP16' 2024-08-20T23:00:03.3738349Z http.https://github.com/.extraheader 2024-08-20T23:00:03.3783179Z Entering 'third_party/FXdiv' 2024-08-20T23:00:03.3828401Z http.https://github.com/.extraheader 2024-08-20T23:00:03.3872333Z Entering 'third_party/NNPACK' 2024-08-20T23:00:03.3919083Z http.https://github.com/.extraheader 2024-08-20T23:00:03.3963856Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T23:00:03.4008095Z http.https://github.com/.extraheader 2024-08-20T23:00:03.4053511Z Entering 'third_party/XNNPACK' 2024-08-20T23:00:03.4098827Z http.https://github.com/.extraheader 2024-08-20T23:00:03.4160809Z Entering 'third_party/benchmark' 2024-08-20T23:00:03.4205841Z http.https://github.com/.extraheader 2024-08-20T23:00:03.4251792Z Entering 'third_party/cpp-httplib' 2024-08-20T23:00:03.4296786Z http.https://github.com/.extraheader 2024-08-20T23:00:03.4341891Z Entering 'third_party/cpuinfo' 2024-08-20T23:00:03.4386052Z http.https://github.com/.extraheader 2024-08-20T23:00:03.4430568Z Entering 'third_party/cudnn_frontend' 2024-08-20T23:00:03.4473815Z http.https://github.com/.extraheader 2024-08-20T23:00:03.4518011Z Entering 'third_party/cutlass' 2024-08-20T23:00:03.4562019Z http.https://github.com/.extraheader 2024-08-20T23:00:03.4613560Z Entering 'third_party/eigen' 2024-08-20T23:00:03.4657451Z http.https://github.com/.extraheader 2024-08-20T23:00:03.4702665Z Entering 'third_party/fbgemm' 2024-08-20T23:00:03.4747255Z http.https://github.com/.extraheader 2024-08-20T23:00:03.4790497Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T23:00:03.4834252Z http.https://github.com/.extraheader 2024-08-20T23:00:03.4877358Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T23:00:03.4924089Z http.https://github.com/.extraheader 2024-08-20T23:00:03.4966658Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T23:00:03.5008674Z http.https://github.com/.extraheader 2024-08-20T23:00:03.5058578Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T23:00:03.5100722Z http.https://github.com/.extraheader 2024-08-20T23:00:03.5144065Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T23:00:03.5186776Z http.https://github.com/.extraheader 2024-08-20T23:00:03.5232581Z Entering 'third_party/flatbuffers' 2024-08-20T23:00:03.5275996Z http.https://github.com/.extraheader 2024-08-20T23:00:03.5322570Z Entering 'third_party/fmt' 2024-08-20T23:00:03.5366020Z http.https://github.com/.extraheader 2024-08-20T23:00:03.5411351Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T23:00:03.5455489Z http.https://github.com/.extraheader 2024-08-20T23:00:03.5499603Z Entering 'third_party/gloo' 2024-08-20T23:00:03.5544461Z http.https://github.com/.extraheader 2024-08-20T23:00:03.5587873Z Entering 'third_party/googletest' 2024-08-20T23:00:03.5632563Z http.https://github.com/.extraheader 2024-08-20T23:00:03.5677825Z Entering 'third_party/ideep' 2024-08-20T23:00:03.5723089Z http.https://github.com/.extraheader 2024-08-20T23:00:03.5765345Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T23:00:03.5807726Z http.https://github.com/.extraheader 2024-08-20T23:00:03.5859981Z Entering 'third_party/ittapi' 2024-08-20T23:00:03.5904051Z http.https://github.com/.extraheader 2024-08-20T23:00:03.5949817Z Entering 'third_party/kineto' 2024-08-20T23:00:03.5993884Z http.https://github.com/.extraheader 2024-08-20T23:00:03.6037421Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T23:00:03.6084205Z http.https://github.com/.extraheader 2024-08-20T23:00:03.6128390Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T23:00:03.6171611Z http.https://github.com/.extraheader 2024-08-20T23:00:03.6217982Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T23:00:03.6260080Z http.https://github.com/.extraheader 2024-08-20T23:00:03.6303936Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T23:00:03.6346443Z http.https://github.com/.extraheader 2024-08-20T23:00:03.6390298Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T23:00:03.6433036Z http.https://github.com/.extraheader 2024-08-20T23:00:03.6475143Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T23:00:03.6523776Z http.https://github.com/.extraheader 2024-08-20T23:00:03.6572137Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T23:00:03.6615473Z http.https://github.com/.extraheader 2024-08-20T23:00:03.6660016Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T23:00:03.6702511Z http.https://github.com/.extraheader 2024-08-20T23:00:03.6747373Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T23:00:03.6792709Z http.https://github.com/.extraheader 2024-08-20T23:00:03.6837568Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T23:00:03.6883655Z http.https://github.com/.extraheader 2024-08-20T23:00:03.6930263Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T23:00:03.6973724Z http.https://github.com/.extraheader 2024-08-20T23:00:03.7020254Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T23:00:03.7064117Z http.https://github.com/.extraheader 2024-08-20T23:00:03.7113246Z Entering 'third_party/mimalloc' 2024-08-20T23:00:03.7157035Z http.https://github.com/.extraheader 2024-08-20T23:00:03.7199707Z Entering 'third_party/nccl/nccl' 2024-08-20T23:00:03.7243711Z http.https://github.com/.extraheader 2024-08-20T23:00:03.7286420Z Entering 'third_party/nlohmann' 2024-08-20T23:00:03.7331063Z http.https://github.com/.extraheader 2024-08-20T23:00:03.7374352Z Entering 'third_party/onnx' 2024-08-20T23:00:03.7419576Z http.https://github.com/.extraheader 2024-08-20T23:00:03.7476351Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T23:00:03.7523244Z http.https://github.com/.extraheader 2024-08-20T23:00:03.7566388Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T23:00:03.7610148Z http.https://github.com/.extraheader 2024-08-20T23:00:03.7656904Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T23:00:03.7700861Z http.https://github.com/.extraheader 2024-08-20T23:00:03.7745428Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T23:00:03.7788404Z http.https://github.com/.extraheader 2024-08-20T23:00:03.7830815Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T23:00:03.7877055Z http.https://github.com/.extraheader 2024-08-20T23:00:03.7921756Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T23:00:03.7966722Z http.https://github.com/.extraheader 2024-08-20T23:00:03.8008561Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T23:00:03.8054057Z http.https://github.com/.extraheader 2024-08-20T23:00:03.8097482Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T23:00:03.8140573Z http.https://github.com/.extraheader 2024-08-20T23:00:03.8183067Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T23:00:03.8227696Z http.https://github.com/.extraheader 2024-08-20T23:00:03.8270313Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T23:00:03.8313291Z http.https://github.com/.extraheader 2024-08-20T23:00:03.8353977Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T23:00:03.8399571Z http.https://github.com/.extraheader 2024-08-20T23:00:03.8445355Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T23:00:03.8490326Z http.https://github.com/.extraheader 2024-08-20T23:00:03.8536394Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T23:00:03.8579323Z http.https://github.com/.extraheader 2024-08-20T23:00:03.8643302Z Entering 'third_party/pocketfft' 2024-08-20T23:00:03.8688490Z http.https://github.com/.extraheader 2024-08-20T23:00:03.8734023Z Entering 'third_party/protobuf' 2024-08-20T23:00:03.8779176Z http.https://github.com/.extraheader 2024-08-20T23:00:03.8826122Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T23:00:03.8869177Z http.https://github.com/.extraheader 2024-08-20T23:00:03.8913110Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T23:00:03.8956233Z http.https://github.com/.extraheader 2024-08-20T23:00:03.9005891Z Entering 'third_party/psimd' 2024-08-20T23:00:03.9050333Z http.https://github.com/.extraheader 2024-08-20T23:00:03.9093772Z Entering 'third_party/pthreadpool' 2024-08-20T23:00:03.9141988Z http.https://github.com/.extraheader 2024-08-20T23:00:03.9185387Z Entering 'third_party/pybind11' 2024-08-20T23:00:03.9229934Z http.https://github.com/.extraheader 2024-08-20T23:00:03.9273641Z Entering 'third_party/python-peachpy' 2024-08-20T23:00:03.9318518Z http.https://github.com/.extraheader 2024-08-20T23:00:03.9362387Z Entering 'third_party/sleef' 2024-08-20T23:00:03.9406525Z http.https://github.com/.extraheader 2024-08-20T23:00:03.9450968Z Entering 'third_party/tensorpipe' 2024-08-20T23:00:03.9494747Z http.https://github.com/.extraheader 2024-08-20T23:00:03.9537861Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T23:00:03.9580097Z http.https://github.com/.extraheader 2024-08-20T23:00:03.9622804Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T23:00:03.9665402Z http.https://github.com/.extraheader 2024-08-20T23:00:03.9706861Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T23:00:03.9754459Z http.https://github.com/.extraheader 2024-08-20T23:00:03.9796774Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T23:00:03.9841597Z http.https://github.com/.extraheader 2024-08-20T23:00:03.9883181Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T23:00:03.9926361Z http.https://github.com/.extraheader 2024-08-20T23:00:04.0086252Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-20T23:00:04.0106050Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-20T23:00:04.0114654Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:00:04.0115181Z ##[endgroup] 2024-08-20T23:00:10.9265896Z Cleaning up orphan processes